Public benchmarks¶

Measured 2026-10-05 on an Apple M4 (10 cores, macOS). Results: benchmarks/results/public_*.json. Harness: benchmarks/public/prepare.py (downloads and embeds data outside the repository), src/bin/public-bench.rs (drives HMS) and benchmarks/public/evaluate.py (scores runs and runs the comparison libraries on the same data).

The machine was not idle: load average ranged from 4 to 37 during the runs, and every result file records it. Recall and nDCG are unaffected by load. Latency and queries/second are indicative; the gaps reported below are far larger than the load effect.

Text retrieval (BEIR)¶

BEIR test splits, scored with pytrec_eval. Embeddings: sentence-transformers/all-MiniLM-L6-v2 at revision 1110a243fdf4706b3f48f1d95db1a4f5529b4d41, L2-normalized, computed by prepare.py.

Re-measured 2026-10-07 (load 70 to 119, which does not affect nDCG) after two document-API changes chosen on NFCorpus dev and SciFact train qrels only (benchmarks/results/proxies_2026-10_beir.json; the test qrels were read once, for this table): Porter (Snowball English) stemming of BM25 terms (+0.024 / +0.032 on dev/train; +0.026 / +0.016 on test) and a min-max score blend of the lexical and semantic lists in place of reciprocal rank fusion (+0.017 / +0.016 on dev/train; RRF remains available as fusion: "rrf"). Chunks stored before stemming keep matching unstemmed query terms.

• HMS's dense search matches the independent NumPy reference. This is a correctness check of the document API; it says nothing new about the embedding model.
• HMS's BM25 is now above the published BM25 on SciFact (+0.023) and 0.002 under it on NFCorpus.
• Hybrid search is the best configuration on both sets: +0.084 / +0.038 nDCG@10 over dense alone.
• The sparse vector path loses about 27% (SciFact) and 34% (NFCorpus) of nDCG@10 relative to exact cosine on the same embeddings. It is not suitable as the primary semantic retrieval path; the document API is.

Vector search (ann-benchmarks)¶

Datasets and ground truth from ann-benchmarks: nytimes-256-angular (290,000 train, 10,000 queries) and glove-100-angular (1,183,514 train, 10,000 queries). Every system searches single-threaded, one query at a time; builds use all cores. FAISS 1.15.1, hnswlib 0.8.0.

HMS's only vector-search path is the sparse one: each dense vector is encoded with from_dense (D=16384, 64 active indices) and searched exactly in the inverted index. Its query time includes encoding, because the other systems take dense input directly.

Full sweeps are in the result files.

• HMS is behind at every operating point. On glove, HNSW reaches higher recall than HMS at roughly 600 times the throughput, and exact FAISS search is faster than HMS while returning the right answer.
• The cause is the encoder, not the index: the inverted index is exact over the sparse codes, but the codes keep only part of the neighbour structure. The true top-10 is inside HMS's top-100 for 54% of results on both sets (experimental_prefilter in the result files), so even an exact re-rank of 100 candidates would not close the gap.
• Encoding dominates build time (155 s for nytimes, 265 s for glove, all cores) and is about half of query time on nytimes.

Quantized graph index (core::qgraph) vs the field¶

VGraph (core::qgraph::vertex) is a Vamana-style graph (out-degree 64 on nytimes, 32 on glove, build_ef 200, alpha 1.0, HNSW-like upper layers for the entry point) with one 8-bit LVQ code per vertex for the traversal and an 8-bit residual code for the re-rank of the best rerank pool candidates (public-bench ann-qgraph --index vertex --residual). An opt-in 4-bit traversal encoding (--vertex-bits 4 --residual-bits 8 --align-rows --reorder --id-bytes 3) costs no recall and saves 24% of the bytes on nytimes and 16.5% on glove. Parameters were chosen on held-out train vectors (--holdout 2000; the test queries are never read for a decision). benchmarks/results/qgraph_merge_verification_heldout.json (2026-10-07, one clean paired run per dataset, 11 rounds under one lock) keeps the current search path (x1.30 / x1.38 over the previous code at recall 0.90 / 0.95 on nytimes, x1.48 / x1.58 on glove, 11 of 11 rounds) and keeps the 8-bit encoding as the default: the 4-bit encoding wins x1.20 / x1.18 on nytimes but only x1.01 / x0.97 on glove (6 and 3 of 11 rounds), so it is a memory-only gain.

nytimes-256-angular, test set, Apple M4 (NEON)¶

benchmarks/results/public_qgraph_nytimes-256-angular.json (commit d985308, 2026-10-07): 10,000 test queries, every system single-threaded one query at a time, 3 repeats (median QPS, the runs are in the file), QPS at a target interpolated in log(QPS) between the Pareto-frontier points that bracket it; builds use all cores; competitors at the parameters their authors recommend (benchmarks/public/ann_extra.py). Every row ran with the 1-minute load above the gate of 3 (other builds and timings shared the machine; load 5 to 103, recorded per row), so the absolute QPS are indicative only, and the two HMS rows ran at different loads (9-28 against 7-9). Table: uv run --script benchmarks/public/qgraph_table.py nytimes-256-angular.

Read as the file reads it:

• At recall 0.90 the 8-bit VGraph is 2.2x FAISS HNSW (M 16 and 32), 3.3x hnswlib, 2.1x RaBitQ HNSW, 4.6x RaBitQ IVF and 4.0x Lucene HNSW on QPS at 0.60-0.67x of their bytes. It LOSES to SymphonyQG (the authors' library, raw-vector refinement) by 0.75x on QPS while using 0.32x of its bytes. The 4-bit VGraph row is 2.2x SymphonyQG, but it ran at load 7-9 while the 8-bit row ran at 9-28; the paired held-out ratio of 4-bit over 8-bit is x1.20, not x3. Treat the 4-bit row as unconfirmed until a rerun at equal load.
• At recall 0.95 no HMS row is bracketed: both sweeps stopped at ef 512 with recall 0.948 (the test set is harder than the held-out train vectors, where ef 384 gave 0.950). FAISS, hnswlib, Lucene and RaBitQ IVF 1+8 reach it (379, 509, 348, 129-278 and 803 QPS).
• Memory: RaBitQ IVF (1+4: 55.5 MB, 1+8: 92.6 MB) and RaBitQ HNSW (95.3 MB) are 2.4-4.1x SMALLER than the 8-bit VGraph (226 MB) and 1.8-3.1x smaller than the 4-bit one (172 MB). That is a loss on index bytes at equal recall.
• Build time: FAISS 51-93 s, SymphonyQG 17 s, RaBitQ 12-29 s, Lucene 243-851 s, VGraph 111 s for the full nytimes graph (measured once at load 47 while building the checked cache; the paired build numbers are in qgraph_vertex_build_heldout.json). Build time is the one metric where losing is accepted; it is still a loss against SymphonyQG and RaBitQ here.
• Not run on this machine: NGT (its quantized graph needs the qbg command, which the macOS Homebrew build lacks; the ONNG stand-in did not build within its 30-minute cap), Glass, DiskANN and ScaNN (x86-only builds). They need an x86 machine.
• glove-100-angular was not run in this pass.

The earlier edge-code index (1,024-bit RaBitQ codes per edge, 2.97 GB on nytimes, 7,297 / 1,427 QPS at load below 3) is superseded by VGraph; its sweeps stay in benchmarks/results/public_qgraph_edge_<set>.json. The competitors are timed through a Python per-query loop (Lucene through a Java loop) and HMS through a native loop; that per-query overhead was not measured.

Stop rule and 1-bit screen (held-out, 2026-10-07)¶

benchmarks/results/qgraph_stop_screen_heldout.json (held-out train vectors, test set unread). VSearchParams::patience ends a search after that many expansions without a change to the 10 best estimates: 17-29% fewer estimates at equal recall on both sets, and in paired rounds (load 11-185, gate unmet, ratios only) x1.33 (4 of 5 rounds) on nytimes and x1.42 (4 of 4) on glove at recall 0.95, neutral at 0.90; off by default, --patience in the bench. The opt-in 1-bit screen before the 8-bit estimate skips 76% of the estimates on nytimes at equal recall but is slower in nearly every round (x0.65-0.87): its scalar popcount costs as much as the SDOT estimate; recorded as a negative result, off by default.

Agent long-term memory (LongMemEval_S)¶

Retrieval only, no LLM. Data: LongMemEval_S from the xiaowu0162/longmemeval-cleaned release (revision 98d7416c24c778c2fee6e6f3006e7a073259d48f, longmemeval_s_cleaned.json, sha256 d6f21ea9d60a0d56f34a05b609c79c88a451d2ae03597821ea3d5a9678c3a442), the release the LongMemEval repository now points to (September 2025 clean-up of the history sessions); the paper's numbers predate it. Embeddings: the same all-MiniLM-L6-v2 revision as above, L2-normalized. Metrics come from the repository's own src/retrieval/eval_utils.py (commit 9e0b455f, hash recorded in the result file), applied the way run_retrieval.py and print_retrieval_metrics.py do: items are user turns only, a session is the concatenation of its user turns, and the 30 abstention questions plus questions with no answer-bearing user turn are excluded, leaving 419 of 500 questions (only 5 of the 56 single-session-assistant questions remain, so that row is not informative). Each question is searched in a fresh store holding only its own haystack (about 48 sessions, 245 user turns). Recall is recall_all (every evidence item in the top k) and nDCG is ndcg_any, the two figures the official script prints; recall_any is in the result file. "Round" is the official user-turn granularity. Turn-level rows score the round run against the labelled turns; session-level rows of a round run use the official turn-to-session conversion. Results: benchmarks/results/public_longmemeval_s.json.

HMS dense reproduces an independent NumPy exact-cosine ranking on the same embeddings to every digit (the result file carries both).

On this data and model, hybrid is above dense on every overall metric at every granularity and level (nDCG@10: +0.089 round/turn, +0.076 round/session, +0.041 session). The runs are single deterministic passes with no confidence intervals or significance test. Caveats that bound the comparison: MiniLM truncates input at 256 tokens, so a session embedding sees only the start of the concatenated user turns (the lexical index sees all of it); HMS BM25 is the engine's own, not the rank_bm25 used by the official flat-bm25; zero-score items follow scored items in haystack order in the lexical run.

Recall@10 and nDCG@10 per question type (recall_all / ndcg_any):

The session-run column is hybrid. The weakest types are multi-session (evidence spread over several sessions, all of which must be retrieved), single-session-preference, and temporal-reasoning. Preference is the one type where hybrid is below dense at round granularity (30 questions; lexical matching works against implicit preference questions). Temporal questions are where round-level hybrid helps most (turn Recall@10 0.614 to 0.787), but they are still weaker than the other types, and HMS has no date-aware filtering that would use the question date.

Published numbers (Wu et al., ICLR 2025, Table 3), LongMemEval_M with Stella V5 1.5B:

These are not comparable with the table above: M has about 500 sessions per history against about 48 in S, the embedding model is a 1.5B-parameter model against a 22M-parameter one, the dataset release differs, and the paper's recall variant and level mapping were not re-derived here. No ranking against the paper is implied.

What this means¶

• For semantic search over text, use the document API: exact cosine reproduces the embedding model's quality, and hybrid search with the built-in BM25 improves on it.
• Do not use the sparse vector path (memorize_vector / query_vector) where nearest-neighbour recall matters. Improving it needs a different encoder; making the current one faster does not.
• As a vector index, VGraph is ahead of FAISS HNSW, hnswlib and Lucene HNSW on single-thread QPS at recall 0.90 and below them on bytes on nytimes (Apple M4, load above the gate: indicative), and behind SymphonyQG on QPS and behind RaBitQ on bytes. No x86 number exists yet, and the distinctive parts of HMS (the compositional VSA algebra, structured memory and provenance) are not what these benchmarks measure.

Reproduce¶

uv run --script benchmarks/public/prepare.py beir scifact
uv run --script benchmarks/public/prepare.py ann glove-100-angular
cargo build --release --bin public-bench
./target/release/public-bench beir --data ~/.cache/hms-bench/scifact --dim 16384 --out scifact.json
uv run --script benchmarks/public/evaluate.py beir scifact scifact.json
./target/release/public-bench ann --data ~/.cache/hms-bench/glove-100-angular --dim 16384 --out glove.json
uv run --script benchmarks/public/evaluate.py ann glove-100-angular glove.json
uv run --script benchmarks/public/prepare.py longmemeval s
./target/release/public-bench longmemeval --data ~/.cache/hms-bench/longmemeval_s --out lme.json
uv run --script benchmarks/public/evaluate.py longmemeval s lme.json

On-device model stages (local-models), Apple M4, 32 GB¶

Measured 2026-10-05 with public-bench lme-model --stage device (Metal; encoders f32, LLM bf16; Qwen3-Embedding-0.6B, Qwen3-Reranker-0.6B, Qwen3-4B-Instruct-2507 at the pinned revisions), every stage on, through the document API. The first 10 LongMemEval_S sessions (mean 127 user words) were ingested one session per document; 10 questions were searched with k = 10. Small sample: these are data points, not a distribution.

Ingest time is dominated by the LLM's output length (sessions with many facts take tens of seconds); query latency by the rewrite generation. The CPU path was checked for correctness, not timed end to end. Embedder and re-ranker parity with the Python stages on 48 turns / 16 questions / 48 pairs: cosine >= 0.9999997, p(yes) max |diff| 8.4e-8 (CPU) and 7.4e-8 (Metal).

Batched model stages on Metal¶

benchmarks/results/local_models_batching.json. HMS now vendors the Qwen3 forward pass so that Metal and CUDA can batch: the decoder runs continuous batching with left-padded rows, the encoders run right-padded batches. Inputs: the first 64 distinct LongMemEval_S haystack sessions (prompts::fact_prompt, greedy, up to 1536 new tokens), 96 user turns, 96 questions and 96 (question, turn) pairs. Timing is paired (arms alternating, 2 rounds, 1-minute load average below 8 before every run). Throughput excludes model load.

On the full 64 sessions, untimed: batch 1 took 829 s (7,851 tokens) and batch 8 434 s (7,812 tokens), peak memory footprint 9.6 GB and 19.2 GB.

Defaults on Metal and CUDA: the decoder stays at batch 1 because batch 8 changed 12.5% of fact outputs, which is above the 5% bar for making it the default; set --batch (or Generator::set_batch) to trade that for about 2x throughput. The embedder defaults to batch 4 and the re-ranker to batch 1. CUDA was not built or run for this measurement, so the CUDA defaults remain unverified with batching.