Retrieval evaluation results

Concrete numbers for gmem recall on three multi-hop QA benchmarks. For the interpretation, see conclusions.md .

Setup

• Scripts: scripts/eval_retrieval.py drives the gmem mcp server over stdio; graphs come from scripts/extract_eval_graphs.py (spaCy en_core_web_sm, no LLM). Datasets come from scripts/download_eval_datasets.py.
• Datasets: HotpotQA (distractor), 2WikiMultihopQA and MuSiQue, validation split, first 100 questions each.
• Corpus: shared (default) — every paragraph of the selected questions, deduplicated, in one scope; each question ranks the whole corpus, as HippoRAG evaluates. (per-question keeps each question’s own paragraphs, where recall@10 is ~1 and nothing is discriminated.)
• Baseline model: sentence-transformers/msmarco-distilbert-cos-v5.
• Binary: release gmem built with --features cuda, GRAPHMEM_EMBEDDING_BACKEND=auto → CUDA (RTX 4060), batch 16. The 50-question run reproduces the earlier CPU batch-1 numbers exactly, so the backend changes speed, not quality.
• Metrics: mean over questions. recall@k = fraction of supporting paragraphs in the top k; MRR = 1 / rank of the first supporting paragraph.

Graphs:

• none: no entities.
• mentions: one entity per paragraph title, plus an edge when a paragraph’s text mentions another title from the same question.
• spacy: spaCy entities and subject-verb-object triples, corpus-filtered (--spacy-max-df 0.02 --spacy-min-df 2).
• oracle: the dataset’s own gold triples. These cover only the answer path, so this leaks the answer and is only a ceiling.

Modes:

• embeddings: recall with use_embeddings=true (semantic seeds + Personalized PageRank).
• fts-raw: use_embeddings=false with the raw question.
• fts-or: use_embeddings=false with the question’s words OR-joined.

MS MARCO DistilBERT: 100 questions, shared corpus

IBM Granite Embedding: 100 questions, shared corpus

Run with GRAPHMEM_EMBEDDING_MODEL=ibm-granite/granite-embedding-97m-multilingual-r2, GRAPHMEM_EMBEDDING_BACKEND=auto, and ~/.cargo/bin/gmem (CUDA-enabled) on an NVIDIA GeForce RTX 4060 Laptop GPU. The harness’s standard matrix (all graphs and modes above, 100 questions per dataset) completed in about 90 seconds.

The lexical rows are identical to the MS MARCO run. Among non-oracle graph settings, mentions leads HotpotQA and 2Wiki recall; on MuSiQue, mentions and spacy tie at recall@10, with mentions ahead at recall@5 and MRR. Oracle results leak gold reasoning paths and are only a ceiling.

all-MiniLM-L6-v2: 100 questions, shared corpus

Run at commit 0db8dcf with GRAPHMEM_EMBEDDING_MODEL=sentence-transformers/all-MiniLM-L6-v2, the same CUDA release configuration, RTX 4060, batch 16, corpus, graph extraction, and first 100 validation questions as the DistilBERT baseline above. The complete sweep (11 reembeds) took 104 s.

MiniLM is faster for this sweep but its embedding recall is lower than the MS MARCO DistilBERT baseline in every non-oracle setting. The lexical rows are unchanged because they do not use embeddings. As with the baseline, graph context substantially improves MiniLM over its none configuration.

all-MiniLM-L12-v2: 100 questions, shared corpus

Run at commit 0db8dcf with GRAPHMEM_EMBEDDING_MODEL=sentence-transformers/all-MiniLM-L12-v2, the same CUDA release configuration, RTX 4060, batch 16, corpus, graph extraction, and first 100 validation questions as the other runs. The complete sweep (11 reembeds) took 217 s.

L12 is about twice as slow as L6 and lower on HotpotQA and MuSiQue. Its 2Wiki spaCy recall@2 improves from 0.557 to 0.593, but recall@5 falls to 0.657 and recall@10 to 0.667. L6 remains the better general-purpose MiniLM choice for this retrieval workload.

msmarco-MiniLM-L6-cos-v5: 100 questions, shared corpus

Run at commit 0db8dcf with GRAPHMEM_EMBEDDING_MODEL=sentence-transformers/msmarco-MiniLM-L6-cos-v5, the same CUDA release configuration, RTX 4060, batch 16, corpus, graph extraction, and first 100 validation questions as the other runs. The complete sweep (11 reembeds) took 112 s.

This MS MARCO-tuned L6 model is close to the DistilBERT baseline while being much faster. mentions is best at HotpotQA and 2Wiki recall, while spaCy is best on MuSiQue and produces the highest 2Wiki MRR.

msmarco-MiniLM-L12-cos-v5: 100 questions, shared corpus

Run at commit 0db8dcf with GRAPHMEM_EMBEDDING_MODEL=sentence-transformers/msmarco-MiniLM-L12-cos-v5, the same CUDA release configuration, RTX 4060, batch 16, corpus, graph extraction, and first 100 validation questions as the other runs. The complete sweep (11 reembeds) took 197 s.

L12 improves selected results over L6 (HotpotQA no-graph recall@2 and MuSiQue MRR), but it is 1.8x slower with no consistent graph-assisted recall gain. L6-cos is the better default for this workload.

Best non-oracle configuration (MS MARCO DistilBERT)

Adding a graph improves every dataset at the top ranks; without one, embeddings are roughly level with BM25 (HotpotQA and 2Wiki FTS even lead at recall@10).

50 vs 100 questions

The ordering above already held at 50 questions; absolute numbers were higher because the harder tail is absent. The 50-question run is also the CPU-parity check (identical recall/MRR to the earlier CPU batch-1 run).

mentions and spacy trade blows: at 100, spacy wins MuSiQue, mentions wins HotpotQA and 2Wiki; the two are within noise of each other.

Lexical recall fix

Before, FTS5 required every whitespace term (implicit AND), so a full natural -language question matched nothing. Now a plain query ORs its terms and BM25 ranks; quoted phrases and prefix* are preserved, and only an uppercase AND / OR / NOT / NEAR switches to strict FTS5 syntax.

Because the OR-join now applies to plain queries, fts-raw is effectively a second OR baseline rather than an all-terms baseline.

Performance

gmem reembed over 100 HotpotQA paragraphs:

• CPU defaults to batch 1: padding to the longest text costs more than batching saves there. CUDA defaults to batch 16.
• The full 100-question sweep (3 datasets, all graphs/modes, 11 reembeds) took 175 s on CUDA vs 72 s for the 50-question sweep.
• The same 100-question CUDA sweep took 104 s with all-MiniLM-L6-v2.
• The same sweep took 217 s with all-MiniLM-L12-v2.
• The same sweep took 112 s with msmarco-MiniLM-L6-cos-v5.
• The same sweep took 197 s with msmarco-MiniLM-L12-cos-v5.
• All backends produce identical recall/MRR; the batching changes speed only.

Notes

• n = 100 questions per dataset; one recall point ≈ one question, so ±0.02–0.03 is noise.
• spacy extracts fewer than 1 relation per paragraph; paragraphs are linked mostly through shared entities.
• oracle confirms the ceiling: with gold triples, 2Wiki reaches recall@10 ≥ 0.965 and MuSiQue 0.885, well above the no-LLM graphs.

Pending

• More questions (or the full validation split) to tighten confidence.
• Reuse memory vectors across graph runs instead of re-embedding per graph.
• Tune [retrieval] (damping, memory_seed_weight, …) side by side.
• Improve relation extraction (broader spaCy patterns or LLM OpenIE), or a combined mentions + spacy graph.
• Measure RUSTFLAGS="-C target-cpu=native" for CPU builds.