Memory retrieval on LoCoMo, without a model

Summary

A deterministic, model-free ranker for agent memory, measured by how much of the annotated evidence it retrieves from the LoCoMo conversations. Coverage, not answer accuracy.

Share of the annotated evidence retrieved, 0 to 1

LoCoMo, 1,982 questions, top 20 results, no model calls. Coverage, not answer accuracy. Source: locomo-retrieval.json.

Key results

MeasurementResultSource
LoCoMo mean evidence coverage at 20 results (1,982 questions)0.759652locomo-retrieval.json
LoCoMo complete evidence recall at 20 results0.716448locomo-retrieval.json
Relative gain in coverage at 20 over v120.75%MEMORY.md
LoCoMo query latency, p5021.060 mslocomo-retrieval.json
Planted-fact regression, Recall@1 (1,000 records, 100 queries)1.0core-microbenchmarks.json
Planted-fact regression latency, p503.621 mscore-microbenchmarks.json

MELRA's memory ranker, melra-hybrid-v2, is deterministic and local. It makes no model, embedding or network calls. It scores records by lexical relevance (BM25-style, with document frequency per query), adds a signal for exact phrases, weighs how confident the record's provenance is and how fresh it is, keeps the first 20 results diverse, and leaves out expired and superseded records. Every search result comes back with its score breakdown. Memory stays strictly scoped, and values are redacted before they are stored.

Two additions matter for conversations. When a caller gives records an explicit episode and sequence, the strongest matches expand to the adjacent records, at most two in each direction. And when a query names a speaker recorded as metadata, that speaker's records are boosted.

What was measured

The public harness ingests every dialogue turn of the ten LoCoMo conversations as one record, then retrieves up to 100 records for each question. Four questions have no evidence IDs and are excluded. The measure is how much of the annotated evidence comes back: the mean share per question, and the share of questions whose evidence comes back complete.

Results retrievedMean evidence coverageComplete evidence recall
50.5904780.554995
200.7596520.716448
500.8416410.796670
1000.8886150.844601

Against the previous public version, melra-hybrid-v1, at the same dataset hash, ingestion unit and cutoff, coverage at 20 rose from 0.629117 to 0.759652, a relative gain of 20.75%. Median latency per question was 21.060 ms, and the 95th percentile rose by 0.086 ms.

A separate local regression plants known facts among 1,000 records and runs 100 queries. The planted fact came first every time (Recall@1 of 1.0), with a median latency of 3.621 ms. That is a component microbenchmark, not a task score.

What it is not

This is objective evidence retrieval, not the accuracy of generated answers. The choice of reader and judge can move end-to-end memory scores dramatically, which is why this release reports coverage with no reader at all. Numbers from other projects compare only when the ingestion unit, the number of results, the dataset revision and the evidence metric all match.

What we learned

  • Filter by scope before ranking. Ranking globally and filtering afterwards can starve valid results.
  • A question turn often carries the useful lexical match while the adjacent answer turn carries the evidence. Explicit episode order recovers that evidence without dataset-specific rules or model calls.
  • Speaker identity belongs in metadata, not inferred from how a value happens to be formatted.
  • Freshness and supersession must be explicit. Overwriting a fact destroys its history; returning both the old and the new fact creates contradictions.
  • Questions that aggregate need a larger evidence budget than single-fact questions.
  • Diversity should shape the head of the ranking without making deep retrieval quadratically expensive.

What comes next

Still open: the official full LongMemEval run, retrieval depth that adapts to aggregation questions, an evaluation of prompt injection and memory poisoning, and end-to-end answer accuracy with pinned reader and judge prompts.

All research