We Ran Niyra's Memory Against LongMemEval. Here's Everything.

<!-- AI-DRAFTED: review before publish -->

Most memory benchmarks published by AI companies are marketing dressed as science. Someone tunes a system specifically for the test, runs it a few times, reports the best number, and buries the config choices that made it look good. I did not want to do that. So we ran Niyra's production memory stack, the exact code serving users right now, against the complete LongMemEval benchmark. One shot. No tuning. Every caveat published.

One distinction does all the work in this post: retrieval recall and question-answering accuracy are different things, and if you cannot keep them separate, every memory benchmark you read is going to mislead you. I am going to tell you exactly what our numbers mean and, just as importantly, what they do not mean.

What LongMemEval actually is

LongMemEval is the hardest public benchmark for long-term conversational memory. It contains 500 questions spread across 246,738 conversation turns. For each question, the system has to find the right piece of information inside a "haystack" of roughly 490 turns. That is a lot of noise. The benchmark stresses different skills: remembering something you said in a single session, tracking a fact across many sessions, handling a preference, reasoning about time, and catching when a fact you stated earlier got updated later.

We scored recall@10 = 98.2% and recall@5 = 92.0%, with an MRR of 0.723, across all 500 questions. The full run took 64 minutes and about $1.20 in embedding spend.

Now the part that matters more than the headline number.

Turn-level, not session-level

Most memory systems publish session-level recall. That means they score whether the right session showed up, out of maybe 40 to 50 sessions. We scored at the turn level. We check whether the exact gold conversation turn showed up, out of roughly 490 candidate turns. That is about ten times finer.

This is not a small footnote. It changes what the number means. You have probably seen the strongest open-source baseline, MemPalace, quoted at 96.6% recall@5. That figure is session-level. Our 92.0% recall@5 is turn-level. These are different units of measurement. There is no honest strict ranking between them, because we are counting different things. I am not going to tell you we beat MemPalace at recall@5, because I cannot say that truthfully. What I can say is that we found the exact right turn among ten times more candidates, and I would rather show you the granularity than pretend the comparison is clean.

If I wanted a bigger headline number, I would report session-level recall too. I am not going to, because the turn-level number is the one that describes what actually happens when Niyra answers you.

The category breakdown

Here is recall@10 by category, with sample sizes:

• Knowledge-update: 100% (n=78)
• Single-session-assistant: 100% (n=56)
• Single-session-user: 98.6% (n=70)
• Multi-session: 98.5% (n=133)
• Single-session-preference: 96.7% (n=30)
• Temporal-reasoning: 96.2% (n=133)

The temporal-reasoning number comes with a disclosed config choice. We measured it without ingesting session timestamps. That handicaps temporal questions, and we still landed at 96.2%. I am telling you this because if we had fed timestamps in, the number would likely be higher, and I would rather you know the exact condition than see a flattered result.

The knowledge-update result is the one I care most about, and I will come back to it.

The four conditions that make this honest

Anyone can post a percentage. The conditions behind it are what separate a real result from a press release.

Turn-level granularity. Already covered. Roughly 10x finer than session-level recall. When we say "found the memory," we mean the specific turn, not the neighborhood it lived in.

Per-record isolation. Each question retrieves against only its own haystack, exactly as the published benchmark defines it. We did not pool everything together to make retrieval easier.

Zero benchmark-specific tuning. These are production defaults. It was the first full-scale attempt, and every one of the 500 questions was touched once. No cherry-picking a good run out of ten. No parameter sweep against the test set.

The system under test is production. Same hybrid RRF retrieval over pgvector plus lexical pools that answers real users. Not a special benchmark build. If you want the mechanics, I wrote about how Niyra's memory actually works and the layered structure behind it lives in the memory feature page.

We also ran a control arm the whole way through: a pure vector-similarity baseline over the exact same store, scored on the exact same questions. The shipped hybrid retriever never fell below it.

The line I will not let you cross

This is the honesty rule I most want you to leave with: recall is not answer accuracy.

Our numbers say the right memory was retrieved and ranked near the top. They say nothing about whether the model then generated a correct answer from it. Those are two separate stages. Retrieval can be perfect and generation can still fumble the final sentence. Generation can be brilliant and useless if retrieval handed it the wrong context.

So when you see an end-to-end question-answering accuracy figure from another system, do not put it next to our recall number and declare a winner. They measure different stages of the same pipeline. Comparing them is a category error, and I would rather lose a favorable-looking comparison than teach you to make a wrong one.

Why the knowledge-update result is the real story

We hit 100% recall@10 on knowledge-update questions. That is the category where a fact you stated earlier gets contradicted or replaced later, and the system has to surface the current version, not the stale one.

This landed alongside a fix we shipped to belief updates. The problem: when a user restated a fact in different words that quietly contradicted an earlier statement, our older approach missed it. On our internal paraphrased-contradiction test, we scored 0 out of 5. After moving to attribute-keyed supersession, we went to 5 out of 5.

Here is what that means in plain terms. Retrieval finding the changed fact is one half. The other half happens at write time. When you tell Niyra your target ship date moved from March to May, she does not just store a second fact and hope retrieval picks the right one later. She retires the March belief structurally, keyed on the attribute, and versions it rather than overwriting it. So the stale fact stops competing for attention, and the retrieval numbers reflect that. Perfect knowledge-update recall is not luck. It is the write-time fix and the read-time retrieval working together.

If you want the reasoning behind superseding facts instead of piling them up, the oversight side of the system and the way personal AI assistants are supposed to track a moving life both depend on this.

What we published

The full technical report, the exact scoring harness, the complete run log, and a runnable reproduce path are public at github.com/Avuvo-AI/niyra-memory-bench. The baseline arm reproduces with nothing but an OpenAI key, and the repo ships a small Retriever interface so you can run your own memory system under identical conditions. If you think we measured something wrong, open an issue. I would genuinely rather someone find a flaw in our method than have the number stand unexamined.

We measured ourselves the hard way, at the fine-grained unit, on production code, with every config choice disclosed. The numbers held. That is the whole point of doing it in public.

FAQ

Does 98.2% recall@10 mean Niyra answers 98.2% of questions correctly? No. It means the correct memory was retrieved in the top ten results 98.2% of the time. Whether the model then writes a correct answer is a separate stage we did not score here. Do not read recall as answer accuracy.

How does turn-level recall compare to the session-level numbers other systems publish? It is stricter. Session-level recall checks if the right session appeared among roughly 40 to 50. Turn-level checks if the exact right turn appeared among roughly 490. They are different units, so a turn-level 92.0% recall@5 and a session-level 96.6% recall@5 cannot be ranked head-to-head.

Was the system tuned for the benchmark? No. We used production defaults, ran all 500 questions once, and did not sweep parameters against the test set. It was the first full-scale attempt on the exact code serving users.

Why measure temporal reasoning without session timestamps? It was a disclosed config choice that made the test harder. We still scored 96.2% recall@10 on that category. Feeding timestamps in would likely improve it, and we wanted you to see the handicapped number.

Can I reproduce this? Yes. Everything is public at github.com/Avuvo-AI/niyra-memory-bench: the report, the harness, the run log, and a standalone runner that reproduces the baseline arm with only an OpenAI key. It also includes an interface to benchmark your own memory system under matching conditions.