Shared Brain Memory Benchmarks

These benchmarks measure how well HivemindOS’s shared brain recalls the right memory, how fast it does it, and how much less text it sends to models. Latest results: September 28, 2026.

Results at a glance

Top-1 means the expected memory ranked first. Top-3 means it was in the first three results. MRR (mean reciprocal rank) rewards ranking the correct memory as high as possible.

LongMemEval Oracle Against Sibyl-Memory

Sibyl-Memory publishes a 95.6% score on LongMemEval Oracle: 500 questions, each given the past conversations that hold its answer. We ran the same 500 questions through the Superbrain the way an agent asks it (hive-brain answer), with Claude reading the answer from what the Superbrain returned.

• Questions: all 500 of longmemeval_oracle.json from the cleaned LongMemEval release. Each question’s conversations go into a fresh, empty Superbrain, and the question is asked with its own date as today.
• Readers: Opus 4.6 (the model behind Sibyl-Memory’s headline) and Sonnet 4.5 (the model of its plugin run). LongMemEval’s official reading prompt, unchanged; temperature 0; no extended thinking.
• Judge: LongMemEval’s official per-type judge prompts with gpt-4o-2024-08-06, for every answer, ours and Sibyl-Memory’s.
• Sibyl-Memory: its published answer files from Sibyl-Labs/memory-bench-kit, graded by the same judge. We did not re-run its system.
• Headline: excludes the 30 preference questions, as Sibyl-Memory’s does. Both figures are shown.

What this shows:

• With the same reading model, HivemindOS scores higher. With Opus 4.6, on the same 470 questions, it answered 16 that Sibyl-Memory missed, and Sibyl-Memory answered 10 that it missed (exact McNemar test, p = 0.33). With Sonnet 4.5 the counts were 21 and 10 (p = 0.07).
• Ahead, but not proven beyond chance. Six questions out of 470 is within what a second run of either system could change. Updated facts and dates carry most of the lead; Sibyl-Memory’s Opus run counts across conversations slightly better.
• What moved the score. The first full HivemindOS run scored 91.1% with Sonnet 4.5. Four changes took it to 95.1%: finding conversations by the days a question names (“two weeks ago”) even when no word matches; labeling each session with how long before today it was; handing over whole conversations when they fit; and no longer repeating an excerpt above its full text.

Limitations. The changes were found by studying misses on these same 500 questions, with no separate held-out set (see the LoCoMo held-out test). They are general product behavior, not per-question rules, and shipped for every user. There was one run per configuration. Every full run is in the receipt: Sonnet 4.5 first version 91.1%, reported 95.1%, with the optional conversation ledgers 94.5%; Opus 4.6 reported 96.6%, with ledgers 96.0%. Sibyl-Memory’s own 95.6% comes from a string-matching scorer that accepts almost any step-by-step answer containing the right number; graded by the official judge, its Opus run scores 95.3%. Oracle gives each question only the conversations that hold its answer, so it tests reading and recall, not search through a large history.

The published receipt holds the exact text the Superbrain gave the reading model for every question, every answer, and every verdict. Two free checks need only the public dataset: one confirms every piece of conversation text the model saw comes word for word from that question’s own history, and one recomputes every number above. Scripts to re-grade or re-read on your own API key are included.

LongMemEval_S Against Sibyl-Memory’s Architecture

LongMemEval_S asks the same 500 questions but buries each answer among about 50 conversations (about 490,000 characters), which is the real job of a memory system. Sibyl-Memory has published no S result, so we ran its published architecture with its public runner: Claude Code reads the question’s whole chronological journal under Sibyl-Memory’s instructions. Its plugin product was not run, because the integration behind it is shared only with beta testers.

• Both sides: Claude Opus 4.6 reading; LongMemEval’s official judge prompts with gpt-4o-2024-08-06.
• Sibyl-Memory: its runner unchanged except the per-question time limit (120 s to 600 s, because S journals are about ten times longer than Oracle’s), run with an empty home directory so no local settings reached it. The plan was published before it answered anything.
• HivemindOS: hive-brain answer as it ships, without model ordering or embeddings.

HivemindOS answered 26 questions Sibyl-Memory missed, and Sibyl-Memory 16 that HivemindOS missed (p = 0.16): higher, not established beyond chance, while giving the model about a ninth of the text. Deviations are disclosed with the receipt: two copies of the run overlapped and answered 200 questions twice (the first answer counts), and requests refused for lack of API credit were re-sent.

LongMemEval_S search accuracy

Answer accuracy depends on the model that reads. This measures the memory alone: for each of the 500 LongMemEval_S questions, does the brain’s recall put the conversations that hold the answer in its first five results? Each question’s ~50 conversations go into a fresh, empty brain through the same archive the app writes chats to, and the question is asked through the brain’s own full-vault recall.

• Metric: sibyl.ist’s open-source LongMemEval_S scorer (hyperb1iss/sibyl, sibyl_core/evals/longmemeval.py), ported line for line: hit@5 (at least one answer conversation in the top 5), strict recall@5 (the share of a question’s answer conversations in the top 5; a question with two scores 0.5 when only one is found) and nDCG@5 (how high they rank). All 500 questions, abstention questions included.
• Before: keyword search with the brain’s weighted score, as shipped until September 28.
• Meaning search: every note and conversation has an embedding; recall now also ranks every conversation by meaning similarity and fuses that ranking with its own by position (reciprocal rank fusion, k = 60). Conversations are embedded when a chat goes quiet. No model calls at search time.
• Reranker: the top 20 are read again, question and note together, by a reranking model, and its order is used. It sends the question and note text to an outside service, so it runs only when asked (hive-brain recall --rerank, and the phone’s memory search through a linked desktop) and only under the same rule as the brain’s other outside calls. If it does not answer, the decision model orders the top 20 instead; if neither does, recall’s own order stands.

Recall@5 by question type, before to after: counting across chats (133) 87.64 to 97.93; dates and time (133) 88.06 to 96.65; preferences (30) 83.33 to 96.67; updated facts (78) 97.44 to 100; user facts (70) and what the assistant said (56) 100 before and after.

• Repeats: three full runs gave the same result to the hundredth for every row above (the reranker’s scores do not vary between calls). A fourth, through the shipped code and scripts/benchmark-longmemeval-s-search.mjs, gave 99.60 / 98.36 / 96.52.
• Halves: split in two by question, the reranker scored 98.07 and 98.63 recall@5. Using its order alone, rather than fused with the decision model’s, was decided after seeing all 500, so treat that difference as a tie; both are well above before.
• What was tried and not shipped: the reranker’s order fused with recall’s and the decision model’s scored 97.98 (three runs; not different from the reranker alone, p 0.33 to 0.61, for a second call); recall’s order fused with the reranker’s only, 97.32; with the decision model’s only, 95.99; a larger reranker from the same family scored the same as the small one (98.15 vs 98.09, both fused with the decision model) at 2.5 times the price and was rate-limited on 157 of 500 questions.
• Speed and cost: recall itself took 58 ms median (170 ms p95) here, four questions at a time; the reranker adds 1.1 s median (2.0 s p95) and about $0.001 a search ($0.47 for all 500).
• sibyl.ist: it reported recall@5 96.97 and nDCG@5 94.63 on the same scorer and withdrew that number on September 25, 2026 (“the pre-1.0 headline run was withdrawn”); it publishes none now, so there is no current number to compare with.

What this does not show: it is search, not answers (the reading model still has to use what was found), and it is one public dataset of synthetic chats. The hosted memory behind the SuperAgent API got the same two changes the same day; its own result is below.

Hosted memory search accuracy

The same 500 questions through the SuperAgent API’s hosted memory (memory.recall with meaning search, on a paid plan), scored with the same metric code. Each question’s conversations were written to a fresh hosted account through the real memory routes and embedded by the production embedding model; workers/paid-agent-gateway/scripts/benchmark-cloud-memory-search.ts in the hosted-services repository.

Recall@5 by question type, before to after: counting across chats (133) 86.75 to 97.31; dates and time (133) 85.03 to 95.11; preferences (30) 86.67 to 96.67; updated facts (78) 97.44 to 99.36; user facts (70) 95.71 to 100; what the assistant said (56) 100 before and after.

Memory Answer uses the second read under Turbo. The share of the 470 answerable questions where every labelled piece of evidence reached the answer model went from 91.91% to 94.47% with fusion and 96.17% with the second read (8 questions better, none worse). The second read adds about 1.2 s (median) and costs about $0.001, inside the existing prices. In this run 37 of 961 second reads failed after retries and kept recall’s order, which is also what production does.

On real prompts

LongMemEval is a clean test. The same week’s changes were also measured on 100 real prompts from Claude Code sessions, run through the real prompt hook against a frozen copy of a live vault, three runs an arm, each attached note labelled useful or not by a judge that agreed with human labels on 95.6% of 551 notes. Before: 92 useful and 381 off-topic notes across the 100 prompts. After (the relevance check running for Claude Code and Codex and with the app closed, follow-ups searched together with the previous reply, one note per program-written series): 118 useful (better on 30 prompts, worse on 12, p = 0.008) and 255 off-topic (fewer on 67, more on 19, p < 0.001). Meaning search fusion was neutral on these prompts (120 to 117 useful, 256 to 249 off-topic); the reranker was not run on them.

LoCoMo held-out test

To check that the LongMemEval changes didn’t just fit LongMemEval’s questions, we pre-registered a comparison on LoCoMo, which none of them was developed on: 500 sampled questions, memory code from before the LongMemEval work against the code after it, Claude Opus 4.6 reading, Mem0’s published LoCoMo prompts and judge.

The newer code answered 13 questions the older missed and missed 8 it answered (p = 0.38). The changes did not overfit LongMemEval, and they are not shown to help on LoCoMo. The newer code gives the model about twice as much text there. A second pre-registered variant (both speakers stored as the user) produced empty memory, because saved conversations need an assistant turn; it is published as a failed run. Receipt.

Public long-conversation suites

HivemindOS also runs the full public LoCoMo, LongMemEval, and BEAM suites through the product’s own conversation archive, vault index, and recall ranking. These are complete Top-50 retrieval runs, not hand-picked samples.

Evidence-session recall asks whether the labeled source conversation was among the retrieved memories. It measures retrieval, not final-answer accuracy. BEAM has no equivalent labels in this adapter, so its rows show only that context came back, not that every result was relevant.

Across the 2,036 eligible LoCoMo and LongMemEval questions, HivemindOS retrieved the labeled source session for 1,931 at Top-10 (94.84%) and 2,032 at Top-50 (99.80%).

Judge-scored answers

The answer phase used gpt-5.4-mini as both answering model and judge through ChatGPT OAuth, with the same Top-50 context from HivemindOS. Every question in all four suites completed.

LoCoMo and LongMemEval are graded pass/fail, so score and pass rate match. BEAM averages a rubric per question; its pass rate counts questions scoring at least 0.5. The adapter does not yet report BEAM’s separate event-ordering (Kendall tau-b) metric.

These results are a HivemindOS snapshot, not a competitor comparison. They predate the conversation recall used in the September head-to-head and use a different reader, judge, and cutoff, so the two LongMemEval numbers can’t be compared. Results with a different answering model, judge, prompt, or cutoff (including GPT-5/Top-200 setups) are not apples-to-apples either. ChatGPT OAuth didn’t expose token counts, so no token or cost estimate is published.

The same runs confirmed a cache change that let BEAM 10M fit in normal memory. Re-running 2,440 LoCoMo, BEAM 1M, and BEAM 10M questions returned the same ranked context, faster:

The full machine-readable result, with category breakdowns and run details, is available as JSON. Methodology follows mem0ai/memory-benchmarks, snap-research/locomo, xiaowu0162/LongMemEval, and mohammadtavakoli78/BEAM. HivemindOS returns at most 50 memories, so these Top-50 results are not compared with other systems’ Top-200 scores.

Recall on messy questions

The 1,000-query live memory matrix didn’t only search exact titles. It also used sparse terms, conversational noise, typos, natural phrasing, and type, project, and tag filters.

Overall: 90% Top-1, 98% Top-3, and 0.94 MRR. Typos are the main weakness: 91% Top-3 but 81% Top-1.

The same run also checked:

• Abstention: unrelated compound questions returned no memory in 10/10 cases.
• Receipt isolation: 40/40 checks passed. Old operational receipts stayed out of normal recall but could still be requested.
• Time: current, historical, and as-of questions reached 96% Top-1 and 100% Top-3.

The 1,000 calls ran at 86.38 sequential queries per second, with 12.27ms p50 and 16.28ms p95 latency.

API speed

This benchmark goes through the authenticated dashboard API on the same machine, using a temporary vault with eight behavior cases. Across 400 requests (50 repetitions, after two warmup passes):

This is not a hosted-service promise. Hardware, vault size, disk load, and sync activity change the timings.

Hosted SuperAgent API memory

Cloud Superbrain was tested through the public SuperAgent API on August 27, 2026. Three fresh accounts each got the same memory corpus and were measured three times after warmup.

Matching local runs also passed 8/8 and 6/6, with a median trial p50 of 7.85ms and p95 of 11.63ms. Hosted recall matched local quality on this corpus, but not local speed.

A separate test added 48 misleading archive memories that reused the same terms (57 hosted memories, 55 local; the hosted test carried two extra knowledge nodes for answer grading). Every case stayed correct. The single hosted run measured 431.67ms lexical p50, 1,674.67ms semantic p50, and 2,594.36ms answer p50; the matching local lexical run measured 14.48ms p50. This shows the distractors didn’t push out the right memories; it is not a full scale or speed comparison.

These cases don’t use a private vault and don’t prove equal recall on any data. Hosted timings include internet transit, account isolation, metering, storage, and any paid semantic or answer work.

The timings above predate the second read (rerank: true and Turbo) and the current recall. Measured again on September 30, 2026 as server time (Server-Timing), on a brain of 20 short memories: keyword search (memory.search) takes about 24 ms; semantic recall (memory.recall) 333 ms median (416 ms p90), most of it the meaning index (about 200 ms) and embedding the question (about 90 ms); a Memory Answer with the default reader 665 ms median, of which the answer model is about 360 ms. rerank: true adds about 1 s. Answers that read long conversation histories take longer, almost all of it the answering model. Network time comes on top: from the Philippines, a semantic recall took about 480 ms end to end.

Past conversations on LongMemEval_S

On September 23, 2026, hosted memory gained the shared brain’s recall of past conversations: the best-matching exchanges under each conversation’s date, ordered by the Turbo decision model for Memory Answer. It was measured before and after on LongMemEval_S through the hosted search, recall, and answer code, with each question’s 50 or so chats saved to a fresh account and semantic recall off.

• The new Memory Answer was right on 98 questions the old one missed, and the old one on 1.
• Retrieval and reader both mattered. With the old answer model, the new recall alone lifted correct answers from 16.7% to 45.3%; that small model often said it had no information. Memory Answer now uses a larger model at the same price per token, which reached 81.9%. It reasons before answering, adding about three seconds to the reader’s median time.
• The decision model helped in less space. Ordered by it, the evidence fit in 12,000 characters for 81.1% of questions, against 74.8% in keyword order, and the reader did better with those 12,000 characters than with 24,000 in keyword order.

Limitations. Answers were graded with LongMemEval’s official judge instructions but by gpt-4o-mini instead of gpt-4o, so compare these rows with each other, not with the head-to-head above. With semantic recall on, as in paid plans, recall can only gain candidates. LongMemEval measures memory of past conversations; typed memories are covered by the hosted cases above.

Typed memory at scale

This benchmark builds separate indexes of 100, 500, and 1,500 memories and runs 200 exact, natural, sparse, and noisy queries on each, with local embeddings off.

The corpus is synthetic and controlled. It shows a bigger index doesn’t push out unique memories; it doesn’t replace the noisier live benchmark above.

Searching the whole vault

Typed memory is the fast path for facts, preferences, and decisions. When that isn’t enough, HivemindOS searches your normal vault notes through an index and loads only the best matches. The eight-case live benchmark covered project decisions, policies, docs, skills, secure references, imported sources, and intake notes.

The index was 15.39× faster with the same first result in every case. The full runtime is slower because it also routes, loads, and ranks the final results.

A larger test indexed 25,995 eligible notes from a 28,549-file vault in about 9.2 seconds, producing a 70.6 MB local index. Across five queries, median search dropped from 2,285ms to 118ms (19.4×) with identical Top-1 results.

Fewer tokens

Memory saves money only when agents load focused context instead of whole files and histories. The live token benchmark sent the same task to the same model twice: once with a broad context pack and once with a targeted Hive pack.

These are provider-reported counts from live OpenRouter calls with openai/gpt-4.1-mini. The broad pack was cut off at 180,000 characters in both runs; the Hive packs were not. They measure context size, not an invoice or answer quality. Full prompts and method are in Token And Cost Savings.

A local estimate over four scenarios (brain recall, software build, chatbot build, workflow reuse) counted 2,906,702 baseline context tokens against 22,764 targeted Hive tokens, a 99.2% reduction. It is a text-size estimate, not provider billing.

Cleaner current truth

Two design choices keep memory clean:

1. Routine receipts, retries, and completions go to a separate operations journal, not long-term memory.
2. Each durable memory has one current version; reviewed updates keep older versions as history.

The current snapshot has 101 active memories and 21 older versions kept as history. Forty-seven old operational records are still available on request but no longer appear in normal recall. This combines the design change with a reviewed cleanup, so it isn’t an automatic cleanup rate for every vault.

Pattern mining, with a gate

Pattern mining suggests new memories, skills, or routines from repeated events, so noise must not become advice. The labeled 47-event test set has three real patterns and many traps (retries, test runs, one-offs, unknown outcomes). The miner returned exactly the three real ones:

This passes the test gate but isn’t production precision. Pattern mining stays a dry run by default and only creates proposals for you to review. Automatic promotion stays off until there is enough real reviewed evidence.

Quoting these results

• Quote the Sibyl-Memory comparisons with their setup: LongMemEval Oracle (or LongMemEval_S), all 500 questions, the same reading model on both sides, the official gpt-4o-2024-08-06 judge for both, and Sibyl-Memory’s own published answers (or, for S, its published architecture run by us with its public runner).
• Say HivemindOS scored higher. Don’t say the difference is statistically significant: it isn’t (p = 0.33 with Opus 4.6 on Oracle, p = 0.16 on S).
• The LongMemEval_S result is not a comparison with Sibyl-Memory’s plugin product.
• Quote LongMemEval_S search accuracy as search, not answers: “the right conversation in the top 5”, with the metric named (recall@5) and “scored with sibyl.ist’s open metric code”. Don’t compare it with sibyl.ist’s withdrawn number. Quote the desktop brain’s 98.36% and the hosted API memory’s 97.68% as separate results.
• Don’t turn these results into an unmeasured competitor comparison, a hosted speed guarantee, a universal token-savings promise, or a claim that pattern proposals are production-perfect.

Reproduce the benchmarks

The local and fixture benchmarks don’t need a paid model:

pnpm benchmark:agent-memory-live-recall -- --vault <vault> --calls 1000
pnpm benchmark:agent-memory-scale -- --calls 200
pnpm benchmark:agent-memory-upgrade -- --base-url <dashboard-url> --iterations 50 --warmup 2
pnpm benchmark:shared-brain-search -- --vault <vault> --limit 8
pnpm benchmark:agent-memory-pattern-mining
pnpm benchmark:context-savings

The API benchmark needs a running HivemindOS dashboard. The full-vault and live-memory benchmarks are read-only. The scale, API, and pattern benchmarks use isolated test data. The public-suite harness is development-only; datasets and raw outputs are not shipped with the app.

The LongMemEval Oracle comparison uses the public longmemeval_oracle.json from the LongMemEval Hugging Face dataset. Building each question’s Superbrain needs no model; the reader and judge call OpenRouter and cost about $20 for all 500 questions:

OPENROUTER_API_KEY=<key> node scripts/benchmark-longmemeval-arms.mjs retrieve --dataset <longmemeval_oracle.json> --out <dir> --arms product-answer-jev
OPENROUTER_API_KEY=<key> node scripts/benchmark-longmemeval-arms.mjs answer --out <dir> --arms product-answer-jev --reader anthropic/claude-sonnet-4.5 --judge openai/gpt-4o-2024-08-06 --reader-max-tokens 1500 --budget-usd 25
node scripts/benchmark-longmemeval-arms.mjs summarize --out <dir>

LongMemEval_S search accuracy uses longmemeval_s_cleaned.json from the same dataset. Embeddings cost about $0.10 for all 500 questions (cached after the first run); --rerank adds about $0.50. The command’s header lists the embedding settings it needs:

OPENROUTER_API_KEY=<key> node scripts/benchmark-longmemeval-s-search.mjs --dataset <longmemeval_s_cleaned.json> --out <dir> --rerank
node scripts/benchmark-longmemeval-s-search.mjs --summarize <dir>

The published answers and a script that recomputes the scores are in hivemindos-memory-benchmarks.

The live token benchmark calls your chosen provider and may cost money:

./scripts/hive-env-run -- pnpm benchmark:e2e-token-savings -- --scenario brain-recall
./scripts/hive-env-run -- pnpm benchmark:e2e-token-savings -- --scenario chatbot-build

Related

• Brain, Vault, And Skills
• Brain Services
• Token And Cost Savings