The memory benchmarks
Did you tune HivemindOS on this test?
Yes, and the benchmarks page says so. The first full run scored 91.1% with Claude Sonnet 4.5. We studied the questions it missed and made four general changes to how memory answers a question: it finds conversations by the dates a question names, labels each session with its age, hands over whole conversations when they fit, and stops repeating an excerpt above its full text. They shipped to every user, and none is a rule for a particular question. LongMemEval has no separate held-out set, so we pre-registered a test on LoCoMo, a public dataset none of the changes were tuned on: the version before them scored 93.4% and the version after 94.4% (p = 0.38). They did not overfit, and on LoCoMo they are not shown to help either; the result is published as run.
Is the lead statistically significant?
No. With Claude Opus 4.6, HivemindOS answered 16 questions Sibyl-Memory missed, and Sibyl-Memory answered 10 that HivemindOS missed. An exact McNemar test gives p = 0.33. With Sonnet 4.5 it was 21 against 10 (p = 0.07). On LongMemEval_S against Sibyl-Memory's own architecture it was 26 against 16 (p = 0.16). That is why we say HivemindOS scored higher, not that it is proven better.
Why does an answer cost less than half as much?
HivemindOS memory hands the reading model its evidence once, so each LongMemEval_S question was one call: 12,096 input tokens and $0.068 at Claude Opus 4.6's list prices. Sibyl-Memory's architecture runs as a Claude Code agent that works through its journal over several turns: 8.9 calls a question on average (median 7) and 111,675 input tokens counting Claude Code's own prompt and cached re-reads, about 10,800 of them read fresh. With those cached reads at their lower price it cost $0.158 a question on average (median $0.109). Its runs took 38 seconds a question on average (median 25); ours went through the batch API, so their time was not measured. Every run's turns, tokens and cost are published with the results.
Did you run Sibyl-Memory yourselves?
For LongMemEval Oracle, no. We graded the answer files Sibyl-Memory published in its own benchmark repository, with the same official judge we used for ours, so nothing about that result depends on our setup. For LongMemEval_S, where Sibyl-Memory has published no result, yes: we ran its own published benchmark runner with the same model, changing only its per-question time limit (S histories are about ten times longer), registered the plan before the run, and published every answer and Claude Code's record of every run. Its plugin was not run, because the setup behind its plugin score is shared only with its beta testers.
Sibyl-Memory reports 95.6%. Why does this page say 95.3%?
Their 95.6% comes from their own string-matching scorer, which accepts an answer containing the right number anywhere: a correct answer of "4" matches "May 14". Graded by LongMemEval's official judge prompts with gpt-4o-2024-08-06, the same published answers score 95.3%. Every number on this page uses that judge.
Why are preference questions left out?
Sibyl-Memory's headline leaves out LongMemEval's 30 preference questions, so the comparison does too. Counting all 500 questions: 96.6% against 95.0% with Opus 4.6, and 94.6% against 92.2% with Sonnet 4.5.
Is LongMemEval Oracle a hard test?
It tests reading and reasoning, not search. Each question comes with only the past conversations that hold its answer, so memory has to pick out the right facts, dates and updates rather than find them in a huge history. Searching a large history is LongMemEval_S: there HivemindOS scored 94.3% against 92.1% for Sibyl-Memory's own architecture, while handing the reading model about a ninth of the text.
Is this the same memory the API gives my agents?
Not exactly, and it matters. The Oracle comparison ran on the Superbrain in the HivemindOS app. The API's hosted memory is measured separately: on a 150-question LongMemEval_S sample, its Memory Answer was right 81.9% of the time, graded by a smaller judge, so that figure cannot be set beside the Oracle one. The Search tab is this API's own memory: with rerank: true, recall puts the right past chats in its top 5 97.68% of the time (the app's memory: 98.36%).
Can I check the numbers without paying for anything?
Yes. The public repository holds the exact text memory handed the reading model for every question, every answer and every verdict. One free script confirms that every piece of conversation text in those contexts appears word for word in that question's own public history, so no answer could have been slipped in. Another recomputes every number. Grading again or reading again on your own API key costs about $1 for all 500, or about $0.04 a question.
Is the HivemindOS source code public?
No. The one thing you cannot check without it is that HivemindOS's own code produced the published contexts. Everything after that point can be reproduced from the repository.
Did you only publish your best run?
No. There was one run per configuration, and every full run is listed with the results, including the worse ones: 91.1% for the first version, and 94.5% and 96.0% with an optional feature that turned out not to help.
Privacy
What does HivemindOS keep in private mode?
Nothing you send reaches a service that keeps it. A private model's reply is held for 15 minutes so a dropped connection can still collect it, then the words are deleted and only the billing record stays. What you ask HivemindOS to store, such as memories, files, database records and run results, is kept as usual, and a memory stays in the tier it was written in.
What happens if I call something that is not private?
It is refused with a clear error before anything runs or is charged. A private request is never quietly sent somewhere less private.
Which services stay out of private mode, and why?
Web search and research, X, email, social publishing, lead data, managed agents and a few others. Each one has to hand what you give it to an outside service that may keep it, so they only run in Standard.
What is the difference between Private and Confidential?
Private uses only models and services that keep nothing. Confidential also runs models in attested hardware that proves what it runs, checked on every request. Media generation and meaning-aware memory recall are not available in Confidential yet.
Using the API
How is it paid for?
Every service draws on one Agent Credit balance. Add credits by card on this page, or let an agent top up with an x402 payment on Base after you approve the amount.
Can an agent use it without an account?
Yes, with an x402 wallet on Base. It can buy a single call, such as a token report, an image or an SVG, at the same price an account pays, and it is charged only if the call succeeds. Or it can pay once for an account with credits and a key. There is no sign-up either way.
Can my agent see every endpoint as its own tool?
Yes. Add ?tools=operations to the MCP address and each operation becomes its own tool, which suits apps that pick from a list, like Meta Muse. The key still decides what may run.
Can my agent use it without custom code?
Yes. Point any MCP client at the remote MCP address for your tier. The agent can list services, search actions and run what its key allows, and nothing more.
How do I stop an app from doing too much?
Give it its own key. A key can be limited to exact services and operations, request and spending rates, and a privacy tier, and wallet moves or publishing can require a separate approval.