HivemindOS manual
Agent Evaluations
An agent saying “done” is not the same as the work being done. HivemindOS checks each managed result, verifies the evidence that matters, and records what actually passed. Risky or customer-facing work gets a separate reviewer, and work HivemindOS cannot see is labeled honestly instead of counted as a success.
How much checking each job gets
Checks scale with risk, so an ordinary chat reply does not pay for a full review.
| Level | Used for | What is checked |
|---|---|---|
| Quick | Ordinary chat | Did it finish? Is the answer empty, very thin, or only a refusal? |
| Verified | Work Board tasks, schedules, and managed runtime jobs | Quick checks, plus proof that claimed files exist and claimed commands really ran. A file path in text or pasted test output is not proof. |
| High assurance | Publishing, payments, deployments, customer-facing or rubric-scored work | Verified checks, plus a different agent reviews the result, scores it with evidence, and states its confidence. If no separate reviewer is available, the work needs evidence or human review. |
Where it applies
| Where | What happens |
|---|---|
| Chat | Quick checks. When a chat turn builds an app, it is judged by the build’s own result, so an unfinished app is never accepted. |
| Work Board | Managed completions get a task evaluation and must pass the loop’s required gates. |
| Zero Human Companies | Outcome evidence is required. Product, design, content, and customer-facing work requires a separate reviewer. |
| Scheduler | Run notes include the verdict and receipt. |
| Managed runtime tasks | Codex, Claude Code, OpenHands, and Aider tasks are evaluated when they finish. |
| AEON company handoff | Marked unobserved until HivemindOS can read and review the actual result. |
| Independently launched CLI | A process you start yourself outside HivemindOS is outside the managed run path, so it is not graded. Bring its result back to a Work Board loop if you need a receipt. |
Results
| Result | Meaning |
|---|---|
| Accepted | Passed the checks for its risk level. |
| Rejected | Finished, but did not meet the bar. |
| Needs evidence | May be valid, but a required file, command result, or reviewer was missing. |
| Abstained | The reviewer could not decide from the evidence. |
| Evaluation error | The check itself failed. The work is not counted as accepted. |
| Unobserved | HivemindOS saw a handoff but not enough of the result to grade it. |
Treat Needs evidence as a useful stop: the system refused to invent certainty.
Rate a chat response
Use the thumbs-up or thumbs-down under a managed response. Click again to change or remove your rating.
Thumbs-down marks that response rejected. Thumbs-up confirms it only if the automatic checks already passed; it cannot turn a failed response into an accepted one. Chat ratings do not affect which agents get future tasks.
Why receipts are hard to fake
Agents can attach evidence, but they cannot approve their own independent review, command checks, file checks, or governance gates. Those decisions belong to HivemindOS. Each passing receipt is tied to the exact output that was checked, so if the output changes, the receipt no longer counts.
Only accepted managed-task results influence which agents get future work. Chat volume and unobserved handoffs do not.
Get better results
- Run important work through the Work Board, Scheduler, a company crew, or a managed runtime task.
- Ask for the actual deliverable and concrete proof.
- Put expected files, URLs, test commands, and success criteria in the loop contract.
- For high-assurance work, have at least two eligible agents: one to build and one to review.
To measure whether a change to context, tools, skills, or instructions really helps, use Harness experiments.