HivemindOS manual
Harness Experiments
Harness experiments are an experimental way to improve an agent without changing its model. HivemindOS keeps the agent and the job the same, compares your current setup with one focused change, and keeps the evidence.
What you can test
- narrower task context instead of a large context dump;
- a new or revised skill;
- clearer tool descriptions or routing;
- an instruction placed next to the files it covers;
- a safer permission or approval boundary;
- stronger proof of the finished result.
Each experiment starts by fixing the agent, target, sample job, what counts as success, required proof, permissions, and budget. The change then names one thing and what it should improve.
Evidence that counts
HivemindOS tracks four separate states for each piece of context:
| State | Meaning |
|---|---|
| Available | It could have been used. |
| Retrieved | It was placed in the task’s context. |
| Invoked | The agent actually used it. |
| Relevant | Evidence ties it to the accepted result. |
A tool showing up in the prompt does not prove the agent used it, and a shorter prompt does not prove the result was right.
When a result is trusted
A comparison needs at least three fresh, isolated runs of the current setup and three of the change, with the same agent, target, environment, job, grader, and permissions. The change must actually be used and must not leak into the baseline runs.
Results are judged in order:
- Did the real task pass?
- Did the agent produce the required proof?
- Did it respect the intended design and permissions?
- Only then: did speed, retries, tool calls, tokens, or cost improve?
You can keep, revise, or drop the change. HivemindOS blocks keeping it if outcome or proof got worse, the runs were not comparable, or there were too few repeats.
Where to see it
Open the Memory and Review view to see recent comparisons next to Context X-Ray, with run counts, readiness, outcome and proof changes, token changes, and the decision.
Skill Autoresearch uses the same method when a skill keeps failing. It never replaces an installed skill automatically.
Cost
Local graders do not call a paid model. Live provider comparisons spend tokens, so they run only when you choose them. A savings figure is shown only when the outcome also passes and there are enough repeats.
The method is adapted from Ryan Lopopolo’s Harness Engineering, licensed under CC BY 4.0.