HivemindOS manual
Web Research
Web Research lets agents search the web, read pages and PDFs, pull out structured data, follow links within a site, spot page changes, and take screenshots. Search, page reading, text PDFs, and crawling work out of the box, with no download or API key.
Searches and page requests still go to public search engines and websites, so this is not offline browsing.
What agents can do
| Capability | Result |
|---|---|
| Search | Ranked, deduplicated results from several keyless sources. Results are leads; agents should read the sources before treating claims as evidence. |
| Fetch | Clean Markdown, text, or article content from a public URL, with paging for long pages. |
| Structured extraction | Metadata, headings, links, tables, JSON-LD, and up to 20 fields you pick with CSS selectors. |
| PDF and OCR | Text PDFs work out of the box. Scanned PDFs and page ranges need the enhanced engine. |
| Crawl | A limited crawl of one site, a sitemap, or a chosen set of URLs. |
| Cache and compare | A private cache of up to 32 MB speeds up repeat reads and can show what changed since the last read. Signed or credential-like URLs are never cached. |
| Screenshot | PNG or JPEG captures, with the enhanced engine. |
Web Research only reads. Clicking, filling forms, signed-in sessions, and uploads go through Computer Interaction, with its own permissions and confirmations.
Optional enhanced engine
The enhanced engine adds screenshots, OCR for scanned PDFs, and a local browser for pages that block or need JavaScript. It is optional, starts only when needed, and uses roughly 1–2 GB. If it’s installed, HivemindOS automatically retries blocked pages with it. Setup installs and upgrades it, and uninstall offers to remove it. If its install fails, everything else keeps working.
Limits and safety
- Only public web addresses are allowed. Local and private network addresses are blocked, including after redirects.
- Agents can’t add cookies, logins, custom headers, or proxies.
robots.txtis always respected.- Crawls are capped by pages, depth, parallel requests, output size, and total time.
- Page content is untrusted. It can’t grant permissions, approve spending, or change an agent’s instructions.
Some sites rate-limit, block bots, or need a login, payment, or CAPTCHA. The agent reports the failure and should find another credible source rather than treating an error page as evidence.