How tooldocstest.dev works

tooldocstest.dev reads a versioned JSON suite describing your tools and the example arguments in your quickstart, then checks each example against the tool's schema and, where possible, runs it through an allowlisted data-only sandbox tool. Everything runs locally; there is no hosted upload and no production credential is ever used.

The v1 input contract

A suite is a v1 envelope with a tools map (name to schema) and an examples list of {tool, arguments, expected_output?}. Schemas use a strict flat object JSON Schema subset: type: object with primitive (string / number / integer / boolean) properties, required, and a boolean additionalProperties. Unknown keywords and nested, union or $ref schemas resolve to unknown — never silently ignored to manufacture a pass.

Tri-state results and exit codes

Pass requires real execution through one of two built-in, data-only sandbox tools: sandbox.echo-v1 and sandbox.sum-v1. Unknown actual tools stay unknown. The optional expected_output comparison is canonical JSON. There is no eval, shell, production credential, arbitrary URL or docs code execution.

Import adapters

Two adapters normalize a vendor manifest into the same v1 tools map. Both are pure deterministic reshapers, not live connections to any provider.

Shared limits

Receipts

Receipts carry an index, a tool/output digest, the result and an execution flag — never raw arguments or outputs. Hashes are pseudonyms, not anonymization. Receipts are deterministic and byte-identical across runs. Persist one with --receipt <file>, which works alongside --markdown and preserves the exit code.

What is disabled

Hosted upload, run history, authentication and billing are disabled. Every route other than /, /docs, /robots.txt, /sitemap.xml and /healthz fails closed with 501 HOSTED_EVALUATION_DISABLED. The hosted dashboard ships only after authentication, isolation, retention and deletion tests pass.

See the acceptance fixture matrix for the exact pass/fail/unknown result of each of the five acceptance fixtures, or go back to the overview.