How tooldocstest.dev works
tooldocstest.dev reads a versioned JSON suite describing your tools and the example arguments in your quickstart, then checks each example against the tool's schema and, where possible, runs it through an allowlisted data-only sandbox tool. Everything runs locally; there is no hosted upload and no production credential is ever used.
The v1 input contract
A suite is a v1 envelope with a tools map
(name to schema) and an examples list of
{tool, arguments, expected_output?}. Schemas use a strict flat
object JSON Schema subset: type: object with primitive
(string / number / integer / boolean) properties, required, and a
boolean additionalProperties. Unknown keywords and nested, union or
$ref schemas resolve to unknown — never silently ignored to
manufacture a pass.
Tri-state results and exit codes
- pass (exit 0): the example executed through an allowlisted sandbox tool and matched the contract.
- fail (exit 1): a schema mismatch, which is reported without calling the tool.
- unknown (exit 2, also used for invalid input): the tool or schema cannot be evaluated. Unknown is distinct from pass and from fail.
Pass requires real execution through one of two built-in, data-only sandbox
tools: sandbox.echo-v1 and sandbox.sum-v1. Unknown
actual tools stay unknown. The optional expected_output comparison
is canonical JSON. There is no eval, shell, production credential, arbitrary URL
or docs code execution.
Import adapters
Two adapters normalize a vendor manifest into the same v1 tools map. Both are pure deterministic reshapers, not live connections to any provider.
mcp-tools-list-v1importsmanifest.tools[].nameandinputSchemafrom an MCPtools/listcapture.openai-functions-v1importsmanifest.tools[].function.nameandfunction.parametersfrom an OpenAI tool-calling definition; entries whosetypeis notfunctionare rejected (UNSUPPORTED_TOOL_TYPE), never silently dropped.
Shared limits
- Names must be strings; prototype-polluting names
(
__proto__/constructor/prototype) and duplicate names are rejected (INVALID_TOOL_NAME). manifest.toolsis capped at 1000 entries.- Input is capped at 1 MiB, 1000 examples, depth 32.
- An imported schema using nested or union shapes stays unknown, never a pass.
Receipts
Receipts carry an index, a tool/output digest, the result and an execution
flag — never raw arguments or outputs. Hashes are pseudonyms, not anonymization.
Receipts are deterministic and byte-identical across runs. Persist one with
--receipt <file>, which works alongside
--markdown and preserves the exit code.
What is disabled
Hosted upload, run history, authentication and billing are disabled. Every
route other than /, /docs, /robots.txt,
/sitemap.xml and /healthz fails closed with
501 HOSTED_EVALUATION_DISABLED. The hosted dashboard ships only after
authentication, isolation, retention and deletion tests pass.
See the acceptance fixture matrix for the exact pass/fail/unknown result of each of the five acceptance fixtures, or go back to the overview.