CruciblePublic

The problem

Point a language model at a codebase and it will hand back a list of bugs. Some are real. Plenty read well and turn out to be wrong anyway, and sorting one from the other costs more time than the review was meant to save. Underneath that sits a second worry. A thing that reads your code and runs commands is doing it on your machine, with your keys in the environment, while you watch a progress bar and hope.

What it does

Three verifiers attack every finding before anyone reads it, and that is where the design puts its weight. A planner divides the ground into four to six lanes. One hunter takes each lane, blind to the others, because agents that can see each other’s work converge on the same easy bug and a review comes back with four copies of it. Everything a hunter raises then goes to three verifiers who are told to refute it, kept blind to each other’s verdicts, and instructed that uncertainty counts as a refutation. Two of the three settle it. A verifier that crashes or falls silent counts against the finding it was judging, because the wrong direction to fail in is the one where an unchecked claim reaches the report on the strength of a dead process.

So a run discards a large share of its own output, and the refuted count keeps its own space on screen for the whole time it works. The run published in the repository raised sixteen findings and kept nine. Results vary between runs and the record says so: three consecutive runs against the same nine planted defects found seven, then seven again, then six, and not the same seven. One run found a cache-invalidation miss and passed over a falsy-role authorisation bypass, and the next did the reverse. The defensible reading is a range, with any single run treated as one sample.

The exit code is the whole interface, which is what lets the arena stand in front of a merge as a pull-request gate. Zero means the run finished with everything refuted, and one means a finding survived verification. Two means the run could not complete or the configuration is wrong, and that third code is the one that matters most. Models are chosen per seat rather than once for the run, so the wide fan-out can sit on hardware you own while the seat where being right matters runs on something stronger. Anything speaking the OpenAI chat shape works as a base URL, and a fresh clone runs the whole arena against a stand-in model so you can watch it work before spending anything.

How it works

A run is four kinds of agent over one workspace. Underneath them sit the parts that hold: a policy checked before any tool code executes, a probe that spends the run attacking that policy, a budget that reserves before it spends, and a ledger that records every one of those decisions in an order it can prove.

• PlannerOne call on the expensive seat. It reads the task and divides the ground into four to six lanes, so the hunt has structure rather than four agents starting at the top of the same file.
• HuntersOne agent per lane on the cheap seat, running in parallel and blind to each other. Volume work, priced accordingly, and the stage whose output is treated as raw material rather than as a result.
• VerifiersThree per finding on the expensive seat, each asked to refute it and each blind to the other two verdicts. A finding needs two of the three to fail to refute it, and a verifier that dies counts as a refutation.
• Policy at the tool boundaryAuthority is declared in one object that is printed during the run and travels in the record. It contains file reads and searches to the resolved workspace path, holds scratch writes to their own directory under a size ceiling, permits five commands, and carries no network tool at all. Every proposed action is checked against it before any tool code executes, and a check that raises counts as a refusal.
• Boundary probeA fourth agent runs alongside the hunt and attacks the boundary rather than the code, making six real attempts through the same door every other agent uses. Each attempt is harmless if it lands, and the day one of them stops being refused, the run says so and fails.
• BudgetA call reserves its worst case up front, priced as though the model returns its full output allowance, then settles to real usage afterwards. Calls that fail release their hold, so the ceiling holds against work in flight and a run stops slightly early rather than at a number nobody chose.
• LedgerEvery event is appended carrying the SHA-256 of the entry before it, so the file proves its own order and completeness. The interface recomputes the whole chain in the visitor’s browser after a run, and the verify command replays every policy decision on the reader’s own machine.

What was hard

Safe to leave running

Every action an agent proposes is checked against a written policy before any tool code executes. Containment is decided on the resolved path, so a request for work/../../etc/passwd is judged as the file it actually opens, and a sibling directory whose name shares the workspace prefix is refused, which is the case a naive prefix check quietly admits. A policy check that raises counts as a refusal. Refusals are first-class events: they appear on screen with the reason, they are counted, they land in the record, and the refused agent gets a sentence it can reason about and carries on working.

A run that trips no limit shows a reader nothing about whether the limits exist, so a fourth agent spends the hunt trying to leave. It attempts six escapes through the same door every other agent uses: reading a file outside the workspace, running a command off the allowlist, handing code to a permitted interpreter, reaching the network, writing outside scratch, and running a script it wrote itself. Each attempt is chosen to be harmless if it lands, each is recorded as an ordinary tool decision in the same ledger, and a boundary that gives way fails the run out loud.

Spend is bounded before it happens rather than after. A call reserves its worst case at the price of a full output allowance and settles to real usage afterwards, with failed calls giving their hold back, so a dozen agents checking at once are all checking against the same honest total. The hosted deployment carries a ceiling per run and per UTC day, sixty US cents and eight dollars by default, and a visitor who attaches their own key spends against a separate per-run ceiling of one dollar with a hard maximum of five above which the number stops being theirs to raise. A vendor declared unmetered is refused by every provider on the shelf, because pricing calls at zero would switch the ceiling off entirely.

A repository named by URL is reviewed under a read-only policy with the test runner withheld, because a stranger’s test files executing inside your process is a wider grant than reading their tree. The clone is one commit deep, lands in a temporary workspace that is removed when the run ends, and has its git directory stripped so the agents read the tree at that commit rather than its history, under caps of 200 MB and 20,000 files from the command line and smaller ones on the hosted box. Every child process a tool starts, wherever the workspace came from, sees a scrubbed environment: what an interpreter needs to start, and never the API key or anything ending in _KEY, _TOKEN, _SECRET or _PASS.

Hollow runs fail rather than passing quietly. A run whose agents all died once exited zero and printed a clean bill of health, because finding nothing and never running both produce an empty list. Every way a run can be hollow now exits two: agents that never answered, a planner that produced no lanes, a budget that ran out, or a boundary that gave way. A deployment that sets neither the public switch nor a credential pair exits two as well, so a server that forgot its password stays shut rather than opening a paid model to whoever finds the URL. The demo target’s answer key is held back by name for reads and searches, because a hunter that reads the answers has found nothing, and that refusal is written to the ledger like any other.

The record is checkable by someone who has reason to doubt it. Each ledger entry carries the hash of the one before, the interface recomputes that chain in the visitor’s own browser one click after a run, and the command line replays every policy decision locally, rebuilding the rules from a directory you name when you want the adversarial version of the check. The repository states plainly that this is tamper-evident rather than tamper-proof, keeps eight findings from its own review open in docs/KNOWN-ISSUES.md with what each one costs, and sets out in SECURITY.md what the guarantees rest on, including that the policy is a boundary inside one operating-system user rather than a sandbox.

Evidence

• Majority-survival reporting
  The run published on 2026-08-17 raised sixteen findings against the demo target and kept nine, and every judged finding in its ledger carries its own split of survivals and refusals. The whole file ships in docs/evidence/ beside a method note that records the task and the six lanes the planner drew, and replaying it reproduces 24 tool decisions and prints "16 raised, 9 survived, $0.2746".
• Client-verifiable ledger
  The verify button walks a finished run line by line in the browser using SHA-256 from the Web Crypto API, hashing the raw bytes with the hash field cut out rather than re-serialising, and names the head it reaches beside the number of entries it recomputed. The command line does the same walk and replays the policy on top of it: over the published ledger it prints "chain intact, head 3e4c243b53b9b660aa198eccdfca8f21", which docs/evidence/README.md writes down as the expected output before you run it.
• Spend ceiling held against calls in flight
  Budget.reserve holds a call’s worst case and Budget.settle books what it actually cost, both under one lock in crucible/providers.py, with the concurrency checks in the orchestrator suite. The published run cost US$0.2746 against the $1.20 ceiling recorded in its own header. Twelve recorded runs on the same seats fell between 21 and 39 US cents with a median of 29, a figure reproduced against the ledgers during the pre-ship audit, where the README’s earlier count of thirteen was corrected to the twelve that reproduce.
• 78% lift at the hunt stage
  A 9B model on one desktop GPU, five runs per arm, pass marks written down before any number existed. Recovery of the nine planted defects went from 1.8 per run when the reply shape was asked for in the prompt to 3.2 when the sampler was constrained by a per-role JSON schema, and the union across five runs went from three of nine to seven of nine. Replies falling short of the documented protocol went from 31 of 239 to 2 of 237. The same record prints FAIL against the pre-registered recovery bar, because that bar scores defects surviving verification and that number was zero in every run of both arms: this model’s verifiers, told that uncertainty is a refutation, refuted everything including correct findings. bench/README.md carries the method and bench/results/ carries every raw reply.
• 854 checks, no network, no spend
  The README’s total for the suite, which runs offline against a stand-in model that answers from the prompt, so a queue of canned replies handed to concurrent agents cannot pass or fail by luck. Each check file is a plain script that prints its own tally when run directly, so the files are the authority: running the nine of them in the current tree returns 907 passed and 0 failed, the count having drifted up as checks were added.
• Forty-five findings raised, eighteen survived
  The arena’s review of its own codebase: four reviewers fanned out, with independent verification on every finding they raised. The early ledgers stayed on the build machine, so this pair of counts is reported rather than checkable, and docs/KNOWN-ISSUES.md says exactly that at the top of the page that lists the eight findings still open.

See it running

The hosted demo runs the whole arena on a free stand-in model, open to anyone, and the button beside a finished run recomputes that run’s hash chain in your own browser.