One prompt injection out of six got through

rada uses a language model to decide the order of a queue of jobs. The queue it shows the model contains command lines, and command lines contain text that came from repositories, which can be hostile. I wanted a number for that rather than a guess, so the test came before the defence. Here is what it found.

The number

rada queues heavy jobs from parallel coding sessions on one laptop, and it uses a language model to decide who goes first. The queue it hands the model contains command lines, which contain text from repositories, which may be hostile. That is a prompt injection surface, and writing “we handle prompt injection” would have been easy.

Instead tools/prova-giudice.py runs six styles of attack through a paired comparison: the same queue with the hostile text and without it, so an ordering that changed can be told apart from an ordering that was going to change anyway. On the recorded run, one of the six worked. An appeal to a deadline in one hour promoted the job, and the model repeated the claim back as its reason.

One in six is the honest number and it sits in the README. The reason it is tolerable has nothing to do with the prompt. A verdict is worth at most three points against an age that earns one every thirty seconds, it expires after three minutes, it has to be a permutation of the exact ids that were asked about, and it cannot touch the set of jobs that have already waited ten minutes. A fully successful injection buys ninety seconds of queue jumping. That is a design claim, and it is testable, which is why both fairness properties are proved by tests rather than asserted in prose, including a four hundred round adversarial simulation.

Checks that refuse

A check nobody is forced to run is a suggestion. Some of these stop the work.

claude-codex-bridge moves a live coding conversation between Claude Code and Codex by writing a native session file for the other agent. Those on-disk formats are not a published cross-vendor API, so every target write is version gated: supported source prefixes are Claude Code 2.1.* and Codex 0.146.*, and an unsupported target blocks instead of guessing. Above that sits tools/verify-drift.py, which mints unpredictable markers, creates a real session, performs the round trip, asks both agents to list the markers back, and exits non-zero if any are lost. It is a mandatory gate before a release or an upgrade.

That probe also documents its own weakness. The instruction to answer without reading any file is given in the prompt and is not enforced by a sandbox, so a marker coming back is evidence on the assumption that the agent obeyed. Naming that is part of the job.

molo has one hard rule: no circumvention code. A test enforces it, so a patch that crosses the line fails CI rather than reaching anyone’s inbox.

Check the copy that ships, not the copy you have

varo exists because of four failures found on sites that were already live while every local check stayed green. One of them: a build step pointing at a folder the host never creates. The minifier finds zero files, reports success, exits 0. Nothing is minified and the log says everything is fine.

So the rule the plugin runs on is: fetch it, do not infer it. A finding counts when the live site answered, and counts for nothing when it came from reading code and reasoning about what the code probably does. When the auditor ran, it also listed what it could not close, including that it never sent a real message, so the mail finding stops at the address and does not claim delivery.

argano learned the same lesson from a bug that shipped. A track header written four bytes too long still parses by name, so QuickTime played the file perfectly on the machine that made it. Players that read by offset showed the video stretched about ten to one. Now the finished file is read back from its byte offsets and checked against three sources that have to agree. In the test suite, frame counts have to match exactly and the media bytes in the output have to be identical to the media bytes in the input.

Where this costs

Writing the check is more work than writing the function, and often much more.

argano ships two implementations of one algorithm, Python and browser JavaScript, and keeping them from drifting apart means both have to answer the same questions about the same bytes. Its browser AAC parser is tested against 1218 real frame boundaries taken from four recordings. The claude-codex-bridge drift probe consumes two Claude calls and one Codex call every time it runs, which is why it is a release gate instead of a per-commit test.

The worse cost is a check that is wrong, because it buys silence. While building the Snapchat importer I batched exiftool arguments through an argfile, and the options were silently dropped. Reading the file back with exiftool showed exactly the value I wanted. It was only wrong inside Photos, which is the only place it mattered. The general lesson: verify the effect in the destination application, and distrust a value read back by the same tool that wrote it.

scriba carries a second checker for the same reason. Its style rules pass line by line, and every document it writes is assembled from string fragments, so each fragment can pass while the finished file is wrong. check-output-style.py renders all six output formats from stand-in data and checks those instead.

None of this makes a tool correct. It makes the claims about it falsifiable by somebody who did not write it, which is the only version of a claim worth publishing.