How often each check is right.
We don’t claim an accuracy number we haven’t measured in public. The benchmark lives in the open source repo, so anyone can re-run it.The benchmark is in bench/ in the public repo. Re-run it yourself; the numbers on this page come from its results file.
The numbers
Measured on 2026-11-10 with sigship 0.3.2. For each check, the benchmark runs it over files that must set it off and look-alike files that must not.Run 2026-11-10, sigship 0.3.2. Per check: planted positives and look-alike negatives. Precision = true positives ÷ flagged. Recall = true positives ÷ planted.
| Check | Right when it flagsPrecision | Finds what is thereRecall | Cases |
|---|---|---|---|
| A live secret is sitting in your code | 90.9% | 83.3% | |
| Your .env file is committed | 70.8% | 85% | |
| A server secret has a browser-visible name | 81.8% | 85.7% | |
| Your AI key runs in the browser | 84.2% | 80% | |
| A database table has no row level security | 94.7% | 90% | |
| A database policy lets anyone write | 94.4% | 85% | |
| Your Stripe webhook accepts anything | 65.2% | 75% | |
| User input may be pasted into a SQL query | 88.2% | 75% | |
| Your AI route has no spending brake | 81.3% | 65% | |
| Admin access is decided in the browser | 94.1% | 76.2% | |
| Raw HTML goes straight onto the page | 71.4% | 71.4% | |
| Any website can call your API | 82.4% | 66.7% | |
| A secret may be going to your logs | 72.2% | 61.9% | |
| One crash takes down the whole screen | 81.8% | 85.7% | |
| You will hear about errors from your users | 75% | 100% | |
| Nothing checks that a change did not break login | 100% | 59.1% | |
| Your deploys are not repeatable | 90.5% | 95% |
How to read it
100% on both lines means the check never cried wolf and never missed a planted problem in the benchmark. Real apps are messier, so treat these as the best case.Planted cases are the upper bound. A check’s recorded precision is a floor for releases: a release fails if any check drops below it.
Something it got wrong?
A false alarm or a missed problem is a bug. Tell us with the smallest example you can.Report false positives and misses with a minimal case; each one becomes a benchmark fixture. Report it on GitHub.