The noise standard
Publish your false-positive rate, and let it be checked.
A static-analysis vendor's noise rate is normally a number it quotes about itself. This is a published method for producing one that somebody else can check: a holdout drawn before anyone runs, a fixed verdict vocabulary, a contract every submission must meet, and a record that publishes whether the result flatters the tool or not.
The standard does not run your scanner and does not judge your findings. It verifies the run and computes the numbers — from counts you submit — and refuses the publication when the contract is unmet.
What the standard does
It verifies and publishes. It does not measure for you.
You measure
You run your tool over the published holdout, at the revisions it pins, and you judge your own findings however you judge them. The standard never sees your engine and takes no position on it.
You submit counts and the findings behind them — never a rate. A participant that computed its own headline would be marking its own homework.
The standard checks, then computes
Nine checks run against the submission: holdout membership, pinned sha, run ordering, claim class, recency declaration, configuration declaration, finding count, re-judge and coverage. Any of them can refuse the run, and the refusal names what failed.
Then it computes the rate, its Wilson interval, the actionability rate, the per-100k absolutes, the minimum detectable difference and both cluster averages — and publishes them with the census they came from.
The vocabulary
Six verdicts, because a binary hides two different failures.
Agreement between two tools answering different questions is not a measurement. The standard owns the verdict set, and every participant implements the same one.
Should not have fired
Untrue at the cited code, or not worth a reader's time. Scores as noise.
True, and actionable
True, and it says enough to act on. Scores as valid.
True, but too thin to act on
Scores as valid, and carries the failure on the actionability axis instead. A correct finding nobody can act on is a true positive for the detector and a failure for the reader — a rater forced to choose picks differently from the next rater.
Both positions wrong
Neither the finding nor its opposite is right. Scores as noise, and escalates.
Cannot tell
The evidence shown was not enough to decide — a process defect, not a verdict. From a human it leaves the rate; from a machine it must escalate, because excluding there would hand a pipeline a way to duck its hardest cases and still report a clean number.
Rubric ambiguous
The rubric has no determinate answer here. Also a process defect: it leaves the rate and files against the method rather than the finding.
The holdout
Drawn before anyone runs, from a pool anyone can read.
The pool is public, or the draw proves nothing
Re-deriving a holdout needs the seed and the pool it was drawn from. Publishing only the seed proves nothing, because the pool could have been chosen afterwards. Both ship in a signed manifest — the rules, the draws with their seeds and timestamps, and every candidate.
Verify it yourself with openssl, against the key the manifest names. If the shipped manifest does not verify, the holdout endpoints answer 503 and serve no draw at all: a holdout that quietly degraded to "here is the pool, unsigned" would be worse than one that stops.
Four repositories are never trained on
The recency strata compare never trained against trained N cycles ago. If every repository is eventually developed against, that bucket empties and the comparison quietly becomes meaningless — so a reserved slice is drawn every period and kept pristine.
Every draw publishes its discards too: drew 22, discarded 4, measured 18. Failure-to-build correlates with size and complexity, which plausibly correlates with noise, so a silent discard is a quiet bias.
What a submission must carry
Every field exists because leaving it out flattered somebody.
The findings, with coordinates
Repository, pinned sha, file and line where the dimension can supply them, rule id, title and claim class. A finding with no file — or a file with no line — cannot be matched across vendors, so the union counts it as unmatchable and says so rather than dropping it.
The configuration
Which ruleset ran, whether that is the product default, and any rule disabled or threshold moved. Every other check constrains the run; none of them constrains how the tool was set up — and a result produced under a tuned profile is not the result a customer gets.
The denominator
Production lines of code, because the absolutes — noise per 100k LoC and valid per 100k LoC — are what expose suppression. A tool that goes quiet by firing less improves its rate and worsens its absolute.
Readable git history
A contained scan without a usable git directory makes the history-derived dimensions emit false verdicts. Their noise is then an environment artefact rather than a capability gap, so a run that cannot say whether history was readable is refused.
A recall counterpart
A noise rate measures precision and nothing else, and precision alone rewards under-firing. Published alone it is an incentive to say less — so a rate arrives with a recall figure, or with the reason there is none.
A census that balances
reported = adjudicated + excluded + unrated. Findings falling out of a funnel are exactly the ones a reader wants, and the exclusion rate is capped at 5 % — above it the run is void, not merely caveated.
The mark
CAI-measured — free, mechanical, and never "certified".
The mark says a measurement happened properly. It is not a statement about how good the tool is: three of its four conditions are about process, so a noisy tool that ran the published draw, submitted in time and published in full earns it. The rate says how it went.
Every condition is a fact the standard already holds — a judgement call anywhere in here would be an unreviewable veto held by one participant over its rivals. A withheld mark names the condition that failed.
ran-against-the-published-holdout
Decided by the membership and sha checks the receipt already reports.
submitted-before-the-deadline
A submission received before the period's published deadline.
the-run-reproduces
The receipt is accepted — every verification check passed.
numbers-published-in-full
A publication exists for the period and met the contract.
It is free, and that commitment is a rule about timing rather than an intention. Any change — a fee, or a new condition — takes effect no earlier than one full published period after it is announced, and never for a period already opened. A change reaching an open period would bind participants who have already spent the compute and cannot withdraw.
The crowd
The one check that comes from outside the model family.
Four judges from one vendor agreeing tells you they are consistent. It does not tell you they are right, and no amount of adding judges converts one into the other — so a sample of what the machines agreed on goes to people, not only the contested tail.
One item at a time
The finding, a link to the code at the revision that was scanned, and the six choices. Nothing about which tool reported it, and nothing about what the judges said — told four judges already agreed, a reasonable person rubber-stamps, and the spot-check exists precisely to catch the case where all four were wrong together.
Nobody rates their own estate
"This isn't a real problem" is a very human reaction to your own code being criticised, and it biases a rate systematically rather than randomly — which no amount of averaging removes.
Calibration is earned, not agreed
Raters are scored against findings settled outside the rating process — an upstream fix merged, a vendor withdrawal, a retracted advisory. Scoring them against what the crowd agreed would measure conformity and call it accuracy.
Scores publish, and are never applied
No answer is dropped for a poor calibration score: excluding raters by the variable being measured is selection on the outcome, and it leaves the subset that agreed — a cleaner number that means less.
Taking part
Seven steps, and every one of them is an endpoint.
No account, no key, no fee. The API is open and rate-limited per client; the crowd's endpoints carry their own budget of one item a minute.
1 · Register intent
POST /api/noise/intent — before the period's holdout is drawn. It closes at the draw, because intent declared with the sample in hand is not intent. The register publishes who said they would take part, so not submitting is visible.
2 · Read the method
GET /api/noise/method — the whole contract as JSON: the verdicts, the checks, the ceilings, the claim classes, the panel rule, the mark's conditions. Machine-readable so your client can be built against it rather than against a PDF.
3 · Fetch the holdout
GET /api/noise/holdout/{period} — the drawn repositories with their pinned revisions, the seed, and the signature over the corpus they were drawn from. Verify the signature before you trust the draw.
4 · Run, at the pins
Your tool, your judging, your machine — starting after the draw was published, at the exact revisions, with git history readable. A run that started before its own holdout answers a different question.
5 · File the run
POST /api/noise/submissions — the findings with their coordinates, the recency of each repository, and the configuration you ran under. The receipt names every check that passed or failed, and the findings it recorded.
6 · Publish the result
POST /api/noise/publication — the counts, the claim-class breakdown, the per-cluster tallies, the denominator. The standard computes the figures and refuses the publication if the contract is unmet, naming each breach.
7 · Read the record
GET /api/noise/record/{period} and /api/noise/mark/{period} — the verdicts, the disputes raised against them, and who earned the mark. Your own register row is withheld from everyone else until the period publishes.
Watchdog owns this standard and competes in it.
That conflict is real and it is not resolved by anybody promising to behave. Four things make it survivable, and all four are mechanical:
- Depth is measured leave-one-out. Each tool is scored against the union of what every other participant found, itself excluded — a designated baseline, especially ours, would make depth mean "how much you agree with Watchdog" and score us 100 % by construction.
- An embargo covers the register. No participant sees another's result before the period publishes, and there is no exemption for us. Early sight of a rival's number is the single most valuable thing this position could be worth.
- The mark's conditions are facts, not judgements. Nothing in granting or withholding it is anyone's opinion, and a withheld mark names the condition that failed.
- The method version takes effect from the next holdout drawn, never the current one. A change published after a draw cannot apply to that draw — including a change we would benefit from.
This is phase one: a published method with an open invitation, and no governance body. A board with one vendor on it is theatre, and it invites exactly the scrutiny it is meant to deflect.
Run it against your own tool, and tell us where it is wrong.
A verdict you disagree with can be contested with a reason at /api/noise/verdicts/{findingId}/dispute — upheld and overturned disputes both publish, because a contest that only appeared when the challenger won would be a complaints box. The scoring standard itself → /spec.