Planted ground truth, shadow runs, Wilson intervals — the calibration story · September 2026
"How did you pick that threshold?" is the first hard question an examiner asks about a surveillance program, and the honest answer at most venues is a shrug wrapped in a vendor's default. This is how DataTap answers it — the same way for all 23 patterns, and in writing, because the method is the control.
Before a detector ships, a fixture plants the exact behaviour it claims to catch into a synthetic market — the front-runner's small print 1.2 seconds ahead of the whale five times, the desk that quotes 200 times for four fills, the account that trades small and regular twelve times and then fifty times its median. The test suite asserts the detector fires on exactly the planted actors and nothing else. A detector that over-fires on the background population fails the build. This is why our thresholds tend to be relative — "ten times its own median", "an order of magnitude past the market's median" — rather than absolute counts that drown in small populations.
The tunable thresholds (the god-mode z floor, the bound-spoof cancel-to-fill ratio, the cluster exposure limit) are workspace guardrails under maker-checker: one person proposes, a different person approves, and the proposal carries a backtest at the candidate value — how many planted scenarios still fire, how many accounts the sweep adds — plus an A/B against the value in force: what the change stops catching and what it starts catching, named.
A candidate value can run as a shadow first: the detector re-runs silently at that value on every scan and counts what would have fired, for seven days, without raising an alert. The proposal that follows carries a number ("would have fired 3 times across 28 scans") instead of a hunch. The recommendation engine, which watches your own dispositions for detectors running at a high false-positive rate, offers the shadow before it offers the proposal.
Quarterly — and on demand — the platform generates a calibration report from the workspace's own disposition history: per pattern, alerts raised, confirmed, dismissed as false positive, and the false-positive rate with a Wilson 95% interval, so a pattern with four decisions is not compared to one with four hundred as if they were equally known. Below that, threshold-sensitivity curves: for each tunable value, how many planted scenarios and how many live accounts fire at each candidate. The value in force should sit where the planted scenarios still fire and the account sweep stops adding noise. The report is write-once and ships inside the quarterly CCO pack.
Not certainty — no threshold is right forever — but a method: ground truth the detector is tested against, a governed path for changing it, evidence attached to every change, and a periodic report that measures the result with honest error bars. The public methodology page is rendered from the same code the detectors run, so the description and the system cannot drift apart. That is the answer to "how did you pick that threshold": here is the planted crime, here is the shadow run, here is the interval.
The methodology, every detector: datatap.org/docs/methodology · for DCM applicants.