Open SAST benchmark · Apache 2.0 · no signup
BenchProctor is a machine-verifiable benchmark corpus for static analysis. Every test case is labeled vulnerable or safe across 9 languages, stripped of every naming hint, and rotated each quarter, so a score reflects detection rather than memorization. Score anything that emits SARIF 2.1.0.
Java ships first, production-ready, full public launch by end of June. Python is next. Why one at a time →
$ python score_sarif.py results.sarif expectedresults-2026.2.csv # 22,400 cases · 9 languages · 234 categories matching SARIF locations to labels…
TP 9,812 FN 388 TPR 96.2%
FP 517 TN 11,283 FPR 4.4% category-averaged Youden J 0.918 flat aggregate 0.917 [ok] wrote score_report.json
Illustrative output. Your numbers are whatever your scanner earns.
Why another benchmark
PROBLEM 01
Most suites ship a fixed set of human-written cases. The tools, and the data they were trained on, overfit to them, and a high score stops meaning real-world accuracy.
→ Cases are generated combinatorially, and the corpus rotates every quarter.
PROBLEM 02
When a path, identifier, or comment says sqli_true_positive_01, you are testing string matching, not analysis. A scanner can score high for entirely the wrong reason.
→ Zero comments, zero CWE tags, file names that reveal nothing. The CSV is the only ground truth.
PROBLEM 03
Real findings cross files, services, and languages, and sit next to sanitizers that almost work. Single-file, Java-only suites never exercise any of that.
→ Nine languages and eighteen frameworks, with broken-sanitizer, evasion, and cross-service variants.
What a test case is
Each category is one vulnerability class expressed as a taint flow. Untrusted input enters at a source, travels to a sink, and a defense either holds or it doesn't. A vulnerable case is missing an effective sanitizer; its safe twin has one. Every case is graded by difficulty, and each category comes out balanced 50/50 vulnerable to safe.
Where untrusted input enters: query params, request bodies, headers, env vars, files, message payloads.
How the data travels to the sink: concatenation, decoding, parsing, indirection, collection passthrough.
What would neutralize it: parameterization, output encoding, validation, allow-lists, strong crypto.
The dangerous call: query execution, command exec, HTML render, file open, redirect, deserialization.
# vulnerable case source http_request_body
propagator json_parse, field_access
sink sql_query
label vulnerable # safe twin, same flow, one defense that holds source http_request_body
propagator json_parse, field_access
sanitizer parameterized_query
sink sql_query
label safe The defense in a safe case always fits the sink it protects. A scanner never gets a free win from a nonsense pairing like an HTML encoder guarding a SQL query, so passing means the tool actually reasoned about the flow.
Quarterly rotation
Every quarter the corpus rotates. The actual test code changes completely, so you can't pre-train against next quarter's files. Everything that decides what a score means stays fixed, so last quarter's number is still comparable to this one. The target moves; the ruler reads the same.
# fresh variants, identical contract
corpus-2026.2/ 234 categories · 50/50 · difficulty held
corpus-2026.3/ 234 categories · 50/50 · difficulty held # a 2026.2 score stays comparable to 2026.3 # nothing you trained on last quarter is in this one [ok] new code, identical contract
Anti-leakage by construction
A benchmark only measures analysis if the file can't be classified by its name or its comments. Every file in the corpus holds to these properties, so nothing is left to authoring discipline.
Emitted code carries none. There is no natural-language hint for a model to latch onto.
Not in the code, not in identifiers, not in filenames. Nothing in a test file names its own vulnerability class.
The name and position of a file tell you nothing about its category or its label. There is no ordering to read a pattern off of.
The answer key lives in one place the scanner never reads: a separate expectedresults CSV that only the scorer ever opens.
Beyond single files
Detecting a bare eval(input) is table stakes. The corpus is
weighted toward the findings that separate a real analyzer from a pattern matcher, and toward
where the threat landscape is actually moving.
Instead of one weakness in one file, a chain threads a group of smaller CWEs across modules and functions so they compound into a larger compromise. These are the low-severity findings that line up into privilege escalation and root access, and tools that reason file-by-file miss the path.
Taint that crosses language and process boundaries the way real systems do: HTTP, gRPC, subprocess, environment variables, and message queues (Kafka, RabbitMQ, Redis), across 2-, 3-, and 4-service topologies with multiple sources, sinks, and paths.
A sanitizer is present but bypassed: a flawed regex, wrong-context escaping, an insufficient length limit. Scanners that trust the mere presence of a sanitizer mislabel these as safe.
Built to defeat naive static analysis and to cover where the threat landscape is actually moving, the age of AI rather than just textbook SQLi. That means SAST-evasion (homoglyphs, encoding round-trips, dynamic dispatch, eval/exec wrappers, Unicode-normalization bypasses), software supply-chain shapes (glassworm-style), and AI/LLM prompt-injection-era patterns.
Polyglot coverage
In each language, the code reads the way a developer in that ecosystem would actually write it: framework DTOs, Pydantic models, form classes, ORM calls, not toy snippets. Adding a language changes nothing about the categories, so coverage stays uniform across the matrix.
Python
Flask · Django · FastAPI
Go
net/http · Gin
Java
Spring · Jakarta EE
JavaScript
Express · Koa
TypeScript
NestJS · Express
PHP
Laravel · Symfony
Ruby
Rails · Sinatra
Bash
standalone
Rust
Actix · Axum
OWASP Top 10 2025
213 of the 249 CWEs mapped by the OWASP Top 10 2025 are covered as SAST code-pattern test cases (85.5%). The remainder is config-level, supply-chain, or runtime-only, none of it expressible as a static code pattern (A03 is composition analysis, a separate discipline from SAST).
Scoring
Run your scanner, hand the SARIF and the answer-key CSV to a one-file scorer, read three numbers. No service, no account, no telemetry. The math is fully in your hands.
The whole formula
detected ignored
vulnerable TP FN
safe FP TN
TPR = TP / (TP + FN) detection rate
FPR = FP / (FP + TN) false-alarm rate
J = TPR − FPR Youden's JWhat the score means
Every category is weighted equally, so a handful of huge categories can't carry the score. A flat aggregate is reported alongside for comparison.
Anything that emits SARIF 2.1.0 can be scored. The scorer is a single standard-library Python file with no dependencies, nothing to trust but the source you can read.
Ground truth you can trust
A benchmark is only as good as its labels. Every label is checked before a release goes out, and the scorer is proven to agree with the answer key it reads. A misplaced label can't reach you, because it fails that verification and the release stops.
# two files, nothing more testcode/ the programs you scan, blind expectedresults.csv test, category, label, CWE # the label for each program safe or vulnerable [ok] verified before release, scorer proven to match
Verified before publication
Every label ships with the code that justifies it and its CWE, all in the open. If you think one is wrong, you don't have to take it on faith. Read the case and make the argument with the file in front of you.
The ecosystem
Five focused tools for shipping with AI agents. Each stands on its own; together they cover code, action, orchestration, memory, and proof.
TheAuditor
Don't let agents guess about your code.
theauditortool.com →Warden
Let agents act safely, across any provider.
wardenclient.com →Arbiter
The command center for your AI operation.
arbitermcp.com →Curator
Don't let agents guess about you.
curatormcp.com →BenchProctor
The SAST benchmark you can't game.
you are hereGet started
Three commands stand between you and a real accuracy number for your scanner. Corpora are versioned and released every quarter; the scorer is a single standard-library Python file.
# 1. run your scanner, export SARIF 2.1.0 $ your-tool scan ./corpus --format sarif -o results.sarif
# 2. score against the answer key $ python score_sarif.py results.sarif corpus/expectedresults-*.csv
# 3. read TPR, FPR, and your Youden J → category-averaged + flat aggregate, per-category breakdown