Backtest · pypa/packaging · 2026 performance PRs

Faster isn't the same as correct.

Breakwater audits performance pull requests the way an attacker would. A Nemotron adversary reads the diff, guesses which assumptions the "faster" code makes, and hunts for inputs where the new version behaves differently or runs slower. Deterministic probes and instruction counts confirm everything it claims.

9/9

documented regressions caught before merge

best no-LLM baseline: 7/9

merged perf PRs audited

pypa/packaging, the version library inside pip

8/10

found regressions labelled likely-bug

Ultra adjudication, evidence-checked

$2.02

Nemotron cost of these results

Token Factory, backtest + adjudication; excludes dev and superseded runs and reused perf

Documented regressions caught before merge

9 regressions in these PRs were later fixed upstream. Each approach got the same PRs, the same probes and no hint of which bugs exist.

The PRs' own tests + benchmarksthese PRs merged green

0/9

Plain Hypothesisst.text() search, no LLM

2/9

Mutations of the PR's own test stringshostile edits of literals from the project's tests, no LLM

7/9

Nemotron plan onlyUltra's example inputs, no search

6/9

BreakwaterUltra plan + Lightning strategies + search + mutations

9/9

Candidates (linked to fixes but not reproduced by hand): Breakwater 1/3. PRs with any behaviour divergence: 9/29.

Every PR

PR	Change	Verdict	Classes	Known issue caught	Perf worst	Backend	Cost
#997	perf: better string building`Version` · 2026-01-06	no divergence		–	–	Docker	$0.033
#1028	perf: don't renormalize extras`Requirement` · 2026-01-06	no divergence		–	–	Docker	$0.021
#1029	perf: use fullmatch for markers too`Marker` · 2026-01-04	no divergence		–	–	Docker	$0.018
#1030	perf: translate for canonicalize_name`canonicalize_name` · 2026-01-05	BLOCK	8	unicode-lowercase	4.52×	Docker	$0.049
#1032	perf: add __slots__ to Nodes`Marker` · 2026-01-02	BLOCK	1	–	–	Docker	$0.026
#1058	perf: 10% faster stripping zeros`Version` · 2026-01-14	no divergence		–	–	Docker	$0.045
#1064	perf: dual replace`canonicalize_name` · 2026-01-19	PASS	7	–	1.10×	Docker	$0.039
#1076	perf: Stream PEP440 filtering in `SpecifierSet.filterSpecifierSet#filter` · 2026-01-30	no divergence		–	–	Docker	$0.048
#1081	perf: Skip `__replace__` in Specifier comparison if not needed`Specifier` · 2026-02-09	no divergence		–	–	Docker	$0.030
#1082	perf: Add fast path for parsing simple versions (digits and dots only)`Version` · 2026-02-11	BLOCK	1 1	oversized-int-exception	2.85×	Docker	$0.10
1082@61967cca	perf: fast path for simple versions (intermediate commit 61967cca)`Version` · 2026-02-11	BLOCK	3 2	inner-whitespace, oversized-int-exception, unicode-digits	1.91×	Docker	$0.097
#1083	perf: Add fast path for `Version` to `Version` comparison by skipping `_key` property`Version` · 2026-03-06	no divergence		–	–	Docker	$0.035
#1105	perf: Speed up complex SpecifierSet filtering by implementing cost based ordering`SpecifierSet#filter` · 2026-03-08	no divergence		–	–	Docker	$0.058
#1106	perf: simply Specifier regex`Specifier` · 2026-03-09	no divergence		–	–	Docker	$0.090
#1108	perf: SpecifierSet use tuple instead of frozenset for `_specsSpecifierSet` · 2026-03-07	BLOCK	12 3	dedup-equality, dedup-str-len, old-pickles-break-comparison	–	Docker	$0.28
#1111	perf: Speed up wildcard comparison`Specifier#contains` · 2026-03-09	no divergence		–	–	Docker	$0.12
#1115	perf: skip hash property`Version` · 2026-03-11	no divergence		–	–	Docker	$0.041
#1116	perf: Overhaul `_cmpkey` to remove use of custom objects`Version` · 2026-03-11	BLOCK	1	old-pickles-fail-to-load	–	Docker	$0.12
#1118	perf: Cache `Version` hash value in dedicated slot`Version` · 2026-03-13	no divergence		none	–	Docker	$0.044
#1147	perf: add __slots__ to Marker`Marker` · 2026-04-05	BLOCK	1	marker-pickle	–	Docker	$0.027
#1236 †	perf: simply work done in parse_tag a bit`parse_tag` · 2026-06-09	PASS	4	–	1.09×	Docker	$0.17
#1250 †	perf(markers): cache the default environment`Marker#evaluate` · 2026-07-10	no divergence		–	–	Docker	$0.031
#1253 †	perf(specifiers): keep range caches across canonicalization`SpecifierSet#contains` · 2026-06-16	no divergence		–	–	Docker	$0.056
#1256 †	perf(utils): precompile the wheel project-name pattern`parse_wheel_filename` · 2026-06-13	no divergence		–	–	Docker	$0.11
#1258 †	perf: add __slots__ to token classes`Requirement` · 2026-06-13	no divergence		–	–	Docker	$0.010
#1259 †	perf(specifiers): push PEP 440 logic into the range engine`SpecifierSet#contains` · 2026-06-15	no divergence		–	–	Docker	$0.082
#1320 †	perf: add __slots__ to Requirement`Requirement` · 2026-07-07	no divergence		none	–	Docker	$0.11
#1371 †	perf(markers): speed up _format_marker and benchmark str(Marker)`Marker` · 2026-09-25	no divergence		–	–	Docker	$0.10
#1429 †	perf(parser): avoid literal_eval for plain marker strings`Marker` · 2026-10-05	no divergence		–	–	Docker	$0.032

† merged after Nemotron 3 Ultra's release (2026-06-04), so it cannot be in the model's training data.
Perf column: worst head/base instruction-count ratio over input families where most inputs are valid. Above 1× means the 'faster' change runs more instructions.
Backend: where the untrusted code ran. local Docker (an image cited by its content-addressed tag). Every image is content-addressed (repo at each commit + harness), so every card re-runs from the image and seed 0.
Perf columns come from backtest packaging-v2 (same corpus and commits; the semantic arms were re-run later with a new planner, which perf doesn't use). Costs shown exclude those perf measurements.
Probe corrections applied after the run (1 items re-scored deterministically, no model calls): see data/backtest.json → corrections.

Audit runs

Single audits from breakwater audit, as shown in the demo.

Run	Change	Verdict	Classes	Backend	Cost
`c1a7799be95f`	perf: fast path for simple versions (intermediate commit 61967cca)`packaging.version:Version`	BLOCK	6	Docker	$0.11
`e780ff68654f`	perf: fast path for simple versions (intermediate commit 61967cca)`packaging.version:Version`	BLOCK	6	Sandboxes	$0.18

Also audited: 10 merged AI-written speedups

PRs by the Codeflash optimiser bot, merged by maintainers. No known bugs to score against: 4 of 10 change behaviour, and the instruction counts check each claim.

json_repair#113: claims 109% faster; a valid input family runs up to 363× the instructions (complexity regression).
Math-Interpreter#1: claims 11% faster; changes behaviour, e.g. Lexer("1١"): base raises Exception, head returns normally.
caddy-feeds#4: claims 17,376% faster; changes behaviour, e.g. is_excluded([":"]): base returns normally, head raises TypeError.
patterns#12: claims 11,547% faster; changes behaviour, e.g. sanitize_pattern([["pattern", "0"]]): base raises AttributeError, head raises TypeError.
rendercv#18: claims 47% faster; changes behaviour, e.g. get_date_object(["!!!!"]): base raises TypeError, head raises ValueError.

See all 10

Checked on Token Factory Sandboxes: 5/8 verdicts agree

8 of these PRs were audited again with every untrusted run in a Token Factory Sandboxes microVM; the 3 perf families measured on both agree in verdict, with Ir ratios up to 44% apart. Each number cites the image UUID that reproduces it.

See the comparison

How a verdict is made

Plan. Nemotron 3 Ultra reads the diff and names the assumptions the fast path makes, with example inputs that would break them.
Search. Nemotron 3.5 Lightning writes Hypothesis strategies; base and head run side by side in isolated sandboxes, and counterexamples shrink to their minimal form. Hostile mutations of the project's own test strings run too.
Probe. Deterministic checks compare outputs, exact exception types, equality / ordering / hashing, and whether objects pickled by the old version still load and compare.
Measure. Slowdown families are confirmed by cachegrind instruction counts across input sizes from 2 to 4,096: deterministic, so a ratio is a fact rather than noise.
Adjudicate. Ultra labels each divergence likely-bug, intended or needs-human. Three votes; a vote counts only if its evidence is quoted verbatim from the PR. The model can label, never delete.