Breakwater / Codeflash
Corpus · merged AI-written speedups · Codeflash
Every one of these PRs says it's faster.
10 speedup PRs written by an AI optimiser and merged by maintainers. There is no list of known bugs here: Breakwater gets the same diff a reviewer saw, and the question is simply how many change behaviour or don't deliver. Claims are the PR titles; measurements are cachegrind instruction counts.
10
merged AI speedup PRs audited
opened by the Codeflash bot, merged by maintainers
4/10
change behaviour
every class adjudicated; regression tests verified
1
reverses on valid input
complexity regression, up to 363× the instructions
$0.88
Nemotron cost of these results
Token Factory, backtest + adjudication; excludes superseded runs
Every PR
| PR | Change | Claim | Verdict | Claimed path | Worst valid input family | Tests | Backend | Cost |
|---|---|---|---|---|---|---|---|---|
| Math-Interpreter#1 | ⚡️ Speed up method Lexer.generate_number by 11%hoopinwhoopin/Math-Interpreter · 2025-02-09 | 11% faster | BLOCK | 0.88× | 1.05×Each call to generate_number now creates a list and calls ''.join for | 3 | Sandboxes | $0.066 |
| micropie#3 | ⚡️ Speed up method Server._parse_cookies by 6%hoopinwhoopin/micropie · 2025-02-03 | 6% faster | REVIEW | 1.01× | 1.60×many_empty_segments | – | Sandboxes | $0.053 |
| caddy-feeds#4 | ⚡️ Speed up function is_excluded by 17,376%fabriziosalmi/caddy-feeds · 2025-02-09 | 17,376% faster | BLOCK | 0.0032× | – | 1 | Sandboxes | $0.055 |
| patterns#9 | ⚡️ Speed up function validate_regex by 9,172%fabriziosalmi/patterns · 2025-02-09 | 9,172% faster | no divergence | 0.23× | – | – | Sandboxes | $0.031 |
| patterns#12 | ⚡️ Speed up function sanitize_pattern by 11,547%fabriziosalmi/patterns · 2025-02-09 | 11,547% faster | BLOCK | 0.058× | – | 1 | Sandboxes | $0.067 |
| rendercv#17 † | ⚡️ Speed up function validate_typst_dimension by 59%IanHollow/rendercv · 2026-08-10 | 59% faster | no divergence | 0.56× | – | – | Sandboxes | $0.032 |
| rendercv#18 † | ⚡️ Speed up function validate_date_field by 47%IanHollow/rendercv · 2026-08-10 | 47% faster | PASS | 0.64× | 1.11×present_literal | – | Sandboxes | $0.084 |
| rendercv#19 † | ⚡️ Speed up function validate_start_and_end_date_fields by 34%IanHollow/rendercv · 2026-08-10 | 34% faster | no divergence | 0.46× | – | – | Sandboxes | $0.13 |
| json_repair#113 | ⚡️ Speed up method ObjectComparer.is_same_object by 109%mangiucugna/json_repair · 2025-05-16 | 109% faster | BLOCK | 0.56× | 363.31× complexityswap position of the mismatched key within dict ordering | – | Sandboxes | $0.23 |
| pyhs3#154 | perf: Speed up function parse_expression by 785%scipp-atlas/pyhs3 · 2026-01-26 | 785% faster | no divergence | unmeasured | unmeasured: caching makes per-call cost depend on hit rate | – | Sandboxes | $0.13 |
- Claimed path: head/base instructions per call on the input family the PR optimised (below 1× is faster). Worst valid input family: the highest head/base ratio on a family where the old code accepts most inputs.
- Memoising PRs (an added
lru_cache) are measured on repeated identical inputs, so the claimed path is their best case: cache hits. When caching makes instruction counts non-linear in repetitions, the PR is marked unmeasured rather than given a number. - Tests: verified regression tests (pass on base, fail on head). † merged after Nemotron 3 Ultra's release.
- Dependencies are resolved as of each PR's merge date; script repos run from source. Raw data: data/.
- Backend: where the untrusted code ran. Token Factory Sandboxes (a microVM branched from a kept image, cited by its UUID). Every image is content-addressed (repo at each commit + harness), so every card re-runs from the image and seed 0.