Breakwater

Codeflash / caddy-feeds#4

⚡️ Speed up function is_excluded by 17,376%

fabriziosalmi/caddy-feeds#4 · merged 2025-02-09 · base f5e3d0c28f → head d94d92d5d9 · ran on Sandboxes

BLOCK

At least one behaviour change looks like an unintended regression.

Counterexamples · 1 class: 1 likely bug

P2

Exception behaviour changed

inputis_excluded([":"])
basereturns normally
headraises TypeError
Likely bugvotes: 3 likely-bug

Base version gracefully handles invalid IPs by catching ValueError and returning True; the new @lru_cache decorator introduces a TypeError for the same input, changing documented behavior without any test, doc, or changelog update.

return True # Treat invalid IPs as excluded
+@lru_cache(maxsize=1024)

Found by: Nemotron strategy search, search sample · no baseline found it

ran onToken Factory Sandboxes · image 1bafd31e-3df4-4239-8ac7-13dd73b31ec0

Regression test verified passes on base, errors on head
def test_caddy_feeds_4_p2_outcome_0():
    """Breakwater P2: base behaviour pinned (found by differential testing; label: likely-bug)."""
    from ip import is_excluded
    is_excluded([':'])  # base returns normally

Regression tests

1 of 1 classes have a generated pytest that pins the old behaviour. Each passed on the base commit and failed on the head commit in the sandbox before it was offered.

Download test_caddy_feeds_4.py

Performance · is_excluded

No realistic input family measured slower.

head slowerhead faster0.0039×0.0078×0.016×0.031×0.062×0.12×0.25×0.5×1×100input size (characters)F3F3: 0.00× at ~11 charsF3: 0.00× at ~11 charsF3: 0.00× at ~11 charsF3: 0.00× at ~11 charsF3: 0.00× at ~11 charsF3: 0.00× at ~11 charsF3: 0.00× at ~11 chars
  • F3
Head/base cachegrind instructions per call, by input size (log-log). Each line is one input family Nemotron proposed; points above 1× are where the "faster" code does more work.
FamilyInputsRatio range Slope base → headVerdict
F3repeated_same_ipclaimed speedup · plan0.00× – 0.00×0.00 → 0.00faster

Receipt

TargetBackendImageRun AuditAdjudication
ip:is_excludedToken Factory Sandboxes1bafd31e-3df4-4239-8ac7-13dd73b31ec0bt-caddy-feeds#4-081228$0.037$0.018

Nemotron cost on Nebius Token Factory. Backend: where the untrusted code ran. Token Factory Sandboxes (a microVM branched from a kept image, cited by its UUID). Every image is content-addressed (repo at each commit + harness), so every card re-runs from the image and seed 0. Raw data: audit, adjudication.