Breakwater

Codeflash / Math-Interpreter#1

⚡️ Speed up method Lexer.generate_number by 11%

hoopinwhoopin/Math-Interpreter#1 · merged 2025-02-09 · base c7c1b7e931 → head 04c05ed375 · ran on Sandboxes

BLOCK

At least one behaviour change looks like an unintended regression.

Counterexamples · 2 groups of 3 classes: 2 likely bug, 1 needs a human

P2

Exception behaviour changed

inputLexer("1١")
baseraises Exception
headreturns normally
Likely bugvotes: 3 likely-bug

The change replaces the ASCII‑digit check (`self.current_char in DIGITS`) with `str.isdigit()`, which now accepts Unicode digits such as '١'. The PR only claims a speed‑up and does not mention expanding the accepted digit set, so the new acceptance of non‑ASCII digits is an unintended regression.

DIGITS = '0123456789'
- while self.current_char is not None and (self.current_char in DIGITS or self.current_char == '.'):

Found by: Ultra's planned example, hostile mutation · also caught by: Nemotron plan only

ran onToken Factory Sandboxes · image 6cbae808-5ef0-4683-8481-a6120ba99109

Regression test verified passes on base, fails on head
def test_math_interpreter_1_p2_outcome_0():
    """Breakwater P2: base behaviour pinned (found by differential testing; label: likely-bug)."""
    import pytest
    from lexer import Lexer
    with pytest.raises(Exception) as excinfo:
        result = Lexer('1١').generate_tokens()
        if hasattr(result, '__next__'):  # generators run when iterated
            list(result)
    assert type(excinfo.value) is Exception  # exact type: a subclass would be a behaviour change
Reproduce
uv run breakwater probe --repo https://github.com/hoopinwhoopin/Math-Interpreter --base c7c1b7e931e8 --head 04c05ed37563 --target lexer:Lexer#generate_tokens --backend contree --input "1١"
1 near-duplicate class same behaviour, inputs with extra traits
  • Lexer("1١ ") base raises Exception · head returns normally test verified
P2

Exception behaviour changed

inputLexer("100000000000000000000000000000000000000000…000000000١" (4,301 chars))
baseraises Exception
headreturns normally
Needs a humanvotes: 2 needs-human, 1 likely-bug (model abstained)

The only difference for the very long all‑zero input is that the old version raises an Exception while the new version succeeds. The provided material does not explain why the old version fails (no test, doc, or changelog mentions this case), so we cannot decide whether the change is intentional or a bug.

Found by: hostile mutation · no baseline found it

ran onToken Factory Sandboxes · image 6cbae808-5ef0-4683-8481-a6120ba99109

Regression test verified passes on base, fails on head
def test_math_interpreter_1_p2_outcome_1():
    """Breakwater P2: base behaviour pinned (found by differential testing; label: needs-human)."""
    import pytest
    from lexer import Lexer
    with pytest.raises(Exception) as excinfo:
        result = Lexer('1' + '0' * 4299 + '١').generate_tokens()
        if hasattr(result, '__next__'):  # generators run when iterated
            list(result)
    assert type(excinfo.value) is Exception  # exact type: a subclass would be a behaviour change

Regression tests

3 of 3 classes have a generated pytest that pins the old behaviour. Each passed on the base commit and failed on the head commit in the sandbox before it was offered.

Download test_Math_Interpreter_1.py

Performance · Lexer#generate_tokens

Worst family with mostly valid inputs: 1.05× the instructions of base (Each call to generate_number now creates a list and calls ''.join for a one‑character number. The overhead of list allocation and join dominates for the many sh).

head slowerhead faster0.84×1×101001,000input size (characters)R1R1: 0.98× at ~7 charsR1: 1.01× at ~11 charsR1: 1.04× at ~35 charsR1: 1.05× at ~131 charsR1: 1.05× at ~515 charsR1: 1.05× at ~2,051 charsR1: 1.05× at ~8,195 charsR2R2: 0.98× at ~7 charsR2: 1.01× at ~11 charsR2: 1.04× at ~35 charsR2: 1.05× at ~131 charsR2: 1.05× at ~515 charsR2: 1.05× at ~2,051 charsR2: 1.05× at ~8,195 charsF5F5: 0.98× at ~7 charsF5: 1.01× at ~11 charsF5: 1.04× at ~35 charsF5: 1.05× at ~131 charsF5: 1.05× at ~515 charsF5: 1.05× at ~2,051 charsF5: 1.05× at ~8,195 charsR3R3: 0.98× at ~7 charsR3: 1.01× at ~11 charsR3: 1.04× at ~35 charsR3: 1.05× at ~131 charsR3: 1.05× at ~515 charsR3: 1.05× at ~2,051 charsR3: 1.05× at ~8,195 charsF1F1: 0.93× at ~6 charsF1: 0.93× at ~8 charsF1: 0.92× at ~20 charsF1: 0.91× at ~68 charsF1: 0.90× at ~260 charsF1: 0.88× at ~1,028 charsF1: 0.88× at ~4,100 chars
  • R1
  • R2
  • F5
  • R3
  • F1
Head/base cachegrind instructions per call, by input size (log-log). Each line is one input family Nemotron proposed; points above 1× are where the "faster" code does more work.
FamilyInputsRatio range Slope base → headVerdict
R1Each call to generate_number now creates a list and calls ''.join for a one‑character number. The overhead of list allocation and join dominates for the many shsuspected slowdown · archive0.98× – 1.05×1.04 → 1.05slower
R2Each call to `generate_number` now creates a Python list, appends a single character, and then joins it. For a huge sequence of tiny numbers the per‑number allosuspected slowdown · archive0.98× – 1.05×1.04 → 1.05slower
F5Many one‑digit numbers separated by operatorssuspected slowdown · plan0.98× – 1.05×1.04 → 1.05slower
R3A sequence of n single‑digit numbers separated by ‘+’ (e.g., "1+2+3+…").suspected slowdown · archive0.98× – 1.05×1.04 → 1.05slower
F1single very long numeric literalclaimed speedup · plan0.88× – 0.93×0.90 → 0.89faster

Receipt

TargetBackendImageRun AuditAdjudication
lexer:Lexer#generate_tokensToken Factory Sandboxes6cbae808-5ef0-4683-8481-a6120ba99109bt-Math-Interpreter#1-1d8136$0.046$0.020

Nemotron cost on Nebius Token Factory. Backend: where the untrusted code ran. Token Factory Sandboxes (a microVM branched from a kept image, cited by its UUID). Every image is content-addressed (repo at each commit + harness), so every card re-runs from the image and seed 0. Raw data: audit, adjudication.