Breakwater

Report / run c1a7799be95f

Breakwater audit · tokenfactory · local Docker

perf: fast path for simple versions (intermediate commit 61967cca)

pypa/packaging · base 59f183fe90 → head 61967cca27 · target packaging.version:Version · ran on Docker

BLOCK

6 behaviour change(s) look like unintended regressions. A valid input family runs 2.76× the instructions of the old code.

Ultra's plan · 5 attacks

Nemotron 3 Ultra read the diff and named the assumptions the change makes. Nothing here is trusted: each attack only counts through the counterexamples below.

  1. A1The fast‑path assumes the version string contains only ASCII digits and dots, so it strips leading/trailing whitespace before parsing.

    base: The original regex‑based parser (`_regex.fullmatch(version)`) is anchored (`^…$`) and does not allow surrounding whitespace, so it raises `InvalidVersion`.; head: The new fast‑path accepts the stripped string, builds a Version object with epoch 0, release tuple, and no pre/post/dev/local parts, and returns normally.

    Inputs: Version strings that are a valid simple numeric dotted version surrounded by leading/trailing ASCII whitespace. · e.g. Version(" 1.2.3 "), Version("\t1.0.0\n") · Lightning strategies: 3/4 valid, 0 hits

  2. A2The fast‑path assumes a simple numeric version never has trailing zero components, so it skips the normal trailing‑zero stripping logic.

    base: Version("1.0.0").release == (1,) and str(Version("1.0.0")) == "1.0".; head: Version("1.0.0").release == (1, 0, 0) and str(Version("1.0.0")) == "1.0.0".

    Inputs: numeric dot‑only versions with trailing zero components · e.g. Version("1.0.0"), Version("2.0.0.0"), Version("0.0") · Lightning strategies: 2/4 valid, 0 hits

  3. A3The fast‑path assumes that surrounding whitespace is harmless because it strips the string before parsing, while the original regex validates the raw string.

    base: Version(" 1.2.3 ") raises `InvalidVersion` because the original regex does a fullmatch on the raw string containing spaces.; head: Version(" 1.2.3 ") is accepted; the fast‑path strips whitespace, splits on '.', converts each part to int, stores epoch=0, pre/post/dev/local=None and returns a valid Version object.

    Inputs: whitespace‑padded numeric version · e.g. Version(" 1.2.3 "), Version("\t1.2.3\n") · Lightning strategies: 3/4 valid, 1 hits

  4. A4The fast‑path assumes the version string contains only ASCII digits and dots, so it strips any Unicode whitespace characters before parsing.

    base: The original regex does not recognise Unicode whitespace as part of the version pattern, so `fullmatch` fails and `InvalidVersion` is raised.; head: The fast‑path creates a Version object from the stripped numeric dotted string, treating it as a valid simple version.

    Inputs: Version strings that are a valid simple numeric dotted version surrounded by leading/trailing Unicode whitespace. · e.g. Version("  1.2.3  ") · Lightning strategies: 4/4 valid, 1 hits

  5. A5The fast‑path assumes numeric dot‑only versions are already in normalized form, so it omits the trailing‑zero stripping step.

    base: Version("3.2.0").release == (3, 2) and str(Version("3.2.0")) == "3.2".; head: Version("3.2.0").release == (3, 2, 0) and str(Version("3.2.0")) == "3.2.0".

    Inputs: numeric dot‑only versions ending with one or more zero components · e.g. Version("3.2.0"), Version("10.0"), Version("0.0.0") · Lightning strategies: 3/4 valid, 0 hits

Counterexamples · 3 groups of 6 classes: 6 likely bug

P2

Exception behaviour changed

inputVersion("0. 0")
baseraises InvalidVersion
headreturns normally
Likely bugvotes: 3 likely-bug

Fast‑path added in src/packaging/version.py strips only leading/trailing whitespace and then splits on '.'; internal spaces such as "0. 0" pass the fast path (since space is not in _NON_SIMPLE_VERSION_INDICATORS) and int(" 0") succeeds, whereas the original PEP‑440 regex rejected internal spaces, raising InvalidVersion. This change was not documented and appears unintentional.

_NON_SIMPLE_VERSION_INDICATORS = frozenset("!+-ABCDEHILOPRSTVW_abcdehiloprstvw")
if _NON_SIMPLE_VERSION_INDICATORS.isdisjoint(version):

Found by: Nemotron strategy search, hostile mutation

ran onlocal Docker · image bw-pair:348e2d05c100a5f9

Regression test verified passes on base, fails on head
def test_p2_outcome_0():
    """Breakwater P2: base behaviour pinned (found by differential testing; label: likely-bug)."""
    import pytest
    from packaging.version import InvalidVersion, Version
    with pytest.raises(InvalidVersion) as excinfo:
        Version('0. 0')
    assert type(excinfo.value) is InvalidVersion  # exact type: a subclass would be a behaviour change
Reproduce
uv run breakwater probe --repo https://github.com/pypa/packaging --base 59f183fe90da --head 61967cca278d --target packaging.version:Version --backend docker --input "0. 0"
1 near-duplicate class same behaviour, inputs with extra traits
  • Version(" ١.2.3 ") base raises InvalidVersion · head returns normally test verified (also under another group)
P2

Exception behaviour changed

inputVersion("١.0.0")
baseraises InvalidVersion
headreturns normally
Likely bugvotes: 3 likely-bug

Arabic‑Indic digit “١” is not in the indicator set, so the fast path accepts "١.0.0" after stripping and splitting, converting the Unicode digit with int(). The original regex only allowed ASCII digits, raising InvalidVersion. Accepting non‑ASCII digits is a regression.

_NON_SIMPLE_VERSION_INDICATORS = frozenset("!+-ABCDEHILOPRSTVW_abcdehiloprstvw")
if _NON_SIMPLE_VERSION_INDICATORS.isdisjoint(version):

Found by: hostile mutation

ran onlocal Docker · image bw-pair:348e2d05c100a5f9

Regression test verified passes on base, fails on head
def test_p2_outcome_4():
    """Breakwater P2: base behaviour pinned (found by differential testing; label: likely-bug)."""
    import pytest
    from packaging.version import InvalidVersion, Version
    with pytest.raises(InvalidVersion) as excinfo:
        Version('١.0.0')
    assert type(excinfo.value) is InvalidVersion  # exact type: a subclass would be a behaviour change
Reproduce
uv run breakwater probe --repo https://github.com/pypa/packaging --base 59f183fe90da --head 61967cca278d --target packaging.version:Version --backend docker --input "١.0.0"
1 near-duplicate class same behaviour, inputs with extra traits
  • Version(" ١.2.3 ") base raises InvalidVersion · head returns normally test verified (also under another group)
P2

Exception behaviour changed

inputVersion("100000000000000000000000000000000000000000…00000000.0" (4,303 chars))
baseraises ValueError
headraises InvalidVersion
Likely bugvotes: 2 likely-bug, 1 intended

Same reasoning as C3 – an extremely long numeric version passes the fast path because it contains only digits and dots, while the original regex raised ValueError. The fast path unintentionally relaxes the version length limits.

_NON_SIMPLE_VERSION_INDICATORS = frozenset("!+-ABCDEHILOPRSTVW_abcdehiloprstvw")
if _NON_SIMPLE_VERSION_INDICATORS.isdisjoint(version):

Found by: hostile mutation

ran onlocal Docker · image bw-pair:348e2d05c100a5f9

Regression test verified passes on base, fails on head
def test_p2_outcome_3():
    """Breakwater P2: base behaviour pinned (found by differential testing; label: likely-bug)."""
    import pytest
    from packaging.version import Version
    with pytest.raises(ValueError) as excinfo:
        Version('1' + '0' * 4300 + '.' + '0')
    assert type(excinfo.value) is ValueError  # exact type: a subclass would be a behaviour change
2 near-duplicate classes same behaviour, inputs with extra traits
  • Version(" 10000000000000000000000000000000000000000…00000.2.3 " (4,307 chars)) base raises ValueError · head raises InvalidVersion test verified
  • Version("  1000000000000000000000000000000000000000…0000.2.3  " (4,309 chars)) base raises ValueError · head raises InvalidVersion test verified

Regression tests

6 of 6 classes have a generated pytest that passed on base and failed on head in the sandbox.

Download test_regressions.py

Performance · Version

Worst family with mostly valid inputs: 2.76× the instructions of base (Each numeric chunk increases digit count linearly while keeping total chunks fixed).

head slowerhead faster1×1.2×1.4×1.7×2×2.4×2.8×101001,00010,000100,000input size (characters)R3.2R3.2: 1.04× at ~16 charsR3.2: 1.07× at ~25 charsR3.2: 1.32× at ~163 charsR3.2: 1.93× at ~2,155 charsR3.2: 2.44× at ~33,163 charsR3.2: 2.23× at ~525,835 charsR3R3: 1.04× at ~10 charsR3: 1.07× at ~19 charsR3: 1.29× at ~157 charsR3: 1.93× at ~2,149 charsR3: 2.44× at ~33,157 charsR3: 2.23× at ~525,829 charsR4R4: 1.04× at ~10 charsR4: 1.07× at ~19 charsR4: 1.27× at ~157 charsR4: 1.89× at ~2,149 charsR4: 2.50× at ~33,157 charsR4: 2.76× at ~525,829 charsR3.3R3.3: 1.03× at ~16 charsR3.3: 1.05× at ~25 charsR3.3: 1.23× at ~163 charsR3.3: 1.88× at ~2,155 charsR3.3: 2.41× at ~33,163 charsR3.3: 2.21× at ~525,835 chars
  • R3.2
  • R3
  • R4
  • R3.3
Head/base cachegrind instructions per call, by input size (log-log). Each line is one input family Nemotron proposed; points above 1× are where the "faster" code does more work.
FamilyInputsRatio range Slope base → headVerdict
R3.2Pre-release label changed from 'c5' to 'alpha.1'suspected slowdown · expand:R31.04× – 2.44×0.70 → 0.78slower
R3Each numeric chunk consists of repeated digits (`'9'*i`) instead of zero-padded numberssuspected slowdown · archive1.04× – 2.44×0.67 → 0.75slower
R4Each numeric chunk increases digit count linearly while keeping total chunks fixedsuspected slowdown · archive1.04× – 2.76×0.63 → 0.73slower
R3.3Repeating triple-digit pattern in each chunksuspected slowdown · expand:R31.03× – 2.41×0.67 → 0.76slower

Receipt

Model (Nebius Token Factory)Cost
Nemotron 3 Ultra$0.11
Nemotron 3.5 Lightning$0.002
Nemotron 3 Super$0.001
Total$0.11

runc1a7799be95f

imagebw-pair:348e2d05c100a5f9

backendlocal Docker (docker)

semantic plan 27s · semantic generate 7s · semantic search 13s · semantic batch 2s · perf plan 15s · perf screen 95s · perf full 154s