Breakwater

Report / run e780ff68654f

Breakwater audit · tokenfactory · Token Factory Sandboxes

perf: fast path for simple versions (intermediate commit 61967cca)

pypa/packaging · base 59f183fe90 → head 61967cca27 · target packaging.version:Version · ran on Sandboxes

BLOCK

4 behaviour change(s) look like unintended regressions. A valid input family runs 2.74× the instructions of the old code.

Ultra's plan · 5 attacks

Nemotron 3 Ultra read the diff and named the assumptions the change makes. Nothing here is trusted: each attack only counts through the counterexamples below.

  1. A1The fast‑path guard assumes that a version lacking the indicator characters contains only ASCII digits and dots, so it splits on '.' and feeds each piece to int().

    base: The old __init__ falls through to the full regex (self._regex.fullmatch). The pattern does not allow whitespace between the dot and the digits, so the match fails and InvalidVersion is raised.; head: The fast‑path executes: version.strip().split('.') yields pieces like '1 ' and '2'; int() accepts the surrounding whitespace, so a Version object is created with release=(1, 2).

    Inputs: versions with spaces adjacent to the dot separator · e.g. Version("1 .2"), Version("1. 2") · Lightning strategies: 3/4 valid, 2 hits

  2. A2The fast‑path guard assumes that any string lacking the indicator characters consists only of ASCII digits and dots, so int() will only ever see ASCII digits.

    base: `Version("1.2")` falls through to the regex path; `VERSION_PATTERN` uses `[0-9]` which matches only ASCII digits, so the fullmatch fails and `InvalidVersion` is raised.; head: The fast‑path is taken, `version.strip().split(".")` yields `["1", "2"]`; `int()` accepts Unicode decimal digits, producing `(1, 2)`. A `Version` instance is created with `release=(1, 2)` and no exception.

    Inputs: PEP‑440‑like release segment written with non‑ASCII decimal digits · e.g. Version("1.2") · Lightning strategies: 3/4 valid, 1 hits

  3. A3The fast‑path assumes `int()` only accepts ASCII decimal digits; it treats any string without the indicator characters as a simple numeric version.

    base: The regex `VERSION_PATTERN` only matches ASCII `[0-9]` in the release segment, so `Version("١.٢.٣")` raises `InvalidVersion`.; head: The fast‑path runs `version.strip().split('.')` and `map(int, …)`. `int()` accepts Unicode decimal digits, so the version is accepted and normalized to `(1, 2, 3)`.

    Inputs: non‑ASCII decimal digits (Arabic‑Indic) · e.g. Version("١.٢.٣") · Lightning strategies: 3/4 valid, 2 hits

  4. A4The fast‑path guard assumes that a version without indicator characters consists solely of ASCII decimal digits, but int() also accepts any Unicode decimal digit.

    base: The regex uses [0-9] for the release segment, which matches only ASCII digits; the fullmatch fails and InvalidVersion is raised.; head: The fast‑path splits on '.' and calls int() on each segment; int() accepts full‑width digits (e.g. U+FF11), so a Version object is created with release=(1, 2).

    Inputs: versions written with non‑ASCII decimal digits · e.g. Version("1.2"), Version("123.456") · Lightning strategies: 2/4 valid, 0 hits

  5. A5The fast‑path guard assumes that a string without indicator characters contains no whitespace inside the release segment.

    base: Regex `\s*` only permits leading/trailing whitespace; the pattern for `release` (`[0-9]+(?:\.[0-9]+)*`) does not allow a space between a digit and a dot, so `fullmatch` fails and `InvalidVersion` is raised.; head: Fast‑path executes; `strip()` removes only leading/trailing whitespace, `split(".")` gives `["1 ", "2"]`; `int("1 ")` succeeds (int ignores surrounding whitespace), producing `release=(1, 2)`. A `Version` instance is returned.

    Inputs: Release segment with ASCII whitespace around the dot separator · e.g. Version("1 .2"), Version("1. 2") · Lightning strategies: 3/4 valid, 1 hits

Counterexamples · 5 groups of 6 classes: 4 likely bug, 2 intended

P2

Exception behaviour changed

inputVersion("0 .0")
baseraises InvalidVersion
headreturns normally
Likely bugvotes: 3 likely-bug

The fast path accepts a version with an internal space (e.g., '0 .0') because space is not in _NON_SIMPLE_VERSION_INDICATORS and strip() only removes leading/trailing whitespace. The base version correctly rejects it via regex. This is an unintended regression.

if _NON_SIMPLE_VERSION_INDICATORS.isdisjoint(version):
self._release = tuple(map(int, version.strip().split('.')))

Found by: Nemotron strategy search, Ultra's planned example, hostile mutation, search sample

ran onToken Factory Sandboxes · image dbe8dafe-78c4-4fc8-b8c2-a5adaad2ed49

Regression test verified passes on base, fails on head
def test_p2_outcome_0():
    """Breakwater P2: base behaviour pinned (found by differential testing; label: likely-bug)."""
    import pytest
    from packaging.version import InvalidVersion, Version
    with pytest.raises(InvalidVersion) as excinfo:
        Version('0 .0')
    assert type(excinfo.value) is InvalidVersion  # exact type: a subclass would be a behaviour change
Reproduce
uv run breakwater probe --repo https://github.com/pypa/packaging --base 59f183fe90da --head 61967cca278d --target packaging.version:Version --backend contree --input "0 .0"
1 near-duplicate class same behaviour, inputs with extra traits
  • Version("١ .2") base raises InvalidVersion · head returns normally test verified (also under another group)
P2

Exception behaviour changed

inputVersion("٠")
baseraises InvalidVersion
headreturns normally
Likely bugvotes: 3 likely-bug

The fast path accepts Arabic-Indic digits (e.g., '٠') because they are not in the indicator set, while the base version rejects them via the ASCII-only regex. This expands accepted characters unintentionally.

_NON_SIMPLE_VERSION_INDICATORS = frozenset('!+-ABCDEHILOPRSTVW_abcdehiloprstvw')
if _NON_SIMPLE_VERSION_INDICATORS.isdisjoint(version):

Found by: Nemotron strategy search, Ultra's planned example, hostile mutation

ran onToken Factory Sandboxes · image dbe8dafe-78c4-4fc8-b8c2-a5adaad2ed49

Regression test verified passes on base, fails on head
def test_p2_outcome_2():
    """Breakwater P2: base behaviour pinned (found by differential testing; label: likely-bug)."""
    import pytest
    from packaging.version import InvalidVersion, Version
    with pytest.raises(InvalidVersion) as excinfo:
        Version('٠')
    assert type(excinfo.value) is InvalidVersion  # exact type: a subclass would be a behaviour change
Reproduce
uv run breakwater probe --repo https://github.com/pypa/packaging --base 59f183fe90da --head 61967cca278d --target packaging.version:Version --backend contree --input "٠"
1 near-duplicate class same behaviour, inputs with extra traits
  • Version("١ .2") base raises InvalidVersion · head returns normally test verified (also under another group)
P2

Exception behaviour changed

inputVersion(["1"])
baseraises TypeError
headraises AttributeError
Likely bugvotes: 2 likely-bug, 1 needs-human

The fast path accepts full-width digits (e.g., '1') because they are not in the ASCII-only indicator set, but then an AttributeError occurs (likely due to incomplete initialization). The base version raises TypeError. This change in exception type and acceptance of non-ASCII digits is unintended.

_NON_SIMPLE_VERSION_INDICATORS = frozenset('!+-ABCDEHILOPRSTVW_abcdehiloprstvw')
if _NON_SIMPLE_VERSION_INDICATORS.isdisjoint(version):

Found by: Nemotron strategy search, search sample

ran onToken Factory Sandboxes · image dbe8dafe-78c4-4fc8-b8c2-a5adaad2ed49

Regression test verified passes on base, errors on head
def test_p2_outcome_1():
    """Breakwater P2: base behaviour pinned (found by differential testing; label: likely-bug)."""
    import pytest
    from packaging.version import Version
    with pytest.raises(TypeError) as excinfo:
        Version(['1'])
    assert type(excinfo.value) is TypeError  # exact type: a subclass would be a behaviour change
P2

Exception behaviour changed

inputVersion("100000000000000000000000000000000000000000…0000000000" (4,301 chars))
baseraises ValueError
headraises InvalidVersion
Intendedvotes: 2 intended, 1 needs-human

The fast path wraps ValueError from int() into InvalidVersion, matching the library's documented exception for invalid versions. The base version let ValueError escape. This makes error handling consistent and deliberate.

self._release = tuple(map(int, version.strip().split('.')))
except ValueError: raise InvalidVersion(f'Invalid version: {version!r}') from None

Found by: hostile mutation

ran onToken Factory Sandboxes · image dbe8dafe-78c4-4fc8-b8c2-a5adaad2ed49

No regression test: adjudicated intended: no test pins the old behaviour.

P2

Exception behaviour changed

inputVersion("100000000000000000000000000000000000000000…0000000 .2" (4,303 chars))
baseraises InvalidVersion
headreturns normally
Intendedvotes: 2 intended, 1 needs-human

The fast path is designed for all simple versions (only numbers and dots) without length limits. The base version's regex likely hit an internal recursion/length limit, causing InvalidVersion. The new behavior correctly accepts arbitrarily long simple versions per PEP 440.

# Fast path: If a version doesn't have one of these characters then
# it is a simple version with just numbers and dots (e.g. 1.2.3).

Found by: hostile mutation

ran onToken Factory Sandboxes · image dbe8dafe-78c4-4fc8-b8c2-a5adaad2ed49

No regression test: adjudicated intended: no test pins the old behaviour.

Regression tests

4 of 6 classes have a generated pytest that passed on base and failed on head in the sandbox.

Download test_regressions.py

Performance · Version

Worst family with mostly valid inputs: 2.74× the instructions of base (Trailing numeric sub-label added).

head slowerhead faster0.5×1×2×101001,00010,000100,000input size (characters)R4.2R4.2: 1.05× at ~14 charsR4.2: 1.08× at ~23 charsR4.2: 1.37× at ~161 charsR4.2: 2.17× at ~2,153 charsR4.2: 2.74× at ~33,161 charsR4.1R4.1: 1.04× at ~13 charsR4.1: 1.07× at ~22 charsR4.1: 1.35× at ~160 charsR4.1: 2.16× at ~2,152 charsR4.1: 2.74× at ~33,160 charsR4R4: 1.04× at ~16 charsR4: 1.06× at ~25 charsR4: 1.29× at ~163 charsR4: 1.98× at ~2,155 charsR4: 2.51× at ~33,163 charsR4: 2.15× at ~525,835 charsF1F1: 1.03× at ~9 charsF1: 1.04× at ~13 charsF1: 1.08× at ~37 charsF1: 1.12× at ~133 charsF1: 1.14× at ~517 charsF1: 1.14× at ~2,053 charsF1: 1.14× at ~8,197 charsF2F2: 0.38× at ~7 charsF2: 0.43× at ~11 charsF2: 0.62× at ~35 charsF2: 0.80× at ~131 charsF2: 0.88× at ~515 charsF2: 0.90× at ~2,051 charsF2: 0.91× at ~8,195 chars
  • R4.2
  • R4.1
  • R4
  • F1
  • F2
Head/base cachegrind instructions per call, by input size (log-log). Each line is one input family Nemotron proposed; points above 1× are where the "faster" code does more work.
FamilyInputsRatio range Slope base → headVerdict
R4.2Trailing numeric sub-label addedsuspected slowdown · expand:R41.05× – 2.74×0.57 → 0.70slower
R4.1Pre-release label variationsuspected slowdown · expand:R41.04× – 2.74×0.55 → 0.68slower
R4Pre-release label changed from 'c5' to 'alpha.1'suspected slowdown · archive1.04× – 2.51×0.69 → 0.78slower
F1Long release segment (many dot-separated numbers) ending with a pre‑release marker (e.g. "1.1.1…a1"). The new fast‑path scans the whole string looking for indicator characters before falling back to the regex, so the disjointness check walks the entire long numeric prefix before finding the trailing 'a'.suspected slowdown · plan1.03× – 1.14×0.80 → 0.82slower
F2many numeric segments, no special charactersclaimed speedup · plan0.38× – 0.91×0.79 → 0.92faster

Receipt

Model (Nebius Token Factory)Cost
Nemotron 3 Ultra$0.18
Nemotron 3.5 Lightning$0.001
Nemotron 3 Super$0.001
Total$0.18

rune780ff68654f

imagedbe8dafe-78c4-4fc8-b8c2-a5adaad2ed49

backendToken Factory Sandboxes (contree)

sandbox$11.92 Sandbox cost (free during beta) · 358 Token Factory Sandboxes runs · 1165.3s CPU

semantic plan 17s · semantic generate 9s · semantic search 12s · semantic batch 7s · perf plan 15s · perf screen 148s · perf full 146s