Breakwater

Backtest / #1108

perf: SpecifierSet use tuple instead of frozenset for _specs

pypa/packaging#1108 · merged 2026-03-07 · base 20f1523cf3 → head f59d6ee9af · ran on Docker

BLOCK

At least one behaviour change looks like an unintended regression.

Known regressions in this PR

Counterexamples · 8 groups of 15 classes: 12 likely bug, 3 needs a human

P1

Output changed

known regression: dedup-str-len
inputSpecifierSet(">=01.0,>=1.0")
baselen: 1
headlen: 2
Likely bugvotes: 3 likely-bug

The change deduplicates on raw specifier strings before parsing, so semantically equal specifiers like '>=01.0' and '>=1.0' are no longer merged. This changes __len__ and __str__ behavior (C1 shows len 1 → 2).

- self._specs = frozenset(map(Specifier, split_specifiers))
+ # Deduplicate strings first.

Found by: Nemotron strategy search, Ultra's planned example, hostile mutation · also caught by: Mutations of the PR's own test strings, Nemotron plan only

ran onlocal Docker · image bw-pair:16f20a2b68c37e6b

Regression test verified passes on base, fails on head
def test_1108_p1_len_0():
    """Breakwater P1: base behaviour pinned (found by differential testing; label: likely-bug)."""
    from packaging.specifiers import SpecifierSet
    assert len(SpecifierSet('>=01.0,>=1.0')) == 1
Reproduce
uv run breakwater probe --repo https://github.com/pypa/packaging --base 20f1523cf37d --head f59d6ee9afe2 --target packaging.specifiers:SpecifierSet --backend docker --input ">=01.0,>=1.0"
3 near-duplicate classes same behaviour, inputs with extra traits
  • SpecifierSet(">= 1.0,>=1.0") base len: 1 · head len: 2dedup-str-len test verified
  • SpecifierSet(" >=1.0,>=1.0.0 ") base len: 1 · head len: 2dedup-str-len test verified
  • SpecifierSet(">=1000000000000000000000000000000000000000….0.0,>=1.0" (4,318 chars)) base len: 2 · head len: 3dedup-str-len test verified
P1

Output changed

known regression: dedup-str-len
inputSpecifierSet(">=01.0,>=1.0")
basestr: ">=01.0"
headstr: ">=01.0,>=1.0"
Likely bugvotes: 3 likely-bug

__str__ now reflects the tuple order with duplicates retained, changing the string representation from '>=01.0' to '>=01.0,>=1.0'.

- self._specs = frozenset(map(Specifier, split_specifiers))
+ # Deduplicate strings first.

Found by: Nemotron strategy search, Ultra's planned example, hostile mutation · also caught by: Mutations of the PR's own test strings, Nemotron plan only

ran onlocal Docker · image bw-pair:16f20a2b68c37e6b

Regression test verified passes on base, fails on head
def test_1108_p1_str_4():
    """Breakwater P1: base behaviour pinned (found by differential testing; label: likely-bug)."""
    from packaging.specifiers import SpecifierSet
    assert str(SpecifierSet('>=01.0,>=1.0')) == '>=01.0'
Reproduce
uv run breakwater probe --repo https://github.com/pypa/packaging --base 20f1523cf37d --head f59d6ee9afe2 --target packaging.specifiers:SpecifierSet --backend docker --input ">=01.0,>=1.0"
3 near-duplicate classes same behaviour, inputs with extra traits
  • SpecifierSet(">= 1.0,>=1.0") base str: ">=1.0" · head str: ">=1.0,>=1.0"dedup-str-len test verified
  • SpecifierSet(" >=1.0,>=1.0.0 ") base str: ">=1.0" · head str: ">=1.0,>=1.0.0"dedup-str-len test verified
  • SpecifierSet(">=1000000000000000000000000000000000000000….0.0,>=1.0" (4,318 chars)) base str: ">=1.0.0,>=10000000000000000000000000000000…00000000.0" · head str: ">=1.0,>=1.0.0,>=10000000000000000000000000…00000000.0"dedup-str-len test verified
P3

Equality / ordering changed

known regression: dedup-equality
aSpecifierSet(">=01.0,>=1.0")
bSpecifierSet(">=1.0,>=1.0")
basea == b → True
heada == b → False
Likely bugvotes: 2 likely-bug

Equality semantics changed: the new __eq__ first checks length of the internal `_specs` tuple (which now reflects raw‑string deduplication) before comparing sorted specs. Two SpecifierSet objects that are semantically equal (e.g., `>=01.0,>=1.0` vs `>=1.0,>=1.0`) now compare False because their raw‑string tuples have different lengths. This is a regression introduced by the refactor.

if len(self._specs) != len(other._specs): return False
return self._sort_specs() == other._sort_specs()

Found by: Ultra's planned example, relations pass · also caught by: Nemotron plan only

ran onlocal Docker · image bw-pair:16f20a2b68c37e6b

Regression test verified passes on base, fails on head
def test_1108_p3_eq_11():
    """Breakwater P3: base behaviour pinned (found by differential testing; label: likely-bug)."""
    from packaging.specifiers import SpecifierSet
    a = SpecifierSet('>=01.0,>=1.0')
    b = SpecifierSet('>=1.0,>=1.0')
    assert (a == b) is True
P4

Old pickles break

known regression: old-pickles-break-comparison
inputSpecifierSet("")
basepickled object loads and compares equal
headcompare with fresh object: raises AttributeError
Likely bugvotes: 2 likely-bug

Same root cause as C6: the added `_sorted` slot is missing on a pickled legacy instance, leading to an AttributeError when the new code accesses it during comparison. This is an unintended regression in serialization compatibility.

__slots__ = (
'_prereleases',

Found by: Ultra's planned example, project's test strings, hostile mutation, search sample · also caught by: Plain Hypothesis, Mutations of the PR's own test strings, Nemotron plan only

ran onlocal Docker · image bw-pair:16f20a2b68c37e6b

Regression test verified passes on base, errors on head
def test_1108_p4_base_head_eq_fresh_14():
    """Breakwater P4: base behaviour pinned (found by differential testing; label: likely-bug)."""
    import base64
    import pickle

    from packaging.specifiers import SpecifierSet
    # pickled by the base version (lazy caches warmed first)
    old = pickle.loads(base64.b64decode('gASVUAAAAAAAAACMFHBhY2thZ2luZy5zcGVjaWZpZXJzlIwMU3BlY2lmaWVyU2V0lJOUKYGUTn2UKIwMX3ByZXJlbGVhc2VzlE6MBl9zcGVjc5QokZR1hpRiLg=='))
    assert old == SpecifierSet('')
Reproduce
uv run breakwater probe --repo https://github.com/pypa/packaging --base 20f1523cf37d --head f59d6ee9afe2 --target packaging.specifiers:SpecifierSet --backend docker --input ""
P4

Old pickles break

known regression: old-pickles-break-comparison
inputSpecifierSet(">=01.0,>=1.0")
basepickled object loads and compares equal
headcompare with fresh object: false
Likely bugvotes: 2 likely-bug

The refactor added a new slot `_sorted` (`__slots__ = (..., '_sorted', '_specs')`). Objects pickled by the old code lack this attribute; when unpickled by the new code and compared to a freshly constructed object, an AttributeError is raised. This breaks pickle compatibility.

__slots__ = (
'_prereleases',

Found by: Ultra's planned example, hostile mutation · also caught by: Mutations of the PR's own test strings, Nemotron plan only

ran onlocal Docker · image bw-pair:16f20a2b68c37e6b

Regression test verified passes on base, fails on head
def test_1108_p4_base_head_eq_fresh_13():
    """Breakwater P4: base behaviour pinned (found by differential testing; label: likely-bug)."""
    import base64
    import pickle

    from packaging.specifiers import SpecifierSet
    # pickled by the base version (lazy caches warmed first)
    old = pickle.loads(base64.b64decode('gASVEAEAAAAAAACMFHBhY2thZ2luZy5zcGVjaWZpZXJzlIwMU3BlY2lmaWVyU2V0lJOUKYGUTn2UKIwMX3ByZXJlbGVhc2VzlE6MBl9zcGVjc5QoaACMCVNwZWNpZmllcpSTlCmBlE59lChoBU6MBV9zcGVjlIwCPj2UjAQwMS4wlIaUjA1fc3BlY192ZXJzaW9ulGgNjBFwYWNrYWdpbmcudmVyc2lvbpSMB1ZlcnNpb26Uk5QpgZROfZQojARfZGV2lE6MBl9lcG9jaJRLAIwKX2tleV9jYWNoZZROjAZfbG9jYWyUTowFX3Bvc3SUTowEX3ByZZROjAhfcmVsZWFzZZRLAUsAhpR1hpRihpR1hpRikZR1hpRiLg=='))
    assert old == SpecifierSet('>=01.0,>=1.0')
Reproduce
uv run breakwater probe --repo https://github.com/pypa/packaging --base 20f1523cf37d --head f59d6ee9afe2 --target packaging.specifiers:SpecifierSet --backend docker --input ">=01.0,>=1.0"
P3

Equality / ordering changed

aSpecifierSet(">=01.0,>=1.0")
bSpecifierSet(">=1.0,>=1.0")
basehash(a) == hash(b) → True
headhash(a) == hash(b) → False
Likely bugvotes: 2 likely-bug

Hash now uses the sorted `_specs` tuple (`hash(self._sort_specs())`). Because the new constructor deduplicates on raw strings, two semantically equal SpecifierSet objects can have different tuple lengths/contents, causing `hash(a) != hash(b)`. This breaks the hash‑equals contract.

return hash(self._sort_specs())
def __hash__(self) -> int: return hash(self._sort_specs())

Found by: Ultra's planned example, relations pass · also caught by: Nemotron plan only

ran onlocal Docker · image bw-pair:16f20a2b68c37e6b

Regression test verified passes on base, fails on head
def test_1108_p3_hash_eq_12():
    """Breakwater P3: base behaviour pinned (found by differential testing; label: likely-bug)."""
    from packaging.specifiers import SpecifierSet
    a = SpecifierSet('>=01.0,>=1.0')
    b = SpecifierSet('>=1.0,>=1.0')
    assert (hash(a) == hash(b)) is True
P2

Exception behaviour changed

inputSpecifierSet("<=1000000000000000000000000000000000000000…000000!0.1" (4,307 chars))
baseraises ValueError
headreturns normally
Needs a humanvotes: 2 needs-human (model abstained)

Base raises ValueError on a very large “<=” specifier; head accepts it. The PR does not mention changing version‑parsing semantics, so the reviewer should decide whether the new permissive behaviour is desired.

Found by: hostile mutation · also caught by: Mutations of the PR's own test strings

ran onlocal Docker · image bw-pair:16f20a2b68c37e6b

Regression test verified passes on base, fails on head
def test_1108_p2_outcome_10():
    """Breakwater P2: base behaviour pinned (found by differential testing; label: needs-human)."""
    import pytest
    from packaging.specifiers import SpecifierSet
    with pytest.raises(ValueError) as excinfo:
        SpecifierSet('<=10000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000!0.1')
    assert type(excinfo.value) is ValueError  # exact type: a subclass would be a behaviour change
P2

Exception behaviour changed

inputSpecifierSet(">=1000000000000000000000000000000000000000…00.0,>=1.0" (4,311 chars))
baseraises AssertionError
headreturns normally
Needs a humanvotes: 2 needs-human (model abstained)

Base raises AssertionError on a very large version string while head returns normally. The PR is described as a performance refactor with no mention of fixing large‑version parsing. The change in behavior is undocumented, so a human should decide whether the new permissive parsing is intended.

Found by: hostile mutation · also caught by: Mutations of the PR's own test strings

ran onlocal Docker · image bw-pair:16f20a2b68c37e6b

Regression test verified passes on base, fails on head
def test_1108_p2_outcome_8():
    """Breakwater P2: base behaviour pinned (found by differential testing; label: needs-human)."""
    import pytest
    from packaging.specifiers import SpecifierSet
    with pytest.raises(AssertionError) as excinfo:
        SpecifierSet('>=10000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000.0,>=1.0')
    assert type(excinfo.value) is AssertionError  # exact type: a subclass would be a behaviour change
1 near-duplicate class same behaviour, inputs with extra traits
  • SpecifierSet(">=1000000000000000000000000000000000000000…0, >=1.0.0" (4,314 chars)) base raises AssertionError · head returns normally test verified

Regression tests

15 of 15 classes have a generated pytest that pins the old behaviour. Each passed on the base commit and failed on the head commit in the sandbox before it was offered.

Download test_1108.py

Performance · SpecifierSet

No realistic input family measured slower.

Measured in backtest packaging-v2 (same commits, run bt-1108-c01604, image bw-pair:47497edc9ebcda96). The semantic results above are from a later run; perf doesn't use the semantic planner, so it was not re-measured.

head slowerhead faster0.35×0.42×0.5×0.59×0.71×0.84×1×1001,00010,000input size (characters)F1F1: 0.43× at ~19 charsF1: 0.39× at ~35 charsF1: 0.37× at ~137 charsF1: 0.37× at ~569 charsF1: 0.35× at ~2,453 charsF1: 0.37× at ~10,157 charsF1: 0.36× at ~43,949 chars
  • F1
Head/base cachegrind instructions per call, by input size (log-log). Each line is one input family Nemotron proposed; points above 1× are where the "faster" code does more work.
FamilyInputsRatio range Slope base → headVerdict
F1small_specifier_setclaimed speedup · plan0.35× – 0.43×0.99 → 0.97faster

Receipt

TargetBackendImageRun AuditAdjudication
packaging.specifiers:SpecifierSetlocal Dockerbw-pair:16f20a2b68c37e6bbt-1108-2f9809$0.050$0.24

Nemotron cost on Nebius Token Factory. Backend: where the untrusted code ran. local Docker (an image cited by its content-addressed tag). Every image is content-addressed (repo at each commit + harness), so every card re-runs from the image and seed 0. Raw data: audit, adjudication.