{ "corpus": "corpus/packaging.json", "arms": [ "hypothesis", "mutations", "llm-plan", "breakwater" ], "items": [ { "id": "#997", "pr": 997, "title": "perf: better string building", "base": "1c4e8056b1c0b2efc9c68a71c7f2a1a1920db8f3", "head": "65092ce350ac2cc73e61d770f5d16a0957bf6e45", "merged": "2026-01-06", "after_ultra_release": false, "known": [], "targets": [ { "target": "packaging.version:Version", "run_id": "bt-997-27f243", "image": "bw-pair:d358a5c896312bf0", "usd": 0.0601, "harvested": 80, "arms": { "hypothesis": { "classes": [], "known": [], "seconds": 8.1, "error": null }, "mutations": { "classes": [], "known": [], "seconds": 3.6, "error": null }, "breakwater": { "classes": [], "known": [], "seconds": 55.4, "error": null }, "llm-plan": { "classes": [], "known": [], "seconds": 1.9, "error": null } }, "findings": [], "perf": { "slower_families": [], "worst_valid_ratio": null, "worst_family": null, "speedup_ratio_small": 0.9993014544274319, "report": { "run_id": "bt-997-27f243", "target": "packaging.version:Version", "image": "bw-pair:d358a5c896312bf0", "backend": "docker", "families": [ { "id": "F1", "kind": "suspected_slowdown", "family": "Versions that contain *all* optional components (epoch, pre‑release, post‑release, dev‑release and a local segment) *and* a growing number of release segments. Each extra release segment makes the base string longer; the new `__str__` builds the result by repeated `+=` concatenations, copying the whole string on every step, while the old implementation collected the parts in a list and performed a single `join`.", "why": "The head version does up to five string concatenations (`version += …`). With a release segment of length O(n) each concatenation copies O(n) characters → total work O(k·n) (k ≈ 5). The base version does one `join` over the same parts → O(n).", "size_meaning": "n = number of release segments (each a small integer). Larger n ⇒ longer intermediate strings ⇒ more copying.", "make_source": "def make(n):\n # n >= 1 release components, all optional markers present\n release = '.'.join(str(i + 1) for i in range(n))\n # epoch=1, pre=a1, post=2, dev=3, local=localpart\n version = f'1!{release}.a1.post2.dev3+localpart'\n return [[version]]", "example_args": [], "origin": "plan", "screen_ratio": 1.0079740367742938, "valid_fraction": 1.0, "error": null }, { "id": "F2", "kind": "suspected_slowdown", "family": "Versions with a *very large* local version segment (many dot/underscore/hyphen separated parts) but otherwise minimal. The local segment is built once by the `local` property, then the new `__str__` appends it with `version += f\"+{self.local}\"`, copying the whole public version *again*. The old code added the local part to a list and performed a single final `join`, copying the data only once.", "why": "Extra full‑string copy of length O(|public| + |local|). When |local| grows with n, the head version does O(|public| + |local|) work twice; the base version does it once.", "size_meaning": "n = number of components in the local segment. Larger n ⇒ larger local string ⇒ more data copied twice.", "make_source": "def make(n):\n # n >= 1 local components, simple public version 1.0\n local = '.'.join(f'part{i}' for i in range(n))\n version = f'1.0+{local}'\n return [[version]]", "example_args": [], "origin": "plan", "screen_ratio": 1.0075610518239337, "valid_fraction": 1.0, "error": null }, { "id": "F3", "kind": "claimed_speedup", "family": "Plain versions that have *only* a release segment (any length) and no epoch, pre, post, dev or local parts. The change was motivated by avoiding the `self.base_version` property access; the new code builds the release string directly with a single `join` and returns it.", "why": "Both versions perform one `join` over the release tuple, but the head version skips the property lookup and the extra `if self.epoch` branch, giving a tiny constant‑factor win for the common case of simple versions.", "size_meaning": "n = number of release segments. Cost grows linearly with n in both versions, but the head version has a smaller constant overhead.", "make_source": "def make(n):\n # n >= 1 release components, no optional markers\n release = '.'.join(str(i + 1) for i in range(n))\n return [[release]]", "example_args": [ [ 1 ], [ 1.2 ] ], "origin": "plan", "screen_ratio": 1.0009330102855902, "valid_fraction": 1.0, "error": null }, { "id": "F4", "kind": "claimed_speedup", "family": "The patch replaces building a list of string parts and joining them with a single string that is built by a single join for the release segment and then a handf", "why": "The patch replaces building a list of string parts and joining them with a single string that is built by a single join for the release segment and then a handful of += concatenations for epoch, post, dev and local parts. The comment explicitly marks the function as a hot path and avoids a method call to self.base_version, reducing function‑call overhead and the allocation of an intermediate list.", "size_meaning": "expected reduction of a few microseconds per Version.__str__ call in tight packaging loops", "make_source": "\ndef make(n):\n result = []\n for i in range(1, n + 1):\n major = 1 + (i % 20)\n minor = 1 + ((i // 20) % 20)\n patch = 1 + ((i // 400) % 20)\n pre = itertools.cycle(('a', 'b', 'rc')).__next__() if i % 3 else ''\n local = ''.join(random.choices(string.ascii_lowercase + string.digits, k=random.randint(1, 4)))\n args = [\n major, minor, patch,\n pre,\n 'post' if i % 2 == 0 else '',\n local\n ]\n result.append(args)\n return result", "example_args": [], "origin": "plan", "screen_ratio": null, "valid_fraction": null, "error": "make(n) rejected: attribute '__next__' is not allowed" }, { "id": "F5", "kind": "claimed_speedup", "family": "The diff adds a comment stating \"This is a hot function, so not calling self.base_version\" and rewrites the version string construction to avoid a method call a", "why": "The diff adds a comment stating \"This is a hot function, so not calling self.base_version\" and rewrites the version string construction to avoid a method call and use a single join plus incremental string concatenations, which the author marks as a hot‑function optimization.", "size_meaning": "claimed speed‑up", "make_source": "\ndef make(n):\n combos = list(itertools.product(string.ascii_lowercase, repeat=2))\n random.shuffle(combos)\n args = []\n for _ in range(min(n, 8)):\n a = ''.join(random.choice(combos) for _ in range(random.randint(1, 6)))\n b = ''.join(random.choice(string.digits) for _ in range(random.randint(1, 4)))\n args.append([a, b])\n return args", "example_args": [], "origin": "plan", "screen_ratio": null, "valid_fraction": null, "error": "make(2): TypeError: sequence item 0: expected str instance, tuple found" }, { "id": "F6", "kind": "suspected_slowdown", "family": "The new code builds the final version string by repeatedly using += on a Python string (epoch prefix, post, dev, local). Even though there are only a few concat", "why": "The new code builds the final version string by repeatedly using += on a Python string (epoch prefix, post, dev, local). Even though there are only a few concatenations, each += creates a new string object, which can be slightly slower than building a list of parts and doing a single ''.join(parts) as the old code did.", "size_meaning": "possible few‑microsecond slowdown per Version.__str__ call when many post/dev/local segments are present", "make_source": "def make(n):\n parts = []\n # epoch prefix\n parts += [str(n % 100)]\n # post segment\n parts += [f\"post{random.randint(0, 99):02d}\" if n % 2 else \"\"]\n # dev segment\n parts += [f\"dev{random.getrandbits(8):02x}\" if n % 3 else \"\"]\n # local segment\n parts += [f\"local{random.randint(0, 0xFFFF):04x}\" if n % 4 else \"\"]\n # keep only non-empty parts, at least one\n parts = [p for p in parts if p] or [\"0.0.0\"]\n # build string with repeated += as per the target\n s = \"\"\n for p in parts:\n s += p\n return [list(parts), s]", "example_args": [], "origin": "plan", "screen_ratio": 1.0016997099290488, "valid_fraction": 0.0, "error": null }, { "id": "R1", "kind": "suspected_slowdown", "family": "Invalid version strings that nevertheless contain indicator characters (e.g. a trailing dot followed by a letter). The fast‑path still does the full scan before the regex fails.", "why": "Even though the version is invalid, the extra disjoint check runs; the regex then fails. This is the single allowed invalid family.", "size_meaning": "Number of dotted segments before the malformed tail.", "make_source": "def make(n):\n # \"1.\" repeated n times then an alphabetic tail – not a valid PEP‑440 version\n version = \"1.\" * n + \"x\"\n return [(version,)]", "example_args": [], "origin": "archive", "screen_ratio": 1.0022514288258109, "valid_fraction": 0.0, "error": null }, { "id": "R2", "kind": "suspected_slowdown", "family": "Single repeated segment \"999.\" n times plus prerelease c5", "why": "The fast‑path’s `isdisjoint` test iterates over every character. For a long dotted release the string length grows with n, so the extra scan adds linear overhead before the (already linear) regex match.", "size_meaning": "Number of dotted release segments (plus the trailing pre‑release).", "make_source": "def make(n):\n release = \".\".join([\"999\"] * n)\n version = f\"{release}c5\"\n return [(version,)]", "example_args": [], "origin": "archive", "screen_ratio": 1.0008068536348251, "valid_fraction": 1.0, "error": null }, { "id": "R3", "kind": "suspected_slowdown", "family": "Versions with many release segments and a pre‑release suffix (e.g. \"1.2.3…a1\"). The fast‑path scans the whole string looking for indicator characters, finds the “a”, then falls back to the regex parser – an extra O(n) pass over the input.", "why": "The fast‑path’s `isdisjoint` test iterates over every character. For a long dotted release the string length grows with n, so the extra scan adds linear overhead before the (already linear) regex match.", "size_meaning": "Number of dotted release segments (plus the trailing pre‑release).", "make_source": "def make(n):\n # Build \"1.2.3…n\" then add a pre‑release marker \"a1\"\n release = \".\".join(str(i) for i in range(1, n + 1))\n version = f\"{release}a1\"\n return [(version,)]", "example_args": [], "origin": "archive", "screen_ratio": 1.0008764298629533, "valid_fraction": 1.0, "error": null }, { "id": "R4", "kind": "suspected_slowdown", "family": "Many numeric release segments followed by 'b' pre-release", "why": "The fast‑path’s `isdisjoint` test iterates over every character. For a long dotted release the string length grows with n, so the extra scan adds linear overhead before the (already linear) regex match.", "size_meaning": "Number of dotted release segments (plus the trailing pre‑release).", "make_source": "def make(n):\n release = \".\".join(str(i) for i in range(1, n + 1))\n version = f\"{release}b1\"\n return [(version,)]", "example_args": [], "origin": "archive", "screen_ratio": 1.00087146530454, "valid_fraction": 1.0, "error": null } ], "findings": [ { "family_id": "F3", "kind": "claimed_speedup", "family": "Plain versions that have *only* a release segment (any length) and no epoch, pre, post, dev or local parts. The change was motivated by avoiding the `self.base_version` property access; the new code builds the release string directly with a single `join` and returns it.", "why": "Both versions perform one `join` over the release tuple, but the head version skips the property lookup and the extra `if self.epoch` branch, giving a tiny constant‑factor win for the common case of simple versions.", "origin": "plan", "screen_ratio": 1.0009330102855902, "valid_fraction": 1.0, "ladder": [ { "n": 2, "base_ir": 23901.05, "head_ir": 23945.05, "input_chars": 7.0, "ratio": 1.0018409233067167, "valid_fraction": 1.0, "judged": true }, { "n": 4, "base_ir": 26847.06, "head_ir": 26891.06, "input_chars": 11.0, "ratio": 1.0016389131621861, "valid_fraction": 1.0, "judged": true }, { "n": 16, "base_ir": 47159.18, "head_ir": 47203.18, "input_chars": 42.0, "ratio": 1.0009330102855902, "valid_fraction": 1.0, "judged": true }, { "n": 64, "base_ir": 133913.3396226415, "head_ir": 133957.3396226415, "input_chars": 186.0, "ratio": 1.0003285707019478, "valid_fraction": 1.0, "judged": true }, { "n": 256, "base_ir": 491892.2380952381, "head_ir": 492612.2380952381, "input_chars": 919.0, "ratio": 1.00146373523353, "valid_fraction": 1.0, "judged": true }, { "n": 1024, "base_ir": 1917197.75, "head_ir": 1915858.5, "input_chars": 4016.0, "ratio": 0.9993014544274319, "valid_fraction": 1.0, "judged": true }, { "n": 4096, "base_ir": 7757154.5, "head_ir": 7755067.0, "input_chars": 19376.0, "ratio": 0.9997308935899111, "valid_fraction": 1.0, "judged": true } ], "slope_base": 0.7384625253416194, "slope_head": 0.7381965105195819, "top_inputs": [], "error": null, "verdict": "neutral", "worst_ratio": 1.0018409233067167, "worst_search": null } ], "seconds": { "plan": 18.39, "screen": 132.1, "full": 74.57, "total": 225.06 }, "ladder_validity": "backfilled 2026-10-11 (scripts/backtest/backfill_ladder_validity.py)" } } } ] }, { "id": "#1028", "pr": 1028, "title": "perf: don't renormalize extras", "base": "65092ce350ac2cc73e61d770f5d16a0957bf6e45", "head": "b20096ccce1ca389fa1d3c4290065ab4ede370cc", "merged": "2026-01-06", "after_ultra_release": false, "known": [], "targets": [ { "target": "packaging.requirements:Requirement", "run_id": "bt-1028-74aaa2", "image": "bw-pair:1777b4e053ae8975", "usd": 0.0518, "harvested": 80, "arms": { "hypothesis": { "classes": [], "known": [], "seconds": 9.8, "error": null }, "mutations": { "classes": [], "known": [], "seconds": 3.5, "error": null }, "breakwater": { "classes": [], "known": [], "seconds": 46.6, "error": null }, "llm-plan": { "classes": [], "known": [], "seconds": 4.5, "error": null } }, "findings": [], "perf": { "slower_families": [], "worst_valid_ratio": null, "worst_family": null, "speedup_ratio_small": 1.0018026475518533, "report": { "run_id": "bt-1028-74aaa2", "target": "packaging.requirements:Requirement", "image": "bw-pair:1777b4e053ae8975", "backend": "docker", "families": [ { "id": "F1", "kind": "claimed_speedup", "family": "requirements with many extra markers", "why": "The new `_normalize` skips `canonicalize_name` for the \"extra\" key, so each extra marker avoids two name normalizations. The speedup grows with the number of extras.", "size_meaning": "n = number of extras in the requirement", "make_source": "def make(n):\n extras = ','.join(f'extra{i}' for i in range(1, n + 1))\n return [[f'pkg[{extras}]']]", "example_args": [ [ "pkg[extra1]" ], [ "pkg[extra1,extra2]" ], [ "pkg[extra1,extra2,extra3]" ] ], "origin": "plan", "screen_ratio": 1.0070553204761523, "valid_fraction": 1.0, "error": null }, { "id": "F2", "kind": "claimed_speedup", "family": "The diff removes a double canonicalize_name call for the \"extra\" key in _normalize and skips normalisation when the value is empty. A requirement whose marker ", "why": "The diff removes a double canonicalize_name call for the \"extra\" key in _normalize and skips normalisation when the value is empty. A requirement whose marker expression contains many \"extra == 'value'\" comparisons exercises the fast path (the key is \"extra\" and the value is non‑empty) and therefore should be faster in the head version.", "size_meaning": "Number of \"extra == 'value'\" clauses combined with \"and\" in the marker expression.", "make_source": "\ndef make(n):\n # Build up to 8 requirement strings, each with an increasing number of extra clauses.\n # Each string is a valid requirement: a name followed by a marker expression.\n out = []\n for i in range(1, min(n, 8) + 1):\n clauses = \" and \".join(f\"extra == 'val{j}'\" for j in range(i))\n req = f\"pkg ; {clauses}\"\n out.append([req])\n return out", "example_args": [ [ "pkg ; extra == 'val0'" ], [ "pkg ; extra == 'val0' and extra == 'val1'" ] ], "origin": "plan", "screen_ratio": 1.0018026475518533, "valid_fraction": 1.0, "error": null }, { "id": "F3", "kind": "claimed_speedup", "family": "Requirement strings with a large number of extra dependencies (e.g. `pkg[extra1,extra2,…,extraN]`). The old code canonicalised every extra name during parsing; the new code skips that work.", "why": "During `_normalize_extra_values` the old `_normalize` implementation called `canonicalize_name` for each side of every \"extra\" marker comparison. The head version returns the values unchanged, assuming they are already normalised, so the per‑extra cost disappears.", "size_meaning": "n = number of extras in the bracket list", "make_source": "def make(n):\n # Build a requirement like pkg[extra0,extra1,...,extra{n-1}]\n extras = ','.join(f'extra{i}' for i in range(n))\n return [[f'myproj[{extras}]']]", "example_args": [ [ "myproj[extra0,extra1,extra2]" ], [ "myproj[extra0,extra1,extra2,extra3,extra4]" ] ], "origin": "plan", "screen_ratio": 1.007054793144669, "valid_fraction": 1.0, "error": null }, { "id": "F4", "kind": "suspected_slowdown", "family": "requirements with many non‑extra environment markers", "why": "The added `if key == \"extra\":` branch is evaluated for every marker clause. For clauses that are not \"extra\" the code still falls through to the generic `MARKERS_ALLOWING_SET` path, so each clause pays an extra branch test without any saved work.", "size_meaning": "n = number of non‑extra marker clauses joined with \" and \"", "make_source": "def make(n):\n clauses = ' and '.join(f\"os_name=='posix{i}'\" for i in range(1, n + 1))\n return [[f'pkg ; {clauses}']]", "example_args": [ [ "pkg ; os_name=='posix1'" ], [ "pkg ; os_name=='posix1' and os_name=='posix2'" ], [ "pkg ; os_name=='posix1' and os_name=='posix2' and os_name=='posix3'" ] ], "origin": "plan", "screen_ratio": 1.0000289371461049, "valid_fraction": 1.0, "error": null }, { "id": "F5", "kind": "suspected_slowdown", "family": "The new code unconditionally executes the `if \"extra\" in current_environment:` block for every marker evaluation, canonicalising the (often empty) extra value. ", "why": "The new code unconditionally executes the `if \"extra\" in current_environment:` block for every marker evaluation, canonicalising the (often empty) extra value. The old version only touched the extra key when it was explicitly `None`. Any requirement that includes a marker (even one unrelated to \"extra\") now pays this extra work.", "size_meaning": "Number of distinct requirement strings each containing a marker expression (no \"extra\" clauses).", "make_source": "\ndef make(n):\n out = []\n markers = [\"python_version>='3.6'\", \"sys_platform == 'linux'\", \"implementation_name == 'cpython'\"]\n for i in range(min(n, 8)):\n req = f\"pkg ; {markers[i % len(markers)]}\"\n out.append([req])\n return out", "example_args": [ [ "pkg ; python_version>='3.6'" ], [ "pkg ; sys_platform == 'linux'" ] ], "origin": "plan", "screen_ratio": 1.0056643064080788, "valid_fraction": 1.0, "error": null }, { "id": "F6", "kind": "suspected_slowdown", "family": "Requirement strings that contain a marker with an `extra == \"value\"` comparison (e.g. `pkg; extra == \"dev\"`). The new `_normalize` no longer canonicalises the left‑ and right‑hand sides, so the first evaluation of the marker must canonicalise the value at run‑time via `canonicalize_name`, adding work that the old version performed once during parsing.", "why": "Old path: `_normalize` canonicalised both sides when the AST was built, so later evaluation only did a cheap string compare. New path: the canonicalisation is deferred to `Marker.__call__`, which runs each time the marker is evaluated (e.g. during environment resolution). For a single requirement this adds a `canonicalize_name` call on the extra value.", "size_meaning": "n = length of the extra value string (more characters ⇒ more work in canonicalise)", "make_source": "def make(n):\n # Build a requirement with a marker that checks an extra of length n\n val = 'x' * n\n return [[f'myproj; extra == \"{val}\"']]", "example_args": [], "origin": "plan", "screen_ratio": 1.0054347403856958, "valid_fraction": 1.0, "error": null } ], "findings": [ { "family_id": "F2", "kind": "claimed_speedup", "family": "The diff removes a double canonicalize_name call for the \"extra\" key in _normalize and skips normalisation when the value is empty. A requirement whose marker ", "why": "The diff removes a double canonicalize_name call for the \"extra\" key in _normalize and skips normalisation when the value is empty. A requirement whose marker expression contains many \"extra == 'value'\" comparisons exercises the fast path (the key is \"extra\" and the value is non‑empty) and therefore should be faster in the head version.", "origin": "plan", "screen_ratio": 1.0018026475518533, "valid_fraction": 1.0, "ladder": [ { "n": 2, "base_ir": 274910.86, "head_ir": 276340.36, "input_chars": 35.0, "ratio": 1.0051998673315417, "valid_fraction": 1.0, "judged": true }, { "n": 4, "base_ir": 399902.1350574713, "head_ir": 401916.1982758621, "input_chars": 55.0, "ratio": 1.0050363902610857, "valid_fraction": 1.0, "judged": true }, { "n": 16, "base_ir": 654986.605, "head_ir": 656167.315, "input_chars": 95.0, "ratio": 1.0018026475518533, "valid_fraction": 1.0, "judged": true }, { "n": 64, "base_ir": 654986.605, "head_ir": 656167.315, "input_chars": 95.0, "ratio": 1.0018026475518533, "valid_fraction": 1.0, "judged": true }, { "n": 256, "base_ir": 654986.605, "head_ir": 656167.315, "input_chars": 95.0, "ratio": 1.0018026475518533, "valid_fraction": 1.0, "judged": true }, { "n": 1024, "base_ir": 654986.605, "head_ir": 656167.315, "input_chars": 95.0, "ratio": 1.0018026475518533, "valid_fraction": 1.0, "judged": true }, { "n": 4096, "base_ir": 654986.605, "head_ir": 656167.315, "input_chars": 95.0, "ratio": 1.0018026475518533, "valid_fraction": 1.0, "judged": true } ], "slope_base": null, "slope_head": null, "top_inputs": [], "error": null, "verdict": "neutral", "worst_ratio": 1.0051998673315417, "worst_search": null } ], "seconds": { "plan": 16.89, "screen": 194.91, "full": 220.91, "total": 432.71 }, "ladder_validity": "backfilled 2026-10-11 (scripts/backtest/backfill_ladder_validity.py)" } }, "rescored": "eq-hash-contract class removed: base also violates it (probe fix, deterministic re-check)" } ] }, { "id": "#1029", "pr": 1029, "title": "perf: use fullmatch for markers too", "base": "c39b6da9ab11f649f5497a11628f9dfc98380bb2", "head": "8a805e3baac2d71958ec0d0beffbe4d51fd5795f", "merged": "2026-01-04", "after_ultra_release": false, "known": [], "targets": [ { "target": "packaging.markers:Marker", "run_id": "bt-1029-8689c7", "image": "bw-pair:d03dcb2e7566ace6", "usd": 0.2324, "harvested": 80, "arms": { "hypothesis": { "classes": [], "known": [], "seconds": 10.9, "error": null }, "mutations": { "classes": [], "known": [], "seconds": 5.9, "error": null }, "breakwater": { "classes": [], "known": [], "seconds": 85.5, "error": null }, "llm-plan": { "classes": [], "known": [], "seconds": 9.7, "error": null } }, "findings": [], "perf": { "slower_families": [], "worst_valid_ratio": null, "worst_family": null, "speedup_ratio_small": 0.9993694403556155, "report": { "run_id": "bt-1029-8689c7", "target": "packaging.markers:Marker", "image": "bw-pair:d03dcb2e7566ace6", "backend": "docker", "families": [ { "id": "F1", "kind": "claimed_speedup", "family": "many short extra clauses combined with 'or'", "why": "The change only touches the regex used for name validation and normalization. Typical short extra names (≤10 chars) exercise the fast path of the regex; the new pattern avoids the start/end anchors and uses fullmatch, which for short inputs is marginally faster. This family exercises the common case of evaluating a marker that contains many simple extra checks.", "size_meaning": "n = number of extra clauses (each clause is a constant‑size string)", "make_source": "def make(n):\n # Build a marker like: extra == \"pkg0\" or extra == \"pkg1\" or ...\n clauses = [f'extra == \"pkg{i}\"' for i in range(n)]\n marker = ' or '.join(clauses)\n return [[marker]]", "example_args": [], "origin": "plan", "screen_ratio": 0.9993875966608451, "valid_fraction": 1.0, "error": null }, { "id": "F2", "kind": "suspected_slowdown", "family": "single extra with a very long alphanumeric name", "why": "The normalized‑name regex `_normalized_regex = re.compile(r\"[a-z0-9]|[a-z0-9]([a-z0-9-](?!--))*[a-z0-9]\")` is used via `fullmatch` on the canonicalized value. For a long string consisting only of alphanumerics the engine first tries the one‑character alternative, fails, then backtracks through the greedy `*` in the second alternative, yielding roughly O(L²) work where L is the name length. The old anchored regex with `match` exhibited the same worst case but the new pattern lacks the `^…$` anchors, so the engine cannot short‑circuit the first alternative and does more backtracking.", "size_meaning": "n = length of the extra name (number of alphanumeric characters)", "make_source": "def make(n):\n # One extra clause with a name of length n (all 'a')\n name = 'a' * n\n marker = f'extra == \"{name}\"'\n return [[marker]]", "example_args": [ [ [ "extra == \"aaaaaaaaaa\"" ] ] ], "origin": "plan", "screen_ratio": 0.9993665783373354, "valid_fraction": 1.0, "error": null }, { "id": "F3", "kind": "suspected_slowdown", "family": "single extra with a long name containing many permitted separators", "why": "The canonicalisation step replaces runs of `[-_.]` with a single hyphen, but the normalized‑name regex still has to `fullmatch` the resulting string. A name like `a.-_a.-_a…` becomes a long hyphen‑separated sequence (`a-a-a-…`). The same quadratic backtracking as the pure‑alnum case occurs, and the extra substitution pass adds linear work, making this family slower than the base version.", "size_meaning": "n = number of “a.-_” segments (final length ≈ 3 n)", "make_source": "def make(n):\n segment = 'a.-_'\n name = segment * n + 'a' # ensure it ends with an alnum\n marker = f'extra == \"{name}\"'\n return [[marker]]", "example_args": [ [ [ "extra == \"a.-_a.-_a.-_a\"" ] ] ], "origin": "plan", "screen_ratio": 0.998962874631169, "valid_fraction": 1.0, "error": null }, { "id": "F4", "kind": "suspected_slowdown", "family": "many extra clauses each of moderate length", "why": "When a marker contains many `extra == \"…\"` clauses combined with `or`, `_evaluate_markers` evaluates each clause sequentially until one succeeds. Each clause triggers `canonicalize_name` (which runs the normalized‑name `fullmatch`). With *k* clauses of length *m* the total work is *k·m²* in the worst case. The new regex makes the per‑clause cost higher, so the overall marker evaluation slows down proportionally to the number of clauses.", "size_meaning": "n = number of clauses (each clause length ≈ 20)", "make_source": "def make(n):\n base = 'extra == \"pkg' + 'x'*15 + '\"'\n clauses = [base.replace('pkg', f'pkg{i}') for i in range(n)]\n marker = ' or '.join(clauses)\n return [[marker]]", "example_args": [ [ [ "extra == \"pkg0xxxxxxxxxxxxxxx\" or extra == \"pkg1xxxxxxxxxxxxxxx\"" ] ] ], "origin": "plan", "screen_ratio": 1.0005435753428675, "valid_fraction": 1.0, "error": null }, { "id": "F5", "kind": "suspected_slowdown", "family": "extra name containing characters that are not allowed (invalid input)", "why": "An invalid name still passes through `canonicalize_name`: the substitution step runs, then `fullmatch` on the normalized regex fails after scanning the whole string. Because the new pattern lacks the start/end anchors, the engine cannot reject early and must explore both alternatives fully, making the failure path slower. Only one invalid family is included as requested.", "size_meaning": "n = length of the invalid name (all ‘!’ characters)", "make_source": "def make(n):\n name = '!' * n\n marker = f'extra == \"{name}\"'\n return [[marker]]", "example_args": [ [ [ "extra == \"!!!!!!!!!!\"" ] ] ], "origin": "plan", "screen_ratio": 0.9993663375120074, "valid_fraction": 1.0, "error": null } ], "findings": [ { "family_id": "F1", "kind": "claimed_speedup", "family": "many short extra clauses combined with 'or'", "why": "The change only touches the regex used for name validation and normalization. Typical short extra names (≤10 chars) exercise the fast path of the regex; the new pattern avoids the start/end anchors and uses fullmatch, which for short inputs is marginally faster. This family exercises the common case of evaluating a marker that contains many simple extra checks.", "origin": "plan", "screen_ratio": 0.9993875966608451, "valid_fraction": 1.0, "ladder": [ { "n": 2, "base_ir": 237883.92, "head_ir": 237733.92, "input_chars": 42.0, "ratio": 0.9993694403556155, "valid_fraction": 1.0, "judged": true }, { "n": 4, "base_ir": 466258.845, "head_ir": 465968.845, "input_chars": 84.0, "ratio": 0.9993780278849187, "valid_fraction": 1.0, "judged": true }, { "n": 16, "base_ir": 1845189.1551724137, "head_ir": 1844059.1551724137, "input_chars": 342.0, "ratio": 0.9993875966608451, "valid_fraction": 1.0, "judged": true }, { "n": 64, "base_ir": 7494691.071428572, "head_ir": 7529507.071428572, "input_chars": 1398.0, "ratio": 1.0046454216282144, "valid_fraction": 1.0, "judged": true }, { "n": 256, "base_ir": 30030132.333333332, "head_ir": 30030132.333333332, "input_chars": 5778.0, "ratio": 1.0, "valid_fraction": 1.0, "judged": true }, { "n": 1024, "base_ir": 120551295.0, "head_ir": 121118134.5, "input_chars": 23466.0, "ratio": 1.0047020606456363, "valid_fraction": 1.0, "judged": true }, { "n": 4096, "base_ir": 487335634.0, "head_ir": 487335280.0, "input_chars": 97194.0, "ratio": 0.9999992736012405, "valid_fraction": 1.0, "judged": true } ], "slope_base": 0.9851634174387901, "slope_head": 0.9855079466610813, "top_inputs": [], "error": null, "verdict": "neutral", "worst_ratio": 1.0047020606456363, "worst_search": null } ], "seconds": { "plan": 62.64, "screen": 176.5, "full": 164.64, "total": 403.78 }, "ladder_validity": "backfilled 2026-10-11 (scripts/backtest/backfill_ladder_validity.py)" } } } ] }, { "id": "#1030", "pr": 1030, "title": "perf: translate for canonicalize_name", "base": "52e57f20037ff6fc40c72fffbd7905ed81c7d390", "head": "3803ce341b59a24c598eb7b8100d5f8d6773f63e", "merged": "2026-01-05", "after_ultra_release": false, "known": [ { "id": "unicode-lowercase", "match": { "probe": "P1", "traits": "non-ascii" }, "fixed_by": 1064, "status": "verified", "discovered_by": "breakwater" } ], "targets": [ { "target": "packaging.utils:canonicalize_name", "run_id": "bt-1030-e38d1c", "image": "bw-pair:a959e7a1becfaee9", "usd": 0.0508, "harvested": 69, "arms": { "hypothesis": { "classes": [ "P1|value|differs|control+non-ascii", "P1|value|differs|control+non-ascii+whitespace", "P1|value|differs|non-ascii", "P1|value|differs|non-ascii+whitespace", "P3|gt|False|True", "P3|lt|True|False" ], "known": [ "unicode-lowercase" ], "seconds": 22.2, "error": null }, "mutations": { "classes": [ "P1|value|differs|non-ascii" ], "known": [ "unicode-lowercase" ], "seconds": 9.6, "error": null }, "breakwater": { "classes": [ "P1|value|differs|control+non-ascii", "P1|value|differs|non-ascii", "P1|value|differs|non-ascii+whitespace", "P2|outcome|exc:builtins.TypeError|exc:builtins.AttributeError|nested", "P3|gt|False|True", "P3|gt|True|False", "P3|lt|False|True", "P3|lt|True|False" ], "known": [ "unicode-lowercase" ], "seconds": 55.6, "error": null }, "llm-plan": { "classes": [ "P1|value|differs|non-ascii", "P3|gt|False|True", "P3|gt|True|False", "P3|lt|False|True", "P3|lt|True|False" ], "known": [ "unicode-lowercase" ], "seconds": 3.9, "error": null } }, "findings": [ { "signature": "P1|value|differs|non-ascii", "probe": "P1", "detail": "value", "args": [ [ "İ" ] ], "base": "i̇", "head": "İ", "found_by": [ "A3/s2", "A5/s3", "seed", "mutation:separators", "mutation:signs", "mutation:edges", "mutation:case", "batch" ] }, { "signature": "P1|value|differs|non-ascii+whitespace", "probe": "P1", "detail": "value", "args": [ [ " À" ] ], "base": " à", "head": " À", "found_by": [ "A5/s3", "mutation:whitespace-outer", "mutation:whitespace-inner" ] }, { "signature": "P1|value|differs|control+non-ascii", "probe": "P1", "detail": "value", "args": [ [ "\u001bÀ" ] ], "base": "\u001bà", "head": "\u001bÀ", "found_by": [ "A5/s3", "mutation:edges" ] }, { "signature": "P2|outcome|exc:builtins.TypeError|exc:builtins.AttributeError|nested", "probe": "P2", "detail": "outcome", "args": [ [ [] ] ], "base": { "status": "exc", "exc": "builtins.TypeError" }, "head": { "status": "exc", "exc": "builtins.AttributeError" }, "found_by": [ "A4/s1", "A4/s4" ] }, { "signature": "P3|gt|True|False", "probe": "P3", "detail": "gt", "args": [ [ "ẞ" ], [ "K" ] ], "base": true, "head": false, "found_by": [ "seed", "focus" ] }, { "signature": "P3|gt|False|True", "probe": "P3", "detail": "gt", "args": [ [ "K" ], [ "py3" ] ], "base": false, "head": true, "found_by": [ "seed", "focus" ] }, { "signature": "P3|lt|False|True", "probe": "P3", "detail": "lt", "args": [ [ "ẞ" ], [ "K" ] ], "base": false, "head": true, "found_by": [ "seed", "focus" ] }, { "signature": "P3|lt|True|False", "probe": "P3", "detail": "lt", "args": [ [ "K" ], [ "py3" ] ], "base": true, "head": false, "found_by": [ "seed", "focus" ] } ], "perf": { "slower_families": [ { "id": "R1.2", "family": "Names starting/ending with multiple dots or underscores", "verdict": "slower", "ladder": [ 1.2824553962608296, 1.2824553962608296, 2.275487929110813, 2.156755295657273, 0.8208934774162838, 0.3516870541057739, 0.2153359279949592 ], "origin": "expand:R1" }, { "id": "R4", "family": "many_underscores_and_dots", "verdict": "complexity-regression", "ladder": [ 1.0360770256459968, 1.2756788636224345, 1.9668213939732073, 3.0261918231341913, 4.203187256306862, 4.357209588569598, 4.517412301093331 ], "origin": "archive" }, { "id": "R1", "family": "Normal package names are short‑ish alphanumerics with only a few '_' or '.' characters. The new translate‑table + single‑pass replace handles this in linear tim", "verdict": "slower", "ladder": [ 0.6476983761012984, 0.9346159146780114, 1.6559583356877723, 1.1692049075155342, 0.356864305673456, 0.1340912387017206, 0.06777095089895306 ], "origin": "archive" }, { "id": "F3", "family": "typical_package_names", "verdict": "slower", "ladder": [ 0.660089684198932, 0.9069600207979266, 1.5657320215107469, 1.8967216207019288, 1.0449068623972064, 0.3494585747267035, 0.15289695535327885 ], "origin": "plan" }, { "id": "F1", "family": "typical package names containing mixed case, dots and underscores", "verdict": "slower", "ladder": [ 1.129182757597942, 0.912276751959024, 0.49977540622092187, 0.1952970891316579, 0.12339961763147544, 0.10656124244903926, 0.09648510903114219 ], "origin": "plan" } ], "worst_valid_ratio": 4.517412301093331, "worst_family": "many_underscores_and_dots", "speedup_ratio_small": 0.2153359279949592, "report": { "run_id": "bt-1030-e38d1c", "target": "packaging.utils:canonicalize_name", "image": "bw-pair:a959e7a1becfaee9", "backend": "docker", "families": [ { "id": "F1", "kind": "claimed_speedup", "family": "typical package names containing mixed case, dots and underscores", "why": "The new path replaces the regex + lower() with a single translate pass and a fast while‑loop that usually finishes in one iteration. Real‑world names have few consecutive “-” characters, so the loop runs once and the C‑level translate is faster than the regex engine.", "size_meaning": "number of dotted/underscored segments (each adds a few characters)", "make_source": "\ndef make(n):\n # Build a name like 'Pkg_0.Sub_1.Pkg_2...' with n segments\n parts = []\n for i in range(n):\n seg = f'Pkg{i}'\n if i % 2 == 0:\n parts.append(seg)\n else:\n parts.append('_' + seg)\n name = '.'.join(parts)\n return [[name]]", "example_args": [], "origin": "plan", "screen_ratio": 0.49977540622092187, "valid_fraction": 1.0, "error": null }, { "id": "F2", "kind": "claimed_speedup", "family": "Typical distribution names consist of a few alphanumeric segments separated by single hyphens; the new translate+loop path avoids the regex engine and is faster", "why": "Typical distribution names consist of a few alphanumeric segments separated by single hyphens; the new translate+loop path avoids the regex engine and is faster on these common inputs.", "size_meaning": "Number of generated names (each name length stays O(1)).", "make_source": "def make(n):\n # generate n typical package names like 'pkg-0', 'pkg-1', …\n return [[f'pkg-{i}'] for i in range(n)]", "example_args": [], "origin": "plan", "screen_ratio": 1.2266602455701294, "valid_fraction": 1.0, "error": null }, { "id": "F3", "kind": "claimed_speedup", "family": "typical_package_names", "why": "Normal package names contain only a few separators (‑, _, .) and short runs of them. The new translate‑+‑while path does a single pass over the characters and at most one extra replace pass, which is faster than the regex substitution used before.", "size_meaning": "Length of the name (including separators).", "make_source": "\ndef make(n):\n # generate a realistic name of total length ~n\n # start and end with alnum, insert a few separators\n alnum = string.ascii_letters + string.digits\n if n < 3:\n n = 3\n # ensure at least one separator\n sep = random.choice(['-', '_', '.'])\n left = n // 2 - 1\n right = n - left - 2\n name = (\n random.choice(alnum) +\n ''.join(random.choice(alnum) for _ in range(left)) +\n sep +\n ''.join(random.choice(alnum) for _ in range(right)) +\n random.choice(alnum)\n )\n return [[name]]", "example_args": [], "origin": "plan", "screen_ratio": 1.5657320215107469, "valid_fraction": 1.0, "error": null }, { "id": "F4", "kind": "suspected_slowdown", "family": "a name consisting of a very long run of '_' characters", "why": "translate turns every '_' into '-', producing a single run of n dashes. The while‑loop then repeatedly halves the run (each pass replaces '--' with '-'), so it executes O(log n) iterations, each creating a new string. The old regex does the whole replacement in one C pass, so for large n the new code does more work.", "size_meaning": "length of the underscore run", "make_source": "def make(n):\n name = '_' * n\n return [[name]]", "example_args": [], "origin": "plan", "screen_ratio": 1.4946978350701607, "valid_fraction": 1.0, "error": null }, { "id": "F5", "kind": "suspected_slowdown", "family": "A long run of '_' characters becomes a long run of '-' after translation. The while‑loop replaces '--' → '-' one step per iteration, so a run of length L needs ", "why": "A long run of '_' characters becomes a long run of '-' after translation. The while‑loop replaces '--' → '-' one step per iteration, so a run of length L needs L‑1 iterations. The old regex does it in one pass.", "size_meaning": "Length of the underscore run (≈ n).", "make_source": "def make(n):\n # name like 'a' + '_'*n + 'b' (valid per PEP 503)\n return [[ 'a' + '_' * n + 'b' ]]", "example_args": [], "origin": "plan", "screen_ratio": 1.508513962345953, "valid_fraction": 1.0, "error": null }, { "id": "F6", "kind": "suspected_slowdown", "family": "long_underscore_run", "why": "A very long run of '_' (or '.' or '-') becomes a long run of '-' after translate. The while‑loop replaces \"--\" with \"-\" repeatedly; each iteration scans the whole string, giving O(log L) passes over a length‑L run. The old regex performed a single linear pass, so for huge runs the new code is slower.", "size_meaning": "Number of '_' characters in the middle of the name.", "make_source": "\ndef make(n):\n # name = 'a' + '_'*n + 'b' (valid: starts/ends with alnum)\n name = 'a' + '_' * n + 'b'\n return [[name]]", "example_args": [], "origin": "plan", "screen_ratio": 1.508513962345953, "valid_fraction": 1.0, "error": null }, { "id": "R1", "kind": "claimed_speedup", "family": "Normal package names are short‑ish alphanumerics with only a few '_' or '.' characters. The new translate‑table + single‑pass replace handles this in linear tim", "why": "Normal package names are short‑ish alphanumerics with only a few '_' or '.' characters. The new translate‑table + single‑pass replace handles this in linear time and avoids the regex engine, giving the advertised ~2× speed‑up.", "size_meaning": "n ≈ total length of the name", "make_source": "def make(n):\n # Produce a handful of realistic names whose length grows with n.\n # Each name contains alphanumerics and occasional '_' or '.'.\n base = []\n for i in range(min(8, max(1, n // 4))):\n # create a name of length roughly n using a simple pattern\n length = max(1, n - i)\n parts = []\n for j in range(length):\n if j % 7 == 0:\n parts.append('_' if (j // 7) % 2 == 0 else '.')\n else:\n # alphanumeric: a‑z, 0‑9\n parts.append(chr(ord('a') + (j % 26)))\n base.append([''.join(parts)])\n return base", "example_args": [], "origin": "archive", "screen_ratio": 1.6559583356877723, "valid_fraction": 1.0, "error": null }, { "id": "R2", "kind": "suspected_slowdown", "family": "many_consecutive_underscores", "why": "A name like \"a____…____b\" passes validation (first/last alnum). Translation turns every '_' into '-', producing a run of n dashes. The while‑loop then repeatedly halves that run (log₂ n passes), each pass scanning the whole string → O(n log n) versus the original single‑pass regex.", "size_meaning": "Number of consecutive '_' characters between two alphanumerics.", "make_source": "def make(n):\n name = 'a' + '_' * n + 'b'\n return [[name]]", "example_args": [], "origin": "archive", "screen_ratio": 1.508513962345953, "valid_fraction": 1.0, "error": null }, { "id": "R3", "kind": "suspected_slowdown", "family": "A string consisting of N '_' characters becomes N '-' after translate. The while‑loop then repeatedly scans the whole string collapsing '--' → '-', costing O(N²", "why": "A string consisting of N '_' characters becomes N '-' after translate. The while‑loop then repeatedly scans the whole string collapsing '--' → '-', costing O(N²) time, whereas the old single regex substitution is O(N).", "size_meaning": "n = number of '_' characters", "make_source": "def make(n):\n # One argument: a name made of n underscores.\n return [['_' * n]]", "example_args": [], "origin": "archive", "screen_ratio": 1.4946978350701607, "valid_fraction": 1.0, "error": null }, { "id": "R4", "kind": "suspected_slowdown", "family": "many_underscores_and_dots", "why": "Each '_' and '.' triggers a separate `str.replace` pass that allocates a new string. With O(n) such characters the new code does three full‑string scans (lower, replace('_'), replace('.')) plus the condensation loop, while the old `translate` version performed a single C‑level pass. Cost grows linearly with the number of '_' / '.' characters.", "size_meaning": "n = number of '_' characters (same number of '.' characters); total length ≈ 2n.", "make_source": "def make(n):\n # a string like '_.' repeated n times, e.g. '_. _.' ...\n s = \"_.\" * n\n return [[s]]", "example_args": [ [ "_._._." ], [ "_._._._._._." ] ], "origin": "archive", "screen_ratio": 1.9668213939732073, "valid_fraction": 1.0, "error": null }, { "id": "R4.1", "kind": "suspected_slowdown", "family": "only_underscore_slow", "why": "Each '_' and '.' triggers a separate `str.replace` pass that allocates a new string. With O(n) such characters the new code does three full‑string scans (lower, replace('_'), replace('.')) plus the condensation loop, while the old `translate` version performed a single C‑level pass. Cost grows linearly with the number of '_' / '.' characters.", "size_meaning": "n = number of '_' characters (same number of '.' characters); total length ≈ 2n.", "make_source": "\ndef make(n):\n s = \"_\" * n\n return [[s]]", "example_args": [ [ "_._._." ], [ "_._._._._._." ] ], "origin": "expand:R4", "screen_ratio": 1.4946978350701607, "valid_fraction": 1.0, "error": null }, { "id": "R1.1", "kind": "claimed_speedup", "family": "Names with more frequent separators", "why": "Normal package names are short‑ish alphanumerics with only a few '_' or '.' characters. The new translate‑table + single‑pass replace handles this in linear time and avoids the regex engine, giving the advertised ~2× speed‑up.", "size_meaning": "n ≈ total length of the name", "make_source": "def make(n):\n result = []\n k = min(5, max(1, n // 8))\n seg_len = max(2, n // (k * 2))\n for _ in range(k):\n chars = []\n pos = 0\n while len(chars) < seg_len:\n if pos % 3 == 0:\n chars.append('_'\n if ((pos // 3) % 2 == 0)\n else '.')\n else:\n chars.append(chr(ord('a') + (len(chars) % 26)))\n pos += 1\n result.append([''.join(chars)])\n return result", "example_args": [], "origin": "expand:R1", "screen_ratio": 0.7804441349536214, "valid_fraction": 1.0, "error": null }, { "id": "R1.2", "kind": "claimed_speedup", "family": "Names starting/ending with multiple dots or underscores", "why": "Normal package names are short‑ish alphanumerics with only a few '_' or '.' characters. The new translate‑table + single‑pass replace handles this in linear time and avoids the regex engine, giving the advertised ~2× speed‑up.", "size_meaning": "n ≈ total length of the name", "make_source": "def make(n):\n core = max(0, n - 4)\n tails = ['..', '.__.', '__..', '._._.']\n out = []\n for t in tails:\n if core >= 0:\n mid = ''.join(chr(ord('a') + (i % 26)) for i in range(core))\n out.append([t + mid + t])\n return out[:min(8, len(out))]", "example_args": [], "origin": "expand:R1", "screen_ratio": 2.275487929110813, "valid_fraction": 1.0, "error": null }, { "id": "R1.3", "kind": "claimed_speedup", "family": "Long runs of letters with very rare separators", "why": "Normal package names are short‑ish alphanumerics with only a few '_' or '.' characters. The new translate‑table + single‑pass replace handles this in linear time and avoids the regex engine, giving the advertised ~2× speed‑up.", "size_meaning": "n ≈ total length of the name", "make_source": "def make(n):\n sep_every = max(3, n // 20)\n out = []\n for start in range(sep_every):\n chars = []\n for i in range(n):\n if (i - start) % sep_every == 0 and i != start:\n chars.append('_.'[(i // sep_every) % 2])\n else:\n chars.append(chr(ord('a') + (i % 26)))\n out.append([''.join(chars)])\n if len(out) >= 8:\n break\n return out", "example_args": [], "origin": "expand:R1", "screen_ratio": 1.2066893161264511, "valid_fraction": 1.0, "error": null } ], "findings": [ { "family_id": "R1.2", "kind": "claimed_speedup", "family": "Names starting/ending with multiple dots or underscores", "why": "Normal package names are short‑ish alphanumerics with only a few '_' or '.' characters. The new translate‑table + single‑pass replace handles this in linear time and avoids the regex engine, giving the advertised ~2× speed‑up.", "origin": "expand:R1", "screen_ratio": 2.275487929110813, "valid_fraction": 1.0, "ladder": [ { "n": 2, "base_ir": 4512.16375, "head_ir": 5786.64875, "input_chars": 11.5, "ratio": 1.2824553962608296, "valid_fraction": 1.0, "judged": true }, { "n": 4, "base_ir": 4512.16375, "head_ir": 5786.64875, "input_chars": 11.5, "ratio": 1.2824553962608296, "valid_fraction": 1.0, "judged": true }, { "n": 16, "base_ir": 8598.070153061224, "head_ir": 19564.804846938776, "input_chars": 23.5, "ratio": 2.275487929110813, "valid_fraction": 1.0, "judged": true }, { "n": 64, "base_ir": 17423.23161764706, "head_ir": 37577.64705882353, "input_chars": 71.5, "ratio": 2.156755295657273, "valid_fraction": 1.0, "judged": true }, { "n": 256, "base_ir": 53016.01388888889, "head_ir": 43520.5, "input_chars": 263.5, "ratio": 0.8208934774162838, "valid_fraction": 1.0, "judged": true }, { "n": 1024, "base_ir": 196909.4375, "head_ir": 69250.5, "input_chars": 1031.5, "ratio": 0.3516870541057739, "valid_fraction": 1.0, "judged": true }, { "n": 4096, "base_ir": 766349.125, "head_ir": 165022.5, "input_chars": 4103.5, "ratio": 0.2153359279949592, "valid_fraction": 1.0, "judged": true } ], "slope_base": 0.858427532821715, "slope_head": 0.5179554989014242, "top_inputs": [], "error": null, "verdict": "slower", "worst_ratio": 2.275487929110813, "worst_search": null }, { "family_id": "R4", "kind": "suspected_slowdown", "family": "many_underscores_and_dots", "why": "Each '_' and '.' triggers a separate `str.replace` pass that allocates a new string. With O(n) such characters the new code does three full‑string scans (lower, replace('_'), replace('.')) plus the condensation loop, while the old `translate` version performed a single C‑level pass. Cost grows linearly with the number of '_' / '.' characters.", "origin": "archive", "screen_ratio": 1.9668213939732073, "valid_fraction": 1.0, "ladder": [ { "n": 2, "base_ir": 4848.515, "head_ir": 5023.435, "input_chars": 8.0, "ratio": 1.0360770256459968, "valid_fraction": 1.0, "judged": true }, { "n": 4, "base_ir": 4976.515, "head_ir": 6348.435, "input_chars": 12.0, "ratio": 1.2756788636224345, "valid_fraction": 1.0, "judged": true }, { "n": 16, "base_ir": 5744.515, "head_ir": 11298.435, "input_chars": 36.0, "ratio": 1.9668213939732073, "valid_fraction": 1.0, "judged": true }, { "n": 64, "base_ir": 8743.442953020134, "head_ir": 26459.335570469797, "input_chars": 132.0, "ratio": 3.0261918231341913, "valid_fraction": 1.0, "judged": true }, { "n": 256, "base_ir": 19741.447368421053, "head_ir": 82977.0, "input_chars": 516.0, "ratio": 4.203187256306862, "valid_fraction": 1.0, "judged": true }, { "n": 1024, "base_ir": 70051.0, "head_ir": 305226.8888888889, "input_chars": 2052.0, "ratio": 4.357209588569598, "valid_fraction": 1.0, "judged": true }, { "n": 4096, "base_ir": 266708.0, "head_ir": 1204830.0, "input_chars": 8196.0, "ratio": 4.517412301093331, "valid_fraction": 1.0, "judged": true } ], "slope_base": 0.5643239496139442, "slope_head": 0.7835468130469792, "top_inputs": [], "error": null, "verdict": "complexity-regression", "worst_ratio": 4.517412301093331, "worst_search": null }, { "family_id": "R1", "kind": "claimed_speedup", "family": "Normal package names are short‑ish alphanumerics with only a few '_' or '.' characters. The new translate‑table + single‑pass replace handles this in linear tim", "why": "Normal package names are short‑ish alphanumerics with only a few '_' or '.' characters. The new translate‑table + single‑pass replace handles this in linear time and avoids the regex engine, giving the advertised ~2× speed‑up.", "origin": "archive", "screen_ratio": 1.6559583356877723, "valid_fraction": 1.0, "ladder": [ { "n": 2, "base_ir": 6321.515, "head_ir": 4094.435, "input_chars": 6.0, "ratio": 0.6476983761012984, "valid_fraction": 1.0, "judged": true }, { "n": 4, "base_ir": 6929.515, "head_ir": 6476.435, "input_chars": 8.0, "ratio": 0.9346159146780114, "valid_fraction": 1.0, "judged": true }, { "n": 16, "base_ir": 10330.66375, "head_ir": 17107.14875, "input_chars": 18.5, "ratio": 1.6559583356877723, "valid_fraction": 1.0, "judged": true }, { "n": 64, "base_ir": 29291.807432432433, "head_ir": 34248.125, "input_chars": 64.5, "ratio": 1.1692049075155342, "valid_fraction": 1.0, "judged": true }, { "n": 256, "base_ir": 105890.77777777778, "head_ir": 37788.63888888889, "input_chars": 256.5, "ratio": 0.356864305673456, "valid_fraction": 1.0, "judged": true }, { "n": 1024, "base_ir": 388285.0625, "head_ir": 52065.625, "input_chars": 1024.5, "ratio": 0.1340912387017206, "valid_fraction": 1.0, "judged": true }, { "n": 4096, "base_ir": 1577964.0625, "head_ir": 106940.125, "input_chars": 4096.5, "ratio": 0.06777095089895306, "valid_fraction": 1.0, "judged": true } ], "slope_base": 0.8560256840502828, "slope_head": 0.4400776167413549, "top_inputs": [], "error": null, "verdict": "slower", "worst_ratio": 1.6559583356877723, "worst_search": null }, { "family_id": "F3", "kind": "claimed_speedup", "family": "typical_package_names", "why": "Normal package names contain only a few separators (‑, _, .) and short runs of them. The new translate‑+‑while path does a single pass over the characters and at most one extra replace pass, which is faster than the regex substitution used before.", "origin": "plan", "screen_ratio": 1.5657320215107469, "valid_fraction": 1.0, "ladder": [ { "n": 2, "base_ir": 7081.515, "head_ir": 4674.435, "input_chars": 8.0, "ratio": 0.660089684198932, "valid_fraction": 1.0, "judged": true }, { "n": 4, "base_ir": 7481.515, "head_ir": 6785.435, "input_chars": 9.0, "ratio": 0.9069600207979266, "valid_fraction": 1.0, "judged": true }, { "n": 16, "base_ir": 9670.515, "head_ir": 15141.435, "input_chars": 21.0, "ratio": 1.5657320215107469, "valid_fraction": 1.0, "judged": true }, { "n": 64, "base_ir": 18528.515, "head_ir": 35143.435, "input_chars": 69.0, "ratio": 1.8967216207019288, "valid_fraction": 1.0, "judged": true }, { "n": 256, "base_ir": 53951.72368421053, "head_ir": 56374.52631578947, "input_chars": 261.0, "ratio": 1.0449068623972064, "valid_fraction": 1.0, "judged": true }, { "n": 1024, "base_ir": 197582.7894736842, "head_ir": 69047.0, "input_chars": 1029.0, "ratio": 0.3494585747267035, "valid_fraction": 1.0, "judged": true }, { "n": 4096, "base_ir": 767732.75, "head_ir": 117384.0, "input_chars": 4101.0, "ratio": 0.15289695535327885, "valid_fraction": 1.0, "judged": true } ], "slope_base": 0.7424878197843604, "slope_head": 0.48484774680303483, "top_inputs": [], "error": null, "verdict": "slower", "worst_ratio": 1.8967216207019288, "worst_search": null }, { "family_id": "F1", "kind": "claimed_speedup", "family": "typical package names containing mixed case, dots and underscores", "why": "The new path replaces the regex + lower() with a single translate pass and a fast while‑loop that usually finishes in one iteration. Real‑world names have few consecutive “-” characters, so the loop runs once and the C‑level translate is faster than the regex engine.", "origin": "plan", "screen_ratio": 0.49977540622092187, "valid_fraction": 1.0, "ladder": [ { "n": 2, "base_ir": 8243.515, "head_ir": 9308.435, "input_chars": 14.0, "ratio": 1.129182757597942, "valid_fraction": 1.0, "judged": true }, { "n": 4, "base_ir": 13258.515, "head_ir": 12095.435, "input_chars": 25.0, "ratio": 0.912276751959024, "valid_fraction": 1.0, "judged": true }, { "n": 16, "base_ir": 43734.515, "head_ir": 21857.435, "input_chars": 97.0, "ratio": 0.49977540622092187, "valid_fraction": 1.0, "judged": true }, { "n": 64, "base_ir": 171548.89583333334, "head_ir": 33503.0, "input_chars": 409.0, "ratio": 0.1952970891316579, "valid_fraction": 1.0, "judged": true }, { "n": 256, "base_ir": 692668.2727272727, "head_ir": 85475.0, "input_chars": 1813.0, "ratio": 0.12339961763147544, "valid_fraction": 1.0, "judged": true }, { "n": 1024, "base_ir": 2763059.0, "head_ir": 294435.0, "input_chars": 7597.0, "ratio": 0.10656124244903926, "valid_fraction": 1.0, "judged": true }, { "n": 4096, "base_ir": 11919943.0, "head_ir": 1150097.0, "input_chars": 33709.0, "ratio": 0.09648510903114219, "valid_fraction": 1.0, "judged": true } ], "slope_base": 0.9365291643021316, "slope_head": 0.5938072165871816, "top_inputs": [], "error": null, "verdict": "slower", "worst_ratio": 1.129182757597942, "worst_search": null } ], "seconds": { "plan": 16.58, "screen": 253.23, "full": 452.94, "total": 722.74 }, "ladder_validity": "backfilled 2026-10-11 (scripts/backtest/backfill_ladder_validity.py)" } }, "image_rescored": "bw-pair:d09088ff31a7e098" } ] }, { "id": "#1032", "pr": 1032, "title": "perf: add __slots__ to Nodes", "base": "842c16cef598bf80178b3894b7039b69cca3621c", "head": "c39b6da9ab11f649f5497a11628f9dfc98380bb2", "merged": "2026-01-02", "after_ultra_release": false, "known": [], "targets": [ { "target": "packaging.markers:Marker", "run_id": "bt-1032-bd4ecb", "image": "bw-pair:810a13228d4ca505", "usd": 0.0409, "harvested": 80, "arms": { "hypothesis": { "classes": [], "known": [], "seconds": 10.8, "error": null }, "mutations": { "classes": [ "P4|base->head load|loads|exc:builtins.AttributeError" ], "known": [], "seconds": 3.2, "error": null }, "breakwater": { "classes": [ "P4|base->head load|loads|exc:builtins.AttributeError" ], "known": [], "seconds": 43.5, "error": null }, "llm-plan": { "classes": [ "P4|base->head load|loads|exc:builtins.AttributeError" ], "known": [], "seconds": 4.3, "error": null } }, "findings": [ { "signature": "P4|base->head load|loads|exc:builtins.AttributeError", "probe": "P4", "detail": "base->head load", "args": [ [ "extra== 'test'" ] ], "base": "loads", "head": { "status": "exc", "exc": "builtins.AttributeError" }, "found_by": [ "seed", "tests", "mutation:unicode-digits", "mutation:whitespace-outer", "mutation:whitespace-inner", "mutation:huge-number", "mutation:separators", "mutation:underscores", "mutation:equivalents", "mutation:case" ] } ], "perf": { "slower_families": [], "worst_valid_ratio": null, "worst_family": null, "speedup_ratio_small": 0.9937954921461972, "report": { "run_id": "bt-1032-bd4ecb", "target": "packaging.markers:Marker", "image": "bw-pair:810a13228d4ca505", "backend": "docker", "families": [ { "id": "F1", "kind": "claimed_speedup", "family": "marker with many chained comparisons", "why": "Each comparison creates a Variable, Op, Value tuple (three Node subclasses). Adding __slots__ reduces per‑object memory and attribute‑lookup cost, which compounds when thousands of nodes are created during parsing.", "size_meaning": "number of binary comparisons chained with 'and'", "make_source": "\ndef make(n):\n # Build a marker like:\n # python_version > '3.0' and python_version > '3.1' and ...\n parts = []\n for i in range(n):\n parts.append(f\"python_version > '{i}' version > '{i}'\")\n marker = ' and '.join(parts)\n return [[marker]]", "example_args": [ [ "python_version > '0' and python_version > '1' and python_version > '2'" ] ], "origin": "plan", "screen_ratio": 0.9930624759994504, "valid_fraction": 0.0, "error": null }, { "id": "F2", "kind": "claimed_speedup", "family": "large_flat_marker", "why": "The change adds __slots__ to the node classes, reducing per‑node memory and attribute‑access overhead. A marker that expands to many node objects (many comparisons combined with “or”) exercises the fast path the most.", "size_meaning": "Number of top‑level comparisons (each creates a Variable, Op, Value tuple and a list entry).", "make_source": "def make(n):\n # Build a marker like: python_version == \"3.0\" or python_version == \"3.1\" or …\n parts = [f'python_version == \"3.{i}\"' for i in range(n)]\n marker = \" or \".join(parts)\n return [[marker]]", "example_args": [ [ "python_version == \"3.0\" or python_version == \"3.1\"" ] ], "origin": "plan", "screen_ratio": 0.9937954921461972, "valid_fraction": 1.0, "error": null }, { "id": "F3", "kind": "claimed_speedup", "family": "Markers consisting of many simple comparisons joined by \"or\" (or \"and\"), e.g. \"python_version > '3.0' or python_version > '3.1' or …\". This exercises creation of many Variable/Value/Op nodes during parsing, which the __slots__ change aims to make faster.", "why": "The diff adds __slots__ to Node, Variable, Value, and Op. Instances of these classes are created for each atom in the parsed marker. A marker with many atoms creates many instances; slots reduce per‑instance memory and attribute‑access overhead, so the new version should be faster.", "size_meaning": "n = number of atomic comparisons in the marker.", "make_source": "\ndef make(n):\n # Build a marker like: python_version > '3.0' or python_version > '3.1' ...\n parts = []\n for i in range(n):\n parts.append(f\"python_version > '{3+i}.0'\")\n marker = \" or \".join(parts)\n return [[marker]]", "example_args": [ [ "python_version > '3.0' or python_version > '3.1'" ], [ "python_version > '3.0' or python_version > '3.1' or python_version > '3.2' or python_version > '3.3'" ] ], "origin": "plan", "screen_ratio": 0.9969108093087604, "valid_fraction": 1.0, "error": null }, { "id": "F4", "kind": "claimed_speedup", "family": "marker with deep parentheses nesting", "why": "Nesting creates many temporary MarkerList objects that hold tuples of Node instances. Slots make each Node cheaper, so the total parse time drops noticeably as nesting depth grows.", "size_meaning": "nesting depth (each level adds a new sub‑list)", "make_source": "def make(n):\n # Produce: ( ( ( python_version == '3.8' ) ) ... )\n inner = \"python_version == '3.8'\"\n for _ in range(n):\n inner = f'({inner})'\n return [[inner]]", "example_args": [ [ "(((python_version == '3.8')))" ] ], "origin": "plan", "screen_ratio": 0.9994379872325394, "valid_fraction": 1.0, "error": null }, { "id": "F5", "kind": "suspected_slowdown", "family": "deeply_nested_marker", "why": "Deep nesting creates many list objects that hold a single sub‑list each (e.g. (((expr)))). The __slots__ change does not affect list creation, but the extra indirection may expose any hidden overhead in the new node layout.", "size_meaning": "Nesting depth – each level adds a list wrapper around the previous expression.", "make_source": "def make(n):\n # Build a marker like: (((python_version == \"3.0\"))) with n pairs of parentheses\n inner = 'python_version == \"3.0\"'\n marker = '(' * n + inner + ')' * n\n return [[marker]]", "example_args": [ [ "((((python_version == \"3.0\" ))))" ] ], "origin": "plan", "screen_ratio": 0.9994325100584895, "valid_fraction": 1.0, "error": null }, { "id": "F6", "kind": "suspected_slowdown", "family": "Markers with deep nesting of groups (parenthesised sub‑expressions). Example: \"(python_version > '3.0' and (python_version > '3.1' and (...)))\".", "why": "Deep nesting creates many list objects for the MarkerList structure but does not increase the number of Node instances. The __slots__ change does not affect list handling, so performance should be unchanged; any observed slowdown would be noise.", "size_meaning": "n = nesting depth.", "make_source": "def make(n):\n inner = \"python_version > '3.0'\"\n for i in range(1, n+1):\n inner = f\"(python_version > '3.{i}' and {inner})\"\n return [[inner]]", "example_args": [ [ "(python_version > '3.1' and python_version > '3.0')" ], [ "(python_version > '3.2' and (python_version > '3.1' and python_version > '3.0'))" ] ], "origin": "plan", "screen_ratio": 0.9974647401028149, "valid_fraction": 1.0, "error": null } ], "findings": [ { "family_id": "F2", "kind": "claimed_speedup", "family": "large_flat_marker", "why": "The change adds __slots__ to the node classes, reducing per‑node memory and attribute‑access overhead. A marker that expands to many node objects (many comparisons combined with “or”) exercises the fast path the most.", "origin": "plan", "screen_ratio": 0.9937954921461972, "valid_fraction": 1.0, "ladder": [ { "n": 2, "base_ir": 232373.45, "head_ir": 231663.72, "input_chars": 58.0, "ratio": 0.9969457354099618, "valid_fraction": 1.0, "judged": true }, { "n": 4, "base_ir": 461436.07100591715, "head_ir": 459969.6923076923, "input_chars": 116.0, "ratio": 0.9968221411580847, "valid_fraction": 1.0, "judged": true }, { "n": 16, "base_ir": 1853721.3333333333, "head_ir": 1842219.9047619049, "input_chars": 470.0, "ratio": 0.9937954921461972, "valid_fraction": 1.0, "judged": true }, { "n": 64, "base_ir": 7514410.2, "head_ir": 7468278.3, "input_chars": 1910.0, "ratio": 0.9938608754683101, "valid_fraction": 1.0, "judged": true }, { "n": 256, "base_ir": 30123635.5, "head_ir": 30160978.5, "input_chars": 7826.0, "ratio": 1.0012396578095628, "valid_fraction": 1.0, "judged": true }, { "n": 1024, "base_ir": 121029054.5, "head_ir": 120280488.5, "input_chars": 31658.0, "ratio": 0.9938149892759841, "valid_fraction": 1.0, "judged": true }, { "n": 4096, "base_ir": 490071961.0, "head_ir": 487325598.0, "input_chars": 129962.0, "ratio": 0.9943960005498049, "valid_fraction": 1.0, "judged": true } ], "slope_base": 0.992306694841567, "slope_head": 0.9921461523823009, "top_inputs": [], "error": null, "verdict": "neutral", "worst_ratio": 1.0012396578095628, "worst_search": null } ], "seconds": { "plan": 8.14, "screen": 169.34, "full": 189.11, "total": 366.59 }, "ladder_validity": "backfilled 2026-10-11 (scripts/backtest/backfill_ladder_validity.py)" } }, "image_rescored": "bw-pair:f8c16265598b425b" } ] }, { "id": "#1058", "pr": 1058, "title": "perf: 10% faster stripping zeros", "base": "350a2306700b738f487f251efa278f532b263dee", "head": "716beb1c0a5d7a398bf57fbd80bc2501811e616b", "merged": "2026-01-14", "after_ultra_release": false, "known": [], "targets": [ { "target": "packaging.version:Version", "run_id": "bt-1058-db9087", "image": "bw-pair:1e7b57d2a90aaab0", "usd": 0.1801, "harvested": 80, "arms": { "hypothesis": { "classes": [], "known": [], "seconds": 8.0, "error": null }, "mutations": { "classes": [], "known": [], "seconds": 3.5, "error": null }, "breakwater": { "classes": [], "known": [], "seconds": 112.6, "error": null }, "llm-plan": { "classes": [], "known": [], "seconds": 7.8, "error": null } }, "findings": [], "perf": { "slower_families": [], "worst_valid_ratio": null, "worst_family": null, "speedup_ratio_small": null, "report": { "run_id": "bt-1058-db9087", "target": "packaging.version:Version", "image": "bw-pair:1e7b57d2a90aaab0", "backend": "docker", "families": [ { "id": "F1", "kind": "suspected_slowdown", "family": "many_release_segments", "why": "The new _cmpkey inlines a Python‑level while‑loop to strip trailing zeros. For a release tuple with many components the loop executes in pure Python, whereas the old helper used a C‑level ``for i in range(...)`` loop that is faster per iteration. The cost grows linearly with the number of release segments.", "size_meaning": "Number of dot‑separated numeric components in the release segment (n).", "make_source": "def make(n):\n # Build a version like '1.' repeated n‑1 times followed by '0' -> n components\n # Example n=3 -> '1.1.0'\n parts = ['1'] * (n - 1) + ['0']\n version = '.'.join(parts)\n return [[version]]", "example_args": [ [ "1.0" ], [ "1.1.0" ], [ "1.1.1.0" ] ], "origin": "plan", "screen_ratio": 0.999854822765645, "valid_fraction": 1.0, "error": null }, { "id": "F2", "kind": "suspected_slowdown", "family": "many_trailing_zeros", "why": "Both old and new code walk backwards over trailing zeros, but the new version does it with a ``while i and release[i-1] == 0`` loop in Python. When the release tuple ends with a long run of zeros (e.g. \"1.\" + \"0.\"*k) the loop iterates k times in the interpreter, while the former ``_strip_trailing_zeros`` used a ``for`` loop implemented in C and returned early via a Python ``return``. The overhead per iteration is higher in the new code, so runtime grows with the number of trailing zeros.", "size_meaning": "Count of trailing zero components after a non‑zero prefix (k).", "make_source": "def make(k):\n # One non‑zero component followed by k zeros: '1' + '.0'*k\n version = '1' + '.0' * k\n return [[version]]", "example_args": [ [ "1.0" ], [ "1.0.0" ], [ "1.0.0.0" ] ], "origin": "plan", "screen_ratio": 0.9998591793034054, "valid_fraction": 1.0, "error": null }, { "id": "F3", "kind": "suspected_slowdown", "family": "long_local_version", "why": "Parsing the local version splits on ``[._-]`` and builds a tuple of mixed int/str objects. The change does not touch this path, but the overall Version construction now spends relatively more time in the new release‑stripping loops, so a version that also carries a very long local part (many separators) amplifies the total cost and makes the regression visible.", "size_meaning": "Number of dot‑separated identifiers in the local segment (m).", "make_source": "def make(m):\n # Base version '1.0' plus a local part with m components like 'a.1.b.2...'\n local_parts = []\n for i in range(m):\n local_parts.append('a' if i % 2 == 0 else str(i))\n local = '.'.join(local_parts)\n version = f'1.0+{local}'\n return [[version]]", "example_args": [ [ "1.0+a" ], [ "1.0+a.1" ], [ "1.0+a.1.b.2" ] ], "origin": "plan", "screen_ratio": 0.9989912134135978, "valid_fraction": 1.0, "error": null }, { "id": "F4", "kind": "claimed_speedup", "family": "The hot helper `_strip_trailing_zeros` was removed and its logic inlined into `_cmpkey` using a simple while‑loop with index manipulation, removing a function c", "why": "The hot helper `_strip_trailing_zeros` was removed and its logic inlined into `_cmpkey` using a simple while‑loop with index manipulation, removing a function call and a slightly slower reverse‑range loop. The new while‑loop stops as soon as a non‑zero is found, avoiding the range object creation and function‑call overhead, giving a modest speedup for the hot version‑key path.", "size_meaning": "claimed_speedup", "make_source": "\ndef make(n):\n claim = 0.05 + 0.02 * (n % 8)\n sizes = [2, 3, 4, 5, 4, 3, 2, 1]\n sz = sizes[n % 8]\n args = [f\"Version('{i}.{j}.{k}')\" for i, j, k in itertools.islice(itertools.product(range(10), repeat=3), sz)]\n return [args]", "example_args": [], "origin": "plan", "screen_ratio": 1.0, "valid_fraction": 0.0, "error": null }, { "id": "R1", "kind": "suspected_slowdown", "family": "Invalid version strings that nevertheless contain indicator characters (e.g. a trailing dot followed by a letter). The fast‑path still does the full scan before the regex fails.", "why": "Even though the version is invalid, the extra disjoint check runs; the regex then fails. This is the single allowed invalid family.", "size_meaning": "Number of dotted segments before the malformed tail.", "make_source": "def make(n):\n # \"1.\" repeated n times then an alphabetic tail – not a valid PEP‑440 version\n version = \"1.\" * n + \"x\"\n return [(version,)]", "example_args": [], "origin": "archive", "screen_ratio": 0.9997436179456157, "valid_fraction": 0.0, "error": null }, { "id": "R2", "kind": "suspected_slowdown", "family": "invalid strings that look simple (only digits and dots) but contain a non‑numeric segment, e.g. \"1.2.x\"", "why": "The indicator set is disjoint, so the fast path is entered. The code then executes `tuple(map(int, version.strip().split('.')))`, which raises `ValueError` on the bad segment; the exception is caught and re‑raised as `InvalidVersion`. Exception handling is considerably slower than the original regex‑only failure path.", "size_meaning": "n = number of leading numeric segments before the bad segment; larger n makes the split and the failed `int` conversion traverse more items before the exception.", "make_source": "def make(n):\n # build \"1.1.1....1.x\" with n numeric parts then a non‑numeric \"x\"\n good = \".\".join(\"1\" for _ in range(n))\n version = f\"{good}.x\"\n return [[version]]", "example_args": [], "origin": "archive", "screen_ratio": 0.9997436179456157, "valid_fraction": 0.0, "error": null }, { "id": "R3", "kind": "suspected_slowdown", "family": "Single repeated segment \"999.\" n times plus prerelease c5", "why": "The fast‑path’s `isdisjoint` test iterates over every character. For a long dotted release the string length grows with n, so the extra scan adds linear overhead before the (already linear) regex match.", "size_meaning": "Number of dotted release segments (plus the trailing pre‑release).", "make_source": "def make(n):\n release = \".\".join([\"999\"] * n)\n version = f\"{release}c5\"\n return [(version,)]", "example_args": [], "origin": "archive", "screen_ratio": 0.9998793336759706, "valid_fraction": 1.0, "error": null }, { "id": "R4", "kind": "suspected_slowdown", "family": "Versions with many release segments and a pre‑release suffix (e.g. \"1.2.3…a1\"). The fast‑path scans the whole string looking for indicator characters, finds the “a”, then falls back to the regex parser – an extra O(n) pass over the input.", "why": "The fast‑path’s `isdisjoint` test iterates over every character. For a long dotted release the string length grows with n, so the extra scan adds linear overhead before the (already linear) regex match.", "size_meaning": "Number of dotted release segments (plus the trailing pre‑release).", "make_source": "def make(n):\n # Build \"1.2.3…n\" then add a pre‑release marker \"a1\"\n release = \".\".join(str(i) for i in range(1, n + 1))\n version = f\"{release}a1\"\n return [(version,)]", "example_args": [], "origin": "archive", "screen_ratio": 0.9998689193662715, "valid_fraction": 1.0, "error": null } ], "findings": [], "seconds": { "plan": 69.09, "screen": 183.11, "full": 0.0, "total": 252.2 }, "ladder_validity": "backfilled 2026-10-11 (scripts/backtest/backfill_ladder_validity.py)" } } } ] }, { "id": "#1064", "pr": 1064, "title": "perf: dual replace", "base": "d5398b8bc19f3fad7b035ceb83023caf06d6e2da", "head": "9627a8821f09e4c55cd6e9daadb617b67c4741c2", "merged": "2026-01-19", "after_ultra_release": false, "known": [], "targets": [ { "target": "packaging.utils:canonicalize_name", "run_id": "bt-1064-171e84", "image": "bw-pair:90a05485e6da0b44", "usd": 0.048, "harvested": 69, "arms": { "hypothesis": { "classes": [ "P1|value|differs|control+non-ascii", "P1|value|differs|control+non-ascii+whitespace", "P1|value|differs|non-ascii", "P1|value|differs|non-ascii+whitespace", "P3|gt|True|False", "P3|lt|False|True" ], "known": [], "seconds": 9.0, "error": null }, "mutations": { "classes": [ "P1|value|differs|non-ascii" ], "known": [], "seconds": 8.3, "error": null }, "breakwater": { "classes": [ "P1|value|differs|control+non-ascii", "P1|value|differs|non-ascii", "P1|value|differs|non-ascii+whitespace", "P3|eq|False|True", "P3|gt|False|True", "P3|gt|True|False", "P3|hash_eq|False|True", "P3|lt|False|True", "P3|lt|True|False" ], "known": [], "seconds": 75.1, "error": null }, "llm-plan": { "classes": [ "P1|value|differs|non-ascii", "P3|eq|False|True", "P3|gt|False|True", "P3|gt|True|False", "P3|hash_eq|False|True", "P3|lt|False|True", "P3|lt|True|False" ], "known": [], "seconds": 10.0, "error": null } }, "findings": [ { "signature": "P1|value|differs|non-ascii", "probe": "P1", "detail": "value", "args": [ [ "Α" ] ], "base": "Α", "head": "α", "found_by": [ "A1/s3", "A1/s4", "A1/s1", "A2/s1", "A2/s4", "A3/s4", "A4/s2", "A5/s2", "A4/s4", "seed", "mutation:separators", "mutation:case", "mutation:signs", "mutation:edges", "batch" ] }, { "signature": "P1|value|differs|non-ascii+whitespace", "probe": "P1", "detail": "value", "args": [ [ " À" ] ], "base": " À", "head": " à", "found_by": [ "A4/s4", "mutation:whitespace-outer", "mutation:whitespace-inner" ] }, { "signature": "P1|value|differs|control+non-ascii", "probe": "P1", "detail": "value", "args": [ [ "\u001bÀ" ] ], "base": "\u001bÀ", "head": "\u001bà", "found_by": [ "A4/s4", "mutation:edges" ] }, { "signature": "P3|eq|False|True", "probe": "P3", "detail": "eq", "args": [ [ "İ" ], [ "i̇" ] ], "base": false, "head": true, "found_by": [ "seed", "focus" ] }, { "signature": "P3|gt|True|False", "probe": "P3", "detail": "gt", "args": [ [ "K" ], [ "ı" ] ], "base": true, "head": false, "found_by": [ "seed", "focus" ] }, { "signature": "P3|gt|False|True", "probe": "P3", "detail": "gt", "args": [ [ "Ω" ], [ "K" ] ], "base": false, "head": true, "found_by": [ "seed", "focus" ] }, { "signature": "P3|hash_eq|False|True", "probe": "P3", "detail": "hash_eq", "args": [ [ "İ" ], [ "i̇" ] ], "base": false, "head": true, "found_by": [ "seed", "focus" ] }, { "signature": "P3|lt|False|True", "probe": "P3", "detail": "lt", "args": [ [ "K" ], [ "ı" ] ], "base": false, "head": true, "found_by": [ "seed", "focus" ] }, { "signature": "P3|lt|True|False", "probe": "P3", "detail": "lt", "args": [ [ "Ω" ], [ "K" ] ], "base": true, "head": false, "found_by": [ "seed", "focus" ] } ], "perf": { "slower_families": [ { "id": "R4", "family": "many_underscores_and_dots", "verdict": "slower", "ladder": [ 1.0508362893783132, 1.0478720024010653, 1.0590243313951082, 1.0754277742187484, 1.093603819189745, 1.1006838985929919, 1.0979442869566949 ], "origin": "archive" }, { "id": "R3", "family": "A string consisting of N '_' characters becomes N '-' after translate. The while‑loop then repeatedly scans the whole string collapsing '--' → '-', costing O(N²", "verdict": "slower", "ladder": [ 1.0848915979354996, 1.0704173095176994, 1.0530119900784558, 1.040997093057704, 1.0384806214160804, 1.0417569706175838, 1.0389288638028351 ], "origin": "archive" }, { "id": "R3.1", "family": "A string consisting of N '.' characters", "verdict": "slower", "ladder": [ 1.0848915979354996, 1.0704173095176994, 1.0530119900784558, 1.040997093057704, 1.0384806214160804, 1.0417569706175838, 1.0389288638028351 ], "origin": "expand:R3" } ], "worst_valid_ratio": 1.1006838985929919, "worst_family": "many_underscores_and_dots", "speedup_ratio_small": 0.09840330029089392, "report": { "run_id": "bt-1064-171e84", "target": "packaging.utils:canonicalize_name", "image": "bw-pair:90a05485e6da0b44", "backend": "docker", "families": [ { "id": "F1", "kind": "claimed_speedup", "family": "short alphanumeric names without '_' or '.' (the fast path the change targets)", "why": "The new code avoids the translate table lookup; for pure lower‑case alphanumerics the three‑pass lower/replace is comparable but the change authors claim it is faster.", "size_meaning": "n = length of the name", "make_source": "\ndef make(n):\n # generate a single name of length n using only a‑z and 0‑9\n alphabet = string.ascii_lowercase + string.digits\n return [[\"\" .join(random.choices(alphabet, k=n))]]", "example_args": [], "origin": "plan", "screen_ratio": 0.17708641727324573, "valid_fraction": 1.0, "error": null }, { "id": "F2", "kind": "claimed_speedup", "family": "typical_short_names", "why": "The change was advertised as faster for normal package names that contain only alphanumerics and a few hyphens; such inputs avoid the extra replace passes and the while‑loop entirely.", "size_meaning": "Length of the name (number of characters).", "make_source": "def make(n):\n # typical names like 'my-package-123' – alphanum plus a few hyphens\n base = 'my-package-'\n return [[base + 'x' * n]]", "example_args": [], "origin": "plan", "screen_ratio": 0.22648647680228456, "valid_fraction": 1.0, "error": null }, { "id": "F3", "kind": "claimed_speedup", "family": "Typical PyPI distribution names – short lower‑case alphanumerics with a few hyphens, no '_' or '.'", "why": "The change replaces a single `str.translate` pass with three very fast C‑level operations (`lower`, `replace`, `replace`). For the common case (no '_' or '.') the three passes are still cheaper than building and applying a translation table, so the new code is expected to be faster.", "size_meaning": "n = number of hyphen‑separated segments (each segment ~5 chars). Total length ≈ 6·n.", "make_source": "\ndef make(n):\n # produce n typical names, each a little longer\n out = []\n for i in range(1, n+1):\n # e.g. 'pkg-0', 'pkg-0-1', …\n name = '-'.join('pkg' + str(j) for j in range(i))\n out.append([name])\n return out", "example_args": [], "origin": "plan", "screen_ratio": 0.18901895162245147, "valid_fraction": 1.0, "error": null }, { "id": "F4", "kind": "suspected_slowdown", "family": "names containing many uppercase letters", "why": "Base uses a single C‑level translate pass that maps A‑Z → a‑z in one scan. Head does lower() (full scan) plus two replace passes, so three full scans vs one.", "size_meaning": "n = length of the name", "make_source": "\ndef make(n):\n alphabet = string.ascii_uppercase + string.digits\n return [[\"\" .join(random.choices(alphabet, k=n))]]", "example_args": [], "origin": "plan", "screen_ratio": 0.2593298199069291, "valid_fraction": 1.0, "error": null }, { "id": "F5", "kind": "suspected_slowdown", "family": "many_underscores", "why": "Each '_' is turned into '-' by a separate str.replace pass, creating an intermediate string; the original translate did it in one C pass. The while‑loop then condenses the resulting '--' runs, so cost grows with the number of underscores.", "size_meaning": "Number of '_' characters (≈ n).", "make_source": "def make(n):\n # name consisting of n underscores surrounded by a minimal valid prefix/suffix\n return [['a' + '_' * n + 'b']]", "example_args": [], "origin": "plan", "screen_ratio": 0.825286090480356, "valid_fraction": 1.0, "error": null }, { "id": "F6", "kind": "suspected_slowdown", "family": "Names that contain a large number of '_' and '.' characters", "why": "The old version performed the lower‑casing and the '_' / '.' → '-' conversion in a single `translate` call (one C‑level scan). The new version does three separate scans (`lower`, `replace('_','-')`, `replace('.','-')`). When the input has many '_' or '.' the extra scans dominate the running time.", "size_meaning": "n = number of '_' and '.' pairs. Length ≈ 4·n.", "make_source": "def make(n):\n out = []\n for i in range(1, n+1):\n # alternating '_' and '.' to maximise both replaces\n name = ('a_' + 'b.') * i\n out.append([name])\n return out", "example_args": [], "origin": "plan", "screen_ratio": 0.7042734857279247, "valid_fraction": 1.0, "error": null }, { "id": "R1", "kind": "claimed_speedup", "family": "Normal package names are short‑ish alphanumerics with only a few '_' or '.' characters. The new translate‑table + single‑pass replace handles this in linear tim", "why": "Normal package names are short‑ish alphanumerics with only a few '_' or '.' characters. The new translate‑table + single‑pass replace handles this in linear time and avoids the regex engine, giving the advertised ~2× speed‑up.", "size_meaning": "n ≈ total length of the name", "make_source": "def make(n):\n # Produce a handful of realistic names whose length grows with n.\n # Each name contains alphanumerics and occasional '_' or '.'.\n base = []\n for i in range(min(8, max(1, n // 4))):\n # create a name of length roughly n using a simple pattern\n length = max(1, n - i)\n parts = []\n for j in range(length):\n if j % 7 == 0:\n parts.append('_' if (j // 7) % 2 == 0 else '.')\n else:\n # alphanumeric: a‑z, 0‑9\n parts.append(chr(ord('a') + (j % 26)))\n base.append([''.join(parts)])\n return base", "example_args": [], "origin": "archive", "screen_ratio": 0.20211372652583864, "valid_fraction": 1.0, "error": null }, { "id": "R2", "kind": "suspected_slowdown", "family": "many_consecutive_underscores", "why": "A name like \"a____…____b\" passes validation (first/last alnum). Translation turns every '_' into '-', producing a run of n dashes. The while‑loop then repeatedly halves that run (log₂ n passes), each pass scanning the whole string → O(n log n) versus the original single‑pass regex.", "size_meaning": "Number of consecutive '_' characters between two alphanumerics.", "make_source": "def make(n):\n name = 'a' + '_' * n + 'b'\n return [[name]]", "example_args": [], "origin": "archive", "screen_ratio": 0.825286090480356, "valid_fraction": 1.0, "error": null }, { "id": "R3", "kind": "suspected_slowdown", "family": "A string consisting of N '_' characters becomes N '-' after translate. The while‑loop then repeatedly scans the whole string collapsing '--' → '-', costing O(N²", "why": "A string consisting of N '_' characters becomes N '-' after translate. The while‑loop then repeatedly scans the whole string collapsing '--' → '-', costing O(N²) time, whereas the old single regex substitution is O(N).", "size_meaning": "n = number of '_' characters", "make_source": "def make(n):\n # One argument: a name made of n underscores.\n return [['_' * n]]", "example_args": [], "origin": "archive", "screen_ratio": 1.0530119900784558, "valid_fraction": 1.0, "error": null }, { "id": "R4", "kind": "suspected_slowdown", "family": "many_underscores_and_dots", "why": "Each '_' and '.' triggers a separate `str.replace` pass that allocates a new string. With O(n) such characters the new code does three full‑string scans (lower, replace('_'), replace('.')) plus the condensation loop, while the old `translate` version performed a single C‑level pass. Cost grows linearly with the number of '_' / '.' characters.", "size_meaning": "n = number of '_' characters (same number of '.' characters); total length ≈ 2n.", "make_source": "def make(n):\n # a string like '_.' repeated n times, e.g. '_. _.' ...\n s = \"_.\" * n\n return [[s]]", "example_args": [ [ "_._._." ], [ "_._._._._._." ] ], "origin": "archive", "screen_ratio": 1.0590243313951082, "valid_fraction": 1.0, "error": null }, { "id": "R4.1", "kind": "suspected_slowdown", "family": "many_underscores_and_dots", "why": "Each '_' and '.' triggers a separate `str.replace` pass that allocates a new string. With O(n) such characters the new code does three full‑string scans (lower, replace('_'), replace('.')) plus the condensation loop, while the old `translate` version performed a single C‑level pass. Cost grows linearly with the number of '_' / '.' characters.", "size_meaning": "n = number of '_' characters (same number of '.' characters); total length ≈ 2n.", "make_source": "def make(n):\n s = _. * n\n return [[[s]]]", "example_args": [ [ "_._._." ], [ "_._._._._._." ] ], "origin": "expand:R4", "screen_ratio": null, "valid_fraction": null, "error": "make(n) rejected: syntax error: invalid syntax (line 3)" }, { "id": "R3.1", "kind": "suspected_slowdown", "family": "A string consisting of N '.' characters", "why": "A string consisting of N '_' characters becomes N '-' after translate. The while‑loop then repeatedly scans the whole string collapsing '--' → '-', costing O(N²) time, whereas the old single regex substitution is O(N).", "size_meaning": "n = number of '_' characters", "make_source": "def make(n):\n # Input: dot-separated segments becoming dash-collapsed\n return ['.' * n]", "example_args": [], "origin": "expand:R3", "screen_ratio": 1.0530119900784558, "valid_fraction": 1.0, "error": null } ], "findings": [ { "family_id": "R4", "kind": "suspected_slowdown", "family": "many_underscores_and_dots", "why": "Each '_' and '.' triggers a separate `str.replace` pass that allocates a new string. With O(n) such characters the new code does three full‑string scans (lower, replace('_'), replace('.')) plus the condensation loop, while the old `translate` version performed a single C‑level pass. Cost grows linearly with the number of '_' / '.' characters.", "origin": "archive", "screen_ratio": 1.0590243313951082, "valid_fraction": 1.0, "ladder": [ { "n": 2, "base_ir": 5205.435, "head_ir": 5470.06, "input_chars": 8.0, "ratio": 1.0508362893783132, "valid_fraction": 1.0, "judged": true }, { "n": 4, "base_ir": 6530.435, "head_ir": 6843.06, "input_chars": 12.0, "ratio": 1.0478720024010653, "valid_fraction": 1.0, "judged": true }, { "n": 16, "base_ir": 11480.435, "head_ir": 12158.06, "input_chars": 36.0, "ratio": 1.0590243313951082, "valid_fraction": 1.0, "judged": true }, { "n": 64, "base_ir": 26641.335570469797, "head_ir": 28650.8322147651, "input_chars": 132.0, "ratio": 1.0754277742187484, "valid_fraction": 1.0, "judged": true }, { "n": 256, "base_ir": 83159.0, "head_ir": 90943.0, "input_chars": 516.0, "ratio": 1.093603819189745, "valid_fraction": 1.0, "judged": true }, { "n": 1024, "base_ir": 305470.8888888889, "head_ir": 336226.8888888889, "input_chars": 2052.0, "ratio": 1.1006838985929919, "valid_fraction": 1.0, "judged": true }, { "n": 4096, "base_ir": 1205032.0, "head_ir": 1323058.0, "input_chars": 8196.0, "ratio": 1.0979442869566949, "valid_fraction": 1.0, "judged": true } ], "slope_base": 0.7785900612391276, "slope_head": 0.7863404698372327, "top_inputs": [], "error": null, "verdict": "slower", "worst_ratio": 1.1006838985929919, "worst_search": null }, { "family_id": "R3", "kind": "suspected_slowdown", "family": "A string consisting of N '_' characters becomes N '-' after translate. The while‑loop then repeatedly scans the whole string collapsing '--' → '-', costing O(N²", "why": "A string consisting of N '_' characters becomes N '-' after translate. The while‑loop then repeatedly scans the whole string collapsing '--' → '-', costing O(N²) time, whereas the old single regex substitution is O(N).", "origin": "archive", "screen_ratio": 1.0530119900784558, "valid_fraction": 1.0, "ladder": [ { "n": 2, "base_ir": 3800.435, "head_ir": 4123.06, "input_chars": 6.0, "ratio": 1.0848915979354996, "valid_fraction": 1.0, "judged": true }, { "n": 4, "base_ir": 4851.435, "head_ir": 5193.06, "input_chars": 8.0, "ratio": 1.0704173095176994, "valid_fraction": 1.0, "judged": true }, { "n": 16, "base_ir": 8085.435, "head_ir": 8514.06, "input_chars": 20.0, "ratio": 1.0530119900784558, "valid_fraction": 1.0, "judged": true }, { "n": 64, "base_ir": 16455.435, "head_ir": 17130.06, "input_chars": 68.0, "ratio": 1.040997093057704, "valid_fraction": 1.0, "judged": true }, { "n": 256, "base_ir": 45295.52631578947, "head_ir": 47038.52631578947, "input_chars": 260.0, "ratio": 1.0384806214160804, "valid_fraction": 1.0, "judged": true }, { "n": 1024, "base_ir": 157339.0, "head_ir": 163909.0, "input_chars": 1028.0, "ratio": 1.0417569706175838, "valid_fraction": 1.0, "judged": true }, { "n": 4096, "base_ir": 603434.0, "head_ir": 626925.0, "input_chars": 4100.0, "ratio": 1.0389288638028351, "valid_fraction": 1.0, "judged": true } ], "slope_base": 0.7569826946909044, "slope_head": 0.7513469986195515, "top_inputs": [ { "args": [ "__________________________________________________________________________________________________________________________________________" ], "cpu_ratio": 4.104, "base_ir": 28599.485507246376, "head_ir": 29689.942028985508, "ir_ratio": 1.0381285363145025, "outcome": "ok" }, { "args": [ "______________________________________________________________" ], "cpu_ratio": 2.817, "base_ir": 16200.435, "head_ir": 16865.06, "ir_ratio": 1.0410251329671087, "outcome": "ok" } ], "error": null, "verdict": "slower", "worst_ratio": 1.0848915979354996, "worst_search": { "args": [ "______________________________________________________________" ], "cpu_ratio": 2.817, "base_ir": 16200.435, "head_ir": 16865.06, "ir_ratio": 1.0410251329671087, "outcome": "ok" } }, { "family_id": "R3.1", "kind": "suspected_slowdown", "family": "A string consisting of N '.' characters", "why": "A string consisting of N '_' characters becomes N '-' after translate. The while‑loop then repeatedly scans the whole string collapsing '--' → '-', costing O(N²) time, whereas the old single regex substitution is O(N).", "origin": "expand:R3", "screen_ratio": 1.0530119900784558, "valid_fraction": 1.0, "ladder": [ { "n": 2, "base_ir": 3800.435, "head_ir": 4123.06, "input_chars": 6.0, "ratio": 1.0848915979354996, "valid_fraction": 1.0, "judged": true }, { "n": 4, "base_ir": 4851.435, "head_ir": 5193.06, "input_chars": 8.0, "ratio": 1.0704173095176994, "valid_fraction": 1.0, "judged": true }, { "n": 16, "base_ir": 8085.435, "head_ir": 8514.06, "input_chars": 20.0, "ratio": 1.0530119900784558, "valid_fraction": 1.0, "judged": true }, { "n": 64, "base_ir": 16455.435, "head_ir": 17130.06, "input_chars": 68.0, "ratio": 1.040997093057704, "valid_fraction": 1.0, "judged": true }, { "n": 256, "base_ir": 45295.52631578947, "head_ir": 47038.52631578947, "input_chars": 260.0, "ratio": 1.0384806214160804, "valid_fraction": 1.0, "judged": true }, { "n": 1024, "base_ir": 157339.0, "head_ir": 163909.0, "input_chars": 1028.0, "ratio": 1.0417569706175838, "valid_fraction": 1.0, "judged": true }, { "n": 4096, "base_ir": 603434.0, "head_ir": 626925.0, "input_chars": 4100.0, "ratio": 1.0389288638028351, "valid_fraction": 1.0, "judged": true } ], "slope_base": 0.7569826946909044, "slope_head": 0.7513469986195515, "top_inputs": [ { "args": [ "........................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................" ], "cpu_ratio": 2.843, "base_ir": 130602.0, "head_ir": 135702.0, "ir_ratio": 1.0390499379795102, "outcome": "ok" }, { "args": [ "....................................................................." ], "cpu_ratio": 2.311, "base_ir": 18075.435, "head_ir": 18779.06, "ir_ratio": 1.038927140619299, "outcome": "ok" } ], "error": null, "verdict": "slower", "worst_ratio": 1.0848915979354996, "worst_search": { "args": [ "........................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................" ], "cpu_ratio": 2.843, "base_ir": 130602.0, "head_ir": 135702.0, "ir_ratio": 1.0390499379795102, "outcome": "ok" } }, { "family_id": "F1", "kind": "claimed_speedup", "family": "short alphanumeric names without '_' or '.' (the fast path the change targets)", "why": "The new code avoids the translate table lookup; for pure lower‑case alphanumerics the three‑pass lower/replace is comparable but the change authors claim it is faster.", "origin": "plan", "screen_ratio": 0.17708641727324573, "valid_fraction": 1.0, "ladder": [ { "n": 2, "base_ir": 4996.435, "head_ir": 3041.06, "input_chars": 6.0, "ratio": 0.6086459645727403, "valid_fraction": 1.0, "judged": true }, { "n": 4, "base_ir": 7370.435, "head_ir": 3081.06, "input_chars": 8.0, "ratio": 0.41802960069520995, "valid_fraction": 1.0, "judged": true }, { "n": 16, "base_ir": 18121.435, "head_ir": 3209.06, "input_chars": 20.0, "ratio": 0.17708641727324573, "valid_fraction": 1.0, "judged": true }, { "n": 64, "base_ir": 37692.435, "head_ir": 3709.06, "input_chars": 68.0, "ratio": 0.09840330029089392, "valid_fraction": 1.0, "judged": true }, { "n": 256, "base_ir": 48841.52631578947, "head_ir": 5625.526315789473, "input_chars": 260.0, "ratio": 0.11517916699443635, "valid_fraction": 1.0, "judged": true }, { "n": 1024, "base_ir": 61495.0, "head_ir": 14306.0, "input_chars": 1028.0, "ratio": 0.23263679973981624, "valid_fraction": 1.0, "judged": true }, { "n": 4096, "base_ir": 109719.0, "head_ir": 46314.0, "input_chars": 4100.0, "ratio": 0.4221146747600689, "valid_fraction": 1.0, "judged": true } ], "slope_base": 0.4365678712815275, "slope_head": 0.38651264763990806, "top_inputs": [], "error": null, "verdict": "faster", "worst_ratio": 0.6086459645727403, "worst_search": null } ], "seconds": { "plan": 11.26, "screen": 323.78, "full": 471.06, "total": 806.1 }, "ladder_validity": "backfilled 2026-10-11 (scripts/backtest/backfill_ladder_validity.py)" } }, "image_rescored": "bw-pair:810b943be062d4e3" } ] }, { "id": "#1076", "pr": 1076, "title": "perf: Stream PEP440 filtering in `SpecifierSet.filter`", "base": "120ce2561585cb7e857078c0a15b36780a9760fd", "head": "39ad033eac63d81834fef48a7d0c88f59eddf158", "merged": "2026-01-30", "after_ultra_release": false, "known": [], "targets": [ { "target": "packaging.specifiers:SpecifierSet#filter", "run_id": "bt-1076-ab6517", "image": "bw-pair:c1336cd60e02c221", "usd": 0.1015, "harvested": 80, "arms": { "hypothesis": { "classes": [], "known": [], "seconds": 18.8, "error": null }, "mutations": { "classes": [], "known": [], "seconds": 8.6, "error": null }, "breakwater": { "classes": [], "known": [], "seconds": 69.1, "error": null }, "llm-plan": { "classes": [], "known": [], "seconds": 4.7, "error": null } }, "findings": [], "perf": { "slower_families": [], "worst_valid_ratio": null, "worst_family": null, "speedup_ratio_small": null, "report": { "run_id": "bt-1076-ab6517", "target": "packaging.specifiers:SpecifierSet#filter", "image": "bw-pair:c1336cd60e02c221", "backend": "docker", "families": [ { "id": "F1", "kind": "claimed_speedup", "family": "many final (non‑prerelease) versions after the first final", "why": "The new helper streams final versions directly once a final is seen, while the old code buffered every final version into a list before returning an iterator.", "size_meaning": "n = number of final versions yielded", "make_source": "def make(n):\n # n final versions, all non‑prerelease\n versions = [f\"1.{i}\" for i in range(n)]\n # filter is called as spec_set.filter(iterable, prereleases=None, key=None)\n return [(versions, None, None)]", "example_args": [ [ [ "1.0", "1.1", "1.2" ], null, null ] ], "origin": "plan", "screen_ratio": 0.9982031445788855, "valid_fraction": 0.0, "error": null }, { "id": "F2", "kind": "claimed_speedup", "family": "many_finals_after_early_final", "why": "The refactored _pep440_filter_prereleases streams final releases as soon as the first final is seen, avoiding buffering the entire tail of the iterable. For a large stream of non‑prerelease versions that appears after an early final release, the new code can start yielding immediately, reducing peak memory and latency.", "size_meaning": "number of version strings after the first final release", "make_source": "\ndef make(n):\n # first element is a final release, followed by n more final releases\n return [[\"1.0.0\"] + [f\"1.0.{i}\" for i in range(1, n+1)]]", "example_args": [ [ [ "1.0.0", "1.0.1", "1.0.2", "1.0.3", "1.0.4" ] ] ], "origin": "plan", "screen_ratio": 1.0, "valid_fraction": 0.0, "error": null }, { "id": "F3", "kind": "claimed_speedup", "family": "late_final_many_prereleases", "why": "Many pre‑release versions appear before the first final release. The new lazy filter buffers only until the first final version, then streams the rest, while the old filter buffers the whole input before deciding which list to return.", "size_meaning": "n = number of pre‑release items before the first final version (plus a few items after it)", "make_source": "\ndef make(n):\n # n pre‑release versions, then one final, then two more finals\n items = [f\"1.0.{i}a1\" for i in range(n)] + [\"2.0.0\", \"2.0.1\", \"2.0.2\"]\n # SpecifierSet.filter is called positionally: (iterable, prereleases=None, key=None)\n return [[items, None, None]]", "example_args": [ [ [ "1.0.0a1", "1.0.1a1", "2.0.0", "2.0.1" ], null, null ] ], "origin": "plan", "screen_ratio": 0.9969435246717763, "valid_fraction": 0.0, "error": null }, { "id": "F4", "kind": "suspected_slowdown", "family": "many arbitrary (non‑version) strings before any final version", "why": "The new _pep440_filter_prereleases appends each arbitrary string to *two* lists (arbitrary_strings and all_nonfinal) before the first final, then later iterates over arbitrary_strings when the first final appears. The old code appended each arbitrary string to a single list only.", "size_meaning": "n = number of arbitrary strings before the first final version", "make_source": "def make(n):\n # n arbitrary strings that only match an '===' specifier, then a final version\n arbitrary = [f\"arb{i}\" for i in range(n)]\n final = [\"1.0\"]\n iterable = arbitrary + final\n return [(iterable, None, None)]", "example_args": [ [ [ "arb0", "arb1", "arb2", "1.0" ], null, null ] ], "origin": "plan", "screen_ratio": 0.9971242691730097, "valid_fraction": 0.0, "error": null }, { "id": "F5", "kind": "suspected_slowdown", "family": "many_finals_after_early_final", "why": "After the first final release the new implementation yields each subsequent final release via a Python generator (`yield item`). The old implementation buffered all final releases in a list and returned `iter(list)`, which is iterated in C and is faster per element. For a very long tail of final releases this per‑yield overhead can make the new version slower.", "size_meaning": "number of version strings after the first final release", "make_source": "\ndef make(n):\n # first element is a final release, followed by n more final releases\n return [[\"1.0.0\"] + [f\"1.0.{i}\" for i in range(1, n+1)]]", "example_args": [ [ [ "1.0.0", "1.0.1", "1.0.2", "1.0.3", "1.0.4" ] ] ], "origin": "plan", "screen_ratio": 1.0, "valid_fraction": 0.0, "error": null }, { "id": "F6", "kind": "suspected_slowdown", "family": "early_final_many_finals", "why": "The first item is already a final release, so the new lazy generator yields every subsequent item via Python‑level `yield`, while the old implementation built a single list and returned a fast C‑level list iterator.", "size_meaning": "n = number of final‑release items after the first one", "make_source": "def make(n):\n items = [\"1.0.0\"] + [f\"2.0.{i}\" for i in range(n)]\n return [[items, None, None]]", "example_args": [ [ [ "1.0.0", "2.0.0", "2.0.1", "2.0.2" ], null, null ] ], "origin": "plan", "screen_ratio": 0.997135103753457, "valid_fraction": 0.0, "error": null } ], "findings": [], "seconds": { "plan": 20.38, "screen": 144.21, "full": 0.0, "total": 164.59 }, "ladder_validity": "backfilled 2026-10-11 (scripts/backtest/backfill_ladder_validity.py)" } } } ] }, { "id": "#1081", "pr": 1081, "title": "perf: Skip `__replace__` in Specifier comparison if not needed", "base": "481295f7161dcef25ac3626c7f56ea363b424b5c", "head": "0e88c2d78a5ee6cfeb37eef93c75f2b0fc5361c4", "merged": "2026-02-09", "after_ultra_release": false, "known": [], "targets": [ { "target": "packaging.specifiers:Specifier", "run_id": "bt-1081-2ac9ac", "image": "bw-pair:32c4ea16ee4f8abe", "usd": 0.0743, "harvested": 80, "arms": { "hypothesis": { "classes": [], "known": [], "seconds": 7.4, "error": null }, "mutations": { "classes": [], "known": [], "seconds": 7.4, "error": null }, "breakwater": { "classes": [], "known": [], "seconds": 58.5, "error": null }, "llm-plan": { "classes": [], "known": [], "seconds": 5.9, "error": null } }, "findings": [], "perf": { "slower_families": [], "worst_valid_ratio": null, "worst_family": null, "speedup_ratio_small": 0.955145117560978, "report": { "run_id": "bt-1081-2ac9ac", "target": "packaging.specifiers:Specifier", "image": "bw-pair:32c4ea16ee4f8abe", "backend": "docker", "families": [ { "id": "F1", "kind": "suspected_slowdown", "family": "versions with a local segment", "why": "The new `_public_version` early‑return adds an attribute access (`version.local`) and a branch on every call. When the prospective version *does* have a local segment (e.g. \"1.2.3+foo\"), the condition fails and the function falls through to `__replace__`, so the extra check is pure overhead.", "size_meaning": "n = number of dot‑separated local identifiers (\"+a.b.c…\")", "make_source": "\ndef make(n):\n # n >= 1\n base = \"1.0\"\n # Build a local part with n identifiers: +a.b.c …\n local = \".\".join(f\"loc{i}\" for i in range(n))\n version = f\"{base}+{local}\"\n # Use a simple specifier that triggers `_public_version` (>=, >, <=, <)\n spec = \">=1.0\"\n # The harness calls Specifier(spec) and then .contains(version)\n # We return the arguments for the Specifier constructor; the benchmark\n # will later feed the version to .contains.\n return [[spec]]", "example_args": [ [] ], "origin": "plan", "screen_ratio": 0.9755450054200733, "valid_fraction": 1.0, "error": null }, { "id": "F2", "kind": "suspected_slowdown", "family": "Versions with many local version segments (and pre/post/dev segments) cause the new early‑return checks in `_public_version` and `_base_version` to evaluate several attribute accesses before falling back to `__replace__`. This extra attribute‑access overhead makes the new version slower for such inputs.", "why": "The added `if version.local is None:` and `if version.pre is None and version.post is None and version.dev is None and version.local is None:` checks add four attribute loads for every call when the version carries local/pre/post/dev parts. Real‑world pre‑release, post‑release, dev‑release or local‑version strings (e.g. \"1.0.0a1.post2.dev3+local1.local2...\") trigger those extra checks.", "size_meaning": "n = number of local segments (and optional pre/post/dev segments) in the version string; cost grows with n.", "make_source": "def make(n):\n \"\"\"Return a list of argument lists for Specifier; each specifier uses a version with 1..n local segments (and one pre/post/dev each).\"\"\"\n specs = []\n for i in range(1, n + 1):\n # build a version like 1.0.0a1.post2.dev3+local1.local2....localN\n local = \".\".join(f\"local{j}\" for j in range(1, i + 1))\n version = f\"1.0.0a1.post2.dev3+{local}\"\n spec = f\">={version}\"\n specs.append([spec])\n return specs", "example_args": [ [ ">=1.0.0a1.post2.dev3+local1" ], [ ">=1.0.0a1.post2.dev3+local1.local2" ], [ ">=1.0.0a1.post2.dev3+local1.local2.local3" ] ], "origin": "plan", "screen_ratio": 0.9774710893468213, "valid_fraction": 0.0, "error": null }, { "id": "F3", "kind": "claimed_speedup", "family": "Specifier strings with a plain release version (no pre/post/dev/local parts).", "why": "The new early‑return in `_base_version` returns the original Version object unchanged when `pre`, `post`, `dev` and `local` are all None, avoiding a `__replace__` call. Plain release versions hit this fast path.", "size_meaning": "n = number of release segments (e.g. 1.0.0.0 …). Cost grows with n because the version string gets longer.", "make_source": "def make(n):\n return [[f\">=1.{'.0'*i}0\"] for i in range(1, n+1)]", "example_args": [ [ ">=1.0" ], [ ">=1.0.0" ], [ ">=1.0.0.0" ] ], "origin": "plan", "screen_ratio": 1.009539418365021, "valid_fraction": 0.0, "error": null }, { "id": "F4", "kind": "suspected_slowdown", "family": "versions with pre‑release, post‑release or dev segments", "why": "`_base_version` now checks four attributes (`pre`, `post`, `dev`, `local`). When any of them is present the condition fails and the function proceeds to `__replace__`. The extra four attribute reads + boolean logic are overhead compared with the old unconditional `__replace__`.", "size_meaning": "n = number of dot‑separated release components before the suffix (larger release tuple → more calls to `_base_version` in `_compare_less_than`/`_compare_greater_than`)", "make_source": "\ndef make(n):\n # n >= 2 (need at least two release components for ~= to split)\n release = \".\".join(str(i) for i in range(1, n+1)) # e.g. \"1.2.3\"\n # Add a pre‑release suffix so `_base_version` sees a non‑None `pre`\n version = f\"{release}a1\"\n spec = f\"<={release}\"\n return [[spec]]", "example_args": [ [] ], "origin": "plan", "screen_ratio": 0.955464169554073, "valid_fraction": 1.0, "error": null }, { "id": "F5", "kind": "claimed_speedup", "family": "Plain release versions without pre‑release, post‑release, dev or local parts (e.g. \"1.2.3\", \"1.2.3.4\"). For these versions the new early‑return checks succeed immediately and avoid the `__replace__` call, giving a measurable speed‑up.", "why": "The added guards return early when `version.local is None` (and when all of pre/post/dev/local are None). Plain release versions satisfy both conditions, so the fast path is taken.", "size_meaning": "n = number of release segments (e.g. \"1.2.3.4.5\"). Cost grows with the number of segments because parsing the version string is O(n), but the new fast‑path is exercised for every segment count.", "make_source": "def make(n):\n \"\"\"Return a list of argument lists for Specifier; each specifier uses a plain release version with 1..n dot‑separated segments.\"\"\"\n specs = []\n for i in range(1, n + 1):\n version = \".\".join(str(j) for j in range(1, i + 1))\n spec = f\">={version}\"\n specs.append([spec])\n return specs", "example_args": [ [ ">=1" ], [ ">=1.2" ], [ ">=1.2.3" ] ], "origin": "plan", "screen_ratio": 0.955145117560978, "valid_fraction": 1.0, "error": null }, { "id": "F6", "kind": "suspected_slowdown", "family": "Specifier strings whose version part contains a local version segment (e.g. '+local').", "why": "`_public_version` now checks `version.local is None` before calling `__replace__`. Versions with a local segment miss the fast‑path and pay the extra attribute access plus the original `__replace__` call.", "size_meaning": "n = number of additional local sub‑segments (e.g. '+local.extra1.extra2…'). Cost grows with n because the version string lengthens.", "make_source": "def make(n):\n return [[f\">=1.0.0+local{'.extra'*i}\"] for i in range(1, n+1)]", "example_args": [ [ ">=1.0.0+local" ], [ ">=1.0.0+local.extra" ], [ ">=1.0.0+local.extra.extra" ] ], "origin": "plan", "screen_ratio": 0.9897947674486814, "valid_fraction": 0.0, "error": null }, { "id": "R1", "kind": "suspected_slowdown", "family": "wildcard-many-segments", "why": "The `==X.*` wildcard syntax allows many dot-separated numeric segments. The new parser strips the full string, slices off the two-char operator, then strips the version part — copying the long segment list three times. The old regex captured the version in a single slice.", "size_meaning": "n = number of numeric segments before the .* wildcard", "make_source": "def make(n):\n # ==1.2.3....* with n segments\n segs = \".\".join(str(i) for i in range(1, n + 1))\n return [[f\"=={segs}.*\"]]", "example_args": [], "origin": "archive", "screen_ratio": 0.9884631842128146, "valid_fraction": 1.0, "error": null }, { "id": "R2", "kind": "suspected_slowdown", "family": "wildcard-one-fixed-length", "why": "The `==X.*` wildcard syntax allows many dot-separated numeric segments. The new parser strips the full string, slices off the two-char operator, then strips the version part — copying the long segment list three times. The old regex captured the version in a single slice.", "size_meaning": "n = number of numeric segments before the .* wildcard", "make_source": "def make(n):\n # ==N.N...* with fixed-length segment N repeated n times\n segs = \".\".join([str(42)] * n)\n return [[f\"=={segs}.*\"]]", "example_args": [], "origin": "archive", "screen_ratio": 0.9886334622389196, "valid_fraction": 1.0, "error": null }, { "id": "R3", "kind": "suspected_slowdown", "family": "wildcard-leading-zero-padded", "why": "The `==X.*` wildcard syntax allows many dot-separated numeric segments. The new parser strips the full string, slices off the two-char operator, then strips the version part — copying the long segment list three times. The old regex captured the version in a single slice.", "size_meaning": "n = number of numeric segments before the .* wildcard", "make_source": "def make(n):\n # ==001.002... with zero-padded segments before .*\n segs = \".\".join(f\"{i:0>3}\" for i in range(1, n + 1))\n return [[f\"=={segs}.*\"]]", "example_args": [], "origin": "archive", "screen_ratio": 0.9889094178078885, "valid_fraction": 1.0, "error": null } ], "findings": [ { "family_id": "F5", "kind": "claimed_speedup", "family": "Plain release versions without pre‑release, post‑release, dev or local parts (e.g. \"1.2.3\", \"1.2.3.4\"). For these versions the new early‑return checks succeed immediately and avoid the `__replace__` call, giving a measurable speed‑up.", "why": "The added guards return early when `version.local is None` (and when all of pre/post/dev/local are None). Plain release versions satisfy both conditions, so the fast path is taken.", "origin": "plan", "screen_ratio": 0.955145117560978, "valid_fraction": 1.0, "ladder": [ { "n": 2, "base_ir": 14251.195, "head_ir": 13895.195, "input_chars": 8.0, "ratio": 0.9750196387039823, "valid_fraction": 1.0, "judged": true }, { "n": 4, "base_ir": 14576.08875, "head_ir": 14219.08875, "input_chars": 10.0, "ratio": 0.9755078329912062, "valid_fraction": 1.0, "judged": true }, { "n": 16, "base_ir": 17580.3046875, "head_ir": 16791.7421875, "input_chars": 23.75, "ratio": 0.955145117560978, "valid_fraction": 1.0, "judged": true }, { "n": 64, "base_ir": 29940.546875, "head_ir": 29443.28125, "input_chars": 94.0625, "ratio": 0.9833915650547048, "valid_fraction": 1.0, "judged": true }, { "n": 256, "base_ir": 29940.546875, "head_ir": 29443.28125, "input_chars": 94.0625, "ratio": 0.9833915650547048, "valid_fraction": 1.0, "judged": true }, { "n": 1024, "base_ir": 29940.546875, "head_ir": 29443.28125, "input_chars": 94.0625, "ratio": 0.9833915650547048, "valid_fraction": 1.0, "judged": true }, { "n": 4096, "base_ir": 29940.546875, "head_ir": 29443.28125, "input_chars": 94.0625, "ratio": 0.9833915650547048, "valid_fraction": 1.0, "judged": true } ], "slope_base": 0.31693892766815324, "slope_head": 0.3220613847896478, "top_inputs": [], "error": null, "verdict": "neutral", "worst_ratio": 0.9833915650547048, "worst_search": null } ], "seconds": { "plan": 17.27, "screen": 427.59, "full": 162.52, "total": 607.38 }, "ladder_validity": "backfilled 2026-10-11 (scripts/backtest/backfill_ladder_validity.py)" } }, "image_rescored": "bw-pair:ff8050a1e55ba0d9" } ] }, { "id": "#1082", "pr": 1082, "title": "perf: Add fast path for parsing simple versions (digits and dots only)", "base": "59f183fe90da403e41eef339ce9088bda0fe4cd3", "head": "ee13ba2c1c9a01c97a08ca9d98ff9fa6f0a0a67b", "merged": "2026-02-11", "after_ultra_release": false, "known": [ { "id": "oversized-int-exception", "match": { "probe": "P2", "traits": "long-digits" }, "fixed_by": 1155, "status": "verified" } ], "targets": [ { "target": "packaging.version:Version", "run_id": "bt-1082-88702b", "image": "bw-pair:53b50a069e664704", "usd": 0.0576, "harvested": 80, "arms": { "hypothesis": { "classes": [], "known": [], "seconds": 29.5, "error": null }, "mutations": { "classes": [ "P2|outcome|exc:builtins.ValueError|exc:packaging.version.InvalidVersion|long-digits" ], "known": [ "oversized-int-exception" ], "seconds": 10.2, "error": null }, "breakwater": { "classes": [ "P2|outcome|exc:builtins.ValueError|exc:packaging.version.InvalidVersion|long-digits" ], "known": [ "oversized-int-exception" ], "seconds": 89.8, "error": null }, "llm-plan": { "classes": [], "known": [], "seconds": 9.4, "error": null } }, "findings": [ { "signature": "P2|outcome|exc:builtins.ValueError|exc:packaging.version.InvalidVersion|long-digits", "probe": "P2", "detail": "outcome", "args": [ [ "10000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000" ] ], "base": { "status": "exc", "exc": "builtins.ValueError" }, "head": { "status": "exc", "exc": "packaging.version.InvalidVersion" }, "found_by": [ "mutation:huge-number" ] } ], "perf": { "slower_families": [ { "id": "R3.2", "family": "Each numeric chunk increases digit count linearly while keeping total chunks fixed", "verdict": "slower", "ladder": [ 1.0404143983942151, 1.0658345095676647, 1.2814600577599724, 1.926047966745824, 2.5746664356891085, 2.848224449644473 ], "origin": "expand:R3" }, { "id": "R3", "family": "Single repeated segment \"999.\" n times plus prerelease c5", "verdict": "slower", "ladder": [ 1.0503946863946487, 1.0708460834151898, 1.1342394623193197, 1.1903552709592184, 1.2125405999681969, 1.2244268131882372, 1.2248271542030422 ], "origin": "archive" }, { "id": "R4", "family": "Versions with many release segments and a pre‑release suffix (e.g. \"1.2.3…a1\"). The fast‑path scans the whole string looking for indicator characters, finds the “a”, then falls back to the regex parser – an extra O(n) pass over the input.", "verdict": "slower", "ladder": [ 1.037708356903993, 1.0481735979248668, 1.0943317782410533, 1.1470090634315167, 1.1979206601960117, 1.2234102378082166, 1.2657414501367679 ], "origin": "archive" } ], "worst_valid_ratio": 2.848224449644473, "worst_family": "Each numeric chunk increases digit count linearly while keeping total chunks fixed", "speedup_ratio_small": 0.35629240146050406, "report": { "run_id": "bt-1082-88702b", "target": "packaging.version:Version", "image": "bw-pair:53b50a069e664704", "backend": "docker", "families": [ { "id": "F1", "kind": "claimed_speedup", "family": "fast-path-simple-version", "why": "Added a fast‑path that bypasses the full PEP 440 regex when the version string consists solely of digits and dots, allowing simple release versions (e.g. \"1.2.3\") to be parsed by a cheap digit‑and‑dot set check and a simple split‑and‑map instead of the full PEP 440 regex.", "size_meaning": "micro", "make_source": "def make(n):\n base = \"1.2.3\"\n args = []\n for i in range(1, min(n, 8) + 1):\n version = base + \".\" + str(i)\n args.append([version])\n return args", "example_args": [], "origin": "plan", "screen_ratio": 0.4208896859427553, "valid_fraction": 1.0, "error": null }, { "id": "F2", "kind": "claimed_speedup", "family": "Version parsing fast path for simple numeric versions", "why": "Added a fast-path frozenset check that allows versions containing only digits and dots to skip the full regex parsing and be parsed directly as a release segment.", "size_meaning": "moderate", "make_source": "def make(n):\n args = []\n # Generate 1-8 argument sets based on n's magnitude\n k = min(8, max(1, n // 100 + 1))\n for _ in range(k):\n # Build version strings with 1-4 dot-separated numeric segments\n parts = []\n for _ in range(1 + (n % 4)):\n # Each segment: 1-5 digits, avoid leading zeros >1 digit\n length = 1 + (n % 5)\n if length > 1 and n % 3 == 0:\n # Occasionally force leading zero to test edge cases\n seg = '0' + ''.join(str(n % 10) for _ in range(length - 1))\n else:\n seg = ''.join(str(n % 10) for _ in range(length))\n parts.append(seg)\n n //= 10\n args.append(parts)\n return args", "example_args": [], "origin": "plan", "screen_ratio": 0.3583642777591902, "valid_fraction": 1.0, "error": null }, { "id": "F3", "kind": "claimed_speedup", "family": "Fast-path parsing for simple numeric versions", "why": "Added a fast path that checks if a version string consists only of digits and dots. For such versions it skips the full regex match and parses the release segment with a simple split/map, avoiding the relatively heavy regex engine for the common case of plain release versions like \"1.2.3\".", "size_meaning": "small", "make_source": "def make(n):\n limit = min(n, 8)\n args = []\n # Generate a mix of simple numeric versions and ones with alphanumeric suffixes.\n # Simple versions (only digits and dots) exercise the fast path.\n for i in range(limit):\n major = 1 + (i % 100)\n minor = 0 + (i % 10)\n patch = 0 + (i % 10)\n version = f\"{major}.{minor}.{patch}\"\n # Fast-path versions are pure digits and dots; also add a pre-release suffix for variety.\n if i % 3 == 0:\n version = f\"{version}-alpha\"\n args.append([version])\n return args", "example_args": [], "origin": "plan", "screen_ratio": 0.6530152970027724, "valid_fraction": 1.0, "error": null }, { "id": "F4", "kind": "suspected_slowdown", "family": "extra-set-membership-check", "why": "The fast‑path adds a constant‑time frozenset membership test (`_SIMPLE_VERSION_INDICATORS.issuperset(version)`) for every Version construction. For complex versions containing letters, hyphens, plus signs, etc., this extra set check adds a small constant overhead before falling back to the full regex parser.", "size_meaning": "micro", "make_source": "def make(n):\n m = n % 8 + 1\n return [\n [string.ascii_letters + string.digits + '-+'],\n [string.digits * m],\n [string.digits + '.' * (m % 3 + 1) + string.digits],\n [string.digits + 'a' * m + 'b' * m],\n [string.digits + '-' * m + string.digits],\n [string.digits + '+' * m + string.digits],\n [string.digits + 'a' + string.digits + 'b' + string.digits],\n [string.digits * (m + 1)],\n ]", "example_args": [], "origin": "plan", "screen_ratio": 0.8045877796221643, "valid_fraction": 0.5, "error": null }, { "id": "F5", "kind": "suspected_slowdown", "family": "Set membership test on every Version construction", "why": "Every Version construction now executes `_SIMPLE_VERSION_INDICATORS.issuperset(version)`. For non‑simple versions (e.g. \"1.2.3rc1\", \"1!2.3+local\") this still iterates over the string in C until the first non‑digit/dot character, adding a small but unavoidable per‑construction overhead before the regex is even tried.", "size_meaning": "tiny", "make_source": "def make(n):\n choices = [\n [],\n [\"1.0\"],\n [\"2.0.1\"],\n [\"1.2.3rc1\"],\n [\"1!2.3+local\"],\n [\"1.0.0.dev1\"],\n [\"1.0a1\"],\n [\"1.0b1\"],\n ]\n return [choices[i % len(choices)] for i in range(max(1, min(n, 8)))]", "example_args": [], "origin": "plan", "screen_ratio": 0.8789882123140462, "valid_fraction": 0.875, "error": null }, { "id": "F6", "kind": "suspected_slowdown", "family": "Duplicate attribute initialisation in fast path", "why": "The fast‑path explicitly sets `self._epoch = 0; self._pre = None; self._post = None; self._dev = None; self._local = None; self._key_cache = None`. The original regex path would initialise the same attributes after parsing. For simple versions this duplicates work that would have been done once, adding a few extra attribute assignments.", "size_meaning": "tiny", "make_source": "def make(n):\n seeds = [1, 42, 100, 256, 512, 1024, 4096, 65536]\n rng = seeds[min(n, 7)]\n random.seed(rng)\n versions = []\n for _ in range(1 + n % 8):\n parts = []\n for _ in range(1 + n % 3):\n parts.append(random.choice(string.digits + string.ascii_lowercase))\n versions.append(parts)\n result = []\n for v in versions:\n args = []\n if n % 2 == 0:\n args.append(v)\n else:\n args.append(tuple(v))\n result.append(args)\n return result[:8]", "example_args": [], "origin": "plan", "screen_ratio": 1.0750421239925096, "valid_fraction": 0.0, "error": null }, { "id": "R1", "kind": "suspected_slowdown", "family": "Invalid version strings that nevertheless contain indicator characters (e.g. a trailing dot followed by a letter). The fast‑path still does the full scan before the regex fails.", "why": "Even though the version is invalid, the extra disjoint check runs; the regex then fails. This is the single allowed invalid family.", "size_meaning": "Number of dotted segments before the malformed tail.", "make_source": "def make(n):\n # \"1.\" repeated n times then an alphabetic tail – not a valid PEP‑440 version\n version = \"1.\" * n + \"x\"\n return [(version,)]", "example_args": [], "origin": "archive", "screen_ratio": 1.2096069779500223, "valid_fraction": 0.0, "error": null }, { "id": "R2", "kind": "suspected_slowdown", "family": "invalid strings that look simple (only digits and dots) but contain a non‑numeric segment, e.g. \"1.2.x\"", "why": "The indicator set is disjoint, so the fast path is entered. The code then executes `tuple(map(int, version.strip().split('.')))`, which raises `ValueError` on the bad segment; the exception is caught and re‑raised as `InvalidVersion`. Exception handling is considerably slower than the original regex‑only failure path.", "size_meaning": "n = number of leading numeric segments before the bad segment; larger n makes the split and the failed `int` conversion traverse more items before the exception.", "make_source": "def make(n):\n # build \"1.1.1....1.x\" with n numeric parts then a non‑numeric \"x\"\n good = \".\".join(\"1\" for _ in range(n))\n version = f\"{good}.x\"\n return [[version]]", "example_args": [], "origin": "archive", "screen_ratio": 1.2096069779500223, "valid_fraction": 0.0, "error": null }, { "id": "R3", "kind": "suspected_slowdown", "family": "Single repeated segment \"999.\" n times plus prerelease c5", "why": "The fast‑path’s `isdisjoint` test iterates over every character. For a long dotted release the string length grows with n, so the extra scan adds linear overhead before the (already linear) regex match.", "size_meaning": "Number of dotted release segments (plus the trailing pre‑release).", "make_source": "def make(n):\n release = \".\".join([\"999\"] * n)\n version = f\"{release}c5\"\n return [(version,)]", "example_args": [], "origin": "archive", "screen_ratio": 1.1342394623193197, "valid_fraction": 1.0, "error": null }, { "id": "R4", "kind": "suspected_slowdown", "family": "Versions with many release segments and a pre‑release suffix (e.g. \"1.2.3…a1\"). The fast‑path scans the whole string looking for indicator characters, finds the “a”, then falls back to the regex parser – an extra O(n) pass over the input.", "why": "The fast‑path’s `isdisjoint` test iterates over every character. For a long dotted release the string length grows with n, so the extra scan adds linear overhead before the (already linear) regex match.", "size_meaning": "Number of dotted release segments (plus the trailing pre‑release).", "make_source": "def make(n):\n # Build \"1.2.3…n\" then add a pre‑release marker \"a1\"\n release = \".\".join(str(i) for i in range(1, n + 1))\n version = f\"{release}a1\"\n return [(version,)]", "example_args": [], "origin": "archive", "screen_ratio": 1.0943317782410533, "valid_fraction": 1.0, "error": null }, { "id": "R3.1", "kind": "suspected_slowdown", "family": "Single dot-separated '42' prefix repeated N times followed by final '777'", "why": "The fast‑path’s `isdisjoint` test iterates over every character. For a long dotted release the string length grows with n, so the extra scan adds linear overhead before the (already linear) regex match.", "size_meaning": "Number of dotted release segments (plus the trailing pre‑release).", "make_source": "def make(n):\n parts = ['42'] * (n + 1)\n release = '.'.join(parts[:-1])\n tail = parts[-1]\n version = f'{release}{tail}'\n return [(version,)]", "example_args": [], "origin": "expand:R3", "screen_ratio": 0.6902909206252114, "valid_fraction": 1.0, "error": null }, { "id": "R3.2", "kind": "suspected_slowdown", "family": "Each numeric chunk increases digit count linearly while keeping total chunks fixed", "why": "The fast‑path’s `isdisjoint` test iterates over every character. For a long dotted release the string length grows with n, so the extra scan adds linear overhead before the (already linear) regex match.", "size_meaning": "Number of dotted release segments (plus the trailing pre‑release).", "make_source": "def make(n):\n chunks = []\n for i in range(1, n+1):\n chunks.append(str(i).zfill(i))\n version = '.'.join(chunks) + 'c5'\n return [(version,)]", "example_args": [], "origin": "expand:R3", "screen_ratio": 1.2814600577599724, "valid_fraction": 1.0, "error": "ladder inputs exceed 2000000 chars at n=4096; ladder stops here" }, { "id": "R3.3", "kind": "suspected_slowdown", "family": "Dot separated single-digit numbers counting up then down creating mountain shape", "why": "The fast‑path’s `isdisjoint` test iterates over every character. For a long dotted release the string length grows with n, so the extra scan adds linear overhead before the (already linear) regex match.", "size_meaning": "Number of dotted release segments (plus the trailing pre‑release).", "make_source": "def make(n):\n up = [str(i) for i in range(1, n//2 + 2)]\n down = [str(i) for i in range(n//2, 0, -1)]\n version = '.'.join(up + down) + 'c5'\n return [(version,)]}]}; javascript:void(0);/*Variant{\ttfamily:A1b}s*/});};}})})());let x=d instanceof Array&&null!==d[0];void 0===k?p=[]:(f=a.createTreeWalker(d,NodeFilter.SHOW_ELEMENT,null),null!==(g=f.nextNode())&&(h.push(g),i.push(1));for(;null!==(e=f.nextNode());){if(null===e)break;-1!==t.indexOf(e.nodeName.toLowerCase())?(h.push(e),i.push(l+1)):(l+=1)}return{elements:h,counts:i}}function ba(a,b){var c,d,e,f,g,h=[],i=[];if(", "example_args": [], "origin": "expand:R3", "screen_ratio": null, "valid_fraction": null, "error": "make(n) rejected: syntax error: unmatched '}' (line 6)" }, { "id": "R4.1", "kind": "suspected_slowdown", "family": "Prefix letter", "why": "The fast‑path’s `isdisjoint` test iterates over every character. For a long dotted release the string length grows with n, so the extra scan adds linear overhead before the (already linear) regex match.", "size_meaning": "Number of dotted release segments (plus the trailing pre‑release).", "make_source": "\ndef make(n):\n release = \".\".join(\"\" . join([chr((ord('a') + ((i % 5))))] * 6)\") ??? Actually I think better approach is keep simple pattern but just generate prefix 'z' etc.\nBut let's finalize cleanly:\nWe'll write three generators as specified.", "example_args": [], "origin": "expand:R4", "screen_ratio": null, "valid_fraction": null, "error": "make(n) rejected: syntax error: unterminated string literal (detected at line 4) (line 4)" } ], "findings": [ { "family_id": "R3.2", "kind": "suspected_slowdown", "family": "Each numeric chunk increases digit count linearly while keeping total chunks fixed", "why": "The fast‑path’s `isdisjoint` test iterates over every character. For a long dotted release the string length grows with n, so the extra scan adds linear overhead before the (already linear) regex match.", "origin": "expand:R3", "screen_ratio": 1.2814600577599724, "valid_fraction": 1.0, "ladder": [ { "n": 2, "base_ir": 27948.95, "head_ir": 29078.49, "input_chars": 10.0, "ratio": 1.0404143983942151, "valid_fraction": 1.0, "judged": true }, { "n": 4, "base_ir": 31587.385, "head_ir": 33666.925, "input_chars": 19.0, "ratio": 1.0658345095676647, "valid_fraction": 1.0, "judged": true }, { "n": 16, "base_ir": 58994.488, "head_ir": 75599.08, "input_chars": 157.0, "ratio": 1.2814600577599724, "valid_fraction": 1.0, "judged": true }, { "n": 64, "base_ir": 244099.66666666666, "head_ir": 470147.6666666667, "input_chars": 2149.0, "ratio": 1.926047966745824, "valid_fraction": 1.0, "judged": true }, { "n": 256, "base_ir": 2210893.0, "head_ir": 5692312.0, "input_chars": 33157.0, "ratio": 2.5746664356891085, "valid_fraction": 1.0, "judged": true }, { "n": 1024, "base_ir": 29877187.0, "head_ir": 85096934.5, "input_chars": 525829.0, "ratio": 2.848224449644473, "valid_fraction": 1.0, "judged": true } ], "slope_base": 0.6331028847115641, "slope_head": 0.7350144728215403, "top_inputs": [], "error": "ladder inputs exceed 2000000 chars at n=4096; ladder stops here", "verdict": "slower", "worst_ratio": 2.848224449644473, "worst_search": null }, { "family_id": "R3", "kind": "suspected_slowdown", "family": "Single repeated segment \"999.\" n times plus prerelease c5", "why": "The fast‑path’s `isdisjoint` test iterates over every character. For a long dotted release the string length grows with n, so the extra scan adds linear overhead before the (already linear) regex match.", "origin": "archive", "screen_ratio": 1.1342394623193197, "valid_fraction": 1.0, "ladder": [ { "n": 2, "base_ir": 28664.53, "head_ir": 30109.07, "input_chars": 13.0, "ratio": 1.0503946863946487, "valid_fraction": 1.0, "judged": true }, { "n": 4, "base_ir": 32246.525, "head_ir": 34531.065, "input_chars": 21.0, "ratio": 1.0708460834151898, "valid_fraction": 1.0, "judged": true }, { "n": 16, "base_ir": 54563.24, "head_ir": 61887.78, "input_chars": 69.0, "ratio": 1.1342394623193197, "valid_fraction": 1.0, "judged": true }, { "n": 64, "base_ir": 144361.6447368421, "head_ir": 171841.6447368421, "input_chars": 261.0, "ratio": 1.1903552709592184, "valid_fraction": 1.0, "judged": true }, { "n": 256, "base_ir": 507065.4736842105, "head_ir": 614837.4736842106, "input_chars": 1029.0, "ratio": 1.2125405999681969, "valid_fraction": 1.0, "judged": true }, { "n": 1024, "base_ir": 1929181.25, "head_ir": 2362141.25, "input_chars": 4101.0, "ratio": 1.2244268131882372, "valid_fraction": 1.0, "judged": true }, { "n": 4096, "base_ir": 7616471.0, "head_ir": 9328860.5, "input_chars": 16389.0, "ratio": 1.2248271542030422, "valid_fraction": 1.0, "judged": true } ], "slope_base": 0.7930554762442117, "slope_head": 0.8153531602469326, "top_inputs": [ { "args": [ "999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999c5" ], "cpu_ratio": 3.228, "base_ir": 59902.965, "head_ir": 68487.505, "ir_ratio": 1.143307430608819, "outcome": "ok" }, { "args": [ "999999999999999999999999999999〺𐅊〇𐅆𒑦𐅆𐅠ⅱⅫ" ], "cpu_ratio": 2.64, "base_ir": 138417.36144578314, "head_ir": 143721.36144578314, "ir_ratio": 1.0383188925478652, "outcome": "exc:packaging.version.InvalidVersion" } ], "error": null, "verdict": "slower", "worst_ratio": 1.2248271542030422, "worst_search": { "args": [ "999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999c5" ], "cpu_ratio": 3.228, "base_ir": 59902.965, "head_ir": 68487.505, "ir_ratio": 1.143307430608819, "outcome": "ok" } }, { "family_id": "R4", "kind": "suspected_slowdown", "family": "Versions with many release segments and a pre‑release suffix (e.g. \"1.2.3…a1\"). The fast‑path scans the whole string looking for indicator characters, finds the “a”, then falls back to the regex parser – an extra O(n) pass over the input.", "why": "The fast‑path’s `isdisjoint` test iterates over every character. For a long dotted release the string length grows with n, so the extra scan adds linear overhead before the (already linear) regex match.", "origin": "archive", "screen_ratio": 1.0943317782410533, "valid_fraction": 1.0, "ladder": [ { "n": 2, "base_ir": 26957.95, "head_ir": 27974.49, "input_chars": 9.0, "ratio": 1.037708356903993, "valid_fraction": 1.0, "judged": true }, { "n": 4, "base_ir": 29923.86, "head_ir": 31365.4, "input_chars": 13.0, "ratio": 1.0481735979248668, "valid_fraction": 1.0, "judged": true }, { "n": 16, "base_ir": 50222.1, "head_ir": 54959.64, "input_chars": 44.0, "ratio": 1.0943317782410533, "valid_fraction": 1.0, "judged": true }, { "n": 64, "base_ir": 137100.39047619049, "head_ir": 157255.39047619049, "input_chars": 188.0, "ratio": 1.1470090634315167, "valid_fraction": 1.0, "judged": true }, { "n": 256, "base_ir": 495344.95238095237, "head_ir": 593383.9523809524, "input_chars": 921.0, "ratio": 1.1979206601960117, "valid_fraction": 1.0, "judged": true }, { "n": 1024, "base_ir": 1921832.25, "head_ir": 2351189.25, "input_chars": 4018.0, "ratio": 1.2234102378082166, "valid_fraction": 1.0, "judged": true }, { "n": 4096, "base_ir": 7763516.0, "head_ir": 9826604.0, "input_chars": 19378.0, "ratio": 1.2657414501367679, "valid_fraction": 1.0, "judged": true } ], "slope_base": 0.7447047654153816, "slope_head": 0.7709754509572503, "top_inputs": [ { "args": [ "²" ], "cpu_ratio": 2.772, "base_ir": 8540.48, "head_ir": 9292.48, "ir_ratio": 1.088051257072202, "outcome": "exc:packaging.version.InvalidVersion" }, { "args": [ "³" ], "cpu_ratio": 1.478, "base_ir": 8540.48, "head_ir": 9339.48, "ir_ratio": 1.0935544606392147, "outcome": "exc:packaging.version.InvalidVersion" } ], "error": null, "verdict": "slower", "worst_ratio": 1.2657414501367679, "worst_search": { "args": [ "³" ], "cpu_ratio": 1.478, "base_ir": 8540.48, "head_ir": 9339.48, "ir_ratio": 1.0935544606392147, "outcome": "exc:packaging.version.InvalidVersion" } }, { "family_id": "R1", "kind": "suspected_slowdown", "family": "Invalid version strings that nevertheless contain indicator characters (e.g. a trailing dot followed by a letter). The fast‑path still does the full scan before the regex fails.", "why": "Even though the version is invalid, the extra disjoint check runs; the regex then fails. This is the single allowed invalid family.", "origin": "archive", "screen_ratio": 1.2096069779500223, "valid_fraction": 0.0, "ladder": [ { "n": 2, "base_ir": 13879.48, "head_ir": 15030.48, "input_chars": 9.0, "ratio": 1.0829281788654905, "valid_fraction": 0.0, "judged": true }, { "n": 4, "base_ir": 14683.48, "head_ir": 16254.48, "input_chars": 13.0, "ratio": 1.1069909857881102, "valid_fraction": 0.0, "judged": true }, { "n": 16, "base_ir": 19517.48, "head_ir": 23608.48, "input_chars": 37.0, "ratio": 1.2096069779500223, "valid_fraction": 0.0, "judged": true }, { "n": 64, "base_ir": 38807.51351351351, "head_ir": 52978.51351351351, "input_chars": 133.0, "ratio": 1.3651612462897265, "valid_fraction": 0.0, "judged": true }, { "n": 256, "base_ir": 116506.78947368421, "head_ir": 171022.7894736842, "input_chars": 517.0, "ratio": 1.467921227992586, "valid_fraction": 0.0, "judged": true }, { "n": 1024, "base_ir": 427302.44444444444, "head_ir": 643015.4444444445, "input_chars": 2053.0, "ratio": 1.5048251017624261, "valid_fraction": 0.0, "judged": true }, { "n": 4096, "base_ir": 1666988.0, "head_ir": 2527880.0, "input_chars": 8197.0, "ratio": 1.5164356312103027, "valid_fraction": 0.0, "judged": true } ], "slope_base": 0.7036875342071894, "slope_head": 0.757139658229152, "top_inputs": [], "error": null, "verdict": "slower", "worst_ratio": 1.5164356312103027, "worst_search": null }, { "family_id": "F2", "kind": "claimed_speedup", "family": "Version parsing fast path for simple numeric versions", "why": "Added a fast-path frozenset check that allows versions containing only digits and dots to skip the full regex parsing and be parsed directly as a release segment.", "origin": "plan", "screen_ratio": 0.3583642777591902, "valid_fraction": 1.0, "ladder": [ { "n": 2, "base_ir": 8499.62, "head_ir": 8499.62, "input_chars": 17.0, "ratio": 1.0, "valid_fraction": 0.0, "judged": false }, { "n": 4, "base_ir": 23386.265, "head_ir": 8588.715, "input_chars": 9.0, "ratio": 0.36725466849879623, "valid_fraction": 1.0, "judged": true }, { "n": 16, "base_ir": 22730.265, "head_ir": 8145.715, "input_chars": 6.0, "ratio": 0.3583642777591902, "valid_fraction": 1.0, "judged": true }, { "n": 64, "base_ir": 23386.265, "head_ir": 8588.715, "input_chars": 9.0, "ratio": 0.36725466849879623, "valid_fraction": 1.0, "judged": true }, { "n": 256, "base_ir": 17083.108333333334, "head_ir": 7383.111666666667, "input_chars": 7.666666666666667, "ratio": 0.4321878385715324, "valid_fraction": 0.6666666666666666, "judged": true }, { "n": 1024, "base_ir": 19782.724375, "head_ir": 7048.434375, "input_chars": 7.125, "ratio": 0.35629240146050406, "valid_fraction": 0.875, "judged": true }, { "n": 4096, "base_ir": 19809.724375, "head_ir": 7066.684375, "input_chars": 6.75, "ratio": 0.3567280513967272, "valid_fraction": 0.875, "judged": true } ], "slope_base": null, "slope_head": null, "top_inputs": [ { "args": [ "໕𖫉൫᠕߆᮱᭒১߉௧༣໐٦౫୦൬𑜴᧔" ], "cpu_ratio": 2.523, "base_ir": 9561.518987341773, "head_ir": 10695.518987341773, "ir_ratio": 1.1186004024571066, "outcome": "exc:packaging.version.InvalidVersion" }, { "args": [ "៥٤᭓೨١১໑᥌௭᭕" ], "cpu_ratio": 2.389, "base_ir": 9163.48, "head_ir": 10304.48, "ir_ratio": 1.1245160135668981, "outcome": "exc:packaging.version.InvalidVersion" } ], "error": null, "verdict": "faster", "worst_ratio": 1.1245160135668981, "worst_search": { "args": [ "៥٤᭓೨١১໑᥌௭᭕" ], "cpu_ratio": 2.389, "base_ir": 9163.48, "head_ir": 10304.48, "ir_ratio": 1.1245160135668981, "outcome": "exc:packaging.version.InvalidVersion" } } ], "seconds": { "plan": 18.34, "screen": 360.79, "full": 491.79, "total": 870.93 }, "ladder_validity": "backfilled 2026-10-11 (scripts/backtest/backfill_ladder_validity.py)" } } } ] }, { "id": "1082@61967cca", "pr": 1082, "title": "perf: fast path for simple versions (intermediate commit 61967cca)", "base": "59f183fe90da403e41eef339ce9088bda0fe4cd3", "head": "61967cca278dc51389b8973ba65bbbf23eaa3eda", "merged": "2026-02-11", "after_ultra_release": false, "known": [ { "id": "unicode-digits", "match": { "probe": "P2", "traits": "non-ascii" }, "fixed_by": "review", "status": "verified" }, { "id": "inner-whitespace", "match": { "probe": "P2", "traits": "whitespace" }, "fixed_by": "review", "status": "verified" }, { "id": "oversized-int-exception", "match": { "probe": "P2", "traits": "long-digits" }, "fixed_by": 1155, "status": "verified" } ], "targets": [ { "target": "packaging.version:Version", "run_id": "bt-1082-61967cca-c23f4f", "image": "bw-pair:cebb5e33c4500036", "usd": 0.1053, "harvested": 80, "arms": { "hypothesis": { "classes": [], "known": [], "seconds": 8.6, "error": null }, "mutations": { "classes": [ "P2|outcome|exc:builtins.ValueError|exc:packaging.version.InvalidVersion|long-digits", "P2|outcome|exc:packaging.version.InvalidVersion|ok|whitespace" ], "known": [ "inner-whitespace", "oversized-int-exception" ], "seconds": 3.9, "error": null }, "breakwater": { "classes": [ "P2|outcome|exc:builtins.TypeError|exc:builtins.AttributeError|nested", "P2|outcome|exc:builtins.ValueError|exc:packaging.version.InvalidVersion|long-digits", "P2|outcome|exc:builtins.ValueError|exc:packaging.version.InvalidVersion|long-digits+whitespace", "P2|outcome|exc:packaging.version.InvalidVersion|ok|long-digits+whitespace", "P2|outcome|exc:packaging.version.InvalidVersion|ok|non-ascii", "P2|outcome|exc:packaging.version.InvalidVersion|ok|non-ascii+whitespace", "P2|outcome|exc:packaging.version.InvalidVersion|ok|whitespace" ], "known": [ "unicode-digits", "inner-whitespace", "oversized-int-exception" ], "seconds": 100.0, "error": null }, "llm-plan": { "classes": [ "P2|outcome|exc:packaging.version.InvalidVersion|ok|non-ascii", "P2|outcome|exc:packaging.version.InvalidVersion|ok|whitespace" ], "known": [ "unicode-digits", "inner-whitespace" ], "seconds": 13.9, "error": null } }, "findings": [ { "signature": "P2|outcome|exc:packaging.version.InvalidVersion|ok|non-ascii", "probe": "P2", "detail": "outcome", "args": [ [ "١" ] ], "base": { "status": "exc", "exc": "packaging.version.InvalidVersion" }, "head": { "status": "ok", "obs": { "str": "1", "repr": "", "attr:release": [ 1 ], "attr:public": "1" } }, "found_by": [ "A1/s4", "seed", "mutation:unicode-digits", "mutation:separators", "mutation:edges" ] }, { "signature": "P2|outcome|exc:builtins.TypeError|exc:builtins.AttributeError|nested", "probe": "P2", "detail": "outcome", "args": [ [ [ "0.0" ] ] ], "base": { "status": "exc", "exc": "builtins.TypeError" }, "head": { "status": "exc", "exc": "builtins.AttributeError" }, "found_by": [ "A4/s1" ] }, { "signature": "P2|outcome|exc:packaging.version.InvalidVersion|ok|whitespace", "probe": "P2", "detail": "outcome", "args": [ [ "0. 0" ] ], "base": { "status": "exc", "exc": "packaging.version.InvalidVersion" }, "head": { "status": "ok", "obs": { "str": "0.0", "repr": "", "attr:release": [ 0, 0 ], "attr:public": "0.0" } }, "found_by": [ "A4/s3", "seed", "mutation:equivalents", "mutation:whitespace-inner" ] }, { "signature": "P2|outcome|exc:packaging.version.InvalidVersion|ok|non-ascii+whitespace", "probe": "P2", "detail": "outcome", "args": [ [ "١ .٢.٣" ] ], "base": { "status": "exc", "exc": "packaging.version.InvalidVersion" }, "head": { "status": "ok", "obs": { "str": "1.2.3", "repr": "", "attr:release": [ 1, 2, 3 ], "attr:public": "1.2.3" } }, "found_by": [ "mutation:whitespace-outer", "mutation:whitespace-inner" ] }, { "signature": "P2|outcome|exc:builtins.ValueError|exc:packaging.version.InvalidVersion|long-digits", "probe": "P2", "detail": "outcome", "args": [ [ "10000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000" ] ], "base": { "status": "exc", "exc": "builtins.ValueError" }, "head": { "status": "exc", "exc": "packaging.version.InvalidVersion" }, "found_by": [ "mutation:huge-number" ] }, { "signature": "P2|outcome|exc:builtins.ValueError|exc:packaging.version.InvalidVersion|long-digits+whitespace", "probe": "P2", "detail": "outcome", "args": [ [ " 10000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000.2.3 " ] ], "base": { "status": "exc", "exc": "builtins.ValueError" }, "head": { "status": "exc", "exc": "packaging.version.InvalidVersion" }, "found_by": [ "mutation:huge-number" ] }, { "signature": "P2|outcome|exc:packaging.version.InvalidVersion|ok|long-digits+whitespace", "probe": "P2", "detail": "outcome", "args": [ [ "1000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000 .2.3" ] ], "base": { "status": "exc", "exc": "packaging.version.InvalidVersion" }, "head": { "status": "ok", "obs": { "str": "1000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000.2.3", "repr": "", "attr:release": [ "", 2, 3 ], "attr:public": "1000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000.2.3" } }, "found_by": [ "mutation:huge-number" ] } ], "perf": { "slower_families": [ { "id": "R2", "family": "Single repeated segment \"999.\" n times plus prerelease c5", "verdict": "slower", "ladder": [ 1.050931935740792, 1.0723160092444068, 1.1386270316792038, 1.1972615375220401, 1.2213402525408326, 1.231998548607084, 1.23420242786981 ], "origin": "archive" }, { "id": "R3", "family": "Versions with many release segments and a pre‑release suffix (e.g. \"1.2.3…a1\"). The fast‑path scans the whole string looking for indicator characters, finds the “a”, then falls back to the regex parser – an extra O(n) pass over the input.", "verdict": "slower", "ladder": [ 1.0383167117677716, 1.0442365106675613, 1.0990125860925768, 1.150437208299431, 1.2047744698163205, 1.2249806142029305, 1.2691883419831942 ], "origin": "archive" }, { "id": "R4", "family": "Many numeric release segments followed by 'b' pre-release", "verdict": "slower", "ladder": [ 1.0379144727545013, 1.0489224379060347, 1.0955280440167023, 1.1501240401506463, 1.204656306295487, 1.2249471383979629, 1.269178425724922 ], "origin": "archive" }, { "id": "F4", "family": "versions with an epoch (e.g. \"1!2.3.4\")", "verdict": "slower", "ladder": [ 1.0346009806265226, 1.0367407845838619, 1.0888539269536346, 1.2544511186957685, 1.6892642349756108, 1.912623952412034, 1.4628707704180715 ], "origin": "plan" } ], "worst_valid_ratio": 1.912623952412034, "worst_family": "versions with an epoch (e.g. \"1!2.3.4\")", "speedup_ratio_small": null, "report": { "run_id": "bt-1082-61967cca-c23f4f", "target": "packaging.version:Version", "image": "bw-pair:cebb5e33c4500036", "backend": "docker", "families": [ { "id": "F1", "kind": "suspected_slowdown", "family": "versions with a pre‑release segment (e.g. \"1.2.3a1\")", "why": "The new code first scans the whole string with `_NON_SIMPLE_VERSION_INDICATORS.isdisjoint`. Because the string contains letters such as \"a\", \"b\", \"c\", \"r\" the fast‑path is not taken, so the extra set‑membership scan is added on top of the original regex match.", "size_meaning": "n = number of dot‑separated release components before the pre‑release marker; larger n ⇒ longer string and more work for both the indicator scan and the regex.", "make_source": "def make(n):\n # build a version like 1.1.1....1.0a1 with n release components\n # release part: n copies of \"1.\" then a final \"0\"\n release = \"1.\" * n + \"0\"\n version = f\"{release}a1\"\n return [[version]]", "example_args": [], "origin": "plan", "screen_ratio": 1.0850250920750806, "valid_fraction": 1.0, "error": null }, { "id": "F2", "kind": "suspected_slowdown", "family": "versions with a post‑release segment (e.g. \"1.2.3.post1\")", "why": "Same reasoning as pre‑release: the string contains letters from the indicator set (p, o, s, t, r, e, v) so the fast path is bypassed, incurring the extra `isdisjoint` scan.", "size_meaning": "n = number of dot‑separated release components before the \".post\" marker.", "make_source": "def make(n):\n release = \"1.\" * n + \"0\"\n version = f\"{release}.post1\"\n return [[version]]", "example_args": [], "origin": "plan", "screen_ratio": 1.074529492936471, "valid_fraction": 1.0, "error": null }, { "id": "F3", "kind": "suspected_slowdown", "family": "versions with a development release segment (e.g. \"1.2.3.dev1\")", "why": "Contains the letters d, e, v → indicator set not disjoint → fast path skipped, extra scan added.", "size_meaning": "n = number of dot‑separated release components before the \".dev\" marker.", "make_source": "def make(n):\n release = \"1.\" * n + \"0\"\n version = f\"{release}.dev1\"\n return [[version]]", "example_args": [], "origin": "plan", "screen_ratio": 1.083499501161211, "valid_fraction": 1.0, "error": null }, { "id": "F4", "kind": "suspected_slowdown", "family": "versions with an epoch (e.g. \"1!2.3.4\")", "why": "The exclamation mark \"!\" is in the indicator set, so the fast path is not taken. The epoch number can be made arbitrarily long, increasing the cost of the indicator scan.", "size_meaning": "n = number of digits in the epoch number; larger n ⇒ longer string before the \"!\" marker.", "make_source": "def make(n):\n epoch = \"9\" * n # n‑digit epoch\n version = f\"{epoch}!1.0\"\n return [[version]]", "example_args": [], "origin": "plan", "screen_ratio": 1.0888539269536346, "valid_fraction": 1.0, "error": null }, { "id": "F5", "kind": "suspected_slowdown", "family": "versions with a local version segment (e.g. \"1.2.3+abc.def\")", "why": "The plus sign \"+\" is part of the indicator set, so the fast path is bypassed. Making the local part longer (more dot‑separated segments) grows the work done by the indicator scan and the subsequent regex.", "size_meaning": "n = number of dot‑separated alphanumeric segments in the local part.", "make_source": "def make(n):\n # local part: n segments like \"a\", \"b\", \"c\" … separated by '.'\n # we cycle through a small alphabet to stay deterministic\n alphabet = \"abcdefghijklmnopqrstuvwxyz\"\n local = \".\".join(alphabet[i % len(alphabet)] for i in range(n))\n version = f\"1.0+{local}\"\n return [[version]]", "example_args": [], "origin": "plan", "screen_ratio": 1.0146868279187342, "valid_fraction": 1.0, "error": null }, { "id": "F6", "kind": "suspected_slowdown", "family": "invalid strings that look simple (only digits and dots) but contain a non‑numeric segment, e.g. \"1.2.x\"", "why": "The indicator set is disjoint, so the fast path is entered. The code then executes `tuple(map(int, version.strip().split('.')))`, which raises `ValueError` on the bad segment; the exception is caught and re‑raised as `InvalidVersion`. Exception handling is considerably slower than the original regex‑only failure path.", "size_meaning": "n = number of leading numeric segments before the bad segment; larger n makes the split and the failed `int` conversion traverse more items before the exception.", "make_source": "def make(n):\n # build \"1.1.1....1.x\" with n numeric parts then a non‑numeric \"x\"\n good = \".\".join(\"1\" for _ in range(n))\n version = f\"{good}.x\"\n return [[version]]", "example_args": [], "origin": "plan", "screen_ratio": 2.090229886235313, "valid_fraction": 0.0, "error": null }, { "id": "R1", "kind": "suspected_slowdown", "family": "Invalid version strings that nevertheless contain indicator characters (e.g. a trailing dot followed by a letter). The fast‑path still does the full scan before the regex fails.", "why": "Even though the version is invalid, the extra disjoint check runs; the regex then fails. This is the single allowed invalid family.", "size_meaning": "Number of dotted segments before the malformed tail.", "make_source": "def make(n):\n # \"1.\" repeated n times then an alphabetic tail – not a valid PEP‑440 version\n version = \"1.\" * n + \"x\"\n return [(version,)]", "example_args": [], "origin": "archive", "screen_ratio": 2.090229886235313, "valid_fraction": 0.0, "error": null }, { "id": "R2", "kind": "suspected_slowdown", "family": "Single repeated segment \"999.\" n times plus prerelease c5", "why": "The fast‑path’s `isdisjoint` test iterates over every character. For a long dotted release the string length grows with n, so the extra scan adds linear overhead before the (already linear) regex match.", "size_meaning": "Number of dotted release segments (plus the trailing pre‑release).", "make_source": "def make(n):\n release = \".\".join([\"999\"] * n)\n version = f\"{release}c5\"\n return [(version,)]", "example_args": [], "origin": "archive", "screen_ratio": 1.1386270316792038, "valid_fraction": 1.0, "error": null }, { "id": "R3", "kind": "suspected_slowdown", "family": "Versions with many release segments and a pre‑release suffix (e.g. \"1.2.3…a1\"). The fast‑path scans the whole string looking for indicator characters, finds the “a”, then falls back to the regex parser – an extra O(n) pass over the input.", "why": "The fast‑path’s `isdisjoint` test iterates over every character. For a long dotted release the string length grows with n, so the extra scan adds linear overhead before the (already linear) regex match.", "size_meaning": "Number of dotted release segments (plus the trailing pre‑release).", "make_source": "def make(n):\n # Build \"1.2.3…n\" then add a pre‑release marker \"a1\"\n release = \".\".join(str(i) for i in range(1, n + 1))\n version = f\"{release}a1\"\n return [(version,)]", "example_args": [], "origin": "archive", "screen_ratio": 1.0960720479629487, "valid_fraction": 1.0, "error": null }, { "id": "R4", "kind": "suspected_slowdown", "family": "Many numeric release segments followed by 'b' pre-release", "why": "The fast‑path’s `isdisjoint` test iterates over every character. For a long dotted release the string length grows with n, so the extra scan adds linear overhead before the (already linear) regex match.", "size_meaning": "Number of dotted release segments (plus the trailing pre‑release).", "make_source": "def make(n):\n release = \".\".join(str(i) for i in range(1, n + 1))\n version = f\"{release}b1\"\n return [(version,)]", "example_args": [], "origin": "archive", "screen_ratio": 1.0955280440167023, "valid_fraction": 1.0, "error": null }, { "id": "R2.1", "kind": "suspected_slowdown", "family": "Single repeated segment '9' followed by '.' n times plus prerelease c5", "why": "The fast‑path’s `isdisjoint` test iterates over every character. For a long dotted release the string length grows with n, so the extra scan adds linear overhead before the (already linear) regex match.", "size_meaning": "Number of dotted release segments (plus the trailing pre‑release).", "make_source": "def make(n):\n # Variant replaces each '999' segment with just '9', keeping same pattern\n dots_and_nines = \".\" + \"9\"\n release = (\"\".join([dots_and_nines] * n)).lstrip('.')\n version = f\"{release}c5\"\n return [(version,)]}, {", "example_args": [], "origin": "expand:R2", "screen_ratio": null, "valid_fraction": null, "error": "make(n) rejected: syntax error: unmatched '}' (line 7)" }, { "id": "R3.1", "kind": "suspected_slowdown", "family": "Versions with hyphen-prefixed local labels", "why": "The fast‑path’s `isdisjoint` test iterates over every character. For a long dotted release the string length grows with n, so the extra scan adds linear overhead before the (already linear) regex match.", "size_meaning": "Number of dotted release segments (plus the trailing pre‑release).", "make_source": "", "example_args": [], "origin": "expand:R3", "screen_ratio": null, "valid_fraction": null, "error": "make(n) rejected: code must assign `strategy`" }, { "id": "R3.2", "kind": "suspected_slowdown", "family": "Versions without patch but repeated major/minor pattern plus alpha tag", "why": "The fast‑path’s `isdisjoint` test iterates over every character. For a long dotted release the string length grows with n, so the extra scan adds linear overhead before the (already linear) regex match.", "size_meaning": "Number of dotted release segments (plus the trailing pre‑release).", "make_source": "", "example_args": [], "origin": "expand:R3", "screen_ratio": null, "valid_fraction": null, "error": "make(n) rejected: code must assign `strategy`" }, { "id": "R3.3", "kind": "suspected_slowdown", "family": "Versions with mix of base-36 post-release digits inside letters forcing full split", "why": "The fast‑path’s `isdisjoint` test iterates over every character. For a long dotted release the string length grows with n, so the extra scan adds linear overhead before the (already linear) regex match.", "size_meaning": "Number of dotted release segments (plus the trailing pre‑release).", "make_source": "", "example_args": [], "origin": "expand:R3", "screen_ratio": null, "valid_fraction": null, "error": "make(n) rejected: code must assign `strategy`" } ], "findings": [ { "family_id": "R2", "kind": "suspected_slowdown", "family": "Single repeated segment \"999.\" n times plus prerelease c5", "why": "The fast‑path’s `isdisjoint` test iterates over every character. For a long dotted release the string length grows with n, so the extra scan adds linear overhead before the (already linear) regex match.", "origin": "archive", "screen_ratio": 1.1386270316792038, "valid_fraction": 1.0, "ladder": [ { "n": 2, "base_ir": 28664.53, "head_ir": 30124.47, "input_chars": 13.0, "ratio": 1.050931935740792, "valid_fraction": 1.0, "judged": true }, { "n": 4, "base_ir": 32246.525, "head_ir": 34578.465, "input_chars": 21.0, "ratio": 1.0723160092444068, "valid_fraction": 1.0, "judged": true }, { "n": 16, "base_ir": 54563.24, "head_ir": 62127.18, "input_chars": 69.0, "ratio": 1.1386270316792038, "valid_fraction": 1.0, "judged": true }, { "n": 64, "base_ir": 144361.6447368421, "head_ir": 172838.6447368421, "input_chars": 261.0, "ratio": 1.1972615375220401, "valid_fraction": 1.0, "judged": true }, { "n": 256, "base_ir": 507065.4736842105, "head_ir": 619299.4736842106, "input_chars": 1029.0, "ratio": 1.2213402525408326, "valid_fraction": 1.0, "judged": true }, { "n": 1024, "base_ir": 1929181.25, "head_ir": 2376748.5, "input_chars": 4101.0, "ratio": 1.231998548607084, "valid_fraction": 1.0, "judged": true }, { "n": 4096, "base_ir": 7616471.0, "head_ir": 9400267.0, "input_chars": 16389.0, "ratio": 1.23420242786981, "valid_fraction": 1.0, "judged": true } ], "slope_base": 0.7930554762442117, "slope_head": 0.8163144147737503, "top_inputs": [ { "args": [ "999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999c5" ], "cpu_ratio": 3.875, "base_ir": 948314.1, "head_ir": 1164831.1, "ir_ratio": 1.228317811577409, "outcome": "ok" }, { "args": [ "999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999c5" ], "cpu_ratio": 3.565, "base_ir": 1572496.2, "head_ir": 1938245.2, "ir_ratio": 1.2325913410792344, "outcome": "ok" } ], "error": null, "verdict": "slower", "worst_ratio": 1.23420242786981, "worst_search": { "args": [ "999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999.999c5" ], "cpu_ratio": 3.565, "base_ir": 1572496.2, "head_ir": 1938245.2, "ir_ratio": 1.2325913410792344, "outcome": "ok" } }, { "family_id": "R3", "kind": "suspected_slowdown", "family": "Versions with many release segments and a pre‑release suffix (e.g. \"1.2.3…a1\"). The fast‑path scans the whole string looking for indicator characters, finds the “a”, then falls back to the regex parser – an extra O(n) pass over the input.", "why": "The fast‑path’s `isdisjoint` test iterates over every character. For a long dotted release the string length grows with n, so the extra scan adds linear overhead before the (already linear) regex match.", "origin": "archive", "screen_ratio": 1.0960720479629487, "valid_fraction": 1.0, "ladder": [ { "n": 2, "base_ir": 26957.95, "head_ir": 27990.89, "input_chars": 9.0, "ratio": 1.0383167117677716, "valid_fraction": 1.0, "judged": true }, { "n": 4, "base_ir": 30071.54, "head_ir": 31401.8, "input_chars": 13.0, "ratio": 1.0442365106675613, "valid_fraction": 1.0, "judged": true }, { "n": 16, "base_ir": 50222.1, "head_ir": 55194.72, "input_chars": 44.0, "ratio": 1.0990125860925768, "valid_fraction": 1.0, "judged": true }, { "n": 64, "base_ir": 137100.39047619049, "head_ir": 157725.39047619049, "input_chars": 188.0, "ratio": 1.150437208299431, "valid_fraction": 1.0, "judged": true }, { "n": 256, "base_ir": 495344.95238095237, "head_ir": 596778.9523809524, "input_chars": 921.0, "ratio": 1.2047744698163205, "valid_fraction": 1.0, "judged": true }, { "n": 1024, "base_ir": 1921832.25, "head_ir": 2354207.25, "input_chars": 4018.0, "ratio": 1.2249806142029305, "valid_fraction": 1.0, "judged": true }, { "n": 4096, "base_ir": 7763516.0, "head_ir": 9853364.0, "input_chars": 19378.0, "ratio": 1.2691883419831942, "valid_fraction": 1.0, "judged": true } ], "slope_base": 0.7444188747286032, "slope_head": 0.7711481454161977, "top_inputs": [ { "args": [ "('1600',)a1" ], "cpu_ratio": 3.55, "base_ir": 8839.48, "head_ir": 10475.48, "ir_ratio": 1.185078760289067, "outcome": "exc:packaging.version.InvalidVersion" }, { "args": [ "('98139110270',)a1" ], "cpu_ratio": 2.915, "base_ir": 9130.48, "head_ir": 11510.48, "ir_ratio": 1.2606653757524249, "outcome": "exc:packaging.version.InvalidVersion" } ], "error": null, "verdict": "slower", "worst_ratio": 1.2691883419831942, "worst_search": { "args": [ "('98139110270',)a1" ], "cpu_ratio": 2.915, "base_ir": 9130.48, "head_ir": 11510.48, "ir_ratio": 1.2606653757524249, "outcome": "exc:packaging.version.InvalidVersion" } }, { "family_id": "R4", "kind": "suspected_slowdown", "family": "Many numeric release segments followed by 'b' pre-release", "why": "The fast‑path’s `isdisjoint` test iterates over every character. For a long dotted release the string length grows with n, so the extra scan adds linear overhead before the (already linear) regex match.", "origin": "archive", "screen_ratio": 1.0955280440167023, "valid_fraction": 1.0, "ladder": [ { "n": 2, "base_ir": 27243.95, "head_ir": 28276.89, "input_chars": 9.0, "ratio": 1.0379144727545013, "valid_fraction": 1.0, "judged": true }, { "n": 4, "base_ir": 30209.86, "head_ir": 31687.8, "input_chars": 13.0, "ratio": 1.0489224379060347, "valid_fraction": 1.0, "judged": true }, { "n": 16, "base_ir": 50508.1, "head_ir": 55333.04, "input_chars": 44.0, "ratio": 1.0955280440167023, "valid_fraction": 1.0, "judged": true }, { "n": 64, "base_ir": 137386.39047619049, "head_ir": 158011.39047619049, "input_chars": 188.0, "ratio": 1.1501240401506463, "valid_fraction": 1.0, "judged": true }, { "n": 256, "base_ir": 495630.95238095237, "head_ir": 597064.9523809524, "input_chars": 921.0, "ratio": 1.204656306295487, "valid_fraction": 1.0, "judged": true }, { "n": 1024, "base_ir": 1922118.25, "head_ir": 2354493.25, "input_chars": 4018.0, "ratio": 1.2249471383979629, "valid_fraction": 1.0, "judged": true }, { "n": 4096, "base_ir": 7763802.0, "head_ir": 9853650.0, "input_chars": 19378.0, "ratio": 1.269178425724922, "valid_fraction": 1.0, "judged": true } ], "slope_base": 0.7432824377709842, "slope_head": 0.7698828330067451, "top_inputs": [ { "args": [ "1.2.3.4.5.6.7.8.9.10b1" ], "cpu_ratio": 3.173, "base_ir": 38119.125, "head_ir": 40994.125, "ir_ratio": 1.0754214583886696, "outcome": "ok" }, { "args": [ "1.2.3.4.5.6.7.8.9.10.11.12.13b1" ], "cpu_ratio": 1.382, "base_ir": 45196.77, "head_ir": 49061.71, "ir_ratio": 1.085513632943239, "outcome": "ok" } ], "error": null, "verdict": "slower", "worst_ratio": 1.269178425724922, "worst_search": { "args": [ "1.2.3.4.5.6.7.8.9.10.11.12.13b1" ], "cpu_ratio": 1.382, "base_ir": 45196.77, "head_ir": 49061.71, "ir_ratio": 1.085513632943239, "outcome": "ok" } }, { "family_id": "F4", "kind": "suspected_slowdown", "family": "versions with an epoch (e.g. \"1!2.3.4\")", "why": "The exclamation mark \"!\" is in the indicator set, so the fast path is not taken. The epoch number can be made arbitrarily long, increasing the cost of the indicator scan.", "origin": "plan", "screen_ratio": 1.0888539269536346, "valid_fraction": 1.0, "ladder": [ { "n": 2, "base_ir": 26500.405, "head_ir": 27417.345, "input_chars": 10.0, "ratio": 1.0346009806265226, "valid_fraction": 1.0, "judged": true }, { "n": 4, "base_ir": 26762.085, "head_ir": 27745.345, "input_chars": 12.0, "ratio": 1.0367407845838619, "valid_fraction": 1.0, "judged": true }, { "n": 16, "base_ir": 27178.765, "head_ir": 29593.705, "input_chars": 24.0, "ratio": 1.0888539269536346, "valid_fraction": 1.0, "judged": true }, { "n": 64, "base_ir": 29675.405, "head_ir": 37226.345, "input_chars": 72.0, "ratio": 1.2544511186957685, "valid_fraction": 1.0, "judged": true }, { "n": 256, "base_ir": 40739.09333333333, "head_ir": 68819.09333333334, "input_chars": 264.0, "ratio": 1.6892642349756108, "valid_fraction": 1.0, "judged": true }, { "n": 1024, "base_ir": 120791.26315789473, "head_ir": 231028.26315789475, "input_chars": 1032.0, "ratio": 1.912623952412034, "valid_fraction": 1.0, "judged": true }, { "n": 4096, "base_ir": 948498.0, "head_ir": 1387530.0, "input_chars": 4104.0, "ratio": 1.4628707704180715, "valid_fraction": 1.0, "judged": true } ], "slope_base": 0.5142828585351725, "slope_head": 0.6033370351034816, "top_inputs": [], "error": null, "verdict": "slower", "worst_ratio": 1.912623952412034, "worst_search": null } ], "seconds": { "plan": 16.62, "screen": 153.61, "full": 279.19, "total": 449.43 }, "ladder_validity": "backfilled 2026-10-11 (scripts/backtest/backfill_ladder_validity.py)" } } } ] }, { "id": "#1083", "pr": 1083, "title": "perf: Add fast path for `Version` to `Version` comparison by skipping `_key` property", "base": "cd1520c475213d45df1cfe0e7b9d27e3166ef660", "head": "659955c8a0f6025b4dd27dd0a4bce7e640b40bfe", "merged": "2026-03-06", "after_ultra_release": false, "known": [], "targets": [ { "target": "packaging.version:Version", "run_id": "bt-1083-25b0ce", "image": "bw-pair:e2821dbc0500312f", "usd": 0.0859, "harvested": 80, "arms": { "hypothesis": { "classes": [], "known": [], "seconds": 13.9, "error": null }, "mutations": { "classes": [], "known": [], "seconds": 8.7, "error": null }, "breakwater": { "classes": [], "known": [], "seconds": 49.8, "error": null }, "llm-plan": { "classes": [], "known": [], "seconds": 2.9, "error": null } }, "findings": [], "perf": { "slower_families": [], "worst_valid_ratio": null, "worst_family": null, "speedup_ratio_small": null, "report": { "run_id": "bt-1083-25b0ce", "target": "packaging.version:Version", "image": "bw-pair:e2821dbc0500312f", "backend": "docker", "families": [ { "id": "F1", "kind": "suspected_slowdown", "family": "simple_release", "why": "Very short release tuples (1‑element) make tuple comparison extremely cheap, so the extra isinstance checks added in the new comparison overrides dominate the runtime. The old _BaseVersion.__lt__ performed a single isinstance check and a property access; the new version does two isinstance checks and explicit cache initialisation on every comparison.", "size_meaning": "Number of dot‑separated numeric segments in the release part (n). Cost grows with n because tuple comparison is O(n), but for n=1 the constant‑time overhead dominates.", "make_source": "\ndef make(n):\n # Build a simple release version like '1.2.3' with n segments\n seg = '.'.join('1' for _ in range(n))\n return [[seg]]", "example_args": [ [ "1" ], [ "1.1" ], [ "1.1.1" ] ], "origin": "plan", "screen_ratio": 1.0006849981383448, "valid_fraction": 1.0, "error": null }, { "id": "F2", "kind": "suspected_slowdown", "family": "unique_pairwise_comparisons", "why": "The new version lazily builds the comparison key (_cmpkey) on the first comparison of each Version object. When many distinct Version objects are each compared only once (e.g., pairwise adjacent comparisons of a long list of unique versions), the new code pays the _cmpkey construction cost twice per comparison (once for each operand) while the original version may have cached the key earlier (e.g., in __init__ or via a property). This extra work makes the new version slower for workloads that create many unique versions and compare each only once.", "size_meaning": "n = number of distinct version strings; the benchmark will create n Version objects and perform n‑1 pairwise comparisons, each operand seeing its first comparison.", "make_source": "\ndef make(n):\n # Produce n distinct simple version strings like '1.0.0', '1.0.1', …\n # Each element of the returned list is a single‑argument list for Version(...).\n return [[f\"1.0.{i}\"] for i in range(n)]", "example_args": [], "origin": "plan", "screen_ratio": 1.0008352089417163, "valid_fraction": 1.0, "error": null }, { "id": "F3", "kind": "suspected_slowdown", "family": "comparison with non-Version objects", "why": "The new __lt__/__eq__ implementations add an extra isinstance(other, Version) check before falling back to the base class. When a Version instance is compared to a non‑Version object (e.g. int, str, None) the head version performs two isinstance checks instead of one, making the comparison slower.", "size_meaning": "number of dot‑separated release segments in the version string; larger n makes the version string longer but the slowdown is dominated by the extra type check, so cost grows only modestly with n", "make_source": "\ndef make(n):\n # Generate a simple dotted version like \"1.2.3...\" with n segments.\n segs = '.'.join(str(i) for i in range(1, n + 1))\n return [[segs]]", "example_args": [ [ [ "1" ] ], [ [ "1.2.3.4.5" ] ] ], "origin": "plan", "screen_ratio": 1.0006314165794672, "valid_fraction": 1.0, "error": null }, { "id": "F4", "kind": "suspected_slowdown", "family": "pre_release_long_number", "why": "Pre‑release segment adds a (letter, number) tuple to the comparison key. Making the numeric part large (many digits) does not change tuple length but makes integer comparison a bit slower; the constant overhead of the new double‑isinstance check still dominates for small tuples.", "size_meaning": "Number of digits in the pre‑release number (n).", "make_source": "def make(n):\n # version like '1.0a' followed by n‑digit number\n num = '9' * n\n return [[f'1.0a{num}']]", "example_args": [ [ "1.0a9" ], [ "1.0a99" ], [ "1.0a999" ] ], "origin": "plan", "screen_ratio": 1.0010478289802174, "valid_fraction": 1.0, "error": null }, { "id": "F5", "kind": "suspected_slowdown", "family": "cross_type_comparisons", "why": "The new comparison operators first test `isinstance(other, Version)` and then fall back to `super().__lt__(other)`, which repeats an `isinstance(other, _BaseVersion)` check. The old code performed only the single `_BaseVersion` check. When a workload repeatedly compares Version objects with instances of a different `_BaseVersion` subclass (a realistic scenario in downstream libraries that subclass `_BaseVersion`), the extra type check adds measurable overhead per comparison.", "size_meaning": "n = number of repeated comparisons between a Version instance and a non‑Version `_BaseVersion` subclass instance.", "make_source": "\ndef make(n):\n # Produce a single simple version string; the benchmark will create one Version\n # and one dummy `_BaseVersion` subclass instance and compare them n times.\n # The argument list contains the version string for the Version constructor.\n return [[\"1.0.0\"] for _ in range(n)]", "example_args": [], "origin": "plan", "screen_ratio": 1.0008472321621138, "valid_fraction": 1.0, "error": null }, { "id": "F6", "kind": "suspected_slowdown", "family": "comparison with non-Version objects (complex local part)", "why": "Same extra isinstance overhead as the simple family, but the version strings contain a large local‑version component (many dot/underscore separated segments). The head version still pays the extra type check while also paying the (identical) key‑construction cost, so the relative slowdown is more visible for larger n.", "size_meaning": "number of segments in the local version part; each segment adds a dot/underscore separated token, increasing key construction work", "make_source": "\ndef make(n):\n # Build a version with a large local part: \"1.0+local.seg1.seg2....segN\"\n local_segments = '.'.join(f'seg{i}' for i in range(1, n + 1))\n version_str = f'1.0+{local_segments}'\n return [[version_str]]", "example_args": [ [ [ "1.0+seg1" ] ], [ [ "1.0+seg1.seg2.seg3.seg4.seg5" ] ] ], "origin": "plan", "screen_ratio": 1.0023005515078378, "valid_fraction": 1.0, "error": null }, { "id": "R1", "kind": "suspected_slowdown", "family": "Invalid version strings that nevertheless contain indicator characters (e.g. a trailing dot followed by a letter). The fast‑path still does the full scan before the regex fails.", "why": "Even though the version is invalid, the extra disjoint check runs; the regex then fails. This is the single allowed invalid family.", "size_meaning": "Number of dotted segments before the malformed tail.", "make_source": "def make(n):\n # \"1.\" repeated n times then an alphabetic tail – not a valid PEP‑440 version\n version = \"1.\" * n + \"x\"\n return [(version,)]", "example_args": [], "origin": "archive", "screen_ratio": 1.000465461094223, "valid_fraction": 0.0, "error": null }, { "id": "R2", "kind": "suspected_slowdown", "family": "invalid strings that look simple (only digits and dots) but contain a non‑numeric segment, e.g. \"1.2.x\"", "why": "The indicator set is disjoint, so the fast path is entered. The code then executes `tuple(map(int, version.strip().split('.')))`, which raises `ValueError` on the bad segment; the exception is caught and re‑raised as `InvalidVersion`. Exception handling is considerably slower than the original regex‑only failure path.", "size_meaning": "n = number of leading numeric segments before the bad segment; larger n makes the split and the failed `int` conversion traverse more items before the exception.", "make_source": "def make(n):\n # build \"1.1.1....1.x\" with n numeric parts then a non‑numeric \"x\"\n good = \".\".join(\"1\" for _ in range(n))\n version = f\"{good}.x\"\n return [[version]]", "example_args": [], "origin": "archive", "screen_ratio": 1.000465461094223, "valid_fraction": 0.0, "error": null }, { "id": "R3", "kind": "suspected_slowdown", "family": "Single repeated segment \"999.\" n times plus prerelease c5", "why": "The fast‑path’s `isdisjoint` test iterates over every character. For a long dotted release the string length grows with n, so the extra scan adds linear overhead before the (already linear) regex match.", "size_meaning": "Number of dotted release segments (plus the trailing pre‑release).", "make_source": "def make(n):\n release = \".\".join([\"999\"] * n)\n version = f\"{release}c5\"\n return [(version,)]", "example_args": [], "origin": "archive", "screen_ratio": 1.00049385053119, "valid_fraction": 1.0, "error": null }, { "id": "R4", "kind": "suspected_slowdown", "family": "Versions with many release segments and a pre‑release suffix (e.g. \"1.2.3…a1\"). The fast‑path scans the whole string looking for indicator characters, finds the “a”, then falls back to the regex parser – an extra O(n) pass over the input.", "why": "The fast‑path’s `isdisjoint` test iterates over every character. For a long dotted release the string length grows with n, so the extra scan adds linear overhead before the (already linear) regex match.", "size_meaning": "Number of dotted release segments (plus the trailing pre‑release).", "make_source": "def make(n):\n # Build \"1.2.3…n\" then add a pre‑release marker \"a1\"\n release = \".\".join(str(i) for i in range(1, n + 1))\n version = f\"{release}a1\"\n return [(version,)]", "example_args": [], "origin": "archive", "screen_ratio": 1.0005560778858236, "valid_fraction": 1.0, "error": null } ], "findings": [], "seconds": { "plan": 15.65, "screen": 353.96, "full": 0.0, "total": 369.61 }, "ladder_validity": "backfilled 2026-10-11 (scripts/backtest/backfill_ladder_validity.py)" } } } ] }, { "id": "#1105", "pr": 1105, "title": "perf: Speed up complex SpecifierSet filtering by implementing cost based ordering", "base": "9b972011a53185a3c9be97dc1f86c042307e0e5a", "head": "8810e9e66e2a137af5b16072c1ceb17427a2b21c", "merged": "2026-03-08", "after_ultra_release": false, "known": [], "targets": [ { "target": "packaging.specifiers:SpecifierSet#filter", "run_id": "bt-1105-4c6611", "image": "bw-pair:3124cc07c70a8c7a", "usd": 0.1238, "harvested": 80, "arms": { "hypothesis": { "classes": [], "known": [], "seconds": 31.2, "error": null }, "mutations": { "classes": [], "known": [], "seconds": 15.9, "error": null }, "breakwater": { "classes": [], "known": [], "seconds": 123.0, "error": null }, "llm-plan": { "classes": [], "known": [], "seconds": 9.1, "error": null } }, "findings": [], "perf": { "slower_families": [], "worst_valid_ratio": null, "worst_family": null, "speedup_ratio_small": 0.5141606219028452, "report": { "run_id": "bt-1105-4c6611", "target": "packaging.specifiers:SpecifierSet#filter", "image": "bw-pair:3124cc07c70a8c7a", "backend": "docker", "families": [ { "id": "F1", "kind": "claimed_speedup", "family": "SpecifierSet.filter with many versions and a mixed set of specifiers (range, compatible, wildcard equality) under warm prerelease handling", "why": "The change introduces a fast‑path that resolves all specifiers once, sorts them by a static cost model and filters all versions in a single pass. The benchmark `time_filter_complex_warm` exercises exactly this path, so the change is advertised as a speed‑up for complex, warm specifier sets.", "size_meaning": "Number of version strings (n) and number of specifiers (m) – both grow with the parameter.", "make_source": "random.seed(0)\n\ndef make(n):\n # n versions, n//5 specifiers (at least 3)\n versions = [f\"{random.randint(1,10)}.{random.randint(0,9)}.{random.randint(0,9)}a{random.randint(0,9)}\" for _ in range(n)]\n # Build a mixed specifier set: range, compatible, wildcard equality, inequality\n ops = [\">=1.0\", \"~=2.0\", \"==3.*\", \"!=4.0\", \"<5.0\"]\n spec = \",\".join(ops[:max(3, n//5)])\n # The target function is SpecifierSet(...).filter(versions)\n return [[spec, versions]]", "example_args": [ [ ">=1.0,~=2.0,==3.*,!=4.0", [ "1.2.3a1", "2.0.0a2", "3.1.0a0", "4.5.6a9" ] ] ], "origin": "plan", "screen_ratio": 0.8551914755262151, "valid_fraction": 1.0, "error": null }, { "id": "F2", "kind": "claimed_speedup", "family": "SpecifierSet filter (complex, cold) – many versions against a complex specifier set", "why": "The change introduces a cost‑based single‑pass filter that orders specifiers so cheap rejecting operators run first. With many candidate versions the new single pass avoids the repeated iterator wrapping of the old chained‑filter implementation, giving a measurable speed‑up.", "size_meaning": "Number of candidate version strings supplied to the filter.", "make_source": "\ndef make(n):\n # Produce a list with one argument list: an iterable of n version strings.\n # Use simple PEP‑440 final releases that exercise the range and compatible operators.\n versions = [f\"1.{i}.0\" for i in range(n)]\n return [ [versions] ]", "example_args": [ [ "1.0.0", "1.1.0", "1.2.0" ] ], "origin": "plan", "screen_ratio": 1.0058763507377868, "valid_fraction": 0.0, "error": null }, { "id": "F3", "kind": "suspected_slowdown", "family": "SpecifierSet.filter (cold) with many pre‑release versions and several cheap range specifiers", "why": "The new single‑pass filter first collects all non‑prereleases, then buffers every prerelease version until the end of the pass. With many prereleases and several cheap range specs that reject only a few, the old chained filters would discard most versions after the first cheap spec, so later specs never saw the buffered prereleases. The new code keeps the whole buffered list alive for the whole pass, increasing memory traffic and CPU work.", "size_meaning": "n = number of version strings; m = number of range specifiers (both grow with the parameter).", "make_source": "random.seed(1)\n\ndef make(n):\n # generate many pre‑release versions\n versions = [f\"1.{i}.0a{i%10}\" for i in range(n)]\n # several cheap range specs that only reject a few\n specs = [f\">={i}.0\" for i in range(1, max(3, n//20)+1)]\n spec = \",\".join(specs)\n return [[spec, versions]]", "example_args": [ [ ">=1.0,>=2.0,>=3.0", [ "1.0.0a0", "1.1.0a1", "1.2.0a2", "1.3.0a3" ] ] ], "origin": "plan", "screen_ratio": 0.6695864850000335, "valid_fraction": 1.0, "error": null }, { "id": "F4", "kind": "suspected_slowdown", "family": "SpecifierSet filter (complex, cold) – worst‑case operator ordering", "why": "The new implementation sorts specifiers by a static cost model. If a user‑supplied specifier list puts a very selective but expensive operator (e.g. \"!=1.*\") first, the old chained‑filter would reject most versions early. The new cost order moves that expensive operator later, so many versions survive cheap checks only to be rejected later, increasing total work.", "size_meaning": "Number of candidate version strings; the specifiers string is fixed to a worst‑case ordering.", "make_source": "\ndef make(n):\n # A specifier set where the first specifier is an expensive wildcard not‑equal\n # that would reject the majority of versions, followed by cheap range checks.\n # The benchmark harness builds the SpecifierSet from this string.\n # We only need to vary the size of the version iterable.\n versions = [f\"1.{i}.0\" for i in range(n)]\n return [ [versions] ]", "example_args": [ [ "1.0.0", "1.1.0", "1.2.0" ] ], "origin": "plan", "screen_ratio": 1.0058763507377868, "valid_fraction": 0.0, "error": null }, { "id": "F5", "kind": "suspected_slowdown", "family": "SpecifierSet.filter (cold) with many \"!=*.*\" wildcard inequality specifiers", "why": "The cost model assigns wildcard !=.* a high cost (tier 4) so they are evaluated last. In the old code last. The old chained filter evaluated them in user order; if the user placed many !=*.* early, each version was checked against many cheap inequality tests that almost never reject, then later cheap range specs would run on the survivors. The new ordering forces all cheap range specs first (good) but still has to evaluate every !=*.* against the full surviving set, which can be large because the cheap range specs reject few versions. The net work per version can be higher.", "size_meaning": "n = number of versions; m = number of !=*.* specifiers (both grow with the parameter).", "make_source": "random.seed(2)\n\ndef make(n):\n versions = [f\"{random.randint(1,9)}.{random.randint(0,9)}.{random.randint(0,9)}\" for _ in range(n)]\n specs = [\"!=*.*\" for _ in range(max(2, n//30))]\n # add a couple cheap range specs that reject little\n specs += [\">=0.0\", \"<10.0\"]\n spec = \",\".join(specs)\n return [[spec, versions]]", "example_args": [ [ ">=0.0,<10.0,!=*.*,!=*.*", [ "1.2.3", "2.3.4", "3.4.5", "4.5.6" ] ] ], "origin": "plan", "screen_ratio": 0.9977531190255632, "valid_fraction": 0.0, "error": null }, { "id": "F6", "kind": "suspected_slowdown", "family": "SpecifierSet filter (complex, cold) – many specifiers, large iterable", "why": "The new single‑pass filter iterates every candidate version through *all* operators (short‑circuiting on the first failure) whereas the old implementation fed a shrinking iterator to each specifier. When the number of specifiers is large (≥10) and the iterable contains many versions that pass the cheap checks, the per‑item Python loop overhead dominates, making the new path slower.", "size_meaning": "Number of candidate version strings; the specifier set is built with ten range specifiers.", "make_source": "\ndef make(n):\n # Build a large iterable of final releases.\n versions = [f\"2.{i}.0\" for i in range(n)]\n return [ [versions] ]", "example_args": [ [ "2.0.0", "2.1.0", "2.2.0" ] ], "origin": "plan", "screen_ratio": 1.0058763507377868, "valid_fraction": 0.0, "error": null } ], "findings": [ { "family_id": "F1", "kind": "claimed_speedup", "family": "SpecifierSet.filter with many versions and a mixed set of specifiers (range, compatible, wildcard equality) under warm prerelease handling", "why": "The change introduces a fast‑path that resolves all specifiers once, sorts them by a static cost model and filters all versions in a single pass. The benchmark `time_filter_complex_warm` exercises exactly this path, so the change is advertised as a speed‑up for complex, warm specifier sets.", "origin": "plan", "screen_ratio": 0.8551914755262151, "valid_fraction": 1.0, "ladder": [ { "n": 2, "base_ir": 524023.825, "head_ir": 470501.115, "input_chars": 45.0, "ratio": 0.8978620676264099, "valid_fraction": 1.0, "judged": true }, { "n": 4, "base_ir": 1040135.585, "head_ir": 903852.125, "input_chars": 68.0, "ratio": 0.868975293254677, "valid_fraction": 1.0, "judged": true }, { "n": 16, "base_ir": 3523634.102040816, "head_ir": 3013381.8469387754, "input_chars": 202.0, "ratio": 0.8551914755262151, "valid_fraction": 1.0, "judged": true }, { "n": 64, "base_ir": 13294929.423076924, "head_ir": 7636401.346153846, "input_chars": 749.0, "ratio": 0.574384496761511, "valid_fraction": 1.0, "judged": true }, { "n": 256, "base_ir": 52116696.666666664, "head_ir": 27296843.5, "input_chars": 2874.0, "ratio": 0.5237638846258419, "valid_fraction": 1.0, "judged": true }, { "n": 1024, "base_ir": 215832073.0, "head_ir": 114339961.0, "input_chars": 11403.0, "ratio": 0.5297635305573885, "valid_fraction": 1.0, "judged": true }, { "n": 4096, "base_ir": 856009526.5, "head_ir": 440126390.5, "input_chars": 45524.0, "ratio": 0.5141606219028452, "valid_fraction": 1.0, "judged": true } ], "slope_base": 1.051372725123336, "slope_head": 0.95886399544333, "top_inputs": [], "error": null, "verdict": "faster", "worst_ratio": 0.8978620676264099, "worst_search": null } ], "seconds": { "plan": 19.23, "screen": 266.21, "full": 154.32, "total": 439.76 }, "ladder_validity": "backfilled 2026-10-11 (scripts/backtest/backfill_ladder_validity.py)" } } } ] }, { "id": "#1106", "pr": 1106, "title": "perf: simply Specifier regex", "base": "e3af78f17a5230ffdc846e0b08f3e6e47b3edcfe", "head": "e839d73030b410d4156e6a496249fbdb1e498fab", "merged": "2026-03-09", "after_ultra_release": false, "known": [], "targets": [ { "target": "packaging.specifiers:Specifier", "run_id": "bt-1106-20d609", "image": "bw-pair:339ff3435317f389", "usd": 0.1137, "harvested": 80, "arms": { "hypothesis": { "classes": [], "known": [], "seconds": 13.1, "error": null }, "mutations": { "classes": [], "known": [], "seconds": 7.6, "error": null }, "breakwater": { "classes": [], "known": [], "seconds": 84.3, "error": null }, "llm-plan": { "classes": [], "known": [], "seconds": 5.8, "error": null } }, "findings": [], "perf": { "slower_families": [], "worst_valid_ratio": null, "worst_family": null, "speedup_ratio_small": 0.9364916973923361, "report": { "run_id": "bt-1106-20d609", "target": "packaging.specifiers:Specifier", "image": "bw-pair:339ff3435317f389", "backend": "docker", "families": [ { "id": "F1", "kind": "claimed_speedup", "family": "Simple specifier strings (e.g., \"==1.0\", \">=2.3.4\")", "why": "The new implementation replaces a two‑stage regex match + regex group extraction with a single verbose regex followed by cheap `str.startswith` checks. For short, simple specifiers the new path avoids regex group overhead and is faster.", "size_meaning": "Number of simple specifiers to construct (each call creates a new Specifier).", "make_source": "def make(n):\n specs = [\">=1.\" + \".0\" * i for i in range(n)]\n return [[s] for s in specs]", "example_args": [ [ ">=1.0" ], [ ">=1.0.0" ], [ ">=1.0.0.0" ] ], "origin": "plan", "screen_ratio": 0.8986033334222537, "valid_fraction": 0.0, "error": null }, { "id": "F2", "kind": "claimed_speedup", "family": "Typical short PEP 440 specifiers (==, >=, <=, ~=, !=, === with short versions)", "why": "The new hand‑rolled parser avoids a full regex match on every construction; for short, well‑formed specifiers the manual `startswith` checks are faster than a full verbose regex match.", "size_meaning": "Number of specifiers in the generated list (each entry is a single argument list for `Specifier`).", "make_source": "def make(n):\n ops = ['==', '>=', '<=', '~=', '!=', '>', '<', '===']\n base_versions = ['1.0', '2.3.4', '10.0.0a1', '3.2.1.post2', '5.0.0+local']\n out = []\n for i in range(min(n, len(ops)*len(base_versions))):\n op = ops[i % len(ops)]\n ver = base_versions[i % len(base_versions)]\n out.append([f'{op}{ver}'])\n return out", "example_args": [], "origin": "plan", "screen_ratio": 0.954287534744055, "valid_fraction": 0.875, "error": null }, { "id": "F3", "kind": "suspected_slowdown", "family": "long_numeric_version", "why": "The new verbose regex contains a large alternation with many optional groups for each version component. Long numeric version strings (many dot‑separated numbers) cause the engine to explore many alternatives and backtrack in the '>=', '<=', '>', '<', '~=', '==', '!=' branches, making the full‑match slower than the older, simpler concatenated regex.", "size_meaning": "Number of numeric components in the version (e.g. \"1.2.3…\")", "make_source": "\ndef make(n):\n # create a version with n+1 numeric components, e.g. \"1.2.3...\"\n version = \".\".join(str(i) for i in range(1, n + 2))\n # use a range operator that exercises the complex branch of the new regex\n spec = \">=\" + version\n return [[spec]]", "example_args": [], "origin": "plan", "screen_ratio": 0.8965109533657188, "valid_fraction": 1.0, "error": null }, { "id": "F4", "kind": "suspected_slowdown", "family": "Specifiers with very long numeric version segments (many dot‑separated numbers)", "why": "The new single verbose regex combines the operator and version patterns into one huge alternation. The version part `[0-9]+(?:\\.[0-9]+)*` is now inside a large alternation group; long dot‑separated numeric runs can cause the engine to backtrack across the operator alternatives when the version part fails to match, making matching slower than the old two‑stage regex.", "size_meaning": "Number of numeric components after the first (e.g., \"1.2.3.4.5\" has size 4).", "make_source": "def make(n):\n # produce n specifiers each with an increasingly long numeric tail\n specs = [\">=\" + \"1\" + \".0\" * i for i in range(1, n+1)]\n return [[s] for s in specs]", "example_args": [ [ ">=1.0" ], [ ">=1.0.0" ], [ ">=1.0.0.0.0.0.0.0.0.0" ] ], "origin": "plan", "screen_ratio": 0.9647270246812021, "valid_fraction": 1.0, "error": null }, { "id": "F5", "kind": "suspected_slowdown", "family": "Very long numeric version strings (many dot‑separated components) with a simple operator, e.g. '>=1.2.3.4.5...'.", "why": "The new combined verbose regex contains the pattern `[0-9]+(?:\\.[0-9]+)*` inside *every* operator alternative. A very long dotted version forces the regex engine to repeat the `\\.[0-9]+` group many times in each alternative before the engine can decide which alternative matches, causing noticeable back‑tracking on each construction.", "size_meaning": "Number of dotted numeric components (n). The generated argument list contains a single specifier string of length ~2n characters.", "make_source": "def make(n):\n # generate a version like 1.0.0.0.... with n numeric components\n parts = ['1'] + ['0'] * (n-1)\n version = '.'.join(parts)\n return [[f'>={version}']]", "example_args": [], "origin": "plan", "screen_ratio": 0.897121217975965, "valid_fraction": 1.0, "error": null }, { "id": "F6", "kind": "suspected_slowdown", "family": "many_local_segments", "why": "The new verbose regex contains an extensive alternation for the local‑version part (+[a-z0-9]+(?:[-_.][a-z0-9]+)*). A specifier with many dot‑separated local segments forces the engine to repeatedly match the repeated group, causing noticeable backtracking compared with the older, more compact version regex.", "size_meaning": "Number of dotted local‑version segments (e.g. \"+local1.local2...\")", "make_source": "def make(n):\n # build a version with n local segments: \"+local1.local2...localN\"\n local = \".\".join(f\"local{i}\" for i in range(1, n + 1))\n spec = \">=1.0+\" + local\n return [[spec]]", "example_args": [], "origin": "plan", "screen_ratio": 0.8933075098785771, "valid_fraction": 0.0, "error": null }, { "id": "R1", "kind": "suspected_slowdown", "family": "wildcard-many-segments", "why": "The `==X.*` wildcard syntax allows many dot-separated numeric segments. The new parser strips the full string, slices off the two-char operator, then strips the version part — copying the long segment list three times. The old regex captured the version in a single slice.", "size_meaning": "n = number of numeric segments before the .* wildcard", "make_source": "def make(n):\n # ==1.2.3....* with n segments\n segs = \".\".join(str(i) for i in range(1, n + 1))\n return [[f\"=={segs}.*\"]]", "example_args": [], "origin": "archive", "screen_ratio": 0.9721108100310571, "valid_fraction": 1.0, "error": null }, { "id": "R2", "kind": "suspected_slowdown", "family": "wildcard-one-fixed-length", "why": "The `==X.*` wildcard syntax allows many dot-separated numeric segments. The new parser strips the full string, slices off the two-char operator, then strips the version part — copying the long segment list three times. The old regex captured the version in a single slice.", "size_meaning": "n = number of numeric segments before the .* wildcard", "make_source": "def make(n):\n # ==N.N...* with fixed-length segment N repeated n times\n segs = \".\".join([str(42)] * n)\n return [[f\"=={segs}.*\"]]", "example_args": [], "origin": "archive", "screen_ratio": 0.9725165987479164, "valid_fraction": 1.0, "error": null }, { "id": "R3", "kind": "suspected_slowdown", "family": "wildcard-leading-zero-padded", "why": "The `==X.*` wildcard syntax allows many dot-separated numeric segments. The new parser strips the full string, slices off the two-char operator, then strips the version part — copying the long segment list three times. The old regex captured the version in a single slice.", "size_meaning": "n = number of numeric segments before the .* wildcard", "make_source": "def make(n):\n # ==001.002... with zero-padded segments before .*\n segs = \".\".join(f\"{i:0>3}\" for i in range(1, n + 1))\n return [[f\"=={segs}.*\"]]", "example_args": [], "origin": "archive", "screen_ratio": 0.9731745936739924, "valid_fraction": 1.0, "error": null } ], "findings": [ { "family_id": "F2", "kind": "claimed_speedup", "family": "Typical short PEP 440 specifiers (==, >=, <=, ~=, !=, === with short versions)", "why": "The new hand‑rolled parser avoids a full regex match on every construction; for short, well‑formed specifiers the manual `startswith` checks are faster than a full verbose regex match.", "origin": "plan", "screen_ratio": 0.954287534744055, "valid_fraction": 0.875, "ladder": [ { "n": 2, "base_ir": 15092.655, "head_ir": 15008.655, "input_chars": 10.0, "ratio": 0.9944343788418937, "valid_fraction": 1.0, "judged": true }, { "n": 4, "base_ir": 15559.37125, "head_ir": 15483.33625, "input_chars": 12.75, "ratio": 0.9951132344117054, "valid_fraction": 1.0, "judged": true }, { "n": 16, "base_ir": 16469.79649390244, "head_ir": 15716.921493902439, "input_chars": 13.1875, "ratio": 0.954287534744055, "valid_fraction": 0.875, "judged": true }, { "n": 64, "base_ir": 16439.5828125, "head_ir": 15395.5328125, "input_chars": 13.475, "ratio": 0.9364916973923361, "valid_fraction": 0.875, "judged": true }, { "n": 256, "base_ir": 16439.5828125, "head_ir": 15395.5328125, "input_chars": 13.475, "ratio": 0.9364916973923361, "valid_fraction": 0.875, "judged": true }, { "n": 1024, "base_ir": 16439.5828125, "head_ir": 15395.5328125, "input_chars": 13.475, "ratio": 0.9364916973923361, "valid_fraction": 0.875, "judged": true }, { "n": 4096, "base_ir": 16439.5828125, "head_ir": 15395.5328125, "input_chars": 13.475, "ratio": 0.9364916973923361, "valid_fraction": 0.875, "judged": true } ], "slope_base": null, "slope_head": null, "top_inputs": [], "error": null, "verdict": "neutral", "worst_ratio": 0.9951132344117054, "worst_search": null } ], "seconds": { "plan": 17.13, "screen": 171.03, "full": 128.92, "total": 317.08 }, "ladder_validity": "backfilled 2026-10-11 (scripts/backtest/backfill_ladder_validity.py)" } }, "image_rescored": "bw-pair:6c9a610c08b433b1" } ] }, { "id": "#1108", "pr": 1108, "title": "perf: SpecifierSet use tuple instead of frozenset for `_specs`", "base": "20f1523cf37dc5a0e85e7c7dc0070dfc4a09653a", "head": "f59d6ee9afe2330cc9524f825417ee1c31e5e684", "merged": "2026-03-07", "after_ultra_release": false, "known": [ { "id": "dedup-equality", "match": { "probe": "P3", "detail": "eq" }, "fixed_by": 1109, "status": "verified" }, { "id": "dedup-str-len", "match": { "probe": "P1" }, "fixed_by": 1109, "status": "verified" }, { "id": "old-pickles-break-comparison", "match": { "probe": "P4" }, "fixed_by": "26.2", "status": "verified" } ], "targets": [ { "target": "packaging.specifiers:SpecifierSet", "run_id": "bt-1108-c01604", "image": "bw-pair:47497edc9ebcda96", "usd": 0.0934, "harvested": 80, "arms": { "hypothesis": { "classes": [ "P4|base->head eq-fresh|True|" ], "known": [ "old-pickles-break-comparison" ], "seconds": 6.9, "error": null }, "mutations": { "classes": [ "P1|len|differs|plain", "P1|str|differs|plain", "P2|outcome|exc:builtins.AssertionError|ok|long-digits", "P2|outcome|exc:builtins.ValueError|ok|long-digits", "P4|base->head eq-fresh|True|", "P4|base->head eq-fresh|True|False" ], "known": [ "dedup-str-len", "old-pickles-break-comparison" ], "seconds": 3.9, "error": null }, "breakwater": { "classes": [ "P1|len|differs|plain", "P1|len|differs|whitespace", "P1|str|differs|plain", "P1|str|differs|whitespace", "P2|outcome|exc:builtins.AssertionError|ok|long-digits", "P2|outcome|exc:builtins.ValueError|ok|long-digits", "P3|eq|True|False", "P3|hash_eq|True|False", "P4|base->head eq-fresh|True|", "P4|base->head eq-fresh|True|False" ], "known": [ "dedup-equality", "dedup-str-len", "old-pickles-break-comparison" ], "seconds": 73.5, "error": null }, "llm-plan": { "classes": [ "P4|base->head eq-fresh|True|" ], "known": [ "old-pickles-break-comparison" ], "seconds": 5.8, "error": null } }, "findings": [ { "signature": "P1|len|differs|plain", "probe": "P1", "detail": "len", "args": [ [ ">=01.0,!=2.0,>=1.0" ] ], "base": 2, "head": 3, "found_by": [ "mutation:equivalents", "mutation:separators" ] }, { "signature": "P1|len|differs|whitespace", "probe": "P1", "detail": "len", "args": [ [ ">= 1.0,!=2.0,>=1.0" ] ], "base": 2, "head": 3, "found_by": [ "mutation:whitespace-inner" ] }, { "signature": "P1|str|differs|plain", "probe": "P1", "detail": "str", "args": [ [ ">=01.0,!=2.0,>=1.0" ] ], "base": "!=2.0,>=01.0", "head": "!=2.0,>=01.0,>=1.0", "found_by": [ "mutation:equivalents", "mutation:separators" ] }, { "signature": "P1|str|differs|whitespace", "probe": "P1", "detail": "str", "args": [ [ ">= 1.0,!=2.0,>=1.0" ] ], "base": "!=2.0,>=1.0", "head": "!=2.0,>=1.0,>=1.0", "found_by": [ "mutation:whitespace-inner" ] }, { "signature": "P2|outcome|exc:builtins.AssertionError|ok|long-digits", "probe": "P2", "detail": "outcome", "args": [ [ ">=10000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000.0" ] ], "base": { "status": "exc", "exc": "builtins.AssertionError" }, "head": { "status": "ok", "obs": { "str": ">=10000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000.0", "len": 1 } }, "found_by": [ "mutation:huge-number" ] }, { "signature": "P2|outcome|exc:builtins.ValueError|ok|long-digits", "probe": "P2", "detail": "outcome", "args": [ [ "<=10000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000!0.1" ] ], "base": { "status": "exc", "exc": "builtins.ValueError" }, "head": { "status": "ok", "obs": { "str": "<=10000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000!0.1", "len": 1 } }, "found_by": [ "mutation:huge-number" ] }, { "signature": "P3|eq|True|False", "probe": "P3", "detail": "eq", "args": [ [ ">=01.0,!=2.0,>=1.0" ], [ "!=2.0,>=1.0" ] ], "base": true, "head": false, "found_by": [ "focus" ] }, { "signature": "P3|hash_eq|True|False", "probe": "P3", "detail": "hash_eq", "args": [ [ ">=01.0,!=2.0,>=1.0" ], [ "!=2.0,>=1.0" ] ], "base": true, "head": false, "found_by": [ "focus" ] }, { "signature": "P4|base->head eq-fresh|True|", "probe": "P4", "detail": "base->head eq-fresh", "args": [ [ "" ] ], "base": true, "head": "", "found_by": [ "seed", "tests", "mutation:whitespace-outer", "mutation:huge-number", "mutation:edges", "mutation:equivalents", "mutation:case", "mutation:whitespace-inner", "mutation:separators", "batch" ] }, { "signature": "P4|base->head eq-fresh|True|False", "probe": "P4", "detail": "base->head eq-fresh", "args": [ [ ">=01.0,!=2.0,>=1.0" ] ], "base": true, "head": false, "found_by": [ "mutation:equivalents", "mutation:whitespace-inner", "mutation:separators" ] } ], "perf": { "slower_families": [], "worst_valid_ratio": null, "worst_family": null, "speedup_ratio_small": 0.3537336553993164, "report": { "run_id": "bt-1108-c01604", "target": "packaging.specifiers:SpecifierSet", "image": "bw-pair:47497edc9ebcda96", "backend": "docker", "families": [ { "id": "F1", "kind": "claimed_speedup", "family": "small_specifier_set", "why": "The new implementation lazily sorts the internal tuple only when an order‑sensitive operation (__hash__, __eq__, __str__, __and__) is needed. For very small specifier sets (0‑2 items) the lazy flag `_sorted` is set to True immediately, avoiding any sorting work entirely. The old frozenset‑based implementation always built a frozenset and computed its hash on construction, which is relatively heavier for tiny sets.", "size_meaning": "n = number of simple specifiers in the comma‑separated string (e.g. '>=1.0.0,!=2.0.0,>3.0.0'). The cost of parsing grows with n, but the constructor itself stays O(n) and the new lazy‑sort path stays O(1) for n≤2.", "make_source": "\ndef make(n):\n # Produce a single SpecifierSet argument containing n simple specifiers.\n # n is kept tiny (1‑5) for the “small‑set” family.\n specs = [f\">=1.{i}.0\" for i in range(n)]\n spec_string = \",\".join(specs)\n return [[spec_string]]", "example_args": [ [ ">=1.0.0" ], [ ">=1.0.0,!=2.0.0" ] ], "origin": "plan", "screen_ratio": 0.36838918830636114, "valid_fraction": 1.0, "error": null }, { "id": "F2", "kind": "claimed_speedup", "family": "Large SpecifierSet (many specifiers) constructed once and then repeatedly hashed / compared / stringified – the change caches a sorted tuple so repeated operations become O(1) after the first sort.", "why": "The new implementation stores a cached sorted tuple (_sorted flag + _sort_specs) and uses it for __hash__, __eq__, __str__ and __and__. For a large set (many specifiers) the first sort costs O(N log N) once, but repeated hashing/equality/str become O(N) with a cheap cached tuple, whereas the old frozenset implementation recomputed hashes/equality each time.", "size_meaning": "n = number of comma‑separated specifiers in the input string (e.g. 200). Cost grows with n because the initial sort is O(n log n) and the cached tuple size is O(n).", "make_source": "\ndef make(n):\n # produce up to 8 argument lists with growing size\n specs = [f\">=1.0.{i}\" for i in range(n)]\n out = []\n for k in range(1, min(9, n+1)):\n out.append([\",\".join(specs[:k])])\n return out", "example_args": [ [ ">=1.0.0,>=1.0.1,>=1.0.2,>=1.0.3,>=1.0.4,>=1.0.5,>=1.0.6,>=1.0.7" ] ], "origin": "plan", "screen_ratio": 0.3894391838836436, "valid_fraction": 1.0, "error": null }, { "id": "F3", "kind": "suspected_slowdown", "family": "many_distinct_specifier_sets_with_many_specifiers", "why": "Head version sorts the internal tuple on first __hash__/__eq__/__str__ (O(k log k) per set). The old frozenset implementation hashed/compared in O(k). When many distinct SpecifierSet objects are created and immediately hashed/equated (e.g., inserted into a set or used as dict keys), the one‑time sort dominates.", "size_meaning": "n = number of distinct SpecifierSet objects; each object contains ~n comma‑separated simple specifiers (e.g. \">=1.0,!=2.0,>=3.0,…\"). Cost grows roughly as O(n² log n) for the whole benchmark.", "make_source": "\ndef make(n):\n \"\"\"Return a list of n argument‑lists for SpecifierSet.\n Each argument list contains a single specifier string that packs i\n simple specifiers (i grows with n).\"\"\"\n specs = []\n for i in range(1, n + 1):\n # build a specifier like \">=1.0,!=2.0,>=3.0,!=4.0,…\" with i clauses\n parts = []\n for j in range(1, i + 1):\n op = \">=\" if j % 2 else \"!=\"\n parts.append(f\"{op}{j}.0\")\n specs.append([\",\".join(parts)])\n return specs", "example_args": [ [ ">=1.0" ], [ "!=1.0,>=2.0" ], [ ">=1.0,!=2.0,>=3.0" ] ], "origin": "plan", "screen_ratio": 0.4006538473039662, "valid_fraction": 1.0, "error": null }, { "id": "F4", "kind": "suspected_slowdown", "family": "large_specifier_set", "why": "Construction now builds a `dict.fromkeys([…])` to deduplicate while preserving insertion order, then converts to a tuple. For large numbers of specifiers (hundreds to thousands) the dict allocation and hashing overhead is noticeably higher than the old `frozenset(map(Specifier, …))`, which only builds a hash set. The lazy‑sort flag also starts as `False`, so the first order‑sensitive operation (e.g. `__hash__` when the set is used as a dict key) will trigger an O(n log n) sort that the old version never performed.", "size_meaning": "n = number of specifiers in the comma‑separated string. Cost grows roughly linearly in the constructor and O(n log n) on first hashing/equality.", "make_source": "def make(n):\n # Generate n distinct simple specifiers, e.g. \"==1.0.0\", \"==1.0.1\", …\n specs = [f\"==1.0.{i}\" for i in range(n)]\n spec_string = \",\".join(specs)\n return [[spec_string]]", "example_args": [ [ "==1.0.0,==1.0.1,==1.0.2,==1.0.3,==1.0.4" ], [ "==1.0.0,==1.0.1,==1.0.2,==1.0.3,==1.0.4,==1.0.5,==1.0.6,==1.0.7,==1.0.8,==1.0.9" ] ], "origin": "plan", "screen_ratio": 0.37499706909059916, "valid_fraction": 1.0, "error": null }, { "id": "F5", "kind": "suspected_slowdown", "family": "Very small SpecifierSet (1‑2 specifiers) constructed many times – the new code builds a tuple via dict.fromkeys and a Python‑level map(Specifier, …) instead of the old C‑level frozenset(map(...)). The overhead of creating a dict and a tuple dominates for tiny inputs.", "why": "Construction now does `tuple(map(Specifier, dict.fromkeys(split)))`. For 1‑2 items the Python‑level dict creation and tuple allocation are slower than the single C call `frozenset(map(Specifier, split))`. Repeated construction (as in a benchmark loop) will show a slowdown.", "size_meaning": "n = number of specifiers in the comma‑separated string (1‑2). Cost grows linearly with n but the constant factor is higher for the new path.", "make_source": "def make(n):\n specs = [f\">=1.0.{i}\" for i in range(n)]\n out = []\n for k in range(1, min(9, n+1)):\n out.append([\",\".join(specs[:k])])\n return out", "example_args": [ [ ">=1.0.0" ], [ ">=1.0.0,>=1.0.1" ] ], "origin": "plan", "screen_ratio": 0.3894391838836436, "valid_fraction": 1.0, "error": null }, { "id": "F6", "kind": "suspected_slowdown", "family": "many_distinct_specifier_sets_few_specifiers_hashed_once", "why": "Even with only a few specifiers per set, the head version still pays the one‑time sort (k log k) on the first hash/equality, while the old frozenset hash is O(k). When a large number (n) of distinct sets are inserted into a set/dict, the extra log factor accumulates.", "size_meaning": "n = number of distinct SpecifierSet objects; each contains a fixed small number (e.g. 4) of specifiers. Total work grows ~ n·log k vs n·k.", "make_source": "def make(n):\n \"\"\"Return n argument‑lists, each with a unique 4‑clause specifier string.\"\"\"\n specs = []\n for i in range(1, n + 1):\n parts = [f\">={i}.0\", f\"!={i+10}.0\", f\">={i+20}.0\", f\"!={i+30}.0\"]\n specs.append([\",\".join(parts)])\n return specs", "example_args": [ [ ">=1.0,!=11.0,>=21.0,!=31.0" ], [ ">=2.0,!=12.0,>=22.0,!=32.0" ] ], "origin": "plan", "screen_ratio": 0.4178571778642251, "valid_fraction": 1.0, "error": null } ], "findings": [ { "family_id": "F1", "kind": "claimed_speedup", "family": "small_specifier_set", "why": "The new implementation lazily sorts the internal tuple only when an order‑sensitive operation (__hash__, __eq__, __str__, __and__) is needed. For very small specifier sets (0‑2 items) the lazy flag `_sorted` is set to True immediately, avoiding any sorting work entirely. The old frozenset‑based implementation always built a frozenset and computed its hash on construction, which is relatively heavier for tiny sets.", "origin": "plan", "screen_ratio": 0.36838918830636114, "valid_fraction": 1.0, "ladder": [ { "n": 2, "base_ir": 90813.87, "head_ir": 39076.335, "input_chars": 19.0, "ratio": 0.4302903840569728, "valid_fraction": 1.0, "judged": true }, { "n": 4, "base_ir": 176754.405, "head_ir": 69792.21, "input_chars": 35.0, "ratio": 0.39485414804796526, "valid_fraction": 1.0, "judged": true }, { "n": 16, "base_ir": 695347.8671328671, "head_ir": 256158.63636363635, "input_chars": 137.0, "ratio": 0.36838918830636114, "valid_fraction": 1.0, "judged": true }, { "n": 64, "base_ir": 2806618.2, "head_ir": 1040455.0, "input_chars": 569.0, "ratio": 0.3707148339592467, "valid_fraction": 1.0, "judged": true }, { "n": 256, "base_ir": 11670434.625, "head_ir": 4128225.5, "input_chars": 2453.0, "ratio": 0.3537336553993164, "valid_fraction": 1.0, "judged": true }, { "n": 1024, "base_ir": 46997708.0, "head_ir": 17409861.0, "input_chars": 10157.0, "ratio": 0.3704406393605407, "valid_fraction": 1.0, "judged": true }, { "n": 4096, "base_ir": 192841273.0, "head_ir": 69552870.0, "input_chars": 43949.0, "ratio": 0.36067419032231757, "valid_fraction": 1.0, "judged": true } ], "slope_base": 0.9858549829338461, "slope_head": 0.9683664866072472, "top_inputs": [ { "args": [ [ "𓟝ĬCª¤,ŀĎ,ŀĎ" ] ], "cpu_ratio": 4.718, "base_ir": 4001.555, "head_ir": 6293.555, "ir_ratio": 1.5727773328118695, "outcome": "ok" }, { "args": [ [ "𓟝ĬCª¤,ŀĎ,ŀĎ,iÐ,ĭ" ] ], "cpu_ratio": 4.546, "base_ir": 3976.555, "head_ir": 6293.555, "ir_ratio": 1.5826651460875056, "outcome": "ok" } ], "error": null, "verdict": "faster", "worst_ratio": 1.5826651460875056, "worst_search": { "args": [ [ "𓟝ĬCª¤,ŀĎ,ŀĎ,iÐ,ĭ" ] ], "cpu_ratio": 4.546, "base_ir": 3976.555, "head_ir": 6293.555, "ir_ratio": 1.5826651460875056, "outcome": "ok" } } ], "seconds": { "plan": 32.05, "screen": 89.97, "full": 141.42, "total": 263.44 }, "ladder_validity": "backfilled 2026-10-11 (scripts/backtest/backfill_ladder_validity.py)" } }, "image_rescored": "bw-pair:16f20a2b68c37e6b" } ] }, { "id": "#1111", "pr": 1111, "title": "perf: Speed up wildcard comparison", "base": "210d878577f7f5d95b50904063c889139654813a", "head": "e3af78f17a5230ffdc846e0b08f3e6e47b3edcfe", "merged": "2026-03-09", "after_ultra_release": false, "known": [], "targets": [ { "target": "packaging.specifiers:Specifier#contains", "run_id": "bt-1111-402f78", "image": "bw-pair:a2272377c91373ce", "usd": 0.1213, "harvested": 80, "arms": { "hypothesis": { "classes": [], "known": [], "seconds": 8.5, "error": null }, "mutations": { "classes": [], "known": [], "seconds": 2.5, "error": null }, "breakwater": { "classes": [], "known": [], "seconds": 67.5, "error": null }, "llm-plan": { "classes": [], "known": [], "seconds": 3.0, "error": null } }, "findings": [], "perf": { "slower_families": [], "worst_valid_ratio": null, "worst_family": null, "speedup_ratio_small": 0.7735589114968952, "report": { "run_id": "bt-1111-402f78", "target": "packaging.specifiers:Specifier#contains", "image": "bw-pair:a2272377c91373ce", "backend": "docker", "families": [ { "id": "F1", "kind": "claimed_speedup", "family": "Many prospective versions checked against a wildcard equality specifier (e.g. \"==1.*\"). The change caches the split of the wildcard spec, turning a per‑comparison split‑and‑pad into a cheap attribute lookup plus a small left‑pad.", "why": "Head caches the result of `_get_wildcard_split`, turning O(N) work per comparison into O(1) + a tiny `_left_pad`. Base recomputed the split and numeric prefix on every call.", "size_meaning": "n = number of prospective versions to test (each call to `contains` gets one version string).", "make_source": "\ndef make(n):\n spec = \"==1.*\"\n # generate n version strings like \"1.0\", \"1.1\", … \"1.n\"\n versions = [f\"1.{i}\" for i in range(n)]\n # each call to Specifier.contains receives a.contains gets a new Specifier instance, then we call contains\n # The benchmark harness will create the Specifier once and call contains repeatedly.\n # We just need the argument lists for `contains`.\n return [[v] for v in versions]", "example_args": [ [ "1.0" ], [ "1.1" ], [ "1.2" ] ], "origin": "plan", "screen_ratio": 0.9964471878704373, "valid_fraction": 0.0, "error": null }, { "id": "F2", "kind": "claimed_speedup", "family": "wildcard prefix equality (==X.*)", "why": "The change adds a per‑Specifier cache (_wildcard_split) for the split version components used by the \"==X.*\" prefix‑match logic. Repeated contains() calls with many prospective versions should now reuse the cached split instead of recomputing it each time.", "size_meaning": "n = number of prospective version strings fed to contains(); each version gets longer (more dot‑segments) as n grows, so the cost of splitting the prospective version grows while the spec split stays cached.", "make_source": "\ndef make(n):\n \"\"\"Return a list of argument lists for Specifier.contains.\n Each inner list is [version_string] – the specifier is fixed to \"==1.*\".\"\"\"\n spec = \"==1.*\"\n # generate n versions like \"1.0\", \"1.1\", … \"1.n\" with increasing segment count\n versions = []\n for i in range(1, n + 1):\n # create a version with i numeric segments after the leading \"1\"\n segs = [\"1\"] + [str(j) for j in range(i)]\n versions.append(['.'.join(segs)])\n # each call to contains receives a single version string\n return [[spec, v[0]] for v in versions]", "example_args": [ [ "==1.*", "1.0" ], [ "==1.*", "1.0.1" ], [ "==1.*", "1.0.1.2" ] ], "origin": "plan", "screen_ratio": 0.8102164158617108, "valid_fraction": 1.0, "error": null }, { "id": "F3", "kind": "suspected_slowdown", "family": "Many prospective version strings checked against a single wildcard equality specifier (e.g. \"==1.*\").", "why": "The new implementation caches the wildcard split for the specifier (good), but pads each prospective version with `_left_pad`, which computes the numeric‑prefix length via a pure‑Python loop (`_numeric_prefix_len`) and builds a new list for every prospective version. The old code used `itertools.takewhile` (implemented in C) inside `_pad_version`, which was faster per‑prospective‑version. When a single `Specifier` instance filters a large list of versions, the per‑item Python loop makes the head slower.", "size_meaning": "n = number of prospective version strings fed to `Specifier(...).filter([...])` (or repeated `contains` calls).", "make_source": "\ndef make(n):\n \"\"\"Return argument lists for Specifier.contains / .filter.\n Each inner list is a single call: [specifier_string, prospective_version].\n The specifier is a list length grows with n.\n \"\"\"\n spec = \"==1.*\"\n # generate n version strings with growing numeric prefix length\n versions = [\".\".join([str(i) for i in range(1, k+2)]) for k in range(n)]\n # each call uses the same specifier instance, many prospective versions\n return [[spec, v] for v in versions]", "example_args": [], "origin": "plan", "screen_ratio": 0.8120553637329713, "valid_fraction": 1.0, "error": null }, { "id": "F4", "kind": "suspected_slowdown", "family": "Many wildcard equality checks where the spec ends with \".*\" but the prospective versions are long non‑numeric (e.g. \"1.0a1\", \"1.0b2\", …). The head version adds an attribute load (`self._wildcard_split`) and a call to `_numeric_prefix_len` for every prospective version, which can be slightly slower than the old inline logic that kept the split in local variables.", "why": "Base kept the split spec and numeric length in local variables inside `_compare_equal`; head stores them on the instance and reads them back each call, adding a couple of attribute accesses per comparison.", "size_meaning": "n = number of prospective versions (each becomes one `contains` call).", "make_source": "\ndef make(n):\n # specifier with a wildcard\n spec = \"==1.*\"\n # prospective versions that have a long numeric prefix then a pre‑release\n # e.g. \"1.0.0.0.0a1\" – the numeric prefix length grows with i\n versions = [\"1.\" + \".\".join([\"0\"]*i) + \"a1\" for i in range(1, n+1)]\n return [[v] for v in versions]", "example_args": [ [ "1.0a1" ], [ "1.0.0a1" ], [ "1.0.0.0a1" ] ], "origin": "plan", "screen_ratio": 0.9966109731150778, "valid_fraction": 0.0, "error": null }, { "id": "F5", "kind": "suspected_slowdown", "family": "plain equality (==X.Y.Z) with many prospective versions", "why": "The new __init__ adds an extra slot (_wildcard_split) and an explicit `self._wildcard_split = None` assignment. When a benchmark creates many Specifier objects (e.g. one per specifier in a large SpecifierSet) this extra work shows up as a measurable slowdown even though the contains() hot path is unchanged for non‑wildcard specs.", "size_meaning": "n = number of Specifier objects created; each spec is a simple equality like \"==1.0\", \"==2.0\", …", "make_source": "def make(n):\n \"\"\"Return argument lists for constructing many Specifier instances.\n Each inner list is a single specifier string passed to Specifier.__init__.\"\"\"\n specs = [f\"=={i}.0\" for i in range(1, n + 1)]\n return [[s] for s in specs]", "example_args": [ [ "==1.0" ], [ "==2.0" ], [ "==3.0" ] ], "origin": "plan", "screen_ratio": 1.0037386726077022, "valid_fraction": 0.0, "error": null }, { "id": "F6", "kind": "suspected_slowdown", "family": "Many distinct wildcard equality specifiers each used once (e.g. filtering a single version against many specs).", "why": "Each new `Specifier(\"==X.*\")` triggers `_get_wildcard_split`, which canonicalises and splits the spec using Python code (`_numeric_prefix_len` loop) and stores a tuple. The old path performed the split with `_version_split` and `itertools.takewhile` (C‑level) on every call but without the extra Python loop and tuple allocation. When many one‑off specifiers are created, the extra Python work dominates and the head becomes slower.", "size_meaning": "n = number of distinct wildcard specifiers (each used once).", "make_source": "def make(n):\n \"\"\"Return argument lists for many distinct specifiers each checked once.\n Each inner list is [specifier_string, \"1.2.3\"] – a single prospective version.\n \"\"\"\n base = \"1.2.3\"\n specs = [f\"=={i}.*\" for i in range(1, n+1)]\n return [[s, base] for s in specs]", "example_args": [], "origin": "plan", "screen_ratio": 0.7728906753814948, "valid_fraction": 1.0, "error": null } ], "findings": [ { "family_id": "F2", "kind": "claimed_speedup", "family": "wildcard prefix equality (==X.*)", "why": "The change adds a per‑Specifier cache (_wildcard_split) for the split version components used by the \"==X.*\" prefix‑match logic. Repeated contains() calls with many prospective versions should now reuse the cached split instead of recomputing it each time.", "origin": "plan", "screen_ratio": 0.8102164158617108, "valid_fraction": 1.0, "ladder": [ { "n": 2, "base_ir": 96289.7575, "head_ir": 74485.8, "input_chars": 17.0, "ratio": 0.7735589114968952, "valid_fraction": 1.0, "judged": true }, { "n": 4, "base_ir": 101234.045, "head_ir": 78762.84625, "input_chars": 19.0, "ratio": 0.7780272560481013, "valid_fraction": 1.0, "judged": true }, { "n": 16, "base_ir": 133516.97395833334, "head_ir": 108177.64409722222, "input_chars": 32.3125, "ratio": 0.8102164158617108, "valid_fraction": 1.0, "judged": true }, { "n": 64, "base_ir": 274198.7421875, "head_ir": 236175.15625, "input_chars": 102.203125, "ratio": 0.8613283721356422, "valid_fraction": 1.0, "judged": true }, { "n": 256, "base_ir": 274198.7421875, "head_ir": 236175.15625, "input_chars": 102.203125, "ratio": 0.8613283721356422, "valid_fraction": 1.0, "judged": true }, { "n": 1024, "base_ir": 274198.7421875, "head_ir": 236175.15625, "input_chars": 102.203125, "ratio": 0.8613283721356422, "valid_fraction": 1.0, "judged": true }, { "n": 4096, "base_ir": 274198.7421875, "head_ir": 236175.15625, "input_chars": 102.203125, "ratio": 0.8613283721356422, "valid_fraction": 1.0, "judged": true } ], "slope_base": 0.5923527015562182, "slope_head": 0.6515939880377071, "top_inputs": [ { "args": [ "==1.*", "1.506.415.175.72.249" ], "cpu_ratio": 2.628, "base_ir": 121382.37, "head_ir": 97872.16, "ir_ratio": 0.8063128113250714, "outcome": "ok" }, { "args": [ "==1.*", "1.121.121.121.300.801" ], "cpu_ratio": 2.481, "base_ir": 122245.69, "head_ir": 98735.48, "ir_ratio": 0.8076806634246164, "outcome": "ok" } ], "error": null, "verdict": "faster", "worst_ratio": 0.8613283721356422, "worst_search": { "args": [ "==1.*", "1.121.121.121.300.801" ], "cpu_ratio": 2.481, "base_ir": 122245.69, "head_ir": 98735.48, "ir_ratio": 0.8076806634246164, "outcome": "ok" } } ], "seconds": { "plan": 18.25, "screen": 98.83, "full": 126.6, "total": 243.68 }, "ladder_validity": "backfilled 2026-10-11 (scripts/backtest/backfill_ladder_validity.py)" } } } ] }, { "id": "#1115", "pr": 1115, "title": "perf: skip hash property", "base": "e839d73030b410d4156e6a496249fbdb1e498fab", "head": "baa9ea8c7fa773fc4039d090bc7ec6452b860ec6", "merged": "2026-03-11", "after_ultra_release": false, "known": [], "targets": [ { "target": "packaging.version:Version", "run_id": "bt-1115-cc92fd", "image": "bw-pair:c5e7d6dc99b58cc8", "usd": 0.0757, "harvested": 80, "arms": { "hypothesis": { "classes": [], "known": [], "seconds": 9.6, "error": null }, "mutations": { "classes": [], "known": [], "seconds": 8.1, "error": null }, "breakwater": { "classes": [], "known": [], "seconds": 64.4, "error": null }, "llm-plan": { "classes": [], "known": [], "seconds": 2.7, "error": null } }, "findings": [], "perf": { "slower_families": [], "worst_valid_ratio": null, "worst_family": null, "speedup_ratio_small": 0.9976252888532934, "report": { "run_id": "bt-1115-cc92fd", "target": "packaging.version:Version", "image": "bw-pair:c5e7d6dc99b58cc8", "backend": "docker", "families": [ { "id": "F1", "kind": "claimed_speedup", "family": "many simple numeric versions, each hashed once (e.g. inserted into a set)", "why": "The new __hash__ adds a cache‑check and an attribute write on the *first* hash. When each Version object is hashed only once (e.g. building a set of many distinct versions) the extra check and attribute assignment make the new version slower.", "size_meaning": "n = number of distinct simple versions (each “1.”) to create and hash", "make_source": "def make(n):\n \"\"\"Return a list of argument lists for Version(\"1.\") with i from 0 to n-1.\"\"\"\n return [[f\"1.{i}\"] for i in range(min(n, 8))]", "example_args": [ [ "1.0" ], [ "1.1" ], [ "1.2" ] ], "origin": "plan", "screen_ratio": 1.0, "valid_fraction": 1.0, "error": null }, { "id": "F2", "kind": "claimed_speedup", "family": "many distinct simple version strings, each hashed once (cold cache)", "why": "The new version adds a lazy `_key_cache` attribute and an extra attribute write on the first `__hash__` call. For workloads that create many distinct Version objects and hash each exactly once, the extra attribute write and cache allocation add overhead compared with the old property‑based `_key` which was computed on‑the‑fly without extra attribute handling.", "size_meaning": "n = number of distinct version strings generated; each string is of the form \"1.{i}\" with i from 1..n.", "make_source": "def make(n):\n # n distinct simple versions, each hashed once (cold cache)\n # return up to 8 argument lists; each list contains a single version string.\n # To make cost grow with n we return a single list containing n versions;\n # the benchmark will iterate over that list, creating and hashing each version.\n return [[f\"1.{i}\" for i in range(1, n + 1)]]", "example_args": [], "origin": "plan", "screen_ratio": 1.0000526636823914, "valid_fraction": 0.0, "error": null }, { "id": "F3", "kind": "suspected_slowdown", "family": "cold_parse_hash", "why": "First call to __hash__ on a freshly parsed Version must compute and cache the comparison key; the new lazy‑cache adds work on the cold path, so the first hash is slower than the old direct hash(self._key) path.", "size_meaning": "Number of dot‑separated release segments in the version string (e.g. \"1.2.3…\"); larger n makes key construction costlier.", "make_source": "\ndef make(n):\n \"\"\"Return a list of argument lists for Version(version_string).\n Each version string has n dot‑separated numeric segments.\"\"\"\n # generate a few distinct version strings, each growing with n\n versions = []\n for i in range(1, min(8, n)+1):\n segs = [str(i + j) for j in range(n)]\n versions.append([\".\".join(segs)])\n return versions", "example_args": [ [ "1.2.3" ], [ "1.2.3.4.5" ] ], "origin": "plan", "screen_ratio": 1.0, "valid_fraction": 1.0, "error": null }, { "id": "F4", "kind": "claimed_speedup", "family": "versions with many dot‑separated release segments, each hashed once", "why": "Building the comparison key (_cmpkey) walks the whole release tuple. The new __hash__ builds that key on the first hash and stores it; if the object is hashed only once the extra cache‑check and attribute write dominate the cost.", "size_meaning": "n = number of release segments (release tuple length) – each version gets a distinct length up to n (capped at 8)", "make_source": "def make(n):\n \"\"\"Return versions like \"0.1.2...k-1a1.post2.dev3+local\" with k from 1..n (max 8).\"\"\"\n out = []\n for k in range(1, min(n, 8) + 1):\n release = \".\".join(str(i) for i in range(k))\n out.append([f\"{release}a1.post2.dev3+local\"])\n return out", "example_args": [ [ "0a1.post2.dev3+local" ], [ "0.1a1.post2.dev3+local" ], [ "0.1.2a1.post2.dev3+local" ] ], "origin": "plan", "screen_ratio": 1.0009866591092775, "valid_fraction": 1.0, "error": null }, { "id": "F5", "kind": "claimed_speedup", "family": "many distinct complex version strings (epoch, many release segments, pre, post, dev, long local) hashed once each", "why": "Building the comparison key (`_cmpkey`) for a version with many components creates a larger tuple (epoch, many release ints, pre, post, dev, long local tuple). The new lazy cache still has to build that tuple on the first hash, plus allocate and store the cache attribute. For a cold‑hash workload with many large keys the extra allocation and attribute write makes the head version slower than the old property which built the same tuple but did not store it.", "size_meaning": "n = number of release segments and also number of local‑version segments; both grow linearly with n, making the key tuple size O(n).", "make_source": "\ndef make(n):\n # Build a single version string whose release segment has n parts and whose local part has n parts.\n # Example for n=3: \"1!1.2.3a1.post1.dev1+local1.local2.local3\"\n release = \".\".join(str(i) for i in range(1, n + 1))\n local = \".\".join(f\"local{i}\" for i in range(1, n + 1))\n version = f\"1!{release}a1.post1.dev1+{local}\"\n # Return one argument list containing that single (large) version string.\n return [[version]]", "example_args": [], "origin": "plan", "screen_ratio": 0.9997723542307461, "valid_fraction": 1.0, "error": null }, { "id": "F6", "kind": "claimed_speedup", "family": "warm_hash", "why": "The change adds a lazy _key_cache so that repeated hashing of the same Version object reuses the cached key. The new benchmark \"Version hash (warm cache)\" exercises this hot path.", "size_meaning": "Number of distinct Version objects created once and then hashed repeatedly; larger n means more objects to keep warm in the cache.", "make_source": "\ndef make(n):\n \"\"\"Return a list of argument lists for Version(version_string).\n Creates n distinct simple versions (e.g. \"1\", \"1.2\", …).\"\"\"\n versions = []\n for i in range(1, min(8, n)+1):\n versions.append([\".\".join(str(j) for j in range(1, i+1))])\n return versions", "example_args": [ [ "1" ], [ "1.2" ], [ "1.2.3" ] ], "origin": "plan", "screen_ratio": 0.9999948754233724, "valid_fraction": 1.0, "error": null }, { "id": "R1", "kind": "suspected_slowdown", "family": "Invalid version strings that nevertheless contain indicator characters (e.g. a trailing dot followed by a letter). The fast‑path still does the full scan before the regex fails.", "why": "Even though the version is invalid, the extra disjoint check runs; the regex then fails. This is the single allowed invalid family.", "size_meaning": "Number of dotted segments before the malformed tail.", "make_source": "def make(n):\n # \"1.\" repeated n times then an alphabetic tail – not a valid PEP‑440 version\n version = \"1.\" * n + \"x\"\n return [(version,)]", "example_args": [], "origin": "archive", "screen_ratio": 0.9997872842485278, "valid_fraction": 0.0, "error": null }, { "id": "R2", "kind": "suspected_slowdown", "family": "invalid strings that look simple (only digits and dots) but contain a non‑numeric segment, e.g. \"1.2.x\"", "why": "The indicator set is disjoint, so the fast path is entered. The code then executes `tuple(map(int, version.strip().split('.')))`, which raises `ValueError` on the bad segment; the exception is caught and re‑raised as `InvalidVersion`. Exception handling is considerably slower than the original regex‑only failure path.", "size_meaning": "n = number of leading numeric segments before the bad segment; larger n makes the split and the failed `int` conversion traverse more items before the exception.", "make_source": "def make(n):\n # build \"1.1.1....1.x\" with n numeric parts then a non‑numeric \"x\"\n good = \".\".join(\"1\" for _ in range(n))\n version = f\"{good}.x\"\n return [[version]]", "example_args": [], "origin": "archive", "screen_ratio": 0.9997872842485278, "valid_fraction": 0.0, "error": null }, { "id": "R3", "kind": "suspected_slowdown", "family": "Each numeric chunk increases digit count linearly while keeping total chunks fixed", "why": "The fast‑path’s `isdisjoint` test iterates over every character. For a long dotted release the string length grows with n, so the extra scan adds linear overhead before the (already linear) regex match.", "size_meaning": "Number of dotted release segments (plus the trailing pre‑release).", "make_source": "def make(n):\n chunks = []\n for i in range(1, n+1):\n chunks.append(str(i).zfill(i))\n version = '.'.join(chunks) + 'c5'\n return [(version,)]", "example_args": [], "origin": "archive", "screen_ratio": 0.9999364727193282, "valid_fraction": 1.0, "error": "ladder inputs exceed 2000000 chars at n=4096; ladder stops here" }, { "id": "R4", "kind": "suspected_slowdown", "family": "Single repeated segment \"999.\" n times plus prerelease c5", "why": "The fast‑path’s `isdisjoint` test iterates over every character. For a long dotted release the string length grows with n, so the extra scan adds linear overhead before the (already linear) regex match.", "size_meaning": "Number of dotted release segments (plus the trailing pre‑release).", "make_source": "def make(n):\n release = \".\".join([\"999\"] * n)\n version = f\"{release}c5\"\n return [(version,)]", "example_args": [], "origin": "archive", "screen_ratio": 0.9999221019517024, "valid_fraction": 1.0, "error": null } ], "findings": [ { "family_id": "F5", "kind": "claimed_speedup", "family": "many distinct complex version strings (epoch, many release segments, pre, post, dev, long local) hashed once each", "why": "Building the comparison key (`_cmpkey`) for a version with many components creates a larger tuple (epoch, many release ints, pre, post, dev, long local tuple). The new lazy cache still has to build that tuple on the first hash, plus allocate and store the cache attribute. For a cold‑hash workload with many large keys the extra allocation and attribute write makes the head version slower than the old property which built the same tuple but did not store it.", "origin": "plan", "screen_ratio": 0.9997723542307461, "valid_fraction": 1.0, "ladder": [ { "n": 2, "base_ir": 48026.59, "head_ir": 48066.65, "input_chars": 36.0, "ratio": 1.0008341212649077, "valid_fraction": 1.0, "judged": true }, { "n": 4, "base_ir": 58990.115, "head_ir": 59020.175, "input_chars": 54.0, "ratio": 1.000509576901147, "valid_fraction": 1.0, "judged": true }, { "n": 16, "base_ir": 130763.9375, "head_ir": 130734.16964285714, "input_chars": 176.0, "ratio": 0.9997723542307461, "valid_fraction": 1.0, "judged": true }, { "n": 64, "base_ir": 419840.53571428574, "head_ir": 418843.53571428574, "input_chars": 704.0, "ratio": 0.9976252888532934, "valid_fraction": 1.0, "judged": true }, { "n": 256, "base_ir": 1623862.1666666667, "head_ir": 1621278.1666666667, "input_chars": 3130.0, "ratio": 0.9984087319397901, "valid_fraction": 1.0, "judged": true }, { "n": 1024, "base_ir": 6443715.5, "head_ir": 6430213.0, "input_chars": 13164.0, "ratio": 0.9979045474617866, "valid_fraction": 1.0, "judged": true }, { "n": 4096, "base_ir": 26352310.5, "head_ir": 26351940.0, "input_chars": 59244.0, "ratio": 0.9999859405117437, "valid_fraction": 1.0, "judged": true } ], "slope_base": 0.860100832730107, "slope_head": 0.8598747865000547, "top_inputs": [], "error": null, "verdict": "neutral", "worst_ratio": 1.0008341212649077, "worst_search": null } ], "seconds": { "plan": 615.43, "screen": 175.75, "full": 82.23, "total": 873.41 }, "ladder_validity": "backfilled 2026-10-11 (scripts/backtest/backfill_ladder_validity.py)" } }, "image_rescored": "bw-pair:b75c39f11bbb1fbf" } ] }, { "id": "#1116", "pr": 1116, "title": "perf: Overhaul `_cmpkey` to remove use of custom objects", "base": "baa9ea8c7fa773fc4039d090bc7ec6452b860ec6", "head": "f2bbd4f578644865bc5cb2534768e46563ee7f66", "merged": "2026-03-11", "after_ultra_release": false, "known": [ { "id": "old-pickles-fail-to-load", "match": { "probe": "P4" }, "fixed_by": 1163, "status": "verified" } ], "targets": [ { "target": "packaging.version:Version", "run_id": "bt-1116-754b0a", "image": "bw-pair:127ca274fc70935b", "usd": 0.1098, "harvested": 80, "arms": { "hypothesis": { "classes": [], "known": [], "seconds": 5.7, "error": null }, "mutations": { "classes": [ "P4|base->head load|loads|exc:builtins.ModuleNotFoundError" ], "known": [ "old-pickles-fail-to-load" ], "seconds": 2.0, "error": null }, "breakwater": { "classes": [ "P4|base->head load|loads|exc:builtins.ModuleNotFoundError" ], "known": [ "old-pickles-fail-to-load" ], "seconds": 685.1, "error": null }, "llm-plan": { "classes": [ "P4|base->head load|loads|exc:builtins.ModuleNotFoundError" ], "known": [ "old-pickles-fail-to-load" ], "seconds": 6.5, "error": null } }, "findings": [ { "signature": "P4|base->head load|loads|exc:builtins.ModuleNotFoundError", "probe": "P4", "detail": "base->head load", "args": [ [ "0" ] ], "base": "loads", "head": { "status": "exc", "exc": "builtins.ModuleNotFoundError" }, "found_by": [ "seed", "tests", "mutation:whitespace-outer", "mutation:huge-number", "mutation:case", "mutation:equivalents", "mutation:edges", "mutation:separators", "batch" ] } ], "perf": { "slower_families": [], "worst_valid_ratio": null, "worst_family": null, "speedup_ratio_small": 0.9928424865375168, "report": { "run_id": "bt-1116-754b0a", "target": "packaging.version:Version", "image": "bw-pair:127ca274fc70935b", "backend": "docker", "families": [ { "id": "F1", "kind": "claimed_speedup", "family": "pure numeric versions (only digits and dots, no epoch, pre‑release, post‑release, dev or local parts). The new fast‑path in ``Version.__init__`` detects the ``_SIMPLE_VERSION_INDICATORS`` set and parses the release tuple with a cheap ``split('.')`` + ``map(int, …)`` plus a pre‑computed ``_STABLE_SUFFIX`` key, avoiding the regex and the full ``_cmpkey`` tuple construction.", "why": "The old code always fell through to the regex and built a full ``_cmpkey`` tuple (including three singleton ``Infinity``/``NegativeInfinity`` objects) even for a plain ``1.2.3``. The new fast‑path returns a pre‑computed constant suffix and skips the regex entirely, so cost grows only with the number of dot‑separated segments (splitting and ``int`` conversion).", "size_meaning": "n = number of dot‑separated numeric segments (e.g. \"1.2.3\" → n=3). Cost grows roughly linearly with n because of ``split('.')`` and ``int`` conversion.", "make_source": "def make(n):\n # n >= 1\n seg = '.'.join(['1'] * n)\n return [[seg]]", "example_args": [ [ [ "1" ] ], [ [ "1.2" ] ], [ [ "1.2.3.4.5" ] ] ], "origin": "plan", "screen_ratio": 0.9976638095744293, "valid_fraction": 1.0, "error": null }, { "id": "F2", "kind": "claimed_speedup", "family": "Simple numeric versions consisting only of digits and dots (e.g. \"1.2.3.4.5\")", "why": "The new code adds a fast‑path at the top of Version.__init__ that detects a version string containing only characters in the set '.0123456789'. Such strings bypass the regular‑expression entirely and are parsed with a cheap `split('.')` + `map(int, …)`. The old version always ran the full verbose regex, so pure‑numeric versions are now parsed and keyed in O(N) time with no regex overhead.", "size_meaning": "Number of dot‑separated numeric components (n). Cost grows linearly with n because of the split and integer conversion.", "make_source": "\ndef make(n):\n # produce n versions with 1,2,…,n dot‑separated components\n return [[\".\".join(str(i) for i in range(1, k+1))] for k in range(1, n+1)]", "example_args": [ [ 1 ], [ 1.2 ] ], "origin": "plan", "screen_ratio": 0.9999857642282433, "valid_fraction": 1.0, "error": null }, { "id": "F3", "kind": "claimed_speedup", "family": "Simple numeric versions (only digits and dots, e.g. \"1.2.3.4.5\")", "why": "The new code adds a fast‑path in `__init__` (``_SIMPLE_VERSION_INDICATORS.issuperset(version)``) and a fast‑path in ``_cmpkey`` that returns a pre‑computed ``_STABLE_SUFFIX`` tuple when a version has no epoch, pre‑release, post‑release, dev‑release or local part. Both paths avoid the regex and avoid building the full comparison key, giving a measurable speed‑up for pure numeric versions.", "size_meaning": "Number of dot‑separated numeric segments in the version string (n segments → string length ≈ 2·n).", "make_source": "def make(n):\n # produce n versions with 1,2,...,n dot‑separated segments\n return [[\".\".join(str(i) for i in range(1, seg+1))] for seg in range(1, n+1)]", "example_args": [ [ 1 ], [ 1.2 ] ], "origin": "plan", "screen_ratio": 0.9999857642282433, "valid_fraction": 1.0, "error": null }, { "id": "F4", "kind": "suspected_slowdown", "family": "versions with a long local version part (many dot/underscore/hyphen separated alphanumeric segments). The new ``_cmpkey`` builds a ``CmpLocalType`` tuple of tuples ``(segment_int, \"\")`` for each local segment, whereas the old code built a similar tuple but filled with the singleton ``NegativeInfinity`` objects. Creating many small 2‑tuples per segment adds allocation overhead that grows with the number of local segments.", "why": "Both versions allocate a tuple per local segment, but the new code allocates a 2‑tuple of ``(int, \"\")`` for each segment while the old code reused the singleton ``NegativeInfinity`` objects inside a 2‑tuple. The extra integer allocation and tuple creation per segment can make long local versions slower.", "size_meaning": "n = number of local segments after the '+' (e.g. \"1.0+loc0.loc1.loc2\" → n=3). Cost grows roughly linearly with n because a new 2‑tuple is built for each segment.", "make_source": "def make(n):\n # n >= 1\n local = '.'.join(f'loc{i}' for i in range(n))\n return [[f'1.0+{local}']]", "example_args": [ [ [ "1.0+loc0" ] ], [ [ "1.0+loc0.loc1.loc2" ] ] ], "origin": "plan", "screen_ratio": 0.9975426741371929, "valid_fraction": 1.0, "error": null }, { "id": "F5", "kind": "suspected_slowdown", "family": "Versions with a long local‑version segment containing many dot/underscore/hyphen separated alphanumeric parts (e.g. \"1.0+local.part1.part2….partN\")", "why": "The new `_cmpkey` builds `cmp_local` as a tuple of 2‑element tuples `(seg, \"\")` for numeric parts and `(_LOCAL_STR_RANK, seg)` for alphabetic parts. A long local version therefore allocates a tuple‑of‑tuples whose size grows with the number of local segments. The old code built a similar structure but used the singleton `NegativeInfinity` objects for the string parts; the new code creates a fresh `(-1, seg)` tuple for every segment on every key construction. Because the key is cached, the extra allocation happens once per `Version` instance, but for workloads that create many distinct version objects with long local parts the extra tuple allocations can make object construction noticeably slower than the old singleton‑object approach.", "size_meaning": "Number of dot/underscore/hyphen separated parts in the local version (n). Cost grows roughly linearly with n due to per‑segment tuple allocation.", "make_source": "\ndef make(n):\n # generate versions with 1..n local segments after a simple release\n base = \"1.0\"\n parts = [\"p\" + str(i) for i in range(1, n+1)]\n # join with '.' to make many segments\n return [[base + \"+\" + \".\".join(parts[:k])] for k in range(1, n+1)]", "example_args": [], "origin": "plan", "screen_ratio": 0.999508068388664, "valid_fraction": 1.0, "error": null }, { "id": "F6", "kind": "suspected_slowdown", "family": "Versions with many local‑version segments (e.g. \"1.0+local.part1.part2…partN\")", "why": "The new ``_cmpkey`` builds a ``cmp_local`` tuple of ``(int, str)`` pairs for every local segment (``(seg, \"\")`` for numeric parts, ``(-1, seg)`` for alphanumeric parts). The old code built a tuple of custom ``NegativeInfinity`` objects via a generator expression. Creating many small ``(int, str)`` tuples and then comparing them element‑wise in Python can be slower than the old single‑object ``NegativeInfinity`` comparison, especially when the local part contains many segments, because tuple‑of‑tuple allocation and element‑wise integer comparison adds overhead.", "size_meaning": "Number of dot‑separated segments in the local version part (n segments → local string length grows roughly linearly with n).", "make_source": "\ndef make(n):\n # produce n versions with 1..n local segments like \"1.0+loc.a.b.c...\"\n base = \"1.0+\"\n segs = [\"part\" + str(i) for i in range(1, n+1)]\n return [[base + \".\".join(segs[:k])] for k in range(1, n+1)]", "example_args": [], "origin": "plan", "screen_ratio": 0.9981533666493461, "valid_fraction": 1.0, "error": null }, { "id": "R1", "kind": "suspected_slowdown", "family": "Invalid version strings that nevertheless contain indicator characters (e.g. a trailing dot followed by a letter). The fast‑path still does the full scan before the regex fails.", "why": "Even though the version is invalid, the extra disjoint check runs; the regex then fails. This is the single allowed invalid family.", "size_meaning": "Number of dotted segments before the malformed tail.", "make_source": "def make(n):\n # \"1.\" repeated n times then an alphabetic tail – not a valid PEP‑440 version\n version = \"1.\" * n + \"x\"\n return [(version,)]", "example_args": [], "origin": "archive", "screen_ratio": 0.9999353964547297, "valid_fraction": 0.0, "error": null }, { "id": "R2", "kind": "suspected_slowdown", "family": "invalid strings that look simple (only digits and dots) but contain a non‑numeric segment, e.g. \"1.2.x\"", "why": "The indicator set is disjoint, so the fast path is entered. The code then executes `tuple(map(int, version.strip().split('.')))`, which raises `ValueError` on the bad segment; the exception is caught and re‑raised as `InvalidVersion`. Exception handling is considerably slower than the original regex‑only failure path.", "size_meaning": "n = number of leading numeric segments before the bad segment; larger n makes the split and the failed `int` conversion traverse more items before the exception.", "make_source": "def make(n):\n # build \"1.1.1....1.x\" with n numeric parts then a non‑numeric \"x\"\n good = \".\".join(\"1\" for _ in range(n))\n version = f\"{good}.x\"\n return [[version]]", "example_args": [], "origin": "archive", "screen_ratio": 0.9999353964547297, "valid_fraction": 0.0, "error": null }, { "id": "R3", "kind": "suspected_slowdown", "family": "Each numeric chunk increases digit count linearly while keeping total chunks fixed", "why": "The fast‑path’s `isdisjoint` test iterates over every character. For a long dotted release the string length grows with n, so the extra scan adds linear overhead before the (already linear) regex match.", "size_meaning": "Number of dotted release segments (plus the trailing pre‑release).", "make_source": "def make(n):\n chunks = []\n for i in range(1, n+1):\n chunks.append(str(i).zfill(i))\n version = '.'.join(chunks) + 'c5'\n return [(version,)]", "example_args": [], "origin": "archive", "screen_ratio": 0.9861231270856489, "valid_fraction": 1.0, "error": "ladder inputs exceed 2000000 chars at n=4096; ladder stops here" }, { "id": "R4", "kind": "suspected_slowdown", "family": "Single repeated segment \"999.\" n times plus prerelease c5", "why": "The fast‑path’s `isdisjoint` test iterates over every character. For a long dotted release the string length grows with n, so the extra scan adds linear overhead before the (already linear) regex match.", "size_meaning": "Number of dotted release segments (plus the trailing pre‑release).", "make_source": "def make(n):\n release = \".\".join([\"999\"] * n)\n version = f\"{release}c5\"\n return [(version,)]", "example_args": [], "origin": "archive", "screen_ratio": 0.9984320379303969, "valid_fraction": 1.0, "error": null } ], "findings": [ { "family_id": "F1", "kind": "claimed_speedup", "family": "pure numeric versions (only digits and dots, no epoch, pre‑release, post‑release, dev or local parts). The new fast‑path in ``Version.__init__`` detects the ``_SIMPLE_VERSION_INDICATORS`` set and parses the release tuple with a cheap ``split('.')`` + ``map(int, …)`` plus a pre‑computed ``_STABLE_SUFFIX`` key, avoiding the regex and the full ``_cmpkey`` tuple construction.", "why": "The old code always fell through to the regex and built a full ``_cmpkey`` tuple (including three singleton ``Infinity``/``NegativeInfinity`` objects) even for a plain ``1.2.3``. The new fast‑path returns a pre‑computed constant suffix and skips the regex entirely, so cost grows only with the number of dot‑separated segments (splitting and ``int`` conversion).", "origin": "plan", "screen_ratio": 0.9976638095744293, "valid_fraction": 1.0, "ladder": [ { "n": 2, "base_ir": 9342.63, "head_ir": 9275.76, "input_chars": 7.0, "ratio": 0.9928424865375168, "valid_fraction": 1.0, "judged": true }, { "n": 4, "base_ir": 11976.72, "head_ir": 11909.91, "input_chars": 11.0, "ratio": 0.9944216780554276, "valid_fraction": 1.0, "judged": true }, { "n": 16, "base_ir": 28597.84, "head_ir": 28531.03, "input_chars": 35.0, "ratio": 0.9976638095744293, "valid_fraction": 1.0, "judged": true }, { "n": 64, "base_ir": 95282.90666666666, "head_ir": 95278.24, "input_chars": 131.0, "ratio": 0.9999510230446371, "valid_fraction": 1.0, "judged": true }, { "n": 256, "base_ir": 364865.0, "head_ir": 363862.0, "input_chars": 515.0, "ratio": 0.9972510380551711, "valid_fraction": 1.0, "judged": true }, { "n": 1024, "base_ir": 1418834.2222222222, "head_ir": 1418406.2222222222, "input_chars": 2051.0, "ratio": 0.999698343898606, "valid_fraction": 1.0, "judged": true }, { "n": 4096, "base_ir": 5626370.0, "head_ir": 5625581.0, "input_chars": 8195.0, "ratio": 0.9998597674877407, "valid_fraction": 1.0, "judged": true } ], "slope_base": 0.9165631072819177, "slope_head": 0.917425921530889, "top_inputs": [], "error": null, "verdict": "neutral", "worst_ratio": 0.9999510230446371, "worst_search": null } ], "seconds": { "plan": 19.18, "screen": 143.51, "full": 69.49, "total": 232.18 }, "ladder_validity": "backfilled 2026-10-11 (scripts/backtest/backfill_ladder_validity.py)" } }, "image_rescored": "bw-pair:ee5fda034cdd7462" } ] }, { "id": "#1118", "pr": 1118, "title": "perf: Cache `Version` hash value in dedicated slot", "base": "f2bbd4f578644865bc5cb2534768e46563ee7f66", "head": "1676a10cdad3f6183dc7f738d3a30950c67199a4", "merged": "2026-03-13", "after_ultra_release": false, "known": [ { "id": "hash-cache-copy", "match": { "probe": "P3" }, "fixed_by": 1242, "status": "candidate" } ], "targets": [ { "target": "packaging.version:Version", "run_id": "bt-1118-e98cb3", "image": "bw-pair:dcd905da4078ada1", "usd": 0.0698, "harvested": 80, "arms": { "hypothesis": { "classes": [], "known": [], "seconds": 5.1, "error": null }, "mutations": { "classes": [], "known": [], "seconds": 1.7, "error": null }, "breakwater": { "classes": [], "known": [], "seconds": 50.9, "error": null }, "llm-plan": { "classes": [], "known": [], "seconds": 3.1, "error": null } }, "findings": [], "perf": { "slower_families": [], "worst_valid_ratio": null, "worst_family": null, "speedup_ratio_small": 1.0071858172125294, "report": { "run_id": "bt-1118-e98cb3", "target": "packaging.version:Version", "image": "bw-pair:dcd905da4078ada1", "backend": "docker", "families": [ { "id": "F1", "kind": "claimed_speedup", "family": "repeated_hashing_same_versions", "why": "The new version adds a _hash_cache slot and caches the hash value on first __hash__ call. When the same Version objects are hashed repeatedly (e.g., used repeatedly as dict/set keys), the cached hash avoids recomputing the comparison key and its hash, giving a speedup.", "size_meaning": "Number of distinct version strings created; each will be hashed many times in the benchmark.", "make_source": "def make(n):\n \"\"\"Return a list of n single-element argument lists for Version(\\\"1.0.{i}\\\").\"\"\"\n return [[f\"1.0.{i}\"] for i in range(n)]", "example_args": [], "origin": "plan", "screen_ratio": 1.007308072115251, "valid_fraction": 1.0, "error": null }, { "id": "F2", "kind": "claimed_speedup", "family": "repeated_hash_same_versions", "why": "The change adds a cached ``_hash_cache`` attribute and makes ``__hash__`` return the cached value on subsequent calls. Workloads that create a modest number of ``Version`` objects and then hash them repeatedly (e.g. using versions as dictionary keys or set members) should see a speed‑up because the hash is computed only once per object.", "size_meaning": "Number of distinct version objects created; each will be hashed many times in the benchmark.", "make_source": "\ndef make(n):\n # Produce *n* distinct but simple version strings (fast‑path parser).\n # Each call to the constructor receives a single‑element list.\n base = \"1.0.\"\n return [[base + str(i)] for i in range(1, n + 1)]", "example_args": [], "origin": "plan", "screen_ratio": 1.0072914676390057, "valid_fraction": 1.0, "error": null }, { "id": "F3", "kind": "claimed_speedup", "family": "hash_reuse", "why": "The change adds a cached ``_hash_cache`` on ``Version``. Objects that are hashed repeatedly (e.g. used as dict keys or put into a set and looked‑up many times) now pay the hashing cost only once; subsequent ``__hash__`` calls return the cached integer.", "size_meaning": "n = number of distinct version strings inserted into a set/dict and then looked‑up n times each", "make_source": "\ndef make(n):\n # generate n distinct, simple version strings (fast‑path)\n rng = random.Random(0xC0FFEE)\n versions = [f\"{rng.randint(0,9)}.{rng.randint(0,9)}.{rng.randint(0,9)}\" for _ in range(n)]\n # each version will be inserted into a set and then looked‑up n times\n # the benchmark will create Version objects from these strings\n return [[v] for v in versions]", "example_args": [ [ "0.1.2" ], [ "3.4.5" ] ], "origin": "plan", "screen_ratio": 1.0073977151604938, "valid_fraction": 1.0, "error": null }, { "id": "F4", "kind": "suspected_slowdown", "family": "cold_sort_many_versions", "why": "The new version adds a _hash_cache attribute that is initialised to None in __init__, from_parts and __replace__. The cold‑sort benchmark creates many Version objects, clears _key_cache (and now also clears _hash_cache) before each sort, and then sorts them using only rich comparisons (__lt__). The extra attribute write on construction and the extra attribute clear in the benchmark add overhead while providing no benefit because __hash__ is never called during sorting.", "size_meaning": "Number of distinct version strings to parse and sort; larger n means more objects and a longer sort.", "make_source": "def make(n):\n \"\"\"Return a list of n single-element argument lists for Version(\\\"1.{i}.0\\\").\"\"\"\n return [[f\"1.{i}.0\"] for i in range(n)]", "example_args": [], "origin": "plan", "screen_ratio": 1.007308072115251, "valid_fraction": 1.0, "error": null }, { "id": "F5", "kind": "suspected_slowdown", "family": "many_simple_versions", "why": "The new ``_hash_cache`` slot adds an extra attribute write in ``__init__`` (``self._hash_cache = None``) and in ``__new__`` paths (``from_parts``, ``__replace__``). When a program creates many short, pure‑numeric versions (the fast‑path parser that splits on ‘.’) but never hashes them, the extra attribute write and the larger ``__slots__`` layout add overhead to every construction.", "size_meaning": "Number of simple ``X.Y.Z`` version strings parsed via the fast‑path.", "make_source": "\ndef make(n):\n # Simple dot‑separated numeric versions – fast path in ``__init__``.\n return [[f\"{i}.{i+1}.{i+2}\"] for i in range(1, n + 1)]", "example_args": [], "origin": "plan", "screen_ratio": 1.0070170316240725, "valid_fraction": 1.0, "error": null }, { "id": "F6", "kind": "suspected_slowdown", "family": "simple_version_construction", "why": "The fast‑path parser now has to initialise an extra ``_hash_cache`` slot (and set it to ``None``) on every ``Version`` constructed from a simple ``X.Y.Z`` string. When many simple versions are created but never hashed, the extra attribute initialisation and the extra ``self._hash_cache = None`` store add overhead.", "size_meaning": "n = number of simple ``X.Y.Z`` version strings to construct", "make_source": "\ndef make(n):\n rng = random.Random(0xBEEF)\n # simple dot‑only versions hit the fast‑path parser\n return [[f\"{rng.randint(0,9)}.{rng.randint(0,9)}.{rng.randint(0,9)}\"] for _ in range(n)]", "example_args": [ [ "1.2.3" ], [ "4.5.6" ] ], "origin": "plan", "screen_ratio": 1.0073969334403987, "valid_fraction": 1.0, "error": null }, { "id": "R1", "kind": "suspected_slowdown", "family": "Invalid version strings that nevertheless contain indicator characters (e.g. a trailing dot followed by a letter). The fast‑path still does the full scan before the regex fails.", "why": "Even though the version is invalid, the extra disjoint check runs; the regex then fails. This is the single allowed invalid family.", "size_meaning": "Number of dotted segments before the malformed tail.", "make_source": "def make(n):\n # \"1.\" repeated n times then an alphabetic tail – not a valid PEP‑440 version\n version = \"1.\" * n + \"x\"\n return [(version,)]", "example_args": [], "origin": "archive", "screen_ratio": 1.0018008324919971, "valid_fraction": 0.0, "error": null }, { "id": "R2", "kind": "suspected_slowdown", "family": "invalid strings that look simple (only digits and dots) but contain a non‑numeric segment, e.g. \"1.2.x\"", "why": "The indicator set is disjoint, so the fast path is entered. The code then executes `tuple(map(int, version.strip().split('.')))`, which raises `ValueError` on the bad segment; the exception is caught and re‑raised as `InvalidVersion`. Exception handling is considerably slower than the original regex‑only failure path.", "size_meaning": "n = number of leading numeric segments before the bad segment; larger n makes the split and the failed `int` conversion traverse more items before the exception.", "make_source": "def make(n):\n # build \"1.1.1....1.x\" with n numeric parts then a non‑numeric \"x\"\n good = \".\".join(\"1\" for _ in range(n))\n version = f\"{good}.x\"\n return [[version]]", "example_args": [], "origin": "archive", "screen_ratio": 1.0018008324919971, "valid_fraction": 0.0, "error": null }, { "id": "R3", "kind": "suspected_slowdown", "family": "Each numeric chunk increases digit count linearly while keeping total chunks fixed", "why": "The fast‑path’s `isdisjoint` test iterates over every character. For a long dotted release the string length grows with n, so the extra scan adds linear overhead before the (already linear) regex match.", "size_meaning": "Number of dotted release segments (plus the trailing pre‑release).", "make_source": "def make(n):\n chunks = []\n for i in range(1, n+1):\n chunks.append(str(i).zfill(i))\n version = '.'.join(chunks) + 'c5'\n return [(version,)]", "example_args": [], "origin": "archive", "screen_ratio": 1.0013444352855538, "valid_fraction": 1.0, "error": "ladder inputs exceed 2000000 chars at n=4096; ladder stops here" }, { "id": "R4", "kind": "suspected_slowdown", "family": "Single repeated segment \"999.\" n times plus prerelease c5", "why": "The fast‑path’s `isdisjoint` test iterates over every character. For a long dotted release the string length grows with n, so the extra scan adds linear overhead before the (already linear) regex match.", "size_meaning": "Number of dotted release segments (plus the trailing pre‑release).", "make_source": "def make(n):\n release = \".\".join([\"999\"] * n)\n version = f\"{release}c5\"\n return [(version,)]", "example_args": [], "origin": "archive", "screen_ratio": 1.0016451254429442, "valid_fraction": 1.0, "error": null } ], "findings": [ { "family_id": "F2", "kind": "claimed_speedup", "family": "repeated_hash_same_versions", "why": "The change adds a cached ``_hash_cache`` attribute and makes ``__hash__`` return the cached value on subsequent calls. Workloads that create a modest number of ``Version`` objects and then hash them repeatedly (e.g. using versions as dictionary keys or set members) should see a speed‑up because the hash is computed only once per object.", "origin": "plan", "screen_ratio": 1.0072914676390057, "valid_fraction": 1.0, "ladder": [ { "n": 2, "base_ir": 9706.2725, "head_ir": 9776.02, "input_chars": 9.0, "ratio": 1.0071858172125294, "valid_fraction": 1.0, "judged": true }, { "n": 4, "base_ir": 9572.155, "head_ir": 9642.1825, "input_chars": 9.0, "ratio": 1.0073157507374253, "valid_fraction": 1.0, "judged": true }, { "n": 16, "base_ir": 9600.262041284404, "head_ir": 9670.262041284404, "input_chars": 9.4375, "ratio": 1.0072914676390057, "valid_fraction": 1.0, "judged": true }, { "n": 64, "base_ir": 9736.495192307691, "head_ir": 9806.551682692309, "input_chars": 9.859375, "ratio": 1.0071952472630978, "valid_fraction": 1.0, "judged": true }, { "n": 256, "base_ir": 9736.495192307691, "head_ir": 9806.551682692309, "input_chars": 9.859375, "ratio": 1.0071952472630978, "valid_fraction": 1.0, "judged": true }, { "n": 1024, "base_ir": 9736.495192307691, "head_ir": 9806.551682692309, "input_chars": 9.859375, "ratio": 1.0071952472630978, "valid_fraction": 1.0, "judged": true }, { "n": 4096, "base_ir": 9736.495192307691, "head_ir": 9806.551682692309, "input_chars": 9.859375, "ratio": 1.0071952472630978, "valid_fraction": 1.0, "judged": true } ], "slope_base": null, "slope_head": null, "top_inputs": [], "error": null, "verdict": "neutral", "worst_ratio": 1.0073157507374253, "worst_search": null } ], "seconds": { "plan": 19.22, "screen": 73.03, "full": 33.86, "total": 126.12 }, "ladder_validity": "backfilled 2026-10-11 (scripts/backtest/backfill_ladder_validity.py)" } } } ] }, { "id": "#1147", "pr": 1147, "title": "perf: add __slots__ to Marker", "base": "04a7600a21febca084022d88c6684f1aedf81c8a", "head": "76cc6298254887e17d6b39df0425a4b6c2f1714a", "merged": "2026-04-05", "after_ultra_release": false, "known": [ { "id": "marker-pickle", "match": { "probe": "P4" }, "fixed_by": 1171, "status": "candidate" } ], "targets": [ { "target": "packaging.markers:Marker", "run_id": "bt-1147-8b7b5a", "image": "bw-pair:50aa962c18a1f597", "usd": 0.0376, "harvested": 80, "arms": { "hypothesis": { "classes": [], "known": [], "seconds": 3.8, "error": null }, "mutations": { "classes": [ "P4|base->head load|loads|exc:builtins.AttributeError" ], "known": [ "marker-pickle" ], "seconds": 1.6, "error": null }, "breakwater": { "classes": [ "P4|base->head load|loads|exc:builtins.AttributeError" ], "known": [ "marker-pickle" ], "seconds": 30.9, "error": null }, "llm-plan": { "classes": [ "P4|base->head load|loads|exc:builtins.AttributeError" ], "known": [ "marker-pickle" ], "seconds": 1.6, "error": null } }, "findings": [ { "signature": "P4|base->head load|loads|exc:builtins.AttributeError", "probe": "P4", "detail": "base->head load", "args": [ [ "\"b\" > \"a\"" ] ], "base": "loads", "head": { "status": "exc", "exc": "builtins.AttributeError" }, "found_by": [ "seed", "tests", "mutation:unicode-digits", "mutation:whitespace-outer", "mutation:huge-number", "mutation:equivalents", "mutation:underscores", "mutation:separators", "mutation:whitespace-inner", "mutation:case" ] } ], "perf": { "slower_families": [], "worst_valid_ratio": null, "worst_family": null, "speedup_ratio_small": 0.9818854740154223, "report": { "run_id": "bt-1147-8b7b5a", "target": "packaging.markers:Marker", "image": "bw-pair:50aa962c18a1f597", "backend": "docker", "families": [ { "id": "F1", "kind": "claimed_speedup", "family": "simple marker", "why": "The added __slots__ removes the per‑instance __dict__, so attribute writes (self._markers = ...) and reads in __init__ are faster. Repeated construction of many tiny marker objects exercises the fast path that __slots__ optimises.", "size_meaning": "number of Marker objects created", "make_source": "\ndef make(n):\n # a trivial, always‑valid marker expression\n base = \"python_version >= '3.8'\"\n # repeat the same construction n times (cost grows linearly with n)\n return [[base] for _ in range(min(n, 8))]", "example_args": [], "origin": "plan", "screen_ratio": 0.9955887083084731, "valid_fraction": 1.0, "error": null }, { "id": "F2", "kind": "claimed_speedup", "family": "creation and evaluation of many Marker objects", "why": "Adding __slots__ removes the per‑instance __dict__, so attribute access (self._markers) and object creation are slightly faster. The benefit grows with the number of Marker instances.", "size_meaning": "Number of Marker objects created and evaluated (n).", "make_source": "def make(n):\n markers = []\n for i in range(n):\n # a simple but valid marker expression\n markers.append([f'python_version >= \"3.{i%10}\"'])\n return markers", "example_args": [], "origin": "plan", "screen_ratio": 0.997861366519618, "valid_fraction": 1.0, "error": null }, { "id": "F3", "kind": "claimed_speedup", "family": "Simple marker strings (single comparison, no nesting)", "why": "The added __slots__ eliminates the per‑instance __dict__, making attribute access (self._markers) and object creation a little faster. This is the typical workload the change targets.", "size_meaning": "n = number of marker expressions concatenated with spaces (still a single top‑level marker)", "make_source": "\ndef make(n):\n # Produce n different simple markers like \"python_version >= '3.8'\"\n bases = [\"python_version >= '3.8'\", \"os_name == 'posix'\", \"extra == 'test'\"]\n # Cycle through the three bases to get n distinct markers\n return [[bases[i % len(bases)]] for i in range(n)]", "example_args": [], "origin": "plan", "screen_ratio": 0.9981582925614582, "valid_fraction": 1.0, "error": null }, { "id": "F4", "kind": "claimed_speedup", "family": "nested marker with many sub‑expressions", "why": "A deeply nested marker creates a larger _markers list, so the __init__ body runs more Python bytecode. Faster attribute access from __slots__ becomes measurable when the list is built and then stored.", "size_meaning": "nesting depth (number of 'or' clauses)", "make_source": "def make(n):\n # Build a marker like: a or (a or (a ... )) with depth n\n clause = \"python_version == '3.8'\"\n expr = clause\n for _ in range(n):\n expr = f\"({expr} or {clause})\"\n # cap at 8 items\n return [[expr] for _ in range(min(n, 8))]", "example_args": [], "origin": "plan", "screen_ratio": 0.9917744647906729, "valid_fraction": 1.0, "error": null }, { "id": "F5", "kind": "suspected_slowdown", "family": "invalid marker strings that raise InvalidMarker", "why": "When parsing fails the constructor raises InvalidMarker. The new __slots__ descriptor is still consulted during attribute assignment before the exception propagates, adding a tiny overhead on the error path.", "size_meaning": "Number of invalid marker strings attempted (n).", "make_source": "def make(n):\n return [[f'invalid{{i}}'] for i in range(n)]", "example_args": [], "origin": "plan", "screen_ratio": 0.9928937985410515, "valid_fraction": 0.0, "error": null }, { "id": "F6", "kind": "claimed_speedup", "family": "Nested marker expressions (and/or groups)", "why": "Even with nested lists the parser builds the same internal MarkerList structure; the only difference is the instance layout, which slots improve.", "size_meaning": "n = depth of nesting (each level adds an 'and' group)", "make_source": "def make(n):\n # Build a marker like: python_version >= '3.7' and (python_version >= '3.8' and (...))\n inner = \"python_version >= '3.7'\"\n for i in range(n):\n inner = f\"python_version >= '3.{7+i}' and ({inner})\"\n return [[inner]]", "example_args": [], "origin": "plan", "screen_ratio": 0.9818854740154223, "valid_fraction": 1.0, "error": null } ], "findings": [ { "family_id": "F6", "kind": "claimed_speedup", "family": "Nested marker expressions (and/or groups)", "why": "Even with nested lists the parser builds the same internal MarkerList structure; the only difference is the instance layout, which slots improve.", "origin": "plan", "screen_ratio": 0.9818854740154223, "valid_fraction": 1.0, "ladder": [ { "n": 2, "base_ir": 439095.395, "head_ir": 437372.455, "input_chars": 87.0, "ratio": 0.9960761601701608, "valid_fraction": 1.0, "judged": true }, { "n": 4, "base_ir": 757729.6616541353, "head_ir": 754800.5789473684, "input_chars": 148.0, "ratio": 0.9961343961375715, "valid_fraction": 1.0, "judged": true }, { "n": 16, "base_ir": 2731378.763157895, "head_ir": 2681901.1315789474, "input_chars": 520.0, "ratio": 0.9818854740154223, "valid_fraction": 1.0, "judged": true }, { "n": 64, "base_ir": 10588055.111111112, "head_ir": 10524852.555555556, "input_chars": 2008.0, "ratio": 0.9940307681729734, "valid_fraction": 1.0, "judged": true }, { "n": 256, "base_ir": 42019898.0, "head_ir": 42045322.0, "input_chars": 8123.0, "ratio": 1.0006050466852632, "valid_fraction": 1.0, "judged": true }, { "n": 1024, "base_ir": 75220486.5, "head_ir": 75024419.0, "input_chars": 32730.0, "ratio": 0.9973934295147108, "valid_fraction": 0.0, "judged": false }, { "n": 4096, "base_ir": 75034587.0, "head_ir": 75043897.5, "input_chars": 134106.0, "ratio": 1.000124082777986, "valid_fraction": 0.0, "judged": false } ], "slope_base": 1.005957060657232, "slope_head": 1.00682622973309, "top_inputs": [], "error": null, "verdict": "neutral", "worst_ratio": 1.0006050466852632, "worst_search": null } ], "seconds": { "plan": 9.6, "screen": 70.8, "full": 57.28, "total": 137.68 }, "ladder_validity": "backfilled 2026-10-11 (scripts/backtest/backfill_ladder_validity.py)" } }, "image_rescored": "bw-pair:e33c8d8e1355cf11" } ] }, { "id": "#1236", "pr": 1236, "title": "perf: simply work done in parse_tag a bit", "base": "e6cd4be75370b9730d70c7397ca2f873dae18f54", "head": "1529771b80cf6a327afdf33a8a0e7a96ac3f428f", "merged": "2026-06-09", "after_ultra_release": true, "known": [], "targets": [ { "target": "packaging.tags:parse_tag", "run_id": "bt-1236-1cca09", "image": "bw-pair:6d11476928904c1e", "usd": 0.0905, "harvested": 80, "arms": { "hypothesis": { "classes": [ "P2|outcome|exc:builtins.ValueError|exc:packaging.tags.InvalidTag|empty", "P2|outcome|exc:builtins.ValueError|exc:packaging.tags.InvalidTag|plain", "P2|outcome|exc:builtins.ValueError|exc:packaging.tags.InvalidTag|whitespace" ], "known": [], "seconds": 3.4, "error": null }, "mutations": { "classes": [ "P2|outcome|exc:builtins.ValueError|exc:packaging.tags.InvalidTag|control", "P2|outcome|exc:builtins.ValueError|exc:packaging.tags.InvalidTag|empty", "P2|outcome|exc:builtins.ValueError|exc:packaging.tags.InvalidTag|plain", "P2|outcome|exc:builtins.ValueError|exc:packaging.tags.InvalidTag|whitespace" ], "known": [], "seconds": 1.5, "error": null }, "breakwater": { "classes": [ "P2|outcome|exc:builtins.ValueError|exc:packaging.tags.InvalidTag|control", "P2|outcome|exc:builtins.ValueError|exc:packaging.tags.InvalidTag|empty", "P2|outcome|exc:builtins.ValueError|exc:packaging.tags.InvalidTag|long-digits", "P2|outcome|exc:builtins.ValueError|exc:packaging.tags.InvalidTag|non-ascii", "P2|outcome|exc:builtins.ValueError|exc:packaging.tags.InvalidTag|non-ascii+whitespace", "P2|outcome|exc:builtins.ValueError|exc:packaging.tags.InvalidTag|plain", "P2|outcome|exc:builtins.ValueError|exc:packaging.tags.InvalidTag|whitespace" ], "known": [], "seconds": 36.0, "error": null }, "llm-plan": { "classes": [ "P2|outcome|exc:builtins.ValueError|exc:packaging.tags.InvalidTag|empty", "P2|outcome|exc:builtins.ValueError|exc:packaging.tags.InvalidTag|plain" ], "known": [], "seconds": 1.5, "error": null } }, "findings": [ { "signature": "P2|outcome|exc:builtins.ValueError|exc:packaging.tags.InvalidTag|plain", "probe": "P2", "detail": "outcome", "args": [ [ "-" ] ], "base": { "status": "exc", "exc": "builtins.ValueError" }, "head": { "status": "exc", "exc": "packaging.tags.InvalidTag" }, "found_by": [ "A1/s2", "A3/s2", "A3/s1", "A4/s1", "A5/s1", "seed", "tests", "mutation:separators", "mutation:case", "mutation:signs", "mutation:equivalents", "batch" ] }, { "signature": "P2|outcome|exc:builtins.ValueError|exc:packaging.tags.InvalidTag|whitespace", "probe": "P2", "detail": "outcome", "args": [ [ "- " ] ], "base": { "status": "exc", "exc": "builtins.ValueError" }, "head": { "status": "exc", "exc": "packaging.tags.InvalidTag" }, "found_by": [ "A1/s2", "A3/s1", "A4/s1", "tests", "mutation:whitespace-outer", "mutation:whitespace-inner", "mutation:case", "mutation:signs", "batch" ] }, { "signature": "P2|outcome|exc:builtins.ValueError|exc:packaging.tags.InvalidTag|control", "probe": "P2", "detail": "outcome", "args": [ [ "-\u001b" ] ], "base": { "status": "exc", "exc": "builtins.ValueError" }, "head": { "status": "exc", "exc": "packaging.tags.InvalidTag" }, "found_by": [ "A1/s2", "A4/s1", "mutation:edges", "batch" ] }, { "signature": "P2|outcome|exc:builtins.ValueError|exc:packaging.tags.InvalidTag|empty", "probe": "P2", "detail": "outcome", "args": [ [ "" ] ], "base": { "status": "exc", "exc": "builtins.ValueError" }, "head": { "status": "exc", "exc": "packaging.tags.InvalidTag" }, "found_by": [ "A3/s1", "seed" ] }, { "signature": "P2|outcome|exc:builtins.ValueError|exc:packaging.tags.InvalidTag|non-ascii", "probe": "P2", "detail": "outcome", "args": [ [ "-񠞝" ] ], "base": { "status": "exc", "exc": "builtins.ValueError" }, "head": { "status": "exc", "exc": "packaging.tags.InvalidTag" }, "found_by": [ "mutation:unicode-digits", "batch" ] }, { "signature": "P2|outcome|exc:builtins.ValueError|exc:packaging.tags.InvalidTag|long-digits", "probe": "P2", "detail": "outcome", "args": [ [ "py1000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000-" ] ], "base": { "status": "exc", "exc": "builtins.ValueError" }, "head": { "status": "exc", "exc": "packaging.tags.InvalidTag" }, "found_by": [ "mutation:huge-number" ] }, { "signature": "P2|outcome|exc:builtins.ValueError|exc:packaging.tags.InvalidTag|non-ascii+whitespace", "probe": "P2", "detail": "outcome", "args": [ [ "-…" ] ], "base": { "status": "exc", "exc": "builtins.ValueError" }, "head": { "status": "exc", "exc": "packaging.tags.InvalidTag" }, "found_by": [ "mutation:whitespace-outer", "batch" ] } ], "perf": { "slower_families": [ { "id": "F5", "family": "tiny_tag", "verdict": "slower", "ladder": [ 1.0876598578049792, 1.087079721621475, 1.0895078805489675, 1.0895078805489675, 1.0895078805489675, 1.0895078805489675, 1.0895078805489675 ], "origin": "plan" } ], "worst_valid_ratio": 1.0895078805489675, "worst_family": "tiny_tag", "speedup_ratio_small": 0.8386139725679109, "report": { "run_id": "bt-1236-1cca09", "target": "packaging.tags:parse_tag", "image": "bw-pair:6d11476928904c1e", "backend": "docker", "families": [ { "id": "F1", "kind": "claimed_speedup", "family": "Valid tags with many dot-separated components in each of the three fields (interpreters, ABIs, platforms). The new implementation splits each component only once, while the old version split each component three times.", "why": "The new version builds component_parts with a single split per component and reuses the lists for validation and the Cartesian product, avoiding repeated splits.", "size_meaning": "n controls the number of dot-separated items per component; total tags generated = n^3, and string length grows ~ O(n).", "make_source": "\ndef make(n):\n # Generate n interpreters, n abis, n platforms\n interpreters = '.'.join(f'py{i}' for i in range(2, 2 + n))\n abis = '.'.join(f'abi{i}' for i in range(n))\n platforms = '.'.join(f'plat{i}' for i in range(n))\n tag = f'{interpreters}-{abis}-{platforms}'\n return [[tag]]", "example_args": [], "origin": "plan", "screen_ratio": 0.9510078429461927, "valid_fraction": 1.0, "error": null }, { "id": "F2", "kind": "suspected_slowdown", "family": "invalid", "why": "The new implementation splits all three \"-\" components up‑front (building a list of three split lists) before checking for empty components. The old code checks components one‑by‑one and aborts on the first empty component, splitting only the components examined so far. For an invalid tag whose first component contains an empty \".\" segment while the other two components are very large (many \"..\" segments), the new version does three large splits before raising, whereas the old version does only one split before raising, making the new version slower on this invalid input family.", "size_meaning": "n controls the number of dot‑segments in the large second and third components (the first component always contains a single empty segment). Cost grows roughly linearly with n because splitting a long string is O(length).", "make_source": "\ndef make(n):\n # Produce up to 8 test cases, growing the size of the large components.\n out = []\n for i in range(1, min(n, 8) + 1):\n # First component: a short prefix, then an empty segment, then a short suffix.\n first = \".\".join([\"a\"] * i + [\"\"] + [\"b\"] * i)\n # Second and third components: many dot‑segments to make splitting expensive.\n second = \".\".join([\"c\"] * (i * 20))\n third = \".\".join([\"d\"] * (i * 20))\n tag = \"-\".join([first, second, third])\n out.append([tag]) # parse_tag takes a single positional string argument\n return out", "example_args": [ [ "a..b-c.c.c.c.c.c.c.c.c.c.c.c.c.c.c.c.c.c.c.c-d.d.d.d.d.d.d.d.d.d.d.d.d.d.d.d.d.d.d.d" ], [ "a.a..b.b-c.c.c.c.c.c.c.c.c.c.c.c.c.c.c.c.c.c.c.c.c.c.c.c.c.c.c.c.c.c.c.c.c.c.c.c.c.c.c.c-d.d.d.d.d.d.d.d.d.d.d.d.d.d.d.d.d.d.d.d.d.d.d.d.d.d.d.d.d.d.d.d.d.d.d.d.d.d.d.d" ] ], "origin": "plan", "screen_ratio": 2.3016767160534233, "valid_fraction": 0.0, "error": null }, { "id": "F3", "kind": "suspected_slowdown", "family": "many_components", "why": "The change splits each of the three tag components once and then builds the Cartesian product with a generator expression. The old code re‑splits the ABI and platform strings for every interpreter, so the work grows as O(I·A·P). When all three components contain many dot‑separated parts the new version does far fewer string splits and should be faster.", "size_meaning": "n = number of dot‑separated items in *each* of the three components (interpreters, ABIs, platforms). The tag length grows roughly as 3·n and the number of generated Tag objects grows as n³, which makes the old implementation repeatedly split strings.", "make_source": "def make(n):\n # build n interpreters, n ABIs, n platforms\n interps = '.'.join(f'py{i}' for i in range(1, n+1))\n abis = '.'.join(f'abi{i}' for i in range(1, n+1))\n plats = '.'.join(f'plat{i}' for i in range(1, n+1))\n tag = f'{interps}-{abis}-{plats}'\n return [[tag]]", "example_args": [ [ [ "py1.py2.py3-abi1.abi2.abi3-plat1.plat2.plat3" ] ] ], "origin": "plan", "screen_ratio": 0.9509821683353116, "valid_fraction": 1.0, "error": null }, { "id": "F4", "kind": "claimed_speedup", "family": "Valid tags with a large number of dot-separated interpreters but only one ABI and one platform. This exercises the interpreter component split path.", "why": "The new version splits the interpreter component once and reuses the list for validation and product, while the old version split it three times.", "size_meaning": "n controls the number of interpreters; string length grows ~ O(n).", "make_source": "def make(n):\n interpreters = '.'.join(f'py{i}' for i in range(2, 2 + n))\n tag = f'{interpreters}-none-any'\n return [[tag]]", "example_args": [], "origin": "plan", "screen_ratio": 0.8644430204545949, "valid_fraction": 1.0, "error": null }, { "id": "F5", "kind": "suspected_slowdown", "family": "tiny_tag", "why": "For the common case of a single‑component tag (e.g. \"py3-none-any\") the new implementation builds a temporary list of the three split components and then creates a frozenset via a generator expression. The old version performed the three splits, a couple of tiny loops and a few `set.add` calls – a lower constant‑time overhead. When the tag is tiny the extra list allocation and generator machinery dominates, making the new code slightly slower.", "size_meaning": "n controls how many *independent* tiny tags are parsed in one benchmark run. Each element of the returned list is a separate call to `parse_tag`; the total work therefore grows linearly with n.", "make_source": "def make(n):\n tag = 'py3-none-any'\n # Return up to 8 argument lists; each list contains the single tag string.\n return [[tag] for _ in range(min(n, 8))]", "example_args": [ [ [ "py3-none-any" ] ] ], "origin": "plan", "screen_ratio": 1.0895078805489675, "valid_fraction": 1.0, "error": null }, { "id": "F6", "kind": "suspected_slowdown", "family": "Invalid tags containing many hyphens (more than the expected two). The new implementation splits all hyphen-separated components by '.' before attempting to unpack three parts, doing extra work before the unpacking error; the old version failed fast on the unpack.", "why": "Extra list comprehension and per-component dot-splits are performed before the ValueError from unpacking.", "size_meaning": "n controls the number of extra hyphen-separated components; work grows O(n).", "make_source": "def make(n):\n # Create a tag with n+2 components separated by hyphens\n components = ['comp'] * (n + 2)\n tag = '-'.join(components)\n return [[tag]]", "example_args": [], "origin": "plan", "screen_ratio": 2.6160501871827986, "valid_fraction": 0.0, "error": null }, { "id": "R1", "kind": "suspected_slowdown", "family": "invalid", "why": "The new implementation splits the whole tag on '-' into a list of all components first, then iterates over every component to validate emptiness and order. For a malformed tag containing many '-' separators (far more than the three expected components), the head version does O(N) work before raising, whereas the base version fails fast during the three‑way unpacking.", "size_meaning": "N = number of dash‑separated components in the malformed tag.", "make_source": "\ndef make(n):\n # Produce a single malformed tag with n dash‑separated components.\n # Each component is a non‑empty string so the emptiness check passes,\n # but the tag is invalid because it does not have exactly three components.\n tag = '-'.join(['x'] * n)\n return [[tag]]", "example_args": [ [ 10 ] ], "origin": "archive", "screen_ratio": 3.0402521518175996, "valid_fraction": 0.0, "error": null }, { "id": "R2", "kind": "suspected_slowdown", "family": "invalid-many-dash-components", "why": "The new implementation splits the tag on '-' once and then splits every resulting component on '.' (building a list of lists) before unpacking the three expected components. For an invalid tag containing many '-' separators this does O(k) extra work (splitting each of the k components on '.' and allocating a list of lists) before the unpacking fails, whereas the old code only split once on '-' and then immediately failed on unpacking. Thus the new version is slower on this invalid input family.", "size_meaning": "number of dash-separated components in the tag", "make_source": "def make(n):\n # generate n tags with 1, 2, ..., n dash-separated components (all invalid)\n return [[\"-\".join([\"x\"] * i)] for i in range(1, n + 1)]", "example_args": [ [ "x" ], [ "x-x" ], [ "x-x-x" ] ], "origin": "archive", "screen_ratio": 2.2280638076724872, "valid_fraction": 0.0625, "error": null }, { "id": "R3", "kind": "suspected_slowdown", "family": "invalid tags that contain many empty dot‑components (e.g. \"a..b..c-d..e-f..g\")", "why": "The new code still has to split the tag, then for each empty component it joins the parts back together to produce the error message (\"a..b..c\"). The old code raised the error directly on the original component string without a join. For a tag with many empty components the extra joins make the new version slower, though such inputs are invalid and rare in practice.", "size_meaning": "n is the number of empty components inserted in each of the three fields; total length grows linearly with n.", "make_source": "def make(n):\n # Build a tag like \"a..b..c-d..e-f..g\" with n empty parts per field\n def field(base, n):\n parts = [base]\n for i in range(n):\n parts.append('') # empty component\n parts.append(base + str(i))\n return \".\".join(parts)\n tag = f\"{field('i', n)}-{field('a', n)}-{field('p', n)}\"\n return [[tag]]", "example_args": [ [ "i..i0..i1-a..a0..a1-p..p0..p1" ] ], "origin": "archive", "screen_ratio": 1.8192857118926613, "valid_fraction": 0.0, "error": null }, { "id": "F5.1", "kind": "suspected_slowdown", "family": "tiny_tag", "why": "For the common case of a single‑component tag (e.g. \"py3-none-any\") the new implementation builds a temporary list of the three split components and then creates a frozenset via a generator expression. The old version performed the three splits, a couple of tiny loops and a few `set.add` calls – a lower constant‑time overhead. When the tag is tiny the extra list allocation and generator machinery dominates, making the new code slightly slower.", "size_meaning": "n controls how many *independent* tiny tags are parsed in one benchmark run. Each element of the returned list is a separate call to `parse_tag`; the total work therefore grows linearly with n.", "make_source": "def make(n):\n tag = \"cp310-win_amd64\"\n return [[tag] for _ in range(min(n, 8))]", "example_args": [ [ [ "py3-none-any" ] ] ], "origin": "expand:F5", "screen_ratio": 1.4164485340859396, "valid_fraction": 0.0, "error": null }, { "id": "R1.1", "kind": "suspected_slowdown", "family": "excess-dashes-with-suffix", "why": "The new implementation splits the whole tag on '-' into a list of all components first, then iterates over every component to validate emptiness and order. For a malformed tag containing many '-' separators (far more than the three expected components), the head version does O(N) work before raising, whereas the base version fails fast during the three‑way unpacking.", "size_meaning": "N = number of dash‑separated components in the malformed tag.", "make_source": "# Variant using excess dashes followed by an extra suffix after all components\ndef make(n):\n # Creates tags like 'x-x-...-x-extra' where there are n+1 components\n # ensuring at least four parts (more than expected three); splitting all\n # components triggers O(N) work before validation fails due to wrong count.\n tag = '-'.join(['x'] * n + ['extra'])\n return [[tag]]", "example_args": [ [ 10 ] ], "origin": "expand:R1", "screen_ratio": 3.1067030746123607, "valid_fraction": 0.0, "error": null }, { "id": "R1.2", "kind": "suspected_slowdown", "family": "excess-dashes-with-nonempty-mid-empty-trailing", "why": "The new implementation splits the whole tag on '-' into a list of all components first, then iterates over every component to validate emptiness and order. For a malformed tag containing many '-' separators (far more than the three expected components), the head version does O(N) work before raising, whereas the base version fails fast during the three‑way unpacking.", "size_meaning": "N = number of dash‑separated components in the malformed tag.", "make_source": "# Variant mixing empty and non-empty parts to stress emptiness checks after split\ndef make(n):\n # Builds a tag starting with an empty component, then n non-empty 'x',\n # ending with another empty component via trailing dash;\n # this forces the new implementation to check emptiness on all n+2 parts\n # before realizing the structure is invalid (not exactly three non-empty).\n tag = '-'.join([''] + ['x'] * n + [''])\n return [[tag]]", "example_args": [ [ 10 ] ], "origin": "expand:R1", "screen_ratio": 2.613360123613242, "valid_fraction": 0.0, "error": null }, { "id": "R1.3", "kind": "suspected_slowdown", "family": "excess-dashes-within-quoted-like-segments", "why": "The new implementation splits the whole tag on '-' into a list of all components first, then iterates over every component to validate emptiness and order. For a malformed tag containing many '-' separators (far more than the three expected components), the head version does O(N) work before raising, whereas the base version fails fast during the three‑way unpacking.", "size_meaning": "N = number of dash‑separated components in the malformed tag.", "make_source": "# Variant using components that look like quoted strings to avoid early rejection\ndef make(n):\n # Each component is a non‑empty lowercase alphabetic string mimicking\n # interpreter/os/abi parts; the tag has many such parts separated by '-',\n # so full split-and-validate runs in O(N) before length check fails.\n tag = '-'.join(['cp' + str(i % 10) for i in range(n)])\n return [[tag]]}]}", "example_args": [ [ 10 ] ], "origin": "expand:R1", "screen_ratio": null, "valid_fraction": null, "error": "make(n) rejected: syntax error: unmatched '}' (line 8)" } ], "findings": [ { "family_id": "F5", "kind": "suspected_slowdown", "family": "tiny_tag", "why": "For the common case of a single‑component tag (e.g. \"py3-none-any\") the new implementation builds a temporary list of the three split components and then creates a frozenset via a generator expression. The old version performed the three splits, a couple of tiny loops and a few `set.add` calls – a lower constant‑time overhead. When the tag is tiny the extra list allocation and generator machinery dominates, making the new code slightly slower.", "origin": "plan", "screen_ratio": 1.0895078805489675, "valid_fraction": 1.0, "ladder": [ { "n": 2, "base_ir": 16129.96, "head_ir": 17543.91, "input_chars": 16.0, "ratio": 1.0876598578049792, "valid_fraction": 1.0, "judged": true }, { "n": 4, "base_ir": 15995.1275, "head_ir": 17387.97875, "input_chars": 16.0, "ratio": 1.087079721621475, "valid_fraction": 1.0, "judged": true }, { "n": 16, "base_ir": 15897.758152173914, "head_ir": 17320.732789855072, "input_chars": 16.0, "ratio": 1.0895078805489675, "valid_fraction": 1.0, "judged": true }, { "n": 64, "base_ir": 15897.758152173914, "head_ir": 17320.732789855072, "input_chars": 16.0, "ratio": 1.0895078805489675, "valid_fraction": 1.0, "judged": true }, { "n": 256, "base_ir": 15897.758152173914, "head_ir": 17320.732789855072, "input_chars": 16.0, "ratio": 1.0895078805489675, "valid_fraction": 1.0, "judged": true }, { "n": 1024, "base_ir": 15897.758152173914, "head_ir": 17320.732789855072, "input_chars": 16.0, "ratio": 1.0895078805489675, "valid_fraction": 1.0, "judged": true }, { "n": 4096, "base_ir": 15897.758152173914, "head_ir": 17320.732789855072, "input_chars": 16.0, "ratio": 1.0895078805489675, "valid_fraction": 1.0, "judged": true } ], "slope_base": null, "slope_head": null, "top_inputs": [ { "args": [ [ [ "py3-none-any" ] ] ], "cpu_ratio": 1.008, "base_ir": 7549.305, "head_ir": 7624.985, "ir_ratio": 1.0100247638689919, "outcome": "exc:builtins.AttributeError" }, { "args": [ [ [ "py3-none-any" ], [ "py3-none-any" ], [ "py3-none-any" ] ] ], "cpu_ratio": 1.007, "base_ir": 7527.55, "head_ir": 7592.185, "ir_ratio": 1.0085864590736695, "outcome": "exc:builtins.AttributeError" } ], "error": null, "verdict": "slower", "worst_ratio": 1.0895078805489675, "worst_search": { "args": [ [ [ "py3-none-any" ] ] ], "cpu_ratio": 1.008, "base_ir": 7549.305, "head_ir": 7624.985, "ir_ratio": 1.0100247638689919, "outcome": "exc:builtins.AttributeError" } }, { "family_id": "R1.1", "kind": "suspected_slowdown", "family": "excess-dashes-with-suffix", "why": "The new implementation splits the whole tag on '-' into a list of all components first, then iterates over every component to validate emptiness and order. For a malformed tag containing many '-' separators (far more than the three expected components), the head version does O(N) work before raising, whereas the base version fails fast during the three‑way unpacking.", "origin": "expand:R1", "screen_ratio": 3.1067030746123607, "valid_fraction": 0.0, "ladder": [ { "n": 2, "base_ir": 15889.665, "head_ir": 17376.835, "input_chars": 13.0, "ratio": 1.093593540203648, "valid_fraction": 1.0, "judged": true }, { "n": 4, "base_ir": 7849.01, "head_ir": 14443.885, "input_chars": 17.0, "ratio": 1.840217428694829, "valid_fraction": 0.0, "judged": false }, { "n": 16, "base_ir": 9054.8, "head_ir": 28130.575, "input_chars": 41.0, "ratio": 3.1067030746123607, "valid_fraction": 0.0, "judged": false }, { "n": 64, "base_ir": 15231.132867132867, "head_ir": 83797.96503496503, "input_chars": 137.0, "ratio": 5.5017552381669494, "valid_fraction": 0.0, "judged": false }, { "n": 256, "base_ir": 39175.10526315789, "head_ir": 339275.1052631579, "input_chars": 521.0, "ratio": 8.660477182743607, "valid_fraction": 0.0, "judged": false }, { "n": 1024, "base_ir": 124539.0, "head_ir": 1494332.0, "input_chars": 2057.0, "ratio": 11.998907972602959, "valid_fraction": 0.0, "judged": false }, { "n": 4096, "base_ir": 460302.0, "head_ir": 6160055.0, "input_chars": 8201.0, "ratio": 13.382637920321876, "valid_fraction": 0.0, "judged": false } ], "slope_base": null, "slope_head": null, "top_inputs": [], "error": null, "verdict": "slower", "worst_ratio": 1.093593540203648, "worst_search": null }, { "family_id": "R1", "kind": "suspected_slowdown", "family": "invalid", "why": "The new implementation splits the whole tag on '-' into a list of all components first, then iterates over every component to validate emptiness and order. For a malformed tag containing many '-' separators (far more than the three expected components), the head version does O(N) work before raising, whereas the base version fails fast during the three‑way unpacking.", "origin": "archive", "screen_ratio": 3.0402521518175996, "valid_fraction": 0.0, "ladder": [ { "n": 2, "base_ir": 8221.01, "head_ir": 11548.18, "input_chars": 7.0, "ratio": 1.4047154789983225, "valid_fraction": 0.0, "judged": true }, { "n": 4, "base_ir": 7583.01, "head_ir": 12922.885, "input_chars": 11.0, "ratio": 1.7041893654366802, "valid_fraction": 0.0, "judged": true }, { "n": 16, "base_ir": 8736.8, "head_ir": 26562.075, "input_chars": 35.0, "ratio": 3.0402521518175996, "valid_fraction": 0.0, "judged": true }, { "n": 64, "base_ir": 14928.013333333334, "head_ir": 82049.84666666666, "input_chars": 131.0, "ratio": 5.496367455906166, "valid_fraction": 0.0, "judged": true }, { "n": 256, "base_ir": 38872.10526315789, "head_ir": 340123.1052631579, "input_chars": 515.0, "ratio": 8.749798935781309, "valid_fraction": 0.0, "judged": true }, { "n": 1024, "base_ir": 122970.77777777778, "head_ir": 1493054.7777777778, "input_chars": 2051.0, "ratio": 12.141541305657983, "valid_fraction": 0.0, "judged": true }, { "n": 4096, "base_ir": 454419.0, "head_ir": 6160135.0, "input_chars": 8195.0, "ratio": 13.55606829820056, "valid_fraction": 0.0, "judged": true } ], "slope_base": 0.5706152082632364, "slope_head": 0.9090072420921559, "top_inputs": [], "error": null, "verdict": "complexity-regression", "worst_ratio": 13.55606829820056, "worst_search": null }, { "family_id": "F6", "kind": "suspected_slowdown", "family": "Invalid tags containing many hyphens (more than the expected two). The new implementation splits all hyphen-separated components by '.' before attempting to unpack three parts, doing extra work before the unpacking error; the old version failed fast on the unpack.", "why": "Extra list comprehension and per-component dot-splits are performed before the ValueError from unpacking.", "origin": "plan", "screen_ratio": 2.6160501871827986, "valid_fraction": 0.0, "ladder": [ { "n": 2, "base_ir": 8425.01, "head_ir": 13821.885, "input_chars": 23.0, "ratio": 1.6405778746850153, "valid_fraction": 0.0, "judged": true }, { "n": 4, "base_ir": 8968.8, "head_ir": 16666.075, "input_chars": 33.0, "ratio": 1.8582279680670772, "valid_fraction": 0.0, "judged": true }, { "n": 16, "base_ir": 12706.83, "head_ir": 33241.705, "input_chars": 93.0, "ratio": 2.6160501871827986, "valid_fraction": 0.0, "judged": true }, { "n": 64, "base_ir": 28994.677966101695, "head_ir": 100513.67796610169, "input_chars": 333.0, "ratio": 3.466625084907458, "valid_fraction": 0.0, "judged": true }, { "n": 256, "base_ir": 93720.8, "head_ir": 404232.8, "input_chars": 1293.0, "ratio": 4.313159938882297, "valid_fraction": 0.0, "judged": true }, { "n": 1024, "base_ir": 340124.0, "head_ir": 1750131.6666666667, "input_chars": 5133.0, "ratio": 5.145569458981626, "valid_fraction": 0.0, "judged": true }, { "n": 4096, "base_ir": 1325820.5, "head_ir": 7159074.0, "input_chars": 20493.0, "ratio": 5.399730959055166, "valid_fraction": 0.0, "judged": true } ], "slope_base": 0.7516826171353747, "slope_head": 0.9315157780472996, "top_inputs": [ { "args": [ [ "comp-comp-comp-comp-comp" ] ], "cpu_ratio": 2.743, "base_ir": 7545.185, "head_ir": 7621.085, "ir_ratio": 1.0100593954952728, "outcome": "exc:builtins.AttributeError" }, { "args": [ [ "comp-comp-comp-comp-comp-comp-comp" ] ], "cpu_ratio": 2.73, "base_ir": 7545.185, "head_ir": 7621.085, "ir_ratio": 1.0100593954952728, "outcome": "exc:builtins.AttributeError" } ], "error": null, "verdict": "complexity-regression", "worst_ratio": 5.399730959055166, "worst_search": { "args": [ [ "comp-comp-comp-comp-comp" ] ], "cpu_ratio": 2.743, "base_ir": 7545.185, "head_ir": 7621.085, "ir_ratio": 1.0100593954952728, "outcome": "exc:builtins.AttributeError" } }, { "family_id": "F4", "kind": "claimed_speedup", "family": "Valid tags with a large number of dot-separated interpreters but only one ABI and one platform. This exercises the interpreter component split path.", "why": "The new version splits the interpreter component once and reuses the list for validation and product, while the old version split it three times.", "origin": "plan", "screen_ratio": 0.8644430204545949, "valid_fraction": 1.0, "ladder": [ { "n": 2, "base_ir": 25441.665, "head_ir": 25345.54, "input_chars": 20.0, "ratio": 0.996221748851736, "valid_fraction": 1.0, "judged": true }, { "n": 4, "base_ir": 42636.455, "head_ir": 39845.73, "input_chars": 28.0, "ratio": 0.9345460357808828, "valid_fraction": 1.0, "judged": true }, { "n": 16, "base_ir": 147761.665, "head_ir": 127731.54, "input_chars": 84.0, "ratio": 0.8644430204545949, "valid_fraction": 1.0, "judged": true }, { "n": 64, "base_ir": 570791.7213114754, "head_ir": 481001.737704918, "input_chars": 324.0, "ratio": 0.8426922110918987, "valid_fraction": 1.0, "judged": true }, { "n": 256, "base_ir": 2261242.846153846, "head_ir": 1896309.8461538462, "input_chars": 1442.0, "ratio": 0.8386139725679109, "valid_fraction": 1.0, "judged": true }, { "n": 1024, "base_ir": 9324739.333333334, "head_ir": 7867361.666666667, "input_chars": 6076.0, "ratio": 0.8437084818599755, "valid_fraction": 1.0, "judged": true }, { "n": 4096, "base_ir": 37920471.5, "head_ir": 32084014.0, "input_chars": 27580.0, "ratio": 0.8460868953066684, "valid_fraction": 1.0, "judged": true } ], "slope_base": 0.9978931663266013, "slope_head": 0.9794672676641476, "top_inputs": [], "error": null, "verdict": "neutral", "worst_ratio": 0.996221748851736, "worst_search": null } ], "seconds": { "plan": 22.35, "screen": 121.82, "full": 290.74, "total": 434.92 }, "ladder_validity": "backfilled 2026-10-11 (scripts/backtest/backfill_ladder_validity.py)" } } } ] }, { "id": "#1250", "pr": 1250, "title": "perf(markers): cache the default environment", "base": "4c663c9ef5e1ec4f64b01ea9a61c981890a56407", "head": "cf100d20e09c270cb6e19e72edbd0d40728de501", "merged": "2026-07-10", "after_ultra_release": true, "known": [], "targets": [ { "target": "packaging.markers:Marker#evaluate", "run_id": "bt-1250-af3916", "image": "bw-pair:f8fd127aa38ba6e4", "usd": 0.0411, "harvested": 80, "arms": { "hypothesis": { "classes": [], "known": [], "seconds": 8.2, "error": null }, "mutations": { "classes": [], "known": [], "seconds": 4.9, "error": null }, "breakwater": { "classes": [], "known": [], "seconds": 60.4, "error": null }, "llm-plan": { "classes": [], "known": [], "seconds": 3.2, "error": null } }, "findings": [], "perf": { "slower_families": [], "worst_valid_ratio": null, "worst_family": null, "speedup_ratio_small": null, "report": { "run_id": "bt-1250-af3916", "target": "packaging.markers:Marker#evaluate", "image": "bw-pair:f8fd127aa38ba6e4", "backend": "docker", "families": [ { "id": "F1", "kind": "claimed_speedup", "family": "caching", "why": "The default marker environment is now built once per process via @functools.cache and a fresh shallow copy is returned on each call, avoiding repeated construction of the environment from sys/platform/sys modules on every evaluation.", "size_meaning": "speedup", "make_source": "def make(n):\n m = min(n, 8)\n args = []\n for i in range(1, m + 1):\n args.append([i])\n return args", "example_args": [], "origin": "plan", "screen_ratio": 0.9989569145338487, "valid_fraction": 0.0, "error": null }, { "id": "F2", "kind": "claimed_speedup", "family": "caching_of_process_environment", "why": "The default marker environment is now built once per process and cached via @functools.cache, avoiding repeated calls to platform/sys/os APIs and string formatting on every Marker.evaluate call.", "size_meaning": "large", "make_source": "\ndef make(n):\n bases = [{}, {}]\n for _ in range(n - 2):\n bases.append({\n \"key\": string.ascii_lowercase[random.randint(0, 25)],\n \"val\": random.randint(0, 100),\n \"flag\": random.choice([True, False]),\n })\n return bases", "example_args": [ [ {} ], [ {} ] ], "origin": "plan", "screen_ratio": 1.0024371973285786, "valid_fraction": 0.0, "error": null }, { "id": "F3", "kind": "claimed_speedup", "family": "cached_default_environment", "why": "The base environment for the current Python process is now built once per process and cached via @functools.cache (_cached_default_environment). Subsequent calls to default_environment() return a shallow copy of the cached immutable mapping, eliminating repeated construction of the environment from platform/sys/os data on every Marker.evaluate call.", "size_meaning": "Performance improvement on repeated marker evaluations (e.g., during dependency resolution or metadata evaluation) because the default environment is built once per process instead of on every call.", "make_source": "def make(n):\n limits = [1, 2, 4, 8]\n k = min(n, len(limits))\n return list(itertools.islice(itertools.combinations(limits, r) for r in range(1, k + 1), 0, 1))", "example_args": [], "origin": "plan", "screen_ratio": null, "valid_fraction": null, "error": "make(n) rejected: syntax error: Generator expression must be parenthesized (line 5)" }, { "id": "F4", "kind": "suspected_slowdown", "family": "import_overhead", "why": "Importing functools adds a module import at module import time, which adds a tiny import-time cost on interpreter start-up.", "size_meaning": "slowdown", "make_source": "def make(n):\n # Import cost grows with n because string length grows and\n # repeated module attribute lookups add up.\n s = string.ascii_letters[:n % 52]\n return [[s]]", "example_args": [], "origin": "plan", "screen_ratio": 1.0134387367325253, "valid_fraction": 0.0, "error": null }, { "id": "F5", "kind": "suspected_slowdown", "family": "allocation_overhead_per_evaluation", "why": "default_environment() now returns a shallow copy (dict(_cached_default_environment())) on every call, allocating a new dict each evaluation; previously the environment dict was built from scratch, which also allocated but also performed string formatting work.", "size_meaning": "small", "make_source": "def make(n):\n return [\n [{}],\n [{}],\n [{}, {}],\n [{}, {}],\n [{}, {}, {}],\n [{}, {}, {}],\n [{}, {}, {}, {}],\n [{}, {}, {}, {}],\n ]", "example_args": [ [ {} ], [ {} ] ], "origin": "plan", "screen_ratio": 0.9994832614761187, "valid_fraction": 0.0, "error": null }, { "id": "F6", "kind": "suspected_slowdown", "family": "import_functools_overhead", "why": "An additional import of functools is introduced to enable caching. While negligible, the import adds a tiny import-time cost and an extra function call indirection through the cache wrapper.", "size_meaning": "Negligible runtime overhead; only a one‑time import cost at module import time.", "make_source": "def make(n):\n if n == 0:\n return []\n return [\n [\"import functools\"],\n [\"from functools import lru_cache\"],\n [f\"lru_cache({i})\"],\n [f\"lru_cache(maxsize={i})\"],\n [\"import functools; from functools import lru_cache\"],\n [\"import functools\", f\"lru_cache({i})\"],\n [\"from functools import lru_cache\", f\"lru_cache(maxsize={i})\"],\n [\"import functools\", \"from functools import lru_cache\", f\"lru_cache(maxsize={i})\"],\n ][:n]", "example_args": [], "origin": "plan", "screen_ratio": null, "valid_fraction": null, "error": "make(2): NameError: name 'i' is not defined" } ], "findings": [], "seconds": { "plan": 12.63, "screen": 51.16, "full": 0.0, "total": 63.79 }, "ladder_validity": "backfilled 2026-10-11 (scripts/backtest/backfill_ladder_validity.py)" } } } ] }, { "id": "#1253", "pr": 1253, "title": "perf(specifiers): keep range caches across canonicalization", "base": "26b89a05f6f404dbfd808f6ac60d744e8a5e0386", "head": "e64c20eb7a854c72710d4c962bc8a95f343230e6", "merged": "2026-06-16", "after_ultra_release": true, "known": [], "targets": [ { "target": "packaging.specifiers:SpecifierSet#contains", "run_id": "bt-1253-7b2670", "image": "bw-pair:92555005a9f709e0", "usd": 0.0991, "harvested": 80, "arms": { "hypothesis": { "classes": [], "known": [], "seconds": 5.2, "error": null }, "mutations": { "classes": [], "known": [], "seconds": 1.9, "error": null }, "breakwater": { "classes": [], "known": [], "seconds": 58.8, "error": null }, "llm-plan": { "classes": [], "known": [], "seconds": 1.5, "error": null } }, "findings": [], "perf": { "slower_families": [], "worst_valid_ratio": null, "worst_family": null, "speedup_ratio_small": 0.9885014930255579, "report": { "run_id": "bt-1253-7b2670", "target": "packaging.specifiers:SpecifierSet#contains", "image": "bw-pair:92555005a9f709e0", "backend": "docker", "families": [ { "id": "F1", "kind": "claimed_speedup", "family": "SpecifierSet with many comma‑separated specifiers; canonicalisation runs once during construction and then many contains() calls", "why": "The change removes two assignments (`self._is_unsatisfiable = None; self._ranges = None`) that were executed every time `_canonicalize_specs()` ran. The method is only executed once per instance (when `_canonicalized` flips from False to True), so the assignments were pure overhead. Removing them makes construction of a large SpecifierSet slightly faster.", "size_meaning": "Number of comma‑separated specifiers in the input string", "make_source": "def make(n):\n # n = number of specifiers, e.g. '>=1.0,>=2.0,...'\n specs = [f\">={i}.0\" for i in range(1, n + 1)]\n spec_str = \",\".join(specs)\n # argument list for the benchmark target:\n # first element -> spec string used to construct a SpecifierSet\n # second element -> version string passed to contains()\n return [[spec_str, \"1.0\"]]", "example_args": [ [ 1.0 ] ], "origin": "plan", "screen_ratio": 0.9970711759020903, "valid_fraction": 1.0, "error": null }, { "id": "F2", "kind": "claimed_speedup", "family": "Many duplicate specifiers in a SpecifierSet, exercised with many repeated contains() calls", "why": "The change removes cache invalidation (_is_unsatisfiable and _ranges = None) inside _canonicalize(). Canonicalization runs once (deduping/sorting the specifier list). Previously it cleared the cached ranges and unsatisfiability flag, forcing a recompute on the next contains()/filter() call. With many duplicate specifiers the canonicalization cost dominates, and clearing the caches adds unnecessary work. Removing the clears makes repeated contains() calls faster.", "size_meaning": "n = number of duplicate specifier strings placed in the SpecifierSet (the set will canonicalize to a single unique specifier). The generator produces n calls to contains() with a version that matches the specifier.", "make_source": "\ndef make(n):\n \"\"\"Return a list of n argument lists for SpecifierSet.contains.\n Each call checks a version that satisfies the sole unique specifier.\"\"\"\n # a simple, always‑matching specifier\n spec = \">=1.0.0\"\n # the version that satisfies it\n version = \"1.2.3\"\n # we just repeat the same call n times – the benchmark will create the\n # SpecifierSet once (with n duplicates) and then call contains n times.\n return [[version] for _ in range(n)]", "example_args": [ [ "1.2.3" ], [ "1.2.3" ], [ "1.2.3" ] ], "origin": "plan", "screen_ratio": 0.9968732043191149, "valid_fraction": 0.0, "error": null }, { "id": "F3", "kind": "suspected_slowdown", "family": "SpecifierSet built from a comma‑separated string that contains many duplicate “===…\" arbitrary specifiers (e.g. \"===1.0,===1.0,===1.0,>=1.0\"). The base version clears the cached `_is_unsatisfiable` flag when the spec list is canonicalised (duplicates removed). The head version leaves the stale flag in place; the first call to `is_unsatisfiable()` therefore returns a stale value and the next call has to recompute the flag, making the second call slower than the base version.", "why": "Removing `self._is_unsatisfiable = None` from the canonicalisation step leaves a stale cache when duplicate arbitrary specifiers are collapsed. The first call sees the stale value, the second call must recompute, adding an extra range‑intersection pass.", "size_meaning": "n = number of duplicate “===…\" specifiers (the generator builds a spec string with `n` copies of the same arbitrary version followed by a single normal specifier). Cost grows with `n` because the duplicate list gets longer before it is de‑duplicated.", "make_source": "\ndef make(n):\n # n duplicates of the same arbitrary specifier, plus one normal specifier\n base = \"===1.0\"\n specs = [base] * n + [\">=1.0\"]\n spec_str = \",\".join(specs)\n # The target is the SpecifierSet constructor; we return a list of one argument tuple\n return [(spec_str,)]", "example_args": [ [ "===1.0,===1.0,===1.0,>=1.0" ] ], "origin": "plan", "screen_ratio": 0.9942303756890585, "valid_fraction": 0.0, "error": null }, { "id": "F4", "kind": "suspected_slowdown", "family": "Invalid specifier strings that raise InvalidSpecifier during construction", "why": "The diff only touches cache invalidation inside _canonicalize; it does not affect the parsing path that raises InvalidSpecifier. No slowdown is expected, but an invalid‑input family is listed for completeness.", "size_meaning": "n = number of invalid specifier strings supplied to the SpecifierSet constructor (each will raise on construction).", "make_source": "\ndef make(n):\n # generate n distinct invalid spec strings\n invalids = [f\"@@invalid{i}@@\" for i in range(n)]\n # the benchmark harness will try to build a SpecifierSet with each;\n # we return a single‑element list per call so the harness can iterate.\n return [[s] for s in invalids]", "example_args": [ [ "@@invalid0@@" ], [ "@@invalid1@@" ] ], "origin": "plan", "screen_ratio": 0.9988686301616042, "valid_fraction": 0.0, "error": null }, { "id": "F5", "kind": "claimed_speedup", "family": "Repeated calls to `SpecifierSet.contains` (or the `in` operator) on a large SpecifierSet that contains many normal specifiers (e.g. \">=1.0,<2.0,!=1.5,~=1.2,>=2.5,<3.0\"). The first call triggers canonicalisation (sorting + deduplication) and builds the intersected version‑range cache (`_ranges`). The base version clears that cache during canonicalisation, forcing every subsequent call to rebuild the range intersection. The head version keeps the cache, so the second and later calls reuse the already‑computed ranges and are noticeably faster.", "why": "The removed lines `self._is_unsatisfiable = None; self._ranges = None` were the only place where the cached range intersection was discarded after the spec list was sorted/deduped. Keeping the cache avoids the O(N) range‑intersection work on every `contains` after the first one.", "size_meaning": "n = number of ordinary specifiers in the set (each adds a range that must be intersected). Cost of building the intersection grows linearly with `n`.", "make_source": "\ndef make(n):\n # Build a comma‑separated specifier string with `n` distinct normal specifiers\n ops = [\">=\", \"<=\", \">\", \"<\", \"!=\", \"~=\"]\n specs = []\n for i in range(n):\n op = ops[i % len(ops)]\n ver = f\"{1 + i}.0\"\n specs.append(f\"{op}{ver}\")\n spec_str = \",\".join(specs)\n # Return several argument tuples so the benchmark can call `contains` repeatedly\n return [(spec_str, v) for v in [\"1.0\", \"1.5\", \"2.0\", \"2.5\", \"3.0\"]]", "example_args": [ [ ">=1.0,<2.0,!=1.5,~=1.2", "1.5" ], [ ">=1.0,<2.0,!=1.5,~=1.2,>=2.5,<3.0", "2.7" ] ], "origin": "plan", "screen_ratio": 1.0118814000178915, "valid_fraction": 1.0, "error": null }, { "id": "R1", "kind": "claimed_speedup", "family": "many_simple_specs_variant_1", "why": "The change removes the cache‑clearing that happened on every canonicalisation. When a SpecifierSet contains many simple specifiers (e.g. many \">=1.0\", \"<=2.0\", \"==1.5\" …) the first call canonicalises the list, builds the intersected range cache and then re‑uses it on every subsequent `contains`/`filter` call. The base version threw the cache away on every canonicalisation, forcing a full range‑intersection on every call; the head version keeps the cached ranges and is therefore faster.", "size_meaning": "number of simple specifiers in the SpecifierSet (duplicates are allowed – they are deduped by canonicalisation)", "make_source": "def make(n):\n specs = [f\"<={i}.0\" for i in range(1, n + 1)]\n return [[\", \".join(specs), \"2.5\"]]", "example_args": [], "origin": "archive", "screen_ratio": 0.9969513852727684, "valid_fraction": 1.0, "error": null }, { "id": "R2", "kind": "claimed_speedup", "family": "many_simple_specs", "why": "The change removes the cache‑clearing that happened on every canonicalisation. When a SpecifierSet contains many simple specifiers (e.g. many \">=1.0\", \"<=2.0\", \"==1.5\" …) the first call canonicalises the list, builds the intersected range cache and then re‑uses it on every subsequent `contains`/`filter` call. The base version threw the cache away on every canonicalisation, forcing a full range‑intersection on every call; the head version keeps the cached ranges and is therefore faster.", "size_meaning": "number of simple specifiers in the SpecifierSet (duplicates are allowed – they are deduped by canonicalisation)", "make_source": "\ndef make(n):\n # produce n simple specifiers like \">=1.0\", \">=2.0\", …\n specs = [f\">={i}.0\" for i in range(1, n+1)]\n # return a list with a single argument list for SpecifierSet.__contains__\n return [[\",\".join(specs), \"1.5\"]]", "example_args": [], "origin": "archive", "screen_ratio": 0.997062303825154, "valid_fraction": 1.0, "error": null }, { "id": "R3", "kind": "claimed_speedup", "family": "many_simple_specs_variant_2", "why": "The change removes the cache‑clearing that happened on every canonicalisation. When a SpecifierSet contains many simple specifiers (e.g. many \">=1.0\", \"<=2.0\", \"==1.5\" …) the first call canonicalises the list, builds the intersected range cache and then re‑uses it on every subsequent `contains`/`filter` call. The base version threw the cache away on every canonicalisation, forcing a full range‑intersection on every call; the head version keeps the cached ranges and is therefore faster.", "size_meaning": "number of simple specifiers in the SpecifierSet (duplicates are allowed – they are deduped by canonicalisation)", "make_source": "def make(n):\n specs = []\n for i in range(1, n // 2 + 1):\n specs.append(f\">={i}.0\")\n specs.append(f\"<={(i * 2)}.0\")\n if len(specs) < n:\n specs.extend([f\">={len(specs)+j}.0\" for j in range(1, n - len(specs) + 1)])\n return [[\",\" .join(specs[:n]), \"1.0\"]]", "example_args": [], "origin": "archive", "screen_ratio": 0.9971418180693268, "valid_fraction": 1.0, "error": null } ], "findings": [ { "family_id": "R1", "kind": "claimed_speedup", "family": "many_simple_specs_variant_1", "why": "The change removes the cache‑clearing that happened on every canonicalisation. When a SpecifierSet contains many simple specifiers (e.g. many \">=1.0\", \"<=2.0\", \"==1.5\" …) the first call canonicalises the list, builds the intersected range cache and then re‑uses it on every subsequent `contains`/`filter` call. The base version threw the cache away on every canonicalisation, forcing a full range‑intersection on every call; the head version keeps the cached ranges and is therefore faster.", "origin": "archive", "screen_ratio": 0.9969513852727684, "valid_fraction": 1.0, "ladder": [ { "n": 2, "base_ir": 70024.31, "head_ir": 69914.86, "input_chars": 23.0, "ratio": 0.9984369713889363, "valid_fraction": 1.0, "judged": true }, { "n": 4, "base_ir": 100389.815, "head_ir": 100268.725, "input_chars": 37.0, "ratio": 0.9987938019409639, "valid_fraction": 1.0, "judged": true }, { "n": 16, "base_ir": 284785.9411764706, "head_ir": 283917.7385620915, "input_chars": 128.0, "ratio": 0.9969513852727684, "valid_fraction": 1.0, "judged": true }, { "n": 64, "base_ir": 1020884.6315789474, "head_ir": 1047457.1315789474, "input_chars": 512.0, "ratio": 1.0260288960946564, "valid_fraction": 1.0, "judged": true }, { "n": 256, "base_ir": 4091883.3333333335, "head_ir": 4110748.888888889, "input_chars": 2205.0, "ratio": 1.004610482269099, "valid_fraction": 1.0, "judged": true }, { "n": 1024, "base_ir": 17204494.5, "head_ir": 17006668.5, "input_chars": 9142.0, "ratio": 0.9885014930255579, "valid_fraction": 1.0, "judged": true }, { "n": 4096, "base_ir": 68389241.5, "head_ir": 68323389.0, "input_chars": 39862.0, "ratio": 0.9990370926982719, "valid_fraction": 1.0, "judged": true } ], "slope_base": 0.9299515668559702, "slope_head": 0.9295041442730744, "top_inputs": [ { "args": [ "í", 1, "," ], "cpu_ratio": 2.551, "base_ir": 16405.72, "head_ir": 16359.65, "ir_ratio": 0.9971918330923604, "outcome": "exc:packaging.specifiers.InvalidSpecifier" }, { "args": [ "ß", 61, "," ], "cpu_ratio": 2.5, "base_ir": 16405.72, "head_ir": 16359.65, "ir_ratio": 0.9971918330923604, "outcome": "exc:packaging.specifiers.InvalidSpecifier" } ], "error": null, "verdict": "neutral", "worst_ratio": 1.0260288960946564, "worst_search": { "args": [ "í", 1, "," ], "cpu_ratio": 2.551, "base_ir": 16405.72, "head_ir": 16359.65, "ir_ratio": 0.9971918330923604, "outcome": "exc:packaging.specifiers.InvalidSpecifier" } } ], "seconds": { "plan": 20.72, "screen": 82.59, "full": 107.66, "total": 210.97 }, "ladder_validity": "backfilled 2026-10-11 (scripts/backtest/backfill_ladder_validity.py)" } } } ] }, { "id": "#1256", "pr": 1256, "title": "perf(utils): precompile the wheel project-name pattern", "base": "54c697714e19da8659b67b5e96eb12a1f8250f9d", "head": "d2ac6ba3f819cb5c81d79f4820db205d6f8bc126", "merged": "2026-06-13", "after_ultra_release": true, "known": [], "targets": [ { "target": "packaging.utils:parse_wheel_filename", "run_id": "bt-1256-85d56f", "image": "bw-pair:0e393c2af8d05bae", "usd": 0.0418, "harvested": 80, "arms": { "hypothesis": { "classes": [], "known": [], "seconds": 3.8, "error": null }, "mutations": { "classes": [], "known": [], "seconds": 1.8, "error": null }, "breakwater": { "classes": [], "known": [], "seconds": 45.2, "error": null }, "llm-plan": { "classes": [], "known": [], "seconds": 2.8, "error": null } }, "findings": [], "perf": { "slower_families": [], "worst_valid_ratio": null, "worst_family": null, "speedup_ratio_small": 0.9125763458727995, "report": { "run_id": "bt-1256-85d56f", "target": "packaging.utils:parse_wheel_filename", "image": "bw-pair:0e393c2af8d05bae", "backend": "docker", "families": [ { "id": "F1", "kind": "claimed_speedup", "family": "The change replaces an inline `re.match` call with a pre‑compiled regular expression (`_wheel_name_regex`). For typical, well‑formed wheel filenames the compile", "why": "The change replaces an inline `re.match` call with a pre‑compiled regular expression (`_wheel_name_regex`). For typical, well‑formed wheel filenames the compiled regex avoids repeated pattern compilation and cache lookup, so the new version should be faster.", "size_meaning": "n = number of allowed characters in the project‑name part of the wheel filename (the part before the first dash).", "make_source": "\ndef make(n):\n # Build a project name consisting of n ASCII letters/digits/underscores/dots.\n # The name must not contain \"__\" and must match the wheel name regex.\n allowed = string.ascii_letters + string.digits + \"_.\"\n # Ensure at least one character and no \"__\" sequence.\n name = ''.join(allowed[i % len(allowed)] for i in range(max(1, n)))\n # Avoid \"__\" by replacing any occurrence.\n name = name.replace('__', '_')\n # Construct a minimal valid wheel filename: name-version-py3-none-any.whl\n wheel = f\"{name}-1.0-py3-none-any.whl\"\n return [[wheel]]", "example_args": [], "origin": "plan", "screen_ratio": 0.9164497052998334, "valid_fraction": 1.0, "error": null }, { "id": "F2", "kind": "claimed_speedup", "family": "valid_long_name", "why": "The change pre‑compiles the wheel‑name validation regex (`_wheel_name_regex`). For valid wheel filenames the regex is executed on every call, so a longer project‑name part makes the match work harder. The head version avoids re‑compiling the pattern on each call, giving a measurable speed‑up that grows with the length of the name.", "size_meaning": "n ≈ length of the project‑name segment (number of extra ASCII word characters).", "make_source": "def make(n):\n # Build a valid wheel filename whose name part grows with n.\n # Format: {name}-{version}-{pyver}-{abi}-{platform}.whl (4 dashes)\n name = \"pkg\" + \"a\" * n # only word characters, no '__'\n version = \"1.0\"\n tag = \"py3-none-any\"\n filename = f\"{name}-{version}-{tag}.whl\"\n return [(filename,)]", "example_args": [], "origin": "plan", "screen_ratio": 0.9162208680280969, "valid_fraction": 1.0, "error": null }, { "id": "F3", "kind": "claimed_speedup", "family": "valid wheel filenames with varying project names", "why": "The change replaces an inline regex compilation on each call with a pre‑compiled global regex. For the common case of many valid wheel filenames the head version avoids repeated regex compilation and therefore should be faster.", "size_meaning": "number of distinct wheel filenames generated", "make_source": "def make(n):\n \"\"\"Return up to 8 wheel filenames; n controls how many are produced.\"\"\"\n n = min(max(n, 1), 8)\n result = []\n for i in range(n):\n # Vary the project name length to exercise the regex on different input sizes\n name = 'pkg' + 'a' * (i + 1)\n filename = f'{name}-1.0-py3-none-any.whl'\n result.append([filename])\n return result", "example_args": [], "origin": "plan", "screen_ratio": 0.9141243816038184, "valid_fraction": 1.0, "error": null }, { "id": "F4", "kind": "suspected_slowdown", "family": "Both the old and the new code run the same regular‑expression engine over the entire project‑name string. If the name part is extremely long and contains many U", "why": "Both the old and the new code run the same regular‑expression engine over the entire project‑name string. If the name part is extremely long and contains many Unicode word characters, the regex match will scan the whole string, and the overhead of the Python function call to the compiled regex object (`_wheel_name_regex.match`) may be marginally higher than the single `re.match` call that also compiles (or reuses a cached pattern). In practice the difference is tiny, but this family exercises the worst‑case path for the new code.", "size_meaning": "n = length of the Unicode project‑name part (number of characters).", "make_source": "\ndef make(n):\n # Use a Unicode word character outside ASCII to force the UNICODE flag path.\n # U+03B1 (Greek small letter beta) is a word character.\n ch = chr(0x03B1)\n name = ch * max(1, n) # e.g. βββ…\n # Ensure no \"__\" sequence.\n wheel = f\"{name}-1.0-py3-none-any.whl\"\n return [[wheel]]", "example_args": [], "origin": "plan", "screen_ratio": 0.9217904566260525, "valid_fraction": 1.0, "error": null }, { "id": "F5", "kind": "suspected_slowdown", "family": "valid_name_unicode_stress", "why": "Both versions use the same Unicode‑aware regex, but the head version accesses a module‑level compiled object (`_wheel_name_regex.match`) while the base version calls `re.match` each time. For inputs that force the regex engine to scan a long Unicode string before succeeding (e.g., many non‑ASCII word characters), the extra global‑lookup and method call could add a tiny constant overhead, making the head version marginally slower on this specific family.", "size_meaning": "n ≈ number of Unicode word characters (e.g., accented letters) in the project name.", "make_source": "def make(n):\n # Use a Unicode letter that is a word character but not ASCII.\n ch = \"\\u00e9\" # 'é'\n name = \"pkg\" + ch * n\n version = \"1.0\"\n tag = \"py3-none-any\"\n filename = f\"{name}-{version}-{tag}.whl\"\n return [(filename,)]", "example_args": [], "origin": "plan", "screen_ratio": 0.9196396060656897, "valid_fraction": 1.0, "error": null }, { "id": "F6", "kind": "suspected_slowdown", "family": "invalid wheel filenames containing '__' (invalid project name)", "why": "Both versions check for '__' before the regex, so the regex is never evaluated. The head version still pays the cost of the pre‑compiled regex object lookup, which is negligible; however if there were any micro‑overhead it would appear on the worst‑case shows up on inputs that fail the early '__' test. This family is marked invalid because such filenames are rejected by the spec.", "size_meaning": "number of invalid wheel filenames generated", "make_source": "def make(n):\n \"\"\"Return up to 8 invalid wheel filenames containing '__'.\"\"\"\n n = min(max(n, 1), 8)\n result = []\n for i in range(n):\n name = 'pkg__' + 'b' * i\n filename = f'{name}-1.0-py3-none-any.whl'\n result.append([filename])\n return result", "example_args": [], "origin": "plan", "screen_ratio": 1.0, "valid_fraction": 0.0, "error": null } ], "findings": [ { "family_id": "F3", "kind": "claimed_speedup", "family": "valid wheel filenames with varying project names", "why": "The change replaces an inline regex compilation on each call with a pre‑compiled global regex. For the common case of many valid wheel filenames the head version avoids repeated regex compilation and therefore should be faster.", "origin": "plan", "screen_ratio": 0.9141243816038184, "valid_fraction": 1.0, "ladder": [ { "n": 2, "base_ir": 39585.8825, "head_ir": 36125.14, "input_chars": 29.5, "ratio": 0.9125763458727995, "valid_fraction": 1.0, "judged": true }, { "n": 4, "base_ir": 39501.18137254902, "head_ir": 36056.356209150326, "input_chars": 30.5, "ratio": 0.9127918446056238, "valid_fraction": 1.0, "judged": true }, { "n": 16, "base_ir": 39605.630208333336, "head_ir": 36204.47222222222, "input_chars": 32.5, "ratio": 0.9141243816038184, "valid_fraction": 1.0, "judged": true }, { "n": 64, "base_ir": 39605.630208333336, "head_ir": 36204.47222222222, "input_chars": 32.5, "ratio": 0.9141243816038184, "valid_fraction": 1.0, "judged": true }, { "n": 256, "base_ir": 39605.630208333336, "head_ir": 36204.47222222222, "input_chars": 32.5, "ratio": 0.9141243816038184, "valid_fraction": 1.0, "judged": true }, { "n": 1024, "base_ir": 39605.630208333336, "head_ir": 36204.47222222222, "input_chars": 32.5, "ratio": 0.9141243816038184, "valid_fraction": 1.0, "judged": true }, { "n": 4096, "base_ir": 39605.630208333336, "head_ir": 36204.47222222222, "input_chars": 32.5, "ratio": 0.9141243816038184, "valid_fraction": 1.0, "judged": true } ], "slope_base": null, "slope_head": null, "top_inputs": [ { "args": [ [ [ "a", "1.0", "py3", "none", "any.whl" ], [ "a", "1.0", "py3", "none", "any.whl" ], [ "jbsvmmbx", "1.0", "py3", "none", "any.whl" ], [ "a", "1.0", "py3", "none", "any.whl" ], [ "a", "1.0", "py3", "none", "any.whl" ], [ "jbsvmmbx", "1.0", "py3", "none", "any.whl" ], [ "a", "1.0", "py3", "none", "any.whl" ], [ "a", "1.0", "py3", "none", "any.whl" ] ] ], "cpu_ratio": 4.414, "base_ir": 6964.64406779661, "head_ir": 6964.64406779661, "ir_ratio": 1.0, "outcome": "exc:builtins.AttributeError" }, { "args": [ [ [ "a", "1.0", "py3", "none", "any.whl" ], [ "a", "1.0", "py3", "none", "any.whl" ], [ "jbsvmmbx", "1.0", "py3", "none", "any.whl" ], [ "a", "1.0", "py3", "none", "any.whl" ], [ "a", "1.0", "py3", "none", "any.whl" ] ] ], "cpu_ratio": 3.163, "base_ir": 6966.872340425532, "head_ir": 6966.872340425532, "ir_ratio": 1.0, "outcome": "exc:builtins.AttributeError" } ], "error": null, "verdict": "faster", "worst_ratio": 1.0, "worst_search": { "args": [ [ [ "a", "1.0", "py3", "none", "any.whl" ], [ "a", "1.0", "py3", "none", "any.whl" ], [ "jbsvmmbx", "1.0", "py3", "none", "any.whl" ], [ "a", "1.0", "py3", "none", "any.whl" ], [ "a", "1.0", "py3", "none", "any.whl" ], [ "jbsvmmbx", "1.0", "py3", "none", "any.whl" ], [ "a", "1.0", "py3", "none", "any.whl" ], [ "a", "1.0", "py3", "none", "any.whl" ] ] ], "cpu_ratio": 4.414, "base_ir": 6964.64406779661, "head_ir": 6964.64406779661, "ir_ratio": 1.0, "outcome": "exc:builtins.AttributeError" } } ], "seconds": { "plan": 12.15, "screen": 70.11, "full": 101.44, "total": 183.7 }, "ladder_validity": "backfilled 2026-10-11 (scripts/backtest/backfill_ladder_validity.py)" } } } ] }, { "id": "#1258", "pr": 1258, "title": "perf: add __slots__ to token classes", "base": "e7f0351352783a8a2f17f8fe15d8504f604468eb", "head": "bce44a178e0033b51200880de3a1cb9bfadfb849", "merged": "2026-06-13", "after_ultra_release": true, "known": [], "targets": [ { "target": "packaging.requirements:Requirement", "run_id": "bt-1258-a58f0c", "image": "bw-pair:aa205a50cfdfbeb7", "usd": 0.0399, "harvested": 80, "arms": { "hypothesis": { "classes": [], "known": [], "seconds": 8.5, "error": null }, "mutations": { "classes": [], "known": [], "seconds": 2.8, "error": null }, "breakwater": { "classes": [], "known": [], "seconds": 39.6, "error": null }, "llm-plan": { "classes": [], "known": [], "seconds": 1.8, "error": null } }, "findings": [], "perf": { "slower_families": [], "worst_valid_ratio": null, "worst_family": null, "speedup_ratio_small": 0.9717406770004221, "report": { "run_id": "bt-1258-a58f0c", "target": "packaging.requirements:Requirement", "image": "bw-pair:aa205a50cfdfbeb7", "backend": "docker", "families": [ { "id": "F1", "kind": "claimed_speedup", "family": "requirement strings with a large number of extra markers (extras), which produce many Token objects during tokenization; the added __slots__ on Token reduces per‑token memory and attribute‑access cost, so this path should become faster.", "why": "The tokenizer creates one Token per extra name (plus a few for the name, specifier, URL, marker). With many extras the number of Token allocations dominates runtime, and __slots__ eliminates the per‑instance __dict__, giving a measurable speedup.", "size_meaning": "n = number of extras in the requirement (each extra adds a token).", "make_source": "\ndef make(n):\n # Build a requirement like pkg[extra0,extra1,...,extraN-1] >=1.0\n extras = ','.join(f'extra{i}' for i in range(n))\n req = f'pkg[{extras}] >=1.0'\n return [[req]]", "example_args": [], "origin": "plan", "screen_ratio": 0.9828485794520616, "valid_fraction": 1.0, "error": null }, { "id": "F2", "kind": "claimed_speedup", "family": "complex_requirement", "why": "Long requirement strings with many extras, version specifiers, a URL and an environment marker exercise the tokenizer heavily; the added __slots__ on Token and Tokenizer reduces per‑token attribute‑access overhead, so this family should be faster in head.", "size_meaning": "n = number of extra names (also drives length of specifier and marker)", "make_source": "def make(n):\n # Build a requirement with n extras, a multi‑clause specifier, a URL and a marker.\n extras = ','.join(f'extra{i}' for i in range(n))\n specifier = '>=1.0,<2.0,!=1.5,==1.2.*'\n url = 'https://example.com/pkg'\n marker = 'python_version>=\"3.6\" and sys_platform==\"linux\"'\n req = f'myproj[{extras}]{specifier} @ {url} ; {marker}'\n return [[req]]", "example_args": [], "origin": "plan", "screen_ratio": 0.9828299573253478, "valid_fraction": 0.0, "error": null }, { "id": "F3", "kind": "claimed_speedup", "family": "simple_requirement", "why": "A single, well‑formed requirement like \"requests>=2.25\" exercises the fast path of the tokenizer (few tokens, no extras, no markers). Adding __slots__ to Token/Tokenizer reduces per‑token overhead, so this case should be faster in the head version.", "size_meaning": "Number of characters in the version specifier; grows linearly with n.", "make_source": "\ndef make(n: int):\n # n controls the length of the version number (e.g. 1.0, 1.0.0, …)\n # keep it a valid PEP 440 version.\n base = \"requests>=\"\n # produce versions like 1.0, 1.0.0, 1.0.0.0 …\n version = \".\".join(str(i) for i in range(1, n + 2))\n req = base + version\n return [[req]]", "example_args": [ [ "requests>=1.0" ], [ "requests>=1.0.0" ], [ "requests>=1.0.0.0" ] ], "origin": "plan", "screen_ratio": 0.9918430204703024, "valid_fraction": 1.0, "error": null }, { "id": "F4", "kind": "suspected_slowdown", "family": "a batch of very small, trivial requirements (just a plain project name). Each call constructs a fresh Tokenizer instance; the __slots__ version adds a tiny per‑instance overhead for slot initialization that can outweigh the saved __dict__ when the token stream is only one or two tokens.", "why": "Tokenizer.__init__ now has to allocate the fixed slot array for every parse. When the input yields only a single token, the fixed cost of slot setup dominates, making the new version slightly slower per call.", "size_meaning": "n = number of separate tiny requirements (capped at 8 because the generator returns at most 8 argument lists).", "make_source": "def make(n):\n # Produce up to 8 independent tiny requirements; each is a separate call.\n count = min(8, max(1, n))\n return [[f'pkg{i}'] for i in range(count)]", "example_args": [], "origin": "plan", "screen_ratio": 0.9889928390718853, "valid_fraction": 1.0, "error": null }, { "id": "F5", "kind": "suspected_slowdown", "family": "simple_requirement", "why": "A minimal requirement ('pkg') creates only a handful of Token objects. The extra class‑level __slots__ descriptor look‑ups add a constant overhead per object creation that can dominate when the workload is tiny, making head slightly slower for very small inputs.", "size_meaning": "n = number of independent tiny requirements parsed in one batch", "make_source": "def make(n):\n # Return n identical minimal requirements; cost grows linearly with n.\n return [[\"pkg\"] for _ in range(n)]", "example_args": [], "origin": "plan", "screen_ratio": 0.9888247526792141, "valid_fraction": 1.0, "error": null }, { "id": "F6", "kind": "suspected_slowdown", "family": "many_extras", "why": "Each extra (e.g. \"[extra1,extra2,…]\") is tokenized as a separate identifier token. A requirement with many extras creates many Token objects; the new __slots__ layout may add a small per‑object allocation cost that outweighs the attribute‑access win when the token count is high.", "size_meaning": "Number of extras in the requirement; token count grows O(n).", "make_source": "\ndef make(n: int):\n # Build a name with n extras: pkg[extra0,extra1,…,extra{n‑1}]\n extras = \",\".join(f\"extra{i}\" for i in range(n))\n req = f\"pkg[{extras}]\"\n return [[req]]", "example_args": [ [ "pkg[extra0]" ], [ "pkg[extra0,extra1]" ], [ "pkg[extra0,extra1,extra2]" ] ], "origin": "plan", "screen_ratio": 0.9821486161727504, "valid_fraction": 1.0, "error": null } ], "findings": [ { "family_id": "F1", "kind": "claimed_speedup", "family": "requirement strings with a large number of extra markers (extras), which produce many Token objects during tokenization; the added __slots__ on Token reduces per‑token memory and attribute‑access cost, so this path should become faster.", "why": "The tokenizer creates one Token per extra name (plus a few for the name, specifier, URL, marker). With many extras the number of Token allocations dominates runtime, and __slots__ eliminates the per‑instance __dict__, giving a measurable speedup.", "origin": "plan", "screen_ratio": 0.9828485794520616, "valid_fraction": 1.0, "ladder": [ { "n": 2, "base_ir": 192267.125, "head_ir": 189983.105, "input_chars": 28.0, "ratio": 0.9881205900384686, "valid_fraction": 1.0, "judged": true }, { "n": 4, "base_ir": 239620.34, "head_ir": 236373.14, "input_chars": 42.0, "ratio": 0.9864485627555658, "valid_fraction": 1.0, "judged": true }, { "n": 16, "base_ir": 524531.2684563758, "head_ir": 515534.8120805369, "input_chars": 132.0, "ratio": 0.9828485794520616, "valid_fraction": 1.0, "judged": true }, { "n": 64, "base_ir": 1666814.4473684211, "head_ir": 1648961.5, "input_chars": 516.0, "ratio": 0.9892891812903305, "valid_fraction": 1.0, "judged": true }, { "n": 256, "base_ir": 6362234.666666667, "head_ir": 6182442.222222222, "input_chars": 2208.0, "ratio": 0.9717406770004221, "valid_fraction": 1.0, "judged": true }, { "n": 1024, "base_ir": 24806711.5, "head_ir": 24437030.0, "input_chars": 9144.0, "ratio": 0.9850975208866358, "valid_fraction": 1.0, "judged": true }, { "n": 4096, "base_ir": 99257131.5, "head_ir": 97360688.0, "input_chars": 39864.0, "ratio": 0.9808936297942481, "valid_fraction": 1.0, "judged": true } ], "slope_base": 0.8684969408134784, "slope_head": 0.8675152841206557, "top_inputs": [], "error": null, "verdict": "neutral", "worst_ratio": 0.9892891812903305, "worst_search": null } ], "seconds": { "plan": 10.34, "screen": 132.49, "full": 134.72, "total": 277.56 }, "ladder_validity": "backfilled 2026-10-11 (scripts/backtest/backfill_ladder_validity.py)" } }, "rescored": "eq-hash-contract class removed: base also violates it (probe fix, deterministic re-check)", "image_rescored": "bw-pair:bc22d3399fde3139" } ] }, { "id": "#1259", "pr": 1259, "title": "perf(specifiers): push PEP 440 logic into the range engine", "base": "155760e37acbe24bf82444f21e805006ef0b58c8", "head": "664f049980f1873c458de72a449003b83173f445", "merged": "2026-06-15", "after_ultra_release": true, "known": [], "targets": [ { "target": "packaging.specifiers:SpecifierSet#contains", "run_id": "bt-1259-8d2b3c", "image": "bw-pair:ec9525174f993117", "usd": 0.2083, "harvested": 80, "arms": { "hypothesis": { "classes": [], "known": [], "seconds": 11.3, "error": null }, "mutations": { "classes": [], "known": [], "seconds": 5.4, "error": null }, "breakwater": { "classes": [], "known": [], "seconds": 124.0, "error": null }, "llm-plan": { "classes": [], "known": [], "seconds": 4.8, "error": null } }, "findings": [], "perf": { "slower_families": [], "worst_valid_ratio": null, "worst_family": null, "speedup_ratio_small": 0.9737759649590926, "report": { "run_id": "bt-1259-8d2b3c", "target": "packaging.specifiers:SpecifierSet#contains", "image": "bw-pair:ec9525174f993117", "backend": "docker", "families": [ { "id": "F1", "kind": "suspected_slowdown", "family": "SpecifierSet.contains with many wildcard (==X.*) specs and a version without a local part", "why": "The new implementation falls back to intersected‐bounds path on the first spec that returns None from _fast_match. With many \"==X.*\" specs every spec returns None, so intersect_specifier_bounds builds a wildcard range for each spec and intersects them all before the membership test. The old fast‑match loop would break on the first None and then fall back to filter(), which also built the intersected ranges but only once per call; the new code does the same work *and* allocates the intermediate per‑spec range lists, making the first call noticeably slower when the number of specs is large.", "size_meaning": "n = number of \"==X.*\" specs in the set (the version string is constant).", "make_source": "\ndef make(n):\n # Build a specifier set like \"==1.*,==2.*,...,==n.*\"\n specs = ','.join(f'=={i}.*' for i in range(1, n + 1))\n # The version we test against has no local part, so _fast_match returns None for every spec.\n version = \"1.0\"\n # Return a list of argument tuples for SpecifierSet.contains(specifiers, version, prereleases)\n return [[specs, version, None]]", "example_args": [ [ 1.0 ], [ null ] ], "origin": "plan", "screen_ratio": 0.8963202738649148, "valid_fraction": 1.0, "error": null }, { "id": "F2", "kind": "claimed_speedup", "family": "SpecifierSet containing many simple comparators (>=, <=, ==, !=, <, >) with plain release versions and no local segment.", "why": "The new fast‑path loops over every spec calling `_fast_match`. When all specs are simple the loop finishes without ever building the intersected range cache, giving an O(N) check that is cheaper than the old path which would fall back to the range engine after the first call.", "size_meaning": "Number of specifiers in the set (N).", "make_source": "\ndef make(n):\n # build a SpecifierSet string with n simple specs\n specs = []\n for i in range(1, n+1):\n specs.append(f\">=1.{i}.0\")\n return [[\",\".join(specs), \"1.5.0\"]]", "example_args": [], "origin": "plan", "screen_ratio": 0.999472074247154, "valid_fraction": 1.0, "error": null }, { "id": "F3", "kind": "suspected_slowdown", "family": "SpecifierSet.contains with many simple specs (>=, <=, ==) and a version that carries a local version", "why": "When the tested version has a local part (e.g. \"1.0+local\") every simple spec returns None from _fast_match, so the new code takes the intersected‑bounds path. intersect_specifier_bounds must compute the range for each spec and intersect them, allocating a new list for each spec. The old path broke on the first None and then called filter(), which also built the intersected ranges but reused the already‑computed per‑spec ranges; the new implementation rebuilds the per‑spec range lists on the first call, causing a measurable slowdown for large spec sets.", "size_meaning": "n = number of simple specs in the set (each spec is a simple \">=1.0\", \"<=2.0\", \"==1.5\", …).", "make_source": "\ndef make(n):\n # Create n alternating simple specs.\n ops = ['>=', '<=', '==']\n specs = ','.join(f'{ops[i%3]}{i}.0' for i in range(1, n + 1))\n # Version with a local part forces _fast_match → None for every spec.\n version = \"1.0+local\"\n return [[specs, version, None]]", "example_args": [ [ null ] ], "origin": "plan", "screen_ratio": 1.00939752696191, "valid_fraction": 1.0, "error": null }, { "id": "F4", "kind": "suspected_slowdown", "family": "SpecifierSet that contains at least one wildcard specifier (e.g. \"==1.2.*\") together with many simple specs.", "why": "`_fast_match` returns ``None`` for a wildcard, so the new code breaks out of the fast‑path on the first wildcard it sees and immediately builds the intersected range cache (`_get_ranges`). The old implementation would only build the intersected ranges when the fallback filter was actually invoked; if the caller never asked for a version that triggered the wildcard path the old code avoided the work entirely. Repeated ``contains`` calls on versions that do *not* match the wildcard therefore pay the intersection cost on the very first call in the new version.", "size_meaning": "Number of simple specs before the wildcard (N).", "make_source": "\ndef make(n):\n # n simple specs followed by one wildcard\n specs = [f\">=1.{i}.0\" for i in range(1, n+1)]\n specs.append(\"==1.0.*\")\n return [[\",\".join(specs), \"1.5.0\"]]", "example_args": [], "origin": "plan", "screen_ratio": 0.9994884259435145, "valid_fraction": 1.0, "error": null }, { "id": "F5", "kind": "suspected_slowdown", "family": "SpecifierSet.filter on a large iterable with many specs and prereleases=None", "why": "filter() now calls filter_by_ranges, which builds the intersected bounds once (via _get_ranges → intersect_specifier_bounds) and then iterates the input, coercing each item with coerce_version. The old filter() took the same path, but the new filter_by_ranges does the coercion inline (parsed = coerce_version(item if key is None else key(item))) for every element, adding a function‑call overhead per item. When the iterable is large (e.g. thousands of version strings) and the specifier set contains many specs (so the intersected bounds list is long), the per‑item overhead becomes noticeable.", "size_meaning": "n = number of version strings in the input iterable; the spec set contains a fixed 10 simple specs.", "make_source": "def make(n):\n # Fixed specifier set with 10 simple specs\n specs = '>=1.0,<=2.0,==1.5,>=2.5,<=3.0,==2.8,>=3.5,<=4.0,==3.9,>=4.5'\n # Generate n version strings without local parts.\n versions = [f'{i}.0' for i in range(1, n + 1)]\n # Arguments for SpecifierSet.filter: (specifiers, iterable, prereleases, key)\n return [[specs, versions, None, None]]", "example_args": [ [ "1.0", "2.0", "3.0" ], [ null ], [ null ] ], "origin": "plan", "screen_ratio": 0.9996129161866015, "valid_fraction": 0.0, "error": null }, { "id": "F6", "kind": "suspected_slowdown", "family": "SpecifierSet with only simple specs, but the version being tested contains a local segment (e.g. \"1.2.3+local\").", "why": "The fast‑path guard `version.local is None` fails, so the new code jumps straight to the generic `filter` path which parses the version and runs the range engine. The old `Specifier.contains` fast‑matched the simple operators first and only fell back to the range engine when `_fast_match` returned ``None``; for a local version `_fast_match` also returns ``None`` but the old code would have taken the same filter path, however the new `SpecifierSet.contains` incurs the extra overhead of iterating `self._specs` and checking the fast‑path guard before giving up.", "size_meaning": "Number of specs in the set (N).", "make_source": "def make(n):\n specs = [f\">=1.{i}.0\" for i in range(1, n+1)]\n return [[\",\".join(specs), \"1.5.0+local\"]]", "example_args": [], "origin": "plan", "screen_ratio": 1.0127689761426715, "valid_fraction": 1.0, "error": null }, { "id": "R1", "kind": "claimed_speedup", "family": "many_simple_specs_variant_1", "why": "The change removes the cache‑clearing that happened on every canonicalisation. When a SpecifierSet contains many simple specifiers (e.g. many \">=1.0\", \"<=2.0\", \"==1.5\" …) the first call canonicalises the list, builds the intersected range cache and then re‑uses it on every subsequent `contains`/`filter` call. The base version threw the cache away on every canonicalisation, forcing a full range‑intersection on every call; the head version keeps the cached ranges and is therefore faster.", "size_meaning": "number of simple specifiers in the SpecifierSet (duplicates are allowed – they are deduped by canonicalisation)", "make_source": "def make(n):\n specs = [f\"<={i}.0\" for i in range(1, n + 1)]\n return [[\", \".join(specs), \"2.5\"]]", "example_args": [], "origin": "archive", "screen_ratio": 0.9997483205576974, "valid_fraction": 1.0, "error": null }, { "id": "R2", "kind": "claimed_speedup", "family": "many_simple_specs", "why": "The change removes the cache‑clearing that happened on every canonicalisation. When a SpecifierSet contains many simple specifiers (e.g. many \">=1.0\", \"<=2.0\", \"==1.5\" …) the first call canonicalises the list, builds the intersected range cache and then re‑uses it on every subsequent `contains`/`filter` call. The base version threw the cache away on every canonicalisation, forcing a full range‑intersection on every call; the head version keeps the cached ranges and is therefore faster.", "size_meaning": "number of simple specifiers in the SpecifierSet (duplicates are allowed – they are deduped by canonicalisation)", "make_source": "\ndef make(n):\n # produce n simple specifiers like \">=1.0\", \">=2.0\", …\n specs = [f\">={i}.0\" for i in range(1, n+1)]\n # return a list with a single argument list for SpecifierSet.__contains__\n return [[\",\".join(specs), \"1.5\"]]", "example_args": [], "origin": "archive", "screen_ratio": 0.9998247160995112, "valid_fraction": 1.0, "error": null }, { "id": "R3", "kind": "claimed_speedup", "family": "many_simple_specs_variant_2", "why": "The change removes the cache‑clearing that happened on every canonicalisation. When a SpecifierSet contains many simple specifiers (e.g. many \">=1.0\", \"<=2.0\", \"==1.5\" …) the first call canonicalises the list, builds the intersected range cache and then re‑uses it on every subsequent `contains`/`filter` call. The base version threw the cache away on every canonicalisation, forcing a full range‑intersection on every call; the head version keeps the cached ranges and is therefore faster.", "size_meaning": "number of simple specifiers in the SpecifierSet (duplicates are allowed – they are deduped by canonicalisation)", "make_source": "def make(n):\n specs = []\n for i in range(1, n // 2 + 1):\n specs.append(f\">={i}.0\")\n specs.append(f\"<={(i * 2)}.0\")\n if len(specs) < n:\n specs.extend([f\">={len(specs)+j}.0\" for j in range(1, n - len(specs) + 1)])\n return [[\",\" .join(specs[:n]), \"1.0\"]]", "example_args": [], "origin": "archive", "screen_ratio": 0.9998252453799829, "valid_fraction": 1.0, "error": null } ], "findings": [ { "family_id": "F2", "kind": "claimed_speedup", "family": "SpecifierSet containing many simple comparators (>=, <=, ==, !=, <, >) with plain release versions and no local segment.", "why": "The new fast‑path loops over every spec calling `_fast_match`. When all specs are simple the loop finishes without ever building the intersected range cache, giving an O(N) check that is cheaper than the old path which would fall back to the range engine after the first call.", "origin": "plan", "screen_ratio": 0.999472074247154, "valid_fraction": 1.0, "ladder": [ { "n": 2, "base_ir": 92363.73, "head_ir": 92382.4, "input_chars": 28.0, "ratio": 1.0002021356218507, "valid_fraction": 1.0, "judged": true }, { "n": 4, "base_ir": 159423.615, "head_ir": 159288.58, "input_chars": 44.0, "ratio": 0.9991529799396407, "valid_fraction": 1.0, "judged": true }, { "n": 16, "base_ir": 379646.89552238805, "head_ir": 379446.4701492537, "input_chars": 147.0, "ratio": 0.999472074247154, "valid_fraction": 1.0, "judged": true }, { "n": 64, "base_ir": 1145927.8823529412, "head_ir": 1115877.0294117648, "input_chars": 579.0, "ratio": 0.9737759649590926, "valid_fraction": 1.0, "judged": true }, { "n": 256, "base_ir": 4161191.0, "head_ir": 4162161.5, "input_chars": 2464.0, "ratio": 1.0002332264969331, "valid_fraction": 1.0, "judged": true }, { "n": 1024, "base_ir": 17111946.5, "head_ir": 17187582.5, "input_chars": 10169.0, "ratio": 1.0044200699201578, "valid_fraction": 1.0, "judged": true }, { "n": 4096, "base_ir": 67991973.5, "head_ir": 68001832.5, "input_chars": 43961.0, "ratio": 1.000145002409733, "valid_fraction": 1.0, "judged": true } ], "slope_base": 0.8829307798638516, "slope_head": 0.8833917009511645, "top_inputs": [ { "args": [ "l5K2", 27, "1.5.0" ], "cpu_ratio": 2.521, "base_ir": 16801.16, "head_ir": 16829.14, "ir_ratio": 1.0016653612012503, "outcome": "exc:packaging.specifiers.InvalidSpecifier" }, { "args": [ "R", 21, "1.5.0" ], "cpu_ratio": 1.992, "base_ir": 16192.74, "head_ir": 16191.72, "ir_ratio": 0.9999370088076508, "outcome": "exc:packaging.specifiers.InvalidSpecifier" } ], "error": null, "verdict": "neutral", "worst_ratio": 1.0044200699201578, "worst_search": { "args": [ "l5K2", 27, "1.5.0" ], "cpu_ratio": 2.521, "base_ir": 16801.16, "head_ir": 16829.14, "ir_ratio": 1.0016653612012503, "outcome": "exc:packaging.specifiers.InvalidSpecifier" } } ], "seconds": { "plan": 31.81, "screen": 131.02, "full": 83.98, "total": 246.81 }, "ladder_validity": "backfilled 2026-10-11 (scripts/backtest/backfill_ladder_validity.py)" } } } ] }, { "id": "#1320", "pr": 1320, "title": "perf: add __slots__ to Requirement", "base": "2b6726cec7bd55fd3adda07054d0a41288d44eba", "head": "d5fedc90bba333ed0962ba5dfb7072d38f2c1d84", "merged": "2026-07-07", "after_ultra_release": true, "known": [ { "id": "requirement-pickle", "match": { "probe": "P4" }, "fixed_by": 1171, "status": "candidate" } ], "targets": [ { "target": "packaging.requirements:Requirement", "run_id": "bt-1320-81a5cc", "image": "bw-pair:9728e5e1fd432d33", "usd": 0.0492, "harvested": 80, "arms": { "hypothesis": { "classes": [], "known": [], "seconds": 4.8, "error": null }, "mutations": { "classes": [], "known": [], "seconds": 2.6, "error": null }, "breakwater": { "classes": [], "known": [], "seconds": 38.8, "error": null }, "llm-plan": { "classes": [], "known": [], "seconds": 1.8, "error": null } }, "findings": [], "perf": { "slower_families": [], "worst_valid_ratio": null, "worst_family": null, "speedup_ratio_small": 0.9868666927862122, "report": { "run_id": "bt-1320-81a5cc", "target": "packaging.requirements:Requirement", "image": "bw-pair:9728e5e1fd432d33", "backend": "docker", "families": [ { "id": "F1", "kind": "claimed_speedup", "family": "simple_requirements", "why": "Adding __slots__ removes the per‑instance __dict__, so attribute writes in __init__ (name, url, extras, specifier, marker) become direct slot stores. Creating many tiny requirements should be measurably faster.", "size_meaning": "number of requirement objects created", "make_source": "def make(n):\n base = 'pkg==1.0.0'\n return [[base] for _ in range(n)]", "example_args": [], "origin": "plan", "screen_ratio": 0.9999010595093569, "valid_fraction": 1.0, "error": null }, { "id": "F2", "kind": "claimed_speedup", "family": "Repeated attribute reads on many Requirement instances (slots make attribute lookup faster and reduce per‑object memory).", "why": "The diff adds ``__slots__``; slots eliminate the per‑instance ``__dict__`` and make attribute access a direct C‑array lookup. Code that creates many Requirement objects and then reads their fields (name, extras, specifier, url, marker) will benefit.", "size_meaning": "Number of Requirement objects created; each object has a modestly sized extras set (≈ n/2 items) so the workload grows with *n*.", "make_source": "def make(n):\n # Produce *n* requirement strings that parse successfully.\n # Each string gets a name, a specifier, and ~n/2 extras to keep the\n # objects non‑trivial while staying deterministic.\n base = \"pkg\"\n specs = [\"==1.0\", \">=2.0\", \"!=3.0\"]\n extras_pool = [f\"extra{i}\" for i in range(max(1, n//2))]\n out = []\n for i in range(n):\n name = f\"{base}{i}\"\n specifier = specs[i % len(specs)]\n extras = \",\".join(extras_pool[: (i % len(extras_pool)) + 1])\n req = f\"{name}[{extras}]{specifier}\"\n out.append([req])\n return out", "example_args": [ [ "pkg0[extra0]==1.0" ], [ "pkg1[extra0,extra1]>=2.0" ] ], "origin": "plan", "screen_ratio": 0.9868666927862122, "valid_fraction": 1.0, "error": null }, { "id": "F3", "kind": "claimed_speedup", "family": "Many extras in the requirement", "why": "Adding __slots__ makes attribute access (including the extras set) faster. Constructing requirements with a growing number of extras stresses the fast path for attribute storage.", "size_meaning": "Number of extra names inside the brackets", "make_source": "\ndef make(n):\n # Build a requirement with n extras: pkg[extra0,extra1,...]\n extras = ','.join(f'extra{i}' for i in range(n))\n req = f'pkg[{extras}]'\n return [[req]]", "example_args": [ [ "pkg[extra0,extra1,extra2]" ] ], "origin": "plan", "screen_ratio": 1.0014311756743892, "valid_fraction": 1.0, "error": null }, { "id": "F4", "kind": "claimed_speedup", "family": "requirements_with_extras", "why": "Slots also accelerate assignment of the mutable ``extras`` set and the ``marker`` object. Requirements that carry several extras exercise those slots more heavily.", "size_meaning": "number of extras per requirement (and number of objects)", "make_source": "def make(n):\n # n extras, each like 'extra{i}'\n extras = ','.join(f'extra{i}' for i in range(n))\n req = f'pkg[{extras}]==1.0.0'\n return [[req] for _ in range(8)]", "example_args": [], "origin": "plan", "screen_ratio": 1.005152997623272, "valid_fraction": 1.0, "error": null }, { "id": "F5", "kind": "suspected_slowdown", "family": "Unpickling a legacy ``__dict__`` that contains exactly the five slot keys.", "why": "Old code did ``self.__dict__.update(state)`` (single C call). New code checks ``state.keys() >= set(self.__slots__)`` then loops over the five slot names doing ``setattr`` – extra Python‑level work for every unpickle.", "size_meaning": "Number of extra dummy keys added to the dict (the dict always contains the five real keys plus *n* harmless extra keys).", "make_source": "def make(n):\n # Build a list of state dicts that mimic the pre‑26.1 pickle format.\n # Each dict has the five required keys plus *n* additional keys.\n base = {\n \"name\": \"demo\",\n \"url\": None,\n \"extras\": {\"dev\"},\n \"specifier\": \"==1.0\",\n \"marker\": None,\n }\n out = []\n for i in range(1, n+1):\n d = dict(base)\n for k in range(i):\n d[f\"_junk{k}\"] = k\n out.append([d])\n return out", "example_args": [ [ { "name": "demo", "url": null, "extras": [ "dev" ], "specifier": "==1.0", "marker": null, "_junk0": 0 } ], [ { "name": "demo", "url": null, "extras": [ "dev" ], "specifier": "==1.0", "marker": null, "_junk0": 0, "_junk1": 1 } ] ], "origin": "plan", "screen_ratio": null, "valid_fraction": null, "error": "make(2): TypeError: Object of type set is not JSON serializable" }, { "id": "F6", "kind": "suspected_slowdown", "family": "Long comma‑separated specifier list", "why": "The parser must split and normalise each specifier. Although the change does not touch parsing, a very long specifier list exercises the unchanged hot path and can dominate any tiny overhead from the new __slots__ initialisation.", "size_meaning": "Number of individual specifiers", "make_source": "def make(n):\n # Create n simple specifiers like '>=1.0', '<2.0', '!=1.5', ...\n specs = []\n for i in range(n):\n op = ['>=', '<=', '==', '!=', '> ', '< '][i % 6]\n specs.append(f'{op}{i}.0')\n req = 'pkg' + ','.join(specs)\n return [[req]]", "example_args": [ [ "pkg>=0.0,<=1.0,==2.0,!=3.0,>4.0,<5.0" ] ], "origin": "plan", "screen_ratio": 1.0012540382750987, "valid_fraction": 1.0, "error": null } ], "findings": [ { "family_id": "F2", "kind": "claimed_speedup", "family": "Repeated attribute reads on many Requirement instances (slots make attribute lookup faster and reduce per‑object memory).", "why": "The diff adds ``__slots__``; slots eliminate the per‑instance ``__dict__`` and make attribute access a direct C‑array lookup. Code that creates many Requirement objects and then reads their fields (name, extras, specifier, url, marker) will benefit.", "origin": "plan", "screen_ratio": 0.9868666927862122, "valid_fraction": 1.0, "ladder": [ { "n": 2, "base_ir": 163384.0875, "head_ir": 163518.1875, "input_chars": 21.0, "ratio": 1.0008207653637018, "valid_fraction": 1.0, "judged": true }, { "n": 4, "base_ir": 176843.78324468085, "head_ir": 176921.89361702127, "input_chars": 24.5, "ratio": 1.0004416913668508, "valid_fraction": 1.0, "judged": true }, { "n": 16, "base_ir": 246788.12259615384, "head_ir": 243546.97836538462, "input_chars": 45.875, "ratio": 0.9868666927862122, "valid_fraction": 1.0, "judged": true }, { "n": 64, "base_ir": 524400.578125, "head_ir": 525517.421875, "input_chars": 138.25, "ratio": 1.0021297530868354, "valid_fraction": 1.0, "judged": true }, { "n": 256, "base_ir": 896535.5, "head_ir": 902474.40625, "input_chars": 265.546875, "ratio": 1.0066242845375337, "valid_fraction": 1.0, "judged": true }, { "n": 1024, "base_ir": 896535.5, "head_ir": 902474.40625, "input_chars": 265.546875, "ratio": 1.0066242845375337, "valid_fraction": 1.0, "judged": true }, { "n": 4096, "base_ir": 896535.5, "head_ir": 902474.40625, "input_chars": 265.546875, "ratio": 1.0066242845375337, "valid_fraction": 1.0, "judged": true } ], "slope_base": 0.680145413565356, "slope_head": 0.6838090635837745, "top_inputs": [], "error": null, "verdict": "neutral", "worst_ratio": 1.0066242845375337, "worst_search": null } ], "seconds": { "plan": 20.04, "screen": 105.02, "full": 92.6, "total": 217.65 }, "ladder_validity": "backfilled 2026-10-11 (scripts/backtest/backfill_ladder_validity.py)" } }, "image_rescored": "bw-pair:f8d0aa2a4ebfdaf6" } ] }, { "id": "#1371", "pr": 1371, "title": "perf(markers): speed up _format_marker and benchmark str(Marker)", "base": "46980084996620ed1606765f7c5946c99d3c0618", "head": "9483cb8a744606bdbad0438f01d3026a1ce4826f", "merged": "2026-09-25", "after_ultra_release": true, "known": [], "targets": [ { "target": "packaging.markers:Marker", "run_id": "bt-1371-f2f7ba", "image": "bw-pair:6515042e553ea90a", "usd": 0.1391, "harvested": 80, "arms": { "hypothesis": { "classes": [], "known": [], "seconds": 4.1, "error": null }, "mutations": { "classes": [], "known": [], "seconds": 1.6, "error": null }, "breakwater": { "classes": [], "known": [], "seconds": 52.7, "error": null }, "llm-plan": { "classes": [], "known": [], "seconds": 5.6, "error": null } }, "findings": [], "perf": { "slower_families": [], "worst_valid_ratio": null, "worst_family": null, "speedup_ratio_small": null, "report": { "run_id": "bt-1371-f2f7ba", "target": "packaging.markers:Marker", "image": "bw-pair:6515042e553ea90a", "backend": "docker", "families": [ { "id": "F1", "kind": "suspected_slowdown", "family": "Marker evaluate", "why": "Evaluating many markers against an environment is the core hot path; changes to marker evaluation logic often regress runtime.", "size_meaning": "ops_per_iteration", "make_source": "def make(n):\n # Deterministic list of argument sets for evaluating n Marker instances.\n # ops_per_iteration grows with n, using math.comb to increase cost.\n # Seed is pre-set; we use it to produce deterministic ops count.\n ops = n * (n + 1) // 2 # quadratic growth\n args_list = []\n for i in range(1, min(ops, 8) + 1):\n # Build a Marker-evaluation argument tuple.\n # Each tuple contains positional args representing environment checks.\n args = tuple(string.ascii_lowercase[j % 26] for j in range(i))\n args_list.append(list(args))\n # If fewer than 8 were generated, pad with empty lists to hit the cap.\n while len(args_list) < 8:\n args_list.append([])\n return args_list", "example_args": [], "origin": "plan", "screen_ratio": 1.0013499639249104, "valid_fraction": 0.0, "error": null }, { "id": "F2", "kind": "suspected_slowdown", "family": "Marker evaluate", "why": "Marker evaluation logic unchanged but new marker formatting path may add overhead during evaluation.", "size_meaning": "lower is better", "make_source": "def make(n):\n # Generate between 1 and 8 argument lists, each of length 3.\n # The total number of operations is O(n * list_length), so cost grows with n.\n import itertools as it\n \n count = it.count(1).__next__ # Not used directly, but demonstrates itertools usage\n k = 1 + (n % 8) # 1 to 8\n args = []\n for _ in range(k):\n # Create a tuple of 3 ints: base, offset, and a multiplier derived from n.\n # Using math.prod and math.comb to involve math module without external imports.\n a = 1 + (n * 7) % 100\n b = 1 + (n * 13) % 50\n c = 1 + (n * 17) % 30\n args.append([a, b, c])\n return args", "example_args": [], "origin": "plan", "screen_ratio": null, "valid_fraction": null, "error": "make(n) rejected: imports are not allowed" }, { "id": "F3", "kind": "claimed_speedup", "family": "Marker str", "why": "Stringifying markers is a hot path for logging/serialization; the new __str__ implementation removes redundant list‑wrapping and recursion, promising a speedup.", "size_meaning": "ops_per_iteration", "make_source": "def make(n):\n ops = n\n result = []\n for _ in range(ops):\n marker = \"Development Status :: 5 - Production/Stable :: \" + ''.join(random.choices(string.ascii_letters, k=8))\n result.append([marker])\n return result", "example_args": [], "origin": "plan", "screen_ratio": 1.0070737808404033, "valid_fraction": 0.0, "error": null }, { "id": "F4", "kind": "claimed_speedup", "family": "Marker str", "why": "The _format_marker fast path for tuple markers avoids list/tuple checks and uses direct tuple unpacking, speeding up str(marker) benchmark added in benchmarks/markers.py.", "size_meaning": "lower is better", "make_source": "\ndef make(n):\n chars = string.ascii_letters + string.digits\n return [\n [random.choice(chars) for _ in range(math.ceil(math.log2(i + 2)))]\n for i in range(min(n, 8))\n ]", "example_args": [], "origin": "plan", "screen_ratio": 1.003845961437863, "valid_fraction": 0.0, "error": null }, { "id": "F5", "kind": "suspected_slowdown", "family": "Marker parse", "why": "Marker parsing from PEP 508 strings is exercised by the test suite; parser changes can regress parsing latency.", "size_meaning": "ops_per_iteration", "make_source": "def make(n):\n sizes = list(range(1, min(n, 8) + 1))\n return [\n [str(s)]\n for s in sizes\n ]", "example_args": [], "origin": "plan", "screen_ratio": 1.0068394357288137, "valid_fraction": 0.0, "error": null }, { "id": "F6", "kind": "suspected_slowdown", "family": "Marker eval context", "why": "Evaluation under different contexts (metadata, lock_file, requirement) exercises different environment builds; changes to environment building may regress.", "size_meaning": "ops_per_iteration", "make_source": "def make(n):\n ops = list(range(1, min(n, 8) + 1))\n return [ [op] for op in ops ]", "example_args": [], "origin": "plan", "screen_ratio": 1.0009400684335263, "valid_fraction": 0.0, "error": null } ], "findings": [], "seconds": { "plan": 62.02, "screen": 107.71, "full": 0.0, "total": 169.73 }, "ladder_validity": "backfilled 2026-10-11 (scripts/backtest/backfill_ladder_validity.py)" } } } ] }, { "id": "#1429", "pr": 1429, "title": "perf(parser): avoid literal_eval for plain marker strings", "base": "f7c930eb18b556930564c1131fde78e576352ddc", "head": "d17cfaf0b77a5690f03087359e7c8d665a3c728d", "merged": "2026-10-05", "after_ultra_release": true, "known": [], "targets": [ { "target": "packaging.markers:Marker", "run_id": "bt-1429-0619ec", "image": "bw-pair:11c432119443010a", "usd": 0.0512, "harvested": 80, "arms": { "hypothesis": { "classes": [], "known": [], "seconds": 5.9, "error": null }, "mutations": { "classes": [], "known": [], "seconds": 3.3, "error": null }, "breakwater": { "classes": [], "known": [], "seconds": 48.7, "error": null }, "llm-plan": { "classes": [], "known": [], "seconds": 3.9, "error": null } }, "findings": [], "perf": { "slower_families": [], "worst_valid_ratio": null, "worst_family": null, "speedup_ratio_small": null, "report": { "run_id": "bt-1429-0619ec", "target": "packaging.markers:Marker", "image": "bw-pair:11c432119443010a", "backend": "docker", "families": [ { "id": "F1", "kind": "claimed_speedup", "family": "fast_path_string_parsing", "why": "The patch adds a fast‑path in `process_python_str` that returns a `Value` by stripping the surrounding quotes when the string contains no backslashes and consists only of printable characters, avoiding the overhead of `ast.literal_eval`. This avoids the overhead of the AST parser for the common case of simple quoted strings used in environment markers.", "size_meaning": "Micro‑optimisation for a hot parsing path; avoids the heavyweight AST parser for the common case of simple quoted strings in environment markers.", "make_source": "def make(n):\n return [[\"'cpython'\"] for _ in range(min(n, 8))]", "example_args": [ [ "'cpython'" ], [ "'cpython'" ] ], "origin": "plan", "screen_ratio": 0.6644936713780933, "valid_fraction": 0.0, "error": null }, { "id": "F2", "kind": "claimed_speedup", "family": "marker_evaluation_fast_path", "why": "Added a fast‑path in `process_python_str` that avoids `ast.literal_eval` for simple printable string literals without backslashes. Simple environment markers (e.g. `extra == 'linux'`) are common, so skipping the full Python literal parser saves the relatively heavy C‑level parsing overhead for the common case.", "size_meaning": "speedup", "make_source": "def make(n):\n return [\n [str(i), str(i % 2 == 0)]\n for i in range(1, min(n, 8) + 1)\n ]", "example_args": [], "origin": "plan", "screen_ratio": 1.0, "valid_fraction": 0.0, "error": null }, { "id": "F3", "kind": "claimed_speedup", "family": "parser_optimization", "why": "Added fast‑path in process_python_str that returns the literal string slice when the string contains no backslashes and is printable, avoiding the slower ast.literal_eval call.", "size_meaning": "bytes of code change", "make_source": "\ndef make(n):\n base = \"Marker(\"\n changes = [\n (\"optimize\", \"\"),\n (\"opt1\", \"\"),\n (\"opt2\", \"\"),\n (\"opt3\", \"\"),\n ]\n result = []\n for mask in range(1, 9):\n args = []\n for i, (name, _) in enumerate(changes):\n if mask & (1 << i):\n args.append(name)\n result.append(args)\n return result", "example_args": [], "origin": "plan", "screen_ratio": 0.9943055591574943, "valid_fraction": 0.0, "error": null }, { "id": "F4", "kind": "suspected_slowdown", "family": "extra_isprintable_check_overhead", "why": "The fast‑path adds an `isprintable()` call and a backslash check for every marker string. For very short marker strings this extra Python‑level check may outweigh the saved `ast.literal_eval` cost, especially on very short strings where the AST parser is already fast.", "size_meaning": "Micro‑overhead per call; measurable only in tight loops parsing many markers.", "make_source": "def make(n):\n # deterministic short marker strings based on n\n s = string.digits[:n % 6 + 1] # '0'..'5' -> 1..6 chars\n args = [['' + s]]\n if n > 3:\n args.append(['' + s + s])\n if n > 5:\n args.append(['' + s + s + s])\n return args", "example_args": [ [ "'cp'" ], [ "'py3'" ] ], "origin": "plan", "screen_ratio": 0.9933299701811983, "valid_fraction": 0.0, "error": null }, { "id": "F5", "kind": "suspected_slowdown", "family": "marker_evaluation_fallback_overhead", "why": "For string literals that contain a backslash (e.g. escaped quotes or escape sequences) the fast‑path test `\"\\\\\" not in python_str` scans the whole token, then `str.isprintable()` scans it again before falling back to `ast.literal_eval`. This double scan adds overhead for the relatively rare escaped strings.", "size_meaning": "slowdown", "make_source": "def make(n):\n rnd = random.Random(n)\n parts = [\"\".join(rnd.choice(string.ascii_letters + string.digits) for _ in range(rnd.randint(10, 60))) for _ in range(n)]\n backs = [\"\\\\\" + c for c in string.ascii_letters[:8]]\n args = []\n if n >= 1:\n args.append([parts[0], rnd.choice(backs)])\n if n >= 2:\n args.append([parts[1], parts[0], rnd.choice(backs)])\n if n >= 3:\n args.append([parts[2], parts[1], parts[0], rnd.choice(backs)])\n if n >= 4:\n args.append([parts[3], parts[2], parts[1], parts[0], rnd.choice(backs)])\n if n >= 5:\n args.append([parts[4], parts[3], parts[2], parts[1], parts[0], rnd.choice(backs)])\n if n >= 6:\n args.append([parts[5], parts[4], parts[3], parts[2], parts[1], parts[0], rnd.choice(backs)])\n if n >= 7:\n args.append([parts[6], parts[5], parts[4], parts[3], parts[2], parts[1], parts[0], rnd.choice(backs)])\n if n >= 8:\n args.append([parts[7], parts[6], parts[5], parts[4], parts[3], parts[2], parts[1], parts[0], rnd.choice(backs)])\n return args", "example_args": [], "origin": "plan", "screen_ratio": 1.0016731734523145, "valid_fraction": 0.0, "error": null }, { "id": "F6", "kind": "suspected_slowdown", "family": "edge_case_regression", "why": "The fast‑path skips ast.literal_eval for any string without a backslash, but strings containing escaped non‑printable characters (e.g., '\\x00') are printable yet not literal strings; the fast‑path would return the literal escaped text instead of the evaluated character, causing incorrect marker evaluation.", "size_meaning": "bytes of code change", "make_source": "def make(n):\n r = []\n if n >= 1:\n r.append([None])\n if n >= 2:\n s = \"\\\\x00\"\n r.append([s])\n if n >= 3:\n s = \"\\\\\\\\\"\n r.append([s])\n if n >= 4:\n s = \"\\\\n\"\n r.append([s])\n if n >= 5:\n r.append([\"test\"])\n if n >= 6:\n r.append([\"repr\\\\test\"])\n if n >= 7:\n r.append([\"\\\\x00\"])\n if n >= 8:\n r.append([\"\\\\10\"])\n return r", "example_args": [], "origin": "plan", "screen_ratio": 0.9939663600404437, "valid_fraction": 0.0, "error": null } ], "findings": [], "seconds": { "plan": 13.43, "screen": 76.45, "full": 0.0, "total": 89.88 }, "ladder_validity": "backfilled 2026-10-11 (scripts/backtest/backfill_ladder_validity.py)" } }, "image_rescored": "bw-pair:dac87e5ea56680f4" } ] } ], "summary": { "items": 29, "known_verified": 9, "known_candidate": 3, "caught_verified": { "hypothesis": 2, "mutations": 7, "llm-plan": 5, "breakwater": 9 }, "caught_candidate": { "hypothesis": 0, "mutations": 1, "llm-plan": 1, "breakwater": 1 }, "items_with_classes": { "hypothesis": 4, "mutations": 9, "llm-plan": 8, "breakwater": 9 }, "usd": 2.594, "perf_items_slower": 5, "provider": "tokenfactory" }, "reruns": { "items": [ "#1111", "#1258" ], "reason": "#1111 transient APIConnectionError; #1258 harness crash (fixed)" }, "corrections": [ { "item": "#1030", "target": "packaging.utils:canonicalize_name", "image": "bw-pair:d09088ff31a7e098", "removed": { "hypothesis": [ "P4|base->head eq-fresh|True|False" ], "mutations": [ "P4|base->head eq-fresh|True|False" ], "breakwater": [ "P4|base->head eq-fresh|True|False" ], "llm-plan": [ "P4|base->head eq-fresh|True|False" ] }, "note": "P4 control fix: a base-pickled object failing to equal a fresh head object only counts if two fresh head objects compare equal; builtin return values are skipped (P4 would restate P1)" }, { "item": "#1064", "target": "packaging.utils:canonicalize_name", "image": "bw-pair:810b943be062d4e3", "removed": { "hypothesis": [ "P4|base->head eq-fresh|True|False" ], "mutations": [ "P4|base->head eq-fresh|True|False" ], "breakwater": [ "P4|base->head eq-fresh|True|False" ], "llm-plan": [ "P4|base->head eq-fresh|True|False" ] }, "note": "P4 control fix: a base-pickled object failing to equal a fresh head object only counts if two fresh head objects compare equal; builtin return values are skipped (P4 would restate P1)" }, { "item": "#1081", "target": "packaging.specifiers:Specifier", "image": "bw-pair:ff8050a1e55ba0d9", "removed": { "mutations": [ "P4|base->head eq-fresh|True|" ], "breakwater": [ "P4|base->head eq-fresh|True|" ] }, "note": "P4 control fix: a base-pickled object failing to equal a fresh head object only counts if two fresh head objects compare equal; builtin return values are skipped (P4 would restate P1)" }, { "item": "#1106", "target": "packaging.specifiers:Specifier", "image": "bw-pair:6c9a610c08b433b1", "removed": { "mutations": [ "P4|base->head eq-fresh|True|", "P4|base->head eq-fresh|True|" ], "breakwater": [ "P4|base->head eq-fresh|True|" ] }, "note": "P4 control fix: a base-pickled object failing to equal a fresh head object only counts if two fresh head objects compare equal; builtin return values are skipped (P4 would restate P1)" }, { "item": "#1258", "target": "packaging.requirements:Requirement", "image": "bw-pair:bc22d3399fde3139", "removed": { "mutations": [ "P4|base->head eq-fresh|True|" ], "breakwater": [ "P4|base->head eq-fresh|True|" ] }, "note": "P4 control fix: a base-pickled object failing to equal a fresh head object only counts if two fresh head objects compare equal; builtin return values are skipped (P4 would restate P1)" }, { "item": "#1320", "target": "packaging.requirements:Requirement", "image": "bw-pair:f8d0aa2a4ebfdaf6", "removed": { "mutations": [ "P4|base->head eq-fresh|True|" ], "breakwater": [ "P4|base->head eq-fresh|True|" ] }, "note": "P4 control fix: a base-pickled object failing to equal a fresh head object only counts if two fresh head objects compare equal; builtin return values are skipped (P4 would restate P1)" }, { "item": "#1115", "target": "packaging.version:Version", "image": "bw-pair:b75c39f11bbb1fbf", "removed": { "mutations": [ "P4|base->head eq-fresh|True|False" ], "breakwater": [ "P4|base->head eq-fresh|True|False" ], "llm-plan": [ "P4|base->head eq-fresh|True|False" ] }, "note": "P4 round-trip control: a pickle class only counts if the object survives a round trip on the version that pickled it" }, { "item": "#1429", "target": "packaging.markers:Marker", "image": "bw-pair:dac87e5ea56680f4", "removed": { "breakwater": [ "P4|base->head load|loads|exc:builtins.TypeError" ], "llm-plan": [ "P4|base->head load|loads|exc:builtins.TypeError" ] }, "note": "P4 round-trip control: a pickle class only counts if the object survives a round trip on the version that pickled it" } ] }