Backend check · Token Factory Sandboxes

Same PRs, same probes, a microVM per run.

8 of the backtest's PRs were audited again with every untrusted run in Token Factory Sandboxes: one microVM branched from a kept image per probe batch, search and cachegrind measurement. The published backtest ran on local Docker. Verdicts should agree; instruction counts should agree to within a few percent (same instruction stream, different CPUs). Each image below reproduces its numbers.

5/8

same verdict on both backends

local Docker vs Token Factory Sandboxes

6/6

same known issues caught

items with documented regressions

44%

largest Ir ratio gap on a shared family

3 families measured on both with identical inputs

$1.12

Nemotron cost of the Token Factory Sandboxes run

Token Factory, backtest + adjudication

Item by item

PR	Change	Docker	Sandboxes	Known caught (Docker / Sandboxes)	Perf worst, Docker	Perf worst, Sandboxes	Images (Docker / Sandboxes)
#1030	perf: translate for canonicalize_name`canonicalize_name`	BLOCK	BLOCK same	unicode-lowercase / unicode-lowercase	4.52×many_underscores_and_dots	6.50×leading_and_alternating_pattern	`bw-pair:d09088ff31a7e098` `a2d83f6b-ca9d-4fca-95ce-558082aac714`
#1064	perf: dual replace`canonicalize_name`	PASS	BLOCK differs	–	1.10×many_underscores_and_dots	1.09×Normal package names are short‑ish alphanumerics w	`bw-pair:810b943be062d4e3` `bfdb727b-8008-42aa-8523-32888cb60192`
#1082	perf: Add fast path for parsing simple versions (digits and dots only)`Version`	BLOCK	REVIEW differs	oversized-int-exception / oversized-int-exception	2.85×Each numeric chunk increases digit count linearly	2.91×Leading zeros in first segment	`bw-pair:3f8cc97a2a8a52dc` `cdd80561-4ef0-4ca1-b645-b13306e326c6`
1082@61967cca	perf: fast path for simple versions (intermediate commit 61967cca)`Version`	BLOCK	BLOCK same	inner-whitespace, oversized-int-exception, unicode-digits / inner-whitespace, oversized-int-exception, unicode-digits	1.91×versions with an epoch (e.g. "1!2.3.4")	2.86×Leading zeros in first segment	`bw-pair:348e2d05c100a5f9` `62d98fe8-9c1e-42e6-b250-ab6f630d8680`
#1108	perf: SpecifierSet use tuple instead of frozenset for `_specsSpecifierSet`	BLOCK	BLOCK same	dedup-equality, dedup-str-len, old-pickles-break-comparison / dedup-equality, dedup-str-len, old-pickles-break-comparison	–	–	`bw-pair:16f20a2b68c37e6b` `052dfee2-22cb-4d9b-b081-e27b77a69db0`
#1116	perf: Overhaul `_cmpkey` to remove use of custom objects`Version`	BLOCK	BLOCK same	old-pickles-fail-to-load / old-pickles-fail-to-load	–	–	`bw-pair:ee5fda034cdd7462` `e7fbcc0c-cc99-4f27-9713-ec040778674e`
#1147	perf: add __slots__ to Marker`Marker`	BLOCK	BLOCK same	marker-pickle / marker-pickle	–	–	`bw-pair:e33c8d8e1355cf11` `583e9e77-230e-4727-b18e-e799efeb03f4`
#1236	perf: simply work done in parse_tag a bit`parse_tag`	PASS	REVIEW differs	–	1.09×tiny_tag	1.09×simple tag with one interpreter, one abi and one p	`bw-pair:d9b82063b49a1e43` `5c5765e3-5a87-48f2-8b21-497e56c1292b`

Verdict: behaviour (adjudicated classes) combined with performance, as on every PR page. Known caught: the documented regressions each run's full pipeline found.
Perf worst: the highest head/base instruction ratio over input families where most inputs are valid. The adversary plans families anew on each run, so the worst family can differ; archived families are shared between runs and are what the ratios are compared on.
Shared families, head/base Ir: Names starting/ending with multiple dots: 2.28× vs 2.46× (8%); many_underscores_and_dots: 4.52× vs 6.50× (44%); many_underscores_and_dots: 1.10× vs 1.08× (2%). The two backends run different CPU architectures, so cachegrind counts different instruction streams; within one backend Ir repeats to the instruction.
Backend: where the untrusted code ran. Token Factory Sandboxes (a microVM branched from a kept image, cited by its UUID); local Docker (an image cited by its content-addressed tag). Every image is content-addressed (repo at each commit + harness), so every card re-runs from the image and seed 0.