- The paper introduces SIDScope, a reproducible resource that evaluates semantic-ID mappings across seven diagnostic dimensions, including code utilization, aliasing, prefix alignment, trie structure, refresh churn, and generated-path resolution.
- The paper finds that prefix alignment strongly predicts prefix-based candidate recall (Spearman ρ=0.976) but transfers inconsistently to trained generators, showing that diagnostic metrics must be interpreted according to the mechanism they measure.
- The paper shows that valid generated paths can exceed unique-item retrieval by 1.2–3.0 percentage points because full-code aliasing creates ambiguous resolutions, while constrained decoding eliminates invalid paths without necessarily improving Recall@20.
Overview and motivation
Semantic-ID (SID) generative recommendation converts item identifiers into discrete code sequences that a sequence generator emits as recommendations. Once a tokenizer exports its item-to-code mapping, that mapping functions as a reusable address space: it must preserve item coverage, expose useful prefixes, avoid harmful aliasing, and remain decodable back to items. SIDScope treats this exported mapping as the primary object of evaluation rather than treating downstream ranking metrics as the sole endpoint. The resource extends the authors' prior CIKM artifact, SIDInspector (2608.18779), from a static mapping audit to a persistent artifact record connecting admission, mapping state, candidate exposure, refreshes, and generated traces.
The paper's central premise is that interface health is multi-signal. Full-code uniqueness is necessary for addressability but does not determine whether useful items are exposed through prefixes; prefix organization can improve exposure while concentrating capacity or increasing collision risk. The resource therefore maintains these coordinates as separate diagnostics D1–D7: code-space utilization (D1), shared-address risk (D2), prefix–behavior alignment (D3), head–tail resolution allocation (D4), trie structure (D5), paired refresh churn (D6), and path-to-item resolution of generated traces (D7).
Artifact contract and admission
SIDScope normalizes an artifact as a record R=(I,c,P;M,X,R,T) joining the item catalog, code assignment, provenance, and optional metadata, interactions, refresh pairs, and traces. Admission requires passing six ordered conformance gates C0–C5 covering source identity and license terms, schema coherence, joins to metadata and interactions, bounded diagnostic execution, replay identity via SHA-256 hashes, and inventory-role consistency. A public failure fixture demonstrates that C1 rejects a row whose full sid disagrees with its sid_level_* fields while the other gates remain independently observable.
The release covers nine source-traced tokenizer exports from seven method families on Amazon and Yelp data—eight executable routes plus one auditable snapshot (ReSID/Musical)—spanning ReSID-GAOQ, GRID, CARD, DIGER (Beauty and Yelp checkpoints), ReSOT, LETTER, and LC-Rec. Stress rows (e.g., a GRID-like Musical export with 23,742 items but only 3,749 unique full codes, a full-code aliasing rate of 0.977) and deterministic category-prefix controls calibrate diagnostic behavior without counting as named-method coverage. Licensing is enforced as an executable package boundary: routes with restricted or undetected licenses publish summaries and hashes only, not raw archives.
The paper is explicit about comparison modes. Same-catalog controlled comparisons can attribute differences to the inspected mapping under a fixed protocol; cross-route profile comparisons are descriptive and explicitly must not be read as a tokenizer leaderboard; longitudinal comparisons require paired mappings with stable item joins.
Mapping profiles across admitted routes
The D1–D5 profiles show that the diagnostic coordinates vary independently across artifacts. CARD and DIGER use similar alphabets on the same Beauty catalog yet differ in collision exposure (0.000 versus 0.070 full-code collision-item rates) and tail uniqueness. LETTER and LC-Rec share item count, D2 rate, and tail ratio but differ in depth-1 weighted co-occurrence recall (0.108 versus 0.055 under the bounded protocol). Within DIGER, the Yelp route shows more collided items (0.102), weaker depth-1 alignment (0.010), and lower tail uniqueness than the Beauty route—a contrast demonstrating contract portability across ecosystems, though the authors caution it is not a causal domain effect because catalogs, checkpoints, and interactions differ.
A robustness check recomputing D3 over m∈{5,10,20} and U∈{50,100,200} leaves route ordering unchanged in all nine configurations (Spearman ρ=1.0 against the primary ordering), supporting stability within the bounded neighborhood family.
Construct calibration and its mechanism-conditional reach
The most consequential empirical result concerns what D3 can and cannot predict. The evidence ladder shows strong calibration when both D3 and the outcome consume prefix organization: per-artifact Spearman correlation between D3 and prefix-candidate recall reaches ρ=0.976 (exact p=0.0004) over eight tokenizer exports after removing controls, and remains ρ=0.976 [0.793, 1.000] when evaluation users are disjoint from D3 construction. A controlled model estimates a standardized D3 coefficient of 0.120 [0.067, 0.172] over 1,080 rows clustered by 12 artifact keys with Rademacher wild-cluster intervals.
The association then degrades systematically as the consumer becomes prefix-independent. With a fixed SID-affinity ranker and hard negatives, correlations fall to 0.645 for Recall@20 and 0.545 for NDCG@20 at the artifact–depth level; collapsing hard-negative NDCG to eight artifacts keeps the point estimate but widens the interval across zero [-0.095, 0.988]. Metadata-category scorers yield ρ=0.207 and random-negative pools ρ=0.001. Critically, Block D finds no stable transfer to trained generators: autoregressive SID generators produce sign-unstable associations (ρ ranging from -0.564 to 0.205, exact m∈{5,10,20}0), partly because the available trained models fail an item-popularity validity check. The paper's conclusion is deliberately narrow: D3 diagnoses prefix-interface organization, and trained-generator quality remains a separate empirical question. This mechanism-conditional framing is the paper's strongest claim about scope, and it is stated as a limitation of the metric rather than a generalization failure.
The five-catalog collapse (m∈{5,10,20}1, exact m∈{5,10,20}2) is reported honestly as directionally stable but insufficient for population-level claims, since several exports share catalogs and interaction logs.
Generated-trace accounting
D7 extends the adapter principle to decoded beams with a constraint-aware label taxonomy. Invalid and unresolved out-of-trie paths are classified as unconstrained-only labels that should disappear under prefix-constrained decoding; the taxonomy separately tracks duplicate items, duplicate paths, ambiguous paths, stale/out-of-catalog resolutions, and high uncertainty, which survive constrained decoding by construction.
Three findings stand out. First, constraint handling works as expected: on the released DACT TIGER/T5 checkpoint, constrained decoding produces zero invalid paths while unconstrained decoding yields 988 invalid rows among 25,000 beams affecting 205 targets; constrained decoding additionally recovers four targets unconstrained decoding misses (39/500 versus 35/500), with Recall@20 unchanged at 0.042. Second, and more substantively, target-path survival exceeds unique-item hit by 1.2–3.0 percentage points across trained constrained-beam runs: 6.4% versus 3.4% on GRID/P5 fold 0, 5.0% versus 3.0% on fold 1, and 6.6% versus 5.4% on the DIGER sensitivity route. A valid generated path can therefore survive decoding without uniquely retrieving the target item, because full-code aliasing permits ambiguous reverse resolution. Ambiguous-row prevalence ranges from 10.1% (DIGER) to 34.0–37.1% (GRID/P5); the authors note they did not intervene on beam width holding mapping fixed, so this establishes sensitivity and portability, not a causal allocation of ambiguity between mapping and generator. Third, the released package includes 125,000 deidentified labeled beam rows bound by hash, enabling external verification of the label distributions.
Lifecycle: refresh and generator handoff
The DACT case exercises D6 on a preregistered single-lifecycle study. Mapping 0.6 leaves 275 catalog items (263 interacted) without a SID; mapping 0.7 closes both gaps but changes 23.6% of common-item codes (2,271 of 9,610), introduces nonzero full-code collision (0.000607), and shifts depth-1 D3 from 0.0280 to 0.0306. The preregistered handoff gate—recovery of at least 90% of common-item NDCG loss plus nonzero new-item Recall@20, with thresholds computed before adapted-seed outcomes were inspected—fails for the unadapted released model: new-item Recall@20 is exactly 0.000 [0.000, 0.000] despite common-item NDCG being statistically indistinguishable from baseline (13.84 vs. 13.94, scaled by m∈{5,10,20}3). All three same-architecture adapted models pass the gate in all 4,999 bootstrap resamples, reaching new-item recall of 0.120–0.127 and common-item NDCG of roughly 20.3–21.0.
Two caveats are stated plainly. The mapping-only common-item change is m∈{5,10,20}4 m∈{5,10,20}5, so the case does not resolve whether the mapping swap itself harms common-item ranking—the point estimate simply misses the predeclared threshold. And the workflow is validated on one lifecycle case only; general repair effectiveness requires additional cases.
Reproducibility and availability
The tagged release (v1.0.1, MIT license) provides adapters, CLI, eight frozen C0–C5 reports, eight table snapshots, four figure records, the D7 trace labels, and CPU-only verifiers. Two replay levels are supported: from the checkout alone, users can validate the nine-route inventory, inspect conformance reports, rerun the failure fixture, and rebuild all tables deterministically; supplying the upstream ReSOT archive enables re-execution of source-dependent intake. Raw upstream inputs remain at their sources with recorded revisions, paths, and hashes, keeping licensing boundaries executable rather than declarative.
Limitations and open questions
The paper is explicit about validated scope. Transfer of D3 to validity-passing trained generators remains an open empirical question, since the available trained models failed the popularity-validity check. The inventory contains one non-Amazon (Yelp) route alongside eight Amazon-derived routes, so broad cross-ecosystem validation awaits additional non-Amazon artifacts. D6 and generator-handoff are exercised on a single released DACT case; causal effects of mapping properties on recommendation errors would require intervention studies the resource does not perform. Variable-length SID routes require explicit termination conventions and length-stratified reporting, which the current fixed-depth matrix does not accommodate. Extensions to temporal, closed-loop, and user-token SID artifacts are outlined as contract extensions but not validated here.
Conclusion
SIDScope contributes a source-traced inspection chain from artifact admission through mapping diagnosis, calibrated exposure analysis, trace accounting, and lifecycle handoff. Its three principal findings—that interface health is multi-signal rather than scalar, that prefix alignment's predictive reach is strictly mechanism-conditional, and that valid-path survival diverges measurably from unique-item retrieval in trained beams—are each bounded by explicit protocols and acknowledged limitations. The resource gives tokenizer, generator, and lifecycle studies a shared, reproducible vocabulary for reporting what changed in the exported address space, complementing rather than replacing downstream model evaluation.
