Test transferability across states, narrative conventions, and model versions

Test whether the Jev crash-narrative coding schema, cost model, recalibration procedure, and reported accuracy and review budgets transfer to other states, agency-specific narrative conventions, and subsequent Jev model versions.

Background

The empirical study covers one state, one commercial model version, and one schema designed around Texas crash-reporting conventions. The paper identifies several possible failure modes: longer narratives could invalidate the cost model, different reporting conventions could reduce schema performance, different prevalence could invalidate the recalibration map, and later model versions could change accuracy and calibration. The authors therefore leave transferability unresolved and call for replication in other states and repeated audits across model versions using the same reference set and metric code.

References

Transfer is a hypothesis here rather than a finding, because this study covers one state, one model version and one schema, and it is stated with the conditions under which it would fail. The cost model would fail where narratives are much longer than the 378 characters of the median here, since the text term would then stop being small beside the schema term. The schema would fail where officers write to a different convention, because the criteria text was written against Texas shorthand and a construct such as an intentional act is recognized through the phrases a reporting culture happens to use. The recalibration map would fail wherever the prevalence differs, since a map fitted at one base rate moves the probabilities toward that rate and not toward another. The accuracy results would fail on any later model version, because the audit measures a model and a schema together rather than a capability. Each of these is testable in one state with a few hundred labeled narratives, which is the cost this study reports, and until that test is run the transfer claim should be read as a design expectation.

— Calibrated Decisions at Scale: Converting Police Crash Narratives into Probabilistic Crash Variables with a System One Model (Jev)  (2609.24052 - Rafe et al., 21 Sep 2026) in Section 5, subsection “Transfer is a hypothesis here rather than a finding”; Section 6, paragraph beginning “Four lines of work follow from those boundaries”