Lightweight observational-setting provenance tests with comparable token efficiency

Design more computationally lightweight independence tests for attributing text to a specific training run in the observational setting—where only generated text is available—than the shuffled-transcript retraining test statistic obs (Algorithm “Training models on shuffled transcript”), while achieving similar token complexity (i.e., requiring a comparable number of observed tokens from the generated text to attain high power).

Background

The paper introduces two families of tests for establishing provenance: query-setting tests that correlate a model’s log-likelihoods of training examples with the training order, and observational-setting tests that attribute text when only generated outputs are available. In the latter, one approach (obs) retrains models on independent reshuffles of the training data and compares likelihoods of the observed text across these models, yielding strong performance but significant computational cost.

The authors note that all tests involve nontrivial cost—either many tokens from the suspected model or retraining multiple models. In the observational setting, tokens cannot be increased via additional queries, so reducing computational overhead without worsening token requirements is especially important. Hence, developing more lightweight tests than obs that preserve token efficiency is identified as an explicit open problem.