Applicability of S Tuning When Verification is Expensive

Determine whether S tuning, which optimizes the initial recurrent state S0 per recurrent layer of hybrid recurrent–attention language models while freezing all model weights, remains effective when applied to tasks where obtaining execution-verified correct solutions for training is expensive or impractical.

Background

S tuning adapts hybrid recurrent–attention models by learning an initial recurrent state S0 for each recurrent layer while keeping all weights frozen, achieving large gains on HumanEval with roughly 48 execution-verified correct solutions used for training.

The method’s demonstrated effectiveness relies on access to execution-verified supervision. The paper explicitly notes that applying S tuning in settings where such verification is costly has not yet been evaluated, leaving open whether comparable benefits can be obtained without inexpensive verification pipelines.