Design reliable rubric generator and verifier for large-scale instruction-following training

Design a reliable rubric generator and a reliable rubric verifier for large-scale reinforcement learning pipelines aimed at improving large language models’ instruction-following capabilities, where the generator synthesizes rubrics for each user prompt from raw training data and the verifier determines whether a given model response satisfies each rubric criterion, so that dependable rubrics and judgments can be provided for training when human labeling is impractical.

Background

The paper introduces AdvancedIF, a human-annotated benchmark and RIFL, a rubric-based reinforcement learning pipeline to improve instruction following in LLMs. A critical component of RIFL is the ability to automatically synthesize rubrics for prompts at scale and reliably verify whether responses satisfy those rubrics, since manual labeling is infeasible for large datasets.

The authors note that while rubrics offer interpretable criteria for evaluation and training, creating a scalable and reliable generator (to produce rubrics per prompt) and verifier (to judge each criterion on model responses) is challenging. This explicit open problem motivates their development of a trained rubric generator and a finetuned rubric verifier, but they acknowledge the broader design question remains open.