---
title: 'RoSE: Round Robin Synthetic Data Evaluation'
url: https://www.emergentmind.com/topics/round-robin-synthetic-data-evaluation-rose
type: topic
---

# RoSE: Round Robin Synthetic Data Evaluation

Searching arXiv for the cited RoSE paper and related work to ground the article in the current literature.
Round Robin Synthetic Data Evaluation (RoSE) is a proxy metric for selecting the best large language model (LLM) generator for synthetic-data creation without relying on human test sets. It was introduced for settings in which LLMs generate labeled synthetic data to train smaller downstream models, especially in low-resource languages where human-labelled data are scarce. The core idea is to assess a candidate generator indirectly: a small classifier is trained on that generator’s synthetic training set and then evaluated on synthetic test sets produced by all other candidate generators; the generator’s RoSE score is the mean cross-evaluation performance. In experiments across six LLMs, eleven languages, and three classification tasks, RoSE identified the optimal generator more often than the intrinsic heuristics considered, achieved the smallest average downstream gap to the optimal generator, and was reported as the only metric with positive correlation to performance on human test data [2510.06143].

## 1. Definition and scope

In the formulation introduced in "RoSE: Round-robin Synthetic Data Evaluation for Selecting LLM Generators without Human Test Sets" [2510.06143], let $\mathcal{M}=\{M_1,\dots,M_K\}$ be a set of $K$ candidate LLM generators. For each $M_i\in\mathcal{M}$ and each classification task–language pair, $M_i$ is used to generate a synthetic training set $D_i^{\text{train}}$ and, separately, synthetic test sets $D_i^{\text{test}}$. A small downstream classifier $C_i$ is trained on $D_i^{\text{train}}$ and evaluated on the synthetic test sets $\{D_j^{\text{test}}: j\neq i\}$ from the other generators. Writing $F1(C_i,D_j^{\text{test}})$ for the $F_1$ score of $C_i$ on $D_j^{\text{test}}$, the RoSE score is defined as

$$
\mathrm{RoSE}(M_i)=\frac{1}{K-1}\sum_{j=1,\;j\neq i}^{K} F1(C_i,D_j^{\text{test}}).
$$

To reduce sampling variance, the complete round-robin procedure is repeated $T$ times with different random seeds; in the paper, $T=10$. This yields $\mathrm{RoSE}^{(t)}(M_i)$ for each repetition, and the final reported score is

$$
\overline{\mathrm{RoSE}}(M_i)=\frac{1}{T}\sum_{t=1}^{T}\mathrm{RoSE}^{(t)}(M_i).
$$

The selected generator is the $M_i$ with the highest $\overline{\mathrm{RoSE}}$ score. Operationally, RoSE is therefore not a direct estimate of human-test performance. It is a proxy selection criterion intended to rank candidate generators by the usefulness of the synthetic data they produce for downstream training.

This construction is designed for extrinsic generator selection rather than for intrinsic quality assessment. A plausible implication is that RoSE treats generator quality as a relational property: a generator is preferred when models trained on its outputs generalize well across the synthetic distributions induced by competing generators.

## 2. Round-robin procedure

The algorithmic workflow is explicit. For each repetition $t=1,\dots,T$, every candidate generator produces its synthetic training set and synthetic test sets. For each generator $M_i$, a classifier $C_i$ is then trained on $D_i^{\text{train}}$. That classifier is evaluated on each $D_j^{\text{test}}$ with $j\neq i$, yielding scores $f_{i\to j}^{(t)}=F1(C_i,D_j^{\text{test}})$. The repetition-level score is the average of these cross-evaluations, and the final score is the mean across repetitions [2510.06143].

The paper’s pseudocode makes three aspects especially clear. First, train and test sets are both synthetic, but they are not drawn from the same generator during evaluation. Second, the procedure is symmetric across the candidate pool: each generator serves both as a source of training data for its own classifier and as a source of test data for the other classifiers. Third, the ranking is defined only relative to a set of candidates $\mathcal{M}$ rather than in isolation.

This round-robin design distinguishes RoSE from conventional intrinsic heuristics such as diversity or entropy measures. Intrinsic metrics summarize properties of the generated corpus itself, whereas RoSE measures whether a downstream model trained on one generator’s outputs transfers to synthetic examples produced by the others. This suggests that RoSE uses disagreement among generators as a form of evaluation substrate when human-labelled test data are unavailable.

## 3. Experimental realization

The reported experiments use six open-weight LLM generators: Gemma-3 with 4B and 27B parameters, Magistral-Small with 24B parameters, Qwen 2.5 with 14B parameters, and Llama 3 with 8B and 70B parameters [2510.06143]. The downstream classifier is XLM-R Base fine-tuned on synthetic data.

Three tasks are evaluated. The first is intent recognition using MASSIVE with the 10 most common intents. The second is topic classification using SIB-200 with 7 topics. The third is sentiment analysis using an aggregate of 12 datasets spanning 11 languages. The languages are Azerbaijani (az), Welsh (cy), German (de), English (en), Hebrew (he), Indonesian (id), Romanian (ro), Slovenian (sl), Swahili (sw), Telugu (te), and Thai (th).

Synthetic data generation uses 100 samples per label via in-context prompting with 10 human examples per label plus self-revision; the sampling configuration is temperature $=0.7$ and top\_p $=0.9$. The dependence on in-context examples is important because the method is described as operating without human test sets, not without any human examples.

The experimental comparison includes several baseline proxy metrics. The intrinsic heuristics are Average Pairwise Cosine Distance, Bigram Diversity, Number of Valid Samples after the revision filter, Silhouette Score, Type–Token Ratio (TTR), Token Entropy, Heuristic: LLM Parameter Size, and Random Selection. For reference, the paper also defines a human-annotated “oracle” ranking by training XLM-R on each generator’s outputs and measuring $F_1$ on human-labelled test sets; the best generator by human $F_1$ is the optimal generator [2510.06143].

| Category | Items |
|---|---|
| Generators | Gemma-3 4B, Gemma-3 27B, Magistral-Small 24B, Qwen 2.5 14B, Llama 3-8B, Llama 3-70B |
| Tasks | Intent recognition, topic classification, sentiment analysis |
| Classifier | XLM-R Base fine-tuned on synthetic data |

## 4. Quantitative behavior

The principal quantitative result is generator selection accuracy. In Table 1, evaluated over 33 task–language cases, RoSE achieves 20 Top-1 correct matches, or 60.6%, and 15 Top-3 correct matches, or 45.5%. Among the listed baselines, the heuristic based on LLM size reaches 12 Top-1 correct matches, or 36.4%, while Average Pairwise Cosine Distance reaches 7, Bigram Diversity 5, and Random Selection 5 [2510.06143].

A second result concerns the downstream performance gap to the optimal generator. Figure 1 reports that RoSE exhibits the smallest average $F_1$ gap, 0.76 percentage points, while the second-best heuristic, LLM size, yields 2.52%. Table 2 further breaks this down by task: for intent, topic, and sentiment respectively, the mean $F_1$ gaps are $-0.64\%$, $-0.77\%$, and $-0.86\%$ for RoSE, compared with $-1.73\%$, $-0.31\%$, and $-5.52\%$ for LLM size. Table 3 reports that RoSE is best in 9 of 11 languages and second-best in Welsh and Indonesian, with gaps under 2%.

The correlation analysis in Figure 2 is central to the paper’s argument. RoSE is reported as the only proxy metric to achieve a positive correlation with performance on human test data; its mean Pearson correlation is approximately $r\approx 0.60$, with a 95% confidence interval entirely above zero. All other intrinsic metrics are described as showing weak or negative correlations [2510.06143].

These results frame RoSE as a proxy whose utility lies less in estimating an absolute performance value than in reliably ranking candidate generators. A plausible implication is that positive proxy–human correlation, combined with a small optimality gap, makes RoSE suitable for model-selection decisions when labelled evaluation sets are unavailable.

## 5. Sensitivity analyses and operational recommendations

The ablation study examines several forms of robustness. When Llama 3-70B is excluded, the RoSE gap is $-1.26\%$, compared with a $-5.78\%$ gap for the LLM-size heuristic. When varying the number of candidate LLMs from 2 to 6, RoSE remains best for all $K$, with the smallest gap at $K=2$ and stable performance when $K\geq 3$. A cost-effective variant evaluates each $C_i$ on a single randomly chosen $D_j^{\text{test}}$ per iteration; this still outperforms heuristics once at least three evaluators are used. By contrast, a zero-shot variant, RoSE-Z, which uses no in-context examples, degrades sharply: the average $F_1$ drop is 8.45%, and RoSE-Z underperforms several intrinsic metrics [2510.06143].

The paper’s recommendations follow directly from these results. At least three candidate generators should be compared to stabilize RoSE scores, and including more evaluators improves reliability. In-context examples are crucial; zero-shot RoSE fails. For higher efficiency, fewer cross-evaluations may be sampled, but at least three evaluators should be maintained.

A common misconception would be to read the method as entirely annotation-free. The limitation section states that it requires a small set of human examples, specifically 10 per label, for generation. The method therefore removes the dependence on human test sets, not all dependence on human supervision.

## 6. Limitations, computational cost, and relation to adjacent work

The stated limitations are fourfold. First, RoSE requires a small set of human examples for generation. Second, it is computationally expensive, with approximately 380 GPU hours for generation and fine-tuning. Third, the impact of data contamination in LLM pretraining is unknown. Fourth, the evaluation covers 6 LLMs, 11 languages, and 3 tasks, and additional studies are needed for broader generalization [2510.06143].

These limitations constrain the scope of the claims. The reported evidence supports RoSE as a generator-selection proxy under the evaluated conditions; broader claims about universality across model families, task types, or contamination regimes would be inferential rather than established.

RoSE also sits within a broader line of work that uses synthetic data for evaluation rather than training alone. "Can You Rely on Your Model Evaluation? Improving Model Evaluation with Synthetic Test Data" introduces 3S Testing, a framework that generates synthetic test sets for subgroup evaluation and distributional-shift assessment [2310.16524]. That paper includes a section titled “Extending to a Round-Robin Synthetic Data Evaluation (RoSE) Protocol,” but the described procedure is different: it is an adaptive, sequential protocol that interleaves data generation, performance estimation, uncertainty reduction, optional generator refinement, and stopping rules based on confidence-interval width. Its purpose is to focus evaluation effort on uncertain or worst-case subgroups and shifts, not to select among LLM generators [2310.16524].

The shared acronym can therefore obscure two distinct uses. In the 2025 LLM paper, RoSE denotes a round-robin cross-evaluation score for choosing a synthetic-data generator without human test sets. In the 2023 3S-testing context, “RoSE” denotes a budget-constrained, sequential audit protocol over subgroup or shift specifications. The overlap is terminological rather than methodological identity.

## 7. Significance for low-resource synthetic-data pipelines

Within synthetic-data pipelines for low-resource languages, RoSE addresses a specific model-selection bottleneck: several LLMs may generate plausible labeled examples, yet intrinsic corpus statistics do not reliably predict which generator will produce the most useful training data for a downstream classifier. The paper’s results indicate that RoSE identifies the optimal generator 60% of the time versus 36% for the LLM-size heuristic, and that its average downstream gap to the optimal generator is 0.76 percentage points [2510.06143].

This places RoSE between two evaluation regimes. It is more task-grounded than intrinsic heuristics because it trains and evaluates an actual downstream classifier. It is less annotation-dependent than oracle evaluation because it does not require human-labelled test sets for ranking. The method is therefore particularly relevant when high-quality human evaluation data are unavailable, but a small amount of labeled prompting data can still be assembled.

Its significance is methodological rather than universal. The available evidence supports RoSE as a practical proxy for generator choice in the evaluated multilingual classification settings. A plausible implication is that similar round-robin cross-evaluation ideas may be useful wherever synthetic-data generators must be compared under annotation scarcity, provided that the dependence on candidate-set composition, computational budget, and in-context exemplars is taken into account.

Source: https://www.emergentmind.com/topics/round-robin-synthetic-data-evaluation-rose