TTScore: Evaluation and Domain Variants

• TTScore is a family of evaluation metrics with variants like TTScore-int and TTScore-pro explicitly defined for speech synthesis and context-specific applications.
• It employs targeted, log-likelihood based assessments from text-conditioned sequence models to evaluate perceptual aspects such as intelligibility and prosody.
• While related approaches like Chronos and TSS provide TTScore-equivalent insights, each method is tailored to its domain with distinct calibration and methodological constraints.

Searching arXiv for papers and naming variants of “TTScore” to ground the article. {"query":"TTScore arXiv Chronos test-time scaling Task-Specificity Score speech synthesis prosody intelligibility", "max_results": 10} {"query":"(Zhang et al., 1 Feb 2026)", "max_results": 5} {"query":"(Ulgen et al., 24 Sep 2025) TTScore speech synthesis", "max_results": 5} {"query":"(Kadasi et al., 3 Feb 2026) Task-Specificity Score", "max_results": 5} In recent arXiv literature, TTScore does not denote a single standardized metric. The name is introduced explicitly in speech synthesis as a targeted, reference-free framework with TTScore-int and TTScore-pro, while Chronos defines a learned per-trajectory quality score for test-time scaling that is described as TTScore-equivalent, and Task-Specificity Score is abbreviated TSS rather than TTScore (Ulgen et al., 24 Sep 2025, Zhang et al., 1 Feb 2026, Kadasi et al., 3 Feb 2026). This suggests a family of related scoring ideas rather than a single canonical definition.

1. Terminological scope and disambiguation

The term is best understood through disambiguation across domains. Several papers explicitly state that they do not use the term “TTScore,” even when they define a scalar score that can be used analogously.

Designation	Domain	Relation to “TTScore”
TTScore	Speech synthesis	Explicitly defined as TTScore-int and TTScore-pro (Ulgen et al., 24 Sep 2025)
Chronos score $\hat{y}$	LLM test-time scaling	Described as TTScore-equivalent, but not named TTScore in the paper (Zhang et al., 1 Feb 2026)
TTSDS	TTS system evaluation	Can serve as a general-purpose TTScore (Minixhofer et al., 2024)
TSS / TSS++	Instruction tuning	Paper uses Task-Specificity Score, not TTScore (Kadasi et al., 3 Feb 2026)
TScore	Music-notation data modeling	Distinct term; “TTScore” is a misnomer (Lepper et al., 2024)
OMOQ / OMOS	Time-scale modification of audio	Not named TTScore; aliasing is external to the paper (Roberts et al., 2020)

A further negative case appears in the Zero Resource Speech Challenge 2019, where submissions were compared with CER, MOS, speaker similarity, ABX, and bitrate; the paper states that it does not define or name a metric called TTScore (Dunbar et al., 2019). The literature therefore separates into three categories: explicit TTScore, TTScore-equivalent scorers, and adjacent but differently named evaluation measures.

2. TTScore in speech synthesis

The paper “Objective Evaluation of Prosody and Intelligibility in Speech Synthesis via Conditional Prediction of Discrete Tokens” defines TTScore as a targeted, reference-free framework for two perceptual aspects of synthesized speech: intelligibility and prosody (Ulgen et al., 24 Sep 2025). The framework uses two text-conditioned sequence-to-sequence predictors over discrete speech tokens. TTScore-int scores content tokens derived from HuBERT-base hidden representations quantized by k-means, while TTScore-pro scores prosody tokens derived from FACodec prosody representations after phoneme-level pooling using Montreal Forced Aligner. The predictors condition on a phoneme sequence obtained by grapheme-to-phoneme conversion.

The formal scores are length-normalized average log-likelihoods:

$$\text{TTScore-int}(\mathbf{c}, \mathbf{x})=\frac{1}{T}\sum_{i=1}^{T}\log p\!\left(c_i \mid \mathbf{c}_{<i}, \mathbf{x};\theta_c\right),$$

$$\text{TTScore-pro}(\mathbf{f}, \mathbf{x})=\frac{1}{L}\sum_{i=1}^{L}\log p\!\left(f_i \mid \mathbf{f}_{<i}, \mathbf{x};\theta_f\right).$$

The logarithm is the natural log, and higher, less negative values indicate better alignment. The paper does not define a combined overall quality score; the two scores remain aspect-specific.

The architecture is shared across both predictors: a BART-style Transformer encoder-decoder with 6 encoder layers, 6 decoder layers, model dimension 512, embedding dimension 256, 8 attention heads, dropout 0.1, AdamW, batch size 8, and maximum sequence length 1024. Training uses LibriSpeech-960 for the seq2seq predictors, while the HuBERT k-means tokenizer is trained on LibriSpeech-100. For content tokens, the explored HuBERT layers are 3, 9, and 12, with $k \in \{50,500\}$. For prosody, FACodec uses a prosody codebook size of 1024 at a 12.5 ms frame rate.

Empirically, TTScore-int achieves markedly higher correlations with WER/CER than unconditioned speech-token LLMs, and it also correlates more strongly with MOS than WER/CER on SOMOS and VoiceMOS22. TTScore-pro yields higher scores for real speech than for synthesized speech, ranks Original F0 above Inverse F0 and Flipped F0 in prosody sanity checks, and shows stronger positive correlations with MOS and TTSArena ELO than F0-RMSE and F0 correlation. The intended usage is diagnostic rather than monolithic: low TTScore-int localizes content and pronunciation failures, whereas low TTScore-pro localizes prosodic inadequacy.

3. Chronos as a TTScore-equivalent for test-time scaling

In “Chronos: Learning Temporal Dynamics of Reasoning Chains for Test-Time Scaling,” the paper states that it does not use the term TTScore, but that Chronos’s learned per-trajectory quality score $\hat{y}$ can be treated as a TTScore-equivalent for test-time scaling (Zhang et al., 1 Feb 2026). Chronos models each sampled reasoning trajectory as a time series over the last $L_{\text{tail}}$ tokens, using the token-level signal

$$s_t=-\frac{1}{k}\sum_{i=1}^{k}\log P_t(i \mid x,y_{<t}),$$

where the paper uses top-$k=20$ and constructs a tail sequence of length $L_{\text{tail}}=2048$. The scorer is a lightweight multi-scale temporal convolutional model inspired by InceptionTime, with $N_{\text{Blk}}=3$ residual multi-scale convolution blocks, a sigmoid output, and optional ensembling of 2–4 scorers.

Chronos replaces uniform majority voting with score-weighted aggregation. After retaining the top-$$\text{TTScore-int}(\mathbf{c}, \mathbf{x})=\frac{1}{T}\sum_{i=1}^{T}\log p\!\left(c_i \mid \mathbf{c}_{<i}, \mathbf{x};\theta_c\right),$$0 fraction of trajectories, with $$\text{TTScore-int}(\mathbf{c}, \mathbf{x})=\frac{1}{T}\sum_{i=1}^{T}\log p\!\left(c_i \mid \mathbf{c}_{<i}, \mathbf{x};\theta_c\right),$$1, the final answer is selected by

$$\text{TTScore-int}(\mathbf{c}, \mathbf{x})=\frac{1}{T}\sum_{i=1}^{T}\log p\!\left(c_i \mid \mathbf{c}_{<i}, \mathbf{x};\theta_c\right),$$2

Training is binary classification over trajectories labeled by whether the extracted final answer is correct, optimized with standard BCE. The scorer is trained exclusively on AIME 2000–2023, with 32 trajectories per training question and an 8:1:1 train/val/test split. The generators scored in the paper are DeepSeek-1.5B, Qwen3-4B-Thinking-2507, and DeepSeek-8B.

The empirical results are framed around AIME25, HMMT25 (Feb), and GPQA-Diamond. On HMMT25 with Qwen3-4B-Thinking-2507, Chronos@128 reaches 74.38%, compared with Pass@1 55.42% and Maj@128 60.62%, corresponding to relative improvements of 34.21% over Pass@1 and 22.70% over Maj@128. Across nine model-dataset combinations, Chronos@128 also exceeds DeepConf@128. The compute overhead is reported as negligible: for a batch of 30 queries, Chronos requires 3.9 BFLOPs versus approximately 2000 TFLOPs for generation with DeepSeek-1.5B, or about 0.0005% of total inference FLOPs.

The score’s practical interpretation is narrower than general quality scoring. The paper states that it requires white-box log-probabilities, depends on the generator’s calibration, was trained on math, and validates internal consistency, not factuality. A plausible implication is that Chronos’s TTScore-equivalent is native to reasoning-trace aggregation rather than to generic response evaluation.

4. Related speech and audio scores often conflated with TTScore

In text-to-speech evaluation, TTSDS — Text-to-Speech Distribution Score is presented as a single, system-level score that can serve as a general-purpose TTScore, although the paper does not define a separate metric called TTScore (Minixhofer et al., 2024). TTSDS evaluates synthetic speech as a combination of prosody, speaker identity, intelligibility, environment, and a general speech representation. For each factor, it computes feature-level scores from Wasserstein distances to sets of real and noise/distractor datasets, and then averages factor scores without explicit weights:

$$\text{TTScore-int}(\mathbf{c}, \mathbf{x})=\frac{1}{T}\sum_{i=1}^{T}\log p\!\left(c_i \mid \mathbf{c}_{<i}, \mathbf{x};\theta_c\right),$$3

with $$\text{TTScore-int}(\mathbf{c}, \mathbf{x})=\frac{1}{T}\sum_{i=1}^{T}\log p\!\left(c_i \mid \mathbf{c}_{<i}, \mathbf{x};\theta_c\right),$$4 and scores in $$\text{TTScore-int}(\mathbf{c}, \mathbf{x})=\frac{1}{T}\sum_{i=1}^{T}\log p\!\left(c_i \mid \mathbf{c}_{<i}, \mathbf{x};\theta_c\right),$$5. The benchmark covers 35 TTS systems from 2008 to 2024, and the paper reports Spearman correlations ranging from 0.60 to 0.83 with human evaluations across historical periods.

A distinct audio-evaluation line appears in “An Objective Measure of Quality for Time-Scale Modification of Audio,” which predicts OMOS on the MOS scale $$\text{TTScore-int}(\mathbf{c}, \mathbf{x})=\frac{1}{T}\sum_{i=1}^{T}\log p\!\left(c_i \mid \mathbf{c}_{<i}, \mathbf{x};\theta_c\right),$$6 using hand-crafted features and a fully connected network (Roberts et al., 2020). The paper explicitly states that it does not introduce the name TTScore. Its reported performance is mean RMSE 0.487 and mean Pearson correlation 0.865, corresponding to the 98th and 82nd percentiles of subjective sessions, respectively. The score is tied to time-scale modified audio, constant-ratio assumptions, and a feature pipeline built from PEAQ Basic and Advanced MOVs plus nine TSM-specific features.

Historical zero-resource TTS work likewise uses different criteria. The Zero Resource Speech Challenge 2019 concentrates on CER as a measure of synthesis quality and supplements it with MOS, speaker similarity, ABX, and bitrate rather than a single TTScore-like scalar (Dunbar et al., 2019). This older configuration is important because it shows that the desire for a single synthesis score predates the explicit naming of TTScore, but was not yet formalized under that label.

5. Task-Specificity Score and neighboring instruction-tuning uses

“Task–Specificity Score: Measuring How Much Instructions Really Matter for Supervision” introduces TSS and TSS++, not TTScore, but explicitly treats this as the relevant term when such a query is mapped into instruction-tuning literature (Kadasi et al., 3 Feb 2026). TSS measures how much an instruction $$\text{TTScore-int}(\mathbf{c}, \mathbf{x})=\frac{1}{T}\sum_{i=1}^{T}\log p\!\left(c_i \mid \mathbf{c}_{<i}, \mathbf{x};\theta_c\right),$$7 matters for predicting output $$\text{TTScore-int}(\mathbf{c}, \mathbf{x})=\frac{1}{T}\sum_{i=1}^{T}\log p\!\left(c_i \mid \mathbf{c}_{<i}, \mathbf{x};\theta_c\right),$$8 for input $$\text{TTScore-int}(\mathbf{c}, \mathbf{x})=\frac{1}{T}\sum_{i=1}^{T}\log p\!\left(c_i \mid \mathbf{c}_{<i}, \mathbf{x};\theta_c\right),$$9 by contrasting the true instruction with plausible alternatives generated for the same input. In the paper’s formulation,

$$\text{TTScore-pro}(\mathbf{f}, \mathbf{x})=\frac{1}{L}\sum_{i=1}^{L}\log p\!\left(f_i \mid \mathbf{f}_{<i}, \mathbf{x};\theta_f\right).$$0

where $$\text{TTScore-pro}(\mathbf{f}, \mathbf{x})=\frac{1}{L}\sum_{i=1}^{L}\log p\!\left(f_i \mid \mathbf{f}_{<i}, \mathbf{x};\theta_f\right).$$1 is a frozen scoring model and $$\text{TTScore-pro}(\mathbf{f}, \mathbf{x})=\frac{1}{L}\sum_{i=1}^{L}\log p\!\left(f_i \mid \mathbf{f}_{<i}, \mathbf{x};\theta_f\right).$$2 are alternative instructions. The paper states that this approximates the pointwise conditional mutual information $$\text{TTScore-pro}(\mathbf{f}, \mathbf{x})=\frac{1}{L}\sum_{i=1}^{L}\log p\!\left(f_i \mid \mathbf{f}_{<i}, \mathbf{x};\theta_f\right).$$3 by Monte Carlo.

TSS++ addresses easy-negative effects by constructing a candidate set $$\text{TTScore-pro}(\mathbf{f}, \mathbf{x})=\frac{1}{L}\sum_{i=1}^{L}\log p\!\left(f_i \mid \mathbf{f}_{<i}, \mathbf{x};\theta_f\right).$$4, selecting hard alternatives $$\text{TTScore-pro}(\mathbf{f}, \mathbf{x})=\frac{1}{L}\sum_{i=1}^{L}\log p\!\left(f_i \mid \mathbf{f}_{<i}, \mathbf{x};\theta_f\right).$$5, using an InfoNCE-style contrast with temperature $$\text{TTScore-pro}(\mathbf{f}, \mathbf{x})=\frac{1}{L}\sum_{i=1}^{L}\log p\!\left(f_i \mid \mathbf{f}_{<i}, \mathbf{x};\theta_f\right).$$6, and adding a small quality term weighted by $$\text{TTScore-pro}(\mathbf{f}, \mathbf{x})=\frac{1}{L}\sum_{i=1}^{L}\log p\!\left(f_i \mid \mathbf{f}_{<i}, \mathbf{x};\theta_f\right).$$7. The paper uses length-normalized log-likelihood, recommends $$\text{TTScore-pro}(\mathbf{f}, \mathbf{x})=\frac{1}{L}\sum_{i=1}^{L}\log p\!\left(f_i \mid \mathbf{f}_{<i}, \mathbf{x};\theta_f\right).$$8 for vanilla TSS, and evaluates on Alpaca, Dolly-15k, and NI-20 with Gemma, Llama, and Qwen families. Under 5% retention, the best strategy beats Random in 8/9 settings, with TSS++ the most frequent winner in 6/9. Under 15% retention, the best strategy improves in 6/9 settings, and the winners diversify across Random, TSS, TSS++, and PPL. In one highlighted sweep, LLaMA on Alpaca peaks at 147.304 SUM with TSS++(E) at $$\text{TTScore-pro}(\mathbf{f}, \mathbf{x})=\frac{1}{L}\sum_{i=1}^{L}\log p\!\left(f_i \mid \mathbf{f}_{<i}, \mathbf{x};\theta_f\right).$$9, which is +7.916 SUM over full-data SFT.

The distinction from speech-synthesis TTScore is structural. TSS is a supervision-selection score over instruction–input–output triples, not an output-quality score over speech or trajectories. The shared element is contrastive ranking, but the scored object and the downstream action—dataset selection rather than inference-time answer aggregation—are different.

6. Limitations, comparability, and usage conventions

The most important encyclopedic point is non-equivalence. TTScore-int and TTScore-pro are raw average log-likelihoods whose absolute values depend on tokenization and model calibration; the paper recommends them for relative comparisons, system-level aggregation, and diagnostics, not as a universal quality number (Ulgen et al., 24 Sep 2025). Chronos outputs a sigmoid trajectory score in $k \in \{50,500\}$0 for weighted voting and top-$k \in \{50,500\}$1 filtering, requires token-level probability distributions, and leaves transfer beyond math and less-structured domains unproven (Zhang et al., 1 Feb 2026). TTSDS is normalized to $k \in \{50,500\}$2, depends on the choice of real and noise reference datasets, and uses an unweighted average even though the paper reports that factor importance varies across eras (Minixhofer et al., 2024). TSS/TSS++ depend on a frozen scoring model, on the generation or retrieval of plausible alternatives, and on budget-sensitive hyperparameters such as $k \in \{50,500\}$3, $k \in \{50,500\}$4, $k \in \{50,500\}$5, and $k \in \{50,500\}$6 (Kadasi et al., 3 Feb 2026).

Adjacent naming confusion extends beyond these four lines. TScore is a semantics-first formalism for time-related musical data and is explicitly stated to be distinct from TTScore (Lepper et al., 2024). The time-scale modification paper defines OMOQ/OMOS, not TTScore, and the Zero Resource Speech Challenge relies on CER as its main synthesis-quality measure rather than on a composite score (Roberts et al., 2020, Dunbar et al., 2019).

This suggests that “TTScore” is best treated as a contextual label whose meaning is fixed by the paper that defines the scored object, the supervisory signal, and the decision rule. In speech synthesis, it denotes targeted token-likelihood metrics; in test-time scaling, it can denote a learned trajectory-quality score; in instruction tuning, the closest named construct is TSS; and in several adjacent literatures the term is absent altogether.