AnimeScore: Evaluating Speech & Popularity

• AnimeScore is a term representing two distinct research systems that quantify anime-related attributes in speech and media popularity.
• The speech system uses pairwise A/B comparisons and acoustic analyses via deep learning models to capture anime-like vocal style.
• The popularity system employs a multimodal approach with text and images to predict pre-release anime success from weighted user scores.

AnimeScore denotes two distinct research systems in recent arXiv literature. In speech technology, "AnimeScore: A Preference-Based Dataset and Framework for Evaluating Anime-Like Speech Style" defines a preference-based dataset and learned ranking framework for evaluating "anime-like" speech style via pairwise judgments rather than absolute opinion scoring (Park et al., 12 Mar 2026). In anime analytics, "Anime Popularity Prediction Before Huge Investments: a Multimodal Approach Using Deep Learning" uses the same name for a continuous, pre-release popularity forecast based on synopsis text, character descriptions, and character portraits, trained against MyAnimeList’s weighted average score (Armenta-Segura et al., 2024). The shared label therefore covers two different problem formulations: perceptual ranking of stylized speech and multimodal regression for anticipated audience reception.

1. Scope and disambiguation

The two uses of AnimeScore differ in modality, supervision, and intended deployment target. One addresses evaluation of anime-like vocal delivery, where no stable absolute perceptual scale is assumed. The other addresses pre-investment forecasting of anime popularity, where a continuous scalar target is available from public platform statistics.

Variant	Input domain	Output target
AnimeScore for speech style	Japanese speech clips	Scalar anime-likeness score learned from pairwise preferences
AnimeScore for popularity forecasting	Synopsis, character descriptions, character portraits	Continuous popularity score aligned with MAL weighted average score

This naming collision is substantive rather than cosmetic. In the speech setting, the core object is a learned ranking function whose absolute scale is arbitrary. In the popularity setting, the target is a calibrated regression value on the same interpretive scale as the source label. A plausible implication is that the two systems are methodologically analogous only at the level of scalar scoring; their labels, data-generating processes, and evaluation semantics are otherwise distinct.

2. Preference-based AnimeScore for anime-like speech

The speech-oriented AnimeScore begins from the claim that Mean Opinion Score is inadequate for anime-likeness because anime-likeness is not a single-dimensional percept with a universally shared absolute scale, whereas comparative judgments are easy, reliable, and aligned with how style preferences are formed (Park et al., 12 Mar 2026). The framework therefore replaces absolute opinion scoring with pairwise A/B judgments and learns a scalar score that reproduces human comparative decisions.

The dataset was constructed from three Japanese speech sources: Anim-400k, ReazonSpeech, and Coco-Nut. After filtering and matching, the final pool contains 3,000 utterances split into 2,500 training and 500 test items. Source counts are reported as Anim-400k: 1,315 total, 1,065 train, 250 test; ReazonSpeech: 948 total, 828 train, 120 test; and Coco-Nut: 737 total, 607 train, 130 test. Bias reduction and quality control include linguistic bias suppression with text screening by Qwen3-30B-Instruct, Sidon enhancement, ASR CER filtering with whisper-large-v3, duration filtering to $[2\,\mathrm{s}, 10\,\mathrm{s}]$, exclusion of degraded audio via UTMOS with predicted MOS $> 3$, and speaker-condition balancing using ECAPA-TDNN speaker embeddings, t-SNE visualization and clustering, and removal of dense redundant clusters. Pair construction uses transcript-embedding cosine similarity from Sentence-Transformers and ECAPA-TDNN speaker-embedding cosine similarity, with greedy selection over a weighted sum of text and speaker similarity and a preference for cross-corpus pairs. The final target is 12,500 train pairs and 2,500 test pairs.

Annotation comprises 15,000 pairwise A/B judgments from 187 evaluators. Reported demographics are age distribution of 20s or younger: 8, 30s: 48, 40s: 80, 50s or older: 51; gender of male: 142 and female: 45; and anime familiarity of low: 9, medium: 103, and high: 75. The task instruction is to choose which of two clips sounds more “anime-like,” focusing on voice style rather than content. Annotators also produced free-form descriptions of cues they personally associate with anime-like voices. Session design was 25 trials per session, approximately 15 minutes, with 1–10 sessions per evaluator.

The corpus analysis reported in the paper includes win-rate histograms in which Anim-400k dominates high-win-rate regions, with 93.2% versus ReazonSpeech and 88.0% versus Coco-Nut. This indicates that the dataset does contain a stylized contrast recognizable to raters rather than only diffuse inter-rater preference noise.

3. Ranking model, acoustic correlates, and optimization use

The learned scoring function maps each clip $x$ to a scalar anime-likeness score $s(x)$ through a RankNet-style logistic pairwise model, equivalently Bradley–Terry with logistic link (Park et al., 12 Mar 2026):

$$P(i \succ j) = \sigma\big(s(x_i) - s(x_j)\big),\quad \sigma(z) = \frac{1}{1 + e^{-z}}.$$

Training minimizes

$$\mathcal{L} = -\sum_{(i, j) \in \mathcal{D}} \log \sigma\big(s(x_i) - s(x_j)\big).$$

The architecture uses a frozen SSL encoder to extract frame-level features $\mathbf{H}$, followed by a BiLSTM, mean pooling, and an MLP:

$$\mathbf{Z} = \mathrm{BiLSTM}(\mathbf{H}),\quad \bar{\mathbf{z}} = \tfrac{1}{T'} \sum_{t=1}^{T'} \mathbf{z}_t,\quad s(x) = \mathrm{MLP}(\bar{\mathbf{z}}).$$

Inference is a single forward pass producing $\hat{s}(x) \in \mathbb{R}$. Because the score is learned from preferences, the absolute scale is arbitrary; for reporting, scores may be z-normalized within a set or rescaled to $[0,1]$ via a logistic transform, while relative ordering remains the primary interpretive target.

The acoustic analysis identifies three dominant correlates of perceived anime-likeness: controlled resonance shaping, prosodic continuity, and deliberate articulation, rather than simple heuristics such as high pitch. Annotator free-form descriptions were categorized into five perceptual dimensions with counts among 187 participants: Emotional Explicitness (62), Timbre Difference (48), Prosodic Salience (38), Articulation Clarity (34), and Temporal Control (5). Operationalized proxies include median formants $> 3$0, $> 3$1, and $> 3$2 over voiced segments; mean $> 3$3; voicing ratio; spectral flux; syllable rate; articulation rate; pause ratio; and mean pause length. Pairwise Concordance Rate shows that lower median $> 3$4 is strongly concordant with preference, with reported values of $> 3$5 median PCR 61.5% $> 3$6, $> 3$7 59.6% $> 3$8, and $> 3$9 60.1% $x$0. Prosodic continuity yields voicing ratio 59.5% $x$1 and spectral flux 60.0% $x$2, while mean $x$3 reaches only 55.1% and in the decreasing direction, directly contradicting the “just raise pitch” heuristic. Articulatory measures show syllable rate 60.3% $x$4, pause ratio 60.0% $x$5, mean pause length 58.7% $x$6, and articulation rate 53.2% $x$7, summarized by the paper as “continuous rapid flow with minimal pausing, but careful segmental enunciation.” Multivariate logistic regression on pairwise feature differences with 5-fold CV over $x$8 pairs yields AUC by group of approximately Emotional 52.9%, Timbre 65.7%, Prosody 66.0%, Articulation 66.8%, and All combined 69.3%, establishing a handcrafted-feature ceiling well below the learned SSL rankers.

The SSL ranking models are evaluated on pairwise negative log-likelihood, pairwise accuracy, and ROC-AUC from pairwise margins. On the 2,500-pair test set, wav2vec2 achieves NLL 0.5139, Acc 74.30%, AUC 82.47%; WavLM achieves NLL 0.4284, Acc 81.05%, AUC 89.44%; HuBERT achieves NLL 0.3852, Acc 82.43%, AUC 90.82%; and data2vec achieves NLL 0.4686, Acc 77.09%, AUC 85.80%. HuBERT is the best reported model, and masked-prediction encoders are described as strongest, consistent with richer encoding of paralinguistics, prosody, and speaker attributes crucial for style.

The same scorer is proposed as a reward model for generative speech systems. With $x$9, the framework supports direct preference optimization through

$s(x)$0

and PPO with objective

$s(x)$1

The paper recommends reward normalization per batch, KL penalties, supervised fine-tuning before preference optimization, small-$s(x)$2 DPO or modest PPO steps, and human spot checks. Domain alignment is explicitly limited to Japanese anime-style speech.

4. AnimeScore as a pre-release popularity forecast

The popularity-oriented AnimeScore is defined as a continuous, pre-release popularity forecast for anime, designed to guide investment decisions using only modalities that are realistically available before large budgets are committed: plot synopses, early character descriptions, and character visual designs (Armenta-Segura et al., 2024). Its label is MyAnimeList’s weighted average score, treated as a proxy for global internet popularity.

The target construction is explicit. If $s(x)$3 is the naive mean of user scores and $s(x)$4 is the number of voters, then with $s(x)$5 and default score $s(x)$6 at the end of scraping on January 3, 2024, the weighted average score is

$s(x)$7

The paper states that MAL aggregates user ratings on a 0–10 scale and applies Bayesian-style weighting to account for the number of voters and a community-wide default score. For training, labels are min–max normalized to $s(x)$8, where 0 maps to the minimum observed score of approximately 1.86 and 1 maps to the maximum of approximately 9.06; predictions are then de-normalized back to the 0–10 interpretation. The task is strictly continuous regression rather than binning or categorical classification.

The dataset was scraped from freely accessible MAL pages between December 28, 2023 and January 3, 2024 using BeautifulSoup4. Raw collection yields 11,873 anime and 21,329 characters; the final curated dataset contains 7,784 anime and 14,682 main characters after removing characters without portrait or with “No description available,” removing anime without score, synopsis, title, or associated main characters, and removing synopses shorter than 20 words. The corpus is described as diverse and unrestricted across genres, studios, and seasons, but with rare extremes below 2 or above 9, producing tail imbalance. To prevent information leakage, all anime sharing any main character are clustered and assigned entirely to train or test, resulting in 6,345 training anime (81.5%) and 1,439 test anime (18.5%).

Preprocessing is straightforward but tightly specified. Synopsis text is tokenized with the Huggingface GPT-2 tokenizer at maximum length 128. Concatenated character descriptions use maximum length 256, truncated because of memory constraints. Character portraits are concatenated into a composite image per anime and processed with Huggingface’s AutoImageProcessor for ResNet-50 defaults, including resize to $s(x)$9 and normalization to ImageNet means and standard deviations. ResNet-50 embedding extraction yields a $$P(i \succ j) = \sigma\big(s(x_i) - s(x_j)\big),\quad \sigma(z) = \frac{1}{1 + e^{-z}}.$$0 feature map that is flattened to 49D for the pipeline. No explicit data augmentation is reported.

5. Multimodal architecture, optimization, and empirical results

The multimodal AnimeScore model uses GPT-2 for synopsis text, GPT-2 for concatenated character descriptions, and ResNet-50 for character portraits (Armenta-Segura et al., 2024). Each GPT-2 branch outputs a 768-dimensional embedding. The paper does not elaborate the pooling strategy beyond reporting an output shape of 768. The image encoder is “microsoft/resnet-50,” with the flattened $$P(i \succ j) = \sigma\big(s(x_i) - s(x_j)\big),\quad \sigma(z) = \frac{1}{1 + e^{-z}}.$$1 feature map providing 49 dimensions.

Character-level fusion combines character-description text and portraits through an MLP with input dimension $$P(i \succ j) = \sigma\big(s(x_i) - s(x_j)\big),\quad \sigma(z) = \frac{1}{1 + e^{-z}}.$$2: Dropout(0.1) $$P(i \succ j) = \sigma\big(s(x_i) - s(x_j)\big),\quad \sigma(z) = \frac{1}{1 + e^{-z}}.$$3 Linear(817$$P(i \succ j) = \sigma\big(s(x_i) - s(x_j)\big),\quad \sigma(z) = \frac{1}{1 + e^{-z}}.$$4768, TanH) $$P(i \succ j) = \sigma\big(s(x_i) - s(x_j)\big),\quad \sigma(z) = \frac{1}{1 + e^{-z}}.$$5 Dropout(0.1) $$P(i \succ j) = \sigma\big(s(x_i) - s(x_j)\big),\quad \sigma(z) = \frac{1}{1 + e^{-z}}.$$6 Linear(768$$P(i \succ j) = \sigma\big(s(x_i) - s(x_j)\big),\quad \sigma(z) = \frac{1}{1 + e^{-z}}.$$7768, TanH). The resulting unified character embedding $$P(i \succ j) = \sigma\big(s(x_i) - s(x_j)\big),\quad \sigma(z) = \frac{1}{1 + e^{-z}}.$$8 is concatenated with the synopsis embedding $$P(i \succ j) = \sigma\big(s(x_i) - s(x_j)\big),\quad \sigma(z) = \frac{1}{1 + e^{-z}}.$$9 to form a 1,536D vector. A larger fusion MLP then reduces dimensionality through Linear(1536$$\mathcal{L} = -\sum_{(i, j) \in \mathcal{D}} \log \sigma\big(s(x_i) - s(x_j)\big).$$0768, TanH) $$\mathcal{L} = -\sum_{(i, j) \in \mathcal{D}} \log \sigma\big(s(x_i) - s(x_j)\big).$$1 Linear(768$$\mathcal{L} = -\sum_{(i, j) \in \mathcal{D}} \log \sigma\big(s(x_i) - s(x_j)\big).$$2384, TanH) $$\mathcal{L} = -\sum_{(i, j) \in \mathcal{D}} \log \sigma\big(s(x_i) - s(x_j)\big).$$3 Linear(384$$\mathcal{L} = -\sum_{(i, j) \in \mathcal{D}} \log \sigma\big(s(x_i) - s(x_j)\big).$$4192, TanH) $$\mathcal{L} = -\sum_{(i, j) \in \mathcal{D}} \log \sigma\big(s(x_i) - s(x_j)\big).$$5 Linear(192$$\mathcal{L} = -\sum_{(i, j) \in \mathcal{D}} \log \sigma\big(s(x_i) - s(x_j)\big).$$696, TanH), followed by ReLU blocks Linear(96$$\mathcal{L} = -\sum_{(i, j) \in \mathcal{D}} \log \sigma\big(s(x_i) - s(x_j)\big).$$748) $$\mathcal{L} = -\sum_{(i, j) \in \mathcal{D}} \log \sigma\big(s(x_i) - s(x_j)\big).$$8 Linear(48$$\mathcal{L} = -\sum_{(i, j) \in \mathcal{D}} \log \sigma\big(s(x_i) - s(x_j)\big).$$924) $\mathbf{H}$0 Linear(24$\mathbf{H}$112) $\mathbf{H}$2 Linear(12$\mathbf{H}$36) $\mathbf{H}$4 Linear(6$\mathbf{H}$53) $\mathbf{H}$6 Linear(3$\mathbf{H}$71, Linear). The paper notes that a table labels the last activation as SoftMax, but training uses a regression loss, so the final head is implemented as a Linear layer to produce a scalar. The core equations are

$\mathbf{H}$8

$\mathbf{H}$9

Training uses mean squared error on normalized labels,

$$\mathbf{Z} = \mathrm{BiLSTM}(\mathbf{H}),\quad \bar{\mathbf{z}} = \tfrac{1}{T'} \sum_{t=1}^{T'} \mathbf{z}_t,\quad s(x) = \mathrm{MLP}(\bar{\mathbf{z}}).$$0

with AdamW, learning rate $$\mathbf{Z} = \mathrm{BiLSTM}(\mathbf{H}),\quad \bar{\mathbf{z}} = \tfrac{1}{T'} \sum_{t=1}^{T'} \mathbf{z}_t,\quad s(x) = \mathrm{MLP}(\bar{\mathbf{z}}).$$1, $$\mathbf{Z} = \mathrm{BiLSTM}(\mathbf{H}),\quad \bar{\mathbf{z}} = \tfrac{1}{T'} \sum_{t=1}^{T'} \mathbf{z}_t,\quad s(x) = \mathrm{MLP}(\bar{\mathbf{z}}).$$2, batch size 16, and 5 epochs for deep models. The implementation is reported in PyTorch 1.10.1 on an NVIDIA Quadro RTX 6000 with 46 GB VRAM. The paper does not explicitly state whether GPT-2 and ResNet-50 are frozen; its practical guidance treats them as trainable modules, with caution about stability.

On the test set, the full multimodal model achieves MSE 0.011, Spearman 0.431, Pearson 0.436, and Kendall’s Tau 0.297. Text-only synopsis yields MSE 0.012, Spearman 0.338, Pearson 0.328, and Kendall 0.230. Text-only character descriptions yield MSE 0.012, Spearman 0.307, Pearson 0.341, and Kendall 0.210. Image-only portraits yield MSE 0.028, Spearman 0.096, Pearson 0.121, and Kendall 0.065. A traditional baseline using TF-IDF texts and PIL-to-tensor images truncated to 750D per modality with a simple MLP yields MSE 0.412, Spearman 0.195, Pearson 0.183, and Kendall 0.130. The paper interprets these results as evidence that images are complementary to text but weak in isolation: adding portraits to text improves Pearson and Spearman by about 0.1 and reduces error from 0.012 to 0.011, whereas vision without textual context has limited predictive value.

The work presents itself as the first proposal to address pre-investment anime popularity prediction with a multimodal dataset. Its end-to-end pipeline includes crawling MAL titles, synopses, weighted scores, main character names, descriptions, and portraits; filtering; tokenization and image preprocessing; clustered train/test split; embedding extraction; fusion; regression; and de-normalization of predictions back to the 0–10 scale. Reproducibility details include seed 42, PyTorch 1.10.1, Huggingface Transformers, and AutoImageProcessor, although the paper does not announce public code or data release.

6. Limitations, misconceptions, and comparative significance

The speech-oriented AnimeScore explicitly rejects a common misconception: anime-likeness is not well approximated by “higher pitch.” Mean $$\mathbf{Z} = \mathrm{BiLSTM}(\mathbf{H}),\quad \bar{\mathbf{z}} = \tfrac{1}{T'} \sum_{t=1}^{T'} \mathbf{z}_t,\quad s(x) = \mathrm{MLP}(\bar{\mathbf{z}}).$$3 shows only weak concordance at 55.1%, while stronger cues arise from lower median formants, higher voicing ratio, sustained spectral activity, and a pattern of higher syllable rate with fewer and shorter pauses (Park et al., 12 Mar 2026). The paper also identifies limitations in dataset scale and demographics, noting 3,000 clips and 15,000 pairs, a male-heavy annotator pool, Japanese and anime-industry-centric style bias, and possible domain shift for unseen languages or speaking styles. Ethical considerations include the possibility that optimizing for anime-like style may over-emphasize aesthetic attributes at the expense of intelligibility or inclusivity.

The popularity-oriented AnimeScore also requires caution in interpretation. MAL’s weighted average score is treated as a robust and widely accepted proxy for global popularity, but it is not demographic-specific, and the label distribution is imbalanced at the tails, with few titles below 2 or above 9 (Armenta-Segura et al., 2024). Character description truncation at 256 tokens can discard information, and lower Kendall’s Tau is attributed to tail sensitivity and ranking inversions at extremes. The vision branch underperforms on its own, which counters any simplistic claim that character portraits alone determine market appeal. The paper also notes possible domain shifts from seasonality, genre cycles, studio reputation, and macrotrends not captured by text and images alone.

Taken together, the two AnimeScore systems illustrate two different ways of formalizing anime-related judgment. One formalizes an aesthetic target as a learnable ranking problem over perceptual comparisons; the other formalizes anticipated audience response as multimodal supervised regression. A plausible implication is that the shared name reflects a broader design pattern—compressing complex anime-associated constructs into operational scalar scores—while the validity of each score remains entirely task-specific.