---
title: Idealized CAT (ICAT) Score
url: https://www.emergentmind.com/topics/idealized-cat-icat-score
type: topic
---

# Idealized CAT (ICAT) Score

The Idealized CAT (ICAT) Score is a metric designed to offer a principled, interpretable summary of model performance by addressing challenges in both language model bias measurement and binary classification evaluation. The term "ICAT" represents two formally unrelated metrics from distinct lines of research: one for assessing language model stereotyping via meaningful/irrelevant triplets, and another for classifier evaluation via indistinguishability precision at a principled threshold. Despite their separate origins, both share rigorous mathematical grounding and address limitations found in alternatives such as AUC or F1. This article delineates both definitions with complete technical fidelity.

## 1. ICAT for Language Model Bias: Definition and Rationale

In evaluation of stereotypical bias in language models, the Idealized CAT Score (iCAT) is formulated to capture a tradeoff between language modeling competence and neutrality between stereotypical and anti-stereotypical completions [2502.01679]. Given a test set $\mathcal D=\{(S_i, S^+_i, S^u_i)\}_{i=1}^N$—where $S_i$ denotes a stereotypical, $S^+_i$ an anti-stereotypical, and $S^u_i$ an irrelevant sentence—the iCAT metric is built on two quantities:

- **Language Model Score (lms):**
  \[
  \mathrm{lms} = \frac{1}{N} \sum_{i=1}^N \mathbf{1}[L(S^u_i) < \max\{L(S_i), L(S^+_i)\}] \times 100
  \]
  $L(\cdot)$ is the model (pseudo-)log-likelihood.

- **Stereotype Score (ss):**
  \[
  \mathrm{ss} = \frac{1}{N} \sum_{i=1}^N \mathbf{1}[L(S_i) > L(S^+_i)] \times 100
  \]

The iCAT score itself is:
\[
\mathrm{iCAT} = \mathrm{lms} \times \frac{\min(100-\mathrm{ss},\,\mathrm{ss})}{50}
\]
This yields a value in $[0, 100]$, maximized only when the model always assigns highest probability to a meaningful (either stereotyped or anti-stereotyped) completion and shows perfect neutrality (ss = 50). The symmetry inherent in $\min(100-\mathrm{ss}, \mathrm{ss})$ ensures indifference between stereotyped and anti-stereotyped preferences is rewarded.

This design simultaneously penalizes language models that are either biased (ss far from 50) or lack discriminative power (low lms), producing a single-number summary that reflects both criteria.

## 2. ICAT in Classification: Precision at the Indistinguishability Threshold

For evaluating binary classifiers, the Idealized CAT Score (ICAT) is independently defined as precision at the "indistinguishability threshold" [2311.11422]. For a classifier assigning real-valued scores to all instances, let $S_P$ be the empirical distribution of true positive scores and $S_N$ the distribution for true negatives.

- **Indistinguishability Threshold $t$:**
  $t$ is the unique solution to:
  \[
  P(\text{score}(X_{\text{real\_cat}}) > \text{score}(X_{\text{labelled\_cat}})) = \frac{1}{2}
  \]
  Formally,
  \[
  B(t) = \frac{P v(t)^2/2 + N \int_{s=t}^{\infty} f_N(s) u(s) ds}{P v(t) + N u(t)} = \frac{1}{2}
  \]
  with $v(t) = P(S_P > t)$ and $u(t) = P(S_N > t)$.

- **ICAT Score (Precision at $t$):**
  \[
  \mathrm{ICAT} = \frac{|\{x: y=+1,\, \mathrm{score}(x) \ge t\}|}{|\{x: \mathrm{score}(x) \ge t\}|}
  \]
  That is, the precision when the threshold is set such that positively-labeled items are statistically indistinguishable from true positives in pairwise comparisons.

## 3. Step-by-Step Computation Procedures

### iCAT (Language Model Bias)

1. For each triplet, compute likelihoods $L(S_i)$, $L(S^+_i)$, $L(S^u_i)$.
2. Compute lms: fraction where $\max[L(S_i), L(S^+_i)] > L(S^u_i)$.
3. Compute ss: fraction where $L(S_i) > L(S^+_i)$.
4. Compute iCAT as $\mathrm{lms} \times \min(100-\mathrm{ss},\,\mathrm{ss})/50$.

### ICAT (Classification)

1. Sort unique classifier scores.
2. For each candidate threshold $r$:
   - Compute $v(r)$ as the positive-label survival fraction above $r$.
   - Compute $u(r)$ for negatives.
   - Evaluate $B(r)$ as above.
3. Find $t$ where $B(r)=0.5$ by interpolation.
4. Compute precision at $t$ for the final ICAT score.

## 4. Interpretation and Numerical Behavior

For language models, iCAT values near 100 indicate both high fluency and neutrality (ss ≈ 50, lms ≈ 100). iCAT collapses to zero for models that are either always biased (ss near 0 or 100) or lack the ability to score meaningful completions above irrelevant ones (lms ≈ 0). Mid-range values reflect partial failures in either attribute.

For classification, ICAT tracks the fraction of predicted positives that are true positives at the threshold where predicted positives are "statistically indistinguishable" from true positives, as formalized via $B(r) = 0.5$. Unlike AUC, ICAT is invariant to strict monotonic rescaling of scores and robust to class imbalance, since the balancing property absorbs the effect of "trivial negatives". In experimental settings with varying overlap between positive and negative distributions, ICAT reflects actual discriminative difficulty rather than being artificially inflated by class distribution [2311.11422]. 

## 5. Comparison to Related Metrics

| Metric        | Domain                | Core Principle                              |
|---------------|----------------------|---------------------------------------------|
| CAT           | Bias eval (CrowS-Pairs) | Biased preference rate, ignores irrelevance & fluency |
| iCAT          | Bias eval (StereoSet, LIBRA) | Combines language ability (lms) and neutrality (ss) |
| EiCAT         | Bias eval (LIBRA)     | Incorporates JSD divergence and local-word knowledge penalty (bbs) |
| AUC           | Classification        | Probability positive ranked above negative, threshold independent   |
| F1            | Classification        | Harmonic mean of precision/recall, ad hoc threshold                |
| ICAT          | Classification        | Precision at indistinguishability threshold (B=0.5), robust to label imbalance |

iCAT in language model bias subsumes ss (CAT Score) and penalizes lack of fluency, while EiCAT (from LIBRA) further incorporates Jensen–Shannon divergence and a "beyond knowledge boundary score" (bbs) to address context in which unfamiliar terms impede meaningful bias measurement [2502.01679]. In classification, ICAT avoids artifacts affecting AUC and F1 by anchoring threshold choice to empirical indistinguishability.

## 6. Illustrative Examples

### Language Model Bias Example

Given $N=4$ triplets, suppose:
- lms = 75 (i.e., model prefers a meaningful option 3/4 times)
- ss = 50 (equal preference for stereotype and anti-stereotype)

Then:
\[
iCAT = 75 \times \frac{\min(100-50,50)}{50} = 75 \times 1 = 75
\]
If a model is maximally fluent but completely biased (ss=100):
\[
iCAT = 100 \times \frac{\min(0,100)}{50} = 0
\]

### Classification ICAT Example *(from [2311.11422])*

In artificial datasets with varying overlap between normals $N(\mu,2)$:
- "Easy" regime: ICAT ≈ 0.85
- "Moderate": ICAT ≈ 0.69
- "Hard": ICAT ≈ 0.50

These reflect intrinsic difficulty and remain stable under label-imbalance manipulations.

## 7. Strengths, Limitations, and Extensions

**Strengths:**
- iCAT: Integrates fairness and language discrimination; symmetric; transparent computation; penalizes extreme preference or lack of competence [2502.01679].
- ICAT: Robust to class imbalance and "easy negatives"; anchored threshold yields direct interpretability; immune to pitfalls of AUC/F1 [2311.11422].

**Limitations:**
- iCAT: Reduces full distributional information to two statistics (ss, lms); may miss preference strength nuance; requires triplet format with a crafted irrelevant case; all test cases equally weighted, lacking stereotype severity adaptation.
- ICAT: Focuses on the single indistinguishability point, does not capture full sensitivity/recall tradeoff.

**Extensions:**
- EiCAT: Combines iCAT’s lms with JSD-based divergence and bbs to measure local context comprehension and bias distributionally.
- For ICAT, the indistinguishability criterion may be replaced by $B(r)=\alpha$ for other balances between positive and predicted classes, generalizing the notion of controlled tradeoff.

A plausible implication is that both ICAT formulations serve as templates for single-number metrics that are robust to frequent artifacts affecting more commonly used measures, provided construction and application align with their rigorous criteria.

Source: https://www.emergentmind.com/topics/idealized-cat-icat-score