CVLAT: Counterfactual Visualization Literacy Assessment

• CVLAT is a counterfactual assessment framework that evaluates causal reasoning using juxtaposed visualizations of factual, counterfactual, and remainder subsets.
• It distinguishes visual correctness from factual correctness, providing a benchmark for both human chart reading and large vision-language models.
• The framework leverages tasks spanning recognition, intervention, analysis, and recall to measure visualization literacy and causal inference competencies.

Searching arXiv for the cited papers to ground the article. Counterfactual Visualization Literacy Assessment Test (CVLAT) denotes a counterfactual assessment framework for evaluating how chart readers reason when visual evidence must be interpreted causally or when visual evidence conflicts with prior factual knowledge. In one formulation, CVLAT is grounded in a preliminary model of causality comprehension derived from visualization and causal inference, with competencies spanning Recognize, Understand, Analyze, and Recall, and with item designs based on static counterfactual visualizations such as juxtaposed small multiples of included, counterfactual, and remainder subsets (Wang et al., 2024). In another formulation, CVLAT is a benchmark for Large Vision-LLMs (LVLMs) that deliberately pits chart evidence against widely shared factual knowledge in order to disentangle visual correctness from factual correctness and to measure arbitration between the two sources (Lee et al., 2 Jun 2026). Taken together, these formulations define CVLAT as an assessment paradigm centered on counterfactual conflict, causal reasoning, and the fidelity with which a respondent follows what a visualization encodes.

1. Conceptual basis and scope

In the causality-oriented formulation, the competency backbone is a preliminary model that maps causal inference levels—association, intervention, and counterfactual—to progressive visual communication objectives: Recognize, Understand, Analyze, and Recall (Wang et al., 2024). Recognize (Association) concerns perceiving statistical relationships rendered in a chart without inferring causality and decoding encodings and simple trends. Understand (Intervention) concerns reasoning about the effect of manipulating one variable on another and making predictions consistent with an intervention framing. Analyze (Counterfactual) concerns integrating cross-variable insights, detecting confounding, distinguishing causal effects from spurious associations, and reasoning about “what would happen if $X$ had been different.” Recall combines counterfactual reasoning with memorability, emphasizing the ability to encode and retrieve causal structures and visual summaries after a delay.

The same formulation specifies core constructs to assess: distinguishing correlation vs. causation; reasoning about interventions and the do-operator; counterfactual reasoning using factual vs. counterfactual outcomes; confounding detection and adjustment; causal explanation quality; and uncertainty interpretation and confidence calibration. Additional constructs for advanced items include mediation, selection bias, Simpson’s paradox, and predictive vs. causal framing (Wang et al., 2024).

In the LVLM-oriented formulation, the conceptual problem is different but related. Conventional visualization literacy benchmarks such as VLAT measure correctness on a single axis and cannot determine whether a correct answer reflects genuine reading of the visual encodings or recall of real-world facts from pretraining (Lee et al., 2 Jun 2026). This formulation therefore formalizes two orthogonal dimensions: visual correctness (VC), meaning that the answer matches what the chart encodes, and factual correctness (FC), meaning that the answer matches real-world facts independent of the chart. This suggests that CVLAT generalizes beyond chart decoding alone and targets the decision rule used when multiple epistemic sources diverge.

A useful way to relate the two formulations is that both treat counterfactual manipulation as an instrument for making latent reasoning observable. In the causal-visualization setting, the counterfactual is used to scaffold causal inference; in the LVLM benchmark, the counterfactual is used to expose whether a model follows the chart or overrides it with factual priors.

2. Causal inference foundations and counterfactual visualization design

The causality-oriented formulation explicitly anchors CVLAT in standard causal inference notation. Individual-level potential outcomes are represented as $Y(1)$ and $Y(0)$, or more generally $Y(x)$, where $Y(1)$ denotes the outcome if treated $(X=1)$ and $Y(0)$ denotes the outcome if not treated $(X=0)$ (Wang et al., 2024). The Average Treatment Effect is defined as

$\mathrm{ATE} = \mathbb{E}[Y(1) - Y(0)],$

and the Conditional Average Treatment Effect as

$$\mathrm{CATE}(z) = \mathbb{E}[Y(1) - Y(0) \mid Z=z].$$

Pearl’s intervention notation is also used:

$Y(1)$0

The back-door criterion is invoked in the standard form that a set $Y(1)$1 blocks all back-door paths from $Y(1)$2 to $Y(1)$3, after which adjustment via $Y(1)$4 identifies causal effect. The assumptions commonly invoked are ignorability,

$Y(1)$5

and SUTVA, defined as no interference and a single version of treatment (Wang et al., 2024).

Within this framework, a counterfactual visualization is defined as a juxtaposition of small multiples showing three subsets: IN, CF, and REM. IN is the included subset satisfying an inclusion criterion. CF is a counterfactual subset drawn from EX but matched to IN on covariates other than the inclusion criterion. REM is the remainder of EX not matched to IN. A second control visualization juxtaposes IN + EX without CF/REM, thereby providing whole-dataset context without matching (Wang et al., 2024).

CF subset creation is specified procedurally. For EX points, Euclidean distance to IN points is computed across all covariates except the inclusion criterion; the $Y(1)$6 points in EX with minimal total distance are selected to form CF; REM consists of the remaining EX points (Wang et al., 2024). The intended scaffolds for causal reasoning are equally explicit: juxtaposed small multiples support mental comparison consistent with do-operator intuition; hypothetical intervention annotations clarify the manipulated condition, such as “do(X=divorced)”; legends and encodings separate factual vs. counterfactual, for example solid marks for factual, dashed marks for counterfactual, and light gray for remainder; and optional overlays, ghost elements, and uncertainty bands can encode estimated $Y(1)$7 vs. $Y(1)$8 and their uncertainty (Wang et al., 2024).

The item construction guidance extends this design logic to assessment stimuli. Datasets should be multivariate and have clear inclusion criteria such as marital status, smoking, or credit limit, with covariates $Y(1)$9 available for confounding exploration. Encodings should use consistent axes across small multiples and common scales; charts should be horizontally aligned with shared legends and minimal occlusion; and at minimum the labels “IN,” “CF,” and “REM,” consistent axis ranges, and a clear legend should be present (Wang et al., 2024). A plausible implication is that CVLAT treats visual design not merely as presentation but as part of the construct being assessed, because weak legends or ambiguous encodings directly alter what can be inferred causally.

3. Assessment architecture and item types

The human-centered CVLAT blueprint adapts four task types from the empirical study: T1 Recognize, T2 Understand interventions, T3 Analyze causality, and T4 Recall (Wang et al., 2024). T1 is phrased as “Describe anything of interest you notice” and is operationalized as the count of correct descriptive findings, such as trends, extrema, and clusters, without causal inference. T2 asks respondents to predict the direction of outcome change if a variable is manipulated; correctness is assessed against a holdout set, and textual reasoning evidence is collected. T3 requires broader predictions about changes under hypothetical interventions and is scored for multi-faceted outcome accuracy and relative impact ratio. T4 occurs after approximately 10 minutes and asks respondents to list datasets and visuals remembered, associate them with stimuli, and count distinct recalls (Wang et al., 2024).

The same blueprint defines proficiency levels. Novice corresponds to Recognize/Association and requires identifying trends and clusters and reading encodings correctly, with thresholds of at least $Y(0)$0 correct descriptive findings on T1-like items regardless of subset type. Intermediate corresponds to Understand/Intervention and requires interpreting do-operator scenarios and predicting outcome direction under manipulation, with thresholds of at least $Y(0)$1 correct on T2 plus coherent textual justification referencing intervention. Advanced corresponds to Analyze/Counterfactual and requires detecting confounding, distinguishing correlation vs. causation, reasoning with matched counterfactuals, interpreting uncertainty, showing appropriate confidence calibration, and demonstrating recall of causal structures, with thresholds of at least $Y(0)$2 composite on T3, relative impact ratio at least $Y(0)$3, and T4 of at least $Y(0)$4 datasets on average across CF stimuli (Wang et al., 2024).

The item bank includes multiple-choice items with rich visuals, short-answer explanations, scenario-based intervention reasoning, confounding and mediation identification, Simpson’s paradox, selection bias, uncertainty and variance interpretation, and limitations of causal claims (Wang et al., 2024). The prompts use the same formal vocabulary as the theory: $Y(0)$5, $Y(0)$6, CI overlap, ignorability, SUTVA, back-door sets, and

$Y(0)$7

This suggests that the intended audience is not general chart readers but respondents expected to reason across visualization, causal inference, and statistical assumptions.

The LVLM benchmark formulation uses a different assessment architecture. It excludes purely perceptual tasks such as Find Anomalies and Find Clusters and focuses on interpretation tasks where visual–factual conflicts are meaningful (Lee et al., 2 Jun 2026). It contains 48 questions, derived from VLAT’s 53, all in multiple-choice format with 3–5 options. Each item includes a visually correct option, a factually correct option, Omit as an explicit uncertainty escape, and distractors unrelated to the main conflict. To neutralize ordering effects, each question is presented in 120 option-permutations: all unique permutations for 5-option items and proportional repetition for 3–4 option items (Lee et al., 2 Jun 2026).

The following table summarizes the two assessment formulations.

Formulation	Primary target	Core item structure
Causality-oriented CVLAT (Wang et al., 2024)	Human causal comprehension in static visualizations	T1–T4 tasks using IN, CF, and REM counterfactual visualizations
LVLM-oriented CVLAT (Lee et al., 2 Jun 2026)	Model arbitration between chart evidence and factual priors	48 multiple-choice counterfactual items with visual-correct and factual-correct options

4. Scoring, metrics, and validation methodology

The causality-oriented formulation uses several measurement dimensions. Accuracy is percentage correct per item; T3 uses multi-component scoring with the relative impact ratio

$Y(0)$8

Explanation quality is scored by axial coding for the presence of intervention framing, confounder reference, and consistency with visual evidence. Confidence is collected on a 5-point Likert scale and can be summarized with calibration measures such as Brier-like scores or an over/underconfidence gap. Response time is captured per item to support cognitive load and efficiency assessments. The proposed weighted composite scores are organized by construct: Recognize uses number of correct findings and time efficiency; Understand uses directional accuracy, explanation alignment to the do-operator, and confidence calibration; Analyze uses multi-component accuracy, relative impact ratio, confounding detection, explanation quality, and time; Recall uses the number of correctly recalled datasets and visuals and the specificity of causal takeaways (Wang et al., 2024).

The psychometric plan includes pilot testing for item difficulty and clarity, cognitive interviews on counterfactual annotations, internal consistency via Cronbach’s $Y(0)$9 per construct subscale, test–retest reliability, and inter-rater reliability via Cohen’s $Y(x)$0/ICC for open-ended explanation scoring and axial coding (Wang et al., 2024). Validity is partitioned into content validity, construct validity, convergent validity, divergent validity, and criterion validity. Item Response Theory is specified using 3PL models for multiple-choice items and graded response models for polytomous explanation items. Reporting includes subscores per construct, an overall CVLAT index on a 0–100 scale, proficiency classification by level thresholds, and a confidence calibration index defined as the difference between mean confidence and accuracy, penalizing overconfidence in causal claims (Wang et al., 2024).

The LVLM-oriented formulation introduces a different metric system intended to disentangle arbitration from capability. Its first component is correction for guessing. For question $Y(x)$1, with target response rate $Y(x)$2, distractor response rate $Y(x)$3, total options $Y(x)$4, and distractor options $Y(x)$5, the corrected score is

$Y(x)$6

In CVLAT, both the visual-correct and factual-correct options are focal categories, so $Y(x)$7; in the capability references Vanon and Q-only, $Y(x)$8 (Lee et al., 2 Jun 2026).

Visualization Fidelity Score (VF) normalizes adherence to visual evidence under conflict by an anonymized visual baseline:

$Y(x)$9

with $Y(1)$0 (Lee et al., 2 Jun 2026). Factual Alignment Score (FA) normalizes reliance on factual priors under conflict by a Q-only factual baseline:

$Y(1)$1

The Visual–Factual Reliance Index (VFRI) summarizes relative preference:

$Y(1)$2

and

$Y(1)$3

Interpretation is fixed: $Y(1)$4 is purely visual-oriented, $Y(1)$5 is purely factual-oriented, and $Y(1)$6 is balanced preference, with the explicit caution that near-zero cases should be interpreted conservatively if false-response rates are high (Lee et al., 2 Jun 2026).

The evaluation protocol for the LVLM benchmark uses per-question accuracy averaged across permutations, bootstrap resampling with 10,000 iterations for 95% confidence intervals, and descriptive grouping by the sign of point-estimate VFRI. Capability references are administered on the same items: Vanon removes domain-identifying cues through placeholders to measure pure chart-reading capability, while Q-only removes the chart entirely to measure factual-prior availability (Lee et al., 2 Jun 2026). Methodologically, this capability normalization is the central difference between CVLAT and single-axis literacy tests.

5. Empirical findings

The empirical study motivating the causal-visualization formulation used $Y(1)$7 participants, with 19 male and 13 female, normal or corrected vision, all with or pursuing university degrees, diverse domains, an average session of approximately 45 minutes, and \$10 compensation (Wang et al., 2024). The design was within-subjects: each participant answered three questions per task, one per visualization group (IN-only, IN+EX, IN+CF+REM), and T4 Recall was asked approximately 10 minutes post-study. Randomization avoided back-to-back tasks of the same type; chart type, participant differences, and trial order were treated as random covariates in analysis. Stimuli included line charts, bar charts, and scatterplots, with datasets such as UCI Credit Card default, Census Income, and selected public datasets used in examples, and a total of 27 visualization groups (Wang et al., 2024).

The primary analyses were 3-factor ANOVA for subset groups with chart type, trial order, and participant as covariates, followed by Tukey’s HSD with Bonferroni correction, with p-values, $Y(1)$8 effect sizes, and 95% bootstrapped confidence intervals reported (Wang et al., 2024). T1 Recognize yielded near-100% correctness across subset groups and no significant differences. T2 Understand showed visually higher average correctness with CF and a significant difference between IN and CF, with $Y(1)$9 and $(X=1)$0; EX showed larger variance. T3 Analyze showed a significant impact of subset groups on correctness,

$(X=1)$1

with CF giving the highest accuracy and the most compact distribution, and the relative impact ratio favoring CF. T4 Recall showed significant differences,

$(X=1)$2

with mean recalls per user of CF $(X=1)$3, EX $(X=1)$4, and IN $(X=1)$5 (Wang et al., 2024).

Response time increased with CF. For T2,

$(X=1)$6

with means of 72s for IN, 117s for EX, and 155s for CF. For T3,

$(X=1)$7

with means of 131s for IN, 170s for EX, and 232s for CF (Wang et al., 2024). Confidence was highest on average for CF in T2 and T3; scatterplots had lower average confidence but also lower incorrect-confidence than bar and line charts. Common errors included confusing observational trends in IN with causal effects, overgeneralizing from EX without matched similarity, and conflating REM with CF when legends or labels were weak (Wang et al., 2024).

The LVLM benchmark evaluated 15 models, including proprietary and open-source systems, across three experiments (Lee et al., 2 Jun 2026). Experiment 1 showed that several models reached or exceeded the human VLAT baseline of 65.50% on aligned VLAT under the Normal prompt, including Gemini-3.1-Pro at 99.73% (SD 1.73), Claude-Opus-4.7 at 94.87% (15.38), Claude-Sonnet-4.6 at 89.43% (26.43), and GPT-5.5 at 80.09% (26.93). On randomized reVLAT, Gemini-3.1-Pro and Claude-Opus-4.7 remained robust but dropped by 6.44 pp and 6.35 pp respectively. The interpretation given is that VLAT scores can be inflated by Source Ambiguity $(X=1)$8, while reVLAT incorrect answers mix Factual Override $(X=1)$9 with true visual failures and may therefore underestimate visualization literacy (Lee et al., 2 Jun 2026).

Experiment 2 used CVLAT to classify models by arbitration behavior. The visualization-oriented cohort, defined by point-estimate VFRI $Y(0)$0, included Gemini-3.1-Pro with VFRI $Y(0)$1 and 95% bootstrap CI $Y(0)$2, Claude-Opus-4.7 with $Y(0)$3 $Y(0)$4, Gemini-3.1-Flash-Lite with $Y(0)$5, Claude-Sonnet-4.6 with $Y(0)$6, GPT-5.5 with $Y(0)$7, Gemma-4-26B-A4B with $Y(0)$8, Gemma-4-31B with $Y(0)$9, Claude-Haiku-4.5 with $(X=0)$0, and Qwen3-VL-8B with $(X=0)$1, the last explicitly marked as a near-boundary case because the CI includes zero (Lee et al., 2 Jun 2026). The factual knowledge-oriented cohort, defined by point-estimate VFRI $(X=0)$2, included Grok-4.20 at $(X=0)$3, Qwen3-VL-32B at $(X=0)$4, Qwen3-VL-235B at $(X=0)$5, Llama4-Scout at $(X=0)$6, Grok-4.3 at $(X=0)$7, and Llama4-Maverick at $(X=0)$8, with Llama4-Maverick also marked as near-boundary because the CI includes zero (Lee et al., 2 Jun 2026).

The human baseline in the same CVLAT experiment used $(X=0)$9 Prolific participants who were not told the data were counterfactual (Lee et al., 2 Jun 2026). Using VLAT-style correction with the visual target, corrected accuracy was 53.71% (SD 14.92) and raw accuracy 60.76% (SD 11.70), statistically equivalent to published VLAT baselines within a $\mathrm{ATE} = \mathbb{E}[Y(1) - Y(0)],$0 pp margin under TOST with $\mathrm{ATE} = \mathbb{E}[Y(1) - Y(0)],$1. The factual-target diagnostic score was $\mathrm{ATE} = \mathbb{E}[Y(1) - Y(0)],$2 (SD 8.97), described as well below chance, and the mean per-participant VFRI without capability normalization was $\mathrm{ATE} = \mathbb{E}[Y(1) - Y(0)],$3 (SD 0.21; range $\mathrm{ATE} = \mathbb{E}[Y(1) - Y(0)],$4 to $\mathrm{ATE} = \mathbb{E}[Y(1) - Y(0)],$5), with all 30 participants visualization-oriented (Lee et al., 2 Jun 2026). The stated implication is that humans overwhelmingly follow the chart under conflict.

Experiment 3 examined prompt-based intervention. It found that prompt controllability is highly model-dependent, sometimes direction-asymmetric, and not predicted by visualization capability (Lee et al., 2 Jun 2026). Claude-Opus-4.7, Gemini-3.1-Flash-Lite, Claude-Sonnet-4.6, Gemma-4-31B, and Gemma-4-26B-A4B showed bidirectional shifts; GPT-5.5 and Gemini-3.1-Pro were V-priority-insensitive despite high chart-reading capability; Grok-4.20, Llama4-Scout, Grok-4.3, Llama4-Maverick, and Qwen3-VL-8B were F-priority-insensitive; and Qwen3-VL-32B and Qwen3-VL-235B exhibited the counter-intuitive pattern called factual-priority collapsing, in which the factual-priority prompt shifted them toward visual reliance (Lee et al., 2 Jun 2026).

6. Interpretation, limitations, and future directions

The empirical findings summarized for the causal-visualization formulation support a specific design conclusion: counterfactual small multiples improved Analyze (T3) and Recall (T4), did not harm Recognize (T1), tended to improve Understand (T2) relative to IN, and increased response time in a manner interpreted as consistent with deeper processing (Wang et al., 2024). The actionable design principles derived from those findings are explicit intervention markers such as “do(X=x),” juxtaposition of factual vs. counterfactual views, uncertainty encodings using error bars or bands around $\mathrm{ATE} = \mathbb{E}[Y(1) - Y(0)],$6 and $\mathrm{ATE} = \mathbb{E}[Y(1) - Y(0)],$7 estimates, clear legends distinguishing factual, counterfactual, and remainder, preference for scatterplots when covariance structure matters, and the use of decluttering tactics to limit visual clutter (Wang et al., 2024).

The LVLM benchmark formulation arrives at a parallel but distinct conclusion: high visualization accuracy alone is not sufficient evidence of faithful visual reasoning, because aligned tests can reflect factual recall and randomized tests can underestimate capability if factual priors dominate answers under conflict (Lee et al., 2 Jun 2026). The guidance given is to interpret VFRI together with VF, FA, and False rates; to treat near-zero VFRI cautiously when false-response rates are high; to select models according to whether the task should privilege the chart or real-world knowledge; and to verify prompt steerability empirically rather than assuming bidirectional controllability (Lee et al., 2 Jun 2026).

Both formulations also specify limitations. For the causal-visualization formulation, the study used static charts and university-attending participants, so broader piloting and accessibility accommodations are advised; evidence concerns static small multiples rather than interactive counterfactual exploration; causal truth is often uncertain, so scoring should emphasize reasoning quality, assumption awareness, and appropriate uncertainty; Euclidean matching may introduce bias and matching quality should therefore be disclosed; and the greater chart count in the CF condition means future assessments should hold the number of panes constant when evaluating cue effectiveness (Wang et al., 2024). For the LVLM benchmark, scope is limited to common chart families and excludes complex charts, domain-specific scientific plots, and non-numeric manipulations; 48 items are sufficient for cohort-level patterns but constrain granular per-task analysis; English-only prompts may miss multilingual behaviors; and future work is proposed on expanded chart types and conflict magnitudes, conflict-aware prompts and few-shot scaffolds, architectural interventions for predictable arbitration control, and mechanism-level probes into factual-priority collapsing (Lee et al., 2 Jun 2026).

A plausible implication of the combined literature is that CVLAT names a broader research program rather than a single immutable instrument. In one branch, it operationalizes visual causal inference literacy through counterfactual comparison, assumption checking, and calibrated confidence. In the other, it operationalizes arbitration between visual evidence and factual priors through capability-normalized metrics and controlled chart–fact conflict. Across both, the central claim is consistent: visualization literacy cannot be reduced to answer accuracy alone when counterfactual structure, confounding, uncertainty, or factual override are part of the inferential problem (Wang et al., 2024, Lee et al., 2 Jun 2026).