Controlled Multimodal Multiplication Benchmark

• The paper introduces a paired evaluation suite that isolates arithmetic competence of multimodal LLMs by controlling operand structure and difficulty.
• It employs a factorial design to systematically vary digit length, sparsity, representation, and modality, ensuring reproducible paired instances.
• The benchmark quantifies performance degradation using arithmetic load metrics and regression analyses, revealing modality-specific and heuristic effects.

Searching arXiv for the cited benchmark paper and closely related work to ground the article. The Controlled Multimodal Multiplication Benchmark is a paired evaluation suite for exact integer multiplication in multimodal LLMs, designed so that the same underlying arithmetic problem can be presented as numerical text, alphabetic text, rendered images, and audio while preserving controlled variation in operand structure and arithmetic difficulty (Balter et al., 20 Apr 2026). Its central purpose is to distinguish arithmetic competence from modality-specific perception, and to quantify how multiplication accuracy degrades as the computational burden of a problem increases. The term “multiplication benchmark” can also denote an unrelated systems-benchmarking tradition exemplified by GEMMbench, which evaluates generic matrix-matrix multiplication kernels of the form $C = \alpha A \times B + \beta C$ under controlled experimental pipelines (Lokhmotov, 2015). In the multimodal-LLM setting, by contrast, the benchmark targets exact multi-digit arithmetic, cross-modal equivalence, and heuristic preference rather than numerical linear algebra throughput.

1. Scope, object of measurement, and conceptual position

The benchmark was introduced to address a specific methodological problem: multimodal LLMs may accurately perceive numerical content across modalities yet fail to perform exact multi-digit multiplication when the identical arithmetic problem is expressed as numerals, number words, images, or audio (Balter et al., 20 Apr 2026). Existing benchmarks were characterized as often lacking systematically paired instances across modalities, making it difficult to compare genuine arithmetic limits both within and across model families. The controlled benchmark therefore factorially varies digit length, digit sparsity, representation, and modality, with paired instances from a reproducible generator.

Its unit of analysis is an exact multiplication problem over positive integers. The benchmark does not study approximate numeracy, symbolic algebra in general, or open-ended reasoning traces. The scoring target is the final integer product, and the design keeps surface-form differences as controlled as possible so that degradation can be attributed to arithmetic load rather than uncontrolled prompt differences.

A common ambiguity arises from the phrase “multiplication benchmark.” In computer systems research, GEMMbench standardizes benchmarking of generic matrix multiplication implementations, reports performance in Gflop/s, captures program, dataset, environment, and parameters through Collective Knowledge pipelines, and studies effects of kernel design, local work size, cache utilization, and energy consumption (Lokhmotov, 2015). The controlled multimodal multiplication benchmark instead evaluates exact multiplication in multimodal LLMs using paired prompts and regression analyses over arithmetic load. The shared vocabulary of “multiplication” therefore masks a substantive distinction between kernel-performance benchmarking and model-level arithmetic evaluation.

2. Benchmark construction, pairing, and released resources

The benchmark uses a factorial design over four factors: digit length, digit sparsity, representation type, and modality (Balter et al., 20 Apr 2026). Digit length is the total number of digits across the two operands, $n+m$, ranging from single-digit cases to long multi-digit values; the extended analyses include pairs up to 10 digits each, with an example of $1{,}632{,}178{,}320 \times 5{,}683{,}473{,}970$. Digit sparsity is controlled by operand templates that determine non-zero positions and trailing zeros. The template family includes dense forms $V$, $VV$, and $VVV$, and structured-sparsity forms $V0$, $V00$, $VV0$, and $V0V$.

Representation is varied between numerals and number words. Modality is varied across text, rendered images of the same prompts, and audio. Instances are paired across modalities and representation types, so that one underlying multiplication problem can appear as numeric text, alphabetic text, images of both texts, and audio where available. This pairing is the benchmark’s central control mechanism: it enables comparison of arithmetic behavior under matched operands rather than under loosely related test items.

The generator samples operand pairs from digit templates, fills non-zero slots from $n+m$0, sets zero slots explicitly, enforces no leading zeros, computes metadata, renders modality-specific variants, and enforces uniqueness up to commutativity, treating $n+m$1 as the same problem as $n+m$2 (Balter et al., 20 Apr 2026). The paper describes this workflow at a high level and notes that exact rendering and audio specifications are in the released dataset; fonts, resolution, and TTS settings are not specified in the paper itself.

The released paired benchmark contains 10,000 shared multiplication instances, each rendered as text, an image, and, where possible, audio. A separate Heuristic-Disagreement Set, or HDS, contains 1,000 problems labeled by an OT/DD/RC cost model and split 70/15/15 into train, validation, and test with enforced separation and uniqueness. An additional Adversarial Traps set contains 30 test-only items designed to make familiar heuristic cues misleading, including anti-rounding traps. Per-instance metadata store operands $n+m$3, the product, template family, representation, modality channel, arithmetic load $n+m$4, and, for HDS items, a target heuristic label and category. The data are released at https://huggingface.co/datasets/cjerzak/MultimodalMathBenchmarks, and a public leaderboard and interactive results are available at https://neuristemic.ai/multiplication-in-multimodal-LLMs/.

3. Arithmetic load, sparsity structure, and problem geometry

The benchmark’s principal scalar difficulty measure is arithmetic load, denoted $n+m$5 (Balter et al., 20 Apr 2026). For positive integers $n+m$6 and $n+m$7 with digit lengths $n+m$8 and $n+m$9, and non-zero digit counts $1{,}632{,}178{,}320 \times 5{,}683{,}473{,}970$0 and $1{,}632{,}178{,}320 \times 5{,}683{,}473{,}970$1, the benchmark defines

$1{,}632{,}178{,}320 \times 5{,}683{,}473{,}970$2

This quantity is intended as a compact proxy for algorithmic burden. It grows with both total length and total non-zero digit count, and can be rewritten as

$1{,}632{,}178{,}320 \times 5{,}683{,}473{,}970$3

The paper relates $1{,}632{,}178{,}320 \times 5{,}683{,}473{,}970$4 to familiar multiplication operation counts through the bound

$1{,}632{,}178{,}320 \times 5{,}683{,}473{,}970$5

so that $1{,}632{,}178{,}320 \times 5{,}683{,}473{,}970$6 upper-bounds the count of non-zero digit-by-digit products. It also notes that $1{,}632{,}178{,}320 \times 5{,}683{,}473{,}970$7 or $1{,}632{,}178{,}320 \times 5{,}683{,}473{,}970$8 correspond to digit-multiplication counts in one-sided expansions, with exactly $1{,}632{,}178{,}320 \times 5{,}683{,}473{,}970$9 of those products being non-zero. This suggests that $V$0 is a deliberately coarse but mechanistically motivated surrogate for intermediate arithmetic work.

The benchmark’s use of digit templates makes sparsity a first-class variable rather than a byproduct of randomly sampled integers. A problem such as $V$1 has $V$2, $V$3, $V$4, and $V$5, giving $V$6, $V$7, and $V$8. By contrast, $V$9 has $VV$0, $VV$1, $VV$2, and $VV$3, giving $VV$4, while $VV$5 has $VV$6, $VV$7, $VV$8, and $VV$9, giving $VVV$0. These examples show that equal or similar digit lengths need not imply equal arithmetic load; the non-zero structure materially changes the benchmark’s difficulty measure.

The paper contrasts $VVV$1 with more complex, carry-aware operation counts and reports that the latter do not materially increase explanatory power in regression fits (Balter et al., 20 Apr 2026). A plausible implication is that, for the tested models and controlled prompts, coarse combinatorial structure already captures most of the variance relevant to exact multiplication accuracy.

4. Evaluation protocol, prompting, and statistical analysis

The benchmark evaluates multiple multimodal model families with modality coverage that varies by system (Balter et al., 20 Apr 2026). The reported models are google-gemini-2.5-flash for text, image, and audio; google-gemini-3.1-pro for extended text, image, and audio trials; qwen-qwen3-vl-30b-a3b-instruct and qwen-qwen3-vl-235b-a22b-instruct for text and image; openai-gpt-4o for text and image; openai-gpt-4o-audio-preview for audio; openai-gpt-5.4 for text and image; and x-ai-grok-4.20 for text and image. Audio was not uniformly available for all model families.

The prompting format is intentionally short and direct. Numeric text uses prompts such as “What is $VVV$2?” Alphabetic text spells out the same operands, as in “forty-seven times thirty-six.” Image inputs render the same textual prompt as a clean equation-like image such as “$VVV$3”, and audio uses spoken number-word prompts. The paper emphasizes that all modalities for a given item reflect the same underlying operands and product, and that wording and formatting are kept short and consistent.

Accuracy is defined as exact-match correctness of the final integer product. To model difficulty effects, the paper fits a logistic regression for each model-modality pair:

$VVV$4

where $VVV$5 is the logistic function and $VVV$6 is $VVV$7 for item $VVV$8. Reported summaries include $VVV$9, $V0$0, the “50% $V0$1” threshold at which predicted accuracy equals $V0$2, and $V0$3, the fraction of outcome variability explained. Standard errors are computed across problem instances, with binomial standard error for accuracy and sample standard error for loss and support metrics. For the LoRA and forced-completion experiments on Qwen3-VL-30B and Qwen3-VL-235B, decoding is deterministic with temperature $V0$4 and a generation budget up to 2,048 tokens; the paper does not detail decoding settings for all other families beyond the LoRA pipeline.

5. Accuracy degradation, modality effects, and the perception-computation split

The dominant empirical finding is that multiplication accuracy declines sharply and monotonically with arithmetic load $V0$5, often nearing zero by $V0$6 on many model-modality pairs (Balter et al., 20 Apr 2026). Across model families, $V0$7 values often exceed $V0$8, indicating that $V0$9 functions as a strong one-dimensional predictor of exact multiplication performance even though it is simpler than more detailed operation-count measures.

Model	Modality	50% $V00$0, $V00$1
google-gemini-2.5-flash	numerical text	53.41, 0.5823
google-gemini-2.5-flash	alphabetic text	52.57, 0.5944
google-gemini-2.5-flash	image-numerical	52.85, 0.5785
google-gemini-2.5-flash	image-alphabetic	52.62, 0.5710
google-gemini-2.5-flash	audio	54.01, 0.5857
qwen-qwen3-vl-30b-a3b-instruct	numerical text	51.29, 0.5560
qwen-qwen3-vl-30b-a3b-instruct	alphabetic text	49.69, 0.5604
qwen-qwen3-vl-30b-a3b-instruct	image-numerical	49.76, 0.5364
qwen-qwen3-vl-30b-a3b-instruct	image-alphabetic	49.56, 0.5884
qwen-qwen3-vl-235b-a22b-instruct	numerical text	74.18, 0.3812
qwen-qwen3-vl-235b-a22b-instruct	alphabetic text	74.45, 0.3693
qwen-qwen3-vl-235b-a22b-instruct	image-numerical	75.15, 0.3451
qwen-qwen3-vl-235b-a22b-instruct	image-alphabetic	74.34, 0.3753
openai-gpt-5.4	numerical text	68.61, 0.6405
openai-gpt-5.4	alphabetic text	67.93, 0.6565
openai-gpt-5.4	image-numerical	66.92, 0.6212
openai-gpt-5.4	image-alphabetic	60.60, 0.6182
openai-gpt-4o	numerical text	52.99, 0.6320
openai-gpt-4o	alphabetic text	52.93, 0.6883
openai-gpt-4o	image-numerical	51.61, 0.6438
openai-gpt-4o	image-alphabetic	52.43, 0.6301
openai-gpt-4o-audio-preview	audio	53.07, 0.6208
x-ai-grok-4.20	numerical text	69.94, 0.5236
x-ai-grok-4.20	alphabetic text	65.27, 0.5024
x-ai-grok-4.20	image-numerical	61.87, 0.5078
x-ai-grok-4.20	image-alphabetic	50.84, 0.4071

These results support several comparative observations. Numerical text is typically strongest, and numerical images often track text closely. Alphabetic images are a consistent weak point in some families, notably x-ai-grok-4.20. Audio does not impose a uniform penalty under the controlled prompt conditions. The paper characterizes many cross-modal differences as threshold or intercept shifts rather than qualitatively different load-response slopes.

The benchmark also includes a perception-versus-computation decomposition. Matched-perception checks require models to read, extract, or repeat the operands across modalities rather than solve the multiplication. Perception accuracy exceeds 99% across modalities, even when multiplication accuracy drops substantially at high $V00$2 (Balter et al., 20 Apr 2026). This directly addresses a common misconception that multimodal degradation primarily reflects OCR, ASR, or prompt-reading failure. In the controlled setting of the benchmark, the degradation is primarily computational rather than perceptual.

An important exception is google-gemini-3.1-pro, which was reported as flawless within $V00$3 in extended trials but exhibited degradation starting near $V00$4. The paper notes that this model required much longer generations and far more output tokens, and therefore used evaluation up to $V00$5 for thorough analysis.

6. Heuristic preference, LoRA interventions, limitations, and implications

Beyond measuring failure rates, the benchmark investigates which multiplication procedures models appear predisposed to follow (Balter et al., 20 Apr 2026). It introduces a forced-completion loss probe that scores heuristic-specific reasoning prefixes under forced completion. For a fixed context $V00$6, the model is asked to continue with one of three heuristic-specific assistant preambles—OT for columnar or long multiplication, DD for distributive decomposition, and RC for rounding-compensation—or with a neutral baseline. The probe computes the length-normalized cross-entropy

$V00$7

and compares it with the neutral baseline through

$V00$8

Lower $V00$9 indicates stronger alignment with the heuristic’s initial trajectory. With equal-length templates, $VV0$0 is a log-likelihood ratio against the neutral baseline, and minimum loss corresponds to maximum likelihood under equal priors.

On a held-out split shared between text and image for Qwen3-VL-30B and Qwen3-VL-235B, DD is the most consistently favored continuation in the minimum-loss sense. The paper also reports a complementary contrastive step probe, using matched surface form with correct versus plausible incorrect next steps, that confirms grounding: preference for correct steps is at or near ceiling, with overall text performance at 100%/100% and image performance at 97.9%/100% for 30B/235B. This suggests that the models are not merely matching stylistic prefixes; they are sensitive to whether a proposed next step is computationally appropriate.

The authors then train heuristic-specific LoRA adapters for OT, DD, and RC on Qwen3-VL-30B and Qwen3-VL-235B using approximately 1,000 synthetic reasoning traces per heuristic, alongside a STYLE control adapter that imitates trace formatting only. The behavioral result is negative in the strict sense: heuristic LoRAs change arithmetic behavior frequently but do not improve out-of-distribution accuracy. Across 432 evaluations per model, the 30B model shows 114 behavioral flips, of which 1 improved and 113 degraded, and 313 heuristic flips; the 235B model shows 121 behavioral flips, of which 4 improved and 117 degraded, and 291 heuristic flips. STYLE-only control also degrades correctness. The paper interprets this as evidence that the base model maintains a well-tuned internal router over strategies that global heuristic forcing disrupts.

The geometric analysis of LoRA updates supports that interpretation. For each adapter, the effective update is flattened as $VV0$1, and cosine similarity is computed as

$VV0$2

The resulting heuristic-pair similarities are near-orthogonal: for 30B, OT-DD is $VV0$3, OT-RC is $VV0$4, and DD-RC is $VV0$5; for 235B, OT-DD is $VV0$6, OT-RC is $VV0$7, and DD-RC is $VV0$8. Sensitivity reruns also show higher same-heuristic than cross-heuristic similarity. This supports the claim that distinct heuristics occupy meaningfully different parameter subspaces.

The benchmark’s limitations are explicit. It focuses only on multiplication rather than addition, division, symbolic algebra, or multi-step word problems. Token-level probing and LoRA analysis are limited to Qwen3-VL-30B and Qwen3-VL-235B, while broader model families are evaluated only through accuracy curves. The images and audio are controlled rather than naturalistic, so the benchmark does not cover handwriting, scanned documents, complex visual scenes, or noisy speech. The template-driven data are synthetic and may not mirror real problem distributions, although the paper treats that synthetic control as methodologically valuable rather than incidental. Taken together, these limitations indicate that the benchmark is best understood as a controlled diagnostic instrument: it isolates arithmetic load, sparsity, representation, and modality sufficiently cleanly to show that multimodal LLM failures on exact multiplication are predominantly computational and strongly load-dependent, while also revealing a cross-modal preference for distributive decomposition and a model-internal routing mechanism that resists simple heuristic overriding (Balter et al., 20 Apr 2026).