MultiTempBench: Multilingual Temporal Benchmark

• MultiTempBench is a benchmark featuring 15,000 Q&A instances across three tasks—date arithmetic, time zone conversion, and temporal relation extraction—in five languages and three calendar systems.
• The benchmark assesses whether LLM temporal reasoning is driven by tokenisation quality or the internal geometry of time representations, contrasting high- and low-resource language effects.
• It employs controlled date-format variations to isolate the impact of surface form and token fragmentation, providing actionable insights into model performance across diverse linguistic settings.

Searching arXiv for the specified paper to ground the article and citation. MultiTempBench is a multilingual, multi-calendar benchmark for evaluating temporal reasoning in LLMs, introduced in "What Really Controls Temporal Reasoning in LLMs: Tokenisation or Representation of Time?" (Bhatia et al., 19 Mar 2026). It comprises $15{,}000$ question–answer instances spanning three temporal tasks—date arithmetic, time zone conversion, and temporal relation extraction—across five languages, English, German, Chinese, Arabic, and Hausa, and across Gregorian, Hijri, and Chinese Lunar calendar conventions. The benchmark is designed to separate two candidate determinants of temporal performance: the surface accessibility of temporal information under tokenisation, and the geometry of internal temporal representations once dates have been encoded.

1. Benchmark Scope and Core Design

MultiTempBench is organised around three axes of variation: task family, language, and calendar system (Bhatia et al., 19 Mar 2026). The task families are date arithmetic, time zone conversion, and temporal relation extraction. The language set is English, German, Chinese, Arabic, and Hausa. The calendar coverage includes Gregorian examples in all languages, Hijri examples in Arabic and Hausa together with English Hijri examples, and Chinese Lunar examples in Chinese.

The benchmark is built from $750$ curated English questions. Each question is translated into four additional languages and expanded into four date-format variants, yielding $750 \times 5 \times 4 = 15{,}000$ instances. Each language therefore contains $3{,}000$ examples, and each task contributes $5{,}000$ instances after expansion.

Dimension	Coverage	Details
Tasks	3	Date arithmetic; time zone conversion; temporal relation extraction
Languages	5	English, German, Chinese, Arabic, Hausa
Calendars	3	Gregorian, Hijri, Chinese Lunar

The benchmark’s stated goals are to probe whether tokenisation preserves meaningful temporal units such as year, month, day, calendar markers, and delimiters; whether dates are internally encoded along linear temporal axes; and how these mechanisms differ between high-resource and low-resource settings. The central research question is whether temporal reasoning is controlled primarily by tokenisation or by representation of time. Because the benchmark holds semantic content fixed while varying surface form, it functions as a controlled environment for disentangling these factors.

2. Task Families and Formal Problem Setup

The date arithmetic task tests symbolic addition and subtraction over dates. Inputs are natural-language questions containing a fully specified date and a duration, and outputs are specific dates. The paper formalises the date as $d = (Y, M, D)$ and the operation as calendar-aware addition $d' = d \oplus (\Delta Y, \Delta M, \Delta D)$, explicitly requiring respect for month lengths, leap years, and the relevant calendar system (Bhatia et al., 19 Mar 2026). Example instances include English Gregorian questions such as “What date is 60 days after 05/01/1225?” and Arabic Hijri questions such as “إذا بدأ الحدث في ٣ رمضان ١٤٤٥ هـ واستمر ٤٠ يومًا، فما تاريخ انتهائه؟”.

The time zone conversion task evaluates handling of time zone offsets together with date rollover. Inputs specify a date, time, source time zone, and target time zone. The benchmark expresses the transformation as $$t_\text{tgt} = t_\text{src} + (\text{offset}_\text{tgt} - \text{offset}_\text{src})$$, applied in the appropriate calendar system. The paper’s canonical example asks for the conversion of “2 AM on 1352-03-02 in Asia/Singapore” to “Europe/Athens,” with the correct output “8 PM on 1352-03-01.”

The temporal relation extraction task requires assigning a temporal label between an event and a reference time or interval. The label inventory includes BEFORE, AFTER, and IS_INCLUDED, and the paper also refers to “before, after, simultaneous” in describing the task family. The formal object is a function $\text{Rel}(t, r)$ over event and reference intervals. The benchmark’s example concerns the relation between the event “redefined” and the time “April 1, 1997,” with gold label IS_INCLUDED.

These three tasks jointly span arithmetic manipulation, offset computation, and relation classification. This suggests that MultiTempBench is intended not merely as a format-robustness test, but as a controlled probe of several distinct computational subroutines that underlie temporal reasoning.

3. Languages, Calendars, and Controlled Surface Variation

The benchmark distinguishes high-resource and low-resource languages using CommonCrawl-derived corpora, specifically CC-100/FineWeb2 (Bhatia et al., 19 Mar 2026). English ($\sim 300$ GB), German ($750$0 GB), and Chinese ($750$1 GB) are treated as high-resource, while Arabic ($750$2 GB) and Hausa ($750$3 GB) are treated as low-resource. This resource distinction is central to the paper’s analysis.

Gregorian formatting varies by language and template. The benchmark includes ISO YYYY-MM-DD, local numeric formats such as [DD](https://www.emergentmind.com/topics/data-diversification-dd)/MM/YYYY and DD.MM.YYYY, and language-specific textual forms such as YYYY年MM月DD日, DD Month YYYY, DD ga Month YYYY, and “Am DD. Month des Jahres YYYY”. The benchmark also varies delimiters, ordering conventions, scripts, and numeral systems, including Latin script, Chinese characters, and Arabic script with Arabic-Indic numerals.

Hijri examples are used for Arabic and Hausa, with English Hijri examples added for cross-calendar stress-testing. English Hijri dates follow the pattern DD Month YYYY AH, while Arabic Hijri dates use the pattern DD Month YYYY هـ. Gregorian-to-Hijri conversion is performed via the HijriDate Python package. Chinese Lunar examples are used in Chinese, with both Gregorian textual Chinese and Lunar textual Chinese formats represented; conversion uses a lunar calendar library.

A central design feature is the use of controlled date-format variants. Every underlying date is represented in multiple surface forms that differ only in format and not in semantic content. The benchmark’s unified Python pipeline first extracts and normalises dates to an internal $750$4 representation and then re-realises them in language-specific formats. Because the underlying question semantics remain constant, observed performance differences across variants can be attributed to surface form and its tokenisation rather than to content.

4. Dataset Construction, Validation, and Evaluation Protocol

The $750$5 English seed questions are drawn evenly from TRAM, ToT, and FreshBench, with $750$6 questions from each source (Bhatia et al., 19 Mar 2026). The curation criteria include fully specified dates with year, month, and day, removal of synthetic entities, removal of internal prompt instructions, and natural-language form. These questions are then translated into German, Chinese, Arabic, and Hausa.

Translation quality control is performed by two native speakers for each target language. Their checks cover meaning preservation, correctness and naturalness of date expressions, and language-specific calendar words and grammar, including German case, Arabic right-to-left behaviour, and Hausa “ga”. Calendar conversion outputs for Hijri and Lunar forms are also manually verified by native speakers.

The evaluation pipeline uses LLM-as-a-judge with GPT-4o. For each prediction, GPT-4o is given the question, gold target or aliases, and the model prediction, and outputs one of A = CORRECT, B = INCORRECT, or C = NOT_ATTEMPTED. The primary reported metric is accuracy. Validation on $750$7 multilingual examples yields $750$8 agreement with majority human labels and Cohen’s $750$9.

The reported inference setting is strict zero-shot: no fine-tuning, no demonstrations, no chain-of-thought prompting, and no external tools. This setup isolates base-model behaviour under direct answer generation. A plausible implication is that the reported scores should be read as measurements of intrinsic multilingual temporal competence under minimally scaffolded prompting rather than as ceilings achievable with retrieval, tools, or specialised adaptation.

5. Tokenisation Analysis and the Multilingual Date Fragmentation Ratio

MultiTempBench introduces the multilingual Date Fragmentation Ratio, or mDFR, as a scalar in $750 \times 5 \times 4 = 15{,}000$0 that measures how badly a tokenizer fragments a date string (Bhatia et al., 19 Mar 2026). The metric extends the Date Fragmentation Ratio of Bhatia et al. (2025) to multilingual, multi-calendar settings. It combines four components: whether semantic roots are split, whether delimiters are lost or merged, token count inflation relative to a language-aware baseline, and a cosine-distance term $750 \times 5 \times 4 = 15{,}000$1 that quantifies structural divergence between model tokenisation and baseline tokenisation. The learned weights are approximately $750 \times 5 \times 4 = 15{,}000$2, making structural divergence and root splitting the dominant contributors.

The metric is calibrated against human severity ratings. The stimulus set contains $750 \times 5 \times 4 = 15{,}000$3 tokenised date strings spanning languages, calendars, tokenisers, and fragmentation severities. Five hypothesis-blind graduate students rate fragmentation severity on a $750 \times 5 \times 4 = 15{,}000$4–$750 \times 5 \times 4 = 15{,}000$5 scale. Inter-annotator reliability is strong, with Krippendorff’s $750 \times 5 \times 4 = 15{,}000$6. Spearman correlation with mDFR is $750 \times 5 \times 4 = 15{,}000$7, compared with $750 \times 5 \times 4 = 15{,}000$8 for original DFR, $750 \times 5 \times 4 = 15{,}000$9 for BLEU, and $3{,}000$0 for character edit distance.

Qualitative examples show substantial cross-linguistic variation. In German, 10. Oktober 2034 may be tokenised as 1 | 0 | . | Oktober | 2 | 0 | 3 | 4, preserving the month and delimiter but splitting digits. In Chinese textual Gregorian format, 2034 年 10 月 10 日 may preserve 年, 月, and 日 while fragmenting the numerical components. In Arabic Hijri and Hausa textual formats, fragmentation is more semantically disruptive: month names and calendar markers may themselves be split, and individual digits are often separated. The paper reports Hausa Gregorian textual Oktoba 10, 2034 tokenised as O | kt | oba | 1 | 0 | , | 2 | 0 | 3 | 4, with mDFR $3{,}000$1 and human rating $3{,}000$2.

The reported pattern is that high-resource languages tend to exhibit moderate fragmentation in which delimiters and month names are mostly preserved, whereas low-resource languages more often exhibit semantic fragmentation of month names, calendar markers, and numerals. Non-Gregorian formats, particularly Chinese Lunar and Hijri variants, frequently have elevated mDFR. This supports the paper’s characterisation of a “token tax” imposed by tokenisers that are optimised for high-resource languages and common temporal formats.

6. Internal Temporal Geometry, Empirical Results, and Interpretive Significance

Beyond tokenisation, the paper studies internal temporal representation geometry through linear probes and trajectory analysis in hidden-state space (Bhatia et al., 19 Mar 2026). For each language, year, and date format, dates are embedded in a simple declarative template, hidden states are extracted at a given layer for the final token, and year centroids are formed by averaging over $3{,}000$3 dates per year. Consecutive-year difference vectors define a temporal path, and linear regressors are trained to decode Year, Month, and Day. Temporal linearity is measured by the resulting $3{,}000$4.

The geometric findings are layer-dependent. At layer $3{,}000$5, trajectories are chaotic and heavily influenced by surface form. Around early layers such as layer $3{,}000$6, language clusters remain separated by syntax without a clear linear temporal dimension. Around middle layers such as layer $3{,}000$7, English, German, and Chinese begin to form smoother, nearly linear year trajectories, while Hausa remains curved or disorganised. At reasoning layers around $3{,}000$8, English, German, and Chinese trajectories are described as almost straight lines, whereas Hausa remains non-linear; output layers around $3{,}000$9 separate again for lexical decoding.

Correlations between overall temporal linearity and average task accuracy differ substantially by language. The paper reports $5{,}000$0 for English, $5{,}000$1 for Chinese, $5{,}000$2 for German, $5{,}000$3 for Arabic, and $5{,}000$4 for Hausa. Year linearity is the most consistent predictor, while month and day linearity are weaker and more variable. This suggests that in higher-resource settings, successful temporal reasoning depends strongly on whether models have organised year information along a stable linear axis.

Model evaluation covers 19–20 decoder-only LLMs, including GPT-4o and open-weight families such as Gemma 3, LLaMA 3/3.1, Phi-4 Mini, Qwen 3, Mistral, LLaMA 2, OLMo, DeepSeek / DS-R1 Qwen 7B, GPT-OSS 20B, and Cohere models. On average accuracy over all three tasks, GPT-4o achieves $5{,}000$5, with language-specific scores of $5{,}000$6 in Arabic, $5{,}000$7 in Chinese, $5{,}000$8 in English, $5{,}000$9 in German, and $d = (Y, M, D)$0 in Hausa. The best open model reported is Gemma 3 4B at $d = (Y, M, D)$1 average, ahead of LLaMA 3.1 8B at $d = (Y, M, D)$2. Qwen 14B is reported at $d = (Y, M, D)$3 average, and Qwen 3 4B records only $d = (Y, M, D)$4 in Hausa.

The paper’s main empirical conclusion is resource-dependent. In low-resource languages, fragmentation is the stronger predictor of performance. For date arithmetic, the correlation between mDFR and accuracy is $d = (Y, M, D)$5 in Hausa and $d = (Y, M, D)$6 in Arabic, while correlations in English, German, and Chinese are weak. Comparable patterns are reported for time zone conversion and temporal relation extraction. In a crossed mixed-effects logistic regression over $d = (Y, M, D)$7 predictions, the three-way interaction among mDFR, linearity, and resource is significant, with $d = (Y, M, D)$8, $d = (Y, M, D)$9, $d' = d \oplus (\Delta Y, \Delta M, \Delta D)$0, and $d' = d \oplus (\Delta Y, \Delta M, \Delta D)$1. In low-resource languages, fragmentation strongly predicts lower accuracy, with $d' = d \oplus (\Delta Y, \Delta M, \Delta D)$2, $d' = d \oplus (\Delta Y, \Delta M, \Delta D)$3. In high-resource languages, temporal linearity is the stronger predictor, with $d' = d \oplus (\Delta Y, \Delta M, \Delta D)$4, $d' = d \oplus (\Delta Y, \Delta M, \Delta D)$5, while mDFR has only a weak effect, with $d' = d \oplus (\Delta Y, \Delta M, \Delta D)$6, $d' = d \oplus (\Delta Y, \Delta M, \Delta D)$7.

A common misconception would be that tokenisation uniformly determines temporal reasoning quality across languages. MultiTempBench rejects that universal claim: tokenisation is a major bottleneck in low-resource and rare-calendar regimes, but not universally. A second misconception would be that scale alone determines success. The benchmark reports several cases in which smaller models outperform larger ones, indicating that multilingual coverage and tokenizer design can matter more than parameter count. The paper also notes explicit limitations: only five languages are covered; only three task families are included; non-English items are translated from English and templated; the setup is zero-shot without tool use or chain-of-thought; and the mechanistic analysis is correlational rather than causal. Within those bounds, MultiTempBench provides a controlled framework for diagnosing whether failures arise at the level of surface accessibility or at the level of internal temporal geometry.