TimePuzzles: Temporal Reasoning Diagnostics

• TimePuzzles are constraint-based tasks that use temporal and spatial clues to rigorously evaluate systematic reasoning in language and vision-language models.
• They are generated using algorithmic methods that blend event facts, calendar arithmetic, and cross-cultural clues to create controlled, multi-solution puzzles.
• Benchmarks indicate LLMs struggle with implicit tasks requiring multi-step inference, highlighting the need for augmented tools like web retrieval and code interpretation.

TimePuzzles refer to a family of constraint-based tasks—originating in both natural language and vision-language modalities—that require iterative multi-step temporal (and sometimes spatial) reasoning. They have recently gained prominence as diagnostic tools for evaluating the systematic, combinatorial, and knowledge-driven inference capabilities of LLMs and vision-LLMs (VLMs). TimePuzzles typically blend factual, calendar, and event-anchored clues, require systematic constraint resolution (often under ambiguity or noise), and expose the limitations of pattern-matching models that lack compositional temporal reasoning. The term spans both algorithmically generated textual tasks such as the TimePuzzles benchmark and vision-based tasks as instantiated in modular systems like PuzzleGPT.

1. Formal Task Structure and Evaluation Protocols

Algorithmically generated TimePuzzles in the text domain are defined as follows. Let $\mathcal{D}$ be the set of all Gregorian dates. Each natural-language temporal fact $t$ is grounded via a function $\mathcal{C}$ to a date subset $\mathcal{C}(t) \subseteq \mathcal{D}$. Given $N$ facts $F = \{t_1, ..., t_N\}$, the target answer set is

$\mathcal{A} = \bigcap_{i=1}^N \mathcal{C}(t_i)$

with the dataset designed to enforce $1 \le |\mathcal{A}| \le 6$ for controlled ambiguity. Each fact $t$ is also annotated with an information-gain score to facilitate difficulty calibration:

$$\mathrm{IG}(t) = \log_2 |\mathcal{D}| - \log_2 |\mathcal{C}(t)|$$

Puzzles are generated through a randomized, search-validated algorithm that selects a real-world event anchor (from a curated template set), samples a seed date by web search, and instantiates $N-1$ additional calendar-based constraints (including cross-cultural relations). The constraints are then validated and filtered to produce a solution set of exact cardinality $M$ ($1 \leq M \leq 6$).

Two parallel datasets are constructed:

• Implicit: event facts (e.g., "the day Kobe Bryant passed away") requiring retrieval.
• Explicit: fully grounded (e.g., "on 2020-01-26"). Each set contains 600 puzzles, evenly distributed over solution counts ($M = 1$ to $6$), with constraints drawn across year-, month-, and day-level fact categories (Wang et al., 12 Jan 2026).

2. Cognitive and Computational Skills Probed

TimePuzzles are explicitly designed to evaluate iterative temporal reasoning, i.e., a model’s ability to chain together constraints, ground facts to dates, and compute intersections to yield complete (possibly multi-element) answer sets. This diagnostic discriminates models that recall or retrieve facts from those that can apply calendar arithmetic, combine leap year and weekday constraints, or reconcile cross-cultural relations (e.g., Chinese lunar months). In the vision-language context (e.g., PuzzleGPT), TimePuzzles require decomposing an image into a set of clues, mapping each to candidate times/locations, combining evidence hierarchically, and invoking retrieval or noise elimination as needed (Ayyubi et al., 24 Jan 2025).

Evaluated skills include:

• Grounding factual or symbolic clues to concrete time/date sets
• Combinatorial integration of orthogonal (and possibly noisy) constraints
• Controlled ambiguity handling (multi-solution inference)
• Iterative usage of tools such as web search or code interpreters for temporal calculation

3. Methodological Implementations

In the text-only modality, state-of-the-art TimePuzzle datasets are solved using LLMs in a zero-shot Chain-of-Thought (CoT) setup, optionally augmented with web retrieval and code interpreter tools. Metrics include Exact Match (EM) between predicted and gold date sets, F1, and Jaccard scores. The test bed spans 13 LLMs, both proprietary (GPT-5, GPT-4.1 variants) and open-weight (e.g., DeepSeek-V3.2, Qwen3-series).

In the multimodal setting, PuzzleGPT exemplifies a modular, interpretable pipeline:

• Perceiver: Extracts candidate entities (landmarks, OCR, people, etc.) from images using a frozen VLM.
• Reasoner: Maps each clue to candidate time or location spans using a frozen LLM.
• Combiner: Implements hierarchical voting across clue subsets, halting when confident candidates reach a threshold.
• Web Retriever: Retrieves external evidence when local clues are insufficient, incorporating results via CLIP similarity and re-query to the Reasoner.
• Noise Filter: Uses the VLM to filter candidates inconsistent with the image content (Ayyubi et al., 24 Jan 2025).

Performance is assessed on datasets (TARA, WikiTilo) that require pinpointing when/where an image was taken, using accuracy, F1, and custom brevity-aware metrics.

4. Empirical Findings and Benchmarking

LLMs, even at the largest scale, display clear limitations under TimePuzzle evaluation. On implicit textual puzzles, GPT-5 achieves 49.3% EM; other models remain at or below 31.0%. Explicit fact version performance is much higher (GPT-5: 80.7% EM; best open-weight: 77.0%), indicating that factual retrieval, not reasoning, is rate-limiting (Wang et al., 12 Jan 2026). Performance declines as the solution set cardinality increases. Web retrieval improves results across models but cannot close the gap between implicit and explicit variants.

In vision-language tasks, PuzzleGPT exceeds frozen and code-driven multimodal baselines by wide margins—Location Standardized Accuracy of 22.99% versus BLIP-2’s 17.41%, and Time X-F1$^\beta$ at 43.72% versus InstructBLIP’s 33.83%. PuzzleGPT matches or beats fine-tuned CLIP classifiers in location prediction and markedly improves F1 scores. Web retrieval (+7.7 X-F1$^\beta$) and noise filtering (+3 X-F1$^\beta$) are empirically critical for robustness and precision (Ayyubi et al., 24 Jan 2025).

5. Design Considerations and Diagnostic Role

TimePuzzles are intentionally lightweight, algorithmically generated, and immune to memorization, thus suitable for continual model evaluation. Difficulty and ambiguity are controlled by varying the number/types of constraints and solution set sizes. Cross-cultural calendar facts simulate real-world event ambiguity. Both generation and validation pipelines are open-source for reproducibility. By comparing performance on implicit (retrieval-needed) versus explicit (fully grounded) variants, diagnostic separation between recall, tool-use, and systematic reasoning is achieved (Wang et al., 12 Jan 2026).

PuzzleGPT’s modular design demonstrates that decomposing perception, reasoning, combination, and retrieval/noise stages is necessary for state-of-the-art performance—end-to-end or prompt-only models either saturate at low accuracy or fail to filter noise. Ablation studies confirm substantial degradation if any module is omitted.

6. Challenges, Limitations, and Future Directions

The main observed bottleneck in LLMs is integrating multiple constraints for systematic inference; most gains from increased model scale or reasoning-tuning plateau at around 50% EM on implicit tasks. Code interpreters may improve efficiency or performance in scenarios involving explicit constraints but show erratic utility on implicit (ambiguous) cases. Agentic, iterative web search outperforms static retrieval (Wang et al., 12 Jan 2026).

In vision-language pipelines, pipeline complexity and reliance on proprietary (GPT-3.5, BLIP-2) modules are limiting factors; open-source LLMs substantially underperform (Location Std.Acc: 7.53% vs. 22.99%). Extending TimePuzzles beyond temporal and spatial inference (e.g., to cause, agency, or narrative structure), richer ambiguity (zero-solution cases), or richer temporal phenomena (interval, uncertainty) are plausible future directions. Learning a differentiable combiner and enhancing retrieval-augmented neural networks are proposed as engineering extensions (Ayyubi et al., 24 Jan 2025).

7. Applications and Impact

TimePuzzles provide a diagnostic environment for evaluating and developing tool-augmented, compositional reasoning in LLMs and vision-language agents. Their fine-grained control over temporal reasoning complexity, ambiguity, and cross-cultural knowledge is particularly relevant for real-world applications such as intelligent scheduling, timeline construction, automated assistants, and historical analysis.

The benchmarks and pipeline methodologies established by TimePuzzles and PuzzleGPT serve as templates for the design of new agentic systems that require robust, interpretable, and scalable temporal (and spatial) reasoning. Released datasets and code bases support reproducibility, ongoing benchmarking, and extension to broader domains within computational reasoning research (Wang et al., 12 Jan 2026, Ayyubi et al., 24 Jan 2025).