Table2LaTeX-RL: High-Fidelity LaTeX Code Generation from Table Images via Reinforced Multimodal Language Models (2509.17589v1)

Abstract: In this work, we address the task of table image to LaTeX code generation, with the goal of automating the reconstruction of high-quality, publication-ready tables from visual inputs. A central challenge of this task lies in accurately handling complex tables -- those with large sizes, deeply nested structures, and semantically rich or irregular cell content -- where existing methods often fail. We begin with a comprehensive analysis, identifying key challenges and highlighting the limitations of current evaluation protocols. To overcome these issues, we propose a reinforced multimodal LLM (MLLM) framework, where a pre-trained MLLM is fine-tuned on a large-scale table-to-LaTeX dataset. To further improve generation quality, we introduce a dual-reward reinforcement learning strategy based on Group Relative Policy Optimization (GRPO). Unlike standard approaches that optimize purely over text outputs, our method incorporates both a structure-level reward on LaTeX code and a visual fidelity reward computed from rendered outputs, enabling direct optimization of the visual output quality. We adopt a hybrid evaluation protocol combining TEDS-Structure and CW-SSIM, and show that our method achieves state-of-the-art performance, particularly on structurally complex tables, demonstrating the effectiveness and robustness of our approach.

• The paper introduces a dual-reward reinforced multimodal LLM that generates high-fidelity LaTeX code from table images.
• It integrates visual (CW-SSIM) and structure-level (TEDS) rewards to address LaTeX ambiguity in complex table structures.
• Extensive experiments show state-of-the-art performance on structurally complex tables, emphasizing robustness and accuracy.

High-Fidelity Table Image to LaTeX Code Generation via Reinforced Multimodal LLMs

Introduction and Motivation

The task of converting table images into high-quality LaTeX code is central to document digitization, scientific publishing, and content reuse. While prior work has focused on HTML-based table recognition, these approaches lack the structural expressiveness and typographic fidelity required for complex tables prevalent in scientific documents. LaTeX, as the de facto standard for scientific typesetting, presents unique challenges due to its syntactic complexity, support for nested structures, and semantic richness. The "Table2LaTeX-RL" paper addresses these challenges by proposing a reinforced multimodal LLM (MLLM) framework that directly generates LaTeX code from table images, with a particular emphasis on handling structurally complex tables.

Task Analysis and Dataset Construction

A key insight of the work is the explicit categorization of tables into simple, medium, and complex subsets based on structural features such as the use of \multirow, \multicolumn, and the total number of cells. This stratification enables fine-grained evaluation and highlights the limitations of existing models, which often fail on large, deeply nested, or semantically rich tables. The authors construct a large-scale dataset of over 1.2 million table–LaTeX pairs by scraping arXiv LaTeX sources, followed by rigorous cleaning and complexity annotation. This dataset forms the basis for both supervised and reinforcement learning stages.

Model Architecture and Training Paradigm

The proposed framework leverages a pre-trained MLLM, which is first fine-tuned via supervised learning (SFT) on the constructed dataset. The SFT stage uses a standard next-token prediction objective, mapping table images and prompts to LaTeX code. However, the authors identify a critical limitation: LaTeX code is inherently ambiguous, with multiple syntactic forms yielding identical renderings. This leads to a mismatch between token-level training objectives and the true evaluation criteria, which are based on rendered visual fidelity and structural correctness.

To address this, the authors introduce a reinforced fine-tuning stage using a novel dual-reward Group Relative Policy Optimization (VSGRPO) scheme. This approach augments the standard text-based reward with a visual fidelity reward, computed by rendering the generated LaTeX and comparing it to the ground-truth image using a refined CW-SSIM metric. Simultaneously, a structure-level reward is computed using TEDS-Structure, which measures tree edit distance between the predicted and ground-truth table structures.

Figure 1: Overview of the VSGRPO framework, illustrating the dual-reward mechanism combining visual fidelity (CW-SSIM) and structure-level (TEDS-Structure) rewards for table image to LaTeX code generation.

Reward Design and Optimization

The dual-reward mechanism is central to the framework's success. The visual reward leverages a Haar wavelet–based CW-SSIM, specifically tuned for binary, high-contrast table images, to robustly measure rendered similarity. The structure reward uses TEDS-Structure, which normalizes the minimum tree edit distance between HTML representations of the predicted and ground-truth LaTeX. The VSGRPO objective combines these rewards within a clipped PPO-style policy optimization, with explicit KL regularization to stabilize training. The advantage function is normalized within each sampled group, and failed LaTeX compilations or HTML conversions are penalized with zero reward, directly incentivizing syntactic correctness.

Evaluation Protocol and Metric Analysis

The authors critique existing evaluation metrics, demonstrating that token-level metrics (e.g., BLEU, TEDS) are insufficient due to LaTeX's syntactic ambiguity. Visual metrics such as CW-SSIM, while robust to such ambiguities, may miss global structural errors. The hybrid evaluation protocol—combining TEDS-Structure and CW-SSIM—provides a more comprehensive assessment. The paper includes illustrative cases where visually identical renderings yield divergent TEDS scores due to innocuous code differences.

Figure 2: Example 1: LaTeX code ambiguity—different code, identical rendering.

Figure 3: Example 2: LaTeX code ambiguity—alternative bold commands, same visual output.

Experimental Results

The framework is evaluated against commercial tools (Mathpix), general-purpose VLMs (GPT-4o, Qwen2.5-VL-72B, Intern2.5-VL-78B), and expert models (Nougat). The VSGRPO-enhanced models (notably Qwen2.5-VL-3B-VSGRPO) achieve state-of-the-art results across all complexity levels, with particularly strong gains on complex tables. For example, on complex tables, Qwen2.5-VL-3B-VSGRPO achieves a CW-SSIM of 0.6145 and a TEDS-Structure of 0.9218, outperforming much larger models and commercial systems. Compile success rates are also high, indicating robust syntactic correctness.

Figure 4: Ground truth example of a complex table.

Figure 5: Prediction example of a complex table generated by VSGRPO.

Figure 6: Visualization of result comparisons between ground truth, VSGRPO, and SFT outputs, with CW-SSIM scores and error highlights.

Human evaluation further corroborates these findings, with VSGRPO outputs preferred by annotators across all table types.

Figure 7: Human evaluation interface for table image selection.

Ablation Studies

Ablation experiments demonstrate that:

• Restricting RL fine-tuning to complex tables yields the best generalization across all metrics.
• Both reward components (TEDS-Structure and CW-SSIM) are necessary; their combination is strictly superior to either alone.
• SFT initialization is essential; direct RL from a pre-trained MLLM without SFT leads to significantly degraded performance.

Limitations and Computational Considerations

The primary limitation is the computational overhead of rendering LaTeX outputs and computing visual rewards during RL. Each output must be compiled and converted to an image, which is non-trivial even with multi-threading. This bottleneck constrained the RL training set to 5,936 complex tables, despite the availability of a much larger dataset. Scaling this approach to even larger corpora or more complex tables will require further optimization of the rendering and reward computation pipeline.

Implications and Future Directions

This work demonstrates that integrating visual-in-the-loop reinforcement learning with MLLMs enables high-fidelity, publication-ready LaTeX code generation from table images, even for structurally complex cases. The dual-reward design directly aligns the training objective with the end-user requirements of visual and structural fidelity. The approach is broadly applicable to other structured document understanding tasks where output ambiguity and non-differentiable rendering are present.

Future research directions include:

• Optimizing the LaTeX rendering and reward computation pipeline for greater scalability.
• Extending the dual-reward RL paradigm to other document elements (e.g., figures, mathematical expressions).
• Investigating more fine-grained or continuous reward signals, potentially leveraging differentiable rendering proxies.
• Exploring transfer learning and domain adaptation for non-scientific or multilingual table corpora.

Conclusion

The "Table2LaTeX-RL" framework establishes a new standard for table image to LaTeX code generation by combining supervised and reinforced fine-tuning of MLLMs with a dual-reward optimization strategy. The method achieves strong empirical results, particularly on complex tables, and provides a robust evaluation protocol that addresses the limitations of prior metrics. The approach is computationally intensive but demonstrates the value of aligning model training with both structural and visual end goals, setting a foundation for future advances in high-fidelity document understanding.