Papers
Topics
Authors
Recent
Search
2000 character limit reached

Scalable Visual Pretraining for Language Intelligence

Published 10 Jul 2026 in cs.CV, cs.AI, and cs.MM | (2607.09657v1)

Abstract: The rapid progress of large foundation models has been driven predominantly by pretraining on large-scale text corpora. However, many forms of knowledge are conveyed through visual representations, where figures, typeset equations, and page layouts carry rich information that cannot be faithfully or completely captured by text alone. Yet current pretraining approaches discard these visual cues by converting visually rich sources, such as documents and web pages, into plain text for learning language intelligence. This paper challenges the default assumption that LLMs must be trained on text-only representations and shows that Visual Pretraining is a scalable learner for foundation model intelligence. To this end, we conduct a systematic study of unsupervised visual pretraining paradigms that directly leverage visual documents without text extraction. Across multiple backbones and benchmarks, visual pretraining on the same underlying corpora consistently outperforms text-only pretraining, offering an efficient pathway to scalable language intelligence.

Summary

  • The paper demonstrates that unsupervised visual pretraining on document images yields up to +4.79 point improvements on reasoning benchmarks such as AIME-25.
  • The paper employs an autoregressive objective with contrastive InfoNCE loss to predict visual latents, achieving scalable training with only 25% of the token budget compared to text parsing.
  • The paper shows enhanced cross-modal alignment by reducing centroid distances and increasing cosine similarity, which leads to better integration of visual and text embeddings in multimodal tasks.

Scalable Visual Pretraining for Language Intelligence: A Technical Essay

Motivation and Problem Statement

The conventional paradigm for foundation model pretraining in NLP has relied solely on large-scale text corpora, operating under the assumption that all essential knowledge can be adequately encoded as text sequences. However, scientific and technical documents contain rich visual structures—figures, mathematical notation, tables, and layout—that carry information inaccessible or lost in text-only representations. The paper "Scalable Visual Pretraining for Language Intelligence" (2607.09657) systematically questions this assumption and advocates for unsupervised visual pretraining directly on document images, without explicit text extraction or image-text pair supervision, as a novel and scalable approach to foundation model training.

Visual Pretraining Methodology

The proposed Visual Pretraining (VP) framework augments standard continued pretraining regimes by introducing an autoregressive objective on foreground document patches. Each scientific PDF page is processed by a frozen ViT-style vision encoder, foreground masks are computed using patch variance and luminance, and non-blank visual features are projected into the LLM hidden space. The backbone is trained to predict the next visual latent in raster-scan order using a contrastive InfoNCE loss. Text and visual examples are interleaved during training, and the visual stream is kept highly compressed (20B visual tokens vs. 80B text tokens). The main formulation is decoder-free, avoiding pixel-level reconstruction and additional trainable parameters.

Empirical Results and Quantitative Analyses

Language Reasoning Performance

On matched scientific-document corpora, VP consistently outperforms text-only pretraining (TP) across state-of-the-art backbones (Qwen 3.5, Llama 3.2 Vision), demonstrating gains on reasoning benchmarks such as MMLU-Pro, GPQA, and AIME-25. For instance, GPQA Diamond improves by up to +3.22 points (from 76.24 to 79.29 on Qwen 3.5), and AIME-25 by +4.79 points (from 13.75 to 18.54 on Llama 3.2 Vision). VP achieves these results while consuming only 25% of the token budget relative to text parsing, thus establishing its superior efficiency in extracting domain-relevant knowledge from visual content.

Scaling and Efficiency

Increasing the visual-token budget produces monotonic gains normalized against TP; VP attains 1.27× improvement on MMLU-Pro, 2.02× on GPQA, and 2.88× on AIME-25 using fewer tokens. Resolution sweeps confirm that VP retains strong performance even under compressed visual input. Analysis by visual structure density further reveals that VP’s advantage is largest on pages rich in figures, equations, and tables—the loss from textualization is recovered when training on the visual modality.

Cross-Modal Alignment

VP improves the compatibility of visual and text embeddings without explicit image–text pair supervision. Metrics show centroid separation reduces from 1.665 to 0.661, paired cosine similarity increases from 0.631 to 0.907, and Linear CKA rises from 0.657 to 0.745. Mutual k-NN overlap at k=1 increases from 0.140 to 0.310, signifying that the shared representation space is better aligned, which is instrumental for multimodal transfer. VP outperforms TP and even the base checkpoints on multimodal benchmarks (MMMU-Pro, SFE, ChartQAPro, MathVista), with gains up to +5.4 on challenging visual QA tasks.

Mechanism: Structural Reasoning

Qualitative analysis of attention patterns highlights that models pretrained with VP focus on semantically relevant regions (constraints, computation steps) in the rendered visual input—paralleling evidence used in text-based reasoning. This suggests the visual objective encourages the backbone to internalize and use document-native spatial structures and symbol topologies for scientific reasoning.

Generative Decoder Variant

A generative-decoder variant equipped with MAR and VAE decoders for pixel-level reconstruction offers only marginal benefit for downstream reasoning relative to the decoder-free VP pipeline, but incurs substantial computational overhead (~1.4× training time, extra parameters, reduced throughput). This confirms that latent-level prediction is highly effective and optimal for scalability.

Implications and Future Directions

VP provides evidence that visual document structure is a vital source of knowledge for foundation models, especially for domains with dense figures and mathematical content. By modeling the native visual form, models can capture structural relations and symbolic correspondences inaccessible to text pipelines. The results support the Platonic Representation Hypothesis (Huh et al., 2024), implying that cross-modal representation learning converges toward shared abstractions.

Practically, VP enables efficient pretraining on highly compressed visual streams, suggesting that scalable foundation models can be trained primarily on visual modalities, using only lightweight text alignment. This paradigm is especially promising for multimodal task domains (scientific QA, chart parsing, math reasoning). Theoretical implications include better understanding of cross-modal representation geometry, effective objectives for multimodal alignment, and architectural considerations for integrating visual and text decoding.

Future work should explore:

  • Coordination between visual-latent decoding and text decoding, including loss scheduling and token selection strategies.
  • Parameter-efficient visual modules for joint pretraining and inference.
  • Transfer of visual pretraining paradigms to broader corpora (natural images, video) beyond scientific documents.
  • Automated corpus construction based on visual structure density for targeted pretraining.

Conclusion

"Scalable Visual Pretraining for Language Intelligence" (2607.09657) challenges the text-only pretraining paradigm and demonstrates that unsupervised, compact visual pretraining on scientific documents yields superior and scalable language intelligence, stronger multimodal alignment, and improved reasoning performance. The findings substantiate that structured visual content carries domain-relevant information lost in textualization, and can be efficiently harnessed for foundation model training. VP thus opens new avenues for training multimodal and LLMs on visually native corpora, offering both practical gains and insights into cross-modal representation learning.

Paper to Video (Beta)

No one has generated a video about this paper yet.

Sign Up to Generate All Videos Subscribe on YouTube

Whiteboard

No one has generated a whiteboard explanation for this paper yet.

Sign Up to Generate

Explain it Like I'm 14

Overview: What this paper is about

This paper asks a simple question: what if we trained AI not just on plain text, but also on how information looks on the page? Many things we learn from (like science papers, textbooks, and websites) use pictures, equations, tables, and layouts to explain ideas. When we convert these into plain text, we throw away a lot of those helpful visual clues. The authors show that training on the visual form of documents (pictures of the pages) can make LLMs smarter at reasoning—sometimes more than training on text alone from the very same documents.

Key goals and questions

The paper focuses on three easy-to-understand goals:

  • Can an AI learn better scientific reasoning by studying the actual pages (with figures, formulas, and layouts), not just the extracted text?
  • Is this “visual pretraining” efficient—can it learn more while reading fewer “tokens” (chunks of data)?
  • Can learning from visuals also improve how well the model handles both images and text together, even without being given matched image–caption pairs?

How they did it (methods, in everyday terms)

Think of a scientific paper page like a poster with many parts: text, figures, tables, and equations. Here’s the basic idea of their training approach, called Visual Pretraining (VP):

  • Step 1: Turn each PDF page into an image and cut it into small tiles (like a mosaic).
  • Step 2: Keep only the interesting tiles (foreground)—the parts with content, not blank margins.
  • Step 3: Use a fixed “image reader” (a vision model they don’t update) to convert each tile into a compact summary number list (a “visual feature”).
  • Step 4: Feed these tile features into the same kind of model that learns from text. But instead of guessing the next word, the model learns to guess the next tile’s feature. This is like predicting the next puzzle piece based on the ones you’ve already seen.
  • Step 5: Mix this visual training with normal text training. Importantly, they do not use OCR to extract text from images, and they don’t rely on image–caption pairs. They just learn from the raw page images plus regular text data.

Key terms explained:

  • Token: a small chunk of data (a word piece for text, or a tile feature for images).
  • Frozen vision model: a pre-trained image encoder that is kept fixed, like a calculator you use but don’t change.
  • Contrastive learning: the model must pick the correct “next tile feature” out of many similar candidates—a bit like finding the right key among many lookalikes.

They carefully compared two training paths using the same document collection:

  • Text Pretraining (TP): convert PDFs to text and train on that text.
  • Visual Pretraining (VP): use the page images directly and train on tile features.

Everything else (the starting model, the extra text data, and the final fine-tuning) was kept the same to make the comparison fair.

Main findings and why they matter

Here are the main results, explained simply:

  • Visual pretraining beats text-only on the same documents:
    • Across several strong models, VP did better on tough science and reasoning tests (like GPQA, MMLU-Pro, and AIME). For example, on one model, GPQA improved by around 3 points and MMLU-Pro by about 2 points. This suggests important knowledge lives in figures, equations, and layouts that pure text misses.
  • It’s more efficient:
    • VP used about 20 billion visual tokens from the PDF pages, while text-only used about 80 billion text tokens from the same PDFs—so VP used roughly 25% of the token budget and still did better. In other words, it learned more from less.
  • It helps most where visuals carry the meaning:
    • On pages full of figures, equations, tables, and careful layouts, VP’s advantage was largest. On text-heavy pages with few visuals, VP and text-only were similar. This shows the gains truly come from modeling visual structure, not from extra training tricks.
  • Better at multimodal tasks without explicit pairing:
    • Even though VP didn’t use labeled image–text pairs, it improved how well the model’s image and text representations align (they occupy a more similar “space” internally). It also boosted performance on multimodal benchmarks (like ChartQAPro and MathVista), sometimes by 4–6 points over text-only.
  • Faster and stronger fine-tuning:
    • After the initial pretraining, the VP models fine-tuned more quickly and reached better final performance, suggesting their internal understanding was more helpful for reasoning.

What this could mean (implications and impact)

  • A new way to train smarter models: This work shows that training on how information looks—not just what it says—can make AI better at scientific reasoning. Equations, diagrams, and layouts aren’t decoration; they are part of the message.
  • Do more with fewer tokens: Because visual features compress information well, future models might learn faster or cheaper by leaning more on visual streams, especially for science, math, and technical content.
  • Stronger multimodal skills, naturally: Without needing image–caption pairs, models become better at connecting images and text. That’s promising for building general-purpose AI assistants that understand documents, slides, lab notes, and textbooks.
  • Not a replacement, but a complement: Text training is still valuable. The best results came from mixing standard text training with this new visual pretraining on the very same documents.
  • Limits and next steps: These results are strongest on scientific PDFs (which are very visual). It remains to be seen how well this strategy scales to other kinds of visuals (like everyday photos or videos). Future research could refine how visual and text training are balanced and explore larger-scale training.

In short, the paper’s main message is simple: if we let AI learn from the full richness of scientific pages—the words, the shapes, and the layout—we can teach it to reason better, using fewer training tokens, and unlock stronger multimodal understanding.

View Paper Prompt  View All Prompts 

Knowledge Gaps

Knowledge gaps, limitations, and open questions

The paper makes a strong case for visual pretraining (VP) on scientific documents but leaves several concrete issues unresolved:

  • Compute-matched comparison: Are VP gains preserved under matched compute (FLOPs), wall‑clock time, and energy, rather than token counts? The study reports token budgets but not compute costs for rendering, the frozen vision tower, or overall throughput.
  • Pre-SFT effects: How much of the downstream improvement is attributable to VP vs. the subsequent SFT stage? Ablations evaluating checkpoints before SFT are missing.
  • Objective design: Would alternative visual objectives (e.g., regression on continuous latents, masked prediction, autoregressive pixel/token decoders, or hybrid CPC+MSE losses) outperform the current InfoNCE next‑latent objective?
  • Negative sampling quality: Do in-batch negatives create false negatives when similar patches from the same or semantically related pages co-occur? What sampling or debiasing strategies mitigate this?
  • Coordination with text decoding: How should visual-latent prediction be scheduled or weighted relative to text next-token prediction (e.g., dynamic mixing ratios, curriculum, or alternating phases) to maximize transfer?
  • Within-sample cross-modal learning: Can the model benefit from interleaving text and visual tokens from the same page instance (without explicit pairing labels) instead of training on separate text-only vs. image-only batches?
  • Vision tower choice and training regime: How sensitive are results to the frozen vision encoder architecture/scale, and would partial or full unfreezing of the vision tower yield larger gains or better alignment?
  • Patch ordering and layout encoding: Does the raster-scan ordering optimally preserve document semantics (e.g., multi-column reading order, footnotes, figure references)? Are graph- or layout-aware orderings superior?
  • Foreground filtering: The simple variance/luminance heuristic may drop faint but semantically important content (e.g., light grid lines, small symbols). How do learned or detector-based foreground selection methods compare?
  • Resolution and compression trade-offs: Beyond the reported sweep, what are the Pareto curves over resolution, patch size, and token budgets at fixed FLOPs, and do multi-scale features improve outcomes?
  • Element-wise attribution: Which visual elements (equations, tables, diagrams, captions, layout cues) contribute most to the gains? Targeted ablations/controlled corpora are needed to isolate causal factors.
  • Generalization beyond scientific PDFs: Does VP transfer to legal documents, patents, multilingual documents, forms, UI/screenshots, natural images, or video? The study focuses on high-density scientific PDFs only.
  • Cross-lingual robustness: How does VP handle non-Latin scripts, right-to-left and vertical layouts, and locale-specific typographic conventions?
  • Long-document modeling: Can VP model inter-page dependencies and document-level structures (e.g., cross-references, section hierarchy) rather than page-local sequences only?
  • Robustness to rendering variability: How sensitive are results to renderer choice, DPI, font embeddings, color profiles, and common PDF artifacts/noise?
  • OCR vs. non-text signals: Are gains driven by learning to implicitly read embedded text from images or by non-text visual structures (equation topology, charts)? Experiments that obfuscate text while preserving structure are needed.
  • Safety, calibration, and hallucination: Does VP alter truthfulness, calibration, or hallucination in scientific QA? No safety or reliability evaluations are reported.
  • Data governance and contamination: Were the evaluation sets deduplicated against the PDF corpus at the page/figure level? What are the licensing, privacy, and provenance controls for the rendered PDFs?
  • Scaling laws: Do VP gains grow or saturate with larger models and much larger visual corpora? The paper presents limited-scale evidence; formal scaling analyses are absent.
  • Compute efficiency at scale: Does the 25% token advantage translate into net efficiency at pretraining scales where image pipelines and memory bandwidth dominate?
  • Alternative latent spaces: Would discretized visual tokens (e.g., VQ or tokenizer-based latents) or stronger learned visual target spaces yield better predictability and transfer than the chosen frozen latent space?
  • Parameter-efficient designs: What is the best way to allocate parameters between LLM, visual projection, and prediction heads (e.g., adapters, LoRA, MoE for visual paths) for maximal return on parameters?
  • Benchmark breadth and stability: Results are reported on a limited set of reasoning and multimodal benchmarks with small alignment probes (100 pairs). More diverse, larger-scale alignment/transfer evaluations and variance analyses are needed.
  • Failure modes and boundary conditions: HLE gains are marginal; what task characteristics limit VP’s benefits, and when might VP be detrimental (e.g., text-dominant tasks, low visual-structure density)?
  • Open pretraining recipe: Full reproducibility would benefit from releasing the rendering pipeline, foreground filtering code, vision tower checkpoints, and detailed hyperparameters; currently, only high-level details are provided.
View Paper Prompt  View All Prompts 

Practical Applications

Practical Applications of “Scalable Visual Pretraining for Language Intelligence”

Below are concrete applications derived from the paper’s findings and method (visual pretraining on raw document pages with next-visual-latent prediction), grouped by deployment horizon. Each item includes sector tags, potential tools/workflows, and key assumptions/dependencies that affect feasibility.

Immediate Applications

These can be prototyped and deployed now using the paper’s VP recipe on document-heavy corpora (e.g., PDFs of reports, scientific papers, manuals) with a frozen vision tower and sparse foreground visual tokens.

  • Layout-aware Retrieval-Augmented Generation (RAG) for PDFs
    • Sectors: software, finance, legal, healthcare, academia, government
    • Tools/Workflow: “VP ingestion layer” that renders pages to images, filters foreground patches, encodes visual latents, and stores pooled embeddings alongside text; hybrid retrieval over text and visual-latent indices; answer synthesis grounded in figure/Equation/table regions
    • Assumptions/Dependencies: High-quality PDF rendering; content is layout/figure/table dense; vector DB support for multimodal embeddings; data governance for sensitive documents
  • Chart/table question answering for BI and analytics
    • Sectors: finance, energy, operations, sales/marketing analytics
    • Tools/Workflow: VP-enhanced BI copilot that directly answers questions over embedded charts/tables in reports (e.g., “What is Q3 YoY growth in Figure 1?”); integrates with dashboard exports and PDF pack reports
    • Assumptions/Dependencies: Access to report images or page renders; permissioning; evaluation/guardrails for numerical faithfulness
  • Scientific paper assistants with figure/equation reasoning
    • Sectors: academia, pharma/biotech R&D, materials science, semiconductors
    • Tools/Workflow: VP-pretrained research copilots that summarize, extract equations, link figures to text claims, and answer GPQA/AIME-style queries; “equation-aware” citation graph builders
    • Assumptions/Dependencies: Licensing for model weights and publisher content; robust equation/figure extraction; domain adaptation for subfields
  • Regulatory filing and contract analysis with layout fidelity
    • Sectors: finance (10-K/10-Q), insurance, legal/compliance, procurement
    • Tools/Workflow: OCR-free analysis of filings and contracts preserving schedules, cross-references, exhibits; clause/risk extraction using visual tokens from tables and footnotes; “Layout-aware redlining copilot”
    • Assumptions/Dependencies: Document heterogeneity handling; compliance/auditability; long-context management for large exhibits
  • Healthcare administrative document processing (not clinical imaging)
    • Sectors: healthcare, insurance
    • Tools/Workflow: Claims/EOB/forms parsing and QA without brittle OCR; table-driven benefit explanations; “prior-authorization assistant” that reads multi-page forms with checkboxes/tables
    • Assumptions/Dependencies: PHI privacy; domain SFT; forms variability and scan quality
  • Education: math and physics tutoring from scanned/typed problems
    • Sectors: education, edtech
    • Tools/Workflow: Tutors that accept worksheet photos or PDF problem sets with diagrams/equations and produce step-by-step solutions; “diagram-aware hint generation”
    • Assumptions/Dependencies: Camera/PDF quality; grading alignment; safeguards against leaking answers during assessments
  • OCR-free enterprise archives search and e-discovery triage
    • Sectors: legal, government, enterprise knowledge management
    • Tools/Workflow: VP-based indexing of scanned archives for semantic and layout-aware retrieval; fast triage of exhibits where tables/figures carry the evidence
    • Assumptions/Dependencies: At-scale page rendering and foreground filtering; deduplication; chain-of-custody and audit trails
  • Developer documentation copilots that understand diagrams and spec sheets
    • Sectors: software, hardware, networking
    • Tools/Workflow: Assistants that reason over API diagrams, timing diagrams, protocol tables, and configuration matrices in PDFs
    • Assumptions/Dependencies: Mixed content (code + diagrams); domain SFT; control of hallucinations on specs
  • Productizing chart- and math-heavy multimodal benchmarks
    • Sectors: AI product teams
    • Tools/Workflow: Integrate VP CPT to improve ChartQAPro/MathVista-like tasks in deployed MLLMs; benchmark-driven fine-tuning and evaluation suites
    • Assumptions/Dependencies: Careful eval to prevent metric gaming; licensing for Qwen/Llama derivatives
  • More compute- and data-efficient model training on document corpora
    • Sectors: AI infra, foundation model labs
    • Tools/Workflow: Replace a portion of text CPT with VP on the same document source to achieve similar or better downstream reasoning at ~25% of token budget; training schedulers that mix ℒ_CE and ℒ_VP
    • Assumptions/Dependencies: Frozen vision tower availability; training stack that supports sparse visual sequences and InfoNCE; hyperparameter tuning for token budgets
  • Cross-modal embedding alignment for DAM and KMS
    • Sectors: digital asset management, enterprise search
    • Tools/Workflow: Use VP-improved alignment to interlink images/figures and text without paired supervision; “figure-to-caption” and “caption-to-figure” retrieval in document repositories
    • Assumptions/Dependencies: Embedding store that handles both modalities; quality control for mismatches

Long-Term Applications

These require additional research, scaling, or generalization beyond scientific PDFs, but are enabled by the paper’s paradigm (visual-latent prediction and sparse foreground training).

  • Visual-first pretraining at scale as a new foundation model regime
    • Sectors: AI research/infra
    • Tools/Workflow: Train general-purpose LLMs with large visual-corpora streams (documents, slides, UI screenshots) and light text alignment to reach or surpass text-only performance at lower token cost
    • Assumptions/Dependencies: Demonstrated transfer beyond scientific PDFs; scalable rendering/ingestion pipelines; stability of training objectives at trillion-token scale
  • Agents for UI understanding and robotic process automation (RPA)
    • Sectors: enterprise software, fintech ops, back-office automation
    • Tools/Workflow: VP-augmented agents that read and act on software UIs, forms, and dashboards using layout-aware reasoning instead of brittle OCR/XPath rules
    • Assumptions/Dependencies: Extension of VP to screenshots and semi-structured GUIs; latency constraints; safe action policies
  • STEM auto-grading and feedback on diagrammatic work
    • Sectors: education
    • Tools/Workflow: Grade and give feedback on student solutions involving graphs, free-body diagrams, geometric constructions, and multi-step derivations captured in scans
    • Assumptions/Dependencies: Robustness to handwriting/sketches; bias/fairness audits; institution approvals
  • Accessibility: faithful narration of complex documents
    • Sectors: public sector, assistive tech, publishing
    • Tools/Workflow: Screen-reader copilots that accurately describe equations, charts, and layouts; “figure-first” navigation and Q&A over accessible PDFs
    • Assumptions/Dependencies: Standards integration (PDF/UA, MathML); guaranteed factual grounding to avoid misleading descriptions
  • Legal discovery and evidence synthesis across heterogeneous exhibits
    • Sectors: legal
    • Tools/Workflow: End-to-end ingestion of scanned exhibits (forms, schematics, tables, photos embedded in filings), cross-referenced with briefs via VP-aligned embeddings
    • Assumptions/Dependencies: Chain-of-custody and explainability; handling of low-quality scans; court acceptance of AI-assisted discovery
  • Scientific knowledge base construction from figures, tables, and equations
    • Sectors: pharma/biotech, materials, climate science
    • Tools/Workflow: Extract structured data and relationships from visual elements (plots, dose–response curves, crystal structures, circuit diagrams) to populate KBs and simulation constraints
    • Assumptions/Dependencies: Domain-specific parsers layered on VP; validation pipelines; licensing of source literature
  • Engineering and energy asset intelligence from technical diagrams
    • Sectors: energy, manufacturing, aerospace
    • Tools/Workflow: Interpret P&IDs, line diagrams, wiring schematics, and performance charts for maintenance planning and anomaly triage
    • Assumptions/Dependencies: Generalization from PDFs to CAD-like drawings; safety-critical verification; secure on-prem deployment
  • Safety and interpretability tooling via cross-modal attention analyses
    • Sectors: AI assurance, regulated industries
    • Tools/Workflow: Use VP’s improved alignment to audit what evidence the model attends to (figures/tables/clauses) when producing answers; “evidence heatmaps” in compliance workflows
    • Assumptions/Dependencies: Reliable attention/attribution methods; human-in-the-loop review; governance policies
  • Low-resource and multilingual document understanding
    • Sectors: public sector, NGOs, international development
    • Tools/Workflow: Leverage layout/visual structure to bootstrap understanding of documents where OCR/text resources are scarce; cross-lingual transfer from visual patterns
    • Assumptions/Dependencies: Adaptation to scripts and local layouts; minimal text alignment for grounding; ethical data sourcing
  • Edge-capable document agents using compressed visual tokens
    • Sectors: mobile, field operations
    • Tools/Workflow: On-device assistants that process forms/reports with sparse foreground tokens for low compute and bandwidth
    • Assumptions/Dependencies: Efficient vision towers; hardware acceleration; privacy-by-design
  • Policy analytics over public records with charts and appendices
    • Sectors: policy, civic tech
    • Tools/Workflow: VP-powered analysis of budgets, census reports, environmental impact statements where evidence sits in tables/figures; “appendix-aware” summarization for policymakers
    • Assumptions/Dependencies: Open data availability; transparency requirements; careful evaluation to prevent misinterpretation

Cross-cutting Assumptions and Dependencies

  • Domain fit: Biggest gains occur on visually dense, structured pages (figures, equations, tables, layouts); benefits may be marginal on text-dominant or natural-image content without further research.
  • Data pipelines: Reliable page rendering, foreground patch filtering, and storage of visual latents are required at scale.
  • Models and licensing: Commercial use depends on licensing of base models (e.g., Qwen, Llama) and third-party content.
  • Privacy/compliance: Many corpora (healthcare, legal, finance) demand strict data governance, auditability, and on-prem options.
  • Evaluation: Numeric faithfulness and hallucination control are critical, especially for chart/table QA and compliance tasks.
  • Compute/latency: Despite sparse tokens, visual streams add cost; batching, caching, and resolution/budget tuning are needed for production SLAs.
View Paper Prompt  View All Prompts 

Glossary

  • autoregressive cross-entropy loss: A training objective that predicts the next token in a sequence and measures error with cross-entropy; standard in language modeling. "autoregressive cross-entropy loss on text tokens"
  • autoregressive visual pretraining: A pretraining approach where the model predicts the next item in a sequence of visual features, analogous to text autoregression. "an autoregressive visual pretraining framework that trains a foundation model to predict document patches in latent space"
  • causal image-only masking: An attention masking scheme that allows each visual token to attend only to prior visual tokens, enforcing autoregressive order in images. "under causal image-only masking"
  • causal objective: A next-step prediction objective that only conditions on past context, not future tokens or patches. "using a causal objective that predicts the next visual latent"
  • centroid separation: A metric measuring distance between modality centroids (e.g., image vs. text embeddings) to assess alignment. "the centroid separation drops from 1.665 to 0.661"
  • contrastive alignment: Training signals that bring matched multimodal pairs closer and push mismatches apart in embedding space. "contrastive alignment"
  • contrastive loss: A loss that encourages similarity of positive pairs and dissimilarity of negatives, often via in-batch negatives. "using a contrastive loss with in-batch negatives"
  • CoT prompting: Chain-of-thought prompting; instructing the model to show intermediate reasoning steps before answers. "use CoT prompting with the instruction “think step by step”"
  • cosine similarity: A similarity measure between vectors based on the cosine of the angle between them, used to assess embedding alignment. "paired cosine similarity rises from 0.631 to 0.907"
  • continued pretraining (CPT): Additional pretraining on new corpora after an initial base pretraining to further adapt the model. "continued pretraining (CPT) stage"
  • cross-modal alignment: The degree to which representations from different modalities (e.g., text and images) occupy a shared, comparable space. "VP improves cross-modal alignment and multimodal transfer"
  • foreground mask: A filter that removes blank/background patches and retains only content-rich regions in page images. "we therefore compute a foreground mask"
  • foreground visual tokens: The retained, non-blank visual patches from documents used as the model’s visual input sequence. "8,192 foreground visual tokens per batch"
  • frozen vision tower: A fixed (non-trainable) visual encoder that produces stable features for downstream prediction. "a frozen vision tower"
  • image–text pair supervision: Training signals derived from paired images and texts (e.g., captions), used to align modalities. "without any image–text pair supervision"
  • in-batch negatives: Other samples within the same batch treated as negative examples for contrastive learning. "in-batch negatives"
  • InfoNCE: A contrastive learning objective (Info Noise-Contrastive Estimation) that maximizes mutual information via positive/negative pairs. "visual InfoNCE training"
  • Linear CKA: Linear Centered Kernel Alignment; a similarity metric for comparing representational structures between networks or modalities. "Linear CKA"
  • mutual k-NN overlap: The proportion of mutual nearest neighbors shared between two representation spaces, indicating local alignment. "Mutual k-NN overlap"
  • next visual latent prediction: Predicting the next embedding (latent) of a visual patch in sequence, analogous to next-token prediction. "introducing next visual latent prediction on raw scientific document pages"
  • OCR-free: Methods that operate directly on images of documents without Optical Character Recognition text extraction. "OCR-free document models"
  • pass@1: An evaluation metric indicating success rate when only the first model attempt is considered. "pass@1 for MMLU-Pro"
  • Platonic Representation Hypothesis: The idea that learned representations across different models/modalities converge toward shared abstractions. "The recently proposed Platonic Representation Hypothesis [10]"
  • positional encodings: Numerical embeddings that encode token positions to preserve order and spatial structure. "Positional encodings are preserved to maintain spatial layout"
  • projection head: A learned layer mapping hidden states to a target feature space for prediction. "an output projection head"
  • raster-scan order: A sequence order that follows row-by-row scanning of image patches to preserve page layout. "ordered in raster-scan order"
  • representation space: The embedding space where inputs (e.g., images, text) are mapped for downstream tasks and comparisons. "shared representation space"
  • sequence packing: The procedure of packing multiple shorter sequences into a fixed-length context while tracking boundaries. "the sequence-packing procedure used in training"
  • t-SNE: t-distributed Stochastic Neighbor Embedding; a technique for visualizing high-dimensional data in 2D/3D. "t-SNE [31] projections"
  • token budget: The total number of tokens allocated for training, constraining compute and data usage. "using only 25% of the token budget"
  • ViT encoder: Vision Transformer encoder; a transformer-based model that processes images as patches. "a frozen ViT encoder"
  • visual-feature space: The embedding space produced by the visual encoder in which visual latents live. "frozen visual-feature space"
  • visual-structure density: A categorization of pages by the amount of visual structure (figures, equations, tables, layout) they contain. "visual-structure density"
  • visual projection module: A learned mapping that projects visual features into the LLM’s hidden space. "via a visual projection module"
  • zero-shot setting: Evaluation without task-specific training, relying solely on pretraining and generalization. "we evaluate VP and the baselines in a zero-shot setting"
View Paper Prompt  View All Prompts 

Open Problems

We haven't generated a list of open problems mentioned in this paper yet.

Generate Now

Collections

Sign up for free to add this paper to one or more collections.

Sign Up

Tweets

Sign up for free to view the 4 tweets with 79 likes about this paper.

Sign Up for Free