---
title: 'NVIDIA Nemotron Parse 1.1: Advanced VLT OCR'
url: https://www.emergentmind.com/topics/nvidia-nemotron-parse-1-1
type: topic
---

# NVIDIA Nemotron Parse 1.1: Advanced VLT OCR

NVIDIA Nemotron Parse 1.1 is a lightweight end-to-end Vision-Language Transformer (VLT) model tailored for advanced document parsing and OCR. As a successor to Nemotron-Parse 1.0 (Eclair), version 1.1 expands the capabilities to structured extraction of formatted text, tables (in LaTeX), and multi-modal text recovery from embedded charts, diagrams, and pictures, while maintaining high throughput suitable for both research and large-scale deployment. Nemotron-Parse 1.1 operates with an encoder-decoder architecture comprising 885M parameters, rapidly processing complex documents while supporting longer output sequence lengths and providing bounding boxes and semantic class labels for extracted text [2511.20478].

## 1. System Architecture and Parameterization

Nemotron-Parse 1.1 is structured as a classical encoder–decoder Transformer:

- **Vision Encoder:** RADIO v2.5 (ViT-H/16 backbone, 657M parameters) encodes the input image $\mathbf{I} \in \mathbb{R}^{3 \times H \times W}$ to dense features $\mathbf{Z} = \mathcal{E}(\mathbf{I}) \in \mathbb{R}^{N \times d}$. 
- **Convolutional Neck:** A horizontal $1 \times 4$ kernel compresses sequence length to 3,200 tokens (for a 1,648 × 2,048 page), with one summary token added (final ≈3,201 encoder tokens).
- **Language Decoder:** A 10-layer mBART (256M parameters, tied weights) predicts output sequence tokens, omitting positional embeddings to enable support for output sequences well beyond 1,000 tokens.
- **No Positional Embeddings in Decoder:** The omission of decoder positional embeddings avoids interpolation artifacts for long sequences by relying solely on causal masking.

**Parameter breakdown:**

| Module        | Parameter Count | Remarks             |
|---------------|----------------|---------------------|
| Vision Encoder| 657M           | RADIO v2.5 (ViT-H/16)|
| Neck          | Negligible     | Convolutional, non-transformer|
| Decoder       | 256M           | 10-layer mBART, tied weights|
| **Total**     | 885,766,720    |                    |

A variant, Nemotron-Parse-1.1-TC ("token-compressed," *Editor's term*), further pixel-shuffles the neck output, reducing vision tokens from 3,200 to 833 (by $4\times$ compression in each spatial dimension).

## 2. Innovations Over Nemotron-Parse 1.0

Nemotron-Parse 1.1 introduces several enhancements relative to version 1.0:

- **Data Expansion:** Leveraged expanded multilingual dense-text data sources (NVpdftex synthetic, Common Crawl, Wikipedia OCR) for broader OCR training.
- **Structured Formatting:** Advanced parsing generates Markdown and LaTeX, with tables reproduced in LaTeX and inline math handled either via LaTeX or super/sub-script encoding.
- **Improved Table Parsing:** End-to-end generation of LaTeX-based table structures, achieving strong TEDS (Tree-Edit Distance-based Similarity) and S-TEDS metrics.
- **Multi-modal Extraction:** Extended training with VQA-style crops and datasets (DocLayNet) to support text extraction from images, charts, and diagrams, not just scanned text.
- **Extended Output Range:** Elimination of decoder positional embeddings enables coherent sequence generation for outputs far exceeding 1,000 tokens.
- **Inference Speed-up:** Multi-token prediction heads trained for parallel next-token generation, achieving inference acceleration by decoding in groups ($\leq 3\times$ reduction in per-token latency).

## 3. Training Methodologies and Objective Functions

Nemotron-Parse 1.1 is trained end-to-end with a composite loss integrating both text generation and bounding box regression:

- **Cross-Entropy Loss for Text Generation:**
  $$
  \mathcal{L}_{\mathrm{CE}} = -\sum_{i=P+1}^L \log P(t_i^* | \mathcal{N}(\mathbf{Z}), t_{<i})
  $$
  where $\{t_1, ..., t_P\}$ are prompt tokens controlling output markup/class prediction.

- **Bounding Box Regression:**
  $$
  \mathcal{L}_{\mathrm{bbox}} = \frac{1}{B} \sum_{b=1}^B \| \hat{\mathbf{b}}_b - \mathbf{b}_b^* \|^2
  $$
  with $(x_{\min}, y_{\min}, x_{\max}, y_{\max})$ as normalized coordinates.

- **Total Loss Function:**
  $$
  \mathcal{L} = \mathcal{L}_{\mathrm{CE}} + \lambda_{\mathrm{bbox}} \mathcal{L}_{\mathrm{bbox}}
  $$

- **Multi-token Prediction:** Auxiliary heads $l_1, l_2$ are applied to combine the last ground-truth and previously predicted hidden state embeddings, enabling group prediction during both training and inference.

## 4. Benchmark Performance and Comparative Results

Nemotron-Parse 1.1 demonstrates competitive or state-of-the-art performance across several public and internal benchmarks. Key comparative benchmark results include:

| Model-Method                | Mask Out | WER ↓  | F1 ↑  |
|-----------------------------|----------|--------|-------|
| Kosmos-2.5 (ocr-mode)       | +        | 0.195  | 0.937 |
| Kosmos-2.5 (md-mode)        | +        | 0.249  | 0.890 |
| GOT (ocr-mode)              | +        | 0.302  | 0.818 |
| Nemotron-Parse-MIP          | -        | 0.109  | 0.958 |
| Nemotron-Parse-MIP          | +        | 0.102  | 0.957 |
| Nemotron-Parse-TC-MIP       | -        | 0.111  | 0.953 |
| Nemotron-Parse-TC-MIP       | +        | 0.121  | 0.949 |

**On the GOT benchmark:**

| Extractor              | OCR/F1 Score ↑ | Text-Only RO/Edit Dist. ↓ | Text-Only RO/METEOR ↑ | Text-Only RO/BLEU ↑ |
|----------------------- |---------------|--------------------------|----------------------|--------------------|
| Pdfium                 | 0.0036        | 0.9993                   | 0.0083               | 0.0000             |
| Docling                | 0.6744        | 0.4300                   | 0.6331               | 0.4651             |
| Nemotron-Parse-1.1     | 0.9785        | 0.014                    | 0.9858               | 0.9623             |
| Nemotron-Parse-1.1-TC  | 0.9755        | 0.014                    | 0.9838               | 0.9582             |

On OmniDocBench 1.0, Nemotron-Parse 1.1 exhibits table, formula, and reading-order scores that outperform all other end-to-end models under a 4,000 vision-token constraint. PubTabNet and RD-TableBench benchmarks show TEDS > 80 and table similarity ≈ 86.

## 5. Token-Compressed Variant (Nemotron-Parse-1.1-TC) and Throughput

The token-compressed (TC) variant utilizes a pixel-shuffle operation to decrease vision encoder output to 833 tokens. This modification yields a 20% throughput gain (from ~3,800 tokens/sec to 4,500 tokens/sec or ~4–5 pages/sec at 1,000 tokens/page) with negligible impact on main accuracy metrics (F1 drop ≤0.005 on GOT; ≤0.003 on OmniDocBench). Notably, reading-order consistency slightly improves due to integrated float-element ordering.

## 6. Released Artifacts, Integrations, and Deployment

NVIDIA provides both model weights (fp32/bf16) and an optimized NIM inference container:

- **Model Weights (HuggingFace):**
  - Nemotron-Parse-v1.1: https://huggingface.co/nvidia/NVIDIA-Nemotron-Parse-v1.1
  - Nemotron-Parse-v1.1-TC: https://huggingface.co/nvidia/NVIDIA-Nemotron-Parse-v1.1-TC

- **NIM Container:** Optimized for production scaling—https://build.nvidia.com/nvidia/nemotron-parse

- **Datasets:** The Nemotron-VLM-Dataset-v2, including NVpdftex synthetic, Common Crawl, DocLayNet, and PubTabNet.

Deployment scenarios encompass batch scientific paper processing, interactive document understanding, and edge/cloud-edge orchestration due to the favorable tradeoff between resource usage and throughput in the TC variant. For in-domain adaptation, continued supervised fine-tuning with customized prompt tokens (JSON, HTML, semantic classes) is supported. Knowledge distillation enables further parameter budget reduction for resource-constrained use cases.

Integration with LLM back-ends enables complex downstream tasks such as semantic question answering or table querying, promoting document understanding workflows from pixels to structured and semantically annotated textual representations [2511.20478].

Source: https://www.emergentmind.com/topics/nvidia-nemotron-parse-1-1