RiverONE: Quantum Calibration VLM

• RiverONE is a compact vision–language model designed for quantum calibration plot understanding, integrating simulated quantum computation during its construction.
• It employs targeted compression methods with Quantum-Generated Parameters (QGP) to recover fine-grained visual reasoning lost through classical model reductions.
• The system combines a specialized visual encoder with an InternVL3.5-based language backbone, enabling efficient classical deployment without runtime quantum simulation.

to=arxiv_search ӡамjson {"query":"(Ma et al., 29 Jun 2026) RiverONE Generating Knowledge-Intensive VLM by Simulated Quantum Machines", "max_results": 5} RiverONE is a compact vision–LLM (VLM) for quantum calibration plot understanding that uses simulated quantum computation during model construction to generate structured parameters for a compressed classical model (Ma et al., 29 Jun 2026). It is designed to connect subtle visual evidence in calibration plots—including oscillation contrast, envelope decay, peak displacement, spectral features, fitting residuals, and cluster separation—to domain conclusions about experiment outcomes, fit reliability, parameter extraction, and calibration diagnosis. The system combines a specialized visual encoder with an InternVL-based language backbone, and introduces Quantum-Generated Parameters (QGP) as layer-specific compensation for compression-induced information loss. At inference time, RiverONE runs entirely on classical GPUs; no quantum hardware and no runtime quantum simulation are required.

1. Problem setting and design rationale

RiverONE addresses a specific scientific multimodal task: quantum calibration plot understanding. In this setting, the relevant input is not merely an image-plus-question pair in the generic VQA sense, but a calibration artifact whose scientifically salient content is distributed across weak, spatially localized, and often multiscale cues. The target outputs include visual description, outcome classification, scientific reasoning, fit reliability assessment, parameter extraction, and calibration diagnosis.

The model is motivated by the observation that large, general-purpose VLMs are costly to deploy in edge or laboratory workflows, while straightforward classical compression can remove precisely the capacity needed for calibration reasoning (Ma et al., 29 Jun 2026). RiverONE therefore combines deployment-oriented compression with a construction-stage simulated quantum parameter generator intended to compensate for lost representational diversity. The stated objective is not quantum inference, but a classical scientific VLM whose compressed architecture retains calibration-sensitive reasoning.

This design places RiverONE in a narrow but technically consequential niche: a knowledge-intensive scientific VLM specialized for a domain where fine-grained visual evidence must be mapped to laboratory-relevant conclusions. A plausible implication is that the model is less a general multimodal assistant than a domain-calibrated reasoning system whose architecture has been shaped around the statistics of calibration imagery and the failure modes of compact multimodal models.

2. System architecture

RiverONE consists of a specialized visual pathway, a multimodal projection interface, and an InternVL3.5-based language backbone (Ma et al., 29 Jun 2026). The visual encoder is IsingViT-800M, a domain-specialized Vision Transformer trained on quantum calibration imagery. It outputs layer-wise hidden states $H$ across $L$ transformer blocks and $N$ patch tokens with hidden size $d$.

The visual pathway includes two explicit transformations intended to preserve calibration-relevant information after compression. First, pixel unshuffle $U$ reduces spatial resolution while increasing channel capacity:

$$H^{u} = U(H), \qquad H^{u} \in \mathbb{R}^{L \times N' \times (d \cdot r^2)}, \quad N' = N/r^2.$$

Second, cross-layer concatenation $C$ merges features from multiple transformer layers along the channel dimension:

$$H^{c} = C(H^{u}), \qquad H^{c} \in \mathbb{R}^{N' \times (L \cdot d \cdot r^2)}.$$

A two-layer MLP projector $P_{\psi}$ maps the concatenated visual representation to the language-model embedding space:

$$Z = P_{\psi}(H^{c}), \qquad Z \in \mathbb{R}^{N' \times d_{lm}}.$$

The language backbone $L$0 is an InternVL3.5-based 4B LLM. Visual token embeddings $L$1 are concatenated with tokenized text and fed into the LLM for autoregressive decoding:

$L$2

The multimodal fusion mechanism is early concatenation of visual and text tokens within the LLM sequence. The tokenizer stack is described as InternVL/Qwen3, and the projected visual tokens occupy the prefix or interleaved positions, allowing the LLM to attend across both modalities to generate answers.

The visual encoder is described as ViT-like and derived from SigLIP2-style representation learning. RiverONE further adapts these representations through pixel unshuffle and cross-layer concatenation to enrich calibration-relevant spatial and semantic cues. Multi-head self-attention uses shared, compressed attention blocks, while QGP produces layer-specific reparameterizations of the shared attention matrices $L$3, $L$4, and $L$5 to restore diversity removed by block sharing.

The deployed RiverONE variant has approximately 1.9 billion parameters. The visual encoder is compressed via weight multiplexing and QGP compensation; the language backbone is compressed with additive codebook quantization (AQLM), bringing the effective deployed footprint into the 1.9B range; and the projector and adapters are small relative to the vision and language backbones.

Component	Role	Stated implementation
IsingViT-800M	Domain-specialized visual encoder	ViT trained on calibration imagery
$L$6 and $L$7	Feature adaptation	Pixel unshuffle and cross-layer concatenation
$L$8	Modality alignment	Two-layer MLP projector
$L$9	Language decoding	InternVL3.5-based 4B LLM
QGP	Compression compensation	VQC-generated layer-specific tensors

The architecture therefore couples domain specialization with compression-aware compensation. This suggests that RiverONE treats visual representation quality, multimodal alignment, and compression recovery as a single design problem rather than as separable stages.

3. Simulated quantum parameter generation

The distinctive mechanism in RiverONE is its construction-stage “quantum-for-LLM” framework, implemented with simulated variational quantum circuits (VQC) (Ma et al., 29 Jun 2026). The VQC is used to generate structured parameters for a compressed VLM; all inference remains classical. The central claim is that parameterized quantum circuits induce expressive, nonlinear feature maps over high-dimensional vectors via superposition and entanglement, and that measurement produces probability distributions encoding cross-entry correlations that classical same-size adapters struggle to capture.

The formal state evolution is given as

$N$0

where $N$1 is a parameterized unitary built from alternating single-qubit rotations $N$2 and entangling CNOT layers, and where $N$3 qubits are used for a flattened $N$4 weight matrix.

Amplitude encoding loads a shared weight $N$5 into a quantum state:

$N$6

The evolved state is

$N$7

with measurement probabilities

$N$8

An optional observable-driven view is also given:

$N$9

where $d$0 may be projectors onto computational basis states or other diagonal observables, and the resulting vector is linearly mapped back to matrix form. In practice, the model is trained end-to-end rather than via a separately optimized observable layer.

The mapping from quantum outputs to classical tensors is performed by a learned readout function from $d$1 to $d$2, typically via probabilities $d$3:

$d$4

Normalization or scaling is applied to maintain numerical stability. These layer-specific matrices are then materialized as classical tensors after training.

The functional purpose of QGP is compensation for compression-induced loss. The VQC readout generates globally correlated features across the entire weight tensor, thereby reinstating layer-specific attention structure removed by MiniViT-style block sharing. This compensation is explicitly applied to shared visual transformer blocks and is described as especially important for detecting fits, peaks, envelopes, and readout clusters.

The per-block attention with QGP-compensated weights is written as

$d$5

Here, $d$6 are produced by the composition $d$7 with layer-specific circuit parameters. A plausible implication is that RiverONE uses the VQC less as a model of quantum data than as a structured nonlinear generator over compressed parameter space.

4. Training data, objectives, and optimization

RiverONE is trained with a two-stage pipeline comprising supervised fine-tuning (SFT) followed by MPO-based preference optimization (Ma et al., 29 Jun 2026). The SFT stage uses general visual QA and scientific plots, while the MPO stage uses preference data targeted at calibration-specific reasoning.

The SFT data sources are listed as follows: ChartQA with 5,000 samples; additional PlotQA samples; a filtered image-dependent subset of ScienceQA; and Qiskit Calibration Drift, constructed QA over time-series drift curves and environmental correlation heatmaps. The MPO stage uses QCalEval-derived augmentation via Qwen3.5-122B-A10B with 4,326 preference samples covering visual description, outcome classification, scientific reasoning, fit reliability, parameter extraction, and diagnosis; 500 retained high-quality ChartQA samples; and filtered science-related subsets from MMPR-Tiny and MMPR-v1.2 containing plots, curves, and fitting figures. The original benchmark images were used for augmentation, but not benchmark training labels.

Preprocessing standardizes calibration plots to the ViT input pipeline, uses InternVL/Qwen grammar for tokenization, and converts image-question pairs into instruction or preference formats for SFT and MPO.

The SFT objective is negative log-likelihood:

$d$8

The MPO objective is a composite of preference, quality, and generation terms:

$d$9

Its DPO-style preference term contrasts chosen and rejected responses relative to a frozen reference policy:

$U$0

QGP parameters are regularized by

$U$1

and the full objective is

$U$2

The optimization schedule is two-stage SFT—Phase 1 with in-context demonstrations, followed by Phase 2 in zero-shot mode to internalize calibration knowledge—and then MPO to refine preferences. During compression and fine-tuning, the VQC is executed on a classical simulator, and gradients flow through the simulator to optimize circuit parameters and the linear readout mapping. After convergence, the quantum-generated weights are materialized as classical tensors.

The paper also provides high-level reproducibility steps: clone the repository; install dependencies including PyTorch, CUDA, tokenizer and LLM backbones, and a quantum simulator; prepare datasets for SFT and MPO; train with two-stage SFT; apply visual compression and integrate QGP for end-to-end fine-tuning; quantize the language backbone with AQLM and run MPO; and finally materialize QGP-generated weights for classical deployment.

5. Empirical performance and efficiency

Evaluation is conducted on QCalEval, which benchmarks six categories: Q1 visual description, Q2 outcome classification, Q3 scientific reasoning, Q4 fit reliability, Q5 parameter extraction, and Q6 calibration diagnosis (Ma et al., 29 Jun 2026). The reported metrics are category scores and the overall average across Q1–Q6, with comparisons described as zero-shot unless noted.

The main disclosed comparison involves NVIDIA Ising Calibration 1 and two RiverONE variants. NVIDIA Ising Calibration 1, described as an approximately 35B MoE model, obtains Q1 87.8, Q2 67.1, Q3 64.7, Q4 90.5, Q5 62.5, Q6 75.3, and average 74.7. RiverONE in an ICL+Zero-Shot uncompressed 4.6B configuration obtains Q1 68.7, Q2 79.0, Q3 41.0, Q4 93.0, Q5 75.8, Q6 77.4, and average 72.5. The paper states that this is about 97% of IC1’s average $U$3 and that it outperforms IC1 on Q2, Q4, Q5, and Q6, while trailing on Q1 and Q3. For RiverONE-1.9B, the disclosed ablation mean is 64.45; category breakdown after MPO is not provided.

Model	Parameters	Reported QCalEval result
NVIDIA Ising Calibration 1	$U$4B MoE	Avg 74.7
RiverONE, uncompressed	4.6B	Avg 72.5
RiverONE-1.9B	$U$5B	Mean 64.45

The paper also reports compression ablations. ViT compression alone, implemented as MiniViT weight multiplexing, reduces the QCalEval mean from 72.5 to 64.17. Adding QGP compensation restores the mean to 72.37, nearly the uncompressed baseline. LLM compression alone with AQLM drops the mean to 42.34, highlighting sensitivity of language-side reasoning. Full compression plus QGP, corresponding to RiverONE-1.9B, recovers to 64.45. Sensitivity analyses focus on category-level impacts and state that QGP most benefits calibration-sensitive tasks—fit reliability, parameter extraction, and diagnosis—where layer-specific visual cues are crucial.

Efficiency figures are given for batch size 1 on reported hardware. NVIDIA Ising Calibration 1 uses 68.1 GB peak GPU memory and 13 s latency on an A100-class server GPU. RiverONE-4.4B uses 9.0 GB peak GPU memory and 0.8 s latency on a consumer 4060 Ti. The 1.9B model is expected to be no worse in memory and latency due to its smaller footprint, but exact numbers are not disclosed. QGP adds no runtime overhead because all quantum-generated parameters are materialized into classical tensors.

An important interpretive point concerns the paper’s summary claims. The abstract and conclusion state that the 1.9B model achieves “at least 95% of the performance of NVIDIA Ising Calibration 1,” but the detailed per-category numbers for that model are not provided, and the reported ablation mean of 64.45 is lower than the uncompressed 72.5. This is not a contradiction that can be resolved from the disclosed data alone; it is an internal reporting tension that should be read cautiously.

6. Deployment characteristics, limitations, and broader context

RiverONE is explicitly designed for classical deployment (Ma et al., 29 Jun 2026). All inference runs on GPUs without quantum hardware or runtime quantum simulation. The VQC is used only during compression and fine-tuning, after which the resulting parameters are saved as ordinary classical tensors. The compressed models are intended for single-GPU deployment; the 4.4B variant reportedly fits in about 9 GB on a 4060 Ti with about 0.8 s latency at batch size 1, and interactive QA batch sizes are described as typically 1–4.

The implementation stack includes Python, PyTorch, the CUDA toolkit, the transformer stack for InternVL/Qwen, and a quantum circuit simulator; candidate simulator families such as PennyLane, Qiskit, or tensor-network-based simulators are mentioned, but the specific package is not disclosed. The repository is listed at https://github.com/THeWakeSystems/RiverOne.

The limitations are domain-specific and architectural. Relative to NVIDIA Ising Calibration 1, the uncompressed RiverONE is weaker in general visual description (Q1) and broader scientific reasoning (Q3). Compressed variants lose additional capacity on language-side reasoning unless compensated. The model is sensitive to heavy language compression with AQLM, and the paper states that preserving calibration-specific reasoning requires careful tuning of codebook size and bases $U$6. Robustness is also scoped narrowly: RiverONE was built and evaluated for quantum calibration plots, and out-of-distribution scientific figures may require further fine-tuning.

The future directions listed in the paper are concrete. They include expanding calibration-specific training data and preferences; comparison against stronger classical generators such as low-rank adapters with sparsity or mixture-of-experts; systematic study of which layers benefit most from QGP and what qubit count and circuit depth are optimal in simulation; and application of materialized quantum-structured parameter generation to other scientific VLM domains.

In the broader context of scientific AI, RiverONE is presented as an example of how knowledge-intensive multimodal reasoning can be retained in compact models through domain pretraining, compression-aware architecture, and simulated quantum parameter generation. It builds on InternVL3.5 for language and ViT/SigLIP2-style visual encoding, but inserts quantum-structured compensation to recover fine-grained visual reasoning essential for calibration. A plausible implication is that the work is best understood not as evidence for practical quantum inference in VLMs, but as a proposal for using simulated quantum computation as a construction-stage mechanism for generating structured classical parameters in small, domain-strong scientific models.