Oryx Architecture Overview

Updated 29 May 2026

Oryx Architecture is a collection of neural and algorithmic designs tailored for vision-language modeling, hybrid sequence processing, and many-agent reinforcement learning.
It leverages adaptive mechanisms like dynamic spatial-temporal compression, sequence-axis token mixing, and linear scalability to efficiently process diverse inputs.
Empirical evaluations show its effectiveness with state-of-the-art benchmarks in multimodal understanding, language modeling, and offline MARL, highlighting practical, scalable impact.

Oryx architecture refers to several distinct neural and algorithmic architectures across vision-language modeling, hybrid sequence modeling, and multi-agent reinforcement learning, each sharing the Oryx designation but targeting diverse domains and problem settings. The following entry organizes and details these three canonical forms: (1) Oryx MLLM for scalable multimodal (2D/image, video, 3D) understanding (Liu et al., 2024), (2) Oryx Multi-Mixer for sequence modeling with hybrid token mixers (Li et al., 27 May 2026), and (3) Oryx for scalable many-agent offline MARL (Formanek et al., 28 May 2025).

1. Oryx MLLM: On-Demand Spatial-Temporal Multimodal Understanding

Oryx MLLM (Liu et al., 2024) introduces a unified multimodal architecture designed for efficient processing and reasoning over images, videos, and multi-view 3D scenes at arbitrary spatial and temporal scales. The key innovations are centered around accommodating native spatial resolution and scalable temporal length, with core components as follows:

OryxViT Encoder: OryxViT is a Vision Transformer variant accepting inputs of any size. Patch embedding proceeds via learnable projections without fixed resizing, yielding variable patch sequences:

$Z_0 = E(\mathrm{patchify}(x)) \in \mathbb{R}^{N \times C}$

where $x \in \mathbb{R}^{H \times W \times 3}$ , $N = \lfloor H/p \rfloor \times \lfloor W/p \rfloor$ , and $E$ is a linear mapping.

Positional Encoding: OryxViT utilizes a large learnable grid $P_{\max}$ , interpolated to arbitrary image dimensions:

$P_{H,W} = \mathrm{BilinearResize}(P_{\max};\, \lfloor H/p \rfloor, \lfloor W/p \rfloor)$

yielding fully adaptive spatial encodings.

Dynamic Compressor Module: Compression ratios from $1\times$ to $16\times$ are available. Downsampling layers $d_1, d_2, d_3$ yield spatial reductions $r_1 = 1$ , $x \in \mathbb{R}^{H \times W \times 3}$ 0, $x \in \mathbb{R}^{H \times W \times 3}$ 1:

$x \in \mathbb{R}^{H \times W \times 3}$ 2

To avoid loss of semantic detail, a localized cross-attention refines compressed features:

$x \in \mathbb{R}^{H \times W \times 3}$ 3

The final visual token stream is projected into the LLM embedding space.

Visual–Textual Fusion: After projection, visual tokens $x \in \mathbb{R}^{H \times W \times 3}$ 4 and text tokens $x \in \mathbb{R}^{H \times W \times 3}$ 5 are concatenated:

$x \in \mathbb{R}^{H \times W \times 3}$ 6

and processed by the LLM for joint spatial-temporal reasoning.

Training Objectives and Data: Pre-training uses 558K image captions; stage two incorporates both image and video QA, needle-in-the-haystack retrieval, and multi-view 3D tasks using ~1.2M multimodal samples. The model is trained only with next-token cross-entropy (no auxiliary heads).
Performance: Oryx-1.5-32B outperforms GPT-4V and other open-source models on a range of benchmarks, especially in long-context retrieval (>1,600 frames), DocVQA/OCR-heavy tasks, and spatial reasoning in 3D settings.

2. Oryx Multi-Mixer: Sequence-Axis Hybrid Mixer Models

Oryx Multi-Mixer (Li et al., 27 May 2026) introduces a novel “sequence-axis hybridization” in neural sequence models by parameter-tying and dynamic mixer selection along the token sequence itself.

Hybridization Strategy: Unlike traditional “inter-layer” (alternating layer type) or “intra-layer” (fusion within a layer) hybrids, Oryx operates at the sequence axis, allowing each token chunk to employ a different token mixing mechanism—softmax attention or a linear recurrent mixer (e.g., Mamba-2, Gated DeltaNet)—while using shared internal key/value/gate/output weights (over 90% shared).
Core Block Computation: For input chunk $x \in \mathbb{R}^{H \times W \times 3}$ $x \in R^{H \times W \times 3}$ 7:
1. Shared $x \in \mathbb{R}^{H \times W \times 3}$ 8, $x \in \mathbb{R}^{H \times W \times 3}$ 9
2. ShortSeq 1D convolution for local context
3. Gating $N = \lfloor H/p \rfloor \times \lfloor W/p \rfloor$ 0
4. If attention, $N = \lfloor H/p \rfloor \times \lfloor W/p \rfloor$ 1, then
$N = \lfloor H/p \rfloor \times \lfloor W/p \rfloor$ 2

If linear, use recurrence (Mamba-2/Gated DeltaNet):

$N = \lfloor H/p \rfloor \times \lfloor W/p \rfloor$ 3

Mamba-2 and GDN have parallelized, decay-masked forms. 5. Output

$N = \lfloor H/p \rfloor \times \lfloor W/p \rfloor$ 4
Mode and State Management: Each chunk’s mode is selected independently (random assignment in training at 1:3 attention:linear ratio). Both the attention KV cache and linear recurrent state $N = \lfloor H/p \rfloor \times \lfloor W/p \rfloor$ 5 are maintained regardless of mode, ensuring representational compatibility during dynamic switching.
Training and Evaluation: Pretraining uses 100B tokens; 2048-token sequences are split into 16 chunks. Baseline learning rates and batch sizes are scaled for stable tied-weight optimization. Oryx variants (TM/TG) outperform pure baselines on downstream language modeling and retrieval by up to +0.7 percentage-points; in mixed inference, retrieval gains up to +38.6 pp are reported over pure linear baselines using <10% attention chunks.
Computational Efficiency: Oryx achieves compute costs strictly between pure attention ( $N = \lfloor H/p \rfloor \times \lfloor W/p \rfloor$ 6) and pure linear ( $N = \lfloor H/p \rfloor \times \lfloor W/p \rfloor$ 7), with empirical FLOP count lower than softmax for long sequences ( $N = \lfloor H/p \rfloor \times \lfloor W/p \rfloor$ 8) and moderate attention fraction ( $N = \lfloor H/p \rfloor \times \lfloor W/p \rfloor$ 9).
Extensibility: The architecture is agnostic to linear mixer specifics and can in principle accommodate learned mode selection, additional non-attention mixers, or routing policies for further efficiency and adaptivity.

3. Oryx for Scalable Many-Agent Offline MARL

Oryx (Formanek et al., 28 May 2025) presents a scalable algorithmic architecture for offline cooperative multi-agent reinforcement learning (MARL), emphasizing multi-step, many-agent coordination under trajectory-level data constraints.

Model Structure:
- Encoder: Per-agent, per-timestep observation embedding via retention blocks (inherited from Sable), yielding $E$ 0 for agent $E$ 1 at time $E$ 2.
- Retention Blocks: Provide linear-time decaying memory over $E$ 3 agents and $E$ 4 timesteps:
$E$ 5

with learned decay $E$ 6 and $E$ 7. - Dual Decoder: Produces both policy logits and Q-value estimates. Policy is auto-regressive across a random permutation of agents, conditioned as:

$E$ 8
Sequential ICQ Algorithm:
- Critic Update via the implicit constraint Q-learning target:
$E$ 9

Extended in sequence across agents (SARSA-like). - Policy Update is a constrained, advantage-weighted log-likelihood minimization, sequentially over agents; Theorem 1 ensures the global optimum is achieved through these subproblems.
Optimization and Regularization:
- Chunks of $P_{\max}$ 0 steps, batch size 64, learning rate $P_{\max}$ 1.
- Temperatures $P_{\max}$ 2 and $P_{\max}$ 3 set by reward sparsity and dataset quality.
- KL divergence constraint is handled implicitly via partition functions in ICQ.
Scalability Features:
- Retention mechanism scales linearly with agents/time, avoiding quadratic attention overhead.
- Auto-regressive structure reduces an intractable $P_{\max}$ 4 joint policy to $P_{\max}$ 5 $P_{\max}$ 6 subproblems.
- Randomized agent orderings mitigate update bias and model symmetry.
Empirical Performance:
- Demonstrated up to 50 agents and $P_{\max}$ 7 timesteps on Connector, SMAC, Multi-Agent MuJoCo, and new benchmark datasets.
- State-of-the-art results on 80%+ of 65 testbeds, with robust generalization in large-scale settings.

4. Core Design Principles and Innovation Across Oryx Variants

Despite domain heterogeneity, several shared principles are prominent:

Adaptive Mechanisms: Whether spatial/temporal compression in vision (MLLM), mixer selection in sequence modeling, or policy coordination in MARL, Oryx prioritizes dynamic, context-adaptive computation and representation.
Linear Scalability: Each Oryx architecture substitutes fixed quadratic computation (attention, joint policies) with linear or chunkwise mechanisms, permitting scale to high context, agent count, or input size.
Parameter Sharing and Unification:
- Multi-Mixer Oryx shares >90% parameters between token mixers.
- OryxViT’s visual representations are unified across spatial and temporal scales.
- MARL Oryx employs a dual-head decoder with shared backbone features.

5. Benchmarking and Empirical Results

The Oryx architectures demonstrate leading results on domain-relevant benchmarks:

Oryx Variant	Key Benchmarks	Notable Results
MLLM	DocVQA, OCRBench	92.7% DocVQA, +4–5 pts vs. SigLIP, >50% needle-retrieval (1.6K frames)
Multi-Mixer	LAMBADA, ARC, QA	+0.7 pp over baselines, +38.6 pp NIAH w/ <10% attention, efficient FLOPs
Many-Agent MARL	SMAC, RWARE, MuJoCo	SOTA on 80%+ of 65 datasets; scalable to 50 agents, T=500

6. Architectural Impact and Future Directions

The Oryx approaches collectively broaden the toolkit for scalable, adaptive, and multimodal machine learning. The sequence-axis mixer hybridization in NLP suggests a new axis of architectural compositionality, while the on-demand vision token compression and MARL scaling mechanisms exemplify robust performance across task scales and contexts. Future development directions include reinforcement-learned or policy-based mixer routing in Multi-Mixer Oryx, further compression/adaptivity in multimodal settings, and new forms of coordinated MARL beyond decaying memory retention. These trajectories aim at richer, more cost-effective models tuned to real-world problem complexity and data heterogeneity.

Papers Referenced: