RynnBrain: Unified Spatiotemporal Model

• RynnBrain is a unified spatiotemporal foundation model for embodied intelligence that combines egocentric understanding with explicit spatial grounding.
• It interleaves textual and spatial tokens on a normalized [0,1000] grid to enable comprehensive reasoning, localization, and physics-aware planning.
• The model family includes various scales and post-trained variants tailored for navigation, manipulation planning, and vision-language-action execution.

Searching arXiv for the cited papers and topic coverage. RynnBrain is an open-source spatiotemporal foundation model family for embodied intelligence that integrates perception, reasoning, and planning within real-world spatial-temporal dynamics (Dang et al., 13 Feb 2026). In the supplied literature, it is defined by four unified capabilities—comprehensive egocentric understanding, diverse spatiotemporal localization, physically grounded reasoning, and physics-aware planning—and by a physically grounded output space in which points, boxes, and trajectories are emitted as discrete coordinate tokens on a unified $[0,1000]$ grid (Dang et al., 13 Feb 2026). The same source positions RynnBrain as a general-purpose embodied “brain” that can be post-trained for navigation, manipulation planning, vision-language-action execution, and complex spatial reasoning, while a separate synthesis uses the label for an RNN-based RSVP P300 BCI pipeline derived from Tal and Friedman’s study of recurrent neural networks for P300 detection (Tal et al., 2019).

1. Conceptual scope and defining properties

RynnBrain was introduced to address a gap between broadly capable vision-LLMs and action-centric embodied systems (Dang et al., 13 Feb 2026). Prior VLMs are described as not physically grounded: they may hallucinate spatial details, lack temporal consistency, and operate purely in text, which impedes actionable plans. Conversely, action-centric embodied models are described as sacrificing high-level semantics and generalization. RynnBrain is presented as a unified, physically grounded spatiotemporal foundation model intended to reconcile these two regimes by integrating perception, reasoning, and planning while remaining adaptable to downstream control (Dang et al., 13 Feb 2026).

Its central design commitment is a shared output space in which textual generation is interleaved with explicit spatial grounding. Spatial predictions are represented as discrete coordinate tokens in a normalized $[0,1000]$ grid for points, boxes, and trajectories, so that reasoning and planning can be expressed in the same autoregressive stream as language (Dang et al., 13 Feb 2026). This physically grounded output space is the principal distinction repeatedly emphasized in the source material. The paper also frames RynnBrain as open infrastructure: code, checkpoints, and datasets are released under the Apache 2.0 license (Dang et al., 13 Feb 2026).

The supplied literature associates the name with two different technical contexts. In its primary and explicit sense, RynnBrain is the embodied foundation model family introduced in 2026 (Dang et al., 13 Feb 2026). In a secondary editorial synthesis, the label is also applied to a practical RSVP P300 BCI configuration centered on CNN, LSTM, and CNN+RNN modeling choices derived from Tal and Friedman’s “Recurrent Neural Networks for P300-based BCI” (Tal et al., 2019). This suggests that the term has been used both as a proper model family name and as a broader organizing label for physically grounded or temporally grounded neural systems.

2. Architecture, model family, and output representation

The RynnBrain family comprises three foundation model scales—RynnBrain-2B, RynnBrain-8B, and RynnBrain-30B-A3B MoE—and four post-trained variants tailored for downstream embodied tasks or complex spatial reasoning (Dang et al., 13 Feb 2026). RynnBrain-2B and RynnBrain-8B are dense decoder-only vision-LLMs initialized from Qwen3-VL-2B/8B-Instruct, while RynnBrain-30B-A3B is a mixture-of-experts variant initialized from Qwen3-VL-30B-A3B-Instruct (Dang et al., 13 Feb 2026). For the MoE model, the active parameter count is 3B per token, whereas the total parameter count is 30B; expert parallelism with world size 2, grouped linear kernels, and DeepEP are used for training efficiency (Dang et al., 13 Feb 2026).

The backbone is a decoder-only vision-language architecture comprising a vision encoder, a vision-language projector, and an LLM, with DeepStack used to enhance visual token depth and Interleaved MRoPE used to improve positional encoding across modalities and time (Dang et al., 13 Feb 2026). Inputs include images and videos, with videos represented as sequences of frames $V=\{I_t\}_{t=1}^T$ and temporal sampling at 2 FPS for long-horizon tasks. Text instructions and interleaved histories are handled in conversational format (Dang et al., 13 Feb 2026).

A concise summary of the model family and major variants is given below.

Model	Type	Primary target
RynnBrain-2B	Dense decoder-only VLM	General embodied foundation model
RynnBrain-8B	Dense decoder-only VLM	General embodied foundation model
RynnBrain-30B-A3B	MoE decoder-only VLM	Higher-capacity embodied foundation model
RynnBrain-CoP	Post-trained variant	Physically grounded reasoning
RynnBrain-Nav	Post-trained variant	Vision-language navigation
RynnBrain-Plan	Post-trained variant	Manipulation planning
RynnBrain-VLA	Post-trained variant	Vision-language-action execution

The output representation is structurally important because localization, reasoning, and planning all share the same tokenization scheme. The inference format includes tags such as <object>, <area>, <affordance>, and <trajectory>, each paired with a <frame n> index and coordinate tuples (Dang et al., 13 Feb 2026). The model does not maintain an explicit 3D map or occupancy grid at inference, and it does not integrate SLAM or explicit $SE(3)$ pose estimation at runtime; instead, its grounding is image-plane and episode-indexed (Dang et al., 13 Feb 2026). Offline 3D reconstructions via MASt3R-SLAM are used for label generation in some datasets but are not part of deployed inference (Dang et al., 13 Feb 2026).

3. Training regime, data mixture, and post-training

RynnBrain is pretrained on approximately 19.89M samples spanning general multimodal instruction data and embodied cognition, localization, and planning data (Dang et al., 13 Feb 2026). The mixture includes 4.8M general MLLM samples; cognition subsets for object understanding, spatial understanding, counting, OCR, and egocentric task understanding; localization subsets for object localization, area localization, affordance localization, trajectory prediction, and grasp pose; and 0.16M planning samples (Dang et al., 13 Feb 2026). This breadth is intended to support long-horizon video understanding, physically grounded prediction, and downstream adaptation within one framework.

The unified training objective is autoregressive next-token prediction over mixed text and discretized coordinate tokens:

$$L = -\sum_{i=1}^L \log P(y_i \mid y_{<i}, V, \Theta).$$

For efficient distributed training under long-tail sequence lengths, the paper also specifies a per-sample loss reduction scheme over distributed workers (Dang et al., 13 Feb 2026). Training infrastructure includes ZeRO-1 for 2B and 8B, ZeRO-2 plus expert parallelism for 30B-A3B, gradient checkpointing, loss-token filtering, DeepEP for token dispatch, grouped linear MoE kernels, and the HuggingFace Transformers training framework (Dang et al., 13 Feb 2026).

Pretraining hyperparameters are stated explicitly. The 2B model uses AdamW with learning rate $5\times10^{-6}$, vision learning rate $1\times10^{-6}$, batch size 512, and 3% warmup. The 8B model uses AdamW with learning rate $2\times10^{-6}$, vision learning rate $2\times10^{-6}$, batch size 1024, and 3% warmup. The 30B-A3B model uses the same learning rate and vision learning rate as 8B, with batch size 1024 and 3% warmup (Dang et al., 13 Feb 2026).

Post-training differentiates the variants. RynnBrain-CoP performs full-parameter SFT on Chain-of-Point-augmented data and then reinforcement learning with GRPO using spatially grounded rewards (Dang et al., 13 Feb 2026). RynnBrain-Nav is fully fine-tuned on VLN-CE datasets reformatted as interleaved image-action sequences and further improved with multi-turn DAgger data collection (Dang et al., 13 Feb 2026). RynnBrain-Plan is fine-tuned on small, multi-turn dialogues that function as explicit memory buffers for long-horizon tasks, while RynnBrain-VLA adds a flow-matching action head and is fine-tuned on teleoperated Franka pick-and-place episodes (Dang et al., 13 Feb 2026).

4. Spatiotemporal localization and Chain-of-Point reasoning

RynnBrain’s localization mechanism predicts temporal key frames and spatial coordinates jointly in a shared token space (Dang et al., 13 Feb 2026). Object localization is emitted as bounding boxes; area localization as polygons or point sets; affordance localization as points; and trajectory localization as ordered coordinate sequences attached to frame indices. The corresponding evaluation metrics are also explicitly defined: object grounding by $\mathrm{Acc}@0.5$, trajectory prediction by Discrete Fréchet distance, area prediction by point-in-polygon accuracy, and affordance prediction by a bidirectional Chamfer-like Euclidean distance (Dang et al., 13 Feb 2026).

Chain-of-Point, implemented in the RynnBrain-CoP variant, is the reasoning formalism used to enforce physical grounding during intermediate inference steps (Dang et al., 13 Feb 2026). It interleaves textual thoughts with explicit spatial annotations such as <object> <frame n>: (x0,y0),(x1,y1) </object>, thereby anchoring intermediate reasoning to specific visual evidence. The training pipeline first performs cold-start SFT from pretrained RynnBrain on curated CoP data. Reasoning chains are auto-generated with Qwen3-VL-235B, entity types are classified as object or area, and human annotators add precise grounding in selected frames (Dang et al., 13 Feb 2026).

The RL stage uses group relative policy optimization. For a query $[0,1000]$0, the objective is

$[0,1000]$1

with group-normalized advantages

$[0,1000]$2

Task-specific rewards are defined for trajectory, affordance, and area grounding:

$[0,1000]$3

These reward definitions formalize the paper’s claim that physically grounded reasoning should be optimized against geometric criteria rather than only textual plausibility (Dang et al., 13 Feb 2026).

A plausible implication is that CoP functions as an alignment mechanism between language-model inference and embodied evidence. The paper explicitly contrasts this with purely textual “thinking,” arguing that interleaved spatial grounding reduces hallucinations and aligns decisions with physical evidence (Dang et al., 13 Feb 2026).

5. Physics-aware planning, navigation, and action execution

RynnBrain-Plan emits step-by-step plans with embedded spatial grounding—object bounding boxes, area points, and affordance points—directly in the decoded sequence (Dang et al., 13 Feb 2026). The source characterizes these outputs as “physics-aware” because each sub-step is tied to explicit spatial references that downstream controllers can execute more precisely in cluttered or multi-object settings. Planning is not formulated through an explicit dynamics model, closed-form trajectory optimization, or MPC objective; rather, it is learned as next-token prediction over mixed text-and-coordinate outputs, with grounded supervision providing the physics-aware bias (Dang et al., 13 Feb 2026).

The high-level planning pipeline is also specified. Given an instruction and recent visual context, the system encodes frames at 2 FPS with temporal embeddings, concatenates instruction and history, autoregressively decodes the next grounded sub-task, passes it to an executor such as UMI or VLA, appends execution feedback to the dialogue history, and repeats until the task is completed (Dang et al., 13 Feb 2026). The use of multi-turn dialogue as an explicit memory buffer is central to long-horizon behavior. The paper reports that multi-turn fine-tuning is critical, with single-turn models collapsing on Medium and Hard settings while multi-turn models deliver large gains, including up to 100% on OOD-Hard for RynnBrain-Plan-MT-30B (Dang et al., 13 Feb 2026).

RynnBrain-Nav specializes the foundation model for instruction-conditioned continuous navigation in photorealistic 3D environments. Its inference loop uses the interleaved history $[0,1000]$4 together with the instruction to decode the next action $[0,1000]$5 (Dang et al., 13 Feb 2026). The variant is trained by full-parameter SFT on R2R/RxR, EnvDrop, and ScaleVLN data reformatted as image-action sequences, followed by multi-turn DAgger iterations for data aggregation (Dang et al., 13 Feb 2026).

RynnBrain-VLA builds on RynnBrain-2B for low-latency control and adds a flow-matching Diffusion Transformer action head with linear projections for noisy actions and time embeddings (Dang et al., 13 Feb 2026). It is fine-tuned on teleoperated Franka pick-and-place episodes across six tasks and three objects, using text-format pointing inputs such as affordances, objects, and the initial frame (Dang et al., 13 Feb 2026). This preserves the instruction-following properties of the VLM backbone while extending the model into direct action generation.

6. Empirical performance, baselines, and limitations

The evaluation suite covers 28 benchmarks: 20 embodied benchmarks and 8 general visual understanding benchmarks (Dang et al., 13 Feb 2026). At 8B scale, RynnBrain-8B is reported as outperforming Qwen3-VL-8B on embodied cognition and localization benchmarks, including VSI-Bench 71.0 vs 60.3, RoboSpatial 73.1 vs 58.2, EgoTaskQA 72.5 vs 57.8, RynnBrain-Object 71.2 vs 41.8, and RynnBrain-Spatial 59.9 vs 35.0 (Dang et al., 13 Feb 2026). On localization tasks, the same model reports RefSpatial-Bench 59.2 vs 53.4, ShareRobot-Affordance 44.7 vs 37.0, Cornell-Grasp 26.6 vs 21.2, VMRD-Grasp 14.1 vs 7.1, RynnBrain-Grounding 81.6 vs 62.8, and RynnBrain-Affordance 90.4 vs 82.9 (Dang et al., 13 Feb 2026). On general visual understanding, the paper reports competitive parity or SOTA on AI2D 86.3, MVBench 69.5, and InfoVQA_test 83.4 (Dang et al., 13 Feb 2026).

At 30B-A3B scale, reported results include VSI-Bench 74.5, EgoTaskQA 78.9, Open-X VQA 83.4, RynnBrain-Object 73.3, RynnBrain-Spatial 59.3, Cornell-Grasp 33.6, VMRD-Grasp 14.5, RynnBrain-Grounding 83.9, RynnBrain-Affordance 90.5, RynnBrain-Area 59.4, and RynnBrain-Trajectory 66.8 (Dang et al., 13 Feb 2026). The source states that these results outperform open 30B-class baselines and approach very large proprietary models on some tasks (Dang et al., 13 Feb 2026).

The CoP variant is evaluated separately on affordance, area, and trajectory prediction. RynnBrain-CoP-8B achieves Affordance 90.3, Area 59.6, Trajectory 71.2, and Average 73.8, outperforming MiMo-Embodied-7B, RoboBrain2.0-32B, Gemini-3-Pro, GPT-5.2, and Qwen3-VL-30B-A3B-Thinking on the reported thinking-mode benchmarks (Dang et al., 13 Feb 2026). For vision-language navigation on Val-Unseen, RynnBrain-Nav-8B reports R2R-CE NE 4.92, OS 71.6, SR 58.6, SPL 49.6, and RxR-CE NE 6.20, SR 56.1, SPL 49.6, nDTW 59.6 (Dang et al., 13 Feb 2026). Multi-turn DAgger improves SR from 50.6% to 56.4% to 58.5% across two iterations (Dang et al., 13 Feb 2026). For manipulation, RynnBrain-Plan-30B-A3B is reported to achieve markedly higher Task Progress on Hard settings, including near 100% Task Progress on OOD Table Bussing across all difficulties (Dang et al., 13 Feb 2026). For real-robot execution, RynnBrain-VLA reports overall success rate 0.77 versus 0.60 for Qwen3-VL-Finetuned and 0.47 for $[0,1000]$6-Finetuned, with recognition success 0.97, pickup success 0.80, and corresponding baseline comparisons as given in the source (Dang et al., 13 Feb 2026).

The paper also identifies several limitations. RynnBrain has no explicit 3D SLAM or $[0,1000]$7 state estimation at inference, so tasks requiring metric 3D reasoning may require external modules (Dang et al., 13 Feb 2026). Navigation shows high OS but lower SR, indicating failure modes in terminal stopping precision. Area prediction remains challenging across models. The 30B-A3B model, despite sparse activation with approximately 3B active parameters, did not initially surpass the 8B model on VLN, suggesting that sparse MoE requires specialized training strategies for navigation (Dang et al., 13 Feb 2026). The authors further state that safety-critical deployment requires additional guardrails such as collision checking, force limits, and human-in-the-loop overrides (Dang et al., 13 Feb 2026).

7. Secondary usage: RynnBrain as an RSVP P300 BCI synthesis

In the supplied material, “RynnBrain” also names an overview that builds an RNN-based RSVP P300 BCI from Tal and Friedman’s “Recurrent Neural Networks for P300-based BCI” (Tal et al., 2019). That system addresses EEG-based P300 spellers using the rapid serial visual presentation paradigm, in which symbols are presented one at a time and rare attended events elicit a P300 ERP (Tal et al., 2019). The synthesis emphasizes that RSVP requires no saccades or eye-gaze control and is therefore suitable for users with severe ocular-motor impairment (Tal et al., 2019).

The summarized pipeline uses a dataset with 11 subjects, 55 EEG channels, approximately 20,000 labeled epochs per subject, 1 target per 30 stimuli, original sampling at 200 Hz, and downsampling to 25 Hz, with 1 s epochs spanning $[0,1000]$8 ms relative to stimulus onset (Tal et al., 2019). The model input shape is $[0,1000]$9, or 1,375 features per epoch. Architectures compared include non-shrinkage LDA using all 1,375 features, a CNN with a 55×1 spatial layer and a 1×5 temporal layer, LSTM-small with 30 hidden units, LSTM-large with 100 hidden units, and a CNN→LSTM hybrid in which spatial convolution produces a sequence of 10-dimensional features over 25 time steps for LSTM processing (Tal et al., 2019).

The reported results show a distinct contrast between subject-specific and cross-subject regimes. For per-subject training and testing, mean letter-level accuracies at $V=\{I_t\}_{t=1}^T$0 are LDA 0.96, CNN 0.98, LSTM-CNN-large 0.93, LSTM-small 0.86, and LSTM-large 0.82 (Tal et al., 2019). For pooled train/test on all subjects, results are CNN 0.92, LSTM-CNN-small 0.93, LSTM-small 0.90, LDA 0.79, and LSTM-large 0.77. For leave-one-subject-out with no calibration, CNN and LSTM-CNN-small both reach 0.84, while LDA reaches 0.65; with fine-tuning, CNN and LSTM-CNN-small both reach 0.97 (Tal et al., 2019). The synthesis therefore concludes that LDA rivals or exceeds RNNs in single-subject settings, whereas CNN and CNN+LSTM generalize better across subjects, with the hybrid showing particular resilience to temporal noise (Tal et al., 2019).

Temporal robustness is the most distinctive property emphasized for the hybrid model. With onset offsets of $V=\{I_t\}_{t=1}^T$1 ms, letter accuracy at $V=\{I_t\}_{t=1}^T$2 is 0.864 and 0.896 for LSTM-CNN versus 0.825 and 0.848 for CNN, while LDA is substantially lower at 0.565 and 0.608 (Tal et al., 2019). Saliency analysis is reported to concentrate CNN+LSTM sensitivity around 250–450 ms, matching P300 latency and reducing susceptibility to noise elsewhere in the epoch (Tal et al., 2019). Computationally, training time on a CPU 4-core i7 with 16 GB RAM is reported as approximately 24 s for CNN and approximately 110 s for LSTM-CNN-small; inference is approximately 0.6 ms per epoch, and model size is below 70 KB for both CNN and LSTM-CNN-small (Tal et al., 2019).

This secondary usage is methodologically distinct from the embodied foundation model family, but both uses of the term emphasize temporally structured perception and explicit handling of uncertainty in sequential signals. That parallel is interpretive rather than terminological: the source data explicitly supports the two technical descriptions, but it does not claim a direct lineage between the 2019 RSVP P300 study and the 2026 embodied foundation model (Tal et al., 2019, Dang et al., 13 Feb 2026).