---
title: 'RynnBrain: Unified Spatiotemporal Model'
url: https://www.emergentmind.com/topics/rynnbrain
type: topic
---

# RynnBrain: Unified Spatiotemporal Model

Searching arXiv for the cited papers and topic coverage.
RynnBrain is an open-source spatiotemporal foundation model family for embodied intelligence that integrates perception, reasoning, and planning within real-world spatial-temporal dynamics [2602.14979]. In the supplied literature, it is defined by four unified capabilities—comprehensive egocentric understanding, diverse spatiotemporal localization, physically grounded reasoning, and physics-aware planning—and by a physically grounded output space in which points, boxes, and trajectories are emitted as discrete coordinate tokens on a unified $[0,1000]$ grid [2602.14979]. The same source positions RynnBrain as a general-purpose embodied “brain” that can be post-trained for navigation, manipulation planning, vision-language-action execution, and complex spatial reasoning, while a separate synthesis uses the label for an RNN-based RSVP P300 BCI pipeline derived from Tal and Friedman’s study of recurrent neural networks for P300 detection [1901.10798].

## 1. Conceptual scope and defining properties

RynnBrain was introduced to address a gap between broadly capable vision-language models and action-centric embodied systems [2602.14979]. Prior VLMs are described as not physically grounded: they may hallucinate spatial details, lack temporal consistency, and operate purely in text, which impedes actionable plans. Conversely, action-centric embodied models are described as sacrificing high-level semantics and generalization. RynnBrain is presented as a unified, physically grounded spatiotemporal foundation model intended to reconcile these two regimes by integrating perception, reasoning, and planning while remaining adaptable to downstream control [2602.14979].

Its central design commitment is a shared output space in which textual generation is interleaved with explicit spatial grounding. Spatial predictions are represented as discrete coordinate tokens in a normalized $[0,1000]$ grid for points, boxes, and trajectories, so that reasoning and planning can be expressed in the same autoregressive stream as language [2602.14979]. This physically grounded output space is the principal distinction repeatedly emphasized in the source material. The paper also frames RynnBrain as open infrastructure: code, checkpoints, and datasets are released under the Apache 2.0 license [2602.14979].

The supplied literature associates the name with two different technical contexts. In its primary and explicit sense, RynnBrain is the embodied foundation model family introduced in 2026 [2602.14979]. In a secondary editorial synthesis, the label is also applied to a practical RSVP P300 BCI configuration centered on CNN, LSTM, and CNN+RNN modeling choices derived from Tal and Friedman’s “Recurrent Neural Networks for P300-based BCI” [1901.10798]. This suggests that the term has been used both as a proper model family name and as a broader organizing label for physically grounded or temporally grounded neural systems.

## 2. Architecture, model family, and output representation

The RynnBrain family comprises three foundation model scales—RynnBrain-2B, RynnBrain-8B, and RynnBrain-30B-A3B MoE—and four post-trained variants tailored for downstream embodied tasks or complex spatial reasoning [2602.14979]. RynnBrain-2B and RynnBrain-8B are dense decoder-only vision-language models initialized from Qwen3-VL-2B/8B-Instruct, while RynnBrain-30B-A3B is a mixture-of-experts variant initialized from Qwen3-VL-30B-A3B-Instruct [2602.14979]. For the MoE model, the active parameter count is 3B per token, whereas the total parameter count is 30B; expert parallelism with world size 2, grouped linear kernels, and DeepEP are used for training efficiency [2602.14979].

The backbone is a decoder-only vision-language architecture comprising a vision encoder, a vision-language projector, and an LLM, with DeepStack used to enhance visual token depth and Interleaved MRoPE used to improve positional encoding across modalities and time [2602.14979]. Inputs include images and videos, with videos represented as sequences of frames $V=\{I_t\}_{t=1}^T$ and temporal sampling at 2 FPS for long-horizon tasks. Text instructions and interleaved histories are handled in conversational format [2602.14979].

A concise summary of the model family and major variants is given below.

| Model | Type | Primary target |
|---|---|---|
| RynnBrain-2B | Dense decoder-only VLM | General embodied foundation model |
| RynnBrain-8B | Dense decoder-only VLM | General embodied foundation model |
| RynnBrain-30B-A3B | MoE decoder-only VLM | Higher-capacity embodied foundation model |
| RynnBrain-CoP | Post-trained variant | Physically grounded reasoning |
| RynnBrain-Nav | Post-trained variant | Vision-language navigation |
| RynnBrain-Plan | Post-trained variant | Manipulation planning |
| RynnBrain-VLA | Post-trained variant | Vision-language-action execution |

The output representation is structurally important because localization, reasoning, and planning all share the same tokenization scheme. The inference format includes tags such as `<object>`, `<area>`, `<affordance>`, and `<trajectory>`, each paired with a `<frame n>` index and coordinate tuples [2602.14979]. The model does not maintain an explicit 3D map or occupancy grid at inference, and it does not integrate SLAM or explicit $SE(3)$ pose estimation at runtime; instead, its grounding is image-plane and episode-indexed [2602.14979]. Offline 3D reconstructions via MASt3R-SLAM are used for label generation in some datasets but are not part of deployed inference [2602.14979].

## 3. Training regime, data mixture, and post-training

RynnBrain is pretrained on approximately 19.89M samples spanning general multimodal instruction data and embodied cognition, localization, and planning data [2602.14979]. The mixture includes 4.8M general MLLM samples; cognition subsets for object understanding, spatial understanding, counting, OCR, and egocentric task understanding; localization subsets for object localization, area localization, affordance localization, trajectory prediction, and grasp pose; and 0.16M planning samples [2602.14979]. This breadth is intended to support long-horizon video understanding, physically grounded prediction, and downstream adaptation within one framework.

The unified training objective is autoregressive next-token prediction over mixed text and discretized coordinate tokens:
$$
L = -\sum_{i=1}^L \log P(y_i \mid y_{<i}, V, \Theta).
$$
For efficient distributed training under long-tail sequence lengths, the paper also specifies a per-sample loss reduction scheme over distributed workers [2602.14979]. Training infrastructure includes ZeRO-1 for 2B and 8B, ZeRO-2 plus expert parallelism for 30B-A3B, gradient checkpointing, loss-token filtering, DeepEP for token dispatch, grouped linear MoE kernels, and the HuggingFace Transformers training framework [2602.14979].

Pretraining hyperparameters are stated explicitly. The 2B model uses AdamW with learning rate $5\times10^{-6}$, vision learning rate $1\times10^{-6}$, batch size 512, and 3% warmup. The 8B model uses AdamW with learning rate $2\times10^{-6}$, vision learning rate $2\times10^{-6}$, batch size 1024, and 3% warmup. The 30B-A3B model uses the same learning rate and vision learning rate as 8B, with batch size 1024 and 3% warmup [2602.14979].

Post-training differentiates the variants. RynnBrain-CoP performs full-parameter SFT on Chain-of-Point-augmented data and then reinforcement learning with GRPO using spatially grounded rewards [2602.14979]. RynnBrain-Nav is fully fine-tuned on VLN-CE datasets reformatted as interleaved image-action sequences and further improved with multi-turn DAgger data collection [2602.14979]. RynnBrain-Plan is fine-tuned on small, multi-turn dialogues that function as explicit memory buffers for long-horizon tasks, while RynnBrain-VLA adds a flow-matching action head and is fine-tuned on teleoperated Franka pick-and-place episodes [2602.14979].

## 4. Spatiotemporal localization and Chain-of-Point reasoning

RynnBrain’s localization mechanism predicts temporal key frames and spatial coordinates jointly in a shared token space [2602.14979]. Object localization is emitted as bounding boxes; area localization as polygons or point sets; affordance localization as points; and trajectory localization as ordered coordinate sequences attached to frame indices. The corresponding evaluation metrics are also explicitly defined: object grounding by $\mathrm{Acc}@0.5$, trajectory prediction by Discrete Fréchet distance, area prediction by point-in-polygon accuracy, and affordance prediction by a bidirectional Chamfer-like Euclidean distance [2602.14979].

Chain-of-Point, implemented in the RynnBrain-CoP variant, is the reasoning formalism used to enforce physical grounding during intermediate inference steps [2602.14979]. It interleaves textual thoughts with explicit spatial annotations such as
`<object> <frame n>: (x0,y0),(x1,y1) </object>`,
thereby anchoring intermediate reasoning to specific visual evidence. The training pipeline first performs cold-start SFT from pretrained RynnBrain on curated CoP data. Reasoning chains are auto-generated with Qwen3-VL-235B, entity types are classified as object or area, and human annotators add precise grounding in selected frames [2602.14979].

The RL stage uses group relative policy optimization. For a query $q$, the objective is
$$
J_{\mathrm{GRPO}}(\theta)=\mathbb{E}\left[\frac{1}{G}\sum_{i=1}^G\left(\min(\rho_i A_i,\mathrm{clip}(\rho_i,1-\epsilon,1+\epsilon)A_i)-\beta D_{KL}(\pi_\theta(o_i|q)\|\pi_{\mathrm{ref}}(o_i|q))\right)\right],
$$
with group-normalized advantages
$$
A_i=\frac{r_i-\mathrm{mean}(\{r_j\})}{\mathrm{std}(\{r_j\})+\epsilon}.
$$
Task-specific rewards are defined for trajectory, affordance, and area grounding:
$$
r_{\mathrm{traj}}=\exp(-\lambda_{\mathrm{traj}}\cdot D_F), \quad
r_{\mathrm{aff}}=\exp(-\lambda_{\mathrm{aff}}\cdot D_{\mathrm{bidir}}(P,G)), \quad
r_{\mathrm{area}}=\frac{1}{|P|}\sum_{p\in P} I(p\in S_G).
$$
These reward definitions formalize the paper’s claim that physically grounded reasoning should be optimized against geometric criteria rather than only textual plausibility [2602.14979].

A plausible implication is that CoP functions as an alignment mechanism between language-model inference and embodied evidence. The paper explicitly contrasts this with purely textual “thinking,” arguing that interleaved spatial grounding reduces hallucinations and aligns decisions with physical evidence [2602.14979].

## 5. Physics-aware planning, navigation, and action execution

RynnBrain-Plan emits step-by-step plans with embedded spatial grounding—object bounding boxes, area points, and affordance points—directly in the decoded sequence [2602.14979]. The source characterizes these outputs as “physics-aware” because each sub-step is tied to explicit spatial references that downstream controllers can execute more precisely in cluttered or multi-object settings. Planning is not formulated through an explicit dynamics model, closed-form trajectory optimization, or MPC objective; rather, it is learned as next-token prediction over mixed text-and-coordinate outputs, with grounded supervision providing the physics-aware bias [2602.14979].

The high-level planning pipeline is also specified. Given an instruction and recent visual context, the system encodes frames at 2 FPS with temporal embeddings, concatenates instruction and history, autoregressively decodes the next grounded sub-task, passes it to an executor such as UMI or VLA, appends execution feedback to the dialogue history, and repeats until the task is completed [2602.14979]. The use of multi-turn dialogue as an explicit memory buffer is central to long-horizon behavior. The paper reports that multi-turn fine-tuning is critical, with single-turn models collapsing on Medium and Hard settings while multi-turn models deliver large gains, including up to 100% on OOD-Hard for RynnBrain-Plan-MT-30B [2602.14979].

RynnBrain-Nav specializes the foundation model for instruction-conditioned continuous navigation in photorealistic 3D environments. Its inference loop uses the interleaved history $\{o_0,a_0,\dots,o_t\}$ together with the instruction to decode the next action $a_t\in\{\leftarrow,\rightarrow,\uparrow,\mathrm{STOP}\}$ [2602.14979]. The variant is trained by full-parameter SFT on R2R/RxR, EnvDrop, and ScaleVLN data reformatted as image-action sequences, followed by multi-turn DAgger iterations for data aggregation [2602.14979].

RynnBrain-VLA builds on RynnBrain-2B for low-latency control and adds a flow-matching Diffusion Transformer action head with linear projections for noisy actions and time embeddings [2602.14979]. It is fine-tuned on teleoperated Franka pick-and-place episodes across six tasks and three objects, using text-format pointing inputs such as affordances, objects, and the initial frame [2602.14979]. This preserves the instruction-following properties of the VLM backbone while extending the model into direct action generation.

## 6. Empirical performance, baselines, and limitations

The evaluation suite covers 28 benchmarks: 20 embodied benchmarks and 8 general visual understanding benchmarks [2602.14979]. At 8B scale, RynnBrain-8B is reported as outperforming Qwen3-VL-8B on embodied cognition and localization benchmarks, including VSI-Bench 71.0 vs 60.3, RoboSpatial 73.1 vs 58.2, EgoTaskQA 72.5 vs 57.8, RynnBrain-Object 71.2 vs 41.8, and RynnBrain-Spatial 59.9 vs 35.0 [2602.14979]. On localization tasks, the same model reports RefSpatial-Bench 59.2 vs 53.4, ShareRobot-Affordance 44.7 vs 37.0, Cornell-Grasp 26.6 vs 21.2, VMRD-Grasp 14.1 vs 7.1, RynnBrain-Grounding 81.6 vs 62.8, and RynnBrain-Affordance 90.4 vs 82.9 [2602.14979]. On general visual understanding, the paper reports competitive parity or SOTA on AI2D 86.3, MVBench 69.5, and InfoVQA_test 83.4 [2602.14979].

At 30B-A3B scale, reported results include VSI-Bench 74.5, EgoTaskQA 78.9, Open-X VQA 83.4, RynnBrain-Object 73.3, RynnBrain-Spatial 59.3, Cornell-Grasp 33.6, VMRD-Grasp 14.5, RynnBrain-Grounding 83.9, RynnBrain-Affordance 90.5, RynnBrain-Area 59.4, and RynnBrain-Trajectory 66.8 [2602.14979]. The source states that these results outperform open 30B-class baselines and approach very large proprietary models on some tasks [2602.14979].

The CoP variant is evaluated separately on affordance, area, and trajectory prediction. RynnBrain-CoP-8B achieves Affordance 90.3, Area 59.6, Trajectory 71.2, and Average 73.8, outperforming MiMo-Embodied-7B, RoboBrain2.0-32B, Gemini-3-Pro, GPT-5.2, and Qwen3-VL-30B-A3B-Thinking on the reported thinking-mode benchmarks [2602.14979]. For vision-language navigation on Val-Unseen, RynnBrain-Nav-8B reports R2R-CE NE 4.92, OS 71.6, SR 58.6, SPL 49.6, and RxR-CE NE 6.20, SR 56.1, SPL 49.6, nDTW 59.6 [2602.14979]. Multi-turn DAgger improves SR from 50.6% to 56.4% to 58.5% across two iterations [2602.14979]. For manipulation, RynnBrain-Plan-30B-A3B is reported to achieve markedly higher Task Progress on Hard settings, including near 100% Task Progress on OOD Table Bussing across all difficulties [2602.14979]. For real-robot execution, RynnBrain-VLA reports overall success rate 0.77 versus 0.60 for Qwen3-VL-Finetuned and 0.47 for $\pi_{0.5}$-Finetuned, with recognition success 0.97, pickup success 0.80, and corresponding baseline comparisons as given in the source [2602.14979].

The paper also identifies several limitations. RynnBrain has no explicit 3D SLAM or $SE(3)$ state estimation at inference, so tasks requiring metric 3D reasoning may require external modules [2602.14979]. Navigation shows high OS but lower SR, indicating failure modes in terminal stopping precision. Area prediction remains challenging across models. The 30B-A3B model, despite sparse activation with approximately 3B active parameters, did not initially surpass the 8B model on VLN, suggesting that sparse MoE requires specialized training strategies for navigation [2602.14979]. The authors further state that safety-critical deployment requires additional guardrails such as collision checking, force limits, and human-in-the-loop overrides [2602.14979].

## 7. Secondary usage: RynnBrain as an RSVP P300 BCI synthesis

In the supplied material, “RynnBrain” also names a synthesis that builds an RNN-based RSVP P300 BCI from Tal and Friedman’s “Recurrent Neural Networks for P300-based BCI” [1901.10798]. That system addresses EEG-based P300 spellers using the rapid serial visual presentation paradigm, in which symbols are presented one at a time and rare attended events elicit a P300 ERP [1901.10798]. The synthesis emphasizes that RSVP requires no saccades or eye-gaze control and is therefore suitable for users with severe ocular-motor impairment [1901.10798].

The summarized pipeline uses a dataset with 11 subjects, 55 EEG channels, approximately 20,000 labeled epochs per subject, 1 target per 30 stimuli, original sampling at 200 Hz, and downsampling to 25 Hz, with 1 s epochs spanning $[-200,+800]$ ms relative to stimulus onset [1901.10798]. The model input shape is $55 \times 25$, or 1,375 features per epoch. Architectures compared include non-shrinkage LDA using all 1,375 features, a CNN with a 55×1 spatial layer and a 1×5 temporal layer, LSTM-small with 30 hidden units, LSTM-large with 100 hidden units, and a CNN→LSTM hybrid in which spatial convolution produces a sequence of 10-dimensional features over 25 time steps for LSTM processing [1901.10798].

The reported results show a distinct contrast between subject-specific and cross-subject regimes. For per-subject training and testing, mean letter-level accuracies at $R=10$ are LDA 0.96, CNN 0.98, LSTM-CNN-large 0.93, LSTM-small 0.86, and LSTM-large 0.82 [1901.10798]. For pooled train/test on all subjects, results are CNN 0.92, LSTM-CNN-small 0.93, LSTM-small 0.90, LDA 0.79, and LSTM-large 0.77. For leave-one-subject-out with no calibration, CNN and LSTM-CNN-small both reach 0.84, while LDA reaches 0.65; with fine-tuning, CNN and LSTM-CNN-small both reach 0.97 [1901.10798]. The synthesis therefore concludes that LDA rivals or exceeds RNNs in single-subject settings, whereas CNN and CNN+LSTM generalize better across subjects, with the hybrid showing particular resilience to temporal noise [1901.10798].

Temporal robustness is the most distinctive property emphasized for the hybrid model. With onset offsets of $\pm40$ ms, letter accuracy at $R=10$ is 0.864 and 0.896 for LSTM-CNN versus 0.825 and 0.848 for CNN, while LDA is substantially lower at 0.565 and 0.608 [1901.10798]. Saliency analysis is reported to concentrate CNN+LSTM sensitivity around 250–450 ms, matching P300 latency and reducing susceptibility to noise elsewhere in the epoch [1901.10798]. Computationally, training time on a CPU 4-core i7 with 16 GB RAM is reported as approximately 24 s for CNN and approximately 110 s for LSTM-CNN-small; inference is approximately 0.6 ms per epoch, and model size is below 70 KB for both CNN and LSTM-CNN-small [1901.10798].

This secondary usage is methodologically distinct from the embodied foundation model family, but both uses of the term emphasize temporally structured perception and explicit handling of uncertainty in sequential signals. That parallel is interpretive rather than terminological: the source data explicitly supports the two technical descriptions, but it does not claim a direct lineage between the 2019 RSVP P300 study and the 2026 embodied foundation model [1901.10798] [2602.14979].

Source: https://www.emergentmind.com/topics/rynnbrain