OmniMouse: Scaling properties of multi-modal, multi-task Brain Models on 150B Neural Tokens
Abstract: Scaling data and artificial neural networks has transformed AI, driving breakthroughs in language and vision. Whether similar principles apply to modeling brain activity remains unclear. Here we leveraged a dataset of 3.1 million neurons from the visual cortex of 73 mice across 323 sessions, totaling more than 150 billion neural tokens recorded during natural movies, images and parametric stimuli, and behavior. We train multi-modal, multi-task models that support three regimes flexibly at test time: neural prediction, behavioral decoding, neural forecasting, or any combination of the three. OmniMouse achieves state-of-the-art performance, outperforming specialized baselines across nearly all evaluation regimes. We find that performance scales reliably with more data, but gains from increasing model size saturate. This inverts the standard AI scaling story: in language and computer vision, massive datasets make parameter scaling the primary driver of progress, whereas in brain modeling -- even in the mouse visual cortex, a relatively simple system -- models remain data-limited despite vast recordings. The observation of systematic scaling raises the possibility of phase transitions in neural modeling, where larger and richer datasets might unlock qualitatively new capabilities, paralleling the emergent properties seen in LLMs. Code available at https://github.com/enigma-brain/omnimouse.
Summary
- The paper introduces a unified transformer-based framework that integrates neural, visual, and behavioral data for multi-task brain modeling.
- It demonstrates that neural prediction performance scales with data volume while saturating with increased model size.
- The study implies that enhancing data diversity is critical for advancing brain-scale models beyond traditional AI scaling laws.
OmniMouse: An Integrated Scaling Analysis of Multi-Modal, Multi-Task Brain Models in Mouse Visual Cortex
Introduction
The paper "OmniMouse: Scaling properties of multi-modal, multi-task Brain Models on 150B Neural Tokens" (2604.18827) presents a comprehensive study of scaling laws in neural predictive modeling, utilizing a unique dataset comprising over 150 billion single-neuron activity tokens collected from more than 3 million neurons across 323 sessions in the mouse visual cortex. OmniMouse introduces a unified transformer-based framework that jointly integrates neural responses, visual stimuli, and behavioral variables, supporting arbitrary masking configurations for flexible multi-task modeling. The principal scientific objective is to determine to what extent model and dataset scaling improve performance on neural prediction, forecasting, and behavioral decoding tasks, and how these scaling properties align or diverge from canonical trends observed in language and vision models.
Figure 1: Overview of the OmniMouse framework, demonstrating unified task support and contrasting effects of model versus dataset scaling on performance.
Dataset and Experimental Setup
OmniMouse leverages one of the largest calcium imaging datasets available, capturing the activity of excitatory neurons in the visual cortex of 73 mice. Recordings were performed with high-throughput two-photon imaging while the animals experienced a rich set of visual stimuli, including dynamic natural movies, images, and parametric patterns. Behavioral parameters, notably locomotion and comprehensive pupil dynamics, were collected synchronously. This multi-modal structure enables systematic study of context dependence in neural activity.
Figure 2: Data acquisition summary: calcium imaging setup, neural/behavioral variables, and visual stimuli distributions.
The dataset is downsampled, temporally aligned, and organized into contiguous chunks suitable for the transformer-based modeling pipeline. The resulting dataset's scale and diversity are unprecedented in neural modeling, supporting nuanced scaling experiments.
Model Architecture
OmniMouse is built on top of a flexible, multi-modal transformer architecture employing per-neuron, per-time-sample tokenization. The model incorporates learned identity embeddings for neuron, session, and animal, and employs a hierarchical vision transformer (Hiera) to encode visual input. The masking strategy enables it to support arbitrary combinations of forecasting, sub-population prediction, stimulus encoding, and behavioral decoding tasks. Each modality (neural, visual, behavioral) is tokenized independently and can be selectively masked or used as a conditional input.
Figure 3: Model schematic highlighting the flexible masking and unified decoding regime across neural, stimulus, and behavioral modalities.
Encoding is based on cross-attention with learned queries, facilitating variable-length input compression; multi-modal fusion layers with interleaved local/global attention integrate context; the decoder reconstructs targeted neural or behavioral outputs using contextualized embeddings. This design aligns closely with advances in language and vision foundation models, but is tailored to neuroscientific constraints (e.g., per-neuron embeddings, explicit behavioral alignment).
Scaling Experiments: Model Size Versus Data Volume
The core contribution resides in a systematic scaling analysis. The authors independently vary model capacity (ranging from 1M to 300M parameters) and training set size (ranging from 8 to 323 sessions), using controlled, nested evaluation splits to ensure fair comparison. Five principal task configurations are evaluated: pure forecasting, population prediction, stimulus encoding, stimulus-conditioned forecasting, and stimulus-conditioned population prediction. Additionally, behavior decoding is assessed across variables.
The key empirical result is that performance reliably scales with data, but strongly saturates with model size (Figure 1, BโC; Figure 4, Figure 5). Unlike the classic scaling observed in LLMs, where model size is the predominant performance driver given sufficient data [kaplan2020scaling, hoffmann2022training], in neural modeling the predictive bottleneck is the dataset, not model width or depth.
Figure 4: Task-specific scaling dynamics with model size for multiple prediction and decoding regimes.
Figure 5: Systematic performance improvement with additional training sessions, across model sizes and prediction tasks.
This inversion of AI scaling laws is robust across all tasks: for model sizes โฅ80M, performance gains flatten even as parameter counts increase fourfold or more. By contrast, expanding dataset size never saturates within the studied regime, particularly for behavioral decoding, where improvements scale linearly with dataset growth.
Multi-Task Performance and Behavioral Decoding
OmniMouse outperforms all specialized baselines in both data-matched and full-dataset settings, confirming that its architectural and masking innovations generalize across modalities and tasks. Importantly, this state-of-the-art performance is achieved even when main baselines are retrained on identical subsets; thus, gains are not attributable merely to increased data, but also to the model's multi-task, unified structure.
Figure 6: Decoding accuracy for behavioral variables scales with model size and continues to benefit from added capacity and data.
The predictive accuracy for pupil location, size, and running speed demonstrates the model's ability to exploit context from large populations to decode overt behavioral outputs. Yet, even here, scaling remains data-limited rather than parameter-limited.
Systematic Context Manipulation and Generalization
Due to flexible masking and unified decoding, OmniMouse enables unprecedented in-silico experiments dissecting the contributions of neural, behavioral, and stimulus context to response variability. Analysis of context length and population size reveals smooth, interpretable trends: additional context (either temporal or population-based) monotonically improves predictive performance, plateauing when available information is exhausted.
Figure 7: Qualitative examples demonstrating the effect of different contextual inputs on neuron and behavior prediction accuracy.
Figure 8: Detailed investigation of context length (neuronal and visual) for forecasting and population prediction tasks; context makes a quantifiable impact on accuracy.
Even for generalization to novel masking configurations, OmniMouse preserves robust performance, indicating it learns deeply generalizable multimodal representations.
Discussion: Implications and Future Directions
The principal theoretical implication of this work is that scaling laws in neural models diverge from the paradigms of language and vision, owing to the higher data constraints and strong code redundancy in biological systems. The observed performance saturation with model size (even at the scale of 3M neurons and 150B tokens) supports the hypothesis that, for current neuroscientific datasets, further architectural or capacity scaling yields diminishing returns unless accompanied by richer, more heterogenous data.
This result is consistent with parallel findings in recent neural decoding and neuroscience foundation modeling literature [jiang2025data, ye2025generalist, gokce2024scaling], and challenges the direct translation of scaling recipes from AI to brain modeling.
Practically, this finding informs the field that the construction of future brain-scale models should prioritize expanding data diversityโacross tasks, stimuli, animal species, and modalitiesโover model depth or parameter count. The authors speculate that the field may see "phase transitions" in neural modeling abilities only with the advent of brain data at the scale, variety, and richness analogous to internet-scale corpora in language and vision domainsโpotentially revealing emergent properties and qualitative shifts, as seen in LLMs [Wei2022-fa].
There are architectural limitations: OmniMouse's parameterization scales linearly with neuron count due to per-neuron embeddings, presenting a bottleneck for extrapolation to larger or more diverse nervous systems. Interpretability in large multi-task transformers also remains a challenge for drawing mechanistic insights.
Conclusion
OmniMouse sets a new empirical and conceptual benchmark for multi-modal brain modeling, demonstrating that data, rather than model size, is the dominant driver of predictive gains in large-scale neural systems. The work motivates a field-wide focus on larger, more varied datasets, and provides a flexible architectural blueprint for future neuroscience foundation models. The systematic scaling analysis, unified multi-task modeling, and flexible evaluation regime will serve as critical references for future work in both theoretical and applied brain modeling.
(2604.18827)
Paper Prompts
Sign up for free to create and run prompts on this paper using GPT-5.
Top Community Prompts
Explain it Like I'm 14
What is this paper about?
This paper introduces OmniMouse, a new kind of AI model that learns from the brains of mice. It looks at how scaling upโusing more brain data and bigger modelsโaffects how well we can predict what the brain will do. The team trained OmniMouse on a huge dataset: signals from about 3.1 million neurons across 73 mice, collected in 323 recording sessions. In total, thatโs more than 150 billion โneural tokensโ (tiny time-slices of brain activity). The model connects three things at once:
- Neural activity in the visual cortex (the part of the brain that sees),
- What the mouse is looking at (images or videos), and
- What the mouse is doing (like running or changes in pupil size).
What questions did the researchers ask?
In simple terms, they wanted to know:
- If we give an AI model more brain data, does it get better at predicting brain activity and behavior?
- Do bigger models always help, or do they eventually stop improving?
- Can one unified model handle multiple tasksโlike predicting future brain activity, guessing behavior from brain signals, and explaining responses to visual inputโbetter than separate specialized models?
How did they do the study?
Think of the dataset as a multi-camera, multi-sensor recording of each mouse:
- โNeural tokensโ are like frames in a video but for neuronsโtiny slices of time telling how much each neuron was active.
- The mice were shown different kinds of visuals (natural images and videos, as well as well-controlled patterns like stripes and dots).
- Behavior was tracked too: running speed, pupil position, and pupil size.
OmniMouse is a multi-modal model, meaning it can take in several types of data at once:
- Neurons: It turns each neuronโs activity over short time windows into tokens (like words in a sentence).
- Video: It encodes frames of the visual stimulus the mouse saw.
- Behavior: It can either use behavior as extra context or try to predict it.
A key idea they used is โmasking,โ like playing hide-and-seek with the data:
- Sometimes the model sees only past neural activity and has to predict the next second (forecasting).
- Sometimes it sees some neurons and predicts the rest (population prediction).
- Sometimes it sees the video and predicts the brainโs response (stimulus encoding).
- Sometimes it uses brain activity to predict behavior (behavior decoding).
- And sometimes it combines these, like seeing both video and past brain activity to predict whatโs next.
They trained models of different sizes (from about 1 million to 300 million parameters) and different amounts of data (from 8 sessions up to all 323). They then tested on standardized datasets so comparisons were fair.
What did they find?
Here are the main findings explained simply:
- More data helps a lot: When they trained with more sessions (more mice, more neurons), performance consistently went up across tasks and model sizes. This means the models are โdata-limitedโ right nowโextra data is the main thing that makes them better.
- Bigger models helpโฆbut only up to a point: Increasing the size of the model improved results until around 80 million parameters. Beyond that, gains were very small or stopped. This is different from many language and vision AI systems, where just making models bigger keeps helping.
- One model beats many specialized ones: OmniMouse performed better than strong baseline models on almost all tasksโpredicting future brain activity, predicting responses to videos, and decoding behaviorโespecially when trained on the same amount of data.
- Behavior decoding shows classic scaling: Predicting things like pupil size and running from brain activity improved smoothly as models got bigger and saw more data, with less sign of hitting a hard limit.
- Video helps: Tasks that included the visual stimulus (video or images) benefited the most from larger datasets, suggesting that richer, more varied visual experiences can unlock better brain modeling.
Why is this important? It flips the usual AI story. In language and computer vision, huge datasets often make model size the main driver of progress. Here, for brain modeling, even with massive recordings, data is still the bottleneck. That hints we may need even larger, more diverse, and richer brain datasets to see the kinds of โemergentโ abilities weโve observed in LLMs.
Why does it matter?
- Toward โfoundation modelsโ for neuroscience: OmniMouse shows that one unified model can learn from many kinds of brain-related data at once and perform many tasks. Thatโs a step toward general-purpose brain models that could help researchers understand how different parts of the brain work together.
- Better predictions could lead to better science: If models can reliably predict brain activity and behavior from stimuli and past signals, scientists can test ideas faster, design better experiments, and explore how brain state (like attention or arousal) changes perception.
- Data, not just bigger models: The results suggest that collecting richer, longer, and more varied brain recordingsโpossibly including more behaviors, tasks, and sensory inputsโwill matter more than simply scaling up the model size.
- Possible โphase transitionsโ: In AI, small performance improvements at large scale sometimes lead to new, surprising abilities. The authors think building larger, better datasets for brain modeling could unlock similarly new capabilities, helping reveal deeper principles of how brains compute.
In short: OmniMouse shows that with lots of brain data, a single, flexible model can predict neural activity and behavior across many tasks. To keep improving, we need more and better dataโsuggesting the next big leap in brain modeling will come from scaling recordings and experiences, not just model size.
Knowledge Gaps
Knowledge gaps, limitations, and open questions
Below is a consolidated list of unresolved issues that are missing, uncertain, or left unexplored in the paper. Each point is phrased to guide concrete follow-up work.
Data and experimental coverage
- Limited species and brain-region scope: all data are from excitatory neurons in mouse visual cortex; it is unknown how the approach transfers to inhibitory cell types, other cortical layers/areas, subcortical regions, or other species.
- Calcium imaging constraints: reliance on deconvolved calcium (6โ14 Hz) upsampled to 30 Hz may distort temporal dynamics; the impact of deconvolution errors, temporal blur, and sampling mismatch on forecasting and stimulus-locked responses is not quantified.
- Stimulus diversity: video/image stimuli are naturalistic but visually constrained (no audio, multisensory cues, or task contingencies). It is unclear what type/amount of stimulus diversity is required to improve scaling or prevent performance saturation.
- Behavior repertoire: behavior is limited to running speed and pupil metrics recorded during passive viewing; generalization to richer, task-engaged behaviors (decisions, reach, locomotion in VR, etc.) is untested.
- Timing and latency calibration: display latencies, retinotopic delays, and calcium kinetics are not explicitly modeled; the consequences for temporal alignment and causal inference remain unknown.
- Cross-lab/device heterogeneity: no evaluation on data from different labs, setups, or imaging hardware to test robustness to acquisition variability.
- OOD generalization: no tests on out-of-distribution stimuli (e.g., novel categories, motion statistics, or perturbations) or behaviors.
Model architecture and training
- Neuron-specific parameter scaling: per-neuron embeddings make parameter count scale linearly with recorded cells; strategies to reduce or amortize per-neuron parameters (e.g., anatomical/retinotopic readouts, cell-type priors, graph constraints, MoE) are not explored.
- Vision encoder pretraining: the Hiera video encoder is randomly initialized; the effect of pretrained spatiotemporal backbones on stimulus encoding and data efficiency is unassessed.
- Masking โrecipeโ design: 119 masking configurations are used, but there is no ablation showing which masks drive gains, how multi-task interference manifests, or how to optimally weight tasks (the 0.1 behavior loss weight is heuristic).
- Forecasting horizon: all neural forecasting uses a 1 s target; scaling to longer horizons and the failure modes (error accumulation, drift) are not studied.
- Temporal tokenization choices: stride/window for neural tokens are fixed; no ablation of tokenization granularity on timescale sensitivity, latency, or context utilization.
- Uncertainty modeling: predictions are point estimates with Poisson/MSE losses; calibrated uncertainty, trial-to-trial variability modeling, and probabilistic decoding (e.g., quantile or Bayesian objectives) are not investigated.
- Measurement models: a principled forward model for calcium measurement (nonlinear dynamics, noise, decay) is absent; it is unclear if explicit measurement models would improve forecasting and stimulus encoding.
- Compute scaling regime: training schedules are not tuned for compute/data-optimal regimes (e.g., Chinchilla-like tradeoffs); it remains unknown whether the observed parameter saturation persists under compute-optimal scaling.
- Compute accounting: FLOPs exclude neuron-specific operations; a standardized and comparable compute budget (including per-neuron readouts) is not reported, complicating fair scaling comparisons.
Evaluation and analysis
- Noise ceiling and explainable variance: single-trial correlation is reported without noise-ceiling adjustments; it is unclear what fraction of explainable variability is captured or how close models are to intrinsic limits.
- Area- and cell-type-specific performance: no breakdown by cortical area, layer, retinotopic location, or cell-type proxies; heterogeneous gains across areas are not quantified.
- Mechanistic interpretability: the modelโs learned features (e.g., tuning properties, receptive fields, temporal kernels) are not analyzed; links to known visual computations are left unexplored.
- Context contributions: although the model can manipulate population/causal context, quantitative decompositions (e.g., variance explained by stimulus vs. population vs. internal dynamics) are not systematically presented.
- Cross-task synergy: benefits (or trade-offs) of multi-task training versus single-task baselines are not ablated; whether joint training improves any task beyond data pooling remains unknown.
- Robustness and calibration: sensitivity to noise, missing channels, corrupted frames, or session dropouts is not characterized; prediction calibration and failure modes are unreported.
Generalization and transfer
- New-neuron/new-session generalization: per-neuron identity embeddings may limit zero-shot performance on unseen neurons/sessions; the extent of adaptation (e.g., few-shot fitting of new embeddings) required for new recordings is unclear.
- Cross-animal transfer: the model includes animal ID embeddings but does not test zero-shot transfer to entirely unseen animals or strain differences.
- Domain adaptation: no evaluation of fine-tuning/adapter strategies for new datasets (e.g., electrophysiology, different imaging rates) to assess portability.
- Anatomical priors: the model does not leverage anatomical/retinotopic metadata; it is unknown whether such priors could reduce per-neuron parameters and improve generalization.
Scaling behavior and theory
- Saturation origins: parameter scaling saturates beyond ~80M, but the root causes (stimulus diversity limits, label noise, architectural bottlenecks, optimization) are not disentangled.
- Data composition needed for phase transitions: the paper hypothesizes phase transitions with larger/richer datasets but does not specify the scale or diversity required; controlled studies varying stimulus/behavior richness are missing.
- Modality-specific scaling: behavior decoding shows better scaling than neural prediction, but the factors (signal-to-noise, supervision strength, alignment with latent state) are not analyzed.
- Architecture scaling paths: alternatives (e.g., MoE, hierarchical temporal modules, state-space models, recurrent hybrids) to overcome saturation are not explored.
Reproducibility and resources
- Dataset availability: the full dataset is not yet released; reproducibility and external benchmarking are blocked pending release.
- Baseline parity: some baselines cannot support all tasks; matched-ability comparisons and stronger task-specific baselines (e.g., pretrained video encoders for stimulus-encoding) are not exhaustively evaluated.
- Training cost accessibility: large-scale training requires substantial compute; guidelines for efficient smaller-scale regimes or distilled models are not provided.
These gaps provide concrete directions for future work, including broader datasets (regions/species/behaviors), improved temporal and measurement modeling, principled multi-task design, pretrained multimodal encoders, uncertainty-aware objectives, and systematic studies of scaling composition and generalization.
Practical Applications
Below is a synthesized set of practical applications derived from the paperโs findings, methods, and innovations. Each item notes relevant sectors, potential tools/workflows, and key assumptions or dependencies.
Immediate Applications
The following use cases can be deployed now with the released codebase and available mouse visual cortex datasets or with modest adaptation to similar lab datasets.
- Pretrained backbone for mouse visual cortex modeling
- Sectors: academia, software
- What: Use OmniMouse (e.g., 80M) as a frozen or fine-tuned backbone to predict neural responses and decode behavior on new mouse imaging datasets.
- Tools/workflows: Fine-tuning pipelines on lab data; neuron/session/animal identity embeddings for rapid adaptation; evaluation via SENSORIUM-style metrics.
- Assumptions/dependencies: Access to time-aligned neural, stimulus, and behavior data; GPU resources; adequate neuron/ID mapping; suitability of calcium imaging tokenization.
- Unified multi-task workflows within a single model
- Sectors: academia, software
- What: Perform neural forecasting, stimulus-conditioned prediction, sub-population prediction, and behavior decoding in one architecture via masking, reducing the need for separate, task-specific models.
- Tools/workflows: Masking configurations for flexible inference; cross-attention encoder/decoder; Hiera-based video encoder for visual context.
- Assumptions/dependencies: Correct masking setup; synchronized stimuli and neuronal sampling.
- Experimental design and data collection planning using scaling curves
- Sectors: academia, policy (research funders, core facilities)
- What: Use data- and compute-scaling analyses to estimate diminishing returns, plan the number of sessions, and allocate compute efficiently.
- Tools/workflows: โScaling dashboardsโ that plot performance vs. sessions/model size; nested dataset splits; checkpoint schedules for budgeted training.
- Assumptions/dependencies: Similar data heterogeneity and quality; access to enough animals/sessions to realize predicted gains.
- Closed-loop and semi-closed-loop pilot experiments (offline-to-online)
- Sectors: academia
- What: Use the model offline to identify stimuli and contexts that maximize predicted responses or behavioral sensitivity; pilot semi-automated stimulus selection for subsequent online (closed-loop) studies.
- Tools/workflows: In-silico screening of natural vs. parametric stimuli; stimulus-conditioned forecasting to rank candidate stimuli before actual presentation.
- Assumptions/dependencies: Latency/throughput constraints for online use; bridging offline predictions to online protocols.
- Behavioral monitoring augmentation from neural data
- Sectors: academia, animal facilities (welfare)
- What: Decode basic behavioral states (pupil position/size, running) when cameras/sensors fail or to cross-validate noisy behavioral measurements.
- Tools/workflows: Behavior-decoding head and masking that reconstructs behavior traces from neural tokens; alerting when behavior deviates from baseline.
- Assumptions/dependencies: Task/behavior limited to spontaneous behaviors in the dataset; generalization to other behaviors requires retraining.
- Cross-session harmonization and โdigital twinโ alignment
- Sectors: academia, software
- What: Leverage neuron/session/animal embeddings to align recordings across sessions/animals and build digital twins for exploratory analyses.
- Tools/workflows: Identity embeddings and linear projection layers; domain adaptation via fine-tuning readouts.
- Assumptions/dependencies: Stable cell registration strategies (or acceptance of per-neuron embedding reinitialization); consistent preprocessing.
- Benchmarking and reproducibility for vision neuroscience
- Sectors: academia, software
- What: Use OmniMouse and the unified evaluation framework to compare methodologies on standard tasks (forecasting, stimulus encoding, etc.).
- Tools/workflows: SENSORIUM-style single-trial correlation metrics; standardized masking recipes; codebase at the provided repository.
- Assumptions/dependencies: Public dataset release timelines; adherence to benchmark protocols.
- Teaching and training materials for computational neuroscience
- Sectors: education
- What: Use OmniMouse as a teaching exemplar for multi-modal brain modeling, attention mechanisms, and scaling laws.
- Tools/workflows: Tutorials using open code; demo notebooks illustrating masking and multi-task inference.
- Assumptions/dependencies: Studentsโ access to small compute instances; subsampling strategies for classroom-scale datasets.
- Infrastructure planning for neuro-AI labs
- Sectors: academia, research IT, cloud/HPC providers
- What: Use reported compute regimes (warmup-constant LR, checkpoint decay) to plan jobs, storage, and GPU scheduling.
- Tools/workflows: Training recipes; FLOP-based budgeting; multi-stage checkpointing for compute-efficient scaling studies.
- Assumptions/dependencies: HPC/cluster availability and scheduling; cost management.
Long-Term Applications
These applications require additional research, larger and more diverse datasets, domain transfer (e.g., human signals), real-time capability, or engineering to meet deployment constraints.
- Foundation models for human brain signals (EEG/MEG/fMRI/ecog)
- Sectors: healthcare, academia, software
- What: Extend OmniMouseโs multi-modal, multi-task approach to human data for clinical decoding/encoding, personalized neuro-diagnostics, and BCI pretraining.
- Tools/products: โNeuroFMโ foundations that integrate video/audio/task with EEG/MEG/fMRI; transfer learning across patients and sites.
- Assumptions/dependencies: Access to large, diverse human datasets; robust cross-site harmonization; regulatory compliance and privacy.
- Real-time closed-loop experiments and neuromodulation control
- Sectors: academia, neurotech
- What: Use near-real-time neural forecasting and stimulus-conditioned prediction to drive closed-loop visual experiments and to guide optogenetic/electrical stimulation protocols.
- Tools/products: Low-latency inference pipelines; on-device or edge accelerators in imaging rigs; controller policies based on predictive uncertainty.
- Assumptions/dependencies: Reduced-latency tokenization and inference; hardware acceleration; reliability under streaming constraints.
- Data-driven stimulus optimization for vision prosthetics and VR/AR
- Sectors: healthcare, XR, neurotech
- What: Optimize visual inputs that maximize targeted cortical responses, potentially informing visual prosthetics or biologically informed rendering.
- Tools/products: Stimulus optimizers co-trained with brain models; pipelines akin to MEI (most exciting inputs) adapted to multi-modal contexts.
- Assumptions/dependencies: Translation from mouse to human; safety/efficacy studies; integration with implantable device constraints.
- Preclinical platforms for CNS drug and gene therapy screening
- Sectors: pharma/biotech
- What: Predict how compounds or gene therapies alter distributed cortical responses and behavior proxies before full-scale in vivo testing, focusing on dose/target selection.
- Tools/products: In-silico responder analysis; learned biomarkers derived from stimulus-conditioned population predictions.
- Assumptions/dependencies: Datasets with pharmacological perturbations; causal validation linking predictions to outcomes; standardized protocols.
- Autonomous or semi-autonomous โexperimentersโ for stimulus discovery
- Sectors: academia, software
- What: Agents that iteratively propose stimuli/contexts based on model predictions, then refine hypotheses from observed recordings (active learning).
- Tools/products: Bayesian optimization or RL agents coupled to OmniMouse-like predictors; uncertainty-aware selection of next trials.
- Assumptions/dependencies: Robust online inference; guardrails for animal welfare; lab integration.
- Cross-species generalist brain models
- Sectors: academia, software
- What: Build generalist models that share tokenization and masking principles to integrate mouse, non-human primate, and human signals for comparative neuroscience.
- Tools/products: Cross-species identity embeddings; shared visual/audio encoders with species-specific readouts.
- Assumptions/dependencies: Large, diverse multi-species datasets; carefully designed alignment strategies; compute scale.
- Brain-anchored evaluation and training for machine vision systems
- Sectors: software, robotics, AI safety
- What: Use cortical prediction/encoding as an alignment signal or regularizer for vision models (e.g., prefer features predictive of neural responses), potentially improving robustness or interpretability.
- Tools/products: โNeuro-alignmentโ losses; co-training computer vision encoders against neural predictivity; deployment in QA for perception stacks.
- Assumptions/dependencies: Robust brain-model generality across stimuli; demonstration of performance/safety gains over baselines.
- Standards and policy for large-scale brain datasets and compute
- Sectors: policy, funding agencies, research consortia
- What: Develop data-sharing and interoperability standards, scaling benchmarks, and compute access policies to enable phase transitions in brain modeling.
- Tools/products: Benchmark suites (tasks, masks, metrics); governance frameworks for multi-lab datasets; equitable compute allocation programs.
- Assumptions/dependencies: Multi-institutional coordination; ethical oversight for animal and human data; sustained funding.
- Interpretable neuroscience from multi-modal attention mechanisms
- Sectors: academia
- What: Translate cross-attention patterns and context sensitivities (population vs. stimulus vs. causal history) into hypotheses about circuit mechanisms and state dependence.
- Tools/products: Attention analytics toolkits; perturbation analyses over masking configurations; causal probing with designed stimuli.
- Assumptions/dependencies: Advances in interpretability methods; careful experimental validation.
- Green and efficient training for neuro-AI at scale
- Sectors: energy, HPC/cloud, software
- What: Develop energy-aware training recipes (data/model co-scaling, low-precision, sparsity) tailored to per-neuron embedding costs and local attention patterns in brain models.
- Tools/products: Energy/compute optimizers; hardware-software co-design for calcium-imaging tokenization pipelines.
- Assumptions/dependencies: Collaboration with hardware vendors; benchmarking on large-scale datasets to verify efficiency.
Notes on feasibility across applications:
- Model capacity vs. data: The paper shows performance saturates with parameter count (~80M) but improves with more data; many long-term gains depend on larger, richer, and more diverse datasets.
- Species transfer: Most translational or clinical applications require domain transfer beyond mouse visual cortex and more complex behavior labels.
- Real-time constraints: Closed-loop/control applications need significantly optimized inference and hardware integration.
- Interpretability: Biological insights depend on new interpretability tools for transformers in multi-modal neuroscience contexts.
- Data governance: Several applications hinge on open, standardized, and ethically governed datasets that are not yet widely available.
Glossary
- 1D-RoPE: A rotary positional embedding variant applied along a 1D temporal axis to encode relative timing in transformers. "All transformer layers throughout the model (including the encoding and decoding cross attention blocks) use 1D-RoPE \citep{su2024roformer} computed from each token's timestamp, to encode relative timing both within and across modalities."
- behavioral decoding: Inferring behavioral variables (e.g., gaze, running) from neural activity. "We train multi-modal, multi-task models that support three regimes flexibly at test time: neural prediction, behavioral decoding, neural forecasting, or any combination of the three."
- CAIMAN: A toolbox for processing calcium imaging data, including denoising and spike inference. "with spiking activity extracted by CAIMAN \citep{giovannucci2019caiman}."
- causal context: Only past (temporally preceding) inputs provided to predict future neural activity. "while causal context provides temporally preceding activity from all neurons."
- Chinchilla scaling: A prescription that balances model size and dataset size for optimal training efficiency. "\citet{hoffmann2022training} refined this with ``Chinchilla scaling'', prescribing proportional growth of model and data size for optimal efficiency."
- contrastive encoder: An encoder trained with a contrastive objective to map related inputs closer in latent space and unrelated ones farther apart. "CEBRA \citep{schneider2023learnable}, a contrastive encoder, also mapped activity to behavior or stimuli"
- cross-attention: An attention mechanism where a set of queries attends to a separate sequence of keys/values to integrate information across modalities. "we compress this variable-length sequence into a fixed set of latent embeddings via cross-attention with learned queries."
- digital twins: Computational models that serve as virtual replicas of biological systems across animals. "culminating in multi-animal ``digital twins'' that capture biological phenomena beyond training data"
- FLOPs: A measure of computational cost in floating point operations. "model FLOPs, excluding FLOPs of neuron-specific parameters"
- Gabor: A sinusoidal grating used as a parametric visual stimulus to probe receptive fields and orientation tuning. "static and drifting Gabors \citep{Petkov2007-kv}"
- global register tokens: Learned tokens added to transformer inputs to facilitate global information aggregation and avoid attention sinks. "we additionally append global register tokens \citep{darcet2023vision} to the query embeddings, which attend to the entire key sequence."
- Hiera vision transformer: A hierarchical Vision Transformer architecture used for efficient spatiotemporal feature extraction from video. "we use the first ten layers of a randomly-initialized Hiera vision transformer \cite{ryali2023hiera}"
- identity embeddings: Learned embeddings that encode categorical identities (e.g., neuron, session, animal) and are added to token features. "We augment each token with learned identity embeddings for its neuron, session, and animal"
- inverse-square-root learning rate decay: A schedule that decreases the learning rate proportional to the inverse square root of training steps. "using an inverse-square-root learning rate decay, which anneals the learning rate to near-zero"
- latent brain state: A hidden variable representation capturing unobserved or slowly varying neural dynamics. "modeled a latent brain state probabilistically"
- masking configuration: A scheme specifying which inputs are visible (unmasked) and which are targets to be reconstructed. "This chunk is paired with a masking configuration specifying which samples from each modality should be encoded (i.e., unmasked) and which subset of the masked samples should serve as reconstruction targets."
- neural forecasting: Predicting future neural activity from past neural signals (and potentially other context). "allowing the model to handle arbitrary combinations of neural forecasting, stimulus-conditioned response prediction, sub-population prediction, and behavioral decoding."
- neural tokens: Discrete tokenized units representing segments of neural recordings used as model inputs. "totaling more than 150 billion neural tokens recorded during natural movies, images and parametric stimuli, and behavior."
- noise ceiling: The theoretical upper bound on prediction performance imposed by measurement noise. "without any correction for the noise ceiling."
- parametric stimuli: Visual inputs defined by controlled parameters to systematically probe neural responses. "parametric stimuli such as static and drifting Gabors"
- pink noise: A 1/f spectral-density noise stimulus often used to probe temporal response properties. "directional pink noise"
- Poisson loss: A likelihood-based loss appropriate for modeling count-like neural responses under a Poisson assumption. "using Poisson loss (averaged across neurons) for neural encoding"
- population context: Simultaneously recorded neural activity from a subset of neurons provided as input to predict others. "Population context provides contemporaneous activity from a subset of non-target neurons"
- population prediction: Predicting the activity of a set of neurons given observed activity from another subset. "Population prediction. Provided with a population context of unmasked neurons over the full 2-second context window, predict a target population."
- random dot kinematograms: Displays of moving random dots used to study motion perception and neural motion sensitivity. "random dot kinematograms \citep{Morrone2000-ov}"
- scaling laws: Empirical relationships describing how performance scales with model size, data size, and compute. "Large-scale models in language and vision exhibit predictable improvements with scale, described by empirical ``scaling laws''."
- single-neuron tokenization: Representing each neuron's activity as its own token sequence for flexible per-neuron modeling. "We adopt single-neuron, single-time-chunk tokenization"
- single-trial correlation: Pearson correlation computed per trial between predicted and observed signals as an evaluation metric. "we opt for single-trial correlation, i.e. the pearson correlation between ground truth and predicted signal"
- sliding-window attention: Attention restricted to a local temporal window to improve efficiency and enforce locality. "we implement local sliding-window attention"
- stimulus-conditioned forecasting: Forecasting neural activity while conditioning on the visual stimulus. "Stimulus-conditioned forecasting: same as forecasting, but also conditioned on the visual stimulus."
- stimulus-conditioned population prediction: Predicting neural activity from a subset of neurons while also conditioning on the visual stimulus. "Stimulus-conditioned population prediction; same as population prediction ( neurons in population context), with additionally provided context of visual stimulus."
- stimulus-encoding: Predicting neural responses conditioned solely on the presented stimulus. "Stimulus-encoding: Neuronal encoding conditioned on the visual stimulus."
- temporal receptive field: The time window over which a feature or unit integrates information. "Each video embedding is assigned the timestamp from the first frame within its temporal receptive field."
- two-photon calcium imaging: An optical method for measuring population neural activity via calcium-sensitive fluorescence. "recorded via wide-field two-photon calcium imaging"
- upsampling artefacts: Distortions introduced when increasing the sampling rate of signals. "to prevent any upsampling artefacts."
Collections
Sign up for free to add this paper to one or more collections.