Neural Latents Benchmark

• Neural Latents Benchmark is a standardized framework that integrates diverse datasets and protocols to rigorously evaluate latent variable models in both real neural data and synthetic environments.
• It employs comprehensive metrics such as co-smoothing, PSTH matching, and representational similarity to benchmark models on interpretability, efficiency, and neural alignment.
• The suite supports practical model selection by assessing performance latency and parsimony, while facilitating reproducibility through centralized evaluation protocols.

The Neural Latents Benchmark constitutes a suite of standardized frameworks, datasets, and quantitative protocols for evaluating models that infer, represent, or estimate latent variables in neural systems or neural network models. It addresses the problem of comparability and interpretability of latent variable models (LVMs), with specific implementations targeting neural population activity analysis, visual model brain-alignment, and performance benchmarking in deep learning pipelines. The term "neural latents" spans both theoretical latent processes governing observed neural activity and synthetic ground-truth latent variables defined in controllable environments.

1. Conceptual Foundations and Definitions

At its core, the Neural Latents Benchmark formalizes shared evaluation criteria for LVMs of neural data or model activations, facilitating rigorous quantitative comparison.

In neural population modeling, let $y_{n, t}$ represent the observed spike count for neuron $n$ at time $t$, with latent process $z_t \in \mathbb{R}^D$ ($D \ll N$) governed by a possibly nonlinear and stochastic prior $p(z_{1:T})$ and likelihood $p(y_t | z_t)$. Typical choices include Poisson observation models and dynamical priors such as LDS, GP, or RNN-driven trajectories (Pei et al., 2021).

For synthetic image settings, ground-truth spatial and categorical latents are fully controlled. Each rendered image records $$L = \{X, Y, Z, R^{xy}, R^{yz}, R^{zx}, C^{cat}, C^{id}\}$$: $X, Y$ as translation coordinates, $Z$ as depth, $n$0 as Euler angles, $n$1 as semantic category, and $n$2 as mesh identity (Xie et al., 2024). The network $n$3 estimates a given latent $n$4 via $n$5.

This benchmark suite thus serves both real neural population datasets (where latents are inferred) and fully specified synthetic environments (where latents are supervised).

2. Dataset Curation and Task Structure

Real Neural Data

The NLB ’21 defines four key datasets, all using intracortical Utah-array spike-sorted data from Macaca mulatta across motor (MC, RTT), sensory (Area 2), and frontal (DMFC) cortices. Each dataset is preprocessed via spike sorting, de-correlation, binning (5 ms bins), and then split into train/validation/test, with explicit "held-in" and "held-out" neuron/time assignments (Pei et al., 2021).

Synthetic Latent Datasets

The controlled synthetic benchmarks leverage rendered images from the ThreeDWorld engine, generating up to 100 million images with precise sampling over object types, spatial configuration, lighting, and background. Latent variability can be constrained (e.g., “one-category” or “zero-translation” variants) to systematically assess representation learning (Xie et al., 2024).

The experimental protocols typically involve randomized 80/20 train/test splits, and, for evaluation, a held-out set of thousands of images or trials traversing the full latent space.

3. Model Classes and Training Objectives

Latent Variable Models of Population Activity

Frameworks model neural population activity through generative LVMs. Baselines span from smoothed spike-derived predictors, through Gaussian Process Factor Analysis (GPFA) and Switching LDS, to deep methods including AutoLFADS (variational autoencoder with generator RNN) and Neural Data Transformer (masked transformer self-attention) (Pei et al., 2021).

Training is unsupervised: parameters $n$6 are optimized to maximize marginal or variational log-likelihood: $n$7

Vision Model Alignment

In synthetic latent settings, CNN backbones (ResNet-18/50) are trained with supervised loss functions: MSE for translational and depth latents, sine-cosine regression for rotation, and cross-entropy for categorical variables (Xie et al., 2024). Compound objectives average over subsets of these loss components.

Performance Latency Benchmarking

Model execution is evaluated at the operator level. The Benanza system parses ONNX models, generates per-layer micro-benchmarks, and produces "sequential lower-bound" (sum of individual idealized kernel times) and "parallel lower-bound" (critical path sum for concurrent branches) latency metrics (Li et al., 2019).

4. Evaluation Metrics and Protocols

Co-smoothing and Few-shot Co-smoothing

Primary population activity benchmarks employ co-smoothing: held-out log-likelihood, typically measured in bits per spike (bps) relative to a mean-rate baseline, and computed via: $n$8 where $n$9 denotes Poisson log-likelihood, $t$0 is total spike count (Pei et al., 2021, Dabholkar et al., 2024).

Few-shot co-smoothing further regresses from latent estimates to new held-out neuron/activity data using only $t$1 samples (e.g., via Poisson-GLM), and scores models on their sample efficiency and parsimony—identifying extraneous latent state usage not penalized by conventional co-smoothing (Dabholkar et al., 2024).

PSTH and Behavioral Decoding

Secondary neural benchmarks include matching model-inferred peri-stimulus time histograms (PSTH) and behavioral decoding (R², Pearson correlation) for movement or task variable prediction.

Neural Alignment to Primate Recordings

Vision CNNs are compared to macaque ventral-stream data using Brain-Score pipelines. The key metric is the noise-corrected Pearson correlation between model activations (at the best-predictive layer per brain area) and recorded neural responses, with average scores reported for V1, V2, V4, IT (Xie et al., 2024).

Representational Similarity

Centered kernel alignment (CKA) measures layerwise similarity between representations learned under different training objectives (e.g., category vs. spatial latent estimation). Early and mid-level layers show CKA > 0.9, while late layers diverge (CKA ≈ 0.7–0.8) (Xie et al., 2024).

Latency and Framework Profiling

For execution performance, lower-bound latency metrics ($t$2, $t$3) and the Benanza Ratio ($t$4) diagnose optimization headroom and framework-induced inefficiency (Li et al., 2019).

5. Key Findings and Quantitative Insights

Population Activity Inference

Deep generative models (AutoLFADS, NDT) achieve best overall co-smoothing (bps: 0.33–0.35) and behavioral R² (0.90–0.91) on NLB ’21, outperforming GPFA and SLDS, which offer only minor gains over simple smoothing on some datasets (Pei et al., 2021).

CNN Spatial-Latent Training

CNNs trained to estimate merely a handful of spatial latents (e.g., distance alone: R² ≈ 0.97) achieve neural alignment mean scores ($t$50.399) indistinguishable from category-trained models ($t$60.412, or 0.430 for ImageNet-1K). Performance-neural alignment correlations are high: Corr(R²_dist, neural_mean) = 0.975, Corr(accuracy_cat, neural_mean) = 0.958. Crucially, introducing non-target latent variability in training increases incidental linear-decode accuracy for these factors, promoting convergent representations.

Model Parsimony via Few-shot Metrics

Among top co-smoothing models, few-shot co-smoothing and latent space cross-decoding identify those with minimal extraneous dynamics, yielding a strong negative correlation with extraneousness measures ($t$7 for LFADS/STNDT on mc_maze_20) (Dabholkar et al., 2024). Standard co-smoothing alone is indifferent to parsimony.

Latency Benchmark

Practical adoption of per-layer micro-benchmarking and lower-bound analysis enables speedups of up to 1.95× on ResNet50+V100, revealing potential from parallel execution, improved cuDNN algorithm selection, layer fusion, and proper Tensor Core utilization. The methodology exposes framework inefficiencies such as unnecessary padding or synchronization (Li et al., 2019).

6. Practical Guidelines, Standardization, and Limitations

All NLB datasets are distributed in NWB format via DANDI, with standardized evaluation protocols and leaderboard-based submissions managed through EvalAI. Co-smoothing is mandatory, with secondary metrics (PSTH match, behavioral decoding) encouraged.

For practical use, few-shot co-smoothing should be used as a secondary metric to screen for models with minimal extraneous latent content after standard co-smoothing selection (Dabholkar et al., 2024). Regularization and $t$8-shot size should be empirically cross-validated. For synthetic latent benchmarks, careful manipulation of non-target latent variability is essential to disentangle task-specific from incidental representation learning (Xie et al., 2024).

Limitations include reliance on fixed datasets and channel splits, potential overfitting to benchmark metrics, and, for real neural data, the open question of latent ground-truth interpretability.

7. Future Implications and Benchmark Evolution

The Neural Latents Benchmark continues to serve as an evolving standard for comparing LVMs in neuroscience, computational vision, and performance engineering domains. Key insights include:

• Data diversity in non-target factors is not noise but a driver of convergent representations across tasks.
• Metrics sensitive to both prediction and parsimony (few-shot co-smoothing) are critical for model selection in latent inference pipelines.
• Neuro-inspired model alignment is not unique to "what" (object categorization) objectives; spatial-"where" tasks suffice for high neural fidelity, challenging assumptions about ventral stream optimization (Xie et al., 2024).
• Systematic benchmark protocols are essential for reproducibility and unbiased innovation assessment.

Integration of expanded datasets, additional behavioral modalities, and multimodal correspondence metrics is anticipated for future iterations of the benchmark framework. Practitioners are encouraged to interface with the benchmark using provided APIs (e.g., nlb_tools), submit models for centralized eval, and contribute improvements in latent variable model design, interpretability, and computational efficiency (Pei et al., 2021, Xie et al., 2024, Dabholkar et al., 2024, Li et al., 2019).