PhysBrain: Scalable Multimodal Brain Pipeline

• PhysBrain Pipeline is a modular, scalable framework for automated processing of diverse brain imaging modalities.
• It integrates state-of-the-art algorithms for segmentation, 3D reconstruction, and egocentric video analysis with rigorous validation metrics.
• Its design supports robust HPC scaling, dynamic orchestration, and reproducible results across connectomics and perinatal MRI analyses.

The PhysBrain pipeline encompasses a set of software and methodological frameworks for large-scale, automated brain data processing across multiple modalities, spanning electron microscopy connectomics (Vescovi et al., 2020), real-time 3D intraoperative shape reconstruction (Hu et al., 2021), egocentric embodied intelligence from human videos (Lin et al., 18 Dec 2025), and perinatal MRI segmentation and analysis (Urru et al., 2022). Each instantiation of PhysBrain integrates modular orchestration, state-of-the-art algorithms, and rigorous evaluation on domain-relevant neural data at scale. The following article covers the core pipeline designs and empirical techniques, with explicit reference to implementation, scalability, data integration, and validation metrics.

1. Pipeline Architectures and Modular Stages

PhysBrain is implemented as a modular, multi-stage system adaptable to diverse neuroscientific imaging and intelligence objectives.

1.1. Electron Microscopy Connectomics

• Stage A (Montage): Ingests overlapping 2D EM tiles (e.g., 10833×14000 px, 8-bit) per section, stitched via TrakEM2 in headless, MPI-wrapped execution. Each MPI rank processes a section, generating montaged images (Vescovi et al., 2020).
• Stage B (Alignment & Normalization): Processes montaged stacks using AlignTK (elastic registration), contrast normalization, and artifact thresholding, aligning neighbor section pairs per rank.
• Stage C (Segmentation): Aligned volumes (optionally downsampled in-plane) are segmented via a Flood-Filling Network (FFN) with MPI GPU-offload, and masked using U-Net and 3D watershed. Segmentation is parallelized per overlapping 3D subvolume (e.g., 512×512×128 voxels).
• Stage D (Post-Processing): Subvolume outputs are reconciled and meshed (marching cubes), skeletonized (TEASAR), and exported in Neuroglancer precomputed formats using the Igneous library.
• Stage E (Visualization): 3D reconstructions are visualized directly in Neuroglancer and Jupyter, with data accessed via the Petrel object store at ALCF.

1.2. Real-Time 3D Shape Perception

• Data Preprocessing: Receives a single 2D preprocessed MRI slice; normalization and canonical alignment to 91×109×91 space (Hu et al., 2021).
• Hierarchical Shape Reconstruction: Uses a ResNet-based adversarial branching predictor to map images to Gaussian latent codes, followed by a Tree-GCN that predicts incomplete point clouds, then encodes/decodes via PointNet++ and hierarchical attention blocks (AGBs) to complete the surface geometry.
• Output: Generates a 2048-point cloud representing a plausible, completed 3D brain surface suitable for surgical visualization and guidance.

1.3. Egocentric Embodied Intelligence

• Egocentric2Embodiment Pipeline: Transforms raw first-person videos (Ego4D, BuildAI, EgoDex) into structured, schema-driven VQA tuples at multiple semantic levels (temporal, spatial, mechanical, etc.) (Lin et al., 18 Dec 2025).
• Quality Control: Rule-based validation for evidence grounding and consistency.
• VLM Fine-Tuning: Multilingual VLMs (e.g., Qwen2.5-VL-7B) are fine-tuned on the E2E-3M dataset, producing the PhysBrain model, with downstream action-conditioned learning via diffusion transformers (Flow-Matching loss).

1.4. Perinatal MRI Segmentation

• Preprocessing: Initiates with raw T2-weighted MRI (stacks/volume); fetal scans include U-Net skull stripping and NiftyMIC super-resolution.
• Atlas-Based Label Propagation: Registration to multi-atlas templates and label fusion via locally weighted voting.
• Surface and Feature Extraction: Deformable mesh fitting yields white/pial cortical surfaces for quantitative morphometrics: curvature, thickness, sulcal depth, LGI (Urru et al., 2022).

2. HPC Scaling, Workflow Orchestration, and Performance

High-throughput and reliability are achieved via integrated parallelism, resource scheduling, and execution granularity controls.

Component	Parallelization Unit	Example Throughput / Runtime
Montage (TrakEM2)	Section / MPI rank	8×1128-section stack: 100–520 min @32 nodes
Alignment	Section-pair / MPI rank	16–32 nodes, 1 rank per pair
Segmentation (FFN)	Subvolume / MPI rank + GPU	8.69×10¹⁰ voxels in 72 h on 32×K80 GPUs

• Throughput and Scaling: Formulas applied include strong scaling $S(p)=T(1)/T(p)$, weak scaling efficiency $E(p)=T(1)/T(p)$, and voxel throughput $R=V/T$.
• Orchestration: Balsam database maps each operation to application/job units, permitting dynamic allocation, automatic retries, and workflow steering from CLI or Jupyter front-ends. Example: Montage throughput achieves 194 GB/hr; FFN segmentation reaches $\approx 1.2\times 10^{9}$ voxels/hr (Vescovi et al., 2020).
• Data Lifecycle: Microscope output is staged to compute, and results are exported to cloud object stores for visualization.

3. Algorithmic Components and Software Integration

The pipeline synthesizes diverse community codes via standardized Python operation wrappers and unified data models.

• Integrated Tools:
  • TrakEM2: 2D tile montage (Java macros, MPI wrapping)
  • AlignTK: Nonlinear elastic section alignment
  • FFN: Dense neuron segmentation (TensorFlow, GPU/MPI offload)
  • U-Net & 3D Watershed: Semantic masks for anatomical features
  • Igneous: Mesh and skeleton generation
• Code Modifications: Minimal changes ensure reproducibility and facilitate new module integration by enforcing I/O conventions (HDF5, Neuroglancer precomputed).
• API/I/O Standards: TIFF ↔ HDF5 ↔ precomputed cubes conversion enables data interoperability; no shared global state; all operations are read-immutable, write-unique.
• Error Handling: Balsam orchestrator manages auto-retries, checksum verification, and idempotent task design.

4. Quantitative Validation and Empirical Benchmarks

Comprehensive validation spans geometric fidelity, segmentation accuracy, and embodied intelligence transfer.

• Electron Microscopy:
  • Montage error rates decrease from 35% to 1% as input range widens and runtime increases (e.g., 520 min at 6% error/1% accumulated error) (Vescovi et al., 2020).
  • FFN edge accuracy after fine-tuning: $\approx 0.91$.
• 3D Surface Completion:
  • Hierarchical Shape-Perception Network (HSPN) achieves Chamfer Distance $4.461\times 10^{-1}$ versus baselines and robustly tolerates point dropout and occlusion (Hu et al., 2021).
• Atlas-Based Segmentation:
  • Mean Dice scores: CSF $0.83\pm0.04$, cortical plate $0.85\pm0.03$, white matter $0.90\pm0.02$, outperform reference (dHCP) especially for challenging tissues (Urru et al., 2022).
  • Three-channel (T2+GM+ventricle) registration outperforms single and two-channel variants, with biggest gains in ventricular segmentation.
  • Pipeline runtime reduced to $\approx$15 min/subject.
• Egocentric Embodiment:
  • PhysBrain achieves 64.3% average on EgoThink, outperforming all models on the Planning category (64.5%) and attaining 53.9% mean success on SimplerEnv robot control tasks (Lin et al., 18 Dec 2025).

5. Data Sets, Domain-Specific Resources, and Extensibility

PhysBrain leverages and produces multi-modal, domain-optimized resources.

• Electron Microscopy: 90×125×52 μm³ tissue, 0.4 Tvox (324 GB), >396 GB raw; workflows transfer-learned on Kasthuri11 and downsampled for tractable inference.
• Perinatal Atlases: Spatiotemporal fetal templates (81 subjects, 19–39 GW) with 7-tissue probability maps and 20-subject multi-atlas for structural parcellation; all templates and codebases are public (Urru et al., 2022).
• Egocentric Video: Ego4D, BuildAI, EgoDex, yielding ≈3 M VQA annotation pairs spanning household, factory, and laboratory scenarios.
• MRI Shape Data: In-house 900 brain MRIs for training, point clouds obtained from voxel-level segmentations; diversified by pathology (Alzheimer's, healthy).

This suggests that a major strength of PhysBrain lies in the creation and open sharing of domain-adapted benchmarks, thereby enabling rigorous, reproducible comparison and further pipeline extensibility.

6. Future Directions and Methodological Impact

Limitations recognized in current PhysBrain deployments guide ongoing methodological development.

• Enrichments include expanded egocentric translation to non-domestic and medical scenarios, integration of scene graphs and state trackers for multi-object reasoning, and unified multi-task loss for end-to-end training (Lin et al., 18 Dec 2025).
• For electron microscopy and perinatal imaging, anticipated enhancements involve higher-dimensional, scalable annotation tools and improved mesh/skeleton reconstruction fidelity.
• Potential integration with reinforcement learning and world-model modules for closed-loop policy optimization is identified as a plausible direction.
• A plausible implication is that continued modularization and community-driven contribution, combined with strict I/O contract enforcement, will enable PhysBrain frameworks to permeate new domains in large-scale neuroscience and embodied cognition pipelines.

References:

• (Vescovi et al., 2020)
• (Hu et al., 2021)
• (Urru et al., 2022)
• (Lin et al., 18 Dec 2025)