CytoNet: Foundation Model for Cortex

• CytoNet is a self-supervised foundation model that encodes high-resolution cortical image patches to capture both general architecture and individual cytoarchitectonic details.
• It employs modified ResNet-50 and hybrid ResNet-50–ViT backbones with a novel SpatialNCE loss, achieving high performance in area classification, layer segmentation, and structural prediction.
• The model extends to a vision–language framework that generates natural-language descriptions of cortical microstructures, facilitating interactive microscopy analysis.

CytoNet is a foundation model for the human cerebral cortex that encodes high-resolution microscopic image patches into feature representations for cytoarchitectonic analysis. In its foundation-model formulation, it is presented as a self-supervised system trained on millions of high-resolution histological image patches, using spatial proximity in a common reference space as the central learning signal rather than manual labelling (Schiffer et al., 21 Oct 2025). In a subsequent technical report on a vision–language extension, CytoNet is also described as a “vision foundation” model that can be coupled to a LLM to generate natural-language descriptions of cortical microstructure through weak, label-mediated supervision (Sutton et al., 26 Feb 2026). Across these materials, CytoNet functions as a shared embedding framework for cortical area classification, cortical layer segmentation, structural variation prediction, unsupervised parcellation, and language-grounded microscopy analysis.

1. Research setting and problem formulation

CytoNet is designed for the study of cortical microarchitecture in cell-body–stained histological sections of the human cerebral cortex. The central objective is to represent laminar and areal organization in a compact feature space that remains useful across downstream tasks. The foundation-model report emphasizes that nearby cortical locations tend to share cytoarchitectonic features and therefore provide a natural self-supervision signal, while the vision–language report emphasizes the scarcity of curated image–text pairs in microscopic brain analysis and uses CytoNet as the visual component in a weakly supervised captioning pipeline (Schiffer et al., 21 Oct 2025, Sutton et al., 26 Feb 2026).

The basic input unit in the foundation-model report is a $2048\times 2048$ patch at $2\,\mu\text{m/px}$, corresponding to a $4\,\text{mm}$ field. These patches are sampled along a reconstructed cortical midsurface and registered into MNI Colin 27 space. The resulting embeddings are intended to capture both general aspects of cortical architecture and unique brain-specific traits. The technical report on the language extension describes a related setting in which $\approx 539\,000$ patches are extracted from BigBrain sections, labelled into $57$ target areas or an “unknown” class, and then connected to literature-derived textual statements (Sutton et al., 26 Feb 2026).

A potential misconception is that CytoNet is defined solely as a supervised area classifier. The self-supervised foundation-model description instead frames it as a representation learner whose downstream utility is tested on multiple tasks without requiring manual labels during pretraining (Schiffer et al., 21 Oct 2025). By contrast, the vision–language technical report summarizes CytoNet with a classification-oriented formulation. This suggests either an evolution of the model family or a task-specific re-description of the same research line (Sutton et al., 26 Feb 2026).

2. Architectural formulations

In the foundation-model paper, all experiments use either a modified ResNet-50 (“R50”) or a hybrid ResNet-50–ViT-B (“R50-ViT”) backbone (Schiffer et al., 21 Oct 2025). In R50, the usual $7\times 7$ stride-2 convolution and $3\times 3$ max-pool are replaced by two convolutions—$5\times 5$, stride $4$; $3\times 3$, stride $2\,\mu\text{m/px}$0; $2\,\mu\text{m/px}$1 channels each—plus $2\,\mu\text{m/px}$2 max-pool, with BatchNorm and ReLU after each convolution. In R50-ViT, the $2\,\mu\text{m/px}$3 input is first processed by R50 to a spatial grid of $2\,\mu\text{m/px}$4 feature maps with $2\,\mu\text{m/px}$5 channels, then tokenized into $2\,\mu\text{m/px}$6-dimensional patches and fed into a ViT-B transformer with $2\,\mu\text{m/px}$7 heads, $2\,\mu\text{m/px}$8 layers, a learnable class token, and positional embeddings.

For contrastive pretraining, the backbone output is mapped through a two-layer projection head to a $2\,\mu\text{m/px}$9-dimensional embedding. The head is specified as Linear $4\,\text{mm}$0 BatchNorm $4\,\text{mm}$1 ReLU with the same hidden size, followed by Linear$4\,\text{mm}$2. Backbone features remain $4\,\text{mm}$3-dimensional for R50 and $4\,\text{mm}$4-dimensional for R50-ViT after global average pooling or class-token extraction, whereas the $4\,\text{mm}$5-dimensional projection features are used only during pretraining and removed before downstream probing (Schiffer et al., 21 Oct 2025).

The vision–language technical report gives a different architectural summary of the original CytoNet (Sutton et al., 26 Feb 2026). There, CytoNet is described as taking $4\,\text{mm}$6-pixel patches sampled at $4\,\text{mm}$7 from reconstructed BigBrain histological sections and producing a $4\,\text{mm}$8-dimensional feature vector $4\,\text{mm}$9, called “CytoEmbed,” with $\approx 539\,000$0, together with an optional softmax classification over Julich-Brain area labels with $\approx 539\,000$1 classes. Its high-level architecture is summarized as a Vision-Transformer backbone with $\approx 539\,000$2 non-overlapping patch embedding into $\approx 539\,000$3 tokens, $\approx 539\,000$4 transformer encoder blocks, global average pooling to a $\approx 539\,000$5-D representation, and a one-layer classification head mapping to $\approx 539\,000$6 logits plus softmax.

The same technical report also specifies supervised training objectives: an area-classification loss $\approx 539\,000$7 based on cross-entropy over $\approx 539\,000$8 classes, an optional supervised contrastive loss $\approx 539\,000$9, and a classification-only setting in which the total pretraining loss is $57$0 (Sutton et al., 26 Feb 2026). Within the cited materials, these formulations coexist with the self-supervised SpatialNCE formulation. This suggests that “CytoNet” refers to a model family whose architecture and training objective are adapted to different experimental contexts.

3. SpatialNCE and the self-supervised training regime

The distinctive methodological contribution of the foundation-model paper is SpatialNCE, an InfoNCE-style contrastive loss in which binary positive/negative definitions are replaced by continuous proximity weights (Schiffer et al., 21 Oct 2025). Each image patch inherits a point $57$1 in MNI Colin 27 reference space through $57$2D–$57$3D registration of each histological section. The core assumption is that patches sampled from nearby $57$4D cortical locations share cytoarchitectonic features and should therefore have similar embeddings, whereas distant locations should be dissimilar. For sample $57$5, the loss is

$57$6

with $57$7 and

$57$8

Training is carried out on ten postmortem brains, nine for pretraining and one held out, scanned at $57$9 with cell-body (silver) stain, with approximately $7\times 7$0 of patches per epoch. The cortical midsurface is computed by $7\times 7$1D rigid alignment of neighbors, Laplacian field, skeleton refining, and projection back onto $7\times 7$2D sections. Sampling uses either $7\times 7$3 or $7\times 7$4 random patches of size $7\times 7$5 at $7\times 7$6 along the midsurface (Schiffer et al., 21 Oct 2025).

Augmentations are applied independently per patch. These comprise rotation $7\times 7$7, translation $7\times 7$8, a $7\times 7$9 vertical flip, unbiased gamma intensity augmentation $3\times 3$0 with $3\times 3$1, $3\times 3$2, and $3\times 3$3 from Pohlen et al., plus blur or sharpen with probability $3\times 3$4 using Gaussian $3\times 3$5. Optimization uses LARS-SGD with Nesterov momentum $3\times 3$6 and weight decay $3\times 3$7, batch size $3\times 3$8 with $3\times 3$9 per GPU across $5\times 5$0 GPUs, learning rate $5\times 5$1 held constant for $5\times 5$2 epochs, and runtime of approximately $5\times 5$3 h on $5\times 5$4 A100 GPUs for the $5\times 5$5-sample setting, or approximately $5\times 5$6 GPU-hours (Schiffer et al., 21 Oct 2025).

4. Representation geometry and biological relevance

The learned feature space is reported to be anatomically sound and biologically relevant (Schiffer et al., 21 Oct 2025). UMAP visualizations of backbone features show strong clustering by brain, indicating inter-brain variability, while cortical areas from the Julich Brain Atlas 3.1 form tight subclusters within each brain cluster. The reported Calinski–Harabasz Index is approximately $5\times 5$7 for brain clustering and approximately $5\times 5$8 for area clustering. Averaged cosine-similarity matrices across atlas areas are block-diagonal, with Pearson $5\times 5$9 across subjects, consistent with within-area cohesion and between-area separation.

The same report attributes interpretive value to the transformer attention maps. These maps highlight the stripe of Gennari in V1, Betz cells in M1, and granular layer IV in S1, indicating attention to canonical laminar landmarks. This is consistent with the claim that CytoNet encodes both general cortical architecture and brain-specific idiosyncrasies (Schiffer et al., 21 Oct 2025).

Linear regressions from the $4$0-D CytoNet-ViT features to BigBrain morphological properties further quantify biological relevance. The reported mean $4$1 reaches up to approximately $4$2 for cortical thickness, approximately $4$3 for curvature, approximately $4$4 for cutting angle, and approximately $4$5–$4$6 for layer thicknesses and cell densities. These values are described as substantially higher than profile-based baselines, for which $4$7 is approximately $4$8–$4$9. In the structural-variation task, CytoNet features yield approximately $3\times 3$0 for thickness, approximately $3\times 3$1 for curvature, approximately $3\times 3$2 for cutting angle, approximately $3\times 3$3–$3\times 3$4 for layer thickness, and approximately $3\times 3$5–$3\times 3$6 for cell density, whereas classical intensity profiles yield approximately $3\times 3$7–$3\times 3$8 for thickness and curvature and approximately $3\times 3$9–$2\,\mu\text{m/px}$00 for laminar measures. Feature-importance analysis indicates that top components encode anterior–posterior variation and layer IV density, while other components capture finer cytoarchitectonic variation (Schiffer et al., 21 Oct 2025).

5. Downstream tasks and empirical performance

The foundation-model paper evaluates CytoNet on cortical area classification, cortical layer segmentation, structural variation prediction, and unsupervised region mapping (Schiffer et al., 21 Oct 2025). In cortical area classification, the task is to predict one of $2\,\mu\text{m/px}$01 cytoarchitectonic areas from a $2\,\mu\text{m/px}$02 patch. The evaluation uses seen brains, a transfer brain with no supervised labels in train, and an unseen brain held out from both pretraining and supervised training. Metrics are macro-F1, top-1 accuracy, and top-3 accuracy. For seen brains, CytoNet-ViT $2\,\mu\text{m/px}$03 achieves macro-F1 $2\,\mu\text{m/px}$04, top-1 $2\,\mu\text{m/px}$05, and top-3 $2\,\mu\text{m/px}$06 under linear probing and finetuning. SimCLR performs poorly, with macro-F1 in the range $2\,\mu\text{m/px}$07–$2\,\mu\text{m/px}$08. For the transfer brain, CytoNet-ViT $2\,\mu\text{m/px}$09 yields linear macro-F1 $2\,\mu\text{m/px}$10 and finetune $2\,\mu\text{m/px}$11, while the best transfer is CytoNet-ViT $2\,\mu\text{m/px}$12 finetune $2\,\mu\text{m/px}$13. For the unseen brain, CytoNet $2\,\mu\text{m/px}$14 linear reaches macro-F1 $2\,\mu\text{m/px}$15, and CytoNet-ViT $2\,\mu\text{m/px}$16 finetune reaches $2\,\mu\text{m/px}$17, whereas scratch and SimCLR are approximately $2\,\mu\text{m/px}$18–$2\,\mu\text{m/px}$19. Approximately $2\,\mu\text{m/px}$20 of misclassifications occur at $2\,\mu\text{m/px}$21-hop neighbors in atlas adjacency and $2\,\mu\text{m/px}$22 within $2\,\mu\text{m/px}$23 hops, with logit margin decreasing systematically with error distance.

Cortical layer segmentation uses $2\,\mu\text{m/px}$24 patches downsampled from $2\,\mu\text{m/px}$25 to $2\,\mu\text{m/px}$26, with a $2\,\mu\text{m/px}$27 mask and seven classes: layers I–VI plus background. The dataset contains $2\,\mu\text{m/px}$28 manually labeled patches split $2\,\mu\text{m/px}$29 into $2\,\mu\text{m/px}$30, with training subsets of $2\,\mu\text{m/px}$31, $2\,\mu\text{m/px}$32, $2\,\mu\text{m/px}$33, $2\,\mu\text{m/px}$34, and $2\,\mu\text{m/px}$35. The metric is macro-F1 averaged over layers. Under linear probing, CytoNet-ViT $2\,\mu\text{m/px}$36 reaches $2\,\mu\text{m/px}$37 with $2\,\mu\text{m/px}$38 data, $2\,\mu\text{m/px}$39 with $2\,\mu\text{m/px}$40 data, and $2\,\mu\text{m/px}$41 with $2\,\mu\text{m/px}$42 data. CytoNet $2\,\mu\text{m/px}$43 reaches $2\,\mu\text{m/px}$44, $2\,\mu\text{m/px}$45, and $2\,\mu\text{m/px}$46 in the same settings. SimCLR $2\,\mu\text{m/px}$47 gives $2\,\mu\text{m/px}$48, $2\,\mu\text{m/px}$49, and $2\,\mu\text{m/px}$50, whereas scratch gives $2\,\mu\text{m/px}$51, $2\,\mu\text{m/px}$52, and $2\,\mu\text{m/px}$53. Finetuning is reported to be less stable in low-data regimes, and linear probing remains robust.

In structural variation prediction, the task is regression of cortical thickness, curvature, laminar thickness L1–L6, cutting angle, and layer-wise cell density at each BigBrain sampling point for subject $2\,\mu\text{m/px}$54, using linear regression with $2\,\mu\text{m/px}$55 PCA components and $2\,\mu\text{m/px}$56-fold cross-validation. The comparison against classical intensity profiles shows a consistent advantage for CytoNet-ViT backbone features. In unsupervised region mapping, the case study concerns frontal pole subdivisions Fp1/Fp2 in subject $2\,\mu\text{m/px}$57. Clustering is performed in $2\,\mu\text{m/px}$58-D backbone space without PCA using k-means. In an atlas-based pre-localization setup with $2\,\mu\text{m/px}$59, label purity is $2\,\mu\text{m/px}$60 and hemisphere purity is $2\,\mu\text{m/px}$61. In an annotation-based setup with $2\,\mu\text{m/px}$62, label purity is $2\,\mu\text{m/px}$63 and hemisphere purity is $2\,\mu\text{m/px}$64, and direct assignment yields $2\,\mu\text{m/px}$65 accuracy in matching Fp1 versus Fp2.

The vision–language technical report also cites an original vision-only benchmark from Schiffer et al. 2025 in which $2\,\mu\text{m/px}$66-way area classification on held-out patches reaches accuracy of approximately $2\,\mu\text{m/px}$67 macro-averaged, again indicating high fidelity of the learned cytoarchitectonic embeddings (Sutton et al., 26 Feb 2026).

6. Weakly supervised vision–language extension

The vision–language extension connects CytoNet to language without curated image–text pairs by using weak supervision through shared area labels (Sutton et al., 26 Feb 2026). The pipeline has four stages. First, approximately $2\,\mu\text{m/px}$68 patches are extracted from BigBrain sections, and the CytoNet classifier assigns each patch a label $2\,\mu\text{m/px}$69. Second, literature mining and statement extraction proceed by identifying seed publications through EBRAINS Knowledge Graph maps, expanding through citation crawling with Scopus, filtering by area keywords, downloading full texts, splitting texts into chunks, and prompting Qwen3-Next to extract stand-alone cytoarchitectonic statements such as “Layer IV is distinctly granular, with small round neurons.” Third, synthetic caption generation constructs a pool $2\,\mu\text{m/px}$70 of canonical statements for each area, samples $2\,\mu\text{m/px}$71–$2\,\mu\text{m/px}$72 statements for each patch with predicted label $2\,\mu\text{m/px}$73, and prompts Llama-3-8B-Instruct with the area name and statements to write a concise caption. For $2\,\mu\text{m/px}$74, the system outputs a generic fallback: “Unknown cortical region; no clear laminar pattern.”

The adapter-training stage freezes the CytoNet encoder and the Llama-3-8B LLM, maps frozen $2\,\mu\text{m/px}$75 to $2\,\mu\text{m/px}$76 vision tokens through a small $2\,\mu\text{m/px}$77-layer projection with GELU, inserts gated cross-attention modules every $2\,\mu\text{m/px}$78 transformer layers in Llama-3-8B as in Flamingo, and trains only the projection and cross-attention modules with token-level cross-entropy loss

$2\,\mu\text{m/px}$79

Training runs for $2\,\mu\text{m/px}$80 epochs on $2\,\mu\text{m/px}$81 train, $2\,\mu\text{m/px}$82 validation, and $2\,\mu\text{m/px}$83 test patches, with batch size $2\,\mu\text{m/px}$84, learning rate $2\,\mu\text{m/px}$85, and $2\,\mu\text{m/px}$86 A100 GPUs using FSDP (Sutton et al., 26 Feb 2026).

Two scalable benchmarks quantify performance. In the label-consistency test, the first sentence of each generated caption is parsed to recover the named area label and compared with the reference. The system achieves a $2\,\mu\text{m/px}$87 match for in-scope areas with $2\,\mu\text{m/px}$88 CI $2\,\mu\text{m/px}$89–$2\,\mu\text{m/px}$90, $2\,\mu\text{m/px}$91 correct on out-of-scope “unknown” with $2\,\mu\text{m/px}$92 CI $2\,\mu\text{m/px}$93–$2\,\mu\text{m/px}$94, and overall F1 $2\,\mu\text{m/px}$95. In masked-label discriminability, all area mentions are redacted from the caption and Qwen3-Next must recover the correct area from $2\,\mu\text{m/px}$96 choices, one correct and seven distractors, each annotated by five literature statements. Retrieval accuracy is $2\,\mu\text{m/px}$97 with $2\,\mu\text{m/px}$98 CI $2\,\mu\text{m/px}$99–$4\,\text{mm}$00, well above chance at $4\,\text{mm}$01. The report interprets these results as evidence that the captions both name the correct area and contain discriminative descriptive content (Sutton et al., 26 Feb 2026).

The report also provides representative outputs. For area 4p, the generated caption states that the patch is “characterized by a high density of large pyramidal neurons in layer V, a thin granular layer IV, and diffuse cell packing in layers II–III, reflecting agranular motor cortex structure.” For area 17, the caption emphasizes a “prominent granular layer IV packed with small round cells,” sparse layer V, and an especially thick layer III. For an unknown area, the fallback description states that laminar boundaries and cell density patterns are ambiguous with no distinct granular layer (Sutton et al., 26 Feb 2026).

7. Applications, limitations, and interpretive issues

CytoNet is presented as a common embedded space for comparing cytoarchitecture across individuals while preserving both shared organization and subject-specific idiosyncrasies (Schiffer et al., 21 Oct 2025). The foundation-model report states that it can be combined with spatial smoothing or Bayesian topological constraints to produce continuous brain maps and refine atlases, and that its proximity-based loss generalizes to multimodal data anchored in a common coordinate system, including MRI, receptor density, and gene expression. The vision–language report adds a complementary use case: interactive, natural-language access to microscopy regions in settings where fine-grained paired annotations are scarce (Sutton et al., 26 Feb 2026).

Several limitations are stated explicitly. Performance drops on entirely unseen brains, which suggests a benefit from including each new brain in pretraining even without labels (Schiffer et al., 21 Oct 2025). High-dimensional embeddings mix factors such as cutting angle, thickness, and curvature, so disentangling them could improve atlas reconstruction. Extension to non-cortical structures such as subcortical nuclei would require analogous spatial-proximity assumptions and registration pipelines. In the vision–language setting, open-set deployment is handled by mapping any patch classified outside the $4\,\text{mm}$02 target areas to an “unknown” label and producing a fallback caption rather than forcing an area name, which addresses one practical failure mode of generative systems (Sutton et al., 26 Feb 2026).

A broader interpretive issue concerns the status of CytoNet as a single model versus a family of related formulations. One source describes a self-supervised foundation model trained with SpatialNCE and no manual labelling during pretraining, while another summarizes an original formulation trained on approximately $4\,\text{mm}$03 million patches with dense area annotations and a classification-only objective (Schiffer et al., 21 Oct 2025, Sutton et al., 26 Feb 2026). This suggests that the term “CytoNet” is being used across adjacent but not identical configurations. For research use, the distinction matters: the self-supervised account foregrounds transferable cortical representations, whereas the technical report foregrounds classifier outputs and their reuse in weakly supervised vision–language modeling.