FoMo4Wheat: Wheat-Specific Vision Foundation Model

• The paper demonstrates that wheat-specific self-supervised pretraining overcomes limitations of general-domain models, achieving superior performance on ten in-field vision tasks.
• FoMo4Wheat is defined as a crop-domain model leveraging a globally curated ImAg4Wheat dataset of 2.5M images spanning diverse genotypes and environmental conditions to ensure robust feature extraction.
• The model’s architecture integrates a Vision Transformer backbone with lightweight adapters and effective distillation, reducing labeled-data requirements and promoting transferability to other crops.

Searching arXiv for the cited FoMo4Wheat papers to ground the article in current records. FoMo4Wheat is a crop-domain vision foundation model specifically pretrained for in-field wheat perception on ImAg4Wheat, a globally curated collection of 2.5 million high-resolution RGB images gathered over a decade at 30 sites across 10 countries and spanning more than 2,000 genotypes and more than 500 environmental conditions (Han et al., 8 Sep 2025). It was introduced to address the mismatch between general-domain visual pretraining and the structural, phenological, and environmental variability of field crops. In the published evaluation, FoMo4Wheat models consistently outperform state-of-the-art models pretrained on general-domain datasets across ten in-field vision tasks at canopy and organ levels, while also reducing labeled-data requirements and transferring to rice, multi-crop, and crop-and-weed settings (Han et al., 8 Sep 2025). A later study on spike volume estimation positions FoMo4Wheat within a broader wheat model pipeline: in that task, a DINOv2-based Regulated Transformer outperforms a FoMo4Wheat-based Regulated Transformer, and cross-modal distillation from 3D geometry is proposed as a route for injecting organ-level shape priors into wheat-specific 2D representations (Zumsteg et al., 20 May 2026).

1. Motivation and problem setting

FoMo4Wheat was developed against a specific failure mode of general-domain pretraining. The reported motivation is that backbones pretrained on datasets such as ImageNet, LVD-142M, and JFT under-represent plant imagery and do not faithfully capture densely packed, fine canopy structures or the strong appearance variability driven by genotype, phenology, management, and outdoor conditions (Han et al., 8 Sep 2025). In the reported field setting, this domain mismatch yields suboptimal features and unreliable transfer across tasks and sites.

The central claim of the model family is therefore not merely that wheat images constitute another downstream domain, but that wheat perception requires a pretraining distribution aligned with crop structure and field acquisition. The paper further states that only about 5% of ImageNet contains plant images, which is presented as one concrete source of the gap between general visual pretraining and agricultural deployment (Han et al., 8 Sep 2025). FoMo4Wheat addresses this by using wheat-specific self-supervised pretraining to obtain representations that are robust for wheat and transferable to other crops and weeds.

This framing has two implications. First, the model is positioned as a foundation model for agricultural vision rather than as a narrowly task-specific backbone. Second, reliability is treated as a function of domain coverage: diverse sites, phenological stages, and acquisition conditions are part of the pretraining design rather than post hoc stress tests. The paper’s broader conclusion is that domain-specific vision foundation models are a practical pathway toward a universal crop foundation model, termed “FoMo4Crop” in the discussion (Han et al., 8 Sep 2025).

2. ImAg4Wheat: data curation, scale, and downstream task suite

ImAg4Wheat is described as the largest and most diverse wheat image dataset to date, with 2.5 million high-resolution RGB images collected from 2010 to 2024 at 30 sites across 10 countries, including Australia, China, Ethiopia, France, India, Japan, Switzerland, the United Kingdom, and the United States, among others (Han et al., 8 Sep 2025). The dataset covers the full wheat growth cycle from emergence to maturity. Acquisition was performed in breeding and experimental fields using ground-based high-throughput phenotyping platforms, including canopy/plot-level, plant-level, multi-angle, and organ-scale imaging. Ground sampling distance ranges from approximately 0.1 mm to 0.4 mm.

Curation is a substantive part of the contribution. The dataset uses standardized field trial protocols across collaborators and a hybrid-scale random cropping strategy from $512 \times 512$ to $1024 \times 1024$ to balance regional, yearly, and scale diversity while preserving the spatial resolution needed for canopy architecture and organ detail. For leaf counting, labels were checked by multiple annotators. A subset of the public FIP 1.0 time-series dataset was also integrated (Han et al., 8 Sep 2025).

The labeled downstream benchmark suite spans ten tasks across classification, detection, counting, and segmentation. It includes wheat growth stage classification, wheat disease classification, wheat head detection, UAV spike detection for cross-platform testing, wheat leaf tip counting, wheat organ segmentation, rice leaf tip counting, rice organ segmentation, multi-crop segmentation, and crop-and-weed segmentation (Han et al., 8 Sep 2025). The downstream subsets vary substantially in size and annotation form, from image-level labels to per-pixel masks and large instance-level detection datasets.

This breadth matters because FoMo4Wheat is evaluated not only on in-domain wheat perception but also on transfer across species and sensing setups. The inclusion of rice tasks, multi-crop segmentation, and crop-and-weed segmentation is used to support the claim that wheat-specific pretraining can still produce features useful beyond wheat (Han et al., 8 Sep 2025). A plausible implication is that the learned representation captures visual regularities shared by field-grown plants and agricultural scenes, not only wheat-specific texture statistics.

3. Architecture, objectives, and distillation regime

FoMo4Wheat uses a Vision Transformer backbone following DINOv2, with three capacity variants: FoMo4Wheat-G at approximately 1.1B parameters, FoMo4Wheat-L at approximately 300M parameters, and FoMo4Wheat-B at approximately 80M parameters (Han et al., 8 Sep 2025). Input tokenization uses patch size 14. The model prepends a learnable class token and a learnable regression token and adds sinusoidal positional encoding. Transformer blocks are stacked Pre-LN blocks with residual connections, Multi-Head Self-Attention, and FFN.

The attention operator is given as

$$\mathrm{Attn}(X)=\mathrm{Softmax}\left(\frac{(XW_q)(XW_k)^\top}{\sqrt{d_k}}\right)(XW_v),$$

and multi-head attention is

$$\mathrm{MHA}(X)=\mathrm{Concat}(\mathrm{Attn}_1(X),\ldots,\mathrm{Attn}_H(X))W_o.$$

When initialized from DINOv2, the nonlinearity is SwiGLU; when distilling, it is ReLU (Han et al., 8 Sep 2025).

For dense prediction tasks, the backbone is coupled with a ViT-Adapter-style lightweight module per block. This adapter adds detail awareness via sparse cross-attention to a multi-scale feature pyramid at resolutions $1/8$, $1/16$, and $1/32$. Downstream heads are task-specific while the backbone remains frozen: an MLP head for classification, Mask R-CNN for detection, PET for point-based counting, and Mask2Former for segmentation (Han et al., 8 Sep 2025).

The self-supervised pretraining objective combines masked image modeling and distillation of class-token invariance across views:

$$L_{rec}=CE(\mathrm{Teacher}(X_u)_{patch},\mathrm{Student}(X_v)_{patch}),$$

$$L_{dis}=CE(\mathrm{Teacher}(X_u)_{cls},\mathrm{Student}(X_v)_{cls}).$$

The Sinkhorn-Knopp clustering loss used in DINOv2 is down-weighted or omitted for agricultural images with repetitive textures to stabilize training (Han et al., 8 Sep 2025). This is one of the paper’s notable methodological adjustments: agricultural textures are treated not as a benign domain shift but as a training-stability issue requiring objective-level modification.

Distillation is also central to the model family’s scaling strategy. The Giant model is initialized from DINOv2 weights and then trained on ImAg4Wheat. The Large and Base variants are then distilled from the trained Giant under identical inputs, and this consistently outperforms training those smaller models from scratch in wheat organ and multi-crop segmentation (Han et al., 8 Sep 2025).

Variant	Parameters	Training pathway
FoMo4Wheat-G	~1.1B	Initialized from DINOv2, then trained on ImAg4Wheat
FoMo4Wheat-L	~300M	Distilled from trained Giant
FoMo4Wheat-B	~80M	Distilled from trained Giant

Training was performed on a 6-node HPC cluster with 48 NVIDIA A100-80GB GPUs using distributed data parallel and mixed precision. The reported resolution curriculum uses $224 \times 224$ global crops for 5 days and then $1024 \times 1024$0 for 3 days, with the explicit goal of learning small-object details such as spikes and leaf tips (Han et al., 8 Sep 2025).

4. Empirical behavior across classification, detection, counting, and segmentation

FoMo4Wheat is evaluated on ten in-field vision tasks using standard metrics for each task family. In growth stage classification at 100% training data, the Giant model achieves balanced accuracy 93.0 versus 91.3 for the state-of-the-art baseline and mAP 96.8 versus 95.6. In disease classification at 100%, Giant achieves balanced accuracy 92.6 versus 87.6 and mAP 96.5 versus 92.9 (Han et al., 8 Sep 2025). Under label scarcity, the reported advantage widens: with 25% training data, Giant reaches growth-stage balanced accuracy $1024 \times 1024$1 versus $1024 \times 1024$2, and disease mAP $1024 \times 1024$3 versus $1024 \times 1024$4.

In detection, the model is tested both on ground-based wheat head detection and on cross-platform UAV spike detection at previously unseen resolutions. For ground-based imagery, Giant achieves AP/AP$1024 \times 1024$5 of 36.6/74.0 versus 36.0/73.4. On UAV imagery at GSD 0.6 mm, Giant reaches 35.4/80.2 versus 34.2/78.2, and at GSD 1.2 mm, 28.3/56.0 versus 27.4/55.4 (Han et al., 8 Sep 2025). These results are used to support the claim of robustness to acquisition changes and resolution loss.

In counting, the gains are reported as more modest but consistent. For wheat leaf tip counting, Giant achieves MAE/MSE/$1024 \times 1024$6 of 9.19/14.11/0.89 versus 9.51/14.26/0.88, and for rice leaf tip counting, 10.77/14.72/0.91 versus 11.16/15.17/0.90 (Han et al., 8 Sep 2025). The cross-species transfer to rice is important because the model is pretrained solely on wheat.

The largest improvements occur in segmentation. On wheat organ segmentation, Giant reaches mIoU/mAcc of 76.49/91.80 versus 70.57/90.63, with especially strong class-level gains for stems, reported as +15.26 IoU. On rice organ segmentation, Giant achieves 74.54/83.36 versus 69.97/79.50. On multi-crop segmentation, Giant obtains 90.71/94.95 versus 78.36/86.42. On crop-and-weed segmentation, Giant reaches 69.99/81.30 versus 64.45/76.84 (Han et al., 8 Sep 2025). The paper interprets these large gains as evidence that crop-specific pretraining is especially beneficial for dense prediction tasks where fine-grained spatial detail and structural priors dominate.

Reliability analyses reinforce the quantitative findings. The paper reports that PCA and t-SNE visualizations show clearer clustering of organs, growth stages, disease severity and location, and crop versus weed categories for FoMo4Wheat than for DINOv2 (Han et al., 8 Sep 2025). This does not by itself prove superior causal structure in the latent space, but it supports the authors’ interpretation that the representation encodes agricultural semantics more coherently than a general-domain baseline.

5. FoMo4Wheat in relation to organ-level geometry and spike volumetry

A later study on wheat spike volume estimation places FoMo4Wheat within a more specialized organ-level pipeline that combines 3D reconstruction, multi-view image modeling, and knowledge distillation (Zumsteg et al., 20 May 2026). That study explicitly frames its contribution to FoMo4Wheat and wheat foundation/model pipelines. It argues that geometry-aware training can be distilled into 2D multi-view Transformers, yielding image-only inference that is orders of magnitude faster than 3D pipelines while preserving accuracy, and that this strategy directly complements FoMo4Wheat’s vision foundation models by injecting shape priors learned from 3D into wheat-specific 2D representations.

The empirical comparison is notable. In the Regulated Transformer setting for spike volume estimation, a DINOv2-based RT achieves MAE 654.31 mm$1024 \times 1024$7 and $1024 \times 1024$8, whereas a FoMo4Wheat-based RT yields MAE 1028.50 and $1024 \times 1024$9 (Zumsteg et al., 20 May 2026). The same study states that this suggests additional pretraining curricula for FoMo4Wheat: multi-view consistency regulation, cross-modal distillation from point clouds or neural reconstruction methods such as NeRF and 3DGS, and large-scale unlabeled multi-view augmentation to transfer geometry priors into wheat organ-level foundation models.

The methodological core of that work is a hybrid 3D-to-2D framework. A rigid-invariant point cloud network uses distance-based histogram features

$$\mathrm{Attn}(X)=\mathrm{Softmax}\left(\frac{(XW_q)(XW_k)^\top}{\sqrt{d_k}}\right)(XW_v),$$0

with pairwise distances $$\mathrm{Attn}(X)=\mathrm{Softmax}\left(\frac{(XW_q)(XW_k)^\top}{\sqrt{d_k}}\right)(XW_v),$$1, so that features remain invariant to translation and rotation. A multi-view Regulated Transformer then combines per-view Gaussian prediction with a spike-level Transformer encoder and Gaussian NLL supervision. Finally, a multimodal ensemble teacher formed from the 3D student and the RT is distilled back into a purely image-based RT via feature-based or pseudo-label distillation (Zumsteg et al., 20 May 2026).

The reported results show why this relationship matters for FoMo4Wheat. The ensemble achieves MAE 578.09 mm$$\mathrm{Attn}(X)=\mathrm{Softmax}\left(\frac{(XW_q)(XW_k)^\top}{\sqrt{d_k}}\right)(XW_v),$$2 and $$\mathrm{Attn}(X)=\mathrm{Softmax}\left(\frac{(XW_q)(XW_k)^\top}{\sqrt{d_k}}\right)(XW_v),$$3; distilled RTs reduce MAE from 654.31 mm$$\mathrm{Attn}(X)=\mathrm{Softmax}\left(\frac{(XW_q)(XW_k)^\top}{\sqrt{d_k}}\right)(XW_v),$$4 for the non-distilled RT to 639.93 mm$$\mathrm{Attn}(X)=\mathrm{Softmax}\left(\frac{(XW_q)(XW_k)^\top}{\sqrt{d_k}}\right)(XW_v),$$5 and 644.62 mm$$\mathrm{Attn}(X)=\mathrm{Softmax}\left(\frac{(XW_q)(XW_k)^\top}{\sqrt{d_k}}\right)(XW_v),$$6, while increasing correlation from 0.76 to 0.77 and 0.82, respectively. Inference time falls from approximately 160 ms per spike for 3D or ensemble inference to 1.38 ms per spike for image-only RT inference (Zumsteg et al., 20 May 2026). The study also reports that distillation mitigates volume-dependent bias and reshapes the latent representation toward a geometry-aware shape.

In the context of FoMo4Wheat, the significance is not that the foundation model fails in general, but that canopy-scale or broadly wheat-specific pretraining does not automatically encode organ-level 3D shape in a form sufficient for volumetry. The later paper’s wording is careful: general canopy-level pretraining, even when crop-specific, may not sufficiently encode organ-level 3D shape (Zumsteg et al., 20 May 2026). This suggests that a next stage of wheat foundation modeling may require explicit geometry-aware supervision or distillation rather than larger RGB-only pretraining alone.

6. Limitations, deployment constraints, and future directions

FoMo4Wheat’s limitations are partly computational and partly modal. Even the Base model at approximately 80M parameters is described as heavy for real-time on-device inference, and the authors explicitly identify compression, quantization, and distillation as necessary for lighter field-ready models (Han et al., 8 Sep 2025). The pretraining corpus is RGB-only, and the paper states that adding modalities such as multispectral or hyperspectral data, LiDAR, and meteorology could further improve performance and generalization.

The scope of generalization is also treated cautiously. Although the model transfers to rice and to crop-and-weed segmentation, the authors argue that a universal cross-crop foundation model would require larger and more diverse multi-crop datasets, new strategies to balance specialization and generalization, and more compute (Han et al., 8 Sep 2025). This is consistent with the model’s design philosophy: reliability is tied to domain coverage, and broad coverage is expensive.

The spike-volume study adds a second set of limitations centered on geometry acquisition and phenological variation. Its data come from one research station across two seasons, and later sampling dates introduce more bending and color variation, increasing MAE. The paper also notes that rigid-invariant features assume shape preservation under rigid motions even though spikes can bend and deform, and it proposes deformable shape priors such as object-centric 3D Gaussian splatting or DeepSDF-like shape families as complementary directions (Zumsteg et al., 20 May 2026). Predominantly nadir views produce partial field point clouds, so additional oblique cameras or temporal aggregation may strengthen geometry supervision.

Taken together, these two papers define a research trajectory rather than a closed system. FoMo4Wheat establishes that globally curated, crop-specific self-supervision yields robust and transferable agricultural vision features (Han et al., 8 Sep 2025). The later spike-volumetry work indicates that organ-level phenotyping may benefit from extending that paradigm with multi-view regulation, epipolar-consistent pairing, geometry-aware distillation, and exploitation of large unlabeled multi-view field datasets (Zumsteg et al., 20 May 2026). A plausible implication is that future wheat foundation models will be hybrid systems in which crop-domain pretraining supplies broad semantic robustness, while cross-modal training injects explicit geometric priors needed for organ-level quantitative traits.