3D Reconstruction and Knowledge Distillation to Improve Multi-View Image Models to Explore Spike Volume Estimation in Wheat

Abstract: Accurate estimation of wheat spike volume is important for yield component analysis and stress resilience assessment, yet field-based measurement remains challenging. Active 3D sensing methods such as Light Detection and Ranging (LiDAR) or time-of-flight (ToF) are sensitive to plant motion or poorly suited to outdoor conditions, while 3D reconstructions are computationally expensive. Direct 2D image processing would offer computational advantages, but image-based models lack explicit geometric information. We therefore propose a hybrid 2D-3D approach with knowledge distillation during training while enabling efficient image-only inference. First, we train a rigid-invariant point cloud network using distance-based histogram features to obtain pose-robust geometric representations. We then combine the 3D model with a proposed multi-view image-based regulated Transformer (RT) in an ensemble architecture. Finally, we distill the ensemble knowledge into a purely image-based student model using either feature-based or label-based distillation. The two distilled RTs reduce the mean absolute error (MAE) from 654.31 mm$^3$ of the non-distilled RT to 639.93 mm$^3$ and 644.62 mm$^3$, and increase correlation from 0.76 to 0.77 and 0.82, respectively. At the same time, inference time is reduced from 160 ms to 1.4 ms per spike. Distillation further mitigates volume-dependent bias and reshapes the latent representation of the image model toward a geometry-aware shape. Our results demonstrate that 3D-informed training of a 2D Transformer allows for scalable and efficient spike volume estimation for high-throughput field phenotyping.

• The paper introduces a multi-stage hybrid framework leveraging multi-view 2D imaging and 3D reconstructions with knowledge distillation for accurate wheat spike volume estimation.
• The approach integrates a regulated Transformer and a rigid-invariant PointNet, achieving a low MAE (578.09 mm³) and high correlation (0.83) compared to traditional methods.
• Knowledge distillation reduces computational cost and volume bias, enabling efficient high-throughput crop phenotyping even in challenging field conditions.

Hybrid 3D-to-2D Deep Learning for In-Field Wheat Spike Volume Estimation

Motivation and Problem Setting

Accurate, high-throughput measurement of wheat spike volume is essential for quantitative phenotyping in plant breeding, crop yield analysis, and the assessment of stress resilience. Traditional field-based 3D sensing technologies, such as LiDAR and ToF, are handicapped in outdoor conditions due to interference from ambient light, plant motion, and significant computational overhead. Pure 2D image-based models, while computationally efficient, typically lack explicit geometric context, resulting in high model bias and poor correlation for volume regression—especially for irregular or occluded structures like wheat spikes.

This work presents a multi-stage hybrid framework that leverages multi-view 2D imaging, field and indoor 3D point cloud reconstruction, and knowledge distillation (KD) techniques to achieve accurate and scalable wheat spike volume estimation in challenging field conditions. The approach allows geometric information from 3D reconstructions to inform efficient, image-only inference via a regulated Transformer (RT), substantially reducing computational cost without sacrificing accuracy.

Figure 1: Overview of the proposed hybrid pipeline combining 2D multi-view imaging, 3D scan supervision via knowledge distillation, ensemble modeling, and image-only efficient RT inference.

Data Acquisition and Pre-processing

Volume-labeled datasets were generated by imaging 1134 tagged wheat spikes from 93 genotypes at different developmental stages and across two years. A custom phenotyping rig equipped with 12 synchronized, nadir-positioned RGB cameras ensured simultaneous multi-view coverage (Figure 7a), with precise genotype-level data aggregation (Figure 7b). Ground truth spike volumes were obtained from high-resolution indoor 3D scanned meshes. Rigorous metric calibration aligned multi-camera field geometries with indoor scans, enabling consistent cross-modal supervision.

Figure 2: Field sensor with 12 RGB cameras for simultaneous multi-view capture (a), and a genotype plot with tagged spikes (b).

Image pre-processing included spike region detection with YOLOv11, instance segmentation using SAM-derived masks, cropping, rotation, and feature extraction via backbone CNN and Vision Transformer models (DINOv2, DINOv3, ResNet, and FoMo4Wheat). To resolve correspondence across views, an epipolar-consistency graph-based multi-view region matching procedure created precise instance-level clusters. After camera calibration and point triangulation, dense field-based point clouds were obtained using OpenMVS; indoor 3D meshes provided high-fidelity reference point clouds.

Model Architecture and Training Paradigm

Baselines and Backbone Selection

Several 2D architectures were compared for image-to-volume regression, including linear models, LSTMs, and a standard Transformer. Among these, DINOv2-based backbones paired with the RT achieved the lowest mean absolute error (MAE) and competitive correlation, outperforming domain-specific (FoMo4Wheat) and deep CNN variants.

Regulated Transformer (RT)

The RT advances over standard Transformer designs by adding a single-image prediction branch to the multi-view encoder setup (Figure 3). Image features extracted via a frozen backbone are fed to two heads: a per-image MLP-based regressor and a global, spike-level Transformer encoder with a learnable volume token. Single-view supervision encourages the emergent representation to encode volume-relevant features at both local and global levels, improving pose robustness and data efficiency.

Figure 3: Regulated Transformer architecture with both single-image (blue) and global, multi-view (red) volume regressors.

Rigid-Invariant PointNet

A specialized PointNet-based network was developed for 3D point cloud-based volume regression (Figure 4). To eliminate pose-variance and leverage superior geometry from indoor scans, per-point histogram feature vectors encoded pairwise distance distributions—explicitly enforcing invariance to rigid transformations. A teacher-student KD paradigm aligned latent features from high-resolution, pose-randomized indoor scans to student models trained on partial, noisy, field-derived point clouds.

Figure 4: Rigid-invariant PointNet using distance-based histograms for pose-invariant feature extraction and feature alignment via knowledge distillation for field-based point clouds.

Ensemble Model and Knowledge Distillation

After independent training, the best-performing RT and rigid-invariant PointNet were fused in an ensemble by concatenating their respective latent representations. A two-layer MLP regressor then predicted spike volume from this joint embedding. This model served as the teacher for two KD strategies distilling knowledge into an image-only RT:

• Feature-based KD: Direct feature matching between the ensemble latent embedding and linearly projected RT representations.
• Bootstrapped pseudo-label KD: Ensemble-generated volume pseudo-labels on 5000 additional unlabeled examples (from new clusters) augmented the RT training set, enabling unlabeled data utilization.

  Figure 5: Overview of the training and distillation pipeline: 3D point cloud and image models are trained independently, fused in an ensemble, and then used for knowledge distillation into a pure image-based RT.

Empirical Results and Analysis

Comparative Performance

The rigid-invariant PointNet outperformed a Point Transformer baseline, demonstrating the efficacy of rigid-invariant histogram features and KD from full scan supervision. The ensemble model combining RT and PointNet achieved the lowest MAE (578.09 $mm^3$), highest correlation (0.83), and lowest MAPE (13.19%).

Both feature- and pseudo-label distilled RTs surpassed the non-distilled image-only RT baseline (MAE reduction from 654.31 $mm^3$ to 639.93 $mm^3$ and 644.62 $mm^3$, and correlation increases from 0.76 to 0.77 and 0.82, respectively). Importantly, the image-only RT inference time is dramatically lower (1.4 ms vs. 160 ms per spike) compared to any point cloud or ensemble approach.

Figure 6: Correlation analysis showing that knowledge distillation reduces volume-dependent bias in MAE compared to the non-distilled RT.

Distillation also reduced volume-dependent bias in estimation errors. The pseudo-label distilled RT maintained MAE stability across sampling dates, with superior robustness to morphological variation.

Latent Space Structure

PCA visualization of spike-level embeddings reveals that rigid-invariant point cloud, ensemble, and distilled RTs all form low-dimensional manifolds aligned with true volume, while the non-distilled RT produces noisier latent structures. Distillation coerces the RT’s representations toward a more geometry-aware embedding, promoting global shape alignment.

Figure 7: PCA of learned embeddings across models; distillation organizes RT latent structure into geometry-aware manifolds.

Practical Implications and Theoretical Developments

This hybrid 3D-to-2D strategy enables high-throughput crop phenotyping by decoupling geometrically informed training from inference, allowing application at scale. The modular use of feature-based and pseudo-label distillation supports training with minimal high-quality 3D scan supervision, enabling practical deployment even when full measurement campaigns are infeasible.

The regulated Transformer architecture, with its per-image regularization, provides new inductive bias for volume regression tasks where cross-view consistency is required but 3D reconstruction is impractical at inference. Future research could:

• Integrate learned shape priors or pose-optimization at test time to further boost robustness for incomplete or occluded field observations.
• Explore memory-based distillation strategies to transfer richer geometric priors from 3D to 2D models, compensating for missing modalities.
• Extend to growth-stage-aware models for improved generalization across developmental diversity.

Conclusion

The work establishes a rigorous pipeline for efficient, non-destructive wheat spike volume estimation by leveraging regulated Transformers informed by 3D geometry during training, but requiring only multi-view images at inference. The use of knowledge distillation—both feature-based and pseudo-label supervision—enables efficient and robust transfer of geometric context, closing the gap between 3D model accuracy and 2D model scalability. The framework has broad implications for plant phenotyping, agricultural monitoring, and any task coupling multi-modal geometric supervision with practical inference constraints.