---
title: 'PlantPose: Dual Agricultural Vision Systems'
url: https://www.emergentmind.com/topics/plantpose
type: topic
---

# PlantPose: Dual Agricultural Vision Systems

PlantPose is the name used for two distinct computer-vision systems in recent agricultural and plant-science literature. One work, written as **PLANTPose**, addresses **category-level 6D object pose estimation in agricultural settings from RGB images only** by jointly estimating rigid pose and mesh deformation relative to a single category-level base mesh [2505.24636]. The other, written as **PlantPose**, addresses **single-image plant skeleton estimation** by generating an arbitrary **tree graph** under explicit botanical topology constraints through the integration of a graph generator, a minimum spanning tree (MST) solver, and a differentiable reparameterization layer [2605.17773]. The shared name can therefore denote either a pose-estimation framework for deformable produce instances or a universal plant-structure estimator for branching skeletons; disambiguation depends on whether the target output is a 6D rigid pose with deformation parameters or a 2D tree graph.

## 1. Dual usage of the name

In the 2025 pose-estimation work, PLANTPose is introduced as a framework for **category-level 6D object pose estimation** in agriculture, motivated by the observation that fruits and vegetables exhibit high intra-class variability in shape, size, and texture, while many existing methods rely on instance-specific CAD models or depth sensors [2505.24636]. Its central representation is a rigid transform in $SE(3)$ together with a learned deformation of a category-level base mesh.

In the 2026 skeleton-estimation work, PlantPose is introduced as a **universal plant skeleton estimator** for single images, motivated by the fact that plant skeleton estimation differs from human pose estimation because plant topology is not fixed and must instead be inferred as an arbitrary tree graph [2605.17773]. Its output is a graph $G=(V,E)$, where $V$ denotes 2D node coordinates and $E$ denotes undirected edges, under the hard tree constraints of connectivity, acyclicity, and $|E|=|V|-1$.

A concise comparison is useful because the two systems address different problem classes.

| System | Core task | Output |
|---|---|---|
| **PLANTPose** [2505.24636] | Category-level 6D object pose estimation in agriculture from RGB images only | Rotation $R$, translation $t$, and deformation parameters relative to a base mesh |
| **PlantPose** [2605.17773] | Single-image plant skeleton estimation via tree-constrained graph generation | Tree graph $G=(V,E)$ with 2D nodes and undirected edges |

This suggests that “PlantPose” functions less as a single method family than as a coincident label for two separate lines of agricultural vision research: one geometric and instance-adaptive, the other structural and topological.

## 2. PLANTPose as category-level 6D object pose estimation

PLANTPose formulates the agricultural pose-estimation problem as **RGB-only category-level 6D object pose estimation without instance-specific CAD models** [2505.24636]. The stated goal is to use a **single base mesh per category** and adapt it to unseen instances via learned deformations, thereby enabling pose estimation across large intra-class variation in shape, size, and texture.

The method defines 6D pose as an element of $SE(3)$ parameterized by rotation and translation:
- $R \in SO(3)$, a $3 \times 3$ orthonormal rotation matrix,
- $t \in \mathbb{R}^3$, a translation vector in the camera frame.

PLANTPose jointly estimates $(R,t)$ and **per-instance mesh deformation parameters** relative to a category base mesh. The deformation model is a **cubic B-spline on a $2 \times 2 \times 2$ control lattice aligned with the base mesh bounding box**, with the network predicting a **24D vector** corresponding to **8 corners $\times$ 3 coordinates** [2505.24636]. This deformation space is intended to adapt a single category-level CAD model to novel instances such as bananas with diverse curvature and thickness.

The architecture comprises a detection-and-cropping stage, a backbone, and three heads. During training, object crops are defined by 2D projections of ground-truth vertices; during inference, a **YOLOv11 detector** finds the object ROI, which is then cropped and resized. The backbone is a **small ViT encoder (30.9M parameters, 32×32 input patch size), pretrained on ImageNet-21k**. The heads predict:
- a **6D rotation representation** later orthonormalized into a valid $R \in SO(3)$,
- a **3D translation vector** in the cropped image frame,
- the **24D lattice corner offsets** for deformation [2505.24636].

PLANTPose is trained with three mean-squared-error terms: a rotation loss, a deformation loss, and a **2D projection loss** based on the perspective projection of deformed, rotated, and translated vertices into the cropped image frame. The overall objective is
$$
L = \lambda_r \lVert r_{6d}^{pred} - r_{6d}^{gt} \rVert^2 + \lambda_d \lVert \delta^{pred} - \delta^{gt} \rVert^2 + \lambda_p \lVert Proj(P \mid R,t,\delta) - v_{2d}^{gt} \rVert^2,
$$
with empirically set weights $\lambda_r,\lambda_d,\lambda_p$ [2505.24636].

## 3. Deformation model, disentanglement, and synthetic-data pipeline in PLANTPose

The deformation model begins from a **category-level reference mesh** and places a tight lattice bounding box around it. Let
- $box_{min}=(min_x,min_y,min_z)$,
- $box_{max}=(max_x,max_y,max_z)$
denote the bounding-box corners of the base mesh. For a vertex $p=(p_x,p_y,p_z)$, normalized coordinates $u,v,w \in [0,1]$ are computed as fractional distances along the $x$, $y$, and $z$ axes within the box [2505.24636].

The lattice has eight corner offsets $C_{i,j,k} \in \mathbb{R}^3$ for $(i,j,k)\in\{0,1\}^3$, which are clamped or extended to a $4 \times 4 \times 4$ grid to ensure correct boundary behavior with cubic B-splines. The 1D cubic B-spline basis functions are
$$
W_0(t)=\frac{1-3t+3t^2-t^3}{6}, \quad
W_1(t)=\frac{4-6t^2+3t^3}{6},
$$
$$
W_2(t)=\frac{1+3t+3t^2-3t^3}{6}, \quad
W_3(t)=\frac{t^3}{6}.
$$
The vertex displacements are then
$$
\delta_x(u,v,w)=\sum_{i=0}^3 \sum_{j=0}^3 \sum_{k=0}^3 W_i(u)W_j(v)W_k(w)C_{i,j,k}^x,
$$
$$
\delta_y(u,v,w)=\sum_{i=0}^3 \sum_{j=0}^3 \sum_{k=0}^3 W_i(u)W_j(v)W_k(w)C_{i,j,k}^y,
$$
$$
\delta_z(u,v,w)=\sum_{i=0}^3 \sum_{j=0}^3 \sum_{k=0}^3 W_i(u)W_j(v)W_k(w)C_{i,j,k}^z,
$$
with displacement vector $\delta(u,v,w)=[\delta_x,\delta_y,\delta_z]$ and deformed vertex
$$
p' = p + \delta(u,v,w).
$$
The paper states that deformation magnitude is constrained to an empirical upper bound to avoid extreme or unhelpful shapes, and that B-splines provide $C^2$ continuity, yielding smooth deformations appropriate for organic produce [2505.24636].

A specific issue addressed by the method is **pose–deformation disentanglement**. Random lattice perturbations can introduce unintended global rotation or translation when cubic B-splines are applied. To prevent deformation from leaking rigid motion, the method estimates a similarity alignment between the original and deformed meshes using **Umeyama**,
$$
(s,R,t)=Umeyama(P_{original},P_{deformed}),
$$
with $s$ fixed to $1$, and then corrects the lattice so that shape change remains independent of pose [2505.24636].

The training data pipeline is synthetic-first. A **BlenderProc-based renderer** generates scenes with procedural rooms, textured surfaces, distractor objects, varied ambient and directional lighting, physically simulated placement, and multiple camera views with fixed intrinsics meant to emulate a real camera. For each rendered image, the annotations include ground-truth $(R,t,\delta)$ and projected vertices [2505.24636].

To narrow the synthetic-to-real gap, the method applies **Stable Diffusion inpainting with ControlNet depth conditioning** only to the object of interest while preserving the background. Curated prompts are used for **unripe, ripe, rotten, and natural color/texture variations** so as to mimic agricultural appearance changes due to lifecycle and environment [2505.24636]. The paper states that this post-render augmentation improves realism while preserving pose.

## 4. Empirical evaluation, ablations, and limitations of PLANTPose

The reported training set contains **4,000 synthetic scenes**, each with a random lattice deformation and **3 images per scene** from varied viewpoints. The real benchmark is an **in-house banana dataset with 6 distinct bananas** covering shapes and ripeness, with **100 RGB images** captured using an **Intel RealSense D435**. Each banana was **3D scanned using an iPhone 14 Pro LiDAR** to obtain real meshes; ground-truth alignment to the base mesh is performed via ICP, and deformation error is computed after removing pose [2505.24636].

Evaluation uses:
- **Chamfer distance (mm)** between scanned meshes and predicted deformed meshes,
- **rotation error (degrees)** as mean and median,
- **translation error (mm)** as mean and median,
- **deformation error (mm)** after removing estimated rotation and translation via ICP [2505.24636].

On the 100-image banana benchmark, the RGB-only baseline **MegaPose** is evaluated using the same single banana mesh for all test images. The reported results are:

| Method | Chamfer | Rotation error | Translation error |
|---|---:|---:|---:|
| MegaPose | 90.1 mm | mean 52.4°, median 43.6° | mean 59.9 mm, median 45.6 mm |
| PLANTPose | 59.8 mm | mean 32.6°, median 23.6° | mean 42.5 mm, median 37.9 mm |

The corresponding **deformation error** is **29.1 mm** for MegaPose and **12.1 mm** for PLANTPose [2505.24636]. The paper attributes these improvements to the ability to adapt the mesh shape via lattice deformation while jointly estimating pose.

Ablation results isolate the effects of **Stable Diffusion augmentation** and **Umeyama correction**. The full model reports **Chamfer 59.8 mm**, **rotation mean/median 32.6°/23.6°**, **translation mean/median 42.5/37.9 mm**, and **deformation 12.1 mm**. Without Stable Diffusion, results become **71.5 mm**, **37.7°/23.0°**, **48.9/48.7 mm**, and **13.9 mm**. Without Umeyama, results become **73.1 mm**, **36.5°/26.2°**, **50.3/45.8 mm**, and **12.7 mm**. Removing both yields **89.7 mm**, **36.1°/26.7°**, **61.1/61.9 mm**, and **26.5 mm** [2505.24636]. The observations reported in the paper are that removing Stable Diffusion increases Chamfer and translation errors, while removing Umeyama notably degrades translation, confirming that cubic B-spline perturbations can leak pose.

The paper also states several limitations. Performance depends on **base mesh representativeness**; extremely atypical instances may remain difficult. Stable Diffusion improves realism but is not perfect, and in rare cases inpainting may slightly alter appearance. Real agricultural settings may involve severe occlusions and foliage, and **RGB-only methods are susceptible to poor visibility**. In addition, the current formulation uses **a separate model per category**, with multi-category conditioning left as future work [2505.24636].

## 5. PlantPose as universal plant skeleton estimation

The 2026 PlantPose work addresses a different problem: estimating a plant’s skeleton from a single image as a **tree graph** rather than as a fixed-topology keypoint set [2605.17773]. The formal task is: given an image $I$ of a plant, estimate a graph $G=(V,E)$ where $V$ is the set of 2D node coordinates, $E$ is the set of undirected edges connecting pairs of nodes, and the resulting graph is connected and acyclic with $|E|=|V|-1$.

The objective is expressed as
$$
G=(V,E)=\Phi(I) \text{ subject to } E \in E_{tree}(V),
$$
where $\Phi$ is the learned image-to-graph predictor and $E_{tree}(V)$ denotes edge sets forming a tree over nodes $V$ [2605.17773].

For each ordered node pair $(i,j)$, the network predicts a binary edge probability vector
$$
y_{(i,j)} = [y^+_{(i,j)}, y^-_{(i,j)}]^\top,
$$
with $\lVert y_{(i,j)} \rVert_1 = 1$, derived by softmax from edge logits
$$
f_{(i,j)} = [f^+_{(i,j)}, f^-_{(i,j)}]^\top.
$$
An unconstrained edge set is formed as
$$
\hat{E} = \{ (i,j) \mid y^+_{(i,j)} > y^-_{(i,j)} \}.
$$
A non-differentiable projection $\Pi$ then enforces the tree constraint, for example by computing the MST of the complete graph defined on $V$ with edge costs
$$
c_{(i,j)} = y^-_{(i,j)}.
$$
This yields the constrained graph
$$
(V,E) = \Pi(V,\{y_{(i,j)}\}).
$$
The work emphasizes that plant skeleton estimation requires handling **arbitrary tree graphs**, which distinguishes it from human-pose settings with fixed topology [2605.17773].

Architecturally, PlantPose builds on **RelationFormer**, described as a state-of-the-art non-autoregressive image-to-graph model. It uses a **deformable DETR-style transformer encoder-decoder over ResNet-50 features**, with decoder queries consisting of **[obj]-tokens** for node hypotheses and a single **[rtn]-token** for global relational context. A node detection head predicts node positions from the [obj]-tokens, and an MLP with layer normalization predicts pairwise edge logits [2605.17773].

## 6. Tree-constrained graph generation, SFS, data curation, and evaluation in PlantPose

The key technical feature of PlantPose is the incorporation of **Kruskal’s MST** into the training loop together with the **Selective Feature Suppression (SFS) layer** [2605.17773]. After the unconstrained prediction stage, the model builds the complete graph on the predicted nodes using edge costs $c_{(i,j)}=y^-_{(i,j)}$, then runs Kruskal’s MST with union-find cycle prevention to obtain constrained edges $E$.

The method defines:
- $E^+ = E \setminus \hat{E}$, the edges added by MST,
- $E^- = \hat{E} \setminus E$, the edges removed by MST.

Because MST is non-differentiable, SFS modifies logits so that the forward pass agrees with the MST decision while gradients still flow. The rules are:
- if $(i,j)\in E^+$, set $f^-_{(i,j)} := -\Lambda$,
- if $(i,j)\in E^-$, set $f^+_{(i,j)} := -\Lambda$,
- otherwise leave $f_{(i,j)}$ unchanged,
with $\Lambda>0$ large [2605.17773].

After softmax on the modified logits, the probabilities become approximately
- $y_{(i,j)} \approx [1-\epsilon,\epsilon]^\top$ for $(i,j)\in E^+$,
- $y_{(i,j)} \approx [\epsilon,1-\epsilon]^\top$ for $(i,j)\in E^-$,
where
$$
\epsilon = \frac{\exp(-\Lambda)}{\exp(f^{kept})+\exp(-\Lambda)} \approx 0.
$$
The paper explicitly states that **no Gumbel-Softmax or relaxation is needed**, because SFS is a deterministic feature-level reparameterization [2605.17773].

The edge loss uses cross-entropy on both unconstrained and constrained probabilities:
$$
L_{CE}(y,t) = -t^+ \log y^+ - t^- \log y^-,
$$
$$
L_{edge} = \sum_{(i,j)} L_{CE}(y^{unconst}_{(i,j)}, t_{(i,j)}) + \sum_{(i,j)} L_{CE}(y^{const}_{(i,j)}, t_{(i,j)}),
$$
and the total loss is
$$
L_{total}=L_{node}+L_{edge}.
$$
The paper’s interpretation is that hard constraints are enforced by MST, while SFS makes the forward pass differentiably consistent with MST so that learning uses the tree-constrained outputs [2605.17773].

Training uses the official RelationFormer code in PyTorch, with **$\Lambda=10$ during training**, **early stopping after 30 epochs without improvement**, and training times on **8× NVIDIA RTX A100 GPUs** of approximately **141 h (synthetic)**, **10 h (root)**, and **98 h (grapevine)** [2605.17773]. The method also notes a complexity of **$O(|V|^2)$** for pairwise edge scoring and **$O(|E|\log|V|)$** for Kruskal MST with $|E|\sim |V|^2$.

The data curation is unusually broad. Domain-specific datasets include:
- **L-system synthetic**: 100,000 train / 1,000 val / 20,000 test,
- **Root (Arabidopsis early roots)**: 625 train / 78 val / 78 test, augmented to 62,500 train,
- **Grapevine (3D2cut Single Guyot)**: 1,185 train / 63 val / 255 test, augmented to 118,500 train [2605.17773].

Generalized training uses six datasets: **grapevine; LVD2021 leaf image; LVD2021 vein mask; MIPDB; self-captured; web-sourced**, standardized to **256×256**, with nodes sampled uniformly at **13 pixels** and a maximum-node cap of **384**. Four out-of-domain test datasets are then used: **thickened L-system**, **LVD with background**, **tree synthetic**, and **root** [2605.17773].

Evaluation uses three principal metrics:
- **Street Mover’s Distance (SMD)** for geometric fidelity of skeletons,
- **TOPO score** for topological node precision, recall, and F1,
- **Tree rate** for the percentage of outputs that satisfy the tree constraint [2605.17773].

## 7. Reported performance, generalization, and conceptual significance

Under **domain-specific training**, PlantPose reports:
- on **synthetic L-system**, **SMD ≈ 4.78×10^−6**, **TOPO F1 ≈ 0.977**, **Tree rate 100%**;
- on **root**, **SMD ≈ 8.82×10^−5**, **TOPO F1 ≈ 0.833**, **Tree rate 100%**;
- on **grapevine**, **SMD ≈ 1.03×10^−4**, **TOPO F1 ≈ 0.870**, **Tree rate 100%** [2605.17773].

The same section reports that the **unconstrained RelationFormer** has much lower tree validity—approximately **36%** tree rate on synthetic L-system and approximately **0%** on grapevine—while **test-time MST** improves performance but remains behind full PlantPose on the reported tasks [2605.17773].

Under **generalized training** across six datasets, PlantPose is reported to maintain **Tree rate 100%** and balanced TOPO and SMD across in-domain benchmarks. The listed typical outcomes are:
- **Grapevine**: **F1 ≈ 0.839**, **SMD ≈ 1.57×10^−4**,
- **LVD leaf image**: **F1 ≈ 0.817**, **SMD ≈ 5.41×10^−4**,
- **LVD vein mask**: **F1 ≈ 0.888**, **SMD ≈ 3.75×10^−4**,
- **MIPDB**: **F1 ≈ 0.686**, **SMD ≈ 1.61×10^−3**,
- **Self-captured**: **F1 ≈ 0.691**, **SMD ≈ 1.92×10^−3**,
- **Web-sourced**: **F1 ≈ 0.594**, **SMD ≈ 8.26×10^−3** [2605.17773].

On the **out-of-domain test sets**, the reported results are:
- **Thickened L-system**: **F1 ≈ 0.841**, **SMD ≈ 5.59×10^−5**, **Tree rate 100%**,
- **LVD with background**: **F1 ≈ 0.594**, **SMD ≈ 2.22×10^−2**, **Tree rate ≈ 99.8%**,
- **Tree synthetic**: **F1 ≈ 0.483**, **SMD ≈ 7.34×10^−3**, **Tree rate 100%**,
- **Root**: **F1 ≈ 0.555**, **SMD ≈ 2.70×10^−2**, **Tree rate ≈ 98.2%** [2605.17773].

The paper attributes this robustness to a combination of **diverse training data** and **constraint-aware learning**. It further reports qualitative generalization to unlabeled web images, sketches, and even retinal vessel images after simple preprocessing, though this is described qualitatively rather than through benchmark metrics [2605.17773].

The work also lists limitations. Training incurs **computational overhead** because MST is computed at every iteration. Performance depends on accurate node detection; missed nodes lead to broken topology. **Very thin or heavily occluded structures**, dense foliage, cluttered backgrounds, and overlapping branches remain challenging because wrong node or edge cues can induce incorrect connections [2605.17773].

Taken together, the two PlantPose systems illustrate two distinct ways of imposing structure in agricultural vision. PLANTPose [2505.24636] imposes **geometric structure** through a low-parameter cubic B-spline deformation around a category-level base mesh, together with pose–deformation disentanglement. PlantPose [2605.17773] imposes **topological structure** through MST-constrained graph generation and SFS-based differentiable consistency. A plausible implication is that the shared name reflects a broader methodological tendency in plant and agricultural vision: embedding explicit structural priors—whether geometric or graph-theoretic—into learning pipelines in order to improve robustness under heterogeneity, domain shift, and sparse supervision.

Source: https://www.emergentmind.com/topics/plantpose