CRISP Pipeline for Pose and Shape Estimation

• CRISP is a category-agnostic framework that estimates object pose and reconstructs implicit shape representations from a single segmented RGB-D image.
• It integrates a ViT-based encoder–decoder with a dense pose estimator employing least-squares alignment and FiLM-conditioned SIREN for accurate shape reconstruction.
• The pipeline refines predictions via an optimization-based corrector and self-training with pseudo-label certification, effectively addressing domain shifts.

Searching arXiv for the specified paper and closely related work to ground the article. Search query: (Shi et al., 2024) CRISP Object Pose and Shape Estimation with Test-Time Adaptation CRISP is a category-agnostic pipeline for estimating object pose and shape from a single segmented RGB-D image. It combines an encoder–decoder model for implicit shape reconstruction with a dense pose estimator based on pose-normalized points, and augments both with an optimization-based corrector and a self-training procedure for test-time domain adaptation. In the formulation reported for "CRISP: Object Pose and Shape Estimation with Test-Time Adaptation" (Shi et al., 2024), the pipeline reconstructs an implicit signed distance field in a pose-normalized frame, estimates pose by least-squares alignment between depth points and predicted pose-normalized coordinates, and then optionally applies a correct-and-certify loop to generate pseudo-labels for self-supervised adaptation under domain shift.

1. Problem setting and system scope

CRISP addresses object pose and shape estimation from a single segmented RGB-D image, with an explicit emphasis on category-agnostic inference and generalization to unseen objects (Shi et al., 2024). The input is a segmented RGB-D observation; the outputs are an object shape represented implicitly as an SDF and a rigid pose $(R,t)$ estimated from pose-normalized point correspondences.

The system is organized around three coupled functions. First, a shape encoder–decoder predicts a latent shape code and decodes it into an implicit field. Second, a dense prediction network estimates pose-normalized points for visible depth samples, from which pose is recovered by least squares. Third, an optimization-based corrector updates the predicted pose-normalized points and shape representation to reduce errors caused by domain gap. A self-training pipeline then uses these corrected outputs as pseudo-labels at test time to adapt the prediction heads while freezing the decoder (Shi et al., 2024).

A notable property of the framework is that it does not require category labels for shape reconstruction or pose estimation. This suggests a design centered on geometric consistency rather than category-specific templates. A plausible implication is that the latent shape space, rather than an explicit taxonomy, acts as the principal structural prior.

2. Core architecture

CRISP uses an encoder–decoder architecture for shape estimation. A ViT backbone, specifically DINOv2 ViT-S, extracts tokens from the segmented RGB image, and an MLP regresses a latent shape code

$h = f_e(I).$

The decoder is a FiLM-conditioned MLP with sinusoidal activations, i.e., a SIREN, and outputs SDF values

$f_d(x \mid h)$

in a pose-normalized frame (Shi et al., 2024).

FiLM conditioning is applied at each hidden layer. For a feature map $f_\ell$ at layer $\ell$,

$$h_\ell = \gamma_\ell(z) \odot f_\ell + \beta_\ell(z),$$

where $z$ is the conditioning input. In CRISP, the shape head produces a high-dimensional conditioning vector that is split per decoder layer, and each layer applies layer-specific $\gamma$ and $\beta$ derived from the latent shape code. This realizes implicit shape reconstruction without explicit category labels (Shi et al., 2024).

Pose estimation is handled by a DPT-based dense prediction network that outputs pose-normalized points

$Z = \Phi(I),$

one per visible depth sample $h = f_e(I).$0, such that

$h = f_e(I).$1

Pose is then solved by least squares using an Arun/SVD procedure. The objective is

$h = f_e(I).$2

with $h = f_e(I).$3 and $h = f_e(I).$4 (Shi et al., 2024).

The DPT-style head uses tokens from ViT layers 3, 6, 9, and 12, processed through reassemble blocks and fusion blocks, followed by a CNN readout. The final head uses $h = f_e(I).$5 kernels with channel progression $h = f_e(I).$6, ReLU activations, and a $h = f_e(I).$7 final kernel to output an $h = f_e(I).$8 pose-normalized coordinate field (Shi et al., 2024).

An important distinction reported in the paper is that CRISP does not normalize $h = f_e(I).$9, specifically to avoid scale degeneracy during self-training (Shi et al., 2024).

3. Training objectives and implementation structure

The pose-normalized point head is trained with a soft L1 loss:

$f_d(x \mid h)$0

with $f_d(x \mid h)$1 on all datasets (Shi et al., 2024).

For SDF learning, the paper reports a loss combining direct SDF supervision, an off-manifold penalty, and Eikonal regularization:

$f_d(x \mid h)$2

with $f_d(x \mid h)$3 and $f_d(x \mid h)$4 (Shi et al., 2024). The combined loss is

$f_d(x \mid h)$5

with dataset-specific $f_d(x \mid h)$6 (Shi et al., 2024).

The code organization described conceptually in the appendix comprises a shape head producing FiLM conditioning vectors, a FiLM-conditioned SIREN shape decoder, a PNC head built from DPT-style reassemble and fusion blocks with a CNN readout, corrector implementations for BCD and LSQ, and a self-training pipeline implementing correct-and-certify followed by SGD with pseudo-labels (Shi et al., 2024).

The implementation stack is based on PyTorch. DINOv2 ViT-S is frozen during training, DPT blocks are used for dense prediction, CVXPY is used for the LSQ interior-point solver, and libigl is used to compute ground-truth SDF values. The reported hardware is 8 Nvidia V100 GPUs for supervised training and 1 Nvidia A6000 for self-training (Shi et al., 2024).

4. Optimization-based correction

The corrector is formulated as a bi-level optimization whose equivalent form minimizes

$f_d(x \mid h)$7

where $f_d(x \mid h)$8 is the least-squares pose recovered from $f_d(x \mid h)$9:

$f_\ell$0

The corrector then solves

$f_\ell$1

using block coordinate descent: update $f_\ell$2 by gradient descent while keeping $f_\ell$3 fixed, then update $f_\ell$4 by projected gradient descent onto the simplex of latent shape codes learned during training (Shi et al., 2024).

The simplex projection is motivated by the observation that the learned decoder behaves reliably in the convex hull of known latent codes. The paper states that this prevents decoder extrapolation. This suggests that the latent geometry of known shapes is treated as a trusted region for correction, while extrapolated latent codes are treated as less reliable.

CRISP provides two solvers for this stage.

Solver	Update strategy	Shape constraint
BCD	Alternating updates for $f_\ell$5 and $f_\ell$6	Projected gradient descent onto simplex
LSQ	Active shape decoder approximation	Constrained linear least squares

The LSQ variant approximates the decoder with an active shape decoder:

$f_\ell$7

with $f_\ell$8, $f_\ell$9, and $\ell$0 (Shi et al., 2024). The update for shape then reduces to

$\ell$1

where $\ell$2 and $\ell$3 stacks decoder evaluations at the pose-normalized points. The paper reports that this constrained linear least-squares problem is solved by an interior-point method, implemented for example through CVXPY (Shi et al., 2024).

The paper also states that $\ell$4 is set by inverse bounding-box diameter per shape decoder instantiation, which empirically improves stability and performance (Shi et al., 2024).

5. Correct-and-certify self-training

CRISP extends correction into a self-supervised domain adaptation procedure based on correct-and-certify. For each detection, the system predicts $\ell$5 and $\ell$6, applies the corrector, certifies the corrected output with an observable consistency test, and uses only certified instances as pseudo-labels (Shi et al., 2024).

The certificate is

$\ell$7

where $\ell$8 denotes the $\ell$9-th quantile over points. The thresholds reported are:

Dataset	$$h_\ell = \gamma_\ell(z) \odot f_\ell + \beta_\ell(z),$$0	$$h_\ell = \gamma_\ell(z) \odot f_\ell + \beta_\ell(z),$$1
YCBV	0.98	$$h_\ell = \gamma_\ell(z) \odot f_\ell + \beta_\ell(z),$$2
SPE3R	0.97	$$h_\ell = \gamma_\ell(z) \odot f_\ell + \beta_\ell(z),$$3

Only detections passing this certificate become pseudo-labels (Shi et al., 2024). The self-training step freezes the shape decoder and updates the encoder heads using

$$h_\ell = \gamma_\ell(z) \odot f_\ell + \beta_\ell(z),$$4

The procedure may also aggregate $$h_\ell = \gamma_\ell(z) \odot f_\ell + \beta_\ell(z),$$5 across views for the shape update in a multi-view setting, while keeping $$h_\ell = \gamma_\ell(z) \odot f_\ell + \beta_\ell(z),$$6 updates per-view (Shi et al., 2024).

The reported effect of certification is that it filters high-error cases without ground truth, and that the correctors substantially increase the certification pass rate (Shi et al., 2024). A plausible implication is that certification is not merely a confidence heuristic but an explicit geometric consistency filter tied to the learned SDF.

6. Datasets, training regimes, and reported results

CRISP is evaluated on YCBV, SPE3R, and NOCS (Shi et al., 2024). The paper reports dataset-specific training schedules and evaluation metrics.

Dataset	Train data	Main metrics
YCBV	Real train split or BlenderProc synthetic renderings	ADD-S, AUC at 1/2/3 cm, $$h_\ell = \gamma_\ell(z) \odot f_\ell + \beta_\ell(z),$$7, AUC at 3/5/10 cm
SPE3R	Official split with 57 satellites, 7 withheld for testing	$$h_\ell = \gamma_\ell(z) \odot f_\ell + \beta_\ell(z),$$8, $$h_\ell = \gamma_\ell(z) \odot f_\ell + \beta_\ell(z),$$9
NOCS	CAMERA (300K) and REAL275 (8K frames across 18 scenes)	$z$0 (mm), mAP at IoU50/IoU75 and pose thresholds

On YCBV, CRISP-Real is trained with Adam, learning rate $z$1, 50 epochs, batch size 10, and weight decay $z$2. CRISP-Syn uses Adam, learning rate $z$3, 500 epochs, batch size 16, and weight decay $z$4. Self-training for CRISP-Syn-ST uses either LSQ or BCD for 5 epochs with decoder frozen; SGD learning rates are $z$5 for the shape head and $z$6 for the PNC head, with batch size 3 and weight decay $z$7 (Shi et al., 2024).

On SPE3R, training uses Adam, learning rate $z$8, 100 epochs, batch size 16, cosine annealing with restarts every 4000 iterations and factor 2, and weight decay $z$9. Self-training runs for 10 epochs with the BCD corrector, certification $\gamma$0, $\gamma$1, and batch size 10 (Shi et al., 2024).

On NOCS, training uses Adam, learning rate $\gamma$2, 50 epochs, batch size 16, cosine annealing with restarts every 4000 iterations, and weight decay $\gamma$3 (Shi et al., 2024).

The reported performance highlights are specific. On YCBV, CRISP-Real achieves mean $\gamma$4, outperforming Shap-E by approximately 73%, and self-training improves CRISP-Syn mean $\gamma$5 by 12–17% and AUC at 5 cm by 11–22%. On pose, CRISP-Real reaches the top AUC at 1–3 cm, while self-training improves CRISP-Syn mean ADD-S by 17–20% and AUC at 2 cm by 38–44% (Shi et al., 2024).

On SPE3R, CRISP outperforms SQRECON and Shap-E on pose, and self-training improves mean $\gamma$6 by 13% and mean $\gamma$7 by 23%, including on unseen satellites (Shi et al., 2024).

On NOCS, CRISP achieves the best $\gamma$8 in millimeters across all categories and overall, with average $\gamma$9 mm, and is reported to be notably strong for Camera at $\beta$0 mm. In category-agnostic pose estimation, it attains the best mAP on IoU50 and IoU75 and competitive mAP at $\beta$1 5 cm, though rotation errors are reported as higher than some category-specific methods (Shi et al., 2024).

These results are accompanied by the claim that CRISP reconstructs and estimates pose for objects unseen during training, and that self-training further improves such cases (Shi et al., 2024).

7. Runtime, code availability, and interpretation

The paper states that code and pre-trained models will be available on the project webpage and identifies the official project page at the MIT SPARK Lab site (Shi et al., 2024). It does not provide explicit repository commands, file paths, or final environment pins in the paper itself.

Approximate runtimes reported in the paper are mean inference time of about 125 ms for CRISP, 251 ms for LSQ, and BCD runtimes depending on iterations, including 3824 ms for 25 iterations and 7630 ms for 50 iterations (Shi et al., 2024). These figures indicate that the corrector is substantially more expensive than forward prediction, with LSQ offering a markedly lower correction cost than iterative BCD.

A common misconception would be to treat CRISP as a purely feed-forward pose-and-shape regressor. The paper instead presents it as a hybrid system in which feed-forward prediction, constrained latent-space correction, and test-time self-training are all first-class components. Another possible misconception is that the self-training stage simply fine-tunes the full model; in fact, the decoder is frozen and only encoder heads are updated during pseudo-label training (Shi et al., 2024).

Taken together, the design reported in (Shi et al., 2024) positions CRISP as an integrated framework in which implicit shape modeling, dense pose-normalized correspondences, convex-hull-constrained shape correction, and observable-consistency certification are mutually dependent. This suggests that its central contribution is less a single architecture than a pipeline that ties representation, optimization, and adaptation into a unified object pose-and-shape estimation system.