---
title: 'Vote2Cap-DETR: 3D Captioning Transformer'
url: https://www.emergentmind.com/topics/vote2cap-detr
type: topic
---

# Vote2Cap-DETR: 3D Captioning Transformer

Vote2Cap-DETR is a fully transformer-based end-to-end framework for 3D dense captioning, designed to unify 3D object localization and region-level caption generation within a one-stage architecture. By leveraging a learnable “vote query” mechanism for spatial object anchoring, a dual-clued language head, and set prediction via DETR-style transformers, Vote2Cap-DETR and its advanced variant Vote2Cap-DETR++ demonstrate state-of-the-art performance on standard benchmarks, substantially improving over detect-then-describe pipelines in complex indoor scenes [2301.02508][2309.02999].

## 1. Model Architecture

Input to Vote2Cap-DETR is a raw point-cloud $PC = \{(p_{in}^i, f_{in}^i)\}_{i=1}^N$ with 3D coordinates $p_{in}^i \in \mathbb{R}^3$ and per-point features $f_{in}^i \in \mathbb{R}^F$. Downsampling to 2,048 points is performed via PointNet++ set abstraction. Scene encoding uses a 3DETR-style backbone: three masked self-attention layers interleaved with local set-abstraction, yielding $M = 1,024$ scene tokens $\{(p_{enc}^j, f_{enc}^j)\}_{j=1}^M$, where $f_{enc}^j \in \mathbb{R}^{256}$.

Vote-query generation identifies $K=256$ seed points $p_{seed}^k$ by Farthest Point Sampling (FPS) over the encoded scene, then applies an MLP $FFN_{vote}$ to seed features $f_{seed}^k$ to predict spatial offsets $\Delta p_{vote}^k$, producing 3D anchors $p_{vq}^k = p_{seed}^k + FFN_{vote}(f_{seed}^k)$. Local PointNet++-style feature aggregation around $p_{vq}^k$ yields $f_{vq}^k$.

These (position, feature) pairs act as queries for an 8-layer transformer decoder. Each decoder layer updates query features as
$$
f_{q}^{(i)} = DecoderLayer_i(f_{q}^{(i-1)} + PE(p_{vq})),
$$
with $PE(\cdot)$ a 3D Fourier positional encoding [2301.02508].

Two heads operate in parallel:
- **Detection head**: Per-query MLPs regress 3D box corners, center offsets, and semantic class scores.
- **Caption head (Dual-Clued Captioner)**: A 2-layer transformer decoder that attends to the query’s final feature $\mathcal{V}^q = f_{q}^{(L)}$ and a local context set $\mathcal{V}^s$ of the $k_s$ nearest scene-token features, decoding captions autoregressively [2309.02999].

## 2. Decoupled Localization and Captioning: Vote2Cap-DETR++

Vote2Cap-DETR++ introduces explicit decoupling of queries for localization ($[LOC]$) and captioning ($[CAP]$). At each decoder layer, matching $[CAP]$ queries are generated from $[LOC]$ via a learned projection:
$$
f_{cap}^{(i)} = W_{proj} f_{loc}^{(i)}, \quad p_{cap}^{(i)} = p_{loc}^{(i)}.
$$
Both sets traverse the same decoding block, but each prediction head operates only over its designated query set, with positions tied for correspondence.

To refine object spatial anchors, iterative spatial refinement updates $p_{loc}^{(i)}$ at each layer:
$$
p_{loc}^{(i)} = p_{loc}^{(i-1)} + FFN_{refine}(f_{loc}^{(i-1)}).
$$
This strategy incrementally drifts queries toward ground-truth centers, empirically increasing mean average precision (mAP) and speeding convergence [2309.02999].

## 3. Spatial Information Injection for Captions

Vote2Cap-DETR++ enhances descriptiveness and localization precision by integrating spatial priors into the captioning process. An absolute position token $PE(p_{cap}^{(L)})$ is prepended to the caption decoder input, explicitly encoding 3D location. For the local context $\{v_j^s\}$, rank-based embeddings $\mathcal{V}^s_{pos}$, learned as a function of each token’s distance rank to $p_{cap}^{(L)}$, are added to the context features:
$$
c^* = \arg\max_c P(c \mid \mathcal{V}^s, \mathcal{V}^s_{pos};\,\mathcal{V}^q, \mathcal{V}^q_{pos}).
$$
This injects geometric hierarchy into linguistic generation and improves spatial phrase accuracy [2309.02999].

## 4. Training Objectives and Loss Functions

Optimization supervises three principal modules:
- **Vote Query Loss**:
  $$
  \mathcal{L}_{vq} = \frac1M \sum_{i=1}^M \sum_{j=1}^{N_{gt}} \|p_{vote}^i - c_j^{center}\|_1\, \mathbf{1}\{p_{enc}^i \in {\rm obj}\,j\}
  $$
- **Detection Set Loss**: After Hungarian matching,
  $$
  \mathcal{L}_{det} = \alpha_1 \mathcal{L}_{giou} + \alpha_2 \mathcal{L}_{cls} + \alpha_3 \mathcal{L}_{center} + \alpha_4 \mathcal{L}_{size}
  $$
- **Query Refinement Loss** (Vote2Cap-DETR++ only): Matches $\mathcal{L}_{vq}$, but applied recursively per decoder layer to $p_{loc}^{(i)}$.

For captioning, two objectives are used:
- **MLE Loss**:
  $$
  \mathcal{L}_{cap}^{\rm MLE} = -\sum_{t=1}^T \log P(w_t \mid w_{1:t-1}, \mathcal{V})
  $$
- **SCST Loss** (Self-Critical Sequence Training, using CIDEr reward):
  $$
  \mathcal{L}_{cap}^{\rm SCST} = -\sum_{i=1}^k (R(\hat{c}_i) - R(\hat{g})) \frac{1}{|\hat{c}_i|} \log P(\hat{c}_i \mid \mathcal{V})
  $$

The multi-task objective is
$$
\mathcal{L}_{\rm total} = \lambda_{loc}\, \mathcal{L}_{loc} + \lambda_{cap}\,\mathcal{L}_{cap},
$$
where $\mathcal{L}_{loc} = \mathcal{L}_{vq} + \mathcal{L}_{det}$ (plus refinement loss as appropriate) and $(\lambda_{loc}, \lambda_{cap})$ are validation-tuned [2301.02508][2309.02999].

## 5. Datasets, Evaluation, and Empirical Performance

Vote2Cap-DETR and Vote2Cap-DETR++ are evaluated on ScanRefer and Nr3D—benchmarks for 3D region-level captioning and localization. Key metrics are:
- **C@0.5**: CIDEr@IoU≥0.5, measuring caption quality conditioned on sufficiently accurate region localization.
- **mAP@0.5**: Standard mean average precision for 3D object detection with IoU≥0.5.

Summarized results:

| Method                   | Dataset   | C@0.5 (SCST) | mAP@0.5 |
|--------------------------|-----------|--------------|---------|
| VoteNet+DtD baseline     | ScanRefer | ≈46%         | ≈38%    |
| Vote2Cap-DETR            | ScanRefer | 73.8%        | 52.1%   |
| Vote2Cap-DETR++          | ScanRefer | 78.2%        | 55.5%   |
| Best prior (SCST)        | Nr3D      | ≈38%         | –       |
| Vote2Cap-DETR            | Nr3D      | 45.5%        | –       |
| Vote2Cap-DETR++          | Nr3D      | 47.6%        | –       |

These models consistently outperform previous detect-then-describe pipelines on both localization and captioning [2301.02508][2309.02999].

## 6. Ablations and Analysis

Ablations confirm the contribution of core architectural elements:
- **Vote Queries**: Lead to a +2.98 mAP@0.5 over a plain 3DETR backbone, with faster convergence.
- **Local Context in Captioning**: Using $k_s=128$ local tokens for the captioner improves C@0.5 by +0.19 over global context.
- **Set-to-Set SCST**: Outperforms set-to-sentence training, with a +2.38 C@0.5 gain.
- **Iterative Refinement and Decoupling (DETR++)**: Further increases in both mAP and C@0.5.
- **NMS Free**: Vote2Cap-DETR is robust, yielding mAP@0.5 = 52.8% regardless of NMS application [2301.02508].

## 7. Position in the Literature and Limitations

Vote2Cap-DETR and Vote2Cap-DETR++ are the first transformer-based, one-stage architectures for 3D dense captioning that replace multi-step, hand-crafted relation modules with set prediction and spatially-aware transformer heads. This approach avoids error accumulation typical in cascade pipelines with duplicated and inaccurate box proposals in cluttered scenes.

Qualitative analysis shows vote queries cluster near object centers, enhancing detection, while the dual-clued captioner improves attribute and relationship selection by exploiting rich 3D context. Limitations include lack of explicit modeling for rotated bounding boxes and a reliance on CIDEr-driven SCST, which may bias toward n-gram matching rather than conceptual diversity [2309.02999][2301.02508].

Source: https://www.emergentmind.com/topics/vote2cap-detr