Object-Oriented Decoder

• Object-oriented decoder is a neural network component that models object instances as fixed, learnable queries with explicit spatial parameters for precise localization.
• It utilizes Rotated RoI Attention and multi-stage refinement with Selective Distinct Queries to align features and reduce redundancy in dense, multi-orientation scenarios.
• Empirical benchmarks on datasets like DIOR-R, DOTA, and COCO demonstrate its state-of-the-art performance by tightly coupling classification and regression tasks.

An object-oriented decoder is a neural network component for end-to-end object detection architectures that directly models object instances as a fixed set of learnable queries. Each query is associated with both a visual content embedding and an explicit parameterization of a bounding box. In the context of oriented object detection, the object-oriented decoder integrates specialized attention mechanisms, multi-stage refinement, and query management strategies, enabling simultaneous localization and classification of complex, arbitrarily oriented objects in dense settings. The Rotated Query Transformer (RQFormer) exemplifies these advancements, incorporating Rotated RoI Attention and Selective Distinct Queries for improved alignment and efficiency, demonstrating high accuracy on leading remote sensing and @@@@1@@@@ benchmarks (Zhao et al., 2023).

1. Rotated RoI Attention Mechanism

Rotated RoI Attention (RRoI Attention) departs from standard cross-attention by spatially aligning features to each query's oriented region. Each query $n$ maintains an explicit oriented box $B_n = (x_n, y_n, w_n, h_n, \theta_n)$, parameterizing center $(x, y)$, width $w$, height $h$, and angle $\theta$. Multi-scale feature maps $x^l$ from a backbone and FPN are warped, using a RoI-Align–style operator $R_l(x^l, B_n, r)$, to extract $r \times r$ feature crops that are locally aligned to these boxes.

For each attention head $m$:

• Projections $W_Q^m$, $W_K^m$, $W_V^m$, $W_O^m$ are used.
• Query features $q_n$ are projected as $Q'_{n,m}$, and similarly for keys $K'_{n,m,k}$ and values $V'_{n,m,k}$ at each spatial location $k$ within the crop.
• Attention weights $A_{n,m,k}$ are computed over $r^2$ locations using softmax normalization.
• Multi-head attended outputs are concatenated and linearly projected to restore embedding dimension $C$.

Compactly, the process can be denoted as:

$$A = \mathrm{softmax}(Q̃ K̃^T / \sqrt{d_k}); \qquad \text{Output} = \mathrm{Concat}_m(A_{:,:,m,:} Ṽ_{:,:,m,:}) W_O$$

This mechanism ensures precise alignment between each query and the key/value features it samples, mitigating misalignment between positional queries and extracted features in oriented detection scenarios (Zhao et al., 2023).

2. Selective Distinct Queries (SDQ)

Selective Distinct Queries (SDQ) dynamically refines the query set passed to each decoder stage by pruning redundant or low-quality queries, optimizing for both diversity and instance coverage.

The criterion for redundancy is based on box overlap: two queries $q_i$, $q_j$ are deemed similar if $\mathrm{IoU}(B_i, B_j) > \tau$, with $\tau$ typically in $[0.8, 0.95]$. Across selected decoder layers $S = \{\ell_1, \ldots, \ell_p\}$, queries and their associated boxes are concatenated and subjected to class-agnostic non-maximum suppression (NMS). Only boxes with the highest classification scores, not overlapping above $\tau$, are retained.

Pseudo-code for SDQ:

Q_collect, B_collect = [], [] for ℓ in S: Q_collect.append(q^ℓ) B_collect.append(B^ℓ) Q_cat = concat(Q_collect) B_cat = concat(B_collect) keep = NMS(boxes=B_cat, scores=cls_scores(Q_cat), iou_threshold=τ) Q_distinct = Q_cat[keep] B_distinct = B_cat[keep] if len(Q_distinct) > n: topk_idx = topk(cls_scores(Q_distinct), k=n) Q_next = Q_distinct[topk_idx] B_next = B_distinct[topk_idx] else: Q_next = Q_distinct B_next = B_distinct

This approach reduces query duplication and enhances optimization of one-to-one assignment between predictions and ground truths (Zhao et al., 2023).

3. Decoder Architecture and Update Mechanism

The object-oriented decoder comprises $d = 6$ identical transformer blocks, each operating in $C = 256$ dimensional embedding space with $M = 8$ attention heads:

1. Content Queries: $Q^0 \in \mathbb{R}^{n \times C}$ are learnable and randomly initialized; similarly, $B^0 \in \mathbb{R}^{n \times 5}$ (box parameters).
2. Layerwise Update:
   • Self-attention: $q' = \mathrm{SelfAttention}(q^{i-1}) + q^{i-1}$
   • RRoI cross-attention: $$q'' = \mathrm{MSRRoIAttn}(q', B^{i-1}, \{x^l\}) + q'$$
   • Feed-forward network (FFN): $q^i = \mathrm{FFN}(q'') + q''$
   • Regression head: Updates $B^i$ for next-layer proposal refinement.

By iteratively refining both content and localization, the decoder maintains tight coupling between classification and spatial reasoning (Zhao et al., 2023).

4. Oriented Bounding Box Parameterization and Regression

Each object proposal adopts the five-parameter oriented box formulation:

$B = (x_c, y_c, w, h, \theta)$

For ground-truth $G = (x^*, y^*, w^*, h^*, \theta^*)$, box regression utilizes:

$$t_x = \frac{x^* - x}{w}, \quad t_y = \frac{y^* - y}{h}, \quad t_w = \log \frac{w^*}{w}, \quad t_h = \log \frac{h^*}{h}, \quad t_\theta = \theta^* - \theta$$

A linear regression head $W_{\text{reg}} \in \mathbb{R}^{5 \times C}$ generates these deltas from the decoder outputs.

This explicit geometric modeling is crucial for tasks requiring high-fidelity orientation prediction, such as remote sensing and scene text detection (Zhao et al., 2023).

5. Training Objectives and Hyperparameters

Three loss components are jointly optimized:

• Classification: Focal Loss, $$L_{\text{cls}} = -\alpha_t (1 - p_t)^\gamma \log(p_t)$$, with $\alpha = 0.25$, $\gamma = 2$.
• Regression: $L_1$ loss on parameterized box deltas and $$L_{\text{iou}} = 1 - \mathrm{RotatedIoU}(B_{\text{pred}}, B_{\text{gt}})$$ (or 1-GIoU).

The final loss is

$$L = \lambda_{\text{cls}} L_{\text{cls}} + \lambda_{L1} L_1 + \lambda_{\text{iou}} L_{\text{iou}}$$

with dataset-dependent coefficients:

• DIOR-R / DOTA: $\lambda_{\text{cls}} = 2.0$, $\lambda_{L1} = 2.0$, $\lambda_{\text{iou}} = 5.0$
• COCO: $\lambda_{\text{cls}} = 2.0$, $\lambda_{L1} = 5.0$, $\lambda_{\text{iou}} = 2.0$

Training utilizes a ResNet-50+FPN backbone, AdamW optimizer with weight decay $1 \times 10^{-4}$, and dataset-/task-specific learning rate and epoch settings. Main experiments use $n=500$ queries; ablations use $n=300$ (Zhao et al., 2023).

6. Empirical Performance and Benchmark Results

The object-oriented decoder paradigm, as instantiated in RQFormer, attains state-of-the-art results on prominent oriented object detection datasets:

Dataset	mAP (%)	Key Baselines	mAP Baseline
DIOR-R test (20cls)	67.31	ARS-DETR, DCFL	65.90, 66.80
DOTA-v1.0 val	75.04 / 77.23 (SS/MS)	ARS-DETR, Deformable DETR-O	73.79, 63.42
DOTA-v1.5 val	67.43	ReDet, DCFL	66.86, 67.37
DOTA-v2.0 val	53.28	Oriented R-CNN, RoI Transformer	53.28, 52.81
COCO 2017 (horiz.)	46.8	Sparse R-CNN	45.0

SS: single-scale, MS: multi-scale.

The combination of (1) explicit feature alignment to learnable boxes via RRoI Attention, (2) distillation of high-quality, non-redundant queries via SDQ, and (3) multi-layer iterative refinement promptly advances the state-of-the-art in both oriented and horizontal detection. For COCO 2017, SDQ provides a +1.8% improvement over the Sparse R-CNN baseline (Zhao et al., 2023).

7. Significance and Implications

The object-oriented decoder framework fundamentally restructures the object detection pipeline: each object query concurrently encodes localization, class hypothesis, and spatial alignment, and is refined iteratively via a tightly coupled attention mechanism. The methods employed, especially RRoI Attention and SDQ, address common challenges in oriented object detection—feature misalignment and query redundancy—without introducing cumbersome auxiliary branches or excessive parameter growth. A plausible implication is that this paradigm will facilitate more efficient and adaptive end-to-end detection architectures for high-density, multi-orientation tasks (Zhao et al., 2023).