OVSegDT: Transformer Policy for Object Navigation

• OVSegDT is a segmenting transformer policy that fuses RGB images, natural language goals, and target masks to enable robust navigation in partially observable indoor environments.
• It leverages an entropy-adaptive loss modulation scheme to seamlessly blend imitation learning and reinforcement learning, reducing sample complexity and collision rates.
• The architecture uses an auxiliary segmentation decoder as a regularizer to reinforce spatial grounding, ensuring strong generalization to unseen categories.

Searching arXiv for OVSegDT and closely related work to ground the article. arxiv_search(query="OVSegDT Segmenting Transformer for Open-Vocabulary Object Goal Navigation", max_results=5)

Searching arXiv for the exact OVSegDT paper title and related terms. <tool name="arxiv_search"> {"query":"OVSegDT Segmenting Transformer for Open-Vocabulary Object Goal Navigation","max_results":5} </tool>

OVSegDT is a lightweight end-to-end transformer policy for open-vocabulary object goal navigation (OVON). It operates in partially observable indoor environments, takes free-form language goals that may name categories unseen during training, and augments a standard RGB+text navigation policy with explicit target-object segmentation masks and an adaptive mixture of imitation learning and reinforcement learning. In the formulation reported in "OVSegDT: Segmenting Transformer for Open-Vocabulary Object Goal Navigation" (Zemskova et al., 15 Aug 2025), the policy is designed to reduce the overfitting, poor unseen-category generalization, and high collision rates that can arise when end-to-end policies are trained on relatively small simulator datasets.

1. Task definition and problem setting

OVSegDT studies open-vocabulary object goal navigation rather than open-vocabulary semantic segmentation in the usual dense-prediction sense. At each timestep $t$, the agent receives

$o_t = \bigl(I_t,\; g,\; a_{t-1},\; M_t\bigr),$

where $I_t \in \mathbb{R}^{H\times W\times 3}$ is the current RGB image, $g$ is a natural-language object-goal description, $a_{t-1}$ is the previous discrete action, and $M_t \in \{0,1\}^{H\times W}$ is a binary mask for the target category in the current view (Zemskova et al., 15 Aug 2025).

The action space consists of six high-level commands: stop, forward $(0.25\ \text{m})$, turn_left $(30^\circ)$, turn_right $(30^\circ)$, look_up $(30^\circ)$, and look_down $o_t = \bigl(I_t,\; g,\; a_{t-1},\; M_t\bigr),$0. Episodes terminate when the agent emits stop or reaches a 500-step horizon. Rewards are sparse: +1 for successful stopping near the goal and a small negative reward for collisions. Evaluation uses standard ObjectNav metrics: Success Rate (SR), SPL, and SoftSPL.

The motivating difficulty is that OVON requires semantic grounding from free-form language while training data in photorealistic simulators are small relative to modern vision-language pretraining corpora. The paper identifies two central weaknesses of earlier mapless end-to-end policies. First, they often rely on a brittle two-stage schedule that manually switches from behavior cloning or DAgger to PPO. Second, they must infer both semantic grounding and navigation control from limited data, which encourages overfitting to training scenes and weak generalization to unseen categories. OVSegDT addresses these issues with two linked components: a semantic branch that injects binary target masks and an Entropy-Adaptive Loss Modulation (EALM) scheduler that continuously balances imitation and reinforcement.

2. Policy architecture and the semantic branch

The architecture follows the HM3D-OVON / DAgRL line closely, but introduces a target-mask pathway and an auxiliary segmentation decoder (Zemskova et al., 15 Aug 2025).

Component	Specification	Role
RGB encoder	Frozen SigLIP image encoder, 768-dim	Encodes current appearance
Text encoder	Frozen SigLIP text encoder, 768-dim	Encodes object-goal text
Action embedding	Learned, 32-dim	Encodes previous action
Goal-mask encoder	ResNet9, 128-dim	Encodes binary target mask
Temporal policy	Decoder-only transformer	Processes recent history
Segmentation head	DCGAN-style decoder	Reconstructs current target mask

The four modality embeddings are concatenated as

$o_t = \bigl(I_t,\; g,\; a_{t-1},\; M_t\bigr),$1

The temporal core is a decoder-only transformer over the most recent 100 observation embeddings $o_t = \bigl(I_t,\; g,\; a_{t-1},\; M_t\bigr),$2. Architecture details given in the paper are 4 transformer layers, 8 attention heads, hidden size 512, MLP size 1024, max context length 100, and an appendix note that the base model is LLaMA-style with shuffle position IDs enabled.

The semantic branch is the distinctive architectural addition. The binary mask $o_t = \bigl(I_t,\; g,\; a_{t-1},\; M_t\bigr),$3 indicates whether the target is visible, where it lies in the frame, and its rough extent and boundary. The paper argues that this provides explicit spatial grounding that standard RGB+text policies must otherwise discover implicitly from limited data. The mask encoder produces a compact representation that is fused with RGB, text, and action history, allowing the transformer state to combine scene appearance, target identity, navigation context, and target localization.

OVSegDT also adds a segmentation decoder that reconstructs the current target mask from the transformer feature. Its objective is

$o_t = \bigl(I_t,\; g,\; a_{t-1},\; M_t\bigr),$4

The paper explicitly presents this auxiliary head as a regularizer for the temporal representation rather than as a standalone segmentation subsystem. Its purpose is to preserve target-aware spatial information inside the hidden state.

3. Entropy-Adaptive Loss Modulation

EALM is the second core contribution. It replaces manual switching between imitation and reinforcement with a continuous, entropy-driven scheduler (Zemskova et al., 15 Aug 2025).

The key signal is the Shannon entropy of the policy’s action distribution. For each mini-batch, the method computes the current entropy and maintains an exponential moving average

$o_t = \bigl(I_t,\; g,\; a_{t-1},\; M_t\bigr),$5

with $o_t = \bigl(I_t,\; g,\; a_{t-1},\; M_t\bigr),$6. High entropy is interpreted as uncertainty, in which case imitation should dominate; low entropy is interpreted as confidence, in which case PPO should dominate.

The EMA entropy is converted into a mixing coefficient by thresholding between $o_t = \bigl(I_t,\; g,\; a_{t-1},\; M_t\bigr),$7 and $o_t = \bigl(I_t,\; g,\; a_{t-1},\; M_t\bigr),$8: $o_t = \bigl(I_t,\; g,\; a_{t-1},\; M_t\bigr),$9 The corresponding mixture weights are $I_t \in \mathbb{R}^{H\times W\times 3}$0 and $I_t \in \mathbb{R}^{H\times W\times 3}$1, yielding

$I_t \in \mathbb{R}^{H\times W\times 3}$2

The full optimization objective is

$I_t \in \mathbb{R}^{H\times W\times 3}$3

This formulation is intended to eliminate brittle manual phase switches. The paper contrasts EALM with pure PPO, pure DAgger, a weighted DAgger+PPO combination, an EarlySwitcher that linearly transitions from 40M to 60M steps, and the DAgRL baseline. The interpretation offered is that entropy provides a simple online estimate of whether the policy still requires expert supervision.

The imitation component uses expert actions from an A* planner with access to a ground-truth map and GPS, plus frontier-based exploration. An implementation detail emphasized in the paper is that the critic is also updated during behavior cloning, which is said to make the transition to RL seamless.

4. Training regime and deployment assumptions

Training is conducted on HM3D-OVON, built on HM3DSem/Habitat, and the paper stresses that HM3DSem contains only 216 annotated scenes, which partly explains why end-to-end navigation policies overfit. Evaluation is reported on val seen, val seen synonyms, and val unseen (Zemskova et al., 15 Aug 2025).

The reported training regime uses 200M environment steps, 40 parallel environments, 2 NVIDIA A100 GPUs, and Variable Experience Rollout (VER). PPO settings in the appendix include $I_t \in \mathbb{R}^{H\times W\times 3}$4, PPO epochs $I_t \in \mathbb{R}^{H\times W\times 3}$5, mini-batches $I_t \in \mathbb{R}^{H\times W\times 3}$6, value loss coefficient $I_t \in \mathbb{R}^{H\times W\times 3}$7, entropy coefficient $I_t \in \mathbb{R}^{H\times W\times 3}$8, learning rate $I_t \in \mathbb{R}^{H\times W\times 3}$9, Adam epsilon $g$0, max grad norm $g$1, steps per update $g$2, $g$3, GAE $g$4, GAE $g$5, linear learning-rate decay $g$6, and normalized advantage $g$7. Appendix simulator settings include turn angle $g$8, RGB sensor width 360, height 640, horizontal field of view $g$9, agent height 1.41, and agent radius 0.17.

The method is presented as a single-stage training pipeline because EALM removes explicit stage switching, even though imitation and RL remain blended during optimization. The abstract reports 130M parameters overall, while the text separately describes the transformer state encoder as 34M parameters. The policy is described as RGB-only at policy input, with no depth, no odometry, no explicit map, and no giant vision-LLM in the control loop.

A practical caveat is that the semantic branch assumes access to a target binary mask $a_{t-1}$0. In simulator experiments, this can be obtained from ground truth. For more realistic settings, the paper evaluates predicted masks.

5. Predicted-mask adaptation and practical calibration

The predicted-mask setting addresses the fact that simulator ground-truth masks are unavailable in real deployments. OVSegDT replaces oracle masks with masks from a pretrained open-vocabulary segmentation model, YOLOE, and then adapts the navigation policy to the noisier supervision (Zemskova et al., 15 Aug 2025).

The paper reports two adaptation procedures. The first is filtered-mask PPO finetuning, which keeps only sufficiently large ground-truth masks during finetuning and discards masks when width or height is less than 32 pixels or total mask area is less than 1000 pixels. This is intended to mimic the difficulty of reliably segmenting distant or heavily occluded targets. The second is per-category confidence calibration and vocabulary cleanup. Thresholds are tuned per class because the optimal detector threshold varies: the paper gives 0.4 for “dishwasher” as an example, while “book” or “flowerpot” may require keeping predictions down to 0.01. It also removes semantically redundant labels such as “rug” versus “carpet.” In practice, 0.3 is said to be a robust threshold for unseen categories.

The predicted-mask ablation on val unseen shows the scale of the deployment gap. With oracle masks, OVSegDT obtains 70.4 SR, 39.4 SPL, and 18.8 collisions. Replacing them with out-of-the-box YOLOE masks drops performance to 35.5 SR, 16.4 SPL, and 28.2 collisions. Filtered-mask finetuning raises SPL to 18.5, and adding YOLOE calibration yields the headline predicted-mask result of 40.1 SR, 20.9 SPL, and 24.7 collisions.

This supports one of the paper’s main practical claims: calibrating the segmentation model is much cheaper than retraining the navigation model for tens of millions of steps. At the same time, it makes clear that segmentation quality is the principal bottleneck once the policy architecture is fixed.

6. Empirical results, ablations, and interpretive significance

On the main HM3D-OVON benchmark, OVSegDT reports 41.0 / 21.6 on val seen SR / SPL, 37.7 / 19.6 on val seen synonyms, and 40.1 / 20.9 on val unseen, which the paper describes as state of the art for this setting (Zemskova et al., 15 Aug 2025). A central claim is the near parity between seen and unseen categories: val seen SR is 41.0, while val unseen SR is 40.1. The abstract also states that the method cuts the sample complexity of training by 33% and reduce collision count in two times.

The switching-strategy ablation isolates EALM. Compared with DagRL at 300M steps, EALM at 200M achieves 42.5 / 21.3 / 25.3 on val seen SR / SPL / collisions and 20.2 / 8.8 / 41.8 on val unseen, whereas DagRL reports 41.3 / 21.2 / 33.4 and 18.3 / 7.9 / 50.2. This is the basis for the stated 33% sample-complexity reduction. The appendix further reports one-sided t-tests such as EALM vs DagRL on val-seen SR: $a_{t-1}$1.

The semantic-branch ablation is even more consequential. On val unseen with ground-truth masks, the paper lists:

• DagRL: 18.3 SR, 7.9 SPL, 50.2 collisions
• OVSegDT with EALM only: 20.2 SR, 8.8 SPL, 41.8 collisions
• OVSegDT + EALM + goal mask: 69.3 SR, 40.8 SPL, 14.5 collisions
• OVSegDT + EALM + goal mask + $a_{t-1}$2: 70.4 SR, 39.4 SPL, 18.8 collisions

These numbers show that the dominant gain comes from the goal mask itself. The auxiliary segmentation loss improves SR from 69.3 to 70.4, but slightly reduces SPL and increases collisions. The appendix reports that the goal mask significantly improves over the no-goal-mask setting with SR $a_{t-1}$3 and SPL $a_{t-1}$4, while adding $a_{t-1}$5 improves SR over goal-mask-only with $a_{t-1}$6.

An especially revealing inference-time ablation tests which input modalities are actually carrying the navigation signal. With Goal Mask + Text Goal, OVSegDT obtains 70.4 SR, 39.4 SPL, 18.8 collisions. With Goal Mask only, it still achieves 65.3 SR, 34.2 SPL, 20.0 collisions. With Text Goal only, performance collapses to 0.1 SR, 0.0 SPL, 40.8 collisions. This strongly suggests that OVSegDT is best understood as a navigation policy whose open-vocabulary generalization is mediated primarily through the visual goal mask rather than through text-only semantic reasoning.

7. Misconceptions, limitations, and relation to adjacent open-vocabulary segmentation work

A common misconception is to read OVSegDT as a classical open-vocabulary segmentation architecture because of its name. The paper does not present it as a new dense prediction system. It is an end-to-end navigation transformer that uses target-object segmentation masks as explicit semantic and spatial cues. The segmentation head is auxiliary; the policy output is an action distribution, not a segmentation map (Zemskova et al., 15 Aug 2025).

A second misconception is that OVSegDT is a text-only OVON policy with improved language grounding. The paper’s own inference-time ablation contradicts that interpretation: with text goal only, navigation essentially fails. The target mask is the decisive signal.

The reported limitations are correspondingly direct. First, performance depends heavily on segmentation quality. Out-of-the-box predicted masks produce a substantial degradation relative to oracle masks. Second, the method does not handle free-form instruction following beyond object-goal naming. Third, the segmentation auxiliary improves success rate but can worsen SPL and collisions slightly, which suggests that the joint optimization of segmentation and navigation remains imperfect.

Within the broader arXiv landscape, OVSegDT is adjacent to but distinct from several open-vocabulary segmentation lines. OpenDAS adapts the CLIP component inside OVSeg and OpenMask3D through parameter-efficient prompt tuning and a triplet-loss strategy for open-vocabulary domain adaptation, improving OVSeg on ADE20K-150 from 29.8 mIoU to 35.8 mIoU and OpenMask3D on ScanNet++ Offices by +4.1 AP (Yilmaz et al., 2024). OVOSE transfers ODISE-style open-vocabulary dense prediction into event cameras through reconstruction-based teacher-student distillation (Rahman et al., 2024). OVBEVSeg introduces open-vocabulary BEV segmentation with geometry-aware pseudo-labeling, per-scene Gaussian optimization, and geometric distillation in autonomous driving (Choi et al., 23 Jun 2026). These works are all genuinely about open-vocabulary segmentation; OVSegDT instead converts segmentation cues into a control signal for navigation.

In that sense, OVSegDT occupies a specific niche. It is not an OVSeg derivative, not a domain-adapted CLIP classifier, and not a general instruction-following embodied model. It is a segmenting transformer policy whose core thesis is that explicit target masks plus entropy-adaptive IL/RL balancing are sufficient to deliver strong unseen-category generalization and substantially lower collision rates in open-vocabulary object-goal navigation.