Frontier-Seg: Unsupervised Terrain Segmentation

• Frontier-Seg is an unsupervised method that segments terrain in mobile robot video by clustering superpixel-level descriptors without relying on labeled data.
• It employs a two-phase clustering pipeline—local temporal aggregation followed by global clustering—to enforce consistent region boundaries over time.
• The approach leverages a frozen DINOv2 backbone to extract dense features, enabling reliable terrain parsing in unstructured off-road environments.

Frontier-Seg is a method for temporally consistent unsupervised terrain segmentation from mobile robot video, introduced for off-road autonomy in unstructured outdoor environments where labeled data may be unavailable and fixed semantic ontologies may be poorly matched to navigation-relevant terrain structure (Ellis et al., 29 Jul 2025). Given a video sequence $\{I_t\}_{t=1}^T$, the method assigns each pixel $p$ in frame $I_t$ a pseudo-label $y_{p,t} \in \{1,\dots,K\}$ without human supervision, with the additional requirement that regions corresponding to the same terrain concept maintain coherent labels across adjacent frames. Its central contribution is a two-phase clustering framework over superpixel-level descriptors extracted from a frozen foundation-model backbone, followed by propagation of region labels back to pixels. In the paper’s terminology, the resulting stable region boundaries are “frontiers”: persistent terrain boundaries between visually distinct terrain regions, rather than the explored–unexplored boundaries that dominate classical frontier-based exploration literature (Ellis et al., 29 Jul 2025).

1. Conceptual scope and problem definition

Frontier-Seg is framed as a perception method for terrain-aware autonomous ground navigation in unrehearsed, unstructured environments. The motivating observation is that supervised semantic segmentation depends on expensive data collection and labor-intensive ground-truth labeling, while off-road terrain categories such as trail, mud, grass, forest floor, brush, or mixed terrain are often ambiguous, context-dependent, or visually heterogeneous (Ellis et al., 29 Jul 2025). The paper therefore emphasizes unsupervised segmentation rather than classification into a pre-specified semantic taxonomy.

The method targets video rather than isolated images. The underlying claim is that a robot observes the world as a stream, not as independent still images, so a framewise unsupervised segmenter that changes labels unpredictably between adjacent frames is unsuitable for downstream planning, traversability estimation, mapping, and behavior selection (Ellis et al., 29 Jul 2025). Frontier-Seg addresses this by inducing temporal consistency through temporal aggregation and sequence-level reclustering, rather than through explicit motion modeling.

The formal task is: $\{I_t\}_{t=1}^T \mapsto y_{p,t}\in\{1,\dots,K\},$ with no supervision and with temporal label coherence for regions corresponding to the same terrain concept. The “frontier” terminology is specific to this paper: frontiers are the boundaries between final clustered terrain regions, especially those that persist coherently across time. This differs from the established exploration definition in SLAM and occupancy-grid mapping, where a frontier is the boundary between explored and unexplored space (Oršulić et al., 2019). That distinction is fundamental, because Frontier-Seg is an unsupervised video segmentation framework rather than a frontier detector for exploration maps.

2. Representation pipeline: superpixels, DINOv2, and region descriptors

The pipeline begins per frame. For each input image $I_t \in \mathbb{R}^{H \times W \times 3}$, Frontier-Seg first computes an oversegmentation into superpixels using SLIC superpixels, specifically SLICO in the experiments (Ellis et al., 29 Jul 2025). The stated role of superpixels is threefold: reducing the computational burden of clustering at pixel level, imposing local spatial coherence, and creating compact regions that roughly respect image boundaries.

In the experimental setup, each image is resized to $512 \times 512$, converted to CIELAB color space, smoothed with a Gaussian blur of $\sigma=0.7$, and segmented with SLICO using region size 30, which yields about 200 superpixels per frame (Ellis et al., 29 Jul 2025). This converts a dense pixelwise problem into clustering over a few hundred region descriptors.

Dense visual features are extracted with a pretrained foundation-model backbone. The primary backbone is DINOv2, specifically the facebook/dinov2-with-registers-base Vision Transformer, with feature dimensionality $D=768$ (Ellis et al., 29 Jul 2025). The paper describes dense per-pixel features $F_t \in \mathbb{R}^{H \times W \times D}$, with each pixel $p$0 associated with a feature vector $p$1.

At the core of the representation step is masked average pooling over each superpixel. If $p$2 are the superpixels in frame $p$3, then the descriptor for superpixel $p$4 is

$p$5

The paper notes that the printed equation is corrupted in formatting, but explicitly identifies this as the intended expression (Ellis et al., 29 Jul 2025). In the supplementary implementation details, region descriptors are not limited to simple pooled patch embeddings: for each superpixel, the method computes a pooled descriptor by masked average pooling of dense features inside the superpixel, aggregating register tokens via attention conditioned on that pooled feature, and blending the result with the CLS token to form the final region descriptor. No formal equation is given for the register-token attention or blending step.

This representation choice is central to the method’s design. Frontier-Seg is essentially zero-shot in the sense that it does not train or fine-tune a segmentation model on labeled data; instead, it relies on a frozen DINOv2 backbone and unsupervised clustering of extracted region descriptors (Ellis et al., 29 Jul 2025). A plausible implication is that the quality and stability of the segmentation are strongly mediated by the geometry of the pretrained feature space.

3. Two-phase clustering and the mechanism of temporal consistency

Frontier-Seg enforces temporal consistency through a two-stage clustering design built on temporally pooled superpixel descriptors. The first stage is local clustering over temporal windows. Rather than clustering descriptors from each frame independently, the method aggregates descriptors over a temporal window $p$6, with

$p$7

It then applies plain $p$8-means in Euclidean feature space: $p$9 The paper explicitly excludes cosine similarity, contrastive learning, learned normalization, and adaptive cluster count; the supported formulation is Euclidean $I_t$0-means with squared Euclidean distance (Ellis et al., 29 Jul 2025). In the experiments, local clustering uses $I_t$1, and videos are divided into non-overlapping temporal windows of 100 frames, i.e. $I_t$2.

These local pseudo-labels are then used to merge superpixels within each frame: all superpixels in a frame that share the same local cluster label are merged into larger regions $I_t$3, where $I_t$4 (Ellis et al., 29 Jul 2025). The descriptor is recomputed on the merged regions as

$I_t$5

The second stage is global clustering over the full sequence. Frontier-Seg collects all recomputed merged-region descriptors $I_t$6 across the entire video and runs another $I_t$7-means step: $I_t$8 In the main experimental setup, global clustering uses $I_t$9 (Ellis et al., 29 Jul 2025). The final per-pixel segmentation is produced by assigning every pixel in merged region $y_{p,t} \in \{1,\dots,K\}$0 the global label $y_{p,t} \in \{1,\dots,K\}$1.

The method’s temporal consistency does not arise from optical flow, tracking, recurrent state, graph matching, temporal smoothing, or explicit correspondence constraints (Ellis et al., 29 Jul 2025). Instead, consistency is induced by two forms of temporal aggregation. First, local clustering jointly processes descriptors from adjacent frames, encouraging nearby-in-time regions with similar semantics to receive the same pseudo-label. Second, global clustering aligns merged descriptors from all windows into a common label space, allowing the same terrain concept to be assigned the same final label across the sequence. The paper is explicit that there is no temporal loss function, no propagation equation, no boundary confidence update, and no explicit frontier score formula.

This architectural choice is also the basis for the method’s offline character. Because the global clustering stage requires all merged region descriptors from the sequence before final labels can be assigned, the current implementation is sequence-dependent and offline rather than online (Ellis et al., 29 Jul 2025).

4. Evaluation protocol, datasets, and reported empirical behavior

Evaluation is performed on RUGD and RELLIS-3D, both chosen as off-road robot-perception benchmarks (Ellis et al., 29 Jul 2025). RUGD contains 18 temporal sequences ranging from 49 to 849 frames, while RELLIS-3D contains five sequences. Ground truth is used only for scoring.

The paper evaluates unsupervised segmentation with four metrics: mean Intersection-over-Union (mIoU), mean pixel accuracy (Acc), over-segmentation entropy (OSE), and under-segmentation entropy (USE) (Ellis et al., 29 Jul 2025). Because unsupervised clusters do not directly correspond to semantic classes, the supplementary material defines a many-to-one mapping from predicted clusters to ground-truth classes using overlap: $y_{p,t} \in \{1,\dots,K\}$2

$y_{p,t} \in \{1,\dots,K\}$3

The main text refers to many-to-one Hungarian matching, but the supplement clarifies that the implemented procedure is overlap-based majority voting rather than one-to-one Hungarian assignment (Ellis et al., 29 Jul 2025).

The segmentation metrics are reported as

$y_{p,t} \in \{1,\dots,K\}$4

The entropy metrics are

$y_{p,t} \in \{1,\dots,K\}$5

$y_{p,t} \in \{1,\dots,K\}$6

Lower OSE corresponds to less fragmentation of a semantic class across predicted clusters, while lower USE corresponds to higher semantic purity of each predicted cluster (Ellis et al., 29 Jul 2025).

The main baseline is DiffCut, evaluated both in its original frame-by-frame mode and in a temporal adaptation that feeds DiffCut’s per-frame regions and SSD-1B features into Frontier-Seg’s local and global clustering pipeline (Ellis et al., 29 Jul 2025). This design isolates the effects of temporal aggregation and feature backbone.

The reported results establish a clear distinction between framewise zero-shot segmentation and temporally consistent sequence-level segmentation. On RUGD in zero-shot mode, DiffCut achieves $y_{p,t} \in \{1,\dots,K\}$7 mIoU/Acc at $y_{p,t} \in \{1,\dots,K\}$8 and $y_{p,t} \in \{1,\dots,K\}$9 at $\{I_t\}_{t=1}^T \mapsto y_{p,t}\in\{1,\dots,K\},$0, while Frontier-Seg with DINOv2 achieves $\{I_t\}_{t=1}^T \mapsto y_{p,t}\in\{1,\dots,K\},$1; on RELLIS-3D zero-shot, DiffCut is stronger, with $\{I_t\}_{t=1}^T \mapsto y_{p,t}\in\{1,\dots,K\},$2 and $\{I_t\}_{t=1}^T \mapsto y_{p,t}\in\{1,\dots,K\},$3 versus Frontier-Seg DINOv2’s $\{I_t\}_{t=1}^T \mapsto y_{p,t}\in\{1,\dots,K\},$4 (Ellis et al., 29 Jul 2025).

The paper’s central empirical claim concerns temporal evaluation. On RUGD temporal evaluation, Frontier-Seg with DINOv2 reaches $\{I_t\}_{t=1}^T \mapsto y_{p,t}\in\{1,\dots,K\},$5 mIoU and $\{I_t\}_{t=1}^T \mapsto y_{p,t}\in\{1,\dots,K\},$6 accuracy, compared with DiffCut temporal at $\{I_t\}_{t=1}^T \mapsto y_{p,t}\in\{1,\dots,K\},$7 for $\{I_t\}_{t=1}^T \mapsto y_{p,t}\in\{1,\dots,K\},$8 and $\{I_t\}_{t=1}^T \mapsto y_{p,t}\in\{1,\dots,K\},$9 for $I_t \in \mathbb{R}^{H \times W \times 3}$0. On RELLIS-3D temporal evaluation, Frontier-Seg with DINOv2 reaches $I_t \in \mathbb{R}^{H \times W \times 3}$1, compared with DiffCut’s $I_t \in \mathbb{R}^{H \times W \times 3}$2 and $I_t \in \mathbb{R}^{H \times W \times 3}$3 (Ellis et al., 29 Jul 2025). The entropy metrics support the same interpretation: on RUGD temporal evaluation, Frontier-Seg DINOv2 has OSE/USE $I_t \in \mathbb{R}^{H \times W \times 3}$4, while DiffCut yields $I_t \in \mathbb{R}^{H \times W \times 3}$5 and $I_t \in \mathbb{R}^{H \times W \times 3}$6; on RELLIS-3D temporal evaluation, Frontier-Seg DINOv2 has $I_t \in \mathbb{R}^{H \times W \times 3}$7, while DiffCut achieves $I_t \in \mathbb{R}^{H \times W \times 3}$8 and $I_t \in \mathbb{R}^{H \times W \times 3}$9 (Ellis et al., 29 Jul 2025). The paper interprets this as evidence that Frontier-Seg’s temporally aggregated clusters are more semantically coherent at sequence level, even when fragmentation behavior varies by dataset.

The supplement further reports ablations over the number of clusters. For Frontier-Seg DINOv2 on RUGD temporal, performance increases from $512 \times 512$0 at $512 \times 512$1 to $512 \times 512$2 at $512 \times 512$3; on RELLIS-3D temporal, it increases from $512 \times 512$4 at $512 \times 512$5 to $512 \times 512$6 at $512 \times 512$7 (Ellis et al., 29 Jul 2025). The supplement explicitly notes that larger $512 \times 512$8 yields finer ontologies that may better capture small structures but can increase fragmentation, whereas smaller $512 \times 512$9 yields more compact but potentially semantically merged outputs.

5. Backbone choice, efficiency profile, and operational limitations

A major ablation concerns the feature backbone. The DINOv2-based Frontier-Seg consistently outperforms the SSD-1B-based version in both zero-shot and temporal settings (Ellis et al., 29 Jul 2025). At the main operating point, on RUGD temporal the DINOv2 variant attains $\sigma=0.7$0 versus $\sigma=0.7$1 for Frontier-Seg with SSD-1B; on RELLIS-3D temporal it attains $\sigma=0.7$2 versus $\sigma=0.7$3. In zero-shot evaluation, DINOv2 also exceeds SSD-1B on both datasets. The paper interprets this as evidence that the ViT-based DINOv2 features are better suited to region clustering for terrain segmentation than the diffusion-derived SSD-1B features used in the comparison.

The computational profile reflects the method’s hybrid status as zero-shot but sequence-optimized. DINOv2 dense feature extraction is reported to take about $\sigma=0.7$4 ms per $\sigma=0.7$5 image in a single pass, compared with approximately $\sigma=0.7$6 s per image for DiffCut’s SSD-1B diffusion backbone with 50 denoising steps on an RTX 4090 (Ellis et al., 29 Jul 2025). This makes the feature-extraction stage substantially lighter than the diffusion baseline. However, the paper identifies the storage and clustering of region descriptors in the global clustering stage as the main computational bottleneck.

The limitations are explicit. Frontier-Seg operates offline, assumes access to the complete video before global clustering, and is not currently suitable for true real-time or streaming deployment (Ellis et al., 29 Jul 2025). The authors identify incremental or online clustering as future work. They also note that the method assumes smooth frame-to-frame continuity and consistent ego-motion, and that performance may degrade under abrupt viewpoint changes, highly dynamic scenes, or disordered temporal input. Because temporal consistency is induced only through feature clustering, instability in the underlying feature representation or collapse of visually similar but semantically distinct terrains in embedding space can degrade cluster stability.

The paper further states that Frontier-Seg has no explicit mechanism to handle moving objects separately from terrain, no explicit geometric reasoning, and no explicit traversability label output (Ellis et al., 29 Jul 2025). It therefore produces persistent terrain-like regions and boundaries, but not direct “safe” versus “unsafe” predictions. This suggests that the method is most naturally interpreted as a label-free terrain parsing module whose outputs require downstream interpretation for navigation.

6. Relation to frontier research, navigation, and downstream robotics use

The word “frontier” has a long-established meaning in robot exploration. In occupancy-grid and SLAM systems, a frontier is the boundary between explored and unexplored space, as in dense frontier detection for 2D graph SLAM (Oršulić et al., 2019), frontier semantic exploration for target-driven navigation (Yu et al., 2023), or frontier management for collaborative active SLAM (Ahmed et al., 2023, Ahmed et al., 2023). Frontier-Seg does not use the term in that sense. Its frontiers are persistent terrain boundaries between clustered regions in robot video (Ellis et al., 29 Jul 2025).

That distinction matters because these two uses of “frontier” support different layers of autonomy. Classical frontier methods select where to explore next in a map, often relying on occupancy, information gain, path entropy, or frontier-point management (Oršulić et al., 2019, Ahmed et al., 2023). Frontier-Seg instead discovers temporally coherent perceptual partitions of terrain. A plausible implication is that the method is better understood as a perceptual substrate for later modules than as an exploration policy.

The paper makes this connection explicit. It motivates persistent terrain boundaries—stable boundaries between trail and vegetation, path and ditch, dirt and mud, or open terrain and obstacles—as structural cues for traversability assessment, costmap generation, route following, and behavior selection (Ellis et al., 29 Jul 2025). The output is a temporally coherent partition of the scene into persistent regions whose boundaries may be interpreted as navigational frontiers, but the method does not itself compute reachability, exploration utility, or planner goals in the sense used by map-based frontier planners (Oršulić et al., 2019, Yu et al., 2023).

Practically, Frontier-Seg is positioned as a label-free terrain parsing module for off-road video when supervised models are unavailable, mismatched to the environment, or overly tied to a fixed ontology (Ellis et al., 29 Jul 2025). Its pseudo-labels are not human-readable semantic names; rather, they are discovered terrain classes or structural regions. The paper suggests that such outputs could feed traversability learning, map building, behavior cloning, novelty detection, or planners that reason over persistent terrain partitions instead of predefined semantic classes.

In this sense, Frontier-Seg occupies a distinct position relative to other uses of segmentation and frontier reasoning in robotics. It is not a map-frontier detector, not a semantic exploration policy, and not a supervised video segmentation network. Its contribution is a temporally aggregated, sequence-level clustering framework that converts foundation-model features into stable terrain partitions for mobile robot perception (Ellis et al., 29 Jul 2025).