PairGS: 3D Gaussian Segmentation

• PairGS is a framework for open-vocabulary 3D Gaussian segmentation that models pairwise relationships between Gaussians to improve instance coherence.
• It uses sparse graph construction with multi-view mask evidence and positional scaling to compute robust affinities efficiently.
• The framework achieves state-of-the-art performance and significant runtime improvements through a hierarchical clustering approach (TreeDBSCAN).

Searching arXiv for papers on PairGS and related open-vocabulary 3D Gaussian segmentation. PairGS is a framework for open-vocabulary 3D Gaussian segmentation that reframes Gaussian segmentation as modeling pairwise relations between Gaussians rather than predicting semantic or instance labels for each Gaussian independently (Cha et al., 1 Jul 2026). It is designed for reconstructed 3D scenes represented by 3D Gaussians and free-form text queries such as object, part, or group descriptions. The framework is motivated by the observation that prior approaches either embed language knowledge into individual Gaussians to improve query responsiveness or optimize per-Gaussian instance features to encode object identity, but these strategies may produce noisy Gaussian segmentations or rely on cost-inefficient per-scene optimization. PairGS instead uses a sparse relation graph, multi-view mask evidence, view contribution weights, and a hierarchical cluster tree for multi-granular querying.

1. Problem setting and formulation

Open-vocabulary 3D Gaussian segmentation requires language understanding for diverse queries and accurate separation of Gaussians along object boundaries. The task is to produce a 3D segmentation of Gaussians corresponding to a free-form text query and optionally project it to 2D. The difficulty is twofold: the method must support open-vocabulary language, and it must separate Gaussians cleanly at thin and touching structures such as a camera strap on the floor or a spoon touching a bowl (Cha et al., 1 Jul 2026).

PairGS is explicitly relation-centric. Its core question for two Gaussians $g_i$ and $g_j$ is how likely they are to belong to the same instance. The framework therefore builds a relation graph whose nodes are Gaussians and whose edges encode same-instance affinity. This differs from language-based 3DGS methods, which lift 2D semantic features into per-Gaussian representations and answer queries by per-Gaussian relevance scores, and from instance-feature-based methods, which optimize per-Gaussian instance features and then cluster them. PairGS uses semantic 3DGS features only to build node descriptors; the central representation is the graph of pairwise affinities.

A recurrent misunderstanding is to treat PairGS as a language-embedding method. The paper instead states that PairGS does not learn per-Gaussian language embeddings and does not optimize any scene-specific 3D network. It uses a pre-trained 3DGS for geometry and appearance, pre-trained SAM for masks, and CLIP for vision-language features, while graph building, affinity computation, and clustering are algorithmic rather than learned.

2. 3D Gaussian signals and relation evidence

The scene representation is a set

$S = \{ g_i \mid i = 1,\dots,N \}.$

Each Gaussian $g_i$ has a center position $p_i \in \mathbb{R}^3$, covariance $\Sigma_i \in \mathbb{R}^{3 \times 3}$, opacity $\alpha_i \in \mathbb{R}_+$, and color parameters, typically via SH features $c_i$. Rendering uses alpha blending along the view ray: $$C = \sum_{i=1}^{N} c_i \, \alpha_i T_i = \sum_{i=1}^{N} c_i w_i,$$ where

$$T_i = \prod_{j=1}^{i-1} (1 - \alpha_j), \quad w_i = \alpha_i T_i.$$

Here $g_j$0 is the contribution weight of Gaussian $g_j$1 to that pixel (Cha et al., 1 Jul 2026).

PairGS exploits signals that are specific to 3D Gaussian representations. The paper identifies two primary sources of relation evidence. The first is the per-view contribution vector: for Gaussian $g_j$2 and camera view $g_j$3, the scalar $g_j$4 summarizes visibility and strength in that view. The second is multi-view mask evidence: each view $g_j$5 has a 2D instance mask map

$g_j$6

with an instance index or special values $g_j$7 for boundary or uncertain regions and $g_j$8 for invisible regions.

From these data, PairGS constructs for each Gaussian a mask index vector $g_j$9 and a contribution vector $S = \{ g_i \mid i = 1,\dots,N \}.$0. This makes the relation evidence explicitly multi-view. Two Gaussians that consistently share the same mask ID in views where both are visible, and with high contribution, are likely to belong to the same instance. This suggests why the method can enforce instance-level coherence without a heavy optimization process.

3. Sparse graph construction and pairwise affinities

PairGS avoids $S = \{ g_i \mid i = 1,\dots,N \}.$1 all-pairs computation by a two-stage strategy: sparse edge proposal using low-dimensional node descriptors, followed by precise pairwise affinity computation only on the proposed edges (Cha et al., 1 Jul 2026). Each Gaussian receives a node descriptor

$S = \{ g_i \mid i = 1,\dots,N \}.$2

combining semantics and position.

The semantic descriptor $S = \{ g_i \mid i = 1,\dots,N \}.$3 is obtained by PCA from lifted SAM-masked CLIP features. PairGS uses contribution-weighted lifting, aggregating CLIP features over all pixels where the Gaussian contributes and weighting by $S = \{ g_i \mid i = 1,\dots,N \}.$4. PairGS-Fast uses center-based lifting at the projected Gaussian center. Position is scaled to balance semantic and positional neighborhoods. With mean $S = \{ g_i \mid i = 1,\dots,N \}.$5-NN distances

$S = \{ g_i \mid i = 1,\dots,N \}.$6

$S = \{ g_i \mid i = 1,\dots,N \}.$7

the scaling factor is

$S = \{ g_i \mid i = 1,\dots,N \}.$8

A directed $S = \{ g_i \mid i = 1,\dots,N \}.$9-NN graph is then built in the $g_i$0 space. For each node $g_i$1, directed edges $g_i$2 are created to its $g_i$3 nearest neighbors. The paper emphasizes that directedness and fixed out-degree stabilize clustering, especially near boundaries.

For each candidate edge $g_i$4, PairGS computes an affinity $g_i$5 from robust multi-view mask consistency. The mask index per Gaussian per view is reconciled from a center-based index and a max-based index: $g_i$6 If center and max disagree, the Gaussian is treated as noise in that view.

Let $g_i$7 be the comparable views in which both Gaussians are visible and non-noise. The affinity is

$g_i$8

This ratio uses the product $g_i$9 as pairwise evidence weight. If masks agree in views with high joint visibility, $p_i \in \mathbb{R}^3$0 approaches $p_i \in \mathbb{R}^3$1; frequent disagreement pushes it toward $p_i \in \mathbb{R}^3$2. Because affinities are computed only on $p_i \in \mathbb{R}^3$3 edges, the procedure is near-linear in $p_i \in \mathbb{R}^3$4 for fixed $p_i \in \mathbb{R}^3$5 and $p_i \in \mathbb{R}^3$6.

4. TreeDBSCAN hierarchy and open-vocabulary querying

Once the sparse directed graph and edge weights are available, PairGS builds a hierarchical cluster tree using TreeDBSCAN, a customized variant of DBSCAN on a directed graph (Cha et al., 1 Jul 2026). Standard DBSCAN uses a distance threshold $p_i \in \mathbb{R}^3$7 and a minimum number of neighbors. PairGS instead uses an affinity threshold $p_i \in \mathbb{R}^3$8, outgoing edges only, and sets $p_i \in \mathbb{R}^3$9 to the out-degree $\Sigma_i \in \mathbb{R}^{3 \times 3}$0, so a core point must have all outgoing edges with affinity at least $\Sigma_i \in \mathbb{R}^{3 \times 3}$1.

TreeDBSCAN uses a monotonic threshold schedule

$\Sigma_i \in \mathbb{R}^{3 \times 3}$2

At the first level it runs DBSCAN on the whole graph. At higher levels it recursively runs DBSCAN inside each parent cluster using a higher threshold. The hierarchy is then pruned with two filters. Redundancy filtering removes levels that do not actually split a parent cluster. Spurious cluster filtering measures the erosion rate

$\Sigma_i \in \mathbb{R}^{3 \times 3}$3

and discards a split if $\Sigma_i \in \mathbb{R}^{3 \times 3}$4, with default $\Sigma_i \in \mathbb{R}^{3 \times 3}$5.

The result is a single cohesive cluster tree with explicit parent-child relationships. Low thresholds produce coarse clusters such as large objects; higher thresholds produce finer clusters such as subparts or thin attachments. This organization is essential for multi-granular querying.

Language enters through CLIP. After clustering, PairGS constructs cluster-level semantic features by rasterizing a cluster map per view, associating each cluster with a dominant SAM mask through majority voting and IoU validation, defining a view weight

$\Sigma_i \in \mathbb{R}^{3 \times 3}$6

refining per-Gaussian semantic features

$\Sigma_i \in \mathbb{R}^{3 \times 3}$7

and aggregating them to a cluster embedding

$\Sigma_i \in \mathbb{R}^{3 \times 3}$8

A text query $\Sigma_i \in \mathbb{R}^{3 \times 3}$9 is encoded by CLIP, and cosine similarity to $\alpha_i \in \mathbb{R}_+$0 selects the most relevant cluster or clusters. The selected Gaussians form the 3D segmentation, and rendering them yields 2D masks. A plausible implication is that PairGS separates geometry-driven instance formation from language-driven retrieval: the cluster tree is built before query time, and the query selects from that hierarchy.

5. Empirical performance and runtime

The reported experiments cover open-vocabulary 3D object selection on LERF-OVS, open-vocabulary semantic segmentation on ScanNet, class-agnostic instance segmentation on ScanNet, ScanNet200, and qualitative examples on KITTI-360, Tanks & Temples, and Mip-NeRF 360 (Cha et al., 1 Jul 2026). PairGS achieves state-of-the-art results on open-vocabulary 3D Gaussian segmentation benchmarks, and the fast variant is over $\alpha_i \in \mathbb{R}_+$1 faster than optimization-based instance-feature approaches.

Benchmark	PairGS result	PairGS-Fast / note
LERF-OVS mIoU / [email protected]	60.4 / 68.2	56.8 / 63.6
ScanNet semantic, 10 classes mIoU / mAcc	53.0 / 68.1	time 0:36
ScanNet instance mIoU / mAcc	51.9 / 56.1	48.9 / 49.4
Runtime on LERF	3:02	0:58

On LERF-OVS, PairGS reports mIoU $\alpha_i \in \mathbb{R}_+$2 and [email protected] $\alpha_i \in \mathbb{R}_+$3, compared with LaGa $\alpha_i \in \mathbb{R}_+$4, COS3D $\alpha_i \in \mathbb{R}_+$5, InstanceGaussian $\alpha_i \in \mathbb{R}_+$6, OpenGaussian $\alpha_i \in \mathbb{R}_+$7, Dr.Splat $\alpha_i \in \mathbb{R}_+$8, Occam’s LGS $\alpha_i \in \mathbb{R}_+$9, and THGS $c_i$0. PairGS-Fast reports $c_i$1. On ScanNet semantic segmentation, PairGS reports mIoU $c_i$2, $c_i$3, and $c_i$4 for 10, 15, and 19 classes respectively, exceeding the listed baselines in those settings. On class-agnostic ScanNet instance segmentation, PairGS reports mIoU $c_i$5 and mAcc $c_i$6, with PairGS-Fast at $c_i$7.

Runtime is a central part of the framework’s positioning. On LERF, PairGS averages about $c_i$8 minutes per scene, with initialization about $c_i$9 seconds, graph plus clustering about $$C = \sum_{i=1}^{N} c_i \, \alpha_i T_i = \sum_{i=1}^{N} c_i w_i,$$0 seconds, and semantic refinement about $$C = \sum_{i=1}^{N} c_i \, \alpha_i T_i = \sum_{i=1}^{N} c_i w_i,$$1 seconds. PairGS-Fast averages about $$C = \sum_{i=1}^{N} c_i \, \alpha_i T_i = \sum_{i=1}^{N} c_i w_i,$$2 minutes. The paper contrasts this with OpenGaussian at about $$C = \sum_{i=1}^{N} c_i \, \alpha_i T_i = \sum_{i=1}^{N} c_i w_i,$$3 minutes, InstanceGaussian at about $$C = \sum_{i=1}^{N} c_i \, \alpha_i T_i = \sum_{i=1}^{N} c_i w_i,$$4 minutes, LaGa at about $$C = \sum_{i=1}^{N} c_i \, \alpha_i T_i = \sum_{i=1}^{N} c_i w_i,$$5 minutes, and COS3D at about $$C = \sum_{i=1}^{N} c_i \, \alpha_i T_i = \sum_{i=1}^{N} c_i w_i,$$6 minutes. THGS is listed at $$C = \sum_{i=1}^{N} c_i \, \alpha_i T_i = \sum_{i=1}^{N} c_i w_i,$$7 minutes. For large scenes with $$C = \sum_{i=1}^{N} c_i \, \alpha_i T_i = \sum_{i=1}^{N} c_i w_i,$$8–$$C = \sum_{i=1}^{N} c_i \, \alpha_i T_i = \sum_{i=1}^{N} c_i w_i,$$9 million Gaussians, full PairGS uses about $$T_i = \prod_{j=1}^{i-1} (1 - \alpha_j), \quad w_i = \alpha_i T_i.$$0–$$T_i = \prod_{j=1}^{i-1} (1 - \alpha_j), \quad w_i = \alpha_i T_i.$$1 minutes runtime and about $$T_i = \prod_{j=1}^{i-1} (1 - \alpha_j), \quad w_i = \alpha_i T_i.$$2 GB VRAM for graph and clustering.

6. Ablations, limitations, and significance

The ablations isolate the contribution of each major design choice. Removing positional scaling from node descriptors harms performance. Converting the graph to an undirected graph, either basic $$T_i = \prod_{j=1}^{i-1} (1 - \alpha_j), \quad w_i = \alpha_i T_i.$$3-NN or mutual $$T_i = \prod_{j=1}^{i-1} (1 - \alpha_j), \quad w_i = \alpha_i T_i.$$4-NN, degrades performance. Removing contribution weights reduces LERF mIoU from $$T_i = \prod_{j=1}^{i-1} (1 - \alpha_j), \quad w_i = \alpha_i T_i.$$5 to $$T_i = \prod_{j=1}^{i-1} (1 - \alpha_j), \quad w_i = \alpha_i T_i.$$6. Using only center-based mask indices reduces LERF mIoU to $$T_i = \prod_{j=1}^{i-1} (1 - \alpha_j), \quad w_i = \alpha_i T_i.$$7. Replacing TreeDBSCAN with single-level DBSCAN reduces LERF mIoU to $$T_i = \prod_{j=1}^{i-1} (1 - \alpha_j), \quad w_i = \alpha_i T_i.$$8, and replacing TreeDBSCAN with HDBSCAN reduces it to $$T_i = \prod_{j=1}^{i-1} (1 - \alpha_j), \quad w_i = \alpha_i T_i.$$9. Without pruning, the number of clusters grows linearly with levels, semantic refinement becomes expensive, and performance slightly degrades. The paper also states that varying hierarchy levels $g_j$00, out-degree $g_j$01, or PCA dimension $g_j$02 from $g_j$03 to $g_j$04 yields only modest performance changes (Cha et al., 1 Jul 2026).

The framework’s limitations are stated in terms of its dependencies. PairGS fundamentally relies on multi-view 2D masks from SAM. If a small object is never separated in any view’s mask, it cannot form a dedicated cluster. Extremely fine parts that only appear in subpart-level masks, when only whole-level masks are used, are treated as noise rather than clusters. Query success also depends on the quality of SAM-masked CLIP features; the paper notes that a query such as “egg” may retrieve one of two eggs because a sub-cluster with one egg has higher similarity than the parent cluster containing both. The hierarchy also has a granularity lower bound because it is built from a single mask level for efficiency.

Within the broader 3DGS literature, PairGS is positioned against three families. Relative to language-embedded 3DGS methods, its novelty is the shift from per-Gaussian semantics to relation-centric segmentation. Relative to instance-feature-based methods, its novelty is that it is training-free and uses pairwise affinities from multi-view masks and visibility rather than learned per-Gaussian instance features. Relative to point-based graph methods such as THGS, it explicitly exploits 3DGS-specific signals, especially view contributions, view-dependent visibility, and robust mask indexing. This suggests that PairGS is best understood not merely as a faster segmentation pipeline, but as a reformulation of open-vocabulary 3D Gaussian segmentation around pairwise same-instance estimation and hierarchical relation structure.