---
title: 'SeqSplatNet: 3D Sequential Affordance Reasoning'
url: https://www.emergentmind.com/topics/seqsplatnet
type: topic
---

# SeqSplatNet: 3D Sequential Affordance Reasoning

SeqSplatNet is the main model introduced in “SeqAffordSplat: Scene-level Sequential Affordance Reasoning on 3D Gaussian Splatting” [2507.23772]. It is proposed for **scene-level sequential 3D Gaussian affordance reasoning**, where a complex natural-language instruction and a 3D Gaussian Splatting (3DGS) scene are mapped to an **ordered sequence of 3D affordance masks** rather than to a single actionable region. The model is an **end-to-end language-to-3D-mask sequence model**: it combines a 3DGS encoder, a causal large language model, and a conditional affordance decoder, and it unifies task decomposition and 3D grounding by interleaving normal text generation with special segmentation tokens.

## 1. Task definition and benchmark setting

SeqSplatNet is defined in tandem with **SeqAffordSplat**, which is the **benchmark/dataset and task setup**, not the model itself. The motivating claim is that prior 3DGS-based affordance reasoning methods are fundamentally limited to **single-object, single-step interactions**, whereas complex embodied tasks require **long-horizon, multi-object** reasoning in cluttered scenes. SeqAffordSplat is introduced to support that setting and contains more than **1,800 3DGS scenes**, over **14,000 affordance masks**, about **8,000 sequential instructions**, **21 object categories**, and **18 affordance types** [2507.23772].

Formally, a scene is represented as a 3DGS model
$$
\mathcal{G} = \{G_i\}_{i=1}^N,
$$
where each Gaussian primitive $G_i$ includes position, opacity, scale, rotation, and spherical harmonics coefficients. Given an instruction $Q_{\text{inst}}$, the objective is to predict an ordered sequence of binary masks
$$
\mathcal{M} = (M_1, M_2, \dots, M_T), \qquad M_t \in \{0,1\}^N,
$$
with each $M_t$ selecting the Gaussians corresponding to the functional region needed at step $t$. The desired mapping is
$$
\mathcal{M} = F(Q_{\text{inst}}, \mathcal{G}).
$$

The benchmark defines three evaluation settings. **Single** evaluates one-step grounding. **Sequential (with GT seq)** provides the ground-truth decomposition into primitive sub-instructions and evaluates perception and localization. **Sequential** requires the model to infer the full action sequence from a single high-level instruction. To evaluate sequential predictions, the benchmark introduces **sIoU, sAUC, sSIM, and sMAE**, computed after aligning predicted and ground-truth sequences by padding the shorter one with empty frames. This evaluation protocol makes temporal order part of the task definition rather than an optional post-processing detail.

## 2. End-to-end autoregressive formulation

At a high level, SeqSplatNet is an **end-to-end language-to-3D-mask sequence model** whose input is a 3DGS scene $\mathcal{G}$ plus a natural-language instruction $Q_{\text{inst}}$, and whose output is an ordered sequence of 3D affordance masks [2507.23772]. Its central mechanism is an interleaved autoregressive process: a causal large language model generates text tokens together with a special segmentation token, **`<SEG>`**. Whenever the model emits a `<SEG>` token, the associated hidden state
$$
h_{\text{seg}} \in \mathbb{R}^d
$$
is extracted and used as a **dynamic instruction/query embedding** for the conditional affordance decoder.

This design collapses planning and grounding into a single generation process. The model does not first output a fully explicit symbolic plan and then invoke a separate segmenter. Instead, primitive reasoning steps emerge during token generation, and segmentation is triggered inline by the special token. Because the language model is causal and autoregressive, each later `$h_{\text{seg}}$` depends on the instruction and on all previously generated context, including earlier primitive descriptions and segmentation events. This gives the architecture a direct mechanism for modeling temporal dependencies across steps and for switching interaction targets across different object instances.

The reported description organizes SeqSplatNet into three core modules—**3DGS encoder**, **large language model**, and **conditional affordance decoder**—plus two augmentations: **Conditional Geometric Reconstruction pre-training** and **VFM Semantic Feature Injection**. This suggests that the distinctive contribution lies not in any single component taken alone, but in the way causal language generation is coupled to query-conditioned 3D affordance decoding.

## 3. Geometric encoding and semantic feature injection

The underlying 3D representation is a set of Gaussians from 3D Gaussian Splatting. For geometric encoding, the model follows 3DAffordSplat in processing primarily the **geometric attributes** of the Gaussians—specifically **position, rotation, and scale**—using a **PointNet-based encoder** [2507.23772]. The encoder outputs per-Gaussian geometric features
$$
F_{\text{geo}} \in \mathbb{R}^{N \times d},
$$
and the reported implementation sets both geometric and semantic projected feature dimensions to **512**.

The language backbone used in the main experiments is **Qwen-3-0.6B**, fine-tuned with **LoRA**. The vocabulary is extended with the special token **`<SEG>`**, and inputs are formatted using the model’s official chat template. The LoRA target modules are `q_proj`, `k_proj`, `v_proj`, and `lm_head`, with **rank 8** and **alpha 16**. The combination of a small causal language model and explicit segmentation tokens makes the architecture structurally different from plan-then-execute pipelines in which language decomposition and 3D perception are trained or run separately.

To address semantic ambiguity, the model injects features from frozen 2D vision foundation models, specifically **DINOv2** and **CLIP**. The scene is rendered from $m$ viewpoints to obtain RGB images $I^{(v)}$, and a frozen VFM $\Psi_{\text{VFM}}$ produces 2D feature maps
$$
F^{(v)} = \Psi_{\text{VFM}}(I^{(v)}) \in \mathbb{R}^{H \times W \times d_{\text{sem}}}.
$$
These 2D semantic features are then lifted back onto the 3D Gaussians by a learning-free aggregation:
$$
f_i^{\text{sem}} =
\frac{\sum_{(v,p)\in \mathcal{S}_i} w_i(v,p)\,F_p^{(v)}}
     {\sum_{(v,p)\in \mathcal{S}_i} w_i(v,p)}.
$$
Here, $\mathcal{S}_i$ is the set of view-pixel pairs to which Gaussian $i$ contributes, and $w_i(v,p)$ is the rendering weight of Gaussian $i$ for pixel $p$ in view $v$. The resulting per-Gaussian semantic bank
$$
F_{\text{sem}} \in \mathbb{R}^{n \times d_{\text{sem}}}
$$
is injected into the conditional affordance decoder at **multiple scales using additive fusion**. Among the tested semantic sources, **DINOv2** performs best.

The reported description does not specify the exact number of decoder blocks, the exact cross-attention layout in the main affordance decoder, or a full transformer-style decoder equation. What is specified is that the decoder follows a **query-based segmentation paradigm**, uses the LLM-derived `$h_{\text{seg}}$` as a latent query, and predicts one 3D affordance mask for each emitted segmentation token.

## 4. Conditional Geometric Reconstruction, losses, and training

One of the main methodological additions is **Conditional Geometric Reconstruction (CGR)** pre-training, introduced because training a strong 3DGS encoder for scene-level affordance reasoning from scratch is difficult and annotation-hungry [2507.23772]. In CGR, given a scene $\mathcal{G}$ and a ground-truth affordance mask
$$
M^{\text{gt}} \in \{0,1\}^N,
$$
a **3DGS encoder** $\Phi_{\text{enc}}$ maps $\mathcal{G}$ to $F_{\text{geo}}$, and a **mask encoder** $\Phi_{\text{mask}}$ maps $M^{\text{gt}}$ to a single embedding $e_{\text{mask}}$. That embedding is used as a query to attend over geometric features:
$$
F_{\text{fused}} = \text{Attention}(Q=e_{\text{mask}}, K=F_{\text{geo}}, V=F_{\text{geo}}),
$$
and a decoder $\Phi_{\text{dec}}$ reconstructs the mask:
$$
\hat{M} = \Phi_{\text{dec}}(F_{\text{fused}}).
$$
The reported interpretation is that this teaches the network to map an abstract affordance concept to its corresponding spatial geometry and to build a stronger geometric prior before the main end-to-end training. The description does **not** provide an explicit standalone CGR loss formula.

For the main task, the training objective combines autoregressive language modeling with segmentation losses at segmentation steps:
$$
\mathcal{L}_{\text{total}} = \mathcal{L}_{\text{lang}} + \lambda_{\text{mask}} \sum_{t=1}^{T} \mathcal{L}_{\text{mask},t}.
$$
The segmentation loss is
$$
\mathcal{L}_{\text{mask}} =
\mathcal{L}_{\text{BCE}}(\hat{M}_t, M_t^{\text{gt}})
+
\mathcal{L}_{\text{Dice}}(\hat{M}_t, M_t^{\text{gt}}).
$$
No additional alignment losses, contrastive losses, or explicit language-vision matching losses are specified beyond these terms, and the exact value of $\lambda_{\text{mask}}$ is not given in the reported description.

The implementation details reported for training are specific. **Pre-training** runs for **10 epochs** with learning rate $1\times10^{-4}$. **Main training** runs for **50 epochs**, with non-LLM parameters using a decaying learning rate of $1\times10^{-4}$. The optimizer is **Adam with weight decay**, and the experiments are run on **8 GeForce RTX 3090 GPUs**. These details place SeqSplatNet in the regime of moderately sized LLM-conditioned 3D models rather than in the scale of very large multimodal foundation models.

## 5. Empirical performance and ablation structure

The benchmark results separate one-step localization, sequential localization with oracle decomposition, and full end-to-end sequential reasoning. The reported numbers are summarized below [2507.23772].

| Evaluation setting | SeqSplatNet | Main comparison |
|---|---|---|
| Single | mIoU **37.0**, AUC **94.0**, SIM **0.470**, MAE **0.049** | 3DAffordSplat: 30.5 / 92.7 / 0.395 / 0.065 |
| Sequential (with GT seq) | sIoU **36.0**, sAUC **95.6**, sSIM **0.457**, sMAE **0.036** | 3DAffordSplat: 26.1 / 91.2 / 0.343 / 0.072 |
| Sequential | sIoU **26.2**, sAUC **80.6**, sSIM **0.312**, sMAE **0.132** | SeqAfford: 12.1 / 73.0 / 0.122 / 0.230 |

In the **Single-step** setting, SeqSplatNet improves over **PointRefer** by **+5.7 mIoU** and over **3DAffordSplat** by **+6.5 mIoU**. In **Sequential (with GT seq)**, it isolates grounding quality under correct decomposition and remains substantially ahead of prior single-step baselines. In the full **Sequential** setting, it exceeds **SeqAfford** by **+14.1 sIoU**, more than doubling that baseline’s score. The drop from **36.0 sIoU** in **Sequential (with GT seq)** to **26.2 sIoU** in **Sequential** indicates that action decomposition remains materially harder than perception and localization alone. A plausible implication is that the architecture’s performance ceiling is jointly constrained by language planning quality and by 3D grounding quality.

The model also generalizes to the prior **3DAffordSplat** benchmark, where it reports **mIoU 40.2**, **AUC 89.3**, **SIM 0.530**, and **MAE 0.169**, compared with **3DAffordSplat** at **30.3 / 83.9 / 0.440 / 0.210**. This is a **+9.9 mIoU** gain, indicating that the method is not only useful for sequential reasoning but is also a stronger 3DGS affordance segmenter in the single-step case.

The ablation results isolate the two principal enhancements. On the **Sequential** task, removing both pre-training and semantic injection yields **20.3 sIoU**; adding **Conditional Geometric Reconstruction** increases this to **24.1**; adding **CLIP** features gives **24.2**; and adding **DINOv2** features yields **26.2**. The LLM ablation further shows that language reasoning quality is critical: **GPT2-small (0.1B)** gives **12.1 sIoU**, **Qwen3-0.6B** gives **26.2**, **Qwen3-1.7B** gives **26.4**, and **Qwen3-8B** gives **24.2**. The reported qualitative findings emphasize better precision on fine-grained actionable parts, coherent ordering of multi-step interactions, and robustness in cluttered multi-object scenes.

## 6. Limitations and interpretive boundaries

Several limits are explicit in the reported description. The model description does **not** provide the exact number of decoder blocks, the exact head structure, the exact cross-attention layout in the affordance decoder, or a precise transformer decoder formulation for the main task [2507.23772]. It also does **not** provide a formal standalone CGR loss formula, an explicit stopping criterion beyond standard autoregressive generation, or a formal account of how primitive sub-instructions are textually represented in the generated sequence. The dataset train/validation/test split ratios are likewise not specified in the reported text.

The qualitative discussion emphasizes the challenge of **semantic ambiguity** and **complex scene geometry**, which motivated semantic feature injection and CGR pre-training, but it does **not** provide a dedicated failure-case analysis. The absence of a systematic failure taxonomy means that issues such as long sequence drift, ambiguous instructions, or cross-object confusion are not catalogued in the reported description. This suggests that SeqSplatNet is best understood as a strong first formulation of sequential 3DGS affordance reasoning rather than as a fully stabilized endpoint for long-horizon embodied affordance inference.

A further interpretive boundary concerns what the model is and is not. SeqSplatNet is not described as a recurrent temporal model over dynamic scene streams, nor as a generic 3DGS feature-lifting framework. Its defining scope is **instruction-conditioned sequential affordance reasoning** in a fixed 3DGS scene, where sequence structure is expressed through autoregressive language generation and `<SEG>`-conditioned mask prediction.

## 7. Nomenclature and relation to adjacent splatting models

The term **SeqSplatNet** is specific and should be distinguished from several nearby but different lines of work. “Sequentially Aggregated Convolutional Networks” introduces **SeqConv**, **SeqConvB**, **WSeqConv**, **SeqResNet**, **SeqResNet-B**, and **SeqResNeXt**; it does **not** introduce an alternative name resembling SeqSplatNet [1811.10798]. “Fitting Segmentation Networks on Varying Image Resolutions using Splatting” introduces a **splat layer** and a UNet-based instantiation called **MeanSpaceNet**; it is technically relevant to splatting-based segmentation, but it does **not** mention SeqSplatNet [2206.06445].

Within later splatting literature, several methods are adjacent rather than identical. **SQS** is a **query-based splatting pre-training** framework for sparse perception models in autonomous driving, and it is explicitly **not** a sequence model [2509.16588]. **Splat Feature Solver** formulates feature lifting onto splats as a **sparse linear inverse problem**, providing a training-free mechanism for semantic assignment rather than a sequential affordance reasoner [2508.12216]. **Segment then Splat** is an object-centric, open-vocabulary Gaussian-splatting framework for static and dynamic scenes, emphasizing object-specific Gaussian sets and dynamic-scene support rather than autoregressive instruction decomposition [2503.22204].

These distinctions clarify the identity of SeqSplatNet. It is most precisely characterized by three coupled properties: it operates on **3D Gaussian Splatting** scenes, it performs **scene-level sequential affordance reasoning**, and it unifies decomposition and grounding through a **causal language model that emits interleaved text and `<SEG>` tokens**. In that sense, its contribution is not merely “splatting” and not merely “sequence modeling,” but the fusion of autoregressive language reasoning with conditional 3D affordance-mask generation.

Source: https://www.emergentmind.com/topics/seqsplatnet