SeqSplatNet: 3D Sequential Affordance Reasoning

• SeqSplatNet is a 3D model that maps natural-language instructions to a sequential set of 3D affordance masks using an end-to-end autoregressive approach.
• It integrates a 3D Gaussian Splatting encoder, a causal large language model, and a conditional affordance decoder to unify planning and 3D grounding.
• The method leverages Conditional Geometric Reconstruction pre-training and semantic feature injection (e.g., DINOv2) to enhance segmentation accuracy in cluttered scenes.

SeqSplatNet is the main model introduced in “SeqAffordSplat: Scene-level Sequential Affordance Reasoning on 3D Gaussian Splatting” (Li et al., 31 Jul 2025). It is proposed for scene-level sequential 3D Gaussian affordance reasoning, where a complex natural-language instruction and a 3D Gaussian Splatting (3DGS) scene are mapped to an ordered sequence of 3D affordance masks rather than to a single actionable region. The model is an end-to-end language-to-3D-mask sequence model: it combines a 3DGS encoder, a causal LLM, and a conditional affordance decoder, and it unifies task decomposition and 3D grounding by interleaving normal text generation with special segmentation tokens.

1. Task definition and benchmark setting

SeqSplatNet is defined in tandem with SeqAffordSplat, which is the benchmark/dataset and task setup, not the model itself. The motivating claim is that prior 3DGS-based affordance reasoning methods are fundamentally limited to single-object, single-step interactions, whereas complex embodied tasks require long-horizon, multi-object reasoning in cluttered scenes. SeqAffordSplat is introduced to support that setting and contains more than 1,800 3DGS scenes, over 14,000 affordance masks, about 8,000 sequential instructions, 21 object categories, and 18 affordance types (Li et al., 31 Jul 2025).

Formally, a scene is represented as a 3DGS model

$\mathcal{G} = \{G_i\}_{i=1}^N,$

where each Gaussian primitive $G_i$ includes position, opacity, scale, rotation, and spherical harmonics coefficients. Given an instruction $Q_{\text{inst}}$, the objective is to predict an ordered sequence of binary masks

$$\mathcal{M} = (M_1, M_2, \dots, M_T), \qquad M_t \in \{0,1\}^N,$$

with each $M_t$ selecting the Gaussians corresponding to the functional region needed at step $t$. The desired mapping is

$\mathcal{M} = F(Q_{\text{inst}}, \mathcal{G}).$

The benchmark defines three evaluation settings. Single evaluates one-step grounding. Sequential (with GT seq) provides the ground-truth decomposition into primitive sub-instructions and evaluates perception and localization. Sequential requires the model to infer the full action sequence from a single high-level instruction. To evaluate sequential predictions, the benchmark introduces sIoU, sAUC, sSIM, and sMAE, computed after aligning predicted and ground-truth sequences by padding the shorter one with empty frames. This evaluation protocol makes temporal order part of the task definition rather than an optional post-processing detail.

2. End-to-end autoregressive formulation

At a high level, SeqSplatNet is an end-to-end language-to-3D-mask sequence model whose input is a 3DGS scene $\mathcal{G}$ plus a natural-language instruction $Q_{\text{inst}}$, and whose output is an ordered sequence of 3D affordance masks (Li et al., 31 Jul 2025). Its central mechanism is an interleaved autoregressive process: a causal LLM generates text tokens together with a special segmentation token, <SEG>. Whenever the model emits a <SEG> token, the associated hidden state

$h_{\text{seg}} \in \mathbb{R}^d$

is extracted and used as a dynamic instruction/query embedding for the conditional affordance decoder.

This design collapses planning and grounding into a single generation process. The model does not first output a fully explicit symbolic plan and then invoke a separate segmenter. Instead, primitive reasoning steps emerge during token generation, and segmentation is triggered inline by the special token. Because the LLM is causal and autoregressive, each later $G_i$0 depends on the instruction and on all previously generated context, including earlier primitive descriptions and segmentation events. This gives the architecture a direct mechanism for modeling temporal dependencies across steps and for switching interaction targets across different object instances.

The reported description organizes SeqSplatNet into three core modules—3DGS encoder, LLM, and conditional affordance decoder—plus two augmentations: Conditional Geometric Reconstruction pre-training and VFM Semantic Feature Injection. This suggests that the distinctive contribution lies not in any single component taken alone, but in the way causal language generation is coupled to query-conditioned 3D affordance decoding.

3. Geometric encoding and semantic feature injection

The underlying 3D representation is a set of Gaussians from 3D Gaussian Splatting. For geometric encoding, the model follows 3DAffordSplat in processing primarily the geometric attributes of the Gaussians—specifically position, rotation, and scale—using a PointNet-based encoder (Li et al., 31 Jul 2025). The encoder outputs per-Gaussian geometric features

$G_i$1

and the reported implementation sets both geometric and semantic projected feature dimensions to 512.

The language backbone used in the main experiments is Qwen-3-0.6B, fine-tuned with LoRA. The vocabulary is extended with the special token <SEG>, and inputs are formatted using the model’s official chat template. The LoRA target modules are q_proj, k_proj, v_proj, and lm_head, with rank 8 and alpha 16. The combination of a small causal LLM and explicit segmentation tokens makes the architecture structurally different from plan-then-execute pipelines in which language decomposition and 3D perception are trained or run separately.

To address semantic ambiguity, the model injects features from frozen 2D vision foundation models, specifically DINOv2 and CLIP. The scene is rendered from $G_i$2 viewpoints to obtain RGB images $G_i$3, and a frozen VFM $G_i$4 produces 2D feature maps

$G_i$5

These 2D semantic features are then lifted back onto the 3D Gaussians by a learning-free aggregation:

$G_i$6

Here, $G_i$7 is the set of view-pixel pairs to which Gaussian $G_i$8 contributes, and $G_i$9 is the rendering weight of Gaussian $Q_{\text{inst}}$0 for pixel $Q_{\text{inst}}$1 in view $Q_{\text{inst}}$2. The resulting per-Gaussian semantic bank

$Q_{\text{inst}}$3

is injected into the conditional affordance decoder at multiple scales using additive fusion. Among the tested semantic sources, DINOv2 performs best.

The reported description does not specify the exact number of decoder blocks, the exact cross-attention layout in the main affordance decoder, or a full transformer-style decoder equation. What is specified is that the decoder follows a query-based segmentation paradigm, uses the LLM-derived $Q_{\text{inst}}$4 as a latent query, and predicts one 3D affordance mask for each emitted segmentation token.

4. Conditional Geometric Reconstruction, losses, and training

One of the main methodological additions is Conditional Geometric Reconstruction (CGR) pre-training, introduced because training a strong 3DGS encoder for scene-level affordance reasoning from scratch is difficult and annotation-hungry (Li et al., 31 Jul 2025). In CGR, given a scene $Q_{\text{inst}}$5 and a ground-truth affordance mask

$Q_{\text{inst}}$6

a 3DGS encoder $Q_{\text{inst}}$7 maps $Q_{\text{inst}}$8 to $Q_{\text{inst}}$9, and a mask encoder $$\mathcal{M} = (M_1, M_2, \dots, M_T), \qquad M_t \in \{0,1\}^N,$$0 maps $$\mathcal{M} = (M_1, M_2, \dots, M_T), \qquad M_t \in \{0,1\}^N,$$1 to a single embedding $$\mathcal{M} = (M_1, M_2, \dots, M_T), \qquad M_t \in \{0,1\}^N,$$2. That embedding is used as a query to attend over geometric features:

$$\mathcal{M} = (M_1, M_2, \dots, M_T), \qquad M_t \in \{0,1\}^N,$$3

and a decoder $$\mathcal{M} = (M_1, M_2, \dots, M_T), \qquad M_t \in \{0,1\}^N,$$4 reconstructs the mask:

$$\mathcal{M} = (M_1, M_2, \dots, M_T), \qquad M_t \in \{0,1\}^N,$$5

The reported interpretation is that this teaches the network to map an abstract affordance concept to its corresponding spatial geometry and to build a stronger geometric prior before the main end-to-end training. The description does not provide an explicit standalone CGR loss formula.

For the main task, the training objective combines autoregressive language modeling with segmentation losses at segmentation steps:

$$\mathcal{M} = (M_1, M_2, \dots, M_T), \qquad M_t \in \{0,1\}^N,$$6

The segmentation loss is

$$\mathcal{M} = (M_1, M_2, \dots, M_T), \qquad M_t \in \{0,1\}^N,$$7

No additional alignment losses, contrastive losses, or explicit language-vision matching losses are specified beyond these terms, and the exact value of $$\mathcal{M} = (M_1, M_2, \dots, M_T), \qquad M_t \in \{0,1\}^N,$$8 is not given in the reported description.

The implementation details reported for training are specific. Pre-training runs for 10 epochs with learning rate $$\mathcal{M} = (M_1, M_2, \dots, M_T), \qquad M_t \in \{0,1\}^N,$$9. Main training runs for 50 epochs, with non-LLM parameters using a decaying learning rate of $M_t$0. The optimizer is Adam with weight decay, and the experiments are run on 8 GeForce RTX 3090 GPUs. These details place SeqSplatNet in the regime of moderately sized LLM-conditioned 3D models rather than in the scale of very large multimodal foundation models.

5. Empirical performance and ablation structure

The benchmark results separate one-step localization, sequential localization with oracle decomposition, and full end-to-end sequential reasoning. The reported numbers are summarized below (Li et al., 31 Jul 2025).

Evaluation setting	SeqSplatNet	Main comparison
Single	mIoU 37.0, AUC 94.0, SIM 0.470, MAE 0.049	3DAffordSplat: 30.5 / 92.7 / 0.395 / 0.065
Sequential (with GT seq)	sIoU 36.0, sAUC 95.6, sSIM 0.457, sMAE 0.036	3DAffordSplat: 26.1 / 91.2 / 0.343 / 0.072
Sequential	sIoU 26.2, sAUC 80.6, sSIM 0.312, sMAE 0.132	SeqAfford: 12.1 / 73.0 / 0.122 / 0.230

In the Single-step setting, SeqSplatNet improves over PointRefer by +5.7 mIoU and over 3DAffordSplat by +6.5 mIoU. In Sequential (with GT seq), it isolates grounding quality under correct decomposition and remains substantially ahead of prior single-step baselines. In the full Sequential setting, it exceeds SeqAfford by +14.1 sIoU, more than doubling that baseline’s score. The drop from 36.0 sIoU in Sequential (with GT seq) to 26.2 sIoU in Sequential indicates that action decomposition remains materially harder than perception and localization alone. A plausible implication is that the architecture’s performance ceiling is jointly constrained by language planning quality and by 3D grounding quality.

The model also generalizes to the prior 3DAffordSplat benchmark, where it reports mIoU 40.2, AUC 89.3, SIM 0.530, and MAE 0.169, compared with 3DAffordSplat at 30.3 / 83.9 / 0.440 / 0.210. This is a +9.9 mIoU gain, indicating that the method is not only useful for sequential reasoning but is also a stronger 3DGS affordance segmenter in the single-step case.

The ablation results isolate the two principal enhancements. On the Sequential task, removing both pre-training and semantic injection yields 20.3 sIoU; adding Conditional Geometric Reconstruction increases this to 24.1; adding CLIP features gives 24.2; and adding DINOv2 features yields 26.2. The LLM ablation further shows that language reasoning quality is critical: GPT2-small (0.1B) gives 12.1 sIoU, Qwen3-0.6B gives 26.2, Qwen3-1.7B gives 26.4, and Qwen3-8B gives 24.2. The reported qualitative findings emphasize better precision on fine-grained actionable parts, coherent ordering of multi-step interactions, and robustness in cluttered multi-object scenes.

6. Limitations and interpretive boundaries

Several limits are explicit in the reported description. The model description does not provide the exact number of decoder blocks, the exact head structure, the exact cross-attention layout in the affordance decoder, or a precise transformer decoder formulation for the main task (Li et al., 31 Jul 2025). It also does not provide a formal standalone CGR loss formula, an explicit stopping criterion beyond standard autoregressive generation, or a formal account of how primitive sub-instructions are textually represented in the generated sequence. The dataset train/validation/test split ratios are likewise not specified in the reported text.

The qualitative discussion emphasizes the challenge of semantic ambiguity and complex scene geometry, which motivated semantic feature injection and CGR pre-training, but it does not provide a dedicated failure-case analysis. The absence of a systematic failure taxonomy means that issues such as long sequence drift, ambiguous instructions, or cross-object confusion are not catalogued in the reported description. This suggests that SeqSplatNet is best understood as a strong first formulation of sequential 3DGS affordance reasoning rather than as a fully stabilized endpoint for long-horizon embodied affordance inference.

A further interpretive boundary concerns what the model is and is not. SeqSplatNet is not described as a recurrent temporal model over dynamic scene streams, nor as a generic 3DGS feature-lifting framework. Its defining scope is instruction-conditioned sequential affordance reasoning in a fixed 3DGS scene, where sequence structure is expressed through autoregressive language generation and <SEG>-conditioned mask prediction.

7. Nomenclature and relation to adjacent splatting models

The term SeqSplatNet is specific and should be distinguished from several nearby but different lines of work. “Sequentially Aggregated Convolutional Networks” introduces SeqConv, SeqConvB, WSeqConv, SeqResNet, SeqResNet-B, and SeqResNeXt; it does not introduce an alternative name resembling SeqSplatNet (Huang et al., 2018). “Fitting Segmentation Networks on Varying Image Resolutions using Splatting” introduces a splat layer and a UNet-based instantiation called MeanSpaceNet; it is technically relevant to splatting-based segmentation, but it does not mention SeqSplatNet (Brudfors et al., 2022).

Within later splatting literature, several methods are adjacent rather than identical. SQS is a query-based splatting pre-training framework for sparse perception models in autonomous driving, and it is explicitly not a sequence model (Zhang et al., 20 Sep 2025). Splat Feature Solver formulates feature lifting onto splats as a sparse linear inverse problem, providing a training-free mechanism for semantic assignment rather than a sequential affordance reasoner (Xiong et al., 17 Aug 2025). Segment then Splat is an object-centric, open-vocabulary Gaussian-splatting framework for static and dynamic scenes, emphasizing object-specific Gaussian sets and dynamic-scene support rather than autoregressive instruction decomposition (Lu et al., 28 Mar 2025).

These distinctions clarify the identity of SeqSplatNet. It is most precisely characterized by three coupled properties: it operates on 3D Gaussian Splatting scenes, it performs scene-level sequential affordance reasoning, and it unifies decomposition and grounding through a causal LLM that emits interleaved text and <SEG> tokens. In that sense, its contribution is not merely “splatting” and not merely “sequence modeling,” but the fusion of autoregressive language reasoning with conditional 3D affordance-mask generation.