SeqAffordSplat: Sequential 3D Affordance

• SeqAffordSplat is a framework that reformulates 3D affordance reasoning from single-step, object-specific masks to sequential, scene-level interactions.
• The method uses an autoregressive LLM to decompose a high-level instruction into ordered primitive actions, each linked with a binary 3D Gaussian mask.
• It bridges geometric features with semantic cues from vision models, achieving state-of-the-art performance on complex, multi-object 3D scenes.

Searching arXiv for the main paper and closely related works mentioned in the provided data. SeqAffordSplat is a framework and benchmark for scene-level sequential affordance reasoning on 3D Gaussian Splatting (3DGS), introduced to address a limitation of prior 3D affordance methods that are fundamentally restricted to single-object, single-step interactions. It defines a setting in which a single high-level instruction is decomposed via an autoregressive process into an ordered sequence of primitive actions, each grounded to a binary mask over a 3D Gaussian scene. In the formulation reported by the paper, given a 3DGS scene $G = \{G_i\}_{i=1}^n$ and a succinct instruction $Q_{inst}$, the objective is to predict an ordered set of masks $M = (M_1,\dots,M_T)$ with $M_t \in \{0,1\}^n$ through a learned mapping $F$, namely $M = F(Q_{inst}, G)$ (Li et al., 31 Jul 2025).

1. Task formulation and problem setting

SeqAffordSplat introduces the Scene-Level Sequential 3D Gaussian Affordance Reasoning task. Traditional 3D affordance reasoning maps a language instruction to a single binary mask on an object’s surface or its point-cloud or 3DGS representation. By contrast, the new task treats a high-level instruction as a long-horizon objective that must be decomposed into an ordered sequence of primitive actions, each associated with a distinct affordance mask over the scene (Li et al., 31 Jul 2025).

The motivation is grounded in the observation that real-world tasks such as “warm up soup then serve” require long-horizon planning over multiple objects. Under this view, single-step, single-object affordance prediction is insufficient because it does not capture inter-object dependencies, temporal order, or shifting regions of interaction. SeqAffordSplat therefore reframes affordance reasoning as a scene-level sequential grounding problem intended for embodied agents operating in complex environments, including robotics and AR/VR (Li et al., 31 Jul 2025).

A plausible implication is that the benchmark targets not merely segmentation accuracy but compositional grounding: the system must infer both what primitive interactions are required and where each interaction should occur in the scene. This suggests a tighter coupling between instruction decomposition and 3D region prediction than in earlier single-step affordance formulations.

2. Benchmark design and evaluation protocol

The SeqAffordSplat benchmark is described as a large-scale resource with 1,800+ complex 3DGS environments, each composed by arranging multiple object instances drawn from 3D-AffordanceNet under varied transforms. The benchmark contains 21 object categories, each with 1–multiple instances per scene, and defines 18 atomic affordance types, including examples such as “sit,” “lift,” and “open.” It provides 14,000+ ground-truth binary masks and approximately 8,000 high-level, long-horizon instruction–sequence pairs generated via GPT-4o + human curation (Li et al., 31 Jul 2025).

The evaluation protocol is organized into three configurations. In Single-step (“Single”), the system predicts an unordered mask for a given primitive instruction. In Sequential w/ GT seq, the ground-truth decomposition is provided and the model predicts each mask in order. In Full Sequential, the model starts from a single high-level instruction and must infer both the decomposition and the masks. The reported metrics are mIoU/AUC/SIM/MAE for the single-step case and sIoU, sAUC, sSIM, sMAE for sequential evaluation, where predicted and ground-truth sequences are aligned by padding shorter sequences to equal length so that length errors are penalized (Li et al., 31 Jul 2025).

Component	Reported specification	Role
Scenes	1,800+ complex 3DGS environments	Benchmark substrate
Categories	21 object categories	Object diversity
Affordance types	18 atomic affordance types	Primitive interaction space
Masks	14,000+ ground-truth binary masks	Supervision target
Instructions	~8,000 high-level instruction–sequence pairs	Sequential language grounding

This benchmark structure makes the dataset substantially different from a standard referring-segmentation corpus. It evaluates long-horizon affordance understanding in scenes containing multiple objects and multiple potentially relevant interaction regions, rather than a single localized target region for a single primitive instruction.

3. SeqSplatNet architecture

SeqAffordSplat is accompanied by SeqSplatNet, described as an end-to-end framework that directly maps an instruction to a sequence of 3D affordance masks. The architecture consists of four components: (a) a pretrained 3DGS encoder, (b) an LLM with a special <SEG> token, (c) a conditional 3D affordance decoder, and (d) VFM Semantic Feature Injection (Li et al., 31 Jul 2025).

The language component receives the high-level instruction $Q_{inst}$ and autoregressively emits a stream of standard word tokens plus the learned special token <SEG>. Each occurrence of <SEG> signals that an affordance mask should now be decoded for the last primitive. The hidden state associated with that token, denoted $h_{seg} \in \mathbb{R}^d$, is used as a dynamic instruction query. The paper states the joint generation process as

$$P(tokens_{1 \dots t}, masks_{1 \dots t} \mid G) = \prod_{k=1}^{t} P(w_k \mid w_{<k}, masks_{<k}, G) \cdot P(mask_k \mid w_{\le k}, G).$$

The conditional 3D affordance decoder performs query-based segmentation. Each $h_{seg}$ acts as a query in a cross-attention mechanism over the 3DGS encoder’s geometric features $Q_{inst}$0. The decoder outputs a per-Gaussian probability vector $Q_{inst}$1, which is then binarized to obtain $Q_{inst}$2. The decoder is formalized as

$Q_{inst}$3

where $Q_{inst}$4 denotes injected semantic features (Li et al., 31 Jul 2025).

Training combines language modeling and mask segmentation in a single objective:

$Q_{inst}$5

Here, $Q_{inst}$6 is autoregressive cross-entropy over text tokens including <SEG>, and

$Q_{inst}$7

This design couples sequential language generation and 3D segmentation tightly: the LLM does not merely produce an instruction decomposition, but emits segmentation triggers whose hidden states directly parameterize the downstream mask decoder. This suggests that decomposition quality and grounding quality are co-dependent within the architecture.

4. Conditional geometric reconstruction pre-training

To endow the 3DGS encoder with a robust geometric prior, SeqSplatNet uses a self-supervised pre-training strategy termed Conditional Geometric Reconstruction. The pre-training task is framed as reconstructing a masked affordance region from an abstract mask embedding. Given a ground-truth affordance mask $Q_{inst}$8, a small mask encoder $Q_{inst}$9 maps it to an embedding $M = (M_1,\dots,M_T)$0, while the 3DGS encoder $M = (M_1,\dots,M_T)$1 produces $M = (M_1,\dots,M_T)$2. These are fused through cross-attention and decoded to a reconstructed mask $M = (M_1,\dots,M_T)$3 (Li et al., 31 Jul 2025).

The fusion and decoding steps are specified as

$M = (M_1,\dots,M_T)$4

and

$M = (M_1,\dots,M_T)$5

The reconstruction loss is

$M = (M_1,\dots,M_T)$6

where $M = (M_1,\dots,M_T)$7 is an optional regularizer, exemplified in the paper as weight decay on $M = (M_1,\dots,M_T)$8 (Li et al., 31 Jul 2025).

The stated purpose of this pre-training is to force the network to reconstruct complete affordance regions from known geometric observations, thereby learning to disentangle semantic concepts from geometry and to build strong geometric feature representations. In the paper’s interpretation, this improves downstream mask decoding. A plausible implication is that the pre-training stage regularizes the 3D representation toward affordance-complete regions rather than sparse local correspondences, which is particularly relevant when interaction regions span nontrivial geometry in cluttered multi-object scenes.

5. Semantic feature lifting from vision foundation models

SeqAffordSplat explicitly addresses the claim that 3DGS geometry alone can be ambiguous under complex instructions. To mitigate this, the method lifts 2D semantic features from a frozen Vision Foundation Model, with DINOv2 given as an example, and injects them into the 3D decoder (Li et al., 31 Jul 2025).

The procedure begins with multi-view feature extraction. The 3DGS scene is rendered from $M = (M_1,\dots,M_T)$9 camera views into RGB images $M_t \in \{0,1\}^n$0, and per-view features are extracted as

$M_t \in \{0,1\}^n$1

These 2D features are then lifted to the Gaussian representation. For each Gaussian $M_t \in \{0,1\}^n$2, the method collects the set $M_t \in \{0,1\}^n$3 of all pixels $M_t \in \{0,1\}^n$4 to which it contributes and accumulates its rendering weight $M_t \in \{0,1\}^n$5. The lifted semantic embedding is defined by alpha-weighting:

$M_t \in \{0,1\}^n$6

Finally, the semantic features are fused into the decoder at multiple layers by adding $M_t \in \{0,1\}^n$7 to the geometric feature $M_t \in \{0,1\}^n$8 through learned linear projections (Li et al., 31 Jul 2025).

This multi-scale semantic injection can be understood as a mechanism for compensating for cases in which geometric salience is insufficient to resolve the referential or functional content of an instruction. The paper’s design therefore combines geometry-dominant 3D encoding with semantics-dominant 2D foundation-model features, mediated through the rendering correspondence structure of 3DGS.

6. Experimental findings, examples, and limitations

The reported experimental results indicate state-of-the-art performance on the proposed benchmark. On SeqAffordSplat itself, the Single-step setting yields mIoU = 37.0 for the proposed method, compared with 30.5 (3DAffordSplat) and 31.3 (PointRefer), reported as a 6.5% gain over the best 3DGS baseline. In Sequential w/ GT seq, the method achieves sIoU = 36.0, which is +5.7 pts over PointRefer. In the Full end-to-end configuration, it reports sIoU = 26.2 versus 12.1 (SeqAfford), corresponding to a +14.1-point improvement. On the original 3DAffordSplat dataset, the method achieves mIoU = 40.2 vs. 30.3, described as setting a new single-step state-of-the-art (Li et al., 31 Jul 2025).

The qualitative examples highlighted by the paper are consistent with the quantitative framing. In a single-step example, the model is reported to precisely segment the liftable part of a bag, whereas baselines over-segment. In a sequential example for “listen to music using a laptop,” it generates a coherent 3-step plan—find plug → open lid → press play—and grounds each step in the corresponding 3D region (Li et al., 31 Jul 2025).

The paper also delineates limitations. First, there is LLM Dependence: performance hinges on the LLM’s ability to parse and decompose instructions, and very large models (e.g. Qwen3-8B) sometimes over-generate or hallucinate irrelevant primitives. Second, Real-Time Deployment remains challenging because the pipeline combines 3D encoding, multi-view rendering + VFM, and LLM inference, making on-robot inference computationally heavy. Future directions named in the paper include jointly fine-tuning end-to-end with reinforcement (robot execution feedback), extending to dynamic scenes and moving agents, and incorporating 4D temporal splatting for action prediction (Li et al., 31 Jul 2025).

These limitations clarify that the contribution is not solely a benchmark extension. It also exposes systems-level constraints that arise when sequential language reasoning, 3D scene grounding, and multi-view semantic lifting are combined in a single affordance pipeline.

7. Position within 3D affordance reasoning

SeqAffordSplat is situated as an explicit departure from prior methods that treat affordance grounding as a single-step interaction problem. The paper states that current methods based on 3D Gaussian Splatting are fundamentally limited to single-object, single-step interactions, and positions its contribution as advancing affordance reasoning from single-step interactions to complex sequential tasks at the scene level (Li et al., 31 Jul 2025).

Within that framing, the benchmark and model jointly target three coupled subproblems: decomposition of a high-level instruction into primitive actions, grounding of each primitive into a binary 3D Gaussian mask, and maintenance of temporal order across the predicted sequence. This suggests that SeqAffordSplat should be read not only as a segmentation benchmark but also as a structured reasoning benchmark over 3DGS scenes.

The broader significance lies in the use of 3D Gaussian Splatting as the underlying scene representation. Because the masks are point-wise over Gaussians and the semantic lifting step uses rendering weights to project 2D features back into the 3DGS domain, the method treats 3DGS as both a geometric carrier and a coordination space for multimodal reasoning. A plausible implication is that SeqAffordSplat contributes to the emerging view of 3DGS as an operational substrate for embodied scene understanding, rather than only a representation for novel-view synthesis.