Sequential 3D Gaussian Affordance Reasoning

• The paper introduces an approach that extends affordance prediction from single-step to sequential, mapping instructions to an ordered series of binary masks in complex 3D scenes.
• It employs an autoregressive decoder with a dual-encoder setup that integrates geometric pre-training and semantic feature injection, yielding significant performance gains.
• The SeqAffordSplat benchmark and SeqSplatNet demonstrate scene-level, long-horizon affordance reasoning, paving the way for adaptive evidence acquisition in multi-object environments.

Searching arXiv for the cited papers and closely related work to ground the article in current literature. Sequential 3D Gaussian affordance reasoning is the task of associating a succinct natural-language instruction with an ordered sequence of affordance regions in a 3D Gaussian Splatting (3DGS) scene, rather than with a single region on a single object. In its scene-level form, the input is a 3DGS scene and a high-level instruction, and the output is a temporally ordered series of binary masks over Gaussian primitives, each mask corresponding to the functional region for one primitive action. The formulation extends earlier affordance reasoning from single-object, single-step, single-mask prediction to scene-level, multi-object, long-horizon grounding, and it is instantiated by the SeqAffordSplat benchmark and the SeqSplatNet framework (Li et al., 31 Jul 2025). Its immediate antecedents include 3DGS-based single-step affordance reasoning in 3DAffordSplat (Wei et al., 15 Apr 2025) and point-cloud-based sequential affordance reasoning in SeqAfford (Yu et al., 2024). A subsequent line of work, A3R, further reframes affordance reasoning in 3D Gaussian scenes as sequential evidence acquisition rather than one-shot prediction (Li et al., 2 Apr 2026).

1. Conceptual emergence and problem shift

Earlier 3D affordance reasoning systems were largely formulated as localization of one functional region from one affordance instruction. In the 3DGS setting, 3DAffordSplat established the first large-scale multi-modal benchmark tailored for 3DGS-based affordance reasoning and introduced AffordSplatNet for language-guided affordance prediction over Gaussian primitives (Wei et al., 15 Apr 2025). In the point-cloud setting, SeqAfford generalized affordance segmentation to instructions that imply multiple actionable steps and multiple objects, introducing the task of Sequential 3D Affordance Reasoning and a benchmark of 180K instruction-point cloud pairs (Yu et al., 2024).

Sequential 3D Gaussian affordance reasoning inherits the sequential intent decomposition of SeqAfford but relocates it to scene-level 3DGS environments. The central change is not merely a new output format. The task assumes that real instructions are often high-level and composite, so the model must recover an ordered series of primitive actions, cope with dynamic changes in target regions across multiple objects, and reason over cluttered scenes rather than isolated objects (Li et al., 31 Jul 2025).

A common misconception is that this setting is equivalent to repeatedly applying a single-step affordance segmenter. The SeqAffordSplat formulation explicitly separates three cases—single-step prediction, sequential prediction with the ground-truth sequence of sub-instructions, and full end-to-end sequential prediction from one high-level instruction—thereby distinguishing language planning from mask prediction quality (Li et al., 31 Jul 2025). This distinction is central to the field’s recent trajectory.

2. Formal task definition in 3D Gaussian scenes

In the SeqAffordSplat formulation, the scene is a 3D Gaussian Splatting representation

$\mathcal{G} = \{G_i\}_{i=1}^{N},$

and the instruction is a succinct natural-language query $Q_{\text{inst}}$. The goal is to predict an ordered sequence of $T$ binary affordance masks

$\mathcal{M} = (M_1, M_2, \ldots, M_T),$

where each $M_t \in \{0,1\}^{N}$ identifies the subset of Gaussians corresponding to the functional region for the $t$-th primitive action. The task is defined as the mapping

$\mathcal{M} = F(Q_{\text{inst}}, \mathcal{G}).$

This definition makes the temporal and causal structure explicit: the model must determine which affordance region is relevant at each step, in the correct order (Li et al., 31 Jul 2025).

The formulation differs from conventional 3D affordance segmentation in three linked ways. First, it requires ordered primitive actions rather than one affordance prediction. Second, it permits multi-object interactions, so distinct steps may target different instances in the same scene. Third, it is explicitly scene-level, meaning that distractors, clutter, and multiple interactive parts are part of the problem rather than noise to be removed (Li et al., 31 Jul 2025).

SeqAfford in point clouds adopts an analogous sequential formulation, but its emphasis is on decomposing “cumbersome user intentions” into a series of segmentation maps over point clouds (Yu et al., 2024). SeqAffordSplat transfers that decomposition logic to 3D Gaussian scenes and couples it to scene composition, long-horizon instructions, and Gaussian-level mask prediction (Li et al., 31 Jul 2025).

3. Benchmark design, annotation, and evaluation

SeqAffordSplat is the benchmark introduced for scene-level sequential 3D Gaussian affordance reasoning. Its main benchmark summary reports 1800+ unique 3DGS scenes, 14,000+ affordance masks, 8,000+ sequential instructions, 21 object categories, and 18 affordance types (Li et al., 31 Jul 2025). The teaser also states “over 1,700 3DGS scenes and 12,000 instruction pairs,” while the main description gives the more complete summary above. The dataset is designed to be scene-level and long-horizon: scenes are manually composed from multiple objects, objects are transformed via translation, rotation, scaling, scenes include multiple interactive instance parts and distractor objects, and instructions describe multi-step action plans (Li et al., 31 Jul 2025).

The annotation format stores the ground truth as an ordered list of affordance masks, one mask per atomic step, with temporal and causal order explicitly encoded. The pipeline proceeds in four stages: object instances in scenes are matched to categories in 3DAffordSplat / 3D-AffordanceNet; point-wise affordance labels are projected onto the 3DGS representation; manual verification is performed with a custom 3D annotation tool; and sequential instructions are generated using GPT-4o, guided by visual context, textual context, role prompting, and goal specification, followed by human curation (Li et al., 31 Jul 2025).

The benchmark defines three evaluation settings.

Setting	Definition
Single	Predict one affordance mask for one explicit instruction
Sequential (with gt seq)	Predict affordance masks given the ground-truth sequence of sub-instructions
Sequential	Infer the sequence from a single high-level instruction

For single-step tasks, the benchmark uses mIoU, AUC, SIM, and MAE. For sequential tasks, it introduces sIoU, sAUC, sSIM, and sMAE. Evaluation aligns predicted and ground-truth sequences to equal length by padding the shorter sequence with empty frames, which penalizes sequence-length mismatch (Li et al., 31 Jul 2025). This evaluation protocol makes explicit that sequential affordance reasoning is not only a segmentation problem but also an ordering problem.

4. SeqSplatNet: autoregressive sequential decoding over 3DGS

SeqSplatNet is an end-to-end framework that directly maps an instruction to a sequence of 3D affordance masks. Its architecture combines an LLM for planning and reasoning, a 3DGS encoder for geometric representation, and a conditional affordance decoder for mask generation, together with two auxiliary components: Conditional Geometric Reconstruction pre-training and VFM semantic feature injection (Li et al., 31 Jul 2025).

The core mechanism is autoregressive generation with a special token <SEG>. The LLM produces an interleaved token sequence consisting of ordinary language tokens and <SEG> tokens. Each <SEG> token has three functions: it marks the end of one primitive action, triggers the affordance decoder, and provides a dynamic instruction embedding $h_{\text{seg}} \in \mathbb{R}^{d}$ from the token’s hidden state. Conceptually, the LLM first reasons over the instruction, emits a sequence of primitive-action text, and each <SEG> causes a corresponding 3D mask to be generated, yielding an ordered affordance-mask sequence (Li et al., 31 Jul 2025).

The 3DGS encoder is based on PointNet++ and consumes Gaussian geometric attributes—position, rotation, and scale—to produce geometric features

$F_{\text{geo}} \in \mathbb{R}^{N \times d}.$

The conditional affordance decoder is query-based: the hidden vector associated with a <SEG> token acts as a latent query to decode the mask for that step, conditioning on encoded geometry $F_{\text{geo}}$ and injected semantic features $Q_{\text{inst}}$0 (Li et al., 31 Jul 2025).

The joint training objective couples autoregressive language generation and segmentation:

$Q_{\text{inst}}$1

where $Q_{\text{inst}}$2 is autoregressive cross-entropy over the token sequence and

$Q_{\text{inst}}$3

The model is therefore jointly optimized for stepwise textual decomposition and mask prediction (Li et al., 31 Jul 2025).

Conditional Geometric Reconstruction

Conditional Geometric Reconstruction is a self-supervised or pre-training strategy for initializing the 3DGS encoder. Its purpose is to build a strong geometric prior for complex scenes, where training an encoder from scratch is difficult. The method uses a dual-encoder setup: $Q_{\text{inst}}$4 encodes scene geometry into $Q_{\text{inst}}$5, and $Q_{\text{inst}}$6 encodes a ground-truth mask into a compact conditional embedding $Q_{\text{inst}}$7. Attention then produces

$Q_{\text{inst}}$8

and the decoder reconstructs the target mask

$Q_{\text{inst}}$9

The intended effect is to teach the model to map an abstract affordance concept to spatial geometry, thereby learning a robust geometric prior (Li et al., 31 Jul 2025).

VFM semantic feature injection

Geometry alone is treated as insufficient for ambiguous language and complex scene semantics. SeqSplatNet therefore renders multi-view 2D images from the 3DGS scene, extracts dense features from frozen VFMs such as DINOv2 and CLIP, and lifts those features back to Gaussian space with a learning-free procedure similar to inverse rendering or LUDVIG. For Gaussian $T$0, the semantic feature is computed by weighted averaging over contributing view-pixel pairs:

$T$1

The resulting semantic feature bank is injected into the conditional decoder at multiple scales via additive fusion, supporting semantically consistent segmentation from coarse to fine stages (Li et al., 31 Jul 2025).

5. Quantitative performance and ablation structure

SeqSplatNet is evaluated against 3DAffordSplat, PointRefer, IAGNet, and SeqAfford on SeqAffordSplat. In the Single setting it achieves mIoU 37.0, AUC 94.0, SIM 0.470, and MAE 0.049. The best baseline, PointRefer, reaches mIoU 31.3, so SeqSplatNet reports +5.7 mIoU over PointRefer and +6.5 mIoU over 3DAffordSplat (Li et al., 31 Jul 2025). In Sequential (with GT seq) it achieves sIoU 36.0, sAUC 95.6, sSIM 0.457, and sMAE 0.036, which is +5.7 sIoU over the next best baseline. In the full end-to-end Sequential setting, it reaches sIoU 26.2, sAUC 80.6, sSIM 0.312, and sMAE 0.132; the baseline SeqAfford reports sIoU 12.1, giving a reported +14.1 sIoU gain (Li et al., 31 Jul 2025).

The model also transfers to the earlier single-step 3DGS benchmark. On 3DAffordSplat, SeqSplatNet reports mIoU 40.2, AUC 89.3, SIM 0.530, and MAE 0.169, exceeding the original 3DAffordSplat result of 30.3 mIoU by 9.9 points (Li et al., 31 Jul 2025, Wei et al., 15 Apr 2025). This positions sequentially trained Gaussian-scene reasoning as useful beyond the specific long-horizon benchmark.

Ablation results decompose the contribution of geometric pre-training and semantic injection. On the sequential task, No pretrain / no feature injection gives sIoU 20.3, sAUC 76.3, sSIM 0.229, sMAE 0.169. Pretrain only improves to sIoU 24.1, sAUC 78.5, sSIM 0.302, sMAE 0.141. Pretrain + CLIP yields sIoU 24.2, sAUC 79.1, sSIM 0.290, sMAE 0.141, while Pretrain + DINOv2 gives the best result: sIoU 26.2, sAUC 80.6, sSIM 0.312, sMAE 0.132 (Li et al., 31 Jul 2025). The reported interpretation is that both geometry prior and semantics matter, and that DINOv2 outperforms CLIP for this task.

The LLM backbone ablation is similarly non-monotonic. GPT2-small (0.1B) obtains sIoU 12.1, Qwen3-0.6B reaches 26.2, Qwen3-1.7B reaches 26.4, and Qwen3-8B drops to 24.2 (Li et al., 31 Jul 2025). The explicit finding is that bigger is not always better in this setting, and that Qwen3-0.6B gives the best trade-off and best overall metrics.

For context, the point-cloud predecessor SeqAfford reports 19.5 mIoU in the Single Affordance Seen setting, 13.8 mIoU in the Single Affordance Unseen setting, and 14.6 mIoU in its Sequential setting, with stronger AUC and SIM gains than prior point-cloud baselines (Yu et al., 2024). The comparison indicates that transferring sequential affordance reasoning into scene-level 3DGS substantially changes both the input representation and the attainable performance regime.

6. Interpretation, limits, and later developments

Qualitative results in SeqAffordSplat show that the framework can handle single-step affordance localization with fine precision, multi-step sequential grounding from a succinct instruction, and instructions requiring multiple distinct objects and actions. Examples in the text include “listen to music using a laptop” and operating a microwave to warm food in a bowl, both of which require decomposition into multiple affordance-grounding steps (Li et al., 31 Jul 2025). This supports the interpretation that the framework is not merely segmenting a static region; it is performing language planning, causal action decomposition, scene-level grounding, and sequential mask generation.

At the same time, the broader literature draws clear boundaries around what these systems do not yet solve. SeqAfford notes that the sequential task is handled through tokenized segmentation reasoning and that the model does not appear to perform explicit symbolic planning or verify action feasibility in a robot dynamics sense (Yu et al., 2024). The same paper identifies future work in scene-level fine-grained reasoning for more sophisticated and context-aware affordance segmentation (Yu et al., 2024). In this sense, sequential 3D Gaussian affordance reasoning is best understood as a representational and reasoning layer for embodied systems, not as a complete manipulation stack.

A later development, A3R, modifies the meaning of “sequential” in 3D Gaussian affordance reasoning. Rather than decomposing a high-level instruction into multiple affordance masks, A3R casts fine-grained affordance localization as a POMDP over a fixed 3DGS scene, maintains a Gaussian-indexed belief map $T$2, and uses an MLLM-based policy to choose among evidence-acquisition operators Inst, Part, SAM2D, Zoom, and Term (Li et al., 2 Apr 2026). Its reported results show that agentic evidence acquisition outperforms static one-shot baselines on both 3DAffordSplat and SeqAffordSplat; for example, on SeqAffordSplat Seen it reports 28.28 mIoU for A3R (RL) versus 21.04 mIoU for AffordSplatNet, and on SeqAffordSplat Unseen it reports 25.56 mIoU versus 9.01 (Li et al., 2 Apr 2026). A plausible implication is that the field is splitting into at least two complementary notions of sequentiality: instruction decomposition into ordered affordance masks and adaptive evidence acquisition under uncertainty.

Taken together, these works define a progression from single-step object-centric affordance segmentation in 3DGS (Wei et al., 15 Apr 2025), to scene-level long-horizon sequential grounding (Li et al., 31 Jul 2025), to agentic cross-dimensional evidence acquisition in complex 3D Gaussian scenes (Li et al., 2 Apr 2026). This suggests that sequential 3D Gaussian affordance reasoning has become a distinct research program centered on how language, geometry, semantics, and temporal structure should be coupled for embodied understanding.