---
title: 'LaSSM: Efficient 3D Instance Segmentation'
url: https://www.emergentmind.com/topics/lassm
type: topic
---

# LaSSM: Efficient 3D Instance Segmentation

LaSSM most commonly denotes a query-based 3D instance segmentation framework for point clouds that is designed to be both simple and highly efficient while still reaching or surpassing state of the art on challenging large-scale indoor benchmarks [2602.11007]. In that usage, it addresses two core issues in current query-based 3D scene instance segmentation: query initialization in sparse point clouds and efficient query refinement without quadratic-cost attention over many queries and points. The acronym also appears in adjacent multimodal literature in two potentially confusing ways: the similarly spelled **LLaSM** refers to a “Large Language and Speech Model” for speech-and-text instruction following [2308.15930], and “LaSSM” is used informally in one paper to denote a **state-space Large Audio Language Model** [2411.15685]. This polysemy makes disambiguation necessary in technical discussion.

## 1. Nomenclature and disambiguation

The primary arXiv paper whose title explicitly uses **LaSSM** is “LaSSM: Efficient Semantic-Spatial Query Decoding via Local Aggregation and State Space Models for 3D Instance Segmentation” [2602.11007]. In that work, LaSSM is a 3D scene instance segmentation framework operating on voxelized point clouds, superpoints, and query decoders.

Two neighboring usages require explicit separation. First, **LLaSM**—with a double “L” as in “LLaMA”—stands for **Large Language and Speech Model** and is a speech–language multimodal large language model designed to follow speech-and-text instructions [2308.15930]. The source explicitly states that the spelling “LaSSM” does not appear in that paper and that the model is consistently named **LLaSM**. Second, “State-Space Large Audio Language Models” introduces the first fully state-space LALM and notes that one can denote it informally as a **LaSSM**, but the paper’s formal terms are **Hybrid-LALM** and **ssLALM** rather than a canonical model name “LaSSM” [2411.15685].

| Term | Meaning | Primary domain |
|---|---|---|
| **LaSSM** | Efficient semantic-spatial query decoding for 3D instance segmentation | 3D point clouds |
| **LLaSM** | Large Language and Speech Model | Speech + text |
| **ssLALM / informal “LaSSM”** | State-space large audio language model | Audio + language |

A common misconception is therefore to treat all three as variants of the same model family. The available evidence indicates instead that they belong to different research lines: query-based 3D scene instance segmentation, speech–language instruction following, and audio–language modeling with state-space backbones.

## 2. Problem setting and design motivation

In the 3D vision usage, LaSSM is situated within query-based 3D scene instance segmentation from point clouds, where the goal is to predict, for each object instance in a scene, a binary mask over points or voxels and a semantic label for that instance [2602.11007]. Query-based methods follow the DETR paradigm: a fixed set of queries is fed into a transformer-like decoder, which interacts with scene features and outputs a set of instance predictions.

The LaSSM paper identifies two bottlenecks. The first is the **query initialization dilemma** in sparse 3D data. Geometry-based strategies such as farthest point sampling provide coverage without discriminativeness, while semantic-confidence selection provides discriminativeness without coverage. Learnable query embeddings also ignore scene content and are the same for all scenes. The second bottleneck is the **computational burden of attention-based decoders**. In 3D, hundreds of queries, multiple decoder layers, and large scene features make cross-attention and self-attention expensive; cross-attention scales roughly as $O(q \times n)$ and self-attention as $O(q^2)$ [2602.11007].

LaSSM is designed to resolve these issues by constructing queries directly from scene-adaptive superpoints, balancing semantic confidence and spatial distribution, and by replacing heavy attention with a more efficient structure based on state space models and local aggregation [2602.11007]. The design goals are to resolve the initialization dilemma, replace heavy attention with a more efficient structure, explicitly integrate coordinates into query refinement, and stay simple and practical while delivering state-of-the-art performance, particularly on large, high-resolution indoor scenes.

A plausible implication is that LaSSM should be read less as a purely architectural novelty in decoder design and more as a coordinated proposal about **where queries come from**, **how they are spatially updated**, and **how refinement complexity is controlled**.

## 3. Core architecture

LaSSM is organized into three main parts: a **Sparse 3D U-Net backbone**, a **hierarchical semantic-spatial query initializer**, and a **coordinate-guided SSM decoder** [2602.11007]. The input is a voxelized point cloud
$$
X \in \mathbb{R}^{n \times 6},
$$
where each voxel has coordinates $(x,y,z)$ and color $(r,g,b)$. The sparse 3D U-Net extracts voxel features
$$
F = Backbone(X) \in \mathbb{R}^{n \times d_o}.
$$

Superpoint pooling then groups voxels into superpoints based on geometry, using Felzenszwalb-Huttenlocher graph segmentation / superpoint pooling:
$$
F_s = SpPool(F), \quad C_s = SpPool([x,y,z]).
$$
Here, $F_s \in \mathbb{R}^{s \times d_o}$ are superpoint features and $C_s \in \mathbb{R}^{s \times 3}$ are their coordinates [2602.11007]. These superpoints are the basis both for query initialization and for local aggregation in the decoder.

From these superpoint features, the initializer selects a subset of superpoints driven by semantic confidence and then applies farthest point sampling to ensure spatial coverage. It outputs query contents
$$
Q \in \mathbb{R}^{q \times d}
$$
and query coordinates
$$
C_c \in \mathbb{R}^{q \times 3}.
$$
These queries are then refined by the coordinate-guided SSM decoder, which applies local aggregation, a spatial dual-path SSM block, and prediction heads for masks, classes, and center offsets [2602.11007].

The mask head $\phi_{\text{mask}}$ generates a per-query mask embedding, with masks computed as
$$
M = \{\sigma( F_s \cdot \phi_\text{mask}(Q_{\text{ref}})^\top )_{i,j} > \tau\},
$$
while the center offset head updates coordinates according to
$$
C_c' = C_c + \phi_\text{cen}(Q_{\text{ref}}).
$$
Query refinement is repeated across $L$ layers, with $L = 6$ in the experiments [2602.11007].

This architecture combines superpoint-level scene abstraction with query-level instance prediction. The paper positions this combination as the core novelty: a superpoint-based, semantic-spatial query initializer, a local aggregation scheme, and a spatial dual-path SSM block [2602.11007].

## 4. Hierarchical semantic-spatial query initialization

The query initializer is intended to resolve the tension between semantic discriminativeness and spatial coverage. It operates on superpoints, which are small, geometrically coherent regions of the scene obtained via graph-based segmentation on the voxel grid [2602.11007].

LaSSM first applies a lightweight MLP classifier $\phi_{\text{sem}}$ to superpoint features:
$$
W = \phi_\text{sem}(F_s) \in \mathbb{R}^{s \times (c+1)},
$$
where $c$ is the number of instance classes and the extra dimension is background. After softmax, each superpoint receives a semantic activation score for being any non-background class:
$$
w_i^* = \underset{j \in \mathcal{C}}{\max}\{w_{i,j} \mid w_{i,j} \in W_i\}, \quad \mathcal{C} = \{1,\dots,c\}.
$$
This yields a scalar confidence of “being part of some instance” for each superpoint [2602.11007].

Instead of applying a fixed threshold, LaSSM keeps the top-$m$ superpoints according to an adaptive ratio $r$:
$$
m = \lfloor r \cdot s \rfloor, \quad \mathbf{ids}_s = TopK(\{w_i^*\}_{i=0}^s, m).
$$
This allows the number of semantic candidates to vary with scene complexity [2602.11007]. To prevent those candidates from concentrating in a small region, farthest point sampling is then applied over the selected superpoints:
$$
\mathbf{ids}_f = FPS(F_s[\mathbf{ids}_s], q), \quad Q = \phi_\text{proj}(F_s[\mathbf{ids}_f]),
$$
and query coordinates are taken as
$$
C_c = C_s[\mathbf{ids}_f] \in \mathbb{R}^{q \times 3}.
$$

The paper describes this as **hierarchical semantic-spatial** selection: semantic hierarchy through ranking and filtering by semantic activation, followed by spatial hierarchy through FPS among semantic candidates [2602.11007]. Empirically, the authors report that query distributions demonstrate better coverage than semantic-only or FPS-only methods, with fewer redundant queries on walls or background, and that training curves show faster convergence than FPS-based and purely semantic-based initialization, given the same decoder [2602.11007].

The initializer is therefore not merely a preprocessing step. The ablations indicate that it materially influences convergence speed, query placement, and final segmentation quality.

## 5. Coordinate-guided SSM decoding and local aggregation

The decoder replaces heavy attention with two coupled mechanisms: **local aggregation** and a **spatial dual-path SSM block** [2602.11007]. The local aggregation module restricts each query’s interaction to a small neighborhood of superpoints defined by geometry, replacing global cross-attention from queries to all superpoints.

Given query contents $Q \in \mathbb{R}^{q \times d}$ and coordinates $C_c \in \mathbb{R}^{q \times 3}$, together with superpoint features and coordinates, the local aggregation proceeds by selecting $k$ nearest superpoints for each query, expanding query features, learning importance weights, and aggregating element-wise query–neighbor interactions [2602.11007]. For each query $i$, the aggregated feature is
$$
f_i = \sum_{j=1}^k K_{i,j} \cdot (Q'_{i,j} \odot N_{i,j}),
$$
followed by
$$
Q_{\text{agg}} \in \mathbb{R}^{q \times d} \gets Q + W_o \cdot F.
$$
Because $k \ll s$, the complexity per query is $O(kd)$ and is independent of the total number of superpoints $s$ [2602.11007].

The SSM component begins from a continuous-time linear state space model:
$$
h^{\prime}(t) = A h(t) + B x(t), \quad y(t) = C h(t) + D x(t),
$$
which is then discretized by zero-order hold into
$$
h_k = \overline{A} h_{k-1} + \overline{B} x_k, \quad y_k = C h_k.
$$
The paper uses the **Mamba-2** variant, which simplifies state space parameters and enables efficient sequence modeling [2602.11007].

Because queries do not have a natural ordering, LaSSM serializes them according to 3D Hilbert space-filling curves and a transposed Hilbert variant. Queries are sorted by those indices, processed by shared SSMs, rearranged back to the original order, averaged across both paths, and added residually:
$$
Q_{\text{ssm}} = Q_{\text{agg}} + \frac{1}{\lVert \mathcal{S} \rVert} \sum_{Q^{(i)}_{\text{agg}} \in \mathcal{S}} RA(SSM(Q^{(i)}_{\text{agg}})).
$$
An FFN then produces
$$
Q_{\text{ref}} = Q_{\text{ssm}} + FFN(Q_{\text{ssm}}).
$$

The paper argues that this design yields linear self-interaction complexity in the number of queries rather than quadratic $O(q^2)$, while also using positional information explicitly through coordinate-based ordering rather than only through positional encodings [2602.11007]. It additionally reports that dual-path Hilbert ordering performs better than single-path or Z-order, and that updating query coordinates after each layer is important for keeping the Hilbert ordering consistent with evolving instance centers [2602.11007].

A plausible implication is that LaSSM’s efficiency does not arise from replacing one global operator with another; it arises from a division of labor in which local geometry is handled by k-NN aggregation and longer-range query dependencies are handled by serialized SSMs.

## 6. Training objectives, empirical results, and efficiency profile

LaSSM is trained with a multi-task loss consisting of superpoint semantic activation supervision and instance-level losses for classification, mask prediction, and center regression across decoder layers [2602.11007]. The semantic activation loss is
$$
\mathcal{L}_\text{sem} = \frac{1}{n}\sum_{i=1}^n CE(W_i, y^*_i),
$$
where superpoint labels are obtained by majority voting within each superpoint. For instance prediction, LaSSM uses Hungarian matching with a cost combining BCE and Dice mask losses, classification cross-entropy, and $\ell_1$ center regression. The total loss is
$$
\begin{aligned}
\mathcal{L} = & \sum_{l=1}^{L} \big[ \lambda_\text{cls} \mathcal{L}^l_\text{cls} +\lambda_\text{cen} \mathcal{L}^l_\text{cen} + \lambda_\text{mask} (\mathcal{L}^l_\text{bce} + \mathcal{L}^l_\text{dice}) \big] \\
& + \lambda_\text{sem} \mathcal{L}_\text{sem},
\end{aligned}
$$
with $\lambda_\text{cls} = 0.5$, $\lambda_\text{mask} = 1.0$, $\lambda_\text{cen} = 1.0$, and $\lambda_\text{sem} = 0.2$ [2602.11007].

The reported benchmark results establish LaSSM’s standing most clearly on **ScanNet++ V2**. The hybrid **LaSSM smpro** variant reaches validation **mAP 29.1**, **AP\(_{50}\) 43.5**, and **AP\(_{25}\) 51.6**, and test **mAP 32.4**, **AP\(_{50}\) 48.0**, and **AP\(_{25}\) 54.8** [2602.11007]. The paper states that the best previous method, SGIFormer, achieved test **mAP 29.9**, **AP\(_{50}\) 45.7**, and **AP\(_{25}\) 54.4**, so LaSSM improves by **+2.5 mAP** and **+2.3 AP\(_{50}\)** on test with approximately **1/3 FLOPs** [2602.11007]. It further states that LaSSM ranks first place on the ScanNet++ V2 leaderboard at submission time.

The framework also reports competitive performance on **ScanNet++ V1**, **ScanNet V2**, **ScanNet200**, and **S3DIS Area 5** [2602.11007]. On ScanNet V2, for example, **smpro LaSSM** reaches validation **mAP 58.4**, **AP\(_{50}\) 78.1**, and **AP\(_{25}\) 86.1**, and test **mAP 57.9**. On ScanNet200, **smpro LaSSM** reports **mAP 29.3**, **AP\(_{50}\) 39.2**, and **AP\(_{25}\) 44.5**. On S3DIS Area 5, **smpro LaSSM** reports **mAP 56.5** and **AP\(_{50}\) 69.4** [2602.11007].

The efficiency analysis in the same paper compares LaSSM against attention-heavy alternatives. On **ScanNet V2**, **LaSSM (smpro)** reports **AP\(_{50}\) 78.1** with **3.711G FLOPs**, compared with **SGIFormer** at **AP\(_{50}\) 78.4** and **4.456G FLOPs**, and **OneFormer3D** at **AP\(_{50}\) 76.3** and **4.082G FLOPs** [2602.11007]. On **ScanNet++ V2**, **SGIFormer** reports **AP\(_{50}\) 41.1**, **13.513G FLOPs**, **15.96M** parameters, and **355.86 ms**, while **LaSSM (smpro)** reports **AP\(_{50}\) 43.5**, **4.777G FLOPs**, **18.38M** parameters, and **379.94 ms** [2602.11007]. The paper additionally notes that OneFormer3D runs out of memory on ScanNet++ V2 in their setting, whereas LaSSM remains feasible.

Ablation studies report that the LaSSM initializer outperforms FPS-based and semantic-only initialization, that **best around 0.8** is observed for the selection ratio $r$, that **best at 8** is observed for the number of local neighbors $k$, that **best performance at 400** queries is obtained, and that performance saturates at **6 layers** [2602.11007]. The paper also states that even with zero decoder layers, the initializer alone gives respectable **AP\(_{50}\) 71.4**, highlighting its strength.

## 7. Related usages in multimodal audio and speech research

Outside 3D segmentation, the most directly confusable name is **LLaSM**, which stands for **Large Language and Speech Model** [2308.15930]. LLaSM combines a frozen **Whisper** speech encoder, a trainable **modal adaptor**, and **Chinese-LLAMA2-7B** as the LLM backbone. Audio is represented by continuous embeddings rather than discrete speech tokens, inserted into a single unified sequence with text by replacing `'<au_patch>'` placeholders between `'<au_start>'` and `'<au_end>'` tokens [2308.15930]. Training proceeds in two stages: modality adaptation on ASR-style data with the LLM frozen, and cross-modal instruction fine-tuning on **LLaSM-Audio-Instructions**, a released bilingual speech–text instruction-following dataset with **199k conversations**, **508k total samples**, **428k English samples**, and **80k Chinese samples** [2308.15930]. The system’s generative modality is text rather than speech.

A separate but conceptually related usage appears in “State-Space Large Audio Language Models,” which introduces the first fully state-space LALM and states that one can denote it informally as a **LaSSM** [2411.15685]. In that work, a transformer-based audio encoder is replaced by **DASS**, a state-space audio encoder, and the transformer-based LLM is replaced by a **Mamba-based state-space LLM-2.8B**. The fully state-space system is called **ssLALM**, while the mixed system is called **Hybrid-LALM** [2411.15685]. The paper emphasizes linear-time sequence complexity for SSMs, reports **2.8B parameters total and ~43–62M trainable parameters** for ssLALM, and shows competitive performance with transformer-based LALMs on close-ended audio tasks [2411.15685].

These neighboring usages are relevant because they share state-space or multimodal design themes while belonging to different application domains. This suggests that the acronym “LaSSM” has become a point of lexical convergence rather than a single established family name.

## 8. Limitations, failure modes, and broader significance

The LaSSM 3D segmentation paper explicitly identifies several limitations [2602.11007]. All scenes use the same fixed number of queries, **400**, which the authors describe as suboptimal because small scenes may have fewer instances while large scenes may need more. Hilbert curve ordering and transposed variants add serialization overhead, and experiments are confined to indoor datasets such as ScanNet, S3DIS, and ScanNet++. The paper also presents failure cases in which a large blackboard with similar color and geometry to the wall is missed, and a door is over-segmented into multiple instances due to texture or structural ambiguities.

The paper’s stated future directions include adaptive query allocation, more direct ways to incorporate positional information with lower latency, extension to outdoor scenes, more advanced or specialized SSM designs for 3D, and application of LaSSM-style decoders to other query-based 3D tasks such as detection and panoptic segmentation [2602.11007]. It also positions LaSSM at the intersection of query-based 3D instance segmentation, efficient decoder design, and state space models in vision and point clouds.

In the broader technical landscape, LaSSM is best understood as a response to the practical scaling problems of query decoders in sparse 3D scenes. Its significance lies not only in the reported first-place ScanNet++ V2 leaderboard result, but also in the specific decomposition of the segmentation problem into superpoint-based query construction, geometry-constrained local refinement, and coordinate-guided linear-time sequence modeling [2602.11007]. A plausible implication is that later work may treat these components as modular design patterns even when not adopting the full LaSSM framework.

Source: https://www.emergentmind.com/topics/lassm