Instance-aware Particle Filling (IPF)

• Instance-aware Particle Filling (IPF) is a set of techniques that populates 3D space with particles based on instance segmentation and probabilistic sampling, ensuring accurate physical simulation and mapping.
• It employs methods like Monte Carlo Importance Sampling and state-augmented sequential Monte Carlo filtering to handle challenges such as thin structures, nonconvex geometries, and multi-object scenarios.
• Empirical evaluations demonstrate that IPF improves simulation stability and mapping accuracy, with key performance gains reflected in metrics like CLIP scores and Hausdorff distances over conventional methods.

Instance-aware Particle Filling (IPF) denotes a family of techniques for populating three-dimensional space with particles, such that structural, semantic, and instance-level properties are preserved for downstream applications in simulation and mapping. Motivated by the limitations of naïve global or uniform filling—such as poor handling of thin regions, geometric concavities, or multi-object scenarios—IPF methods couple object-level instance segmentation with probabilistically principled particle placement. The goals are to enhance physical plausibility for dynamic simulation, maintain high-fidelity geometric detail, and support robust semantic and instance-aware 3D mapping in noisy or dynamic scenes (Ma et al., 2 Feb 2026, Chen et al., 2024).

1. Foundational Principles and Motivation

IPF arises from challenges in correctly representing object interiors for physics simulation (e.g., Material Point Method, MPM) and instance-aware semantic mapping. Surface-based representations such as 3D Gaussian Splatting (3DGS) are efficient for rendering but inherently hollow, which causes physical simulation instability or semantic incompleteness. Traditional filling strategies either ignore object boundaries or assume simplistic surface-to-interior mappings, leading to incorrect mass or occupancy distributions, especially for thin or nonconvex objects and multi-material settings.

In physics simulation, IPF generates interior Lagrangian particles conditional on segmented object instances, preserving fine geometric detail and consistent mass. In semantic mapping, it fuses instance tracking and semantic labeling via a particle filter framework augmented at the state level to encode instance and semantic information, allowing robust, temporally consistent environmental modeling even under perceptual noise (Ma et al., 2 Feb 2026, Chen et al., 2024).

2. Methodologies for Instance-aware Particle Filling

Two representative instantiations of IPF are (a) importance-sampled physical interior filling for 3DGS-based simulation (Ma et al., 2 Feb 2026), and (b) instance-augmented sequential Monte Carlo (SMC) filtering for semantic occupancy mapping (Chen et al., 2024).

a. Importance Sampling Interior Completion in FastPhysGS

• Instance segmentation: Density-based clustering (e.g., DBSCAN) partitions the set of 3DGS gaussians $P = \{p_i\}_{i=1}^N$ into $K$ object instances $\{C_k\}$.
• Geometry extraction: For each $C_k$, compute the convex hull $H_k$ (via Quickhull) and axis-aligned bounding box $B_k$.
• Candidate sampling and pruning: Uniformly sample $M \gg |C_k|$ points $U_k$ in $B_k$; prune using ray-casting occupancy threshold $\tau_{occ}$ to produce $K$0.
• Monte Carlo Importance Sampling (MCIS):
  • Assign importance weights to candidate points based on proximity to the instance center set:

  $K$1

  with $K$2, kernel $K$3 controls locality, and minimum weight $K$4 avoids zero-probability zones. - Normalize and sample $K$5 points according to probabilities $K$6. - Set opacity $K$7 for these points, preserving rendering but enabling MPM simulation.

b. State-augmented SMC-PHD Filter in Semantic Mapping

• Particle representation: Each particle encodes 3D position and a discrete instance ID, storing semantic label at the instance level.
• Prediction: Propagate particles with per-instance rigid transformations from tracked objects, modeling process noise.
• Update: Re-weight particles using a measurement likelihood encompassing position uncertainty, ID confusion resilience (through a confusion kernel $K$8), and a memory-based forgetting factor $K$9.
• Birth and Resampling: New observations spawn “newborn” particles; per-voxel resampling maintains capacity and effects “death” of outdated hypotheses.
• Memory module: Fully-observed instances are cached as templates, which are later reused to fill in occlusions or match new instances.

3. Algorithmic Workflow and Pseudocode Structures

Both IPF variants share a commitment to instance-level granularity and adaptivity; their core workflows are summarized below.

Pipeline	Key Steps	Critical Parameters
FastPhysGS IPF	Segmentation → Hull & BBox → Prune → MCIS → Fill($\{C_k\}$0)	$\{C_k\}$1, $\{C_k\}$2, $\{C_k\}$3, $\{C_k\}$4, $\{C_k\}$5
SMC-PHD IPF	Predict → Project/Update → Birth → Memory Fill → Resample	$\{C_k\}$6, $\{C_k\}$7, $\{C_k\}$8, $\{C_k\}$9, $C_k$0

Example pseudocode for FastPhysGS IPF, as presented in (Ma et al., 2 Feb 2026):

1. DBSCAN clustering $C_k$1 instance sets $C_k$2.
2. For each $C_k$3, compute $C_k$4, sample/prune $C_k$5 to $C_k$6.
3. For $C_k$7, compute $C_k$8, $C_k$9, normalize $H_k$0.
4. Draw $H_k$1 samples via multinomial over $H_k$2.
5. Set their $H_k$3; union with surface gaussians for simulation.

In SMC-PHD, prediction and update leverage conventional RFS/particle filter operational steps but are extended to track instance and semantic information, and enhanced by memory-templated instance filling.

4. Integration of Instance-level Priors and Geometric Reasoning

Instance-aware segmentation is central to IPF. In FastPhysGS, DBSCAN is used to automatically partition the 3DGS gaussians, dispensing with any 2D image-based segmentation—a key step in preserving object-wise integrity during filling. Each object's convex hull, computed post-clustering, provides a geometric prior for determining the region of candidate samples.

Occupancy pruning via ray-casting further restricts interior candidates, enhancing computational efficiency and reducing spurious samples in free space. The MCIS kernel accentuates regions near boundaries or in thin geometric configurations, ensuring that higher importance is given to fine physical structures, which global or uniform fills cannot represent accurately. In particle-based mapping, the instance-augmented state combined with a “collective filtering” update ruleset (using confusion and forgetting factors) improves resilience to ID noise and occlusions, while the memory module enables long-term consistency for re-entered or occluded objects (Chen et al., 2024).

5. Complexity, Memory Overhead, and Scalability

IPF implementation is distinguished by its efficiency and scalability. In FastPhysGS (Ma et al., 2 Feb 2026):

• Complexity per instance: $H_k$4 for Quickhull, $H_k$5 for ray-casting, and $H_k$6 for nearest-neighbor computations using KD-trees.
• Full pipeline memory: $H_k$7 GB for typical scenes, processing time $H_k$8 s on a single RTX A6000 GPU.
• Scalability: The per-instance independence and moderate memory footprint permit near-linear parallelization or batching across objects.

In mapping (Chen et al., 2024):

• Particle cap: Typical occupancy map (51.2 m, 0.2 m grid) yields CPU inference rates $H_k$910 Hz.
• Per-voxel bounded particle storage ensures predictable memory and update costs.

6. Empirical Performance and Evaluation

Quantitative ablations and benchmarks substantiate the efficacy of IPF approaches.

FastPhysGS IPF (Ma et al., 2 Feb 2026):

• Ablation (“w/o IPF”): MPM simulations collapse without interior particles. CLIP score drops from 0.292 (full model) to 0.263; aesthetic, semantic, and physical commonsense scores also decline.
• Qualitative: Baselines (e.g., PhysGaussian’s global fill) distort thin structures; IPF preserves them.

SMC-PHD IPF (Chen et al., 2024):

• Occupancy mapping: On Virtual KITTI 2, achieves absolute Hausdorff Distance (AHD) 0.067, F1 of 0.945, and mean Absolute Distance ($B_k$0) 0.276—markedly outperforming previous methods.
• Semantic mIoU: Static 0.629/0.834; moving 0.680/0.596, with the next best at 0.464/0.212.
• Instance accuracy: mF1 of 0.667/0.588 vs. top baselines at 0.381/0.154.
• Efficiency: ~10 Hz frame rates on large-scale 3D scenes; memory module confers further boosts in completeness.

7. Implementation Hyperparameters and Design Choices

Critical hyperparameters and architectural choices include:

• DBSCAN radius ($B_k$1): 0.05–0.07 (FastPhysGS).
• Sampling parameters: Candidate fill ratio $B_k$2; fill count $B_k$3.
• Occupancy/pruning: Threshold $B_k$4.
• MCIS kernel: $B_k$5 for locality sensitivity, $B_k$6 minimum weight.
• SMC-PHD filter: Detection probability $B_k$7, noise covariances $B_k$8, per-voxel cap $B_k$9.
• Memory parameters: Template extraction upon sufficient observation completeness, RANSAC-based instance matching.

Each parameter is tuned via empirical ablation to balance fidelity, resiliency, and computational cost. Instance-aware segmentation and memory-augmented filling are essential for robustness in both simulation and real-world mapping.

---

Instance-aware Particle Filling, through principled exploitation of instance segmentation, geometric priors, importance sampling, and probabilistic filtering, addresses fundamental limitations in surface-based 3D representation for both physical simulation and dynamic semantic mapping. The methodology achieves superior geometric fidelity and instance robustness under practical runtime and memory budgets, as established in recent works (Ma et al., 2 Feb 2026, Chen et al., 2024).