Object-Centric Sampling (OCS)

• Object-Centric Sampling (OCS) is a strategy that focuses on allocating samples and computation to object regions, enhancing efficiency and interpretability.
• OCS is applied in generative models, fine-grained classification, video editing, and 3D reconstruction, achieving substantial improvements in performance and runtime efficiency.
• Techniques involve leveraging object detectors, slot-based priors, and adaptive sampling strategies to markedly improve segmentation, reconstruction fidelity, and downstream task performance.

Object-Centric Sampling (OCS) refers to a family of methodologies that preferentially allocate data samples, computational resources, or generative capacity toward regions, features, or events associated with distinct objects, as opposed to distributed or uniform (background-centric, view-centric, or event-centric) approaches. OCS arises in multiple areas of machine learning and computer vision, including generative modeling, discriminative training, video editing, process mining, 3D reconstruction, and multi-view 3D perception. The central motivation for OCS is the empirical and theoretical gain in sample efficiency, interpretability, or computation when foreground objects rather than undifferentiated data regions constitute the primary modeling focus.

1. OCS in Generative Models and Scene Synthesis

Object-centric sampling in generative settings is best exemplified by the GENESIS model, which parameterizes scene images as decompositions into $K$ explicit “object slots,” each governed by a mask-latent $z^m_k$ and a content-latent $z^c_k$ (Engelcke et al., 2019). Scene generation proceeds by sequentially sampling each slot’s latent variables from an autoregressive prior:

$$p(z^m_{1:K}) = \prod_{k=1}^{K} p(z^m_k|z^m_{1:k-1}), \qquad p(z^c_{1:K}|z^m_{1:K}) = \prod_{k=1}^K p(z^c_k|z^m_k)$$

Each object’s mask and content are decoded, with a stick-breaking process ensuring mutually exclusive spatial masks, yielding pixel-wise mixture weights $\pi_k$ and image components $\mu_k$. Critically, the autoregressive structure encodes object-object dependencies (such as mutual exclusion or semantic ordering), so OCS not only decomposes and reconstructs but also allows generation of coherent multi-object scenes: the model reliably draws “sky,” then “floor,” followed by separate objects, as opposed to independent priors (e.g., MONet) which produce incoherent image fragments.

Empirical evaluation shows OCS in this sense drastically improves compositional plausibility (FID on GQN: GENESIS 80.5, MONet 176.4), segmentation (ARI 0.73 vs. 0.63), and downstream task fitness (block stability classification: 64% vs. 60%) (Engelcke et al., 2019). Thus, OCS under autoregressive priors is foundational for compositional, sample-efficient scene generators.

2. OCS in Discriminative Training and Image Classification

In fine-grained image classification, OCS directs data sampling to image regions with high object presence, leveraging detection cues to sample patches or crops. In the Object-Centric Sampling for Fine-grained Image Classification pipeline (Wang et al., 2014), a saliency-aware Regionlet detector is trained to localize the most visually salient object of interest per image. Object-centric samples are then generated by multinomially drawing $s\times s$ crops such that the probability of selecting a crop is proportional to its overlap with the detected bounding box:

$p(x,y)\propto |R_{x, y} \cap R_o|$

This non-uniform, overlap-weighted sampling focuses the model’s gradient updates on foreground regions, substantially mitigating overfit and class confusion arising from background clutter. Critically, robustness to detector imperfections is maintained by allowing translations and low-probability background samples, providing data augmentation and mild invariance.

Empirically, on a 333-class car dataset, OCS yields an increase from 81.6% (uniform sampling) to 89.3% top-1 accuracy. “Hard cropping” to the box alone achieves only ~87.0%, highlighting the efficacy of the mixture strategy (Wang et al., 2014).

3. OCS in Diffusion-Based Video Editing

OCS has also been adopted for accelerating diffusion-based video editing by focusing denoising steps on user-salient (foreground) regions (Kahatapitiya et al., 2024). Here, OCS splits latent tokens into foreground and background using a segmentation mask. Foreground tokens are processed with the standard fine step-schedule ($\Delta T$ per iteration), while background tokens are advanced on a coarser schedule (step-size $\phi\Delta T$, $\phi > 1$). After a blending time $T_b = \gamma T$, the regions are merged, and remaining steps proceed as usual.

The count of expensive UNet passes is reduced by a factor of $$N_{\text{OCS}} \sim N[\gamma + (1-\gamma)(1+1/\phi)]$$ vs. $N$ in standard pipelines, yielding runtime gains of $2 \times$ to $10 \times$ with negligible loss in semantic or perceptual metrics. Experiments show a reduction in generation time (FateZero: 41.3 s to 9.3 s) while maintaining or even slightly improving CLIP and temporal consistency scores (Kahatapitiya et al., 2024).

OCS is most effective for local edits and becomes less effective when edits are global (masks cover most pixels). Adaptivity in scheduling and mask computation, as well as sparse UNet support, are current areas for enhancement.

4. OCS in 3D Reconstruction and Neural Rendering

In the context of sparse-view 3D object reconstruction, OCS manifests as object-centric ray sampling (Cerkezi et al., 2023). Rather than casting rays per camera pixel (view-centric), rays are emitted per mesh vertex along the vertex normal, drawing $K$ samples per ray. Vertex updates are weighted by local shape densities returned by an implicit neural network:

$$R_i(\alpha) = V_i + \alpha N_i,\quad \alpha_k = -t_i^{\text{in}} + \frac{k-1}{K-1}(t_i^{\text{in}} + t_i^{\text{out}})$$

$$w_{i,k} = \frac{\exp \sigma_{i,k}}{\sum_j \exp \sigma_{i,j}}$$

$\hat{V}_i = \sum_{k=1}^{K} w_{i,k} X_{i,k}$

This scheme concentrates samples along the actual object surface, drastically reducing redundant queries (sample count independent of number of views) and producing high-fidelity reconstructions without requiring explicit object masks. On Google Scanned Objects (8 views), PSNR of 29.03 dB and Chamfer-L2 of 8.69 × $10^{-4}$ are state-of-the-art among methods with similar supervision (Cerkezi et al., 2023).

5. OCS in Multi-View 3D Perception and Transformers

In multi-view 3D detection with BEV Transformers, OCS appears as object-focused multi-view sampling (Qi et al., 2023). Instead of sampling along a global vertical column, sample density is concentrated in an adaptive, local height band predicted per BEV query, which aligns with typical object heights in the scene. The BEV query feature $f_q$ predicts a shift $\Delta h$ for the local band:

$$\hat{Z}_{h,l} = [h_{\text{min}}^l + \Delta h, h_{\text{max}}^l + \Delta h]$$

Half of spatial samples are drawn from the global range, half from the adaptive local slab, boosting signal from object regions in 3D–2D attention. Ablations show a +6 point NDS gain (0.430 vs. 0.371) on the nuScenes validation split with this strategy (Qi et al., 2023).

6. OCS in Object-Centric Event Log Sampling

In process mining, OCS refers to various strategies for subsetting large multi-object event logs to make downstream analysis tractable (Berti, 2022). These include:

• SS1: Sampling a random subset of events, risking truncated or broken object lifecycles.
• SS2: Sampling objects (and their incident events), preserving object lifecycles but possibly missing interactions.
• SS3: Sampling object types, retaining complete interactions for sampled types.
• SS4: Sampling all events within connected components of the event–event overlap graph, ensuring all interactions within a block are preserved.

Procedural details, costs, and trade-offs are formalized, but no quantitative quality guarantees are shown beyond intuitive behavioral preservation (Berti, 2022). Combination with filtering operations is standard for practical log reduction.

7. Synthesis and Theoretical Considerations

Across modalities, OCS replaces uniform or view-/event-centric sampling with strategies driven by spatial, temporal, or structural objectness priors. The primary empirical advantages demonstrated are:

• Higher sample/computational efficiency for fixed-quality outputs
• Enhanced compositional or discriminative power by suppressing background signal
• Mitigation of overfitting in small-sample or redundancy-prone regimes
• Improved interpretability and structuredness of learned representations

In most settings, implementation involves an auxiliary object detector, slot-based prior, segmentation mask, or geometric proxy to define the object-centric regions or slots. While OCS is provably more efficient in terms of sample or computational complexity only in specific regimes (e.g., when object occupancy is sparse), its practicality and plug-in nature have seen rapid adoption in various domains.

---

References

• GENESIS: Generative Scene Inference and Sampling with Object-Centric Latent Representations (Engelcke et al., 2019)
• Object-centric Sampling for Fine-grained Image Classification (Wang et al., 2014)
• Object-Centric Diffusion for Efficient Video Editing (Kahatapitiya et al., 2024)
• Sparse 3D Reconstruction via Object-Centric Ray Sampling (Cerkezi et al., 2023)
• OCBEV: Object-Centric BEV Transformer for Multi-View 3D Object Detection (Qi et al., 2023)
• Filtering and Sampling Object-Centric Event Logs (Berti, 2022)