Scene-Object Controller (SOC) Framework

• SOC is a framework that decomposes scenes into discrete object representations, integrating perception, prediction, and decision-making for effective control.
• SOC architectures vary in object encoding, employing methods from 2D image-space features and graph dynamics to 3D latent grids and neural scattering functions.
• SOC approaches leverage model-predictive control and continuous state correction to achieve superior performance in robotic manipulation, scene synthesis, and detection tasks.

Searching arXiv for recent and foundational papers on scene-object controllers and related object-centric control frameworks. I’ll look up relevant arXiv entries to ground the article in current and foundational work. Scene-Object Controller (SOC) denotes a family of systems that organize perception, prediction, and decision-making around explicit scene decomposition into objects and their relations. In the manipulation literature, an SOC is a closed-loop control system that plans and executes actions using an explicit, object-centric scene representation and a learned forward dynamics model within a model predictive control framework. Closely related formulations extend the same controller logic beyond robot actuation: scene generation systems use planner and specialist agents to control object layout and environmental scaffolds, while scene-conditioned detection systems use scene-level priors to modulate object decoding. Across these uses, the unifying principle is that control operates on object states, object interactions, or object hypotheses rather than on a monolithic scene latent alone (Ye et al., 2019, Tian et al., 2023, Kim et al., 7 Jun 2026, Zheng et al., 2022).

1. Definition and terminological scope

The most explicit control-oriented definition appears in object-centric model-predictive control: the scene is represented as a set of discrete entities, each with an explicit spatial state and an implicit visual feature, and the controller uses a learned forward model to search for action sequences that achieve a desired goal configuration. In that formulation, the full scene state is

$s_t=\{(x_i^t,z_i^t)\}_{i=1}^N,$

and the learned transition is

$s_{t+1}=f_\theta(s_t,a_t).$

The resulting controller combines object-centric state estimation, relational forward prediction, model-predictive planning, and closed-loop correction (Ye et al., 2019).

Other works widen the meaning of SOC while preserving the same scene-object logic. SceneConductor describes an SOC as a system that governs scene-object relationships, controls layout and placement, and maintains environmental context through three stages: scene initialization, environment construction, and multi-agent refinement. HyperDet3D states that the paper does not use the term “Scene-Object Controller (SOC),” but that its scene-conditioned hypernetwork and Multi-head Scene-Conditioned Attention module together implement precisely the function of an SOC by using scene embeddings to control decoder-layer parameters at test time. This establishes SOC as a functional category rather than a single benchmarked architecture (Kim et al., 7 Jun 2026, Zheng et al., 2022).

A common misconception is that SOC refers only to robot manipulation. The cited literature uses the same structural idea in at least three settings: long-horizon visuomotor control, scene-conditioned 3D detection, and multi-agent 3D scene generation. A second misconception is that SOC implies a specific optimization regime. In fact, the reported systems use Cross-Entropy Method, MPPI, CMA, energy-based imitation, and planner-dispatch policies, all while retaining object-centric control structure (Ye et al., 2019, Tian et al., 2023, Heravi et al., 2022).

2. Scene factorization and object representations

SOC architectures differ primarily in how they encode objects and the ambient scene. In object-centric forward modeling for MPC, each object has an explicit spatial state and an implicit feature. In the reported implementation, the explicit state is the 2D image-space location $b_i^t\in\mathbb{R}^2$, while orientation is implicitly encoded in a ResNet-18 feature $z_i^t$ extracted from a crop centered at $b_i^t$. This yields a compact object-wise state that is directly amenable to graph-based interaction modeling and terminal-goal comparison (Ye et al., 2019).

A more geometric factorization appears in viewpoint-invariant 3D simulators. 3D-OES lifts RGB-D observations and camera poses into a world-aligned latent 3D feature map $M_t\in\mathbb{R}^{w\times h\times d\times c}$, then detects objects as 3D boxes and masks. Its latent state is

$s_t=(M_t,\{o_i^t\}_{i=1..N}),$

with

$o_i^t=(M_i,m_i,p_i^t,r_i^t,v_i^t).$

A key property is that object appearance features $M_i$ do not change with viewpoint or time; simulation moves objects by transforming their 3D features without resynthesizing appearance. The paper characterizes this as viewpoint-invariant object factorization with non-interference between objects (Tung et al., 2020).

The most lighting-aware representation is the Object-Centric Neural Scattering Function. Here each object is modeled by an implicit neural field

$$f_\theta:(x,\omega_{\text{light}},\omega_{\text{out}})\rightarrow (\rho,\sigma),$$

where $s_{t+1}=f_\theta(s_t,a_t).$0 is expressed in the object’s local frame, $s_{t+1}=f_\theta(s_t,a_t).$1 is the incoming light direction, and $s_{t+1}=f_\theta(s_t,a_t).$2 is the outgoing or viewing direction. OSFs model per-object light transport and volumetric density, enabling relighting and compositional scene re-rendering under rearrangement and varying illumination. The scene image is rendered compositionally from per-object fields and poses, with shadowing between objects handled using shadow mapping for speed (Tian et al., 2023).

Scene generation work uses a different but still explicit object state. SceneConductor places each canonical object mesh $s_{t+1}=f_\theta(s_t,a_t).$3 in a floor-aligned scene using pose and scale parameters $s_{t+1}=f_\theta(s_t,a_t).$4 and a shared floor rotation $s_{t+1}=f_\theta(s_t,a_t).$5, with camera-frame vertices parameterized as

$s_{t+1}=f_\theta(s_t,a_t).$6

The global scene graph also stores boundary walls, support surfaces, material skins, and light sources. This extends object factorization from movable items to the environmental scaffold itself (Kim et al., 7 Jun 2026).

Self-supervised visuomotor SOCs factor scenes into slots rather than boxes or meshes. Slot Attention encodes an RGB image into a dense feature map and then produces $s_{t+1}=f_\theta(s_t,a_t).$7 object-centric slots with masks. Reconstruction follows

$s_{t+1}=f_\theta(s_t,a_t).$8

which enforces spatially localized object decomposition without labels. The paper reports that such object-aware representations capture all scene components, including blocks, pole, robot arm, end-effector, table, and background, whereas object-agnostic embeddings tend to over-focus on frequently moving elements (Heravi et al., 2022).

Taken together, these formulations show that SOC does not require a single canonical state space. The scene can be factorized in 2D image coordinates, world-aligned 3D latent grids, relightable neural fields, floor-aligned mesh layouts, or slot-based masks and embeddings. What is invariant is the commitment to object-indexed state.

3. Predictive models and control laws

In the manipulation setting, SOC typically couples object state with relational dynamics. Object-centric forward modeling for MPC constructs a graph with one node per object and one additional node for the robot or action. Initial node features concatenate spatial state and visual feature,

$s_{t+1}=f_\theta(s_t,a_t).$9

and an Interaction Network-style update performs message passing:

$b_i^t\in\mathbb{R}^2$0

Planning minimizes a horizon cost that combines action regularization and terminal goal matching. The paper reports a planning cost

$b_i^t\in\mathbb{R}^2$1

with $b_i^t\in\mathbb{R}^2$2, and uses CEM with $b_i^t\in\mathbb{R}^2$3 samples, $b_i^t\in\mathbb{R}^2$4 elites, $b_i^t\in\mathbb{R}^2$5 iterations, and horizon $b_i^t\in\mathbb{R}^2$6 (Ye et al., 2019).

3D-OES implements a related but 3D-native control law. A GNN predicts per-object translations and rotations from object-centric 3D features, kinematics, and action input, then future scenes are synthesized by rotating and translating the original object features. For control, the paper uses sampling-based MPC over action sequences and evaluates a cost containing goal distance, collision penalty, and action magnitude,

$b_i^t\in\mathbb{R}^2$7

Because appearance is fixed and only placement changes, long-horizon rollouts avoid feature drift (Tung et al., 2020).

The OSF-based controller integrates object-centric rendering directly into the MPC objective. Object states are $b_i^t\in\mathbb{R}^2$8, where $b_i^t\in\mathbb{R}^2$9 and $z_i^t$0 is a unit quaternion; control is the pusher’s Cartesian motion command $z_i^t$1. A graph dynamics model predicts

$z_i^t$2

with adjacency determined by an OSF-derived proximity threshold $z_i^t$3. Planning is finite-horizon optimal control, but the primary cost is visual:

$z_i^t$4

where $z_i^t$5 is rendered by OSF composition from the predicted states. Optimization uses sampling-based MPC with MPPI: sample $z_i^t$6 action sequences, roll out the GNN, render predicted images from the goal view, score with image MSE, execute the first control, and replan every step with $z_i^t$7 (Tian et al., 2023).

Not all SOCs act through motion commands. HyperDet3D instantiates controller behavior as scene-conditioned parameter modulation. After self- and cross-attention compute candidate object features $z_i^t$8, a scene-conditioned hypernetwork generates layer parameters and biases, and the controlled update is

$z_i^t$9

The scene-agnostic prior $b_i^t$0 and scene-specific prior $b_i^t$1 are fused elementwise as $b_i^t$2, then tiled or concatenated to match the target decoder layer. This is a control law over detector interpretation rather than over robot actuation, but it remains scene-object control in the literal sense that global scene priors steer object-level decisions (Zheng et al., 2022).

These variants show that an SOC is best characterized by where control is applied: to object trajectories, to relightable object states, to local scene revisions, or to object proposal decoding.

4. Perception, inversion, and state updating

Closed-loop SOCs depend on continual state refinement rather than one-shot prediction. In object-centric forward modeling for MPC, the forward model is paired with a learned correction module $b_i^t$3 that updates predicted object locations after each executed action:

$b_i^t$4

The corrected locations recenter crops, refresh features, and supply the next MPC iteration. The paper attributes robust closed-loop execution to this correction step, and reports that performance drops without it (Ye et al., 2019).

The OSF pipeline makes perception itself an inverse problem. From multi-view RGB at $b_i^t$5 and object masks, CMA estimates object poses and lighting; at later steps, the same optimizer refines state from RGB alone. The pose-only loss uses masked object renders,

$b_i^t$6

and the light-only loss uses the full compositional render,

$b_i^t$7

The full SOC loop alternates inversion, graph construction, rollout, visual scoring, action execution, and re-estimation. This design is explicitly motivated by harsh directional lighting, hard shadows, and object rearrangement (Tian et al., 2023).

SceneConductor likewise couples initialization with repeated auditing and repair. The initialization stage starts with Grounded-SAM masks, refines them by overlap suppression, fragment merging, and mask splitting, reconstructs meshes with SAM3D, and predicts $b_i^t$8 and $b_i^t$9 with a geometry-aware predictor supervised by point maps from MoGE. Environment construction then estimates the floor plane using RANSAC, builds boundary walls and support surfaces, and assigns materials and illumination. In the refinement stage, a planner agent evaluates structural and visual metrics, applies simple deterministic corrections when possible, and dispatches specialist agents when constraints are jointly coupled. The system uses a blackboard or state store with a versioned scene graph and transactional merges (Kim et al., 7 Jun 2026).

Self-supervised object-aware visuomotor control emphasizes representation freezing rather than online inversion. Slot Attention is pretrained on unlabeled interaction videos, then its dense features and masks are frozen for downstream policy learning and object localization. For localization, slot mask centers

$M_t\in\mathbb{R}^{w\times h\times d\times c}$0

are passed to a two-layer MLP. For policy learning, the controller consumes RGB, pretrained features, slot-derived information, or their concatenation, and is trained with Implicit Behavioral Cloning through an energy-based softmax loss over candidate actions (Heravi et al., 2022).

Across these systems, state estimation is not ancillary. It is a controller component, sometimes implemented as residual correction, sometimes as gradient-free inversion, sometimes as geometric auditing, and sometimes as pretraining that stabilizes downstream control.

5. Representative instantiations and reported performance

The robotic manipulation literature provides the clearest empirical demonstrations of SOC. Object-centric forward modeling for MPC achieves lower distance-to-goal over time than all baselines in both 1-object and 2-object tests, and the reported controller matches analytic performance when oracle states are available at every step. In real-world demonstrations, it pushes a block around another and generalizes to novel objects such as a measuring tape (Ye et al., 2019). 3D-OES reports translation errors of $M_t\in\mathbb{R}^{w\times h\times d\times c}$1 mm at $M_t\in\mathbb{R}^{w\times h\times d\times c}$2, $M_t\in\mathbb{R}^{w\times h\times d\times c}$3 mm at $M_t\in\mathbb{R}^{w\times h\times d\times c}$4, and $M_t\in\mathbb{R}^{w\times h\times d\times c}$5 mm at $M_t\in\mathbb{R}^{w\times h\times d\times c}$6, with corresponding rotation errors of $M_t\in\mathbb{R}^{w\times h\times d\times c}$7, $M_t\in\mathbb{R}^{w\times h\times d\times c}$8, and $M_t\in\mathbb{R}^{w\times h\times d\times c}$9 for pushing with two objects and novel views. In pushing-to-goal MPC, success rates are graph-XYZ $s_t=(M_t,\{o_i^t\}_{i=1..N}),$0, graph-XYZ-image $s_t=(M_t,\{o_i^t\}_{i=1..N}),$1, VF $s_t=(M_t,\{o_i^t\}_{i=1..N}),$2, PlaNet $s_t=(M_t,\{o_i^t\}_{i=1..N}),$3, and Ours $s_t=(M_t,\{o_i^t\}_{i=1..N}),$4; real robot transfer yields Ours-Real $s_t=(M_t,\{o_i^t\}_{i=1..N}),$5 from a single view and an unseen camera pose (Tung et al., 2020). The OSF-based controller reports that OSF+GNN+MPPI surpasses pixel-space MPC with FitVid and compositional NeRF-based MPC across 2-, 3-, and 4-object scenarios, that OSF-based SOC consistently dominates across thresholds, and that it generalizes to 2 and 4 objects despite training with 3 objects (Tian et al., 2023).

Object-aware visuomotor SOCs demonstrate the same principle from a representation-learning perspective. In low-data policy training with Implicit Behavioral Cloning, Slot Attention raises success from $s_t=(M_t,\{o_i^t\}_{i=1..N}),$6 for RGB to $s_t=(M_t,\{o_i^t\}_{i=1..N}),$7 at $s_t=(M_t,\{o_i^t\}_{i=1..N}),$8 episodes, while the oracle RGB+ground-truth segmentation input reaches $s_t=(M_t,\{o_i^t\}_{i=1..N}),$9. For object localization with $o_i^t=(M_i,m_i,p_i^t,r_i^t,v_i^t).$0 blocks, Slot Attention achieves mean PCK $o_i^t=(M_i,m_i,p_i^t,r_i^t,v_i^t).$1, compared with MoCo at $o_i^t=(M_i,m_i,p_i^t,r_i^t,v_i^t).$2 and Autoencoder at $o_i^t=(M_i,m_i,p_i^t,r_i^t,v_i^t).$3; with $o_i^t=(M_i,m_i,p_i^t,r_i^t,v_i^t).$4 blocks, Slot Attention yields mean PCK $o_i^t=(M_i,m_i,p_i^t,r_i^t,v_i^t).$5 versus $o_i^t=(M_i,m_i,p_i^t,r_i^t,v_i^t).$6 for MoCo and $o_i^t=(M_i,m_i,p_i^t,r_i^t,v_i^t).$7 for Autoencoder (Heravi et al., 2022).

Scene-conditioned detection and scene generation instantiate SOC outside robot control. HyperDet3D reports state-of-the-art results on ScanNet V2 with $o_i^t=(M_i,m_i,p_i^t,r_i^t,v_i^t).$8 mAP@0.25 and $o_i^t=(M_i,m_i,p_i^t,r_i^t,v_i^t).$9 [email protected], outperforming GroupFree3D $M_i$0 and 3DETR-m $M_i$1, and on SUN RGB-D with $M_i$2 [email protected] and $M_i$3 [email protected], outperforming GroupFree3D $M_i$4. In cross-dataset evaluation, HyperDet3D achieves mAP$M_i$5 and mAP$M_i$6, compared with GF3D at $M_i$7 and $M_i$8, and VoteNet at $M_i$9 and $$f_\theta:(x,\omega_{\text{light}},\omega_{\text{out}})\rightarrow (\rho,\sigma),$$0 (Zheng et al., 2022). SceneConductor reports, on 3D-FUTURE, CD $$f_\theta:(x,\omega_{\text{light}},\omega_{\text{out}})\rightarrow (\rho,\sigma),$$1 vs $$f_\theta:(x,\omega_{\text{light}},\omega_{\text{out}})\rightarrow (\rho,\sigma),$$2, F-Score $$f_\theta:(x,\omega_{\text{light}},\omega_{\text{out}})\rightarrow (\rho,\sigma),$$3 vs $$f_\theta:(x,\omega_{\text{light}},\omega_{\text{out}})\rightarrow (\rho,\sigma),$$4, IoU-B $$f_\theta:(x,\omega_{\text{light}},\omega_{\text{out}})\rightarrow (\rho,\sigma),$$5 vs $$f_\theta:(x,\omega_{\text{light}},\omega_{\text{out}})\rightarrow (\rho,\sigma),$$6, VLM $$f_\theta:(x,\omega_{\text{light}},\omega_{\text{out}})\rightarrow (\rho,\sigma),$$7 vs $$f_\theta:(x,\omega_{\text{light}},\omega_{\text{out}})\rightarrow (\rho,\sigma),$$8, and CLIP-S $$f_\theta:(x,\omega_{\text{light}},\omega_{\text{out}})\rightarrow (\rho,\sigma),$$9 vs $s_{t+1}=f_\theta(s_t,a_t).$00 relative to the best baseline SAM3D; on ScanNet it reports CD $s_{t+1}=f_\theta(s_t,a_t).$01, F-Score $s_{t+1}=f_\theta(s_t,a_t).$02, IoU-B $s_{t+1}=f_\theta(s_t,a_t).$03, VLM $s_{t+1}=f_\theta(s_t,a_t).$04, and CLIP-S $s_{t+1}=f_\theta(s_t,a_t).$05 as best results; and on MIT-Indoor-67 it reports realism $s_{t+1}=f_\theta(s_t,a_t).$06, functionality $s_{t+1}=f_\theta(s_t,a_t).$07, layout $s_{t+1}=f_\theta(s_t,a_t).$08, image-alignment $s_{t+1}=f_\theta(s_t,a_t).$09, average $s_{t+1}=f_\theta(s_t,a_t).$10, and CLIP $s_{t+1}=f_\theta(s_t,a_t).$11 (Kim et al., 7 Jun 2026).

System	Scene-object representation	Reported outcome
Object-centric Forward Modeling for MPC	2D object locations + implicit visual features	Lower distance-to-goal than all baselines
3D-OES	World-aligned 3D feature map + object crops and masks	MPC success $s_{t+1}=f_\theta(s_t,a_t).$12; Ours-Real $s_{t+1}=f_\theta(s_t,a_t).$13
OSF-based multi-object manipulation	Per-object KiloOSFs + graph dynamics + MPPI	Surpasses FitVid and compositional NeRF-based MPC
Object-aware visuomotor control	Slot masks + dense pretrained features + IBC	$s_{t+1}=f_\theta(s_t,a_t).$14 at $s_{t+1}=f_\theta(s_t,a_t).$15 episodes
HyperDet3D	Scene-conditioned hypernetwork modulation	$s_{t+1}=f_\theta(s_t,a_t).$16 mAP on ScanNet V2
SceneConductor	Planner, specialist agents, scene graph, scaffold	Best CD/F-Score/IoU-B/VLM on reported benchmarks

A plausible implication is that SOC gains do not depend on one modality alone. The reported improvements arise in RGB, RGB-D, multi-view rendering, point-map supervision, and point-cloud detection, provided that scene structure is controlled through object factorization and relational reasoning.

6. Limitations, misconceptions, and future directions

The literature also makes clear that SOC is not equivalent to complete scene understanding or universally robust control. OSF-based SOC has significant computational cost: training a KiloOSF per object takes time and requires multi-view data with light labels, while online rendering and CMA-based inversion are more expensive than simple feature-based pose estimation. Its lighting model is primarily directional, complex environment lighting and interreflections are approximated, the dynamics model may err under heavy occlusions or contact-rich events not captured by the training distribution, and inverse estimation can get stuck in local minima such as rotational symmetries and glossy materials (Tian et al., 2023).

3D-OES assumes rigid objects, quasi-static or moderately dynamic pushes, and detector supervision from ground-truth 3D boxes during training. The model is deterministic, does not explicitly represent long-horizon uncertainty, and can fail under heavy occlusions or contact-rich and deformable interactions. The paper explicitly proposes stochastic dynamics, explicit contact modeling, online adaptation of $s_{t+1}=f_\theta(s_t,a_t).$17, uncertainty-aware MPC, and integration with classical physics engines as extensions (Tung et al., 2020). Object-centric MPC in 2D is likewise limited by perception noise, occlusions, out-of-distribution interactions, large contact-rich dynamics, and the supervision requirement for object locations and correspondences (Ye et al., 2019).

Slot-based SOCs are vulnerable to slot ambiguity. With $s_{t+1}=f_\theta(s_t,a_t).$18 blocks, objects with the same color but slightly different shapes are harder, and the paper attributes degradation to slot assignment swapping under an $s_{t+1}=f_\theta(s_t,a_t).$19 reconstruction loss that is relatively insensitive to subtle geometric details. The method is also sensitive to the number of slots $s_{t+1}=f_\theta(s_t,a_t).$20, with oversplitting reported when $s_{t+1}=f_\theta(s_t,a_t).$21 is too large, and heavy occlusions remain challenging (Heravi et al., 2022). SceneConductor identifies additional failure modes specific to scene synthesis: severe occlusion or reflective surfaces may degrade point-map quality, early-stage mask errors can propagate, runtime overhead arises from multiple foundation models and Blender integration, and the current focus is on indoor, Manhattan-like layouts (Kim et al., 7 Jun 2026). HyperDet3D notes that scene-conditioned priors can still mis-handle fine local geometry, such as merging closely connected objects, and categories with scarce scene context in training may transfer poorly across datasets (Zheng et al., 2022).

A final misconception is that object-centric factorization alone guarantees good control. The reported systems rely on complementary mechanisms: graph message passing for interaction structure, reconstruction or rendering losses for perceptual grounding, repeated replanning or correction to limit drift, and scene-level priors for disambiguation. Future work named in the cited papers is correspondingly modular: broader lighting models and parameter identification for OSFs, stochastic or hybrid residual physics for 3D-OES, unsupervised emergence of objects for object-centric MPC, non-Manhattan priors and multi-light estimation for SceneConductor, and stronger geometry modeling for scene-conditioned detection (Tian et al., 2023, Tung et al., 2020, Ye et al., 2019, Kim et al., 7 Jun 2026, Zheng et al., 2022).

In this sense, SOC is best understood not as a single algorithm but as an organizing doctrine for scene-structured inference and control. The doctrine is consistent across robotic manipulation, scene synthesis, visuomotor imitation, and scene-conditioned detection: represent objects explicitly, model their interactions, couple prediction to a scene-aware decision rule, and keep the loop closed through correction, inversion, or iterative refinement.