Functional Object Arrangement (FORM)

• FORM is defined as the task of arranging a set of objects so they serve a latent high-level function, emphasizing functionality, physical feasibility, and aesthetic coherence.
• Researchers employ a range of methods—from explicit relational graphs and energy-based models to embedding spaces and neuro-symbolic approaches—to infer and generate functional object layouts.
• Challenges in FORM include transitioning from 2D to complex 6-DoF 3D arrangements, integrating context with robotic execution, and ensuring that inferred goals align with practical manipulation constraints.

Functional Object Arrangement (FORM) is the task of placing a set of objects $\mathcal{O}=\{o_1,\dots,o_N\}$ within a container or environment so that they collectively fulfill a high-level function, such as “set up a dining table for two.” In the contemporary literature, FORM is characterized by under-specified instructions, the need to infer object goal poses rather than receive them explicitly, and the requirement that an arrangement be functional, physically feasible, and aesthetically coherent rather than merely collision-free or visually regular (Xu et al., 4 Aug 2025, Xu et al., 2024). The field spans symbolic representations of functional preconditions, context-aware completion of partially arranged scenes, learned priors over continuous layouts, and robotic pipelines that ground inferred arrangements into executable manipulation plans.

1. Conceptual scope and task structure

FORM is broader than conventional goal-conditioned rearrangement. In goal-conditioned settings, the desired end state is usually specified by logical predicates, a goal image, or a natural-language description. FORM instead asks how a robot should arrange objects when the intended organization is latent, under-specified, or only partially demonstrated. ConSOR makes this distinction explicit by treating the pre-arranged subset of a scene as contextual evidence of a latent organizational schema and requiring the robot to infer a mapping from unarranged objects to receptacles without any explicit user goal specification (Ramachandruni et al., 2023).

The literature also distinguishes FORM from neighboring tasks such as semantic rearrangement, tidying, and open-set scene composition. StructFormer studies language-guided semantic rearrangement of novel objects into structures such as circles, lines, towers, and table settings, thereby addressing a concrete instance of arranging objects into semantically meaningful structures under continuous geometric constraints (Liu et al., 2021). By contrast, Knolling Bot learns a generalized notion of “tidiness” from scratch and produces collision-free, space-efficient tabletop layouts, but its stated goal is aesthetic organization rather than explicitly functional or task-oriented placement (Hu et al., 2023). Lay-A-Scene similarly targets plausible 3D scenes with physical coherence and non-penetration, while explicitly noting that it does not model human actors, reachable space, or explicit semantic relations (Rahamim et al., 2024).

A recurring implication is that FORM cannot be reduced to any single objective. Some systems emphasize latent schema inference from context, some emphasize functional priors extracted from demonstrations or large models, and some emphasize explicit relational constraints. The unifying theme is that arrangement quality is judged by how well the final configuration supports intended use, not just by geometric legality or visual neatness.

2. Representations of arrangements, relations, and functionality

A central design choice in FORM is the representation of what makes an arrangement “correct.” One line of work uses explicit relational structure. SetItUp introduces a grounding graph composed of abstract spatial relations among objects, with each relation associated with a classifier $h_R$ and a generative distribution $q_R(p_{1:k}\mid g_{1:k}) \propto \mathds{1}[h_R(g_{1:k},p_{1:k})]$. This decomposes the task into semantic inference over relations and geometric grounding of those relations into poses (Xu et al., 4 Aug 2025).

A second line uses implicit cost functions over continuous pose spaces. SceneScore defines a scene by fixed semantic identities $s$ and variable poses $x$, then learns an energy function $E_\theta(x\mid s)$ so that human-like arrangements have low cost. The induced energy-based model is

$$p_\theta(x\mid s)=\frac{\exp(-E_\theta(x\mid s))}{Z_\theta(s)}.$$

Its graph neural network representation separates “what” each object is from “where” it is located by placing semantic embeddings on nodes and relative pose features on fully connected edges (Kapelyukh et al., 2023).

A third line uses embedding spaces in which functionally related objects become nearby under contextual conditioning. ConSOR represents each object instance as a triple $(o_i,r_i,i)$, concatenates category, receptacle, and instance encodings, and applies a Transformer encoder to produce normalized object-centric latent vectors. At inference time, each unarranged object is assigned to the receptacle whose centroid embedding has maximum cosine similarity with the object’s embedding, so partially arranged context acts as the organizing signal (Ramachandruni et al., 2023).

A fourth, more symbolic tradition represents functional prerequisites for manipulation rather than geometric layouts directly. FOON models knowledge as a directed bipartite graph whose nodes are object states and motion types, and whose atomic building blocks are functional units $(\{N_O^{in}\},N_M,\{N_O^{out}\})$. In this formulation, arrangement is encoded as the set of objects and object states that must be present before a motion can be executed. FOON therefore captures functional dependencies and task structure, but not explicit metric relations such as “under,” “left of,” or “near,” which are deferred to motion generation (Paulius et al., 2019).

These representations are complementary rather than mutually exclusive. Explicit relation graphs are interpretable and composable, energy functions are differentiable and rank arbitrary scenes, contextual embeddings support implicit goal inference, and symbolic functional graphs expose object-state preconditions for downstream planning.

3. Learning paradigms and data regimes

FORM research employs a notably diverse set of supervision strategies. SceneScore learns entirely from example images of human-arranged tabletop scenes, without environment interaction or human supervision, by fitting an energy-based model with an InfoNCE-style objective and generating negative samples through Langevin Dynamics (Kapelyukh et al., 2023). Knolling Bot is also self-supervised, but from a different source: it synthesizes 2.4 million alternate tidy layouts in PyBullet using a stochastic area-minimization optimizer, then trains a transformer decoder to predict a Gaussian Mixture Model over object positions under a weighted sum of log-likelihood, MSE, MSE-min, overlap, and entropy terms (Hu et al., 2023).

Another regime uses large models as data generators and then distills their outputs into compact arrangement priors. LVDiffusor constructs synthetic prompts for functional scenes, generates top-down layouts with a VLM, refines missing or duplicated objects and coordinates with an LLM, and distills the resulting dataset into a conditional score-based diffusion model implemented as a graph neural network over a fully connected object graph (Zeng et al., 2023). GOPLA similarly addresses data scarcity through synthetic augmentation, but in 3D: human arrangement demonstrations are abstracted into scene graphs, expanded through crossover and mutation over functionally similar meshes, refined by collision optimization, and used to train an affordance mapper and a diffusion-based SE(3) planner (Zhong et al., 16 Oct 2025).

Few-shot neuro-symbolic learning is a distinct and influential pattern. SetItUp assumes only a small number of training examples per task family together with a human-written task specification or program sketch. LLMs induce structured rules or Python programs that produce a grounding graph for a novel instruction, while diffusion models trained for primitive relations are composed online to generate the final object poses (Xu et al., 2024). StructFormer, by contrast, learns directly from simulated demonstrations of structured language instructions and segmented point clouds, using self-attention to infer both which objects should move and the 6-DoF offsets required to realize the instructed structure (Liu et al., 2021).

Taken together, these systems suggest a broad methodological spectrum: unannotated image-only density estimation, synthetic self-supervision, large-model data distillation, few-shot program induction, and direct end-to-end geometric prediction all appear as viable routes into FORM, with different trade-offs in compositionality, personalization, and data efficiency.

4. Inference, optimization, and robotic execution

At inference time, FORM systems typically solve one of three problems: scoring candidate arrangements, completing partial arrangements, or synthesizing entirely new object poses. SceneScore is designed primarily as a cost function. Any candidate layout is evaluated by computing $E_\theta(x\mid s)$, and missing-object placement or full-scene synthesis is performed by minimizing this energy with Langevin Dynamics or gradient descent. Because the energy is differentiable, it can be augmented at test time with additional costs $h_R$0, yielding $h_R$1, for example to enforce collision penalties, reachability, or stability (Kapelyukh et al., 2023).

ConSOR solves a different inference problem: given partially arranged receptacles, it infers where new objects belong by embedding the initial scene, computing centroids over pre-arranged objects in each container, and assigning each unarranged object to the best-matching centroid. Knolling Bot, meanwhile, autoregressively decodes GMM parameters for object positions and can either sample diverse tidy layouts or choose the highest-probability component for deterministic output (Ramachandruni et al., 2023, Hu et al., 2023).

Diffusion-based FORM systems ground constraints into continuous poses in more explicit generative terms. SetItUp samples from the product of relation-specific pose distributions by summing learned denoisers for the relations in the grounding graph and running a ULA-style reverse process, so arbitrary sets of abstract constraints can be composed at test time (Xu et al., 2024). LVDiffusor instead samples planar goal poses from a distilled functional rearrangement prior by solving the probability-flow ODE of a conditional score model (Zeng et al., 2023). GOPLA extends this pattern to 6-DoF placement: a multi-modal LLM translates visual and language inputs into pairwise directional relations, a spatial mapper decodes them into 3D affordance activation maps, and a diffusion planner samples SE(3) poses while being steered by test-time affordance and collision costs (Zhong et al., 16 Oct 2025).

Execution on physical robots generally follows a perception-to-planning pipeline. SceneScore assumes overhead RGB perception, Mask R-CNN detections, pose estimation, and standard motion planning after pose optimization. StructFormer uses segmented point clouds, predicts a structure frame and object pose offsets, then hands the targets to GraspNet and RRT-Connect for real-world rearrangement by a Franka Panda (Kapelyukh et al., 2023, Liu et al., 2021). In this sense, FORM is typically not the low-level controller itself; it is the goal-specification and pose-generation layer that supplies actionable geometric targets to a task-and-motion planner.

5. Evaluation protocols, benchmarks, and empirical findings

FORM systems are evaluated on heterogeneous task families, which has shaped the empirical landscape of the field. SceneScore studies missing-object pose prediction, semantic generalization to novel objects through CLIP features, composition with collision constraints, and a real-world tabletop demonstration involving a book, pen, and mug. Its reported baselines include Nearest-Neighbor retrieval, NeatNet VAE, and an absolute-pose ablation, and its errors are measured in centimeters for translation and degrees for orientation (Kapelyukh et al., 2023). ConSOR evaluates context-aware semantic completion with Scene Edit Distance and dataset metrics $h_R$2 and $h_R$3; it reports near-perfect success, with at least $h_R$4 $h_R$5 in all four schemas on both seen-object and unseen-object tests (Ramachandruni et al., 2023).

SetItUp provides the most explicit multi-domain FORM evaluation among the surveyed systems. Across dining tables, bookshelves, and bedrooms, it measures physical feasibility, basic functionality, and human judgments of functionality and aesthetics. Its reported averages are 94.0% feasibility and 89.1% functionality on dining tables, 98.1% and 94.4% on bookshelves, and 93.8% and 91.5% on bedrooms, with corresponding human scores of 2.86/1.87, 2.92/1.95, and 2.87/1.87 for functionality and aesthetics. These figures exceed direct LLM prediction, end-to-end diffusion, and ablations that remove either the intermediate graph or the full program-induction pipeline (Xu et al., 4 Aug 2025).

Other systems validate specific advantages of their modeling choices. LVDiffusor reduces Coverage Score by more than 50% relative to DALL-E-Bot across four planar domains and maintains user-study scores of about 9.0 as scene complexity increases, whereas DALL-E-Bot drops from 8.5 to 4.2 (Zeng et al., 2023). GOPLA reports an average success-rate gain of about 30.04 percentage points over the runner-up across synthetic and real scenes and achieves 76% versus 32% in real-robot trials on a Hello Robot Stretch (Zhong et al., 16 Oct 2025). StructFormer reports 3.5 cm and $h_R$6 error on the circle task, compared with 8.3 cm and $h_R$7 for a no-structure ablation and 11.4 cm and $h_R$8 for a pairwise baseline, while end-to-end simulation success rises from 20% to 37% (Liu et al., 2021).

Benchmarking work has also exposed the gap between arrangement competence and general manipulation competence. RoboCAS defines Scattered, Orderly, and Stacked scenarios and reports that RT-1 achieves 34.2% success in Picking, 23.3% in Selecting, and 10.2% in Searching, while RoboFlamingo attains 15.6%, 13.3%, and 0%, respectively. The benchmark’s sub-step metrics—RA, PA, RR, and CO—show that current policies often reach some object but fail to pick the correct one or to manage occlusion and chain reactions in stacked scenes (Zheng et al., 2024).

6. Limitations, misconceptions, and future directions

A persistent misconception is that tidy, plausible, or semantically clustered layouts are equivalent to functional arrangements. The surveyed literature repeatedly rejects this equivalence. Knolling Bot explicitly focuses on neatness rather than task-driven placement, and semantic preferences enter only through input ordering heuristics rather than end-to-end functional reasoning (Hu et al., 2023). Lay-A-Scene enforces common support and non-penetration but states that affordances, ergonomics, and explicit functional groupings are absent, so visually coherent scenes may still be functionally awkward (Rahamim et al., 2024). SceneScore learns a powerful static cost over layouts, yet it does not explicitly model long-horizon, multi-step rearrangement (Kapelyukh et al., 2023).

Another recurring limitation is the dominance of 2D tabletop assumptions. SceneScore, Knolling Bot, SetItUp, and LVDiffusor all formulate poses primarily on planar workspaces; ConSOR reasons about container assignment but not precise placement within containers; and many systems assume direct top-down visibility, reliable category recognition, or fixed semantic identities (Ramachandruni et al., 2023, Xu et al., 2024, Zeng et al., 2023). GOPLA moves into 6-DoF placement but requires a partial mesh of the to-be-placed object and still reports difficulty with high-precision assembly below the centimeter scale (Zhong et al., 16 Oct 2025). More broadly, multi-modality remains challenging: ConSOR notes ambiguity when multiple valid schemas explain the same partial arrangement, while SetItUp notes that independently trained relation models can cause slight collisions when many constraints intersect tightly (Ramachandruni et al., 2023, Xu et al., 4 Aug 2025).

Recent work points toward several extensions. The literature repeatedly proposes richer semantics, explicit affordances, support and stability reasoning, and integration with sequential planning or hierarchical RL. SceneScore identifies 3D extensions and coupling with physics simulators as open directions; Knolling Bot highlights larger environments and richer semantics; SetItUp identifies automatic sketch generation and 3D relation modeling; FOON-based work points to explicit spatial predicates and probabilistic planning (Kapelyukh et al., 2023, Hu et al., 2023, Xu et al., 4 Aug 2025, Saini, 2022). A more recent 3D-oriented direction, VULCAN, treats arrangement as an iterative “Plan → Execute → Evaluate” loop with tool-mediated grounding and geometric constraints such as Contact, NoOverhang, FaceTo, Distance, and Rotate, suggesting one route beyond one-shot planar layout prediction toward robust multi-step 3D arrangement (Kuang et al., 26 Dec 2025).

The resulting picture is that FORM is not a single algorithmic problem but a layered one. It joins latent goal inference, semantic abstraction, geometric grounding, physical validation, and executable manipulation. Progress has been fastest where these layers are explicitly separated and then recomposed—through grounding graphs, energy functions, affordance maps, or functional object-motion networks—rather than collapsed into a single undifferentiated predictor.