---
title: Decompose–Localize–Recompose Pattern
url: https://www.emergentmind.com/topics/decompose-localize-recompose
type: topic
---
# Decompose–Localize–Recompose Pattern
Decompose–Localize–Recompose denotes a recurring design pattern in which a complex object, task, or computation is first factored into smaller units, then the units relevant to a particular input are selected, grounded, or routed, and finally a task-specific result is assembled from those units. Recent work instantiates this pattern under closely related names, including “decompose–recompose–decide” for cross-domain agentic workflow generation, “Decompose, Look, and Reason” for visual reasoning, “Deconstruct-Recompose” for reinforcement-learning pre-training from videos, and “collapse, copy and compress” for algebraic computation [2602.11114] [2604.07518] [2607.00808] [2504.04660]. Across these formulations, the framework is used to replace monolithic processing with structured intermediate representations whose composition can be adapted to new tasks, domains, or decomposition axes.
## 1. Conceptual organization
At its most general, the framework has three stages. **Decompose** introduces an intermediate basis, graph, skill inventory, atomic motion set, premise list, tag set, or typed algebraic structure. **Localize** determines which subset, weighting, or substructure is relevant for the current input. **Recompose** builds the final executable workflow, scene, action sequence, latent state, proof, or software artifact from the localized components. In cross-domain workflow generation, for example, the basis consists of reusable workflow capability bases; in open-vocabulary temporal action detection it consists of phase descriptions; in robotic manipulation it consists of atomic skill–action pairs; in graph transformation it consists of colimit components of a global state [2602.11114] [2603.24030] [2605.01448] [1008.2111].
The framework is not tied to one representational level. In some systems the decomposition is architectural, as with low-rank capability bases or per-object Gaussian sets. In others it is symbolic, as with scene graphs, skill sequences, or datatype/interface duals. In still others it is categorical or process-theoretic, as with semigroupoids, relational functors, pullback-stable colimits, or Linear Process Equations. Taken together, these formulations show that Decompose–Localize–Recompose is better understood as an organizational principle than as a single algorithm.
## 2. Forms of decomposition
The decomposition stage varies with the substrate being modeled. In agentic workflow generation, the decomposition is a library of low-rank workflow capability bases added to a frozen base LLM, with each basis parameterized as $\Delta B_k = c_k U_k V_k^\top$ and later recomposed by task-dependent coefficients in $M(q) = M + \sum_{k=1}^K \alpha_k(q)\Delta B_k$ [2602.11114]. In text-to-3D generation, the decomposition is a scene graph $G=(V,E)$ together with object prompts $t_i^o$, edge prompts $t_{ij}^e$, and object-wise Gaussian subsets $\mathcal{G}_{o_i}$ initialized from Point-E point clouds and aligned by VLM-inferred position, scale, and orientation [2503.11981]. In controllable image-text synthesis, the decomposition is a tag set $E=\{E_{\text{obj}},E_{\text{attr}},E_{\text{rel}}\}$ produced by a hybrid Vision Tagging Model that combines CatLIP with Florence-large captioning and Qwen2-7B-Instruct extraction [2410.11963].
Other works decompose along temporal, motor, or linguistic axes. Cross-task robotic manipulation converts demonstrations into atomic skill–action pairs $\{(s_k,a_k)\}_{k=1}^K$, where $a=[i_x,i_y,i_z,i_r,i_p,i_y,g]$ and $s_k$ is an atomic skill label such as Reach[obj], Move[obj], Grasp[obj], Release[obj], Insert[obj$_1$,obj$_2$], or Push[obj$_1$,obj$_2$] [2605.01448]. RL video pre-training deconstructs global motion into Atomic Actions defined by local optical-flow patches $u_t^{(k)}$ around tracked keypoints, then encodes them as spatiotemporal tokens [2607.00808]. DLR decomposes a visual question into a trajectory $\tau=\{(p^{(t)},\mathbf{z}^{(t)},r^{(t)})\}_{t=1}^N$ of textual premises, premise-conditioned visual latents, and grounded rationales [2604.07518]. OV-TAD decomposes action labels into start, middle, end, and global descriptions through CoT-Prompting Semantic Decomposition [2603.24030].
Formal work pushes the same idea into abstraction layers where “parts” are not neural or visual objects. Representation-independent decomposition models computation as semigroupoids with typed arrows, then separates a system into a top level induced by a surjective relational functor and a bottom level induced by the kernel $K_\varphi$ [2504.04660]. Structural decomposition of reactions of graph-like objects fixes a colimit decomposition $\xi:\mathbb{X}\to\Delta X$ of a state and asks whether a global DPO transformation can be reconstructed as the colimit of local ones [1008.2111]. Decompositional minimisation of monolithic processes partitions the parameter vector of a Linear Process Equation into subsets $V$ and $W$, inducing component processes that each govern only part of the monolithic state [2012.06468].
## 3. Localization as routing, grounding, and selection
Localization is the stage at which a decomposition becomes input-specific. In CapFlow, the composer maps a task embedding $z(q)$ to routing weights $\alpha(q)=g_\psi(z(q))$, sharpens or diffuses them with a task-dependent temperature $T(q)$, keeps only top-$m$ bases, and uses counterfactual capability attribution to adjust routing by each basis’s marginal contribution $\Delta_k$ [2602.11114]. In DecompDreamer, localization occurs through relation-guided optimization: joint and targeted FMD losses are applied to pairs of object Gaussian sets, and scene-aware translations $T_i^o=(x_i,y_i,z_i)$ correct imperfect VLM layouts [2503.11981]. In open-vocabulary temporal action detection, localization is phase-wise and temporal: Text-infused Foreground Filtering computes similarity between snippet features and phase text embeddings, derives binary foreground masks $\hat S_{fg}^p$, and filters video features into phase-specific foreground representations $F_v^p$ [2603.24030].
Several papers make explicit that localization need not be literal spatial cropping. In DLR, localization is premise-conditioned latent grounding: after a ` ... ` segment is generated, a visual grounder uses the hidden state at the closing premise token to extract continuous visual latents $\mathbf{z}^{(t)}\in\mathbb{R}^{L\times d}$ by cross-attending over the whole image, and SGLP samples on a hypersphere rather than over Euclidean magnitudes [2604.07518]. In CtrlSynth, localization is semantic rather than pixel-level: tags, relations, and controller policies raise or lower the influence of concepts without explicit bounding boxes or segmentation masks [2410.11963]. In semigroupoid decomposition, localization is carried by hierarchical coordinates and encode/decode morphisms that “locate the repeating patterns in the compression” [2504.04660].
Robotic and control-oriented systems make the same stage operational through retrieval or structured attention. Cross-task manipulation builds a task-adaptive dynamic demonstration library using visual similarity from DINOv3 and plan-based similarity from skill-sequence Jaccard scores, then augments it with a coverage-aware static library driven by IDF-weighted skill tokens [2605.01448]. RL pre-training from videos uses a Dual-Attention Encoder with Intra-Frame Attention and Inter-Frame Attention so that local motion tokens encode both spatial relations and temporal evolution before they are aggregated [2607.00808]. The shared theme is that localization converts a reusable but overcomplete decomposition into a sparse, grounded, or context-specific working subset.
## 4. Recompositional operators
Recomposition assembles localized components into a usable system. In CapFlow, sparse routing weights are broadcast across adapted layers to define a task-conditioned parameter set $\Theta(q)$, after which the model performs standard LLM decoding and generates an executable workflow in a single pass, $W \sim p_{\Theta(q)}(W\mid x(q))$ [2602.11114]. In DecompDreamer, recomposition is progressive rather than instantaneous: the optimization focus shifts from pairwise relationship modeling to object-focused refinement, with targeted edge optimization, object-view-aware FMD, negative prompts, and scene-level FMD preserving the relational scaffold while improving object quality [2503.11981]. In OV-TAD, Adaptive Phase-wise Alignment computes per-phase class scores and then aggregates them with learned phase weights $\omega_p$ to produce a final prediction over time [2603.24030].
In reasoning systems, recomposition takes the form of text generation conditioned on localized evidence. DLR inserts premise-conditioned latent tokens between ` ... ` tags and then generates grounded rationales and the final answer, so that textual reasoning is repeatedly reconnected to continuous visual evidence rather than relying on a single early textual summary [2604.07518]. In robotic manipulation, recomposition occurs in skill space: an LLM policy receives retrieved examples of the form $[\text{instr}_i,\text{obs}_i,\text{plan}_i]\to[(s_1,a_1),\ldots,(s_K,a_K)]$ and must output only low-level actions for the new task, effectively reassembling known skills into unseen action sequences without parameter updates [2605.01448]. In RL pre-training from videos, recomposition is realized by the Motion Aggregation Token and a latent dynamics model, later bridged to downstream action-conditioned dynamics by an adapter and an Action-Specific Dynamics Model [2607.00808].
Formal systems also define explicit recompositional operators. In representation-independent decomposition, the pinhole cascade product $T\wr_\varphi K_\varphi$ recomposes a top-level abstraction and a compressed bottom-level kernel, with Theorem 4.3 establishing an injective relational functor $S\hookrightarrow T\wr_\varphi K_\varphi$ [2504.04660]. In decompositional minimisation of monolithic processes, recomposition uses communication, allow, hide, and parallel composition operators to rebuild a strongly bisimilar process from parameter-restricted components [2012.06468]. In FOOD, recomposition is a bidirectional transformation between functional and object-oriented decompositions, so that a program can be re-expressed under the opposite decomposition axis while preserving type and semantics [2204.10411].
## 5. Empirical behavior
A recurring empirical pattern is that decomposition helps when the baseline must either search globally or align globally. CapFlow’s 1-pass generator surpasses refinement systems that consume 20 iterations; PDA improves over label-level open-vocabulary alignment; DLR improves over text-only, interleaved multimodal CoT, and latent reasoning baselines; DecompDreamer improves compositional 3D generation against graph- and SDS-based baselines; CtrlSynth improves data efficiency and long-tail recognition for CLIP-style training [2602.11114] [2603.24030] [2604.07518] [2503.11981] [2410.11963].
| Setting | Representative result | Citation |
|---|---|---|
| Cross-domain agentic workflows | CapFlow (1-pass): Solve 74.19%, Exec 98.03%; AFlow (20 iter): Solve 72.93%, Exec 88.13% | [2602.11114] |
| Text-to-3D generation | DecompDreamer: CLIP 34.5, Pick-A-Pic 22.5, User study 54%; next best LucidDreamer 32.34, HiFA 16.9, GraphDreamer 18.3 | [2503.11981] |
| Zero-shot cross-task manipulation | Ours: Level-1 32.5%, Level-2 18.5%, Overall 26.4%; X-ICM: 28.6%, 16.9%, 23.5% | [2605.01448] |
| Vision-language reasoning | DLR: V* 83.8, MathVista 67.5, MMMU-Pro 56.1, MMStar 65.2 | [2604.07518] |
| Open-vocabulary temporal action detection | THUMOS14 50/50 Avg mAP: PDA 46.9, Ti-FAD 41.2; 75/25 Avg mAP: PDA 52.1, Ti-FAD 46.8 | [2603.24030] |
| Data-efficient multimodal learning | CC3M zero-shot average: 27.1 vs 19.4; CC3M retrieval mean R@1: 37.1 vs 13.7 | [2410.11963] |
Ablation results indicate that the middle stage is usually decisive. In CapFlow, sparse top-2 or top-3 compositions outperform dense mixtures and single-basis routing, and removing CCA degrades solve rate across all domains [2602.11114]. In DLR, removing $\mathcal{J}_{\text{latent}}$ drops MathVista from 67.5% to 57.1%, isolating latent localization as the critical factor [2604.07518]. In PDA, the progression from global label alignment to CSD, then TIF, then full APA yields 40.3, 42.1, 43.6, and 46.9 Avg mAP on THUMOS14 50/50, indicating that decomposition alone is weaker than decomposition plus localization plus adaptive recomposition [2603.24030]. In DecompDreamer, only $\mathcal{L}_{\text{joint}}$ yields good high-level layout but heavy object blending, while the progressive schedule yields the best balance [2503.11981]. In cross-task manipulation, the overall score rises from 21.6% for random demos with no skill labels, to 23.3% with the Dynamic Library, to 24.9% with the Coverage-aware Library, to 26.4% with Skill-Augmented ICL [2605.01448].
## 6. Formal guarantees, misconceptions, and limitations
A common misconception is that decomposition must produce explicit, human-readable modules. Several papers contradict that assumption directly. CapFlow’s learned capabilities are latent low-rank weight updates rather than explicit workflow operators [2602.11114]. DLR localizes continuous visual latents rather than fixed patches or cropped regions [2604.07518]. RL pre-training from videos uses generic tracked keypoints rather than semantic joints [2607.00808]. CtrlSynth localizes edits at the level of semantic tags and relations, not pixel masks [2410.11963]. Taken together, these works show that decomposition may be parametric, latent, semantic, temporal, or categorical.
Another misconception is that recomposition is merely additive concatenation. In practice it may be sparse linear mixing, attention-based aggregation, graph-conditioned optimization, retrieval-conditioned prompting, a latent dynamics transition, a cascade product, or a type-directed program transformation. Representation-independent decomposition formalizes this through collapse, copy, and compress, with hierarchical connections used only to locate repeating patterns [2504.04660]. FOOD proves that its bidirectional transformation preserves type and semantics, while the monolithic-process framework proves preservation under strong bisimulation for a suitable synchronization context [2204.10411] [2012.06468]. Structural decomposition of graph-like reactions gives soundness of colimit decomposition for arbitrary DPO transformations and an accommodated completeness theorem for a class of local decomposition problems [1008.2111]. These formal results establish that the topic is not confined to heuristic modularization; it also has categorical, semantic, and process-theoretic formulations with explicit correctness claims.
The limitations reported in the literature are equally consistent. CapFlow relies on a fixed operator set and workflow DSL, and attribution adds training cost even with subsampling and periodic updates [2602.11114]. DecompDreamer remains object-level rather than part-level, and its quality is bounded by SDS/FMD limitations such as Janus artifacts and view bias [2503.11981]. Cross-task manipulation depends on VLM skill labeling and on the planning agent $T_{\text{plan}}$, so errors in either can corrupt retrieval and recomposition [2605.01448]. DLR remains image-centric and uses a frozen external model for oracle attention in the focus reward [2604.07518]. DRP assumes that transferable local motion patterns exist across morphologies, an assumption that the paper argues empirically but still frames as its central premise [2607.00808]. CtrlSynth has no explicit spatial localization and inherits errors from pretrained taggers, captioners, and diffusion models [2410.11963]. In the formal papers, applicability depends on structural conditions such as accommodation, mono side conditions, or the choice of strong bisimulation and restricted OO/FP fragments [1008.2111] [2012.06468] [2204.10411].
These limitations do not define a single failure mode; they instead indicate that each stage can break independently. Decomposition can be too coarse, localization can be misgrounded, and recomposition can preserve the wrong invariants. The overall literature therefore presents Decompose–Localize–Recompose not as a universal recipe but as a disciplined way to structure search, transfer, and reconstruction across domains ranging from agentic workflows and 3D generation to robotics, video understanding, software design, algebraic computation, and graph transformation.
Source: https://www.emergentmind.com/topics/decompose-localize-recompose