Cross-category Object Fusion (COF)

• COF is a dual-usage concept that fuses objects across categories, applied in both dynamic-scene RGB-D reconstruction and hierarchical text-to-image synthesis.
• In RGB-D reconstruction, COF segments and tracks moving objects using motion and semantic cues, enabling real-time 3D model fusion with precise pose estimation.
• In text-to-image generation, the COF dataset provides a structured benchmark with 95 superclasses and 451,250 fusion pairs, ensuring balanced semantic fusion and evaluation.

Cross-category Object Fusion (COF) denotes two related but technically distinct usages in the supplied literature. In RGB-D dynamic-scene reconstruction, it refers to a Co-Fusion-style approach that takes a live stream of RGB-D images as input, segments the scene into different objects using motion or semantic cues, independently tracks each object and the background, and fuses only the pixels associated with each object label into that object’s 3D model in real time (Rünz et al., 2017). In text-to-image generation, COF denotes a large-scale, hierarchically structured dataset built on ImageNet-1K and WordNet for cross-category object fusion, with 95 superclasses, 10 subclasses per superclass, and up to $\binom{950}{2}=451{,}250$ distinct fusion pairs (Zhang et al., 23 Sep 2025). The common theme is object-level fusion across category or identity boundaries, but the operational meaning differs substantially between geometric reconstruction and semantic image synthesis.

1. Terminological scope and conceptual split

A common source of confusion is terminological: in the supplied literature, COF does not denote a single canonical framework. One usage is a recasting of Co-Fusion as a cross-category object fusion approach for dense SLAM in dynamic scenes, where multiple rigid models are maintained, segmented, tracked, and reconstructed concurrently. The other usage is the COF dataset introduced alongside AGSwap, where the term names a benchmark for fusing cross-category concepts in text-to-image generation (Rünz et al., 2017).

In the reconstruction setting, “fusion” refers to geometric and photometric integration of per-object RGB-D observations into surfel clouds or, in an alternative implementation, TSDF volumes. In the generative setting, “fusion” refers to synthesizing a single coherent image from two categories or concepts, with evaluation driven by CLIP-based alignment, balance, VQA consistency, and human or LLM scores (Zhang et al., 23 Sep 2025). This distinction is essential: the former is a multi-model estimation and mapping problem under $SE(3)$ motion, whereas the latter is a benchmarked compositional synthesis problem over semantic categories.

2. COF in dynamic-scene RGB-D reconstruction

In the Co-Fusion formulation, the input is a live stream of RGB-D frames, specifically color image $\mathbf{C}_t$ and depth $\mathbf{D}_t$. The system initializes by fusing the very first few frames into a single background model and stabilizing the camera pose. Thereafter, the per-frame loop comprises tracking, segmentation, and fusion (Rünz et al., 2017).

Tracking independently estimates a 6-DOF transform $T_{t,m}$ for each active model $m$, whether background or object. Segmentation assigns each pixel or super-pixel to one of the active models by either motion segmentation or semantic instance segmentation. Motion segmentation uses a CRF whose unary cost is the ICP-alignment error under each $T_{t,m}$, while its pairwise term encourages spatial, color, and depth smoothness. Semantic segmentation uses a pretrained SharpMask network trained on COCO to obtain per-pixel class and instance masks, which are then fused over time for temporal consistency. Fusion then integrates only the depth and color of pixels labeled $m$ into that model’s 3D representation.

This formulation differs from earlier dynamic-scene systems that treated moving regions as outliers and therefore did not model their shape or track their motion over time. By contrast, Co-Fusion maintains 3D models for segmented objects and improves them over time through fusion. The stated implication is that a robot can maintain a scene description at the object level, with the potential to allow interactions with its working environment even in dynamic scenes (Rünz et al., 2017).

3. Multi-model fitting, tracking energy, and per-object fusion

At time $t$, the system maintains $M_t$ active rigid models, each with map $SE(3)$0 and pose $SE(3)$1. Pose estimation for model $SE(3)$2 minimizes the tracking cost

$SE(3)$3

The point-to-plane ICP term is

$SE(3)$4

where $SE(3)$5 is the back-projected vertex $SE(3)$6 from the live depth $SE(3)$7, and $SE(3)$8 are the corresponding vertex and normal from the model’s predicted view at $SE(3)$9. The photometric term is

$\mathbf{C}_t$0

where $\mathbf{C}_t$1 denotes pixels currently labelled as object $\mathbf{C}_t$2, and $\mathbf{C}_t$3 is the grayscale intensity of the live color frame. Optimization is performed by Gauss-Newton on a 4-level image pyramid, on GPU (Rünz et al., 2017).

For 3D fusion, Co-Fusion uses a surfel-based representation. Each model $\mathbf{C}_t$4 is a set of surfels $\mathbf{C}_t$5 with attributes

$\mathbf{C}_t$6

Given a new measurement at pixel $\mathbf{C}_t$7 with depth $\mathbf{C}_t$8, normal $\mathbf{C}_t$9, color $\mathbf{D}_t$0, and weight $\mathbf{D}_t$1, fusion updates are

$\mathbf{D}_t$2

$\mathbf{D}_t$3

$\mathbf{D}_t$4

An alternative TSDF-style implementation is also specified. For per-voxel $\mathbf{D}_t$5,

$\mathbf{D}_t$6

where $\mathbf{D}_t$7 is the truncated signed distance at $\mathbf{D}_t$8 and $\mathbf{D}_t$9.

Segmentation under motion cues is written as a CRF energy over super-pixels $T_{t,m}$0,

$T_{t,m}$1

using fully connected CRF inference on super-pixels with features $T_{t,m}$2. The supplied summary further states that motion cues discover any rigidly moving cluster of pixels without pre-training, whereas semantic cues use SharpMask trained on COCO to propose instance masks for approximately 80 object classes, even if they remain static. Once a pixel is assigned to an instance mask or motion cluster, all future fusions inherit that object’s label or class; a per-surfel label distribution can be maintained and fused via simple histogram voting (Rünz et al., 2017).

4. Model management, real-time behavior, and reported reconstruction quality

The Co-Fusion system maintains two lists of models: active models $T_{t,m}$3 that are currently visible, and inactive models that were previously seen but are out of view. Each $T_{t,m}$4 may be represented either as a surfel cloud with position, normal, color, and weight, or as a TSDF volume. The pool of rigid transforms $T_{t,m}$5 is updated every frame. New models are spawned when the “outlier” label occupies more than $T_{t,m}$6 of pixels, and models are retired when they disappear for $T_{t,m}$7 frames (Rünz et al., 2017).

The reported runtime is approximately $T_{t,m}$8 Hz on a single GPU for fusion and tracking, plus a multicore CPU for segmentation. Synthetic benchmarks on ToyCar3 and Room4 report camera-pose AT-RMSE of approximately $T_{t,m}$9–$m$0 mm and object-pose AT-RMSE of approximately $m$1–$m$2 mm. Intersection-over-Union for motion segmentation is typically greater than $m$3. For real hand-held scans of unknown objects, the mean 3D reconstruction error is approximately $m$4 mm, with fewer than $m$5 of points outside a $m$6 cm threshold. Demonstrations include up to five simultaneous objects, including bins, bottles, plush toys, tools, and cars, in dynamic scenes such as handover, placing multiple items, and driving (Rünz et al., 2017).

These reported characteristics situate the approach as an object-level dense SLAM system rather than a background-only tracker. A plausible implication is that the active/inactive model split and per-object fusion policy are as important as the tracking objective itself for maintaining persistent identities in dynamic scenes.

5. COF as a hierarchically structured dataset for text-to-image fusion

In the AGSwap literature, COF is introduced as a dataset built on the $m$7 fine-grained categories of ImageNet-1K and the hypernym/hyponym graph of WordNet. Its assembly follows three steps: superclass candidate mining, manual curation, and subclass balancing (Zhang et al., 23 Sep 2025).

Superclass candidate mining extracts, for each ImageNet class $m$8, its WordNet hypernym path

$m$9

and collects all intermediate nodes whose descendant sets intersect the ImageNet label set. This yields $T_{t,m}$0 candidates. Manual curation prunes visually incoherent or overly fine-grained candidates, such as “wild_dog” versus “dog,” so that each superclass groups visually and semantically consistent subclasses. The final set has $T_{t,m}$1 superclasses. Subclass balancing assigns exactly $T_{t,m}$2 subclasses to each superclass, using random down-sampling when a superclass has more than ten leaf nodes and WordNet hyponym expansion when it has fewer.

The resulting hierarchy contains WordNet synsets such as “canine,” “marine mammals,” “tool,” and “furniture,” each with ten ImageNet leaf-node labels. Under “canine,” for example, listed subclasses include “red wolf,” “fox,” and “dingo” (Zhang et al., 23 Sep 2025).

Item	Value
Superclasses $T_{t,m}$3	95
Subclasses per superclass $T_{t,m}$4	10
Total leaf categories $T_{t,m}$5	950
Full unique unordered fusion pairs $T_{t,m}$6	451,250
Intra-superclass pair count $T_{t,m}$7	4,275

The distinction between the two pair counts is important. The full cross-category set is $T_{t,m}$8, while $T_{t,m}$9 counts only fusion within each superclass for intra-group evaluation. A common misunderstanding is to treat the smaller figure as the dataset’s total size; the supplied definition explicitly assigns it only to the within-superclass setting.

COF supports exhaustive combination over all $m$0 cross-category text pairs $m$1. Sampling may be uniform or superclass-balanced to avoid long-tail bias. The supplied hierarchy also supports inter-superclass sampling, where two distinct superclasses are chosen before selecting one subclass from each, and intra-superclass sampling, where two subclasses from the same superclass are chosen to study “fine-tuned” fusions. For large-scale evaluation, a subset such as COF-tiny may be formed with one subclass per superclass, giving $m$2 labels and $m$3 pairs (Zhang et al., 23 Sep 2025).

6. Evaluation protocol, AGSwap, and downstream uses

COF is designed as a benchmark with two primary tasks: simple prompt fusion and complex prompt fusion. In simple prompt fusion, given text labels $m$4, a method generates an image $m$5. Evaluation uses four metrics. The first is average concept alignment,

$m$6

for which higher is better because both concepts should be present. The second is balance score,

$m$7

for which lower is better because no one concept should dominate. The third is VQA consistency, where a CLIP-based VQA model answers “A realistic photo of creative hybrid with $m$8 and $m$9,” and the answer confidence is measured. The fourth comprises Human/LLM scores, where ChatGPT-4o assigns $t$0–$t$1 ratings for Surprise, Value, Novelty, and an Overall creativity score (Zhang et al., 23 Sep 2025).

In complex prompt fusion, GPT-4o is used to craft multi-sentence or richly descriptive prompts for $t$2, and the same metrics are applied. AGSwap is described as comprising two key components: Group-wise Embedding Swapping, which fuses semantic attributes from different concepts through feature manipulation, and Adaptive Group Updating, a dynamic optimization mechanism guided by a balance evaluation score to ensure coherent synthesis. When fusing two input images $t$3, AGSwap computes

$t$4

where $t$5 is cosine similarity in CLIP space, and iteratively swaps embedding groups to drive $t$6 (Zhang et al., 23 Sep 2025).

The stated uses of COF include training embedding-fusion models or prompt-finetuning methods across semantic gaps such as animal+insect, vehicle+tool, and food+electrical device; evaluating zero-shot or few-shot fusion generalization across $t$7 held-out pairs; organizing benchmark tasks and leaderboards for simple fusion, creativity, and hierarchical fusion; and supporting downstream applications such as virtual product design, game asset generation, and digital art and storytelling (Zhang et al., 23 Sep 2025). This suggests that the dataset is intended not only as a benchmark for output quality but also as a stress test for semantic balance across large taxonomic distances.

7. Relation between the two COF usages

The two COF usages share an object-centric perspective but operate on different primitives. The Co-Fusion line works on RGB-D frames, rigid transforms, surfels or TSDFs, CRF-based segmentation, and real-time map maintenance. The AGSwap line works on ImageNet-1K labels, WordNet hierarchy, CLIP-space similarities, prompt construction, and semantic attribute fusion (Rünz et al., 2017).

Their notion of “category” also differs. In the reconstruction setting, categories may arise from motion clusters or semantic instance masks, and once assigned, future fusions inherit an object’s label or class. In the dataset setting, categories are fixed by a curated hierarchy of superclasses and subclasses, and fusion pairs are generated explicitly over that label space (Zhang et al., 23 Sep 2025). The former addresses dynamic-scene perception with independent rigid motion; the latter addresses compositional generation with semantic coherence and balance.

Accordingly, “cross-category object fusion” should not be treated as a single research problem with one evaluation regime. In one strand it denotes real-time segmentation, tracking, and per-object 3D fusion; in the other it denotes category-pair construction and benchmarking for creative hybrid synthesis. What unifies them is the rejection of single-object or single-category assumptions in favor of explicit modeling of multiple object identities and their interactions.