InstaFormer: Instance-Aware Vision Transformers

• InstaFormer is a suite of Transformer-based vision architectures that integrate global and instance-level tokens for tasks like image translation and order prediction.
• It employs adaptive instance normalization and instance-level contrastive losses to maintain detailed object structure and style coherence during domain translation.
• The later variant uses object queries with latent mask descriptors to efficiently predict occlusion and depth ordering from a single RGB image.

InstaFormer is a name used for distinct Transformer-based vision architectures centered on instance-level reasoning. In one line of work, InstaFormer denotes an instance-aware image-to-image translation framework that integrates global tokens and instance tokens, replaces LayerNorm with adaptive instance normalization, and introduces an instance-level content contrastive loss for multimodal unpaired translation (Kim et al., 2022). In a later line of work, InstaFormer denotes a network for holistic order prediction in natural scenes, where object queries and latent mask descriptors interact to predict full occlusion and depth order matrices for all instances in a single forward pass (Musacchio et al., 2 Oct 2025). Across these usages, the name is consistently associated with architectures that combine per-instance structure with a global Transformer stage, although the target tasks, supervision, and inference regimes differ substantially.

1. Nomenclature and scope

The exact term “InstaFormer” refers most directly to the 2022 model titled “InstaFormer: Instance-Aware Image-to-Image Translation with Transformer” and to the 2025 architecture proposed for holistic order prediction in natural scenes. The former addresses multimodal, unpaired image-to-image translation with explicit instance awareness during training; the latter predicts complete occlusion and depth ordering relations among all object instances in a scene from a single RGB image (Kim et al., 2022, Musacchio et al., 2 Oct 2025).

The term should be distinguished from several near-homonymous models in adjacent literatures. “InterFormer” addresses real-time interactive image segmentation with an offline ViT encoder and an online interactive multi-head self-attention decoder (Huang et al., 2023). “InstanceFormer” denotes an online video instance segmentation framework built on Deformable-DETR with memory cross-attention and temporal contrastive loss (Koner et al., 2022). “InstFormer” denotes an open-vocabulary video instance segmentation framework combining open-world mask proposals, InstCLIP, and rollout association (Guo et al., 2023). “In-Context Former” is a context-compression model for LLMs based on digest tokens and cross-attention rather than a computer-vision instance model (Wang et al., 2024).

2. InstaFormer for instance-aware image-to-image translation

In the 2022 formulation, InstaFormer is a one-sided, unpaired image-to-image translation framework operating between domains $\mathcal{X}$ and $\mathcal{Y}$. Its main components are a content encoder $\mathcal{E}$, a style encoder $\mathcal{S}$, a Transformer-based aggregator $\mathcal{T}$, a generator $\mathcal{G}$, and a discriminator $\mathcal{D}$. Given a source image $\mathbf{x}$, the content encoder produces $$\mathbf{c} = \mathcal{E}(\mathbf{x}) \in \mathbb{R}^{h \times w \times l_c}$$, a style code is sampled from $\mathcal{N}(0,I)$, instance-level features are extracted with RoIAlign from training-time bounding boxes, and global patch embeddings together with one token per instance are concatenated into a single sequence processed by an encoder-only ViT stack (Kim et al., 2022).

The architectural novelty lies in how global and instance content are mixed. Global patch tokens come from an overlapped convolutional embedding of the content feature map, while each training-time bounding box yields a single instance token after RoIAlign and a convolutional projection. The model adds positional encodings for both global grid patches and instances using sinusoidal high-frequency mappings over center coordinates and box sizes. Self-attention over the joint token set allows global tokens to attend to instance tokens and vice versa, so that scene-level appearance and object-level structure are optimized jointly.

A second modification is the replacement of standard Transformer LayerNorm with adaptive instance normalization. In each Transformer block,

$\mathcal{Y}$0

Here the style code is converted into affine parameters that modulate token normalization before both attention and MLP sublayers. This turns style injection into a first-class operation of the Transformer rather than an auxiliary conditioning pathway. The result is a multimodal translation model in which different sampled style codes produce different target-domain appearances while content is preserved by the token interactions and contrastive objectives.

3. Objectives, multimodality, and empirical behavior of the 2022 model

The 2022 InstaFormer optimizes four losses: adversarial loss $\mathcal{Y}$1, global content contrastive loss $\mathcal{Y}$2, instance-level content contrastive loss $\mathcal{Y}$3, style reconstruction loss $\mathcal{Y}$4, and image reconstruction loss $\mathcal{Y}$5, combined as

$\mathcal{Y}$6

with $\mathcal{Y}$7, $\mathcal{Y}$8, $\mathcal{Y}$9, and $\mathcal{E}$0 (Kim et al., 2022).

The global contrastive term follows CUT/F-LSeSim-style patchwise InfoNCE and preserves scene layout. The instance-level contrastive term applies InfoNCE within each RoIAlign-extracted instance region, comparing translated instance patches to their corresponding source patches while using other patches in the same instance as negatives. This explicitly constrains local object structure and is the mechanism most directly responsible for the paper’s “instance-aware” designation. The style reconstruction loss enforces consistency between sampled style codes and re-encoded styles from generated images, while the image reconstruction loss preserves invertibility on target-domain images.

Implementation details are tightly specified. The content encoder downsamples a $\mathcal{E}$1 input to an $\mathcal{E}$2 feature map with 256 channels; the style encoder produces a style code of dimension $\mathcal{E}$3; the Transformer aggregator uses $\mathcal{E}$4 blocks, hidden dimension $\mathcal{E}$5, feedforward dimension 4096, and 4 attention heads; training uses Adam for 200 epochs with initial learning rate $\mathcal{E}$6, batch size 8, and a single 24GB RTX 3090 GPU. The main benchmarks are INIT for domain translation and KITTI $\mathcal{E}$7 Cityscapes for domain-adaptive detection.

Quantitatively, the model reports the highest CIS and IS across INIT domain pairs, better average FID than MGUIT on sunny$\mathcal{E}$8night, and instance SSIM around 0.845 versus MGUIT’s approximately 0.842. On KITTI$\mathcal{E}$9Cityscapes, translated images generated by InstaFormer support a Faster-RCNN detector that reaches mAP $\mathcal{S}$0, compared with $\mathcal{S}$1 for MGUIT and $\mathcal{S}$2 for DUNIT. Ablations show that removing the instance-level NCE loss weakens attention on objects and blurs small instances, replacing the Transformer with ResBlocks worsens FID and SSIM, and replacing AdaIN with LayerNorm yields almost single-modal outputs. This suggests that the three defining ingredients—joint global/instance tokenization, AdaIN-modulated Transformer blocks, and instance-level contrastive supervision—operate as a coupled design rather than as interchangeable modules.

4. InstaFormer for holistic order prediction

In the 2025 formulation, InstaFormer addresses holistic order prediction: given a single RGB image, it predicts the complete occlusion and depth ordering relations among all object instances in one forward pass. Occlusion is represented by a binary adjacency matrix $\mathcal{S}$3, where $\mathcal{S}$4 means instance $\mathcal{S}$5 occludes instance $\mathcal{S}$6. Depth order is represented by a ternary matrix $\mathcal{S}$7, where $\mathcal{S}$8 means $\mathcal{S}$9 is in front of $\mathcal{T}$0, $\mathcal{T}$1 means $\mathcal{T}$2 is not in front of $\mathcal{T}$3, and $\mathcal{T}$4 means the two instances have overlapping depth ranges (Musacchio et al., 2 Oct 2025).

The architecture is built on a frozen Mask2Former backbone. From the image, the segmentation stage yields object embeddings $\mathcal{T}$5 and per-pixel embeddings $\mathcal{T}$6. Dynamic mask decoding produces binary masks

$\mathcal{T}$7

During training, Hungarian matching selects the subset corresponding to real instances, producing $\mathcal{T}$8 and aligned masks $\mathcal{T}$9. The per-pixel embeddings are then masked instance-wise to obtain one feature map per object.

A central concept is the latent mask descriptor. Rather than directly pooling masked pixels, the model applies masked self-attention within each instance region and then max-pools over the masked spatial support to obtain $\mathcal{G}$0. These descriptors are intended to preserve spatial, texture, and local-geometry cues that are not fully captured by Mask2Former’s object queries. InstaFormer then concatenates the two aligned token sets,

$\mathcal{G}$1

and processes them with a Transformer stack comprising 1 encoder layer and 8 decoder layers.

After the final layer, updated query and descriptor branches are separately projected:

$\mathcal{G}$2

All pairwise order scores are then obtained by a single matrix multiplication,

$\mathcal{G}$3

Small task-specific heads map $\mathcal{G}$4 to occlusion logits $\mathcal{G}$5 and depth logits $\mathcal{G}$6. The model therefore replaces the classical pairwise regime—one forward pass per ordered pair—with adjacency-matrix prediction over the entire instance set.

5. Supervision, efficiency, and evaluation of the 2025 model

The 2025 InstaFormer is trained with standard Mask2Former segmentation losses together with geometry supervision. Occlusion uses binary cross-entropy over off-diagonal matrix entries, depth uses three-class cross-entropy, and the total loss for the joint model is

$\mathcal{G}$7

with $\mathcal{G}$8. Because the Mask2Former backbone is frozen, the model introduces AdaptFormer-style adapters in each FFN of the segmentation transformer decoder and pixel decoder, trained jointly with the geometry module (Musacchio et al., 2 Oct 2025).

The evaluation dataset is InstaOrder, built on COCO images with instance segmentation masks and pairwise occlusion and depth labels. Occlusion is measured by precision, recall, and F1, while depth is measured by weighted human disagreement rate (WHDR) for distinct, overlap, and all pairs. To isolate ordering quality from segmentation quality, predicted instances are re-aligned to ground truth by Hungarian matching at evaluation time. Training uses Mask2Former backbones with Swin-T/S/B/L variants; the geometry module uses 8-head attention, 512-dimensional projections, and 2-layer FFNs with hidden size 2048; the ordering module has 34M parameters; training runs for 120k iterations on 4 RTX A6000 GPUs with AdamW and learning rate $\mathcal{G}$9 decayed at 80k and 110k steps.

The reported results separate occlusion-only, depth-only, and joint models. For occlusion, InstaFormer$\mathcal{D}$0 with Swin-L$\mathcal{D}$1 reports precision $\mathcal{D}$2, recall $\mathcal{D}$3, and F1 $\mathcal{D}$4, slightly exceeding the strongest pairwise baseline InstaOrderNet$\mathcal{D}$5 at F1 $\mathcal{D}$6. For depth, InstaFormer$\mathcal{D}$7 reports WHDR(all) $\mathcal{D}$8, and the joint InstaFormer$\mathcal{D}$9 improves further to WHDR(all) $\mathbf{x}$0, with distinct WHDR $\mathbf{x}$1 and overlap WHDR $\mathbf{x}$2. These values substantially outperform pairwise baselines such as InstaOrderNet$\mathbf{x}$3 at WHDR(all) approximately $\mathbf{x}$4, Mask2Order$\mathbf{x}$5 at approximately $\mathbf{x}$6, MiDaS at approximately $\mathbf{x}$7, and a Depth Anything V2 plus SAM heuristic pipeline at approximately $\mathbf{x}$8.

Efficiency is one of the paper’s primary claims. InstaFormer’s runtime is described as essentially constant with respect to the number of instances, whereas InstaOrderNet’s runtime grows dramatically because it performs repeated pairwise passes. InstaFormer becomes faster than InstaOrderNet once there are approximately 7 or more instances and can be up to approximately $\mathbf{x}$9 faster when there are approximately 20 instances. Memory grows linearly with instance count, while a batched pairwise baseline becomes infeasible beyond approximately 20 instances because of out-of-memory behavior. Ablations further show that the cross interaction $$\mathbf{c} = \mathcal{E}(\mathbf{x}) \in \mathbb{R}^{h \times w \times l_c}$$0 is markedly better than queries-to-queries or descriptors-to-descriptors scoring, that one masked self-attention layer before max pooling gives the best depth performance, and that removing adapters degrades depth WHDR(all) from $$\mathbf{c} = \mathcal{E}(\mathbf{x}) \in \mathbb{R}^{h \times w \times l_c}$$1 to $$\mathbf{c} = \mathcal{E}(\mathbf{x}) \in \mathbb{R}^{h \times w \times l_c}$$2. The paper also notes residual failure modes such as instance mixup and segmentation failure, indicating that holistic order prediction remains coupled to upstream mask quality.

6. Conceptual relation and surrounding literature

The two exact InstaFormer models solve different problems, but they share a recognizable design pattern. The 2022 system combines global content tokens with instance tokens inside a style-conditioned Transformer for image translation, whereas the 2025 system combines object queries with latent mask descriptors inside a geometry Transformer for adjacency-matrix prediction (Kim et al., 2022, Musacchio et al., 2 Oct 2025). This suggests a recurring use of the name for architectures in which instance-level descriptors are not treated as auxiliary metadata but as first-class tokens participating in the main Transformer computation.

The surrounding literature clarifies the specificity of this usage. InterFormer addresses click-based interactive segmentation with cached ViT features and an interactive multi-head self-attention decoder rather than instance-aware generation or geometric ordering (Huang et al., 2023). InstanceFormer is an online video instance segmentation framework focused on prior propagation, memory cross-attention, and temporal contrastive loss for long videos (Koner et al., 2022). InstFormer is an open-vocabulary video instance segmentation system that couples open-world mask proposals, InstCLIP, and universal rollout association (Guo et al., 2023). In-Context Former is a digest-token cross-attention compressor for long-context LLMs and belongs to an entirely different problem class (Wang et al., 2024).

Within that broader naming field, the two InstaFormer variants occupy complementary positions. The 2022 model is a generative architecture for multimodal, instance-aware image translation under training-time box supervision. The 2025 model is a discriminative architecture for scene-geometry reasoning that predicts full occlusion and depth graphs from RGB alone. Together they illustrate two different research programs for instance-centric Transformers: one emphasizes translation quality and instance preservation under style control, and the other emphasizes holistic relational prediction, single-pass inference, and scaling beyond pairwise pipelines.