---
title: 'CoFormer: Multiple Transformer Architectures'
url: https://www.emergentmind.com/topics/coformer
type: topic
---

# CoFormer: Multiple Transformer Architectures

CoFormer is a name used for multiple transformer-based architectures in distinct research areas rather than a single canonical model family. In the literature provided here, the name denotes at least three unrelated systems: **Collaborative Glance-Gaze TransFormer** for grounded situation recognition [2203.16518], **Compatible Transformer** for irregularly sampled multivariate time series [2310.11022], and a collaborative inference system for scalable transformer inference on heterogeneous edge devices [2508.20375]. Closely related names, including **CoCoFormer**, **CountFormer**, and a **decoder-only Conformer** for ASR, are separate methods and should not be conflated with CoFormer [2310.09843] [2407.02047] [2602.12546].

## 1. Naming, scope, and disambiguation

The term **CoFormer** is therefore polysemous in arXiv-era usage. One line of work uses it as an acronym for **Collaborative Glance-Gaze TransFormer**, a model specialized for predicting verbs, role-filling nouns, and bounding-box groundings in images [2203.16518]. A second line uses **Compatible Transformer** for direct modeling of irregular multivariate observations without imputation [2310.11022]. A third uses **CoFormer** for collaborative deployment of large transformers across heterogeneous edge devices through decomposition, aggregation, and calibration [2508.20375].

| Usage | Expansion | Domain |
|---|---|---|
| CoFormer | Collaborative Glance-Gaze TransFormer | Grounded situation recognition |
| CoFormer | Compatible Transformer | Irregularly sampled multivariate time series |
| CoFormer | Collaborating with heterogeneous edge devices for scalable transformer inference | Edge inference systems |

This multiplicity matters because later papers sometimes mention “CoFormer” without domain qualification. In vision papers on situation recognition, “CoFormer” typically refers to the collaborative Glance-Gaze model of Cho, Yoon, and Kwak. In time-series papers, it refers to the compatible attention encoder for asynchronous measurements. In systems work, it refers to transformer decomposition and collaborative execution on edge hardware. A plausible implication is that the name should always be resolved by task context rather than by acronym alone.

## 2. CoFormer as Collaborative Glance-Gaze TransFormer

In grounded situation recognition, CoFormer is the model introduced in **“Collaborative Transformers for Grounded Situation Recognition”** [2203.16518]. The task is to predict the main activity verb, the semantic roles associated with that verb, the noun filling each role, and a bounding box grounding for each entity when visually localizable. The paper formulates the grounded situation as $S=(v,\mathcal{F}_v)$, where the fulfilled roles are
$$
\mathcal F_v = \{(r_i, n_i, \mathbf b_i)\mid r_i \in \mathcal R_v,\; n_i \in \mathcal N \cup \{\emptyset_n\},\; \mathbf b_i \in \mathbb R^4 \cup \{\emptyset_b\}\}.
$$

The central architectural claim is that **activity classification** and **entity estimation** should be interactive and complementary rather than a one-way pipeline. CoFormer implements this with two modules. The **Glance transformer** predicts the main activity, while the **Gaze transformer** estimates grounded entities. Gaze assists Glance by analyzing entities and their relations; Glance assists Gaze by restricting entity estimation to the roles relevant to the predicted activity [2203.16518].

The model uses a **ResNet-50** backbone pretrained on ImageNet, producing $22\times 22\times 2048$ features that are projected to transformer dimension $d=512$. Glance is a single encoder operating on flattened image tokens together with a learnable **Image-Looking (IL) token**. Gaze-Step1 uses learnable role tokens for **all predefined roles**, attends to image features, and then aggregates role evidence with a **Role-Looking (RL) token**. The IL and RL token features are concatenated for verb classification:
$$
\mathbf p_v = \mathrm{Softmax}\!\left( \mathrm{FFN}_{\mathrm{Verb}}([\mathbf z_{\text{IL}};\mathbf z_{\text{RL}}]) \right).
$$
Gaze-Step2 then instantiates verb-conditioned frame-role queries
$$
\mathbf q_r = \mathbf w_r + \mathbf w_{\hat v}, \qquad r \in \mathcal R_{\hat v},
$$
and predicts nouns, box existence, and bounding boxes for the roles in the predicted frame.

The training objective combines verb classification, three noun-classification losses, a box-existence loss, an $L1$ box regression loss, and a GIoU loss. Training uses teacher forcing in Gaze-Step2: frame-role queries are built from the **ground-truth verb** during training for stability, while inference uses the predicted verb [2203.16518].

The model is evaluated on **SWiG**, which contains **75K train**, **25K dev**, and **25K test** images, with **504 verbs**, **11K nouns**, and **190 roles**. On the SWiG test set under **Top-1 Predicted Verb**, CoFormer reports **44.66** verb, **35.98** value, **22.22** value-all, **29.05** grounded-value, and **12.21** grounded-value-all. Under **Top-5 Predicted Verbs**, it reports **73.31**, **57.76**, **33.98**, **46.25**, and **18.37**. Under **Ground-Truth Verb**, it reports **75.95** value, **41.87** value-all, **60.11** grounded-value, and **22.12** grounded-value-all [2203.16518].

The ablations in that paper make the collaboration mechanism concrete. Removing Gaze-Step1 lowers verb and grounded prediction; removing Gaze-Step2 causes a larger grounded prediction drop; removing auxiliary noun classifiers on Gaze-Step1 degrades both verb and grounded metrics; and blocking gradient flow from Gaze-Step2 back to Glance also hurts performance. This suggests that CoFormer’s contribution is not merely transformer capacity, but a structured coupling between noun-aware verb prediction and verb-conditioned grounded role decoding [2203.16518].

## 3. CoFormer as Compatible Transformer

A distinct usage appears in **“Compatible Transformer for Irregularly Sampled Multivariate Time Series”** [2310.11022]. Here CoFormer is an encoder for multivariate time series in which different variates are observed at different timestamps, with nonuniform intervals and unequal numbers of observations. The paper characterizes the core difficulty as **misalignment along both temporal and variate dimensions** and argues against preprocessing by interpolation or imputation.

The key modeling shift is sample-centric. Each observed sample is treated as a unique **variate-time point**
$$
p_\ell^{(i)} = (i, t_\ell^{(i)}),
$$
with an embedding built from three components: a measurement embedding $e=f_{\rm ME}(m_\ell^{(i)})$, a learned variate encoding $v^{(i)}$, and a continuous sinusoidal time encoding $u_\ell^{(i)}$. The initial feature is
$$
h = v \oplus \mathrm{Linear}(e \oplus u).
$$

CoFormer then alternates two attention operators. **Intra-variate attention** attends over all samples from the same variate,
$$
t = \mathrm{MHA}_{\rm Intra}(h, H, H),
$$
to capture long-range temporal structure within that variate. **Inter-variate attention** attends over temporally neighboring samples from other variates,
$$
s = \mathrm{MHA}_{\rm Inter}(t, T, T),
$$
to capture cross-variate interactions. Inter-variate neighborhoods can be defined by a temporal radius
$$
N_{\text{inter}}^{(i,\ell)} = \left\{ p_m^{(j)} \;\middle|\; |t_m^{(j)} - t_\ell^{(i)}| \le \Delta \tau \right\},
$$
or by temporal $K$-nearest neighbors; the experiments use the **$K$-nearest-neighbor implementation** by default [2310.11022].

This architecture is called “compatible” because its attention mechanisms and neighborhood definitions are well-defined for both irregular and regular sampling. The paper explicitly states that when $\Delta\tau=0$, inter-variate neighborhoods reduce to synchronized interactions in the regular case [2310.11022].

For classification, CoFormer outputs sample-wise features, aggregates them temporally within each variate, then aggregates across variates to produce an observation-wise feature fed to an MLP classifier. The reported implementation uses **Adam**, learning rate **0.0001**, batch size **128**, **20** epochs, **8** attention heads, **4** successive layers, and **30** neighbors [2310.11022].

The reported experiments cover three datasets. On **P19**, CoFormer achieves **89.2 AUROC / 57.3 AUPRC**. On **P12**, it achieves **76.5 AUROC / 97.5 AUPRC**. On the irregularized **PAMAP2** task, it achieves **91.2 accuracy / 92.4 precision / 93.7 recall / 92.8 F1**. The paper highlights improvement over Raindrop with average gains of **4.0% AUROC** and **5.5% AUPRC** on P19 and P12 [2310.11022].

The ablations indicate that both intra-variate and inter-variate attention matter, as do time encoding, variate encoding, and the final attention-based aggregator. On P19, the inter-variate-only ablation outperforms the intra-variate-only ablation, which the paper interprets as evidence that cross-variate interaction is particularly important on that dataset [2310.11022].

## 4. CoFormer as a collaborative edge inference system

A third usage appears in **“CoFormer: Collaborating with Heterogeneous Edge Devices for Scalable Transformer Inference”** [2508.20375]. This CoFormer is not a new backbone for prediction tasks; it is a **collaborative inference system for general transformer models**. The target problem is deployment of large transformers on resource-constrained edge devices under latency, memory, and energy constraints.

The paper’s core insight is that transformers exhibit **divisibility** and **integrability**. A large transformer can be decomposed into multiple smaller models by reducing the number of layers, embedding dimension, attention heads, and MLP hidden dimensions, and those smaller models’ outputs can be aggregated to recover much of the original model’s performance [2508.20375].

A submodel architecture is written as
$$
C_n = \{l_n, d_n, h_n^{1:l_n}, D_n^{1:l_n}\},
$$
and the decomposition policy is $\mathcal{C}=\{C_1,\dots,C_N\}$. The runtime design is parallel rather than sequential: all submodels execute concurrently on different devices, each emits a compact final-layer feature, and a central node aggregates the received features once using
$$
X_{agg} = \text{Pool}\big(\mathbf{W} \cdot \text{Concat}(\mathbf{X}_1,\mathbf{X}_2,\ldots,\mathbf{X}_N) + \mathbf{b} \big).
$$
This one-shot aggregation is the paper’s alternative to pipeline split inference and communication-heavy tensor-parallel execution [2508.20375].

The formal optimization objective is to minimize validation-loss degradation and latency under compute and memory constraints:
$$
\min_{\mathcal{C}} \ \mathcal{L}_{val}(\mathcal{C})+\delta T(\mathcal{C}),
$$
subject to constraints on layer count, total embedding dimension, total attention heads, total MLP widths, and device-level computation and memory budgets. The latency model is
$$
T=\max_{n\in\mathcal N}(t_n^1+t_n^2)+t_n^3,
$$
with backbone execution time, transmission time, and aggregation time modeled separately [2508.20375].

To solve this, the paper proposes **DeBo**, which combines **Bayesian optimization** for decomposition-policy search with **progressive boosting distillation** for calibration. The decomposition search uses a Gaussian Process with a Matern kernel and **Expected Improvement** as acquisition function. Calibration then sequentially distills each decomposed submodel from the original large transformer using both ground-truth labels and teacher decisions [2508.20375].

The evaluation spans **ViT**, **DeiT**, **Swin**, **BERT-Large**, **GPT2-XL**, and **Flan-T5-Large**, on **ImageNet-1K**, **CIFAR-100**, **MS COCO 2017**, and **GLUE**, using **Jetson Nano**, **Jetson TX2**, and **Jetson Orin Nano** devices. The abstract reports up to **3.1$\times$ inference speedup**, **76.3%** memory reduction for **GPT2-XL**, and approximately **40%** lower energy consumption while maintaining satisfactory inference performance. On ImageNet-1K with a decomposition based on **DeiT-B**, CoFormer reports **2.35$\times$** speedup, **45.3%** lower energy, **35.6%** lower memory, and **1.14%** accuracy drop. The paper also reports that CoFormer enables efficient inference of **GPT2-XL with 1.6 billion parameters** on **4 GB** edge devices [2508.20375].

The same paper emphasizes limitations as well. Performance still depends on bandwidth, synchronization is still determined by the slowest branch, and aggressive decomposition can degrade accuracy. The manuscript also notes diminishing returns as the number of devices increases [2508.20375].

## 5. Later references and comparative positioning

Later work in situation recognition treats the 2022 CoFormer as the principal prior baseline. **“ClipSitu: Effectively Leveraging CLIP for Conditional Predictions in Situation Recognition”** describes CoFormer as the **state-of-the-art CoFormer, a Transformer-based model**, and states that it combines ideas from SituFormer and GSRTR, with transformer encoder and decoder predicting verbs and nouns, respectively, plus another encoder-decoder to refine verb prediction based on noun-decoder outputs [2307.00586].

That paper compares several CLIP-based models against CoFormer on **imSitu** and **SWiG**. On the test set, it reports CoFormer at **44.66** verb top-1 and **73.31** verb top-5, with **35.98 / 22.22** for top-1-predicted-verb value/value-all and **29.05 / 12.21** for top-1-predicted-verb grnd value/grnd value-all. Its strongest model, ClipSitu XTF, reports **58.19** verb top-1, **85.69** verb top-5, **47.23 / 29.73** for top-1 value/value-all, and **40.01 / 15.03** for top-1 grnd value/grnd value-all [2307.00586].

The same paper repeatedly states a **14.1%** improvement on top-1 value and **9.6%** on top-1 value-all, while its tabulated test numbers imply smaller absolute gains over CoFormer. This suggests that CoFormer remained the reference point for state-of-the-art comparison even when CLIP-based alternatives began to surpass it [2307.00586].

## 6. Related names and common sources of confusion

Several similarly named methods are distinct from CoFormer. **CoCoFormer** is explicitly expanded as **Condition Choir Transformer** or **Condition-Choir-Transformer** and is presented as a controllable polyphonic music generation model rather than a CoFormer alias. The paper itself states that **CoCoFormer is not presented as “CoFormer,” nor as an alternate spelling of it** [2310.09843].

**CountFormer** is likewise a different method: a **Multi-View Crowd Counting Transformer** for 3D scene-level density estimation from synchronized multi-camera images. Its inclusion in CoFormer-like searches is a naming confusion rather than an identity claim [2407.02047].

Finally, **“Decoder-only Conformer with Modality-aware Sparse Mixtures of Experts for ASR”** is relevant only in a looser “CoFormer-style” sense. The paper proposes a **decoder-only Conformer** that unifies speech and text in a single stack, but the provided text states explicitly that the paper **does not use the name CoFormer**; the association is only that a decoder-only Conformer can be read as a **Conformer-as-unified-decoder** architecture [2602.12546].

Taken together, these cases show that “CoFormer” is not a stable universal acronym. In current arXiv usage, the name most directly denotes one of three domain-specific systems: a collaborative Glance-Gaze model for grounded situation recognition, a compatible attention encoder for irregular multivariate time series, or a collaborative edge-inference system for transformer decomposition and aggregation [2203.16518] [2310.11022] [2508.20375].

Source: https://www.emergentmind.com/topics/coformer