CoFormer: Multiple Transformer Architectures

• CoFormer is a polysemous term for distinct transformer-based models applied in grounded situation recognition, irregular multivariate time series analysis, and collaborative edge inference.
• Each variant employs specialized attention mechanisms—whether through Glance-Gaze collaboration or intra/inter-variate interactions—to enhance prediction accuracy and inference efficiency.
• Evaluations across diverse datasets demonstrate competitive performance, though naming confusions persist with similarly named models like CoCoFormer or CountFormer.

CoFormer is a name used for multiple transformer-based architectures in distinct research areas rather than a single canonical model family. In the literature provided here, the name denotes at least three unrelated systems: Collaborative Glance-Gaze TransFormer for grounded situation recognition (Cho et al., 2022), Compatible Transformer for irregularly sampled multivariate time series (Wei et al., 2023), and a collaborative inference system for scalable transformer inference on heterogeneous edge devices (Xu et al., 28 Aug 2025). Closely related names, including CoCoFormer, CountFormer, and a decoder-only Conformer for ASR, are separate methods and should not be conflated with CoFormer (Zhou et al., 2023, Mo et al., 2024, Lee et al., 13 Feb 2026).

1. Naming, scope, and disambiguation

The term CoFormer is therefore polysemous in arXiv-era usage. One line of work uses it as an acronym for Collaborative Glance-Gaze TransFormer, a model specialized for predicting verbs, role-filling nouns, and bounding-box groundings in images (Cho et al., 2022). A second line uses Compatible Transformer for direct modeling of irregular multivariate observations without imputation (Wei et al., 2023). A third uses CoFormer for collaborative deployment of large transformers across heterogeneous edge devices through decomposition, aggregation, and calibration (Xu et al., 28 Aug 2025).

Usage	Expansion	Domain
CoFormer	Collaborative Glance-Gaze TransFormer	Grounded situation recognition
CoFormer	Compatible Transformer	Irregularly sampled multivariate time series
CoFormer	Collaborating with heterogeneous edge devices for scalable transformer inference	Edge inference systems

This multiplicity matters because later papers sometimes mention “CoFormer” without domain qualification. In vision papers on situation recognition, “CoFormer” typically refers to the collaborative Glance-Gaze model of Cho, Yoon, and Kwak. In time-series papers, it refers to the compatible attention encoder for asynchronous measurements. In systems work, it refers to transformer decomposition and collaborative execution on edge hardware. A plausible implication is that the name should always be resolved by task context rather than by acronym alone.

2. CoFormer as Collaborative Glance-Gaze TransFormer

In grounded situation recognition, CoFormer is the model introduced in “Collaborative Transformers for Grounded Situation Recognition” (Cho et al., 2022). The task is to predict the main activity verb, the semantic roles associated with that verb, the noun filling each role, and a bounding box grounding for each entity when visually localizable. The paper formulates the grounded situation as $S=(v,\mathcal{F}_v)$, where the fulfilled roles are

$$\mathcal F_v = \{(r_i, n_i, \mathbf b_i)\mid r_i \in \mathcal R_v,\; n_i \in \mathcal N \cup \{\emptyset_n\},\; \mathbf b_i \in \mathbb R^4 \cup \{\emptyset_b\}\}.$$

The central architectural claim is that activity classification and entity estimation should be interactive and complementary rather than a one-way pipeline. CoFormer implements this with two modules. The Glance transformer predicts the main activity, while the Gaze transformer estimates grounded entities. Gaze assists Glance by analyzing entities and their relations; Glance assists Gaze by restricting entity estimation to the roles relevant to the predicted activity (Cho et al., 2022).

The model uses a ResNet-50 backbone pretrained on ImageNet, producing $22\times 22\times 2048$ features that are projected to transformer dimension $d=512$. Glance is a single encoder operating on flattened image tokens together with a learnable Image-Looking (IL) token. Gaze-Step1 uses learnable role tokens for all predefined roles, attends to image features, and then aggregates role evidence with a Role-Looking (RL) token. The IL and RL token features are concatenated for verb classification:

$$\mathbf p_v = \mathrm{Softmax}\!\left( \mathrm{FFN}_{\mathrm{Verb}}([\mathbf z_{\text{IL}};\mathbf z_{\text{RL}}]) \right).$$

Gaze-Step2 then instantiates verb-conditioned frame-role queries

$$\mathbf q_r = \mathbf w_r + \mathbf w_{\hat v}, \qquad r \in \mathcal R_{\hat v},$$

and predicts nouns, box existence, and bounding boxes for the roles in the predicted frame.

The training objective combines verb classification, three noun-classification losses, a box-existence loss, an $L1$ box regression loss, and a GIoU loss. Training uses teacher forcing in Gaze-Step2: frame-role queries are built from the ground-truth verb during training for stability, while inference uses the predicted verb (Cho et al., 2022).

The model is evaluated on SWiG, which contains 75K train, 25K dev, and 25K test images, with 504 verbs, 11K nouns, and 190 roles. On the SWiG test set under Top-1 Predicted Verb, CoFormer reports 44.66 verb, 35.98 value, 22.22 value-all, 29.05 grounded-value, and 12.21 grounded-value-all. Under Top-5 Predicted Verbs, it reports 73.31, 57.76, 33.98, 46.25, and 18.37. Under Ground-Truth Verb, it reports 75.95 value, 41.87 value-all, 60.11 grounded-value, and 22.12 grounded-value-all (Cho et al., 2022).

The ablations in that paper make the collaboration mechanism concrete. Removing Gaze-Step1 lowers verb and grounded prediction; removing Gaze-Step2 causes a larger grounded prediction drop; removing auxiliary noun classifiers on Gaze-Step1 degrades both verb and grounded metrics; and blocking gradient flow from Gaze-Step2 back to Glance also hurts performance. This suggests that CoFormer’s contribution is not merely transformer capacity, but a structured coupling between noun-aware verb prediction and verb-conditioned grounded role decoding (Cho et al., 2022).

3. CoFormer as Compatible Transformer

A distinct usage appears in “Compatible Transformer for Irregularly Sampled Multivariate Time Series” (Wei et al., 2023). Here CoFormer is an encoder for multivariate time series in which different variates are observed at different timestamps, with nonuniform intervals and unequal numbers of observations. The paper characterizes the core difficulty as misalignment along both temporal and variate dimensions and argues against preprocessing by interpolation or imputation.

The key modeling shift is sample-centric. Each observed sample is treated as a unique variate-time point

$p_\ell^{(i)} = (i, t_\ell^{(i)}),$

with an embedding built from three components: a measurement embedding $e=f_{\rm ME}(m_\ell^{(i)})$, a learned variate encoding $v^{(i)}$, and a continuous sinusoidal time encoding $$\mathcal F_v = \{(r_i, n_i, \mathbf b_i)\mid r_i \in \mathcal R_v,\; n_i \in \mathcal N \cup \{\emptyset_n\},\; \mathbf b_i \in \mathbb R^4 \cup \{\emptyset_b\}\}.$$0. The initial feature is

$$\mathcal F_v = \{(r_i, n_i, \mathbf b_i)\mid r_i \in \mathcal R_v,\; n_i \in \mathcal N \cup \{\emptyset_n\},\; \mathbf b_i \in \mathbb R^4 \cup \{\emptyset_b\}\}.$$1

CoFormer then alternates two attention operators. Intra-variate attention attends over all samples from the same variate,

$$\mathcal F_v = \{(r_i, n_i, \mathbf b_i)\mid r_i \in \mathcal R_v,\; n_i \in \mathcal N \cup \{\emptyset_n\},\; \mathbf b_i \in \mathbb R^4 \cup \{\emptyset_b\}\}.$$2

to capture long-range temporal structure within that variate. Inter-variate attention attends over temporally neighboring samples from other variates,

$$\mathcal F_v = \{(r_i, n_i, \mathbf b_i)\mid r_i \in \mathcal R_v,\; n_i \in \mathcal N \cup \{\emptyset_n\},\; \mathbf b_i \in \mathbb R^4 \cup \{\emptyset_b\}\}.$$3

to capture cross-variate interactions. Inter-variate neighborhoods can be defined by a temporal radius

$$\mathcal F_v = \{(r_i, n_i, \mathbf b_i)\mid r_i \in \mathcal R_v,\; n_i \in \mathcal N \cup \{\emptyset_n\},\; \mathbf b_i \in \mathbb R^4 \cup \{\emptyset_b\}\}.$$4

or by temporal $$\mathcal F_v = \{(r_i, n_i, \mathbf b_i)\mid r_i \in \mathcal R_v,\; n_i \in \mathcal N \cup \{\emptyset_n\},\; \mathbf b_i \in \mathbb R^4 \cup \{\emptyset_b\}\}.$$5-nearest neighbors; the experiments use the $$\mathcal F_v = \{(r_i, n_i, \mathbf b_i)\mid r_i \in \mathcal R_v,\; n_i \in \mathcal N \cup \{\emptyset_n\},\; \mathbf b_i \in \mathbb R^4 \cup \{\emptyset_b\}\}.$$6-nearest-neighbor implementation by default (Wei et al., 2023).

This architecture is called “compatible” because its attention mechanisms and neighborhood definitions are well-defined for both irregular and regular sampling. The paper explicitly states that when $$\mathcal F_v = \{(r_i, n_i, \mathbf b_i)\mid r_i \in \mathcal R_v,\; n_i \in \mathcal N \cup \{\emptyset_n\},\; \mathbf b_i \in \mathbb R^4 \cup \{\emptyset_b\}\}.$$7, inter-variate neighborhoods reduce to synchronized interactions in the regular case (Wei et al., 2023).

For classification, CoFormer outputs sample-wise features, aggregates them temporally within each variate, then aggregates across variates to produce an observation-wise feature fed to an MLP classifier. The reported implementation uses Adam, learning rate 0.0001, batch size 128, 20 epochs, 8 attention heads, 4 successive layers, and 30 neighbors (Wei et al., 2023).

The reported experiments cover three datasets. On P19, CoFormer achieves 89.2 AUROC / 57.3 AUPRC. On P12, it achieves 76.5 AUROC / 97.5 AUPRC. On the irregularized PAMAP2 task, it achieves 91.2 accuracy / 92.4 precision / 93.7 recall / 92.8 F1. The paper highlights improvement over Raindrop with average gains of 4.0% AUROC and 5.5% AUPRC on P19 and P12 (Wei et al., 2023).

The ablations indicate that both intra-variate and inter-variate attention matter, as do time encoding, variate encoding, and the final attention-based aggregator. On P19, the inter-variate-only ablation outperforms the intra-variate-only ablation, which the paper interprets as evidence that cross-variate interaction is particularly important on that dataset (Wei et al., 2023).

4. CoFormer as a collaborative edge inference system

A third usage appears in “CoFormer: Collaborating with Heterogeneous Edge Devices for Scalable Transformer Inference” (Xu et al., 28 Aug 2025). This CoFormer is not a new backbone for prediction tasks; it is a collaborative inference system for general transformer models. The target problem is deployment of large transformers on resource-constrained edge devices under latency, memory, and energy constraints.

The paper’s core insight is that transformers exhibit divisibility and integrability. A large transformer can be decomposed into multiple smaller models by reducing the number of layers, embedding dimension, attention heads, and MLP hidden dimensions, and those smaller models’ outputs can be aggregated to recover much of the original model’s performance (Xu et al., 28 Aug 2025).

A submodel architecture is written as

$$\mathcal F_v = \{(r_i, n_i, \mathbf b_i)\mid r_i \in \mathcal R_v,\; n_i \in \mathcal N \cup \{\emptyset_n\},\; \mathbf b_i \in \mathbb R^4 \cup \{\emptyset_b\}\}.$$8

and the decomposition policy is $$\mathcal F_v = \{(r_i, n_i, \mathbf b_i)\mid r_i \in \mathcal R_v,\; n_i \in \mathcal N \cup \{\emptyset_n\},\; \mathbf b_i \in \mathbb R^4 \cup \{\emptyset_b\}\}.$$9. The runtime design is parallel rather than sequential: all submodels execute concurrently on different devices, each emits a compact final-layer feature, and a central node aggregates the received features once using

$22\times 22\times 2048$0

This one-shot aggregation is the paper’s alternative to pipeline split inference and communication-heavy tensor-parallel execution (Xu et al., 28 Aug 2025).

The formal optimization objective is to minimize validation-loss degradation and latency under compute and memory constraints:

$22\times 22\times 2048$1

subject to constraints on layer count, total embedding dimension, total attention heads, total MLP widths, and device-level computation and memory budgets. The latency model is

$22\times 22\times 2048$2

with backbone execution time, transmission time, and aggregation time modeled separately (Xu et al., 28 Aug 2025).

To solve this, the paper proposes DeBo, which combines Bayesian optimization for decomposition-policy search with progressive boosting distillation for calibration. The decomposition search uses a Gaussian Process with a Matern kernel and Expected Improvement as acquisition function. Calibration then sequentially distills each decomposed submodel from the original large transformer using both ground-truth labels and teacher decisions (Xu et al., 28 Aug 2025).

The evaluation spans ViT, DeiT, Swin, BERT-Large, GPT2-XL, and Flan-T5-Large, on ImageNet-1K, CIFAR-100, MS COCO 2017, and GLUE, using Jetson Nano, Jetson TX2, and Jetson Orin Nano devices. The abstract reports up to 3.1$22\times 22\times 2048$3 inference speedup, 76.3% memory reduction for GPT2-XL, and approximately 40% lower energy consumption while maintaining satisfactory inference performance. On ImageNet-1K with a decomposition based on DeiT-B, CoFormer reports 2.35$22\times 22\times 2048$4 speedup, 45.3% lower energy, 35.6% lower memory, and 1.14% accuracy drop. The paper also reports that CoFormer enables efficient inference of GPT2-XL with 1.6 billion parameters on 4 GB edge devices (Xu et al., 28 Aug 2025).

The same paper emphasizes limitations as well. Performance still depends on bandwidth, synchronization is still determined by the slowest branch, and aggressive decomposition can degrade accuracy. The manuscript also notes diminishing returns as the number of devices increases (Xu et al., 28 Aug 2025).

5. Later references and comparative positioning

Later work in situation recognition treats the 2022 CoFormer as the principal prior baseline. “ClipSitu: Effectively Leveraging CLIP for Conditional Predictions in Situation Recognition” describes CoFormer as the state-of-the-art CoFormer, a Transformer-based model, and states that it combines ideas from SituFormer and GSRTR, with transformer encoder and decoder predicting verbs and nouns, respectively, plus another encoder-decoder to refine verb prediction based on noun-decoder outputs (Roy et al., 2023).

That paper compares several CLIP-based models against CoFormer on imSitu and SWiG. On the test set, it reports CoFormer at 44.66 verb top-1 and 73.31 verb top-5, with 35.98 / 22.22 for top-1-predicted-verb value/value-all and 29.05 / 12.21 for top-1-predicted-verb grnd value/grnd value-all. Its strongest model, ClipSitu XTF, reports 58.19 verb top-1, 85.69 verb top-5, 47.23 / 29.73 for top-1 value/value-all, and 40.01 / 15.03 for top-1 grnd value/grnd value-all (Roy et al., 2023).

The same paper repeatedly states a 14.1% improvement on top-1 value and 9.6% on top-1 value-all, while its tabulated test numbers imply smaller absolute gains over CoFormer. This suggests that CoFormer remained the reference point for state-of-the-art comparison even when CLIP-based alternatives began to surpass it (Roy et al., 2023).

6. Related names and common sources of confusion

Several similarly named methods are distinct from CoFormer. CoCoFormer is explicitly expanded as Condition Choir Transformer or Condition-Choir-Transformer and is presented as a controllable polyphonic music generation model rather than a CoFormer alias. The paper itself states that CoCoFormer is not presented as “CoFormer,” nor as an alternate spelling of it (Zhou et al., 2023).

CountFormer is likewise a different method: a Multi-View Crowd Counting Transformer for 3D scene-level density estimation from synchronized multi-camera images. Its inclusion in CoFormer-like searches is a naming confusion rather than an identity claim (Mo et al., 2024).

Finally, “Decoder-only Conformer with Modality-aware Sparse Mixtures of Experts for ASR” is relevant only in a looser “CoFormer-style” sense. The paper proposes a decoder-only Conformer that unifies speech and text in a single stack, but the provided text states explicitly that the paper does not use the name CoFormer; the association is only that a decoder-only Conformer can be read as a Conformer-as-unified-decoder architecture (Lee et al., 13 Feb 2026).

Taken together, these cases show that “CoFormer” is not a stable universal acronym. In current arXiv usage, the name most directly denotes one of three domain-specific systems: a collaborative Glance-Gaze model for grounded situation recognition, a compatible attention encoder for irregular multivariate time series, or a collaborative edge-inference system for transformer decomposition and aggregation (Cho et al., 2022, Wei et al., 2023, Xu et al., 28 Aug 2025).