MGHFT: Cross-Modal Sticker Emotion Recognition

• The paper demonstrates that MGHFT improves sticker emotion recognition by combining multi-view textual interpretation with a Pyramid Vision Transformer backbone.
• It leverages hierarchical multi-granularity fusion and contrastive alignment to integrate local visual features with global semantic context.
• Empirical results show enhanced accuracy and F1 scores on SER30K and MET-MEME datasets compared to prior models.

Searching arXiv for the cited paper and related multi-granularity hierarchical fusion work. arXiv search query: id:([2507.18929](/papers/2507.18929)) OR ti:MGHFT OR all:"Multi-Granularity Hierarchical Fusion Transformer" Multi-Granularity Hierarchical Fusion Transformer (MGHFT) is a vision–language architecture designed specifically for cross-modal sticker emotion recognition (SER), that is, predicting the emotion a user intends to express with a sticker image. In "MGHFT: Multi-Granularity Hierarchical Fusion Transformer for Cross-Modal Sticker Emotion Recognition" (Chen et al., 25 Jul 2025), the model addresses the difficulty of sticker emotion understanding by coupling a Pyramid Vision Transformer with textual interpretations generated by a Multimodal LLM, and by organizing cross-modal interaction through multi-view description, multi-stage hierarchical fusion, contrastive alignment, and a Text-Guided Fusion Attention module.

1. Problem formulation and motivation

Sticker Emotion Recognition is formulated as a multi-class image classification problem with input sticker image $x \in \mathbb{R}^{H \times W \times 3}$ and output emotion label $y$ from a fixed set of categories. In SER30K and MET-MEME, the categories are Anger, Disgust, Fear, Happiness, Neutral, Sadness, and Surprise (Chen et al., 25 Jul 2025). The task is described as cross-modal because effective prediction requires integrating visual content with textual or contextual knowledge that is not explicitly present in the pixels.

The motivation for MGHFT derives from the observation that stickers are harder than standard facial expression recognition or generic affective image classification. The paper identifies several sources of difficulty: implicit, multi-view emotional cues; stylistic diversity and abstraction; and strong context dependence. Emotional interpretation may depend on communicative intent such as sarcasm, mockery, affection, or apology, on visual style and design, on the depicted roles and posture, and on background or cultural knowledge associated with meme templates or internet culture (Chen et al., 25 Jul 2025). Purely visual models, including VGG, ResNet, ViT, affective CNNs, and PVT-based SER models, are described as robust visual feature extractors that nonetheless do not access background knowledge, metaphorical meaning, or stylistic intent. Vision–language pre-trained models such as CLIP, BLIP, LLaVA, and GPT-4o can describe sticker content, but their generic image–text alignment objectives are not tuned for nuanced sticker emotions, and direct classification with multimodal LLMs gives much lower accuracy and F1 than SER-specific models (Chen et al., 25 Jul 2025).

MGHFT is therefore framed as an SER-specific response to a particular failure mode of generic models: the inability to integrate implicit, stylistic, and culturally mediated semantics with localized visual evidence. A plausible implication is that the architecture treats sticker understanding not as captioning or generic image encoding, but as structured multimodal affective reasoning in which high-level intent and low-level expression cues must be coordinated throughout the backbone.

2. System architecture

The MGHFT pipeline consists of three main parts: a multi-view sticker interpreter, a Pyramid Vision Transformer backbone with hierarchical multi-granularity fusion, and a Text-Guided Fusion Attention classifier head (Chen et al., 25 Jul 2025). The architecture uses LLaVA-NeXT-Mistral-7B to produce textual descriptions, a frozen BERT encoder to obtain text embeddings, and PVT-Small as the visual backbone.

Component	Main elements	Role
Multi-view sticker interpreter	LLaVA-NeXT-Mistral-7B, frozen BERT	Produces four textual views and their embeddings
Hierarchical fusion backbone	PVT-Small, Soft-Fusion, contrastive learning, MLCE	Injects text into local and global visual features across four stages
Final multimodal head	TGFA, MLP, FC, softmax	Produces the final 7-class emotion prediction

At a high level, the model first queries the Multimodal LLM for four views: intention $D_I$, overall style $D_S$, main roles $D_{MR}$, and fine-grained character details $D_{PE}$. The LLM outputs four descriptions $\{C_I, C_S, C_{MR}, C_{PE}\}$, and frozen BERT converts them into text embeddings $T = \{T_1, T_2, T_3, T_4\}$ (Chen et al., 25 Jul 2025). The visual stream uses PVT-Small with four stages, and at each stage produces a global feature $V_g^i$ and local features $V_l^i$. One textual view is injected into the corresponding stage through local Soft-Fusion, while global visual features are aligned with text using contrastive learning and Multi-Level Cross-Entropy Loss. After stagewise processing, multi-stage visual and textual features are fused again and passed to TGFA, which applies two cross-attention steps before classification (Chen et al., 25 Jul 2025).

This organization makes the term “multi-granularity” concrete. Local granularity refers to token-level or region-level visual features $y$0, while global granularity refers to stage-level semantic summaries $y$1. The term “hierarchical” refers to the assignment of distinct textual views to successive backbone stages and to the repeated interaction of text and vision at both local and global levels. The result is not a late-fusion classifier but a multi-stage fusion transformer in which textual context modulates feature extraction itself.

3. Multi-view sticker interpreter

The multi-view sticker interpreter is based on the claim that humans interpret stickers from multiple views. MGHFT operationalizes this claim by querying the MLLM in four rounds: intention, overall style, main roles, and fine-grained character details (Chen et al., 25 Jul 2025). These views correspond respectively to communicative intent, visual tone, central characters or entities and their overall emotion, and local evidence such as facial expression details, posture, gestures, small objects, and text within the image.

The mapping from prompts to descriptions is written as

$y$2

and the mapping from descriptions to embeddings is

$y$3

Each $y$4 is a sequence representation with maximum length 512, and each is linearly projected when needed to match the dimensionality of the corresponding visual stage (Chen et al., 25 Jul 2025).

The design choice is not merely descriptive. The paper reports that injecting only $y$5 at all stages is worse than the full four-view setup, and that concatenating all descriptions into a single representation for all stages also degrades performance. The best performance is obtained when the views are assigned in the order $y$6 across stages, corresponding to intention $y$7 style $y$8 main roles $y$9 details (Chen et al., 25 Jul 2025). This suggests that the model treats the four textual views as stage-specific supervisory signals rather than as interchangeable captions. A plausible implication is that the architecture imposes a progression from coarse communicative semantics to fine-grained visual disambiguation.

4. Hierarchical fusion and optimization

The visual backbone is PVT-Small with four stages. At each stage $D_I$0, the feature map is flattened, prepended with a CLS token, and processed by transformer blocks. The global feature $D_I$1 is the final CLS token of that stage, while local features $D_I$2 are selected patch tokens chosen according to attention weights following SER30K’s “LORA” idea (Chen et al., 25 Jul 2025). The backbone therefore produces a feature pyramid with high-resolution, low-channel early stages and low-resolution, high-channel later stages.

The core local fusion operator is Soft-Fusion. For local visual features $D_I$3 and a stage-matched textual view $D_I$4, the text is projected to $D_I$5 and fused as

$D_I$6

Here each local visual token attends over textual tokens, receives a text-aware update, and preserves the original representation through residual addition (Chen et al., 25 Jul 2025). This mechanism is applied at every stage, but with a different semantic view at each stage.

At the global level, MGHFT uses text both as a fusion signal and as an alignment objective. After normalizing global visual and textual features, the model computes a similarity matrix $D_I$7 and optimizes an InfoNCE-style contrastive loss

$D_I$8

where positives are image–text matches within the batch and all other texts serve as negatives (Chen et al., 25 Jul 2025). For robustness, this is combined with Multi-Level Cross-Entropy Loss using self-similarity distillation. The global alignment loss is

$D_I$9

After all stages, global features are stacked into $D_S$0, text features into $D_S$1, and a second Soft-Fusion is applied: $D_S$2

The final multimodal module is Text-Guided Fusion Attention (TGFA). TGFA first lets vision query text to obtain a text-aware representation $D_S$3, then lets $D_S$4 query vision again to obtain fused feature $D_S$5, and finally integrates this through an MLP with residual connection: $D_S$6 The final classifier maps $D_S$7 to seven emotion classes with a fully connected layer and softmax (Chen et al., 25 Jul 2025).

The full objective is

$D_S$8

This coupling of classification loss with stagewise alignment loss is central to MGHFT’s formulation. The architecture does not simply append text to vision at the end; it repeatedly injects, aligns, and re-fuses text at multiple semantic resolutions before final decision formation.

5. Empirical performance and analytical findings

MGHFT is evaluated on two public sticker emotion datasets. SER30K contains 30,739 static English stickers with 7 emotion labels and is split 70% train, 10% validation, and 20% test. MET-MEME English contains 4,000 memes or stickers with 7 emotion categories and is split 60% train, 20% validation, and 20% test (Chen et al., 25 Jul 2025). Implementation is in PyTorch on $D_S$9 NVIDIA A800 80GB, with PVT-Small as a trainable pre-trained backbone, frozen pre-trained BERT as text encoder, batch size 16, 50 epochs, AdamW, learning rate $D_{MR}$0, and maximum text length 512 (Chen et al., 25 Jul 2025).

On SER30K, the strongest prior model reported in the main comparison is TGCA-PVT (Image+Text), with accuracy 71.63% and F1 70.93%. MGHFT achieves accuracy 73.31% and F1 72.52%, improving over TGCA-PVT by +1.68 percentage points in accuracy and +1.59 percentage points in F1 (Chen et al., 25 Jul 2025). Compared to the best pre-trained visual models, the paper states that MGHFT improves over BLIP2 by 5.4% on F1 and 4.0% on accuracy. It also reports strong per-category precision on SER30K, including Happiness 79.79, Neutral 70.82, and Sadness 71.80 (Chen et al., 25 Jul 2025).

On MET-MEME, the strongest prior baseline is MGMCF with accuracy 34.36%, precision 37.77%, and recall 34.38%. MGHFT achieves accuracy 35.13%, precision 34.75%, and recall 35.12%, improving accuracy by +0.77 percentage points and yielding slightly higher recall with more balanced precision and recall (Chen et al., 25 Jul 2025). The paper interprets this as robustness on a smaller, more metaphor-heavy dataset.

Ablation studies isolate four components: CL, GF, LF, and TGFA. Removing any single component reduces performance, and all four together produce the best accuracy and F1. Adding only CL improves F1 from 69.89% to 70.98%, indicating that global alignment is especially impactful. Replacing Soft-Fusion with standard cross-attention yields worse performance and higher cost, and the paper characterizes Soft-Fusion as both effective and lightweight (Chen et al., 25 Jul 2025). Additional ablations show that the best stage assignment is again $D_{MR}$1, while detail-only or concatenated descriptions underperform.

The qualitative analyses are consistent with the quantitative picture. In t-SNE visualizations, MGHFT embeddings form more compact intra-class clusters and clearer inter-class boundaries than CLIP, BLIP, and LLaVA. In attention visualizations, MGHFT concentrates on emotionally salient regions such as faces, eyes, gestures, and key objects, whereas a model without multi-view descriptions attends more to background or less relevant areas (Chen et al., 25 Jul 2025). The paper also notes remaining limitations: cultural ambiguity and metaphor remain difficult, especially on MET-MEME, and MLLM outputs can be noisy for rare cultural references or highly context-dependent memes.

6. Position within multi-granularity hierarchical fusion research

MGHFT for sticker emotion recognition belongs to a broader methodological family in which multiple feature granularities are modeled jointly and fused hierarchically, often with attention-based modules. Related arXiv work shows the same structural pattern in different domains, though under different model names and with different task-specific semantics.

In relation extraction, the SMS framework models entity mention-level, segment-level, and sentence-level features and ties them together hierarchically without external knowledge (Liang et al., 2022). Its mention-aware segment attention and entity-aware global semantic attention provide an explicit example of top-down conditioning across granularities. In medical time-series classification, Medformer uses multiple patch lengths, per-granularity router tokens, and two-stage intra- and inter-granularity self-attention, thereby instantiating a multi-granularity patching transformer with hierarchical fusion across scales (Wang et al., 2024). PrismWF extends the same pattern to website fingerprinting through parallel CNN branches, router tokens, inter-granularity detail supplementation, and router-guided intra- and cross-granularity fusion (Pan et al., 22 Mar 2026). In zero-shot Chinese character recognition, Hi-GITA uses image and text multi-granularity encoders, fusion modules on both sides, and fine-grained decoupled image–text contrastive loss to align stroke-, radical-, and structure-level representations (Zhu et al., 30 May 2025). In hierarchical image classification, query-based fusion transformers combine multi-scale feature fusion with coarse-to-fine query fusion and a CAMP block to reduce error propagation across label granularities (Sahoo et al., 2023).

These works do not define a single canonical MGHFT architecture across domains, but they converge on a recurring design logic: represent data at multiple resolutions or semantic levels, let higher-level summaries guide lower-level selection or refinement, and use structured fusion rather than flat concatenation. In the specific case of sticker emotion recognition, MGHFT (Chen et al., 25 Jul 2025) adapts this logic to a cross-modal setting by assigning distinct textual views to different PVT stages and by aligning global and local visual semantics with MLLM-generated context. This suggests that the distinguishing feature of MGHFT is not merely the presence of multiple scales, but the coupling of multi-view semantic decomposition with stagewise multimodal supervision.

Within affective computing and vision–language modeling, MGHFT therefore occupies a specific position: it is an SER-specific multimodal transformer in which Multimodal LLMs are used as semantic interpreters rather than as direct classifiers, and in which hierarchical fusion is used to integrate intent, style, roles, and details into the visual backbone itself (Chen et al., 25 Jul 2025).