Multimodal Bag-of-Words Representation

• Multimodal BoW is a representation method that quantizes features from diverse modalities into compact, fixed-dimensional histograms.
• It employs techniques like k-means clustering, vector quantization, and codebook learning to discretize continuous and structured data.
• Practical applications include emotion recognition, sentiment analysis, and image retrieval via effective cross-modal fusion and normalization.

Multimodal bag-of-words (BoW) representation is a family of techniques for transforming multisource data—text, image, audio, and other modalities—into discrete, fixed-dimensional histograms capturing the occurrence patterns of modality-specific quantized atoms. While the original BoW representation was formulated for text documents as histograms over a vocabulary, modern multimodal BoW approaches extend the quantization principle to arbitrary numeric features via vector quantization (VQ), codebook learning, clustering, and combinatorial fusion. This enables compact and flexible feature vectors that support information fusion, machine learning, retrieval, and classification across heterogeneous data types.

1. Discretization and Codebook Construction

The central theoretical underpinning of multimodal BoW is the discretization of continuous or structured data streams via a modality-specific or shared codebook. For vector-valued features (e.g., visual descriptors, audio features), standard practice is to first amass low-level descriptor vectors $X = \{ x_i \}_{i=1}^n \subset \mathbb{R}^d$ from each modality. A discrete codebook $C = \{ c_j \}_{j=1}^K$ is induced via unsupervised k-means clustering, minimizing within-cluster squared error:

$$\min_C \sum_{i=1}^n \min_{1 \leq j \leq K} \| x_i - c_j \|^2$$

Serviceable alternatives include k-means++ initialization and supervised per-class clustering, as implemented in the openXBOW toolkit (Schmitt et al., 2016). For purely symbolic data, the inherent vocabulary defines the codebook.

Advanced frameworks implement cross-modal codebook sharing. “Cross-Modal Discrete Representation Learning” introduces a learnable codebook $E = \{ e_k \}_{k=1}^K$ in $\mathbb{R}^d$ accessible to all modalities (Liu et al., 2021). Each modality $M$ is equipped with a fine-grained encoder $f_{\text{fine}}^M$ producing a sequence of embeddings, projected to the common codebook space and then quantized by nearest-neighbor assignment:

$k^*(l) = \arg\min_{k=1}^K \| z_l^M - e_k \|_2$

Penalties akin to vector-quantized VAEs are applied to regularize the codebook and assignments: $$L_{\text{quant}} = \| \text{sg}[z] - e_{k^*} \|_2^2$$ and $$L_{\text{commit}} = \beta \| z - \text{sg}[e_{k^*}] \|_2^2$$ (Liu et al., 2021).

For text present in images, PHOC descriptors (Mafla et al., 2020) encode position-sensitive character histograms at multiple levels, which are then PCA-reduced, clustered via GMM, and softly assigned to codewords.

2. Feature Quantization and Histogramming

Post-codebook induction, every feature vector is assigned to codebook atom(s) using hard, soft, or multiple-assignment strategies:

• Hard assignment: For each $x_i$, $q(i) = \arg\min_j \| x_i - c_j \|^2$, yielding weight $w_{ij} = 1 \{ q(i) = j \}$.
• Multiple-assignment: Each $x_i$ is assigned to its $a$ nearest codewords: $w_{ij} = 1$ if $c_j \in \mathcal{N}_a(x_i)$.
• Soft Gaussian weighting: $$w_{ij} = \exp \left( -\tfrac{ \| x_i - c_j \|^2 }{ 2 \sigma^2 } \right )$$, with optional normalization (Schmitt et al., 2016).
• Soft assignments for VQ: $$P(e_k | z) = \exp(-\| z - e_k \|_2) / \sum_j \exp(-\| z - e_j \|_2)$$ (Liu et al., 2021).
• Fisher Vector Encoding: Gradients of the log-likelihood of a GMM fitted to local features, aggregating higher-order statistics for each codeword (Mafla et al., 2020).

Aggregated counts or soft weights over codewords yield BoW histograms. For a segment containing $N$ vectors, the histogram is $h_j = \sum_{i=1}^N w_{ij}$ (Schmitt et al., 2016), providing a compact representation of the feature distribution. For textual modalities detected in images, Fisher vectors are computed over collections of PHOC descriptors, using gradient pooling with respect to GMM means and variances (Mafla et al., 2020).

3. Cross-Modal Fusion Strategies

BoW representations for each modality may be concatenated to yield a composite feature vector:

$$H = [h^{(1)}; h^{(2)}; \dots; h^{(M)}] \in \mathbb{R}^{\sum_m K_m}$$

Optionally, modality-specific scaling factors $\alpha_m$ can be applied: $H = [\alpha_1 h^{(1)}; ...; \alpha_M h^{(M)}]$ (Schmitt et al., 2016). To mitigate the dominance of frequent or common patterns, TF-IDF weighting or power and L2 normalization are frequently employed.

Recent approaches move beyond concatenation towards shared latent spaces or attention-based fusion. Cross-modal discrete representation learning aligns empirical codeword distributions $P(e_k | H^M)$ between modalities using a symmetric cross-entropy:

$$S_{\text{code}}(x_i^A, x_i^B) = \sum_k P(e_k | H_i^A) \log P(e_k | H_i^B) + \sum_k P(e_k | H_i^B) \log P(e_k | H_i^A)$$

A contrastive InfoNCE wrapper $L_{\text{CMCM}}$ encourages distinct pairs to be separated (Liu et al., 2021).

Other methods employ deep models to map modality-specific histogram and feature vectors (BoW, color, tags) with type-specific RBMs, fusing them in a higher-level binary RBM that models both intra-type and inter-type correlations (Tran et al., 2016). Attention-based fusion, such as softmax of $\tanh(V_{fa}^\top W T_f)$ for visual and textual Fisher vectors, further supports learning deep multimodal representations (Mafla et al., 2020).

4. Training and Inference Pipelines

Typical multimodal BoW pipelines proceed as follows:

1. Preprocessing
   • For numeric features: framing, normalization, and silence/activity detection are standard steps.
   • For text: tokenization, n-gram extraction, vocabulary pruning, and optional PCA reduction for structured descriptors.
2. Codebook Induction
   • Codebooks are learned via k-means, GMM, or end-to-end differentiable mechanisms such as VQ.
3. Encoding
   • Compute embeddings for input segments.
   • Quantize to codewords using hard/soft assignment rules.
   • Aggregate to produce modality-specific histograms or Fisher vectors.
4. Fusion
   • Concatenate, weight, or attend across histograms or Fisher vectors.
   • Normalize the resulting vectors.
5. Downstream Learning
   • Features are used for classification, retrieval, clustering, or regression.
   • Additional supervision or multitask heads (e.g., for tagging, attribute prediction) may regularize the joint space (Tran et al., 2016, Liu et al., 2021).
6. Inference
   • Feature extraction follows the same pipeline: compute code assignments and form BoW or Fisher vectors for new data.

Example toolkits such as openXBOW provide end-to-end pipelines with modifiable parameters, supporting ARFF, CSV, and LIBSVM formats (Schmitt et al., 2016).

5. Interpretability and Semantic Alignment

A key empirical property of joint discrete codebooks with cross-modal objectives is cluster interpretability. When trained with cross-modal code-matching, codebook atoms evolve into modality-invariant semantic atoms—each code index behaves like a discrete “word” semantically grounded across modalities. For instance, on video–speech datasets, certain codes fire only for particular annotated actions and the same codes correspond to aligned concepts in audio (e.g., juggling in both modalities). Visualization (e.g., t-SNE) of codebook usage reveals unified cluster structure only when cross-modal matching is applied (Liu et al., 2021).

Similarly, in deep factor analysis, topic factors learned from multimodal data localize to meaningful clusters across Poisson (count), Gaussian (continuous), and multinomial (text BoW) modalities (Yilmaz et al., 2015). In deep fusion architectures, pretraining and auxiliary tasks further refine the semantic correspondence between text and non-textual information (Tran et al., 2016).

6. Representative Applications and Empirical Results

Multimodal BoW and its variants have been validated in applied contexts:

• Emotion recognition in continuous speech: High-dimensional BoAW features using openXBOW, with TF-IDF and normalization, yield Concordance Correlation Coefficient improvements over baselines, particularly enhancing valence modeling (Schmitt et al., 2016).
• Twitter sentiment analysis: Multimodal BoW with TF-IDF and feature selection achieves or exceeds the performance of more complex neural models on large-scale datasets (Schmitt et al., 2016).
• Fine-grained image classification/retrieval: Fusion of textual Fisher vectors built over PHOC descriptors with CNN features attains state-of-the-art accuracy and mAP in fine-grained tasks. Notably, combining visual and PHOC-based textual FV yields large improvements over individual modalities and other fusion baselines (Mafla et al., 2020).
• Social image retrieval/concept prediction: Deep fusion models using Poisson RBMs with BoW input and multitask heads provide substantial retrieval and multilabel prediction gains, especially when multiple modalities are fused (Tran et al., 2016).

Summary performance table (examples):

Task	Modality	Best BoW Approach	Performance Metric	Citation
Speech Emotion Recognition	Audio (LLDs)	openXBOW, top-a=20 TF-IDF	CCC: .793 (valence)	(Schmitt et al., 2016)
Twitter Sentiment	Text	BoW + TF-IDF + SVM	77.28% acc.	(Schmitt et al., 2016)
Fine-grained Image Class/Ret.	Vision+Text (PHOC FV)	FV(PHOC)+CNN+attention fusion	80.2% accuracy (Con-Text)	(Mafla et al., 2020)
Social Image Retrieval	Multiview (BoW, color)	Deep RBM fusion	MAP: 0.420	(Tran et al., 2016)

7. Unified Generative Models

Multimodal BoW can be subsumed under generative factor models combining Poisson (counts), Gaussian (continuous), and multinomial (text) likelihoods sharing a latent space (Yilmaz et al., 2015). The object representation $z_t \sim \mathcal{N}(0, I_K)$ is mapped to each modality via individual loading matrices. The multinomial component is a direct text BoW model, with softmax of the latent factors yielding a document-specific word distribution.

EM-based inference interleaves posterior approximation for the latent $\{ z_t \}$ (e.g., via Laplace approximation) and closed-form or quasi-Newton updates for loadings. This establishes a direct connection between topic models, multimodal dimensionality reduction, and classical BoW (Yilmaz et al., 2015).

---

Recent multimodal BoW developments demonstrate that code quantization, cross-modal alignment, and flexible histogram aggregation yield robust, interpretable, and semantically meaningful representations applicable to a wide range of high-level multimodal understanding tasks (Liu et al., 2021, Schmitt et al., 2016, Tran et al., 2016, Mafla et al., 2020, Yilmaz et al., 2015).