---
title: Multimodal Bag-of-Words Representation
url: https://www.emergentmind.com/topics/multimodal-bag-of-words-representation
type: topic
---

# Multimodal Bag-of-Words Representation

Multimodal bag-of-words (BoW) representation is a family of techniques for transforming multisource data—text, image, audio, and other modalities—into discrete, fixed-dimensional histograms capturing the occurrence patterns of modality-specific quantized atoms. While the original BoW representation was formulated for text documents as histograms over a vocabulary, modern multimodal BoW approaches extend the quantization principle to arbitrary numeric features via vector quantization (VQ), codebook learning, clustering, and combinatorial fusion. This enables compact and flexible feature vectors that support information fusion, machine learning, retrieval, and classification across heterogeneous data types.

## 1. Discretization and Codebook Construction

The central theoretical underpinning of multimodal BoW is the discretization of continuous or structured data streams via a modality-specific or shared codebook. For vector-valued features (e.g., visual descriptors, audio features), standard practice is to first amass low-level descriptor vectors $X = \{ x_i \}_{i=1}^n \subset \mathbb{R}^d$ from each modality. A discrete codebook $C = \{ c_j \}_{j=1}^K$ is induced via unsupervised k-means clustering, minimizing within-cluster squared error:

$$
\min_C \sum_{i=1}^n \min_{1 \leq j \leq K} \| x_i - c_j \|^2
$$

Serviceable alternatives include k-means++ initialization and supervised per-class clustering, as implemented in the openXBOW toolkit [1605.06778]. For purely symbolic data, the inherent vocabulary defines the codebook.

Advanced frameworks implement cross-modal codebook sharing. “Cross-Modal Discrete Representation Learning” introduces a learnable codebook $E = \{ e_k \}_{k=1}^K$ in $\mathbb{R}^d$ accessible to all modalities [2106.05438]. Each modality $M$ is equipped with a fine-grained encoder $f_{\text{fine}}^M$ producing a sequence of embeddings, projected to the common codebook space and then quantized by nearest-neighbor assignment:

$$
k^*(l) = \arg\min_{k=1}^K \| z_l^M - e_k \|_2
$$

Penalties akin to vector-quantized VAEs are applied to regularize the codebook and assignments: $L_{\text{quant}} = \| \text{sg}[z] - e_{k^*} \|_2^2$ and $L_{\text{commit}} = \beta \| z - \text{sg}[e_{k^*}] \|_2^2$ [2106.05438].

For text present in images, PHOC descriptors [2001.04732] encode position-sensitive character histograms at multiple levels, which are then PCA-reduced, clustered via GMM, and softly assigned to codewords.

## 2. Feature Quantization and Histogramming

Post-codebook induction, every feature vector is assigned to codebook atom(s) using hard, soft, or multiple-assignment strategies:

- **Hard assignment**: For each $x_i$, $q(i) = \arg\min_j \| x_i - c_j \|^2$, yielding weight $w_{ij} = 1 \{ q(i) = j \}$.
- **Multiple-assignment**: Each $x_i$ is assigned to its $a$ nearest codewords: $w_{ij} = 1$ if $c_j \in \mathcal{N}_a(x_i)$.
- **Soft Gaussian weighting**: $w_{ij} = \exp \left( -\tfrac{ \| x_i - c_j \|^2 }{ 2 \sigma^2 } \right )$, with optional normalization [1605.06778].
- **Soft assignments for VQ**: $P(e_k | z) = \exp(-\| z - e_k \|_2) / \sum_j \exp(-\| z - e_j \|_2)$ [2106.05438].
- **Fisher Vector Encoding**: Gradients of the log-likelihood of a GMM fitted to local features, aggregating higher-order statistics for each codeword [2001.04732].

Aggregated counts or soft weights over codewords yield BoW histograms. For a segment containing $N$ vectors, the histogram is $h_j = \sum_{i=1}^N w_{ij}$ [1605.06778], providing a compact representation of the feature distribution. For textual modalities detected in images, Fisher vectors are computed over collections of PHOC descriptors, using gradient pooling with respect to GMM means and variances [2001.04732].

## 3. Cross-Modal Fusion Strategies

BoW representations for each modality may be concatenated to yield a composite feature vector:

$$
H = [h^{(1)}; h^{(2)}; \dots; h^{(M)}] \in \mathbb{R}^{\sum_m K_m}
$$

Optionally, modality-specific scaling factors $\alpha_m$ can be applied: $H = [\alpha_1 h^{(1)}; ...; \alpha_M h^{(M)}]$ [1605.06778]. To mitigate the dominance of frequent or common patterns, TF-IDF weighting or power and L2 normalization are frequently employed.

Recent approaches move beyond concatenation towards shared latent spaces or attention-based fusion. Cross-modal discrete representation learning aligns empirical codeword distributions $P(e_k | H^M)$ between modalities using a symmetric cross-entropy:

$$
S_{\text{code}}(x_i^A, x_i^B) = \sum_k P(e_k | H_i^A) \log P(e_k | H_i^B) + \sum_k P(e_k | H_i^B) \log P(e_k | H_i^A)
$$

A contrastive InfoNCE wrapper $L_{\text{CMCM}}$ encourages distinct pairs to be separated [2106.05438].

Other methods employ deep models to map modality-specific histogram and feature vectors (BoW, color, tags) with type-specific RBMs, fusing them in a higher-level binary RBM that models both intra-type and inter-type correlations [1603.01359]. Attention-based fusion, such as softmax of $\tanh(V_{fa}^\top W T_f)$ for visual and textual Fisher vectors, further supports learning deep multimodal representations [2001.04732].

## 4. Training and Inference Pipelines

Typical multimodal BoW pipelines proceed as follows:

1. **Preprocessing**
   - For numeric features: framing, normalization, and silence/activity detection are standard steps.
   - For text: tokenization, n-gram extraction, vocabulary pruning, and optional PCA reduction for structured descriptors.

2. **Codebook Induction**
   - Codebooks are learned via k-means, GMM, or end-to-end differentiable mechanisms such as VQ.

3. **Encoding**
   - Compute embeddings for input segments.
   - Quantize to codewords using hard/soft assignment rules.
   - Aggregate to produce modality-specific histograms or Fisher vectors.

4. **Fusion**
   - Concatenate, weight, or attend across histograms or Fisher vectors.
   - Normalize the resulting vectors.

5. **Downstream Learning**
   - Features are used for classification, retrieval, clustering, or regression.
   - Additional supervision or multitask heads (e.g., for tagging, attribute prediction) may regularize the joint space [1603.01359, 2106.05438].

6. **Inference**
   - Feature extraction follows the same pipeline: compute code assignments and form BoW or Fisher vectors for new data.

Example toolkits such as openXBOW provide end-to-end pipelines with modifiable parameters, supporting ARFF, CSV, and LIBSVM formats [1605.06778].

## 5. Interpretability and Semantic Alignment

A key empirical property of joint discrete codebooks with cross-modal objectives is cluster interpretability. When trained with cross-modal code-matching, codebook atoms evolve into modality-invariant semantic atoms—each code index behaves like a discrete “word” semantically grounded across modalities. For instance, on video–speech datasets, certain codes fire only for particular annotated actions and the same codes correspond to aligned concepts in audio (e.g., juggling in both modalities). Visualization (e.g., t-SNE) of codebook usage reveals unified cluster structure only when cross-modal matching is applied [2106.05438].

Similarly, in deep factor analysis, topic factors learned from multimodal data localize to meaningful clusters across Poisson (count), Gaussian (continuous), and multinomial (text BoW) modalities [1508.00408]. In deep fusion architectures, pretraining and auxiliary tasks further refine the semantic correspondence between text and non-textual information [1603.01359].

## 6. Representative Applications and Empirical Results

Multimodal BoW and its variants have been validated in applied contexts:

- **Emotion recognition in continuous speech:** High-dimensional BoAW features using openXBOW, with TF-IDF and normalization, yield Concordance Correlation Coefficient improvements over baselines, particularly enhancing valence modeling [1605.06778].
- **Twitter sentiment analysis:** Multimodal BoW with TF-IDF and feature selection achieves or exceeds the performance of more complex neural models on large-scale datasets [1605.06778].
- **Fine-grained image classification/retrieval:** Fusion of textual Fisher vectors built over PHOC descriptors with CNN features attains state-of-the-art accuracy and mAP in fine-grained tasks. Notably, combining visual and PHOC-based textual FV yields large improvements over individual modalities and other fusion baselines [2001.04732].
- **Social image retrieval/concept prediction:** Deep fusion models using Poisson RBMs with BoW input and multitask heads provide substantial retrieval and multilabel prediction gains, especially when multiple modalities are fused [1603.01359].

Summary performance table (examples):

| Task                                  | Modality                 | Best BoW Approach              | Performance Metric          | Citation      |
|---------------------------------------|--------------------------|-------------------------------|----------------------------|---------------|
| Speech Emotion Recognition            | Audio (LLDs)             | openXBOW, top-a=20 TF-IDF      | CCC: .793 (valence)         | [1605.06778]  |
| Twitter Sentiment                     | Text                     | BoW + TF-IDF + SVM             | 77.28% acc.                | [1605.06778]  |
| Fine-grained Image Class/Ret.         | Vision+Text (PHOC FV)    | FV(PHOC)+CNN+attention fusion  | 80.2% accuracy (Con-Text)  | [2001.04732]  |
| Social Image Retrieval                | Multiview (BoW, color)   | Deep RBM fusion                | MAP: 0.420                 | [1603.01359]  |

## 7. Unified Generative Models

Multimodal BoW can be subsumed under generative factor models combining Poisson (counts), Gaussian (continuous), and multinomial (text) likelihoods sharing a latent space [1508.00408]. The object representation $z_t \sim \mathcal{N}(0, I_K)$ is mapped to each modality via individual loading matrices. The multinomial component is a direct text BoW model, with softmax of the latent factors yielding a document-specific word distribution.

EM-based inference interleaves posterior approximation for the latent $\{ z_t \}$ (e.g., via Laplace approximation) and closed-form or quasi-Newton updates for loadings. This establishes a direct connection between topic models, multimodal dimensionality reduction, and classical BoW [1508.00408].

---

Recent multimodal BoW developments demonstrate that code quantization, cross-modal alignment, and flexible histogram aggregation yield robust, interpretable, and semantically meaningful representations applicable to a wide range of high-level multimodal understanding tasks [2106.05438, 1605.06778, 1603.01359, 2001.04732, 1508.00408].

Source: https://www.emergentmind.com/topics/multimodal-bag-of-words-representation