Bag of Whole-Words (BoW) Overview

• Bag of Whole-Words (BoW) is an orderless representation that encodes documents or images as fixed-length sparse vectors based on token counts and weighting schemes like TF-IDF.
• It is widely used in text mining, image retrieval, and multimodal analysis to facilitate efficient similarity computations and integration with standard classifiers.
• Despite its computational simplicity and effectiveness, BoW’s disregard for word order and structure motivates research into semantic, geometric, and neural extensions.

Bag of Whole-Words (BoW), usually written Bag-of-Words, is a representation in which a document is encoded as an unordered multiset of whole-word tokens, or equivalently as a vector indexed by a vocabulary whose coordinates record presence, counts, or weighted counts of terms. In text, a document $d$ is represented in $\mathbb{R}^{|V|}$, where $V$ is the vocabulary; in vision, the same abstraction is instantiated over “visual words,” i.e., codebook entries obtained by quantizing local descriptors, so that an image or video is represented by a histogram over those entries. Across these settings, BoW is valued for fixed-length sparse representations, simple similarity computations, and compatibility with inverted files and standard classifiers, but it also discards order and structural relations, which has motivated semantic, geometric, temporal, and neural extensions (Shimomoto et al., 2018, Cakir et al., 2014, Schmitt et al., 2016).

1. Formal representation and weighting

In the conventional vector space model for text, a corpus with $D$ documents and a vocabulary of size $V$ yields a document-term matrix $X \in \mathbb{R}^{V \times D}$, where entry $X_{i,j}$ describes how term $w_i$ occurs in document $d_j$. A single document is therefore a vector $x \in \mathbb{R}^V$ whose $\mathbb{R}^{|V|}$0-th component encodes the contribution of word $\mathbb{R}^{|V|}$1 to that document. This representation is explicitly orderless: it preserves occurrence information, but not syntax or sequence (Shimomoto et al., 2018, Lebret et al., 2014, Zhang et al., 2017).

The literature instantiates several weighting schemes. One formulation defines binary weighting as presence or absence, term-frequency as $\mathbb{R}^{|V|}$2, inverse document-frequency as $\mathbb{R}^{|V|}$3, and TF-IDF as $\mathbb{R}^{|V|}$4, with a damped variant $\mathbb{R}^{|V|}$5 in very large corpora (Shimomoto et al., 2018). Another formulation writes $\mathbb{R}^{|V|}$6, $\mathbb{R}^{|V|}$7, and $\mathbb{R}^{|V|}$8 for TF-IDF (Lebret et al., 2014). In openXBOW, text post-weighting includes $\mathbb{R}^{|V|}$9 and $V$0 (Schmitt et al., 2016).

Scheme	Definition in the cited literature	Sources
Binary	Presence/absence; $V$1	(Shimomoto et al., 2018, Aryal et al., 2019)
TF	$V$2; or $V$3	(Shimomoto et al., 2018, Lebret et al., 2014)
IDF	$V$4; or $V$5	(Shimomoto et al., 2018, Lebret et al., 2014)
TF-IDF	$V$6; damped $V$7; or $V$8	(Shimomoto et al., 2018, Lebret et al., 2014)

Outside text, BoW generalizes by replacing lexical tokens with quantized descriptors. In Bag-of-Visual-Words pipelines, local descriptors $V$9 are assigned to a learned codebook $D$0, and the image-level representation is obtained by average pooling or histogramming over assignment vectors (Cakir et al., 2014). The same principle is used for covariance descriptors on SPD manifolds (Faraki et al., 2014), 3D façade detail recognition from projected point clouds (Froech et al., 2024), and multimodal acoustic and visual low-level descriptors in openXBOW (Schmitt et al., 2016).

2. Similarity, indexing, and decision rules

Cosine similarity is described as the most widely used inter-document similarity measure in text mining. For weighted vectors $D$1 and $D$2, it is

$D$3

The normalization compensates for document length and makes sparse BoW vectors directly comparable (Aryal et al., 2019). Weighted Jaccard is also used,

$D$4

and BM25 appears in inter-document similarity experiments as a length-normalized, IDF-based alternative (Aryal et al., 2019).

A more specialized alternative is the interval-based probabilistic measure $D$5, which, for each shared term, scores similarity by how rare it is in the corpus to observe document frequencies lying between the two documents’ counts:

$D$6

where $D$7 counts the documents whose frequency for term $D$8 lies in the interval $D$9. In the binary case, this reduces to an IDF-like contribution per shared term and empirically gives better results than standard baselines on several datasets (Aryal et al., 2019).

In retrieval-oriented BoW systems, sparse histograms are particularly valuable because they support inverted files. Visual BoW systems on Oxford Buildings, landmark recognition, ORB-SLAM, and ground-texture localization all rely on this property. In those systems, tf-idf weighted BoW vectors are inserted into an inverted index, query-time lookups traverse only the posting lists of words present in the query, and scoring is performed by weighted voting or cosine-style similarity over sparse histograms (Amato et al., 2016, Wilhelm et al., 16 May 2025, Lee et al., 4 Mar 2026).

BoW features also integrate easily with standard classifiers. In the Reuters experiments on text classification, baselines included Multi-variate Bernoulli Naive Bayes, Multinomial Naive Bayes, Latent Semantic Analysis, and Support Vector Machine using binBOW, tfBOW, tfidfBOW, or word2vec-derived features (Shimomoto et al., 2018). In activity recognition, BoW histograms were paired with $V$0-NN, HMMs, and SVMs (Bettadapura et al., 2015). In openXBOW, BoW or Bag-of-Audio-Words representations were exported for downstream SVM or SVR models (Schmitt et al., 2016).

3. Core assumptions and limitations

The classical BoW abstraction assumes word independence, orderlessness, and a discrete feature space. In text, this means that semantically related words are treated as orthogonal basis elements, multi-word expressions are split apart, and context-sensitive relations such as synonymy or polysemy are not explicitly encoded (Zhang et al., 2017, Lebret et al., 2014). A standard criticism therefore appears repeatedly: BoW loses word order, ignores proximity and position of words, and does not encode semantic relations among words that arise from context (Shimomoto et al., 2018).

High dimensionality and sparsity are the second persistent limitation. With unigrams alone, the vocabulary can already contain tens or hundreds of thousands of dimensions, and adding $V$1-grams causes exponential growth in dictionary size. The result is expensive storage, a higher risk of overfitting, and large sparse vectors in which most coordinates are zero (Lebret et al., 2014, Karami, 2017). In visual search, the analogous difficulty is that a single image may contain on the order of $V$2–$V$3 visual words, so each query probes thousands of posting lists; this is one reason that visual word reduction and codebook pruning have been studied intensively (Amato et al., 2016, Cakir et al., 2014).

The same orderlessness becomes problematic when structure is intrinsic to the task. In activity recognition, standard BoW discards temporal order, durations, and causal structure, even though “what happened” and “when” jointly define the class (Bettadapura et al., 2015). In cross-modal representation learning, CLIP has been shown to behave like a BoW model when comparing images and captions: it often matches them by their shared set of concepts rather than by the specific attribute-object bindings, even though the binding information is linearly recoverable within each modality (Koishigarina et al., 5 Feb 2025). This suggests that BoW-like behavior can arise not only from explicit histogramming, but also from similarity functions that collapse structure into unordered concept overlap.

A further misconception is that BoW is inherently textual. The literature instead treats it as a general quantization-and-counting principle. In vision it counts visual words, in audio it counts quantized low-level descriptors, in 3D geometry it counts codewords derived from projected point-cloud features, and in neural late-interaction retrieval it can denote a set of contextualized whole-word vectors rather than scalar counts (Schmitt et al., 2016, Froech et al., 2024, Hofstätter et al., 2022).

4. Semantic and low-dimensional text extensions

A major line of work preserves the BoW intuition while replacing whole-word counts with semantic or low-dimensional structures. One example is the word subspace model for text classification. In that formulation, words are mapped to pretrained word2vec vectors in $V$4 with $V$5, and a class or document is represented by a low-dimensional linear subspace obtained by PCA without data centering from the autocorrelation matrix

$V$6

A TF-weighted extension forms $V$7 and takes the leading left singular vectors of its SVD as the subspace basis. Similarity between class and query is then measured by canonical angles under the Mutual Subspace Method. On Reuters-8, MSM with unweighted word subspaces reached $V$8, Multinomial Naive Bayes with tfBOW reached $V$9, and TF-MSM reached the best accuracy, $X \in \mathbb{R}^{V \times D}$0, with a $X \in \mathbb{R}^{V \times D}$1-test against MNB yielding $X \in \mathbb{R}^{V \times D}$2 (Shimomoto et al., 2018).

Another line replaces word histograms with continuous bags of word embeddings. The Spherical Paragraph Model represents both words and documents on the unit hypersphere and uses the von Mises–Fisher distribution to generate word embeddings from a latent document direction. This produces fixed-length dense document vectors with a corpus-wide prior and a probabilistic interpretation absent from simple weighted averages. Empirically, SPM achieved strong performance on sentiment analysis and several topical classification settings, although BoW remained better on Reuters and on the 20Newsgroups-same split (Zhang et al., 2017).

Low-dimensional BoW alternatives also include clustering over compositional units rather than raw words. In the n-gram-based low-dimensional representation of documents, 1-, 2-, and 3-gram vectors are formed by averaging Skip-gram word embeddings, then clustered by $X \in \mathbb{R}^{V \times D}$3-means into semantic concepts. Documents are represented as bags of these concepts, optionally with Naive Bayes log-count-ratio weighting. On sentiment classification, the representation outperformed LSA and LDA and achieved results similar to a traditional BOW model with far fewer features; with $X \in \mathbb{R}^{V \times D}$4, the reduction was approximately $X \in \mathbb{R}^{V \times D}$5 on Pang & Lee and $X \in \mathbb{R}^{V \times D}$6 on the Maas dataset relative to unigram BOW (Lebret et al., 2014).

Dimensionality reduction can also be applied directly to BoW matrices. A fuzzy clustering–based Unsupervised Feature Transformation maps documents to soft cluster-membership vectors, using soft spherical $X \in \mathbb{R}^{V \times D}$7-means on normalized document rows. On Reuters-21578 and Ohsumed, the fuzzy-clustering representations were reported to outperform PCA and SVD in average classification accuracy and to show greater stability across reduced dimensions (Karami, 2017).

Neural retrieval has recently reintroduced the notion of “whole words” in a different form. ColBERTer’s Neural Bag of Whole-Words aggregates all subword token embeddings belonging to the same surface word into a single contextualized whole-word vector, then applies late interaction by MaxSim over these whole-word representations. The method reduces the number of stored vectors per document by learning unique whole-word representations and by pruning non-essential words through contextualized stopword gating. The reported storage reduction reaches up to $X \in \mathbb{R}^{V \times D}$8, while the smallest 1-dimensional setting achieves index storage parity with plaintext size and maintains strong retrieval effectiveness (Hofstätter et al., 2022).

BoW can also serve as supervision rather than as a downstream feature. In contrastive image-text training, replacing intact captions with short BoW captions built from shuffled content words improved zero-shot ImageNet-1k performance when combined with word balancing, and adding pseudo-BoW captions for uncaptioned images further improved results. The best model, trained with $X \in \mathbb{R}^{V \times D}$9M aligned pairs and $X_{i,j}$0M unaligned images, reached $X_{i,j}$1 zero-shot ImageNet-1k accuracy, compared with $X_{i,j}$2 for a CLIP model trained on $X_{i,j}$3M image-caption pairs (Tejankar et al., 2021).

5. Visual, geometric, and multimodal generalizations

In computer vision, the BoW analogy is exact at the level of representation but different at the level of tokens. A “word” is a codebook entry in descriptor space, learned typically by $X_{i,j}$4-means, and an image is represented by a histogram over the assigned visual words. This structure underlies landmark retrieval, action recognition, person re-identification, SLAM, and 3D façade matching (Cakir et al., 2014, Tian et al., 2017, Wilhelm et al., 16 May 2025).

Variant	Domain	Key idea
LE-BoW	Human action recognition	Map SPD covariance descriptors by matrix logarithm and run BoW in the Log-Euclidean space (Faraki et al., 2014)
Visual word selection without re-coding	Scene and object recognition	Evaluate pruned codebooks from precomputed quantities under hard or soft coding (Cakir et al., 2014)
Metric learning in codebook generation	Person re-identification	Learn a Mahalanobis metric for local features and run $X_{i,j}$5-means under that metric (Tian et al., 2017)
Improved BoW for ground texture localization	Localization and loop closure	Use AKM, soft assignment, size bins, and orientation verification (Wilhelm et al., 16 May 2025)
HBRB-BoW	ORB-SLAM vocabulary training	Preserve real-valued statistics during hierarchical training and binarize only at leaves (Lee et al., 4 Mar 2026)
openXBOW	Crossmodal text, audio, and visual data	Generate per-modality subbags and concatenate them into a final bag (Schmitt et al., 2016)

Geometric corrections are a particularly important theme. In LE-BoW, local descriptors are covariance matrices, which lie on the SPD manifold rather than in a Euclidean vector space. The method therefore applies the principal matrix logarithm, vectorizes the symmetric log-mapped matrix with a scale-preserving half-vectorization, and then performs Euclidean $X_{i,j}$6-means, histogram encoding, and SVM classification in the embedded space. This avoids Euclidean distortions such as the swelling effect and achieved $X_{i,j}$7 CCR on KTH with sparse coding, $X_{i,j}$8 MAP on Olympic Sports with spatio-temporal pyramids, and $X_{i,j}$9 CCR on ADL with sparse coding (Faraki et al., 2014).

Visual BoW has also been adapted to computational constraints. In large-scale image retrieval, reducing the number of visual words per image is important because distinct-word count largely determines inverted-index traversal cost. Descriptor-level pruning by keypoint scale and word-level pruning by tf, idf, or tf*idf were shown to retain strong effectiveness while improving efficiency; on Oxford Buildings, keeping only the top $w_i$0 of words preserved approximately $w_i$1 of effectiveness relative to the full representation (Amato et al., 2016). A related approach studies pruning a codebook without re-coding or re-pooling by computing altered image representations from precomputed quantities under hard and soft assignment (Cakir et al., 2014).

Supervision can be pushed into the codebook itself. In person re-identification, local superpixel descriptors were clustered under a KISSME-learned Mahalanobis metric instead of Euclidean distance, producing identity-aware visual words. With four low-level feature types and strip-wise pooling, the method improved Rank-1 accuracy by $w_i$2 over the unsupervised BoW baseline on VIPeR and reached $w_i$3 Rank-1 on VIPeR, $w_i$4 on PRID450S, and $w_i$5 Rank-1 with $w_i$6 mAP on Market1501 (Tian et al., 2017).

BoW remains central in SLAM and localization. For ground texture localization, an AKM vocabulary with soft assignment, keypoint size binning, and orientation verification raised mAP from $w_i$7 for the DBoW baseline to $w_i$8 for the high-accuracy variant on the HD Ground dataset, while also improving loop-closure score separability and global localization within $w_i$9 mm and $d_j$0 (Wilhelm et al., 16 May 2025). In ORB-SLAM vocabulary training, HBRB-BoW uses real-valued $d_j$1-means throughout the hierarchy and delays binarization until the leaves, improving loop-closing and relocalization; on KITTI, translation ATE improved from $d_j$2 m to $d_j$3 m (Lee et al., 4 Mar 2026).

The same quantize-and-count principle extends beyond conventional images. For façade detail reconstruction, ORB descriptors extracted from 2D projections of 3D point clouds were quantized into a visual dictionary with $d_j$4, and the resulting BoW histogram was concatenated with HOG features. On TUM-FACADE, ORB + HOG + JSD improved overall accuracy from $d_j$5 to $d_j$6 relative to ORB-only matching (Froech et al., 2024). openXBOW generalizes this strategy further by learning codebooks over arbitrary numeric low-level descriptors, supporting hard or Gaussian soft assignment, and concatenating subbags from text, audio, and visual modalities into a unified feature vector (Schmitt et al., 2016).

6. Empirical behavior, scope, and continuing relevance

BoW remains a strong baseline because explicit frequency signals often remain highly discriminative. On Reuters-8, Multinomial Naive Bayes with tfBOW achieved $d_j$7, outperforming unweighted word-subspace MSM and trailing TF-MSM only slightly (Shimomoto et al., 2018). In Twitter sentiment analysis with openXBOW, a unigram BoW with stopping and TF-IDF weighting reached weighted accuracy $d_j$8 and unweighted accuracy $d_j$9 on $x \in \mathbb{R}^V$0 English tweets (Schmitt et al., 2016). These results show that the classical representation is not merely a historical baseline; it remains competitive when the task aligns well with sparse lexical evidence.

At the same time, empirical results repeatedly show where BoW must be augmented. In activity recognition, adding temporal events, local $x \in \mathbb{R}^V$1-gram structure, and randomly sampled regular expressions produced large gains over standard BoW on surveillance, surgical skill assessment, soccer analysis, and wide-area airborne surveillance (Bettadapura et al., 2015). In image-text modeling, BoW-style caption deformation improved zero-shot classification but reduced zero-shot MSCOCO image-to-text Recall@1 from $x \in \mathbb{R}^V$2 to $x \in \mathbb{R}^V$3, indicating that the discarded syntax matters more for retrieval than for category recognition (Tejankar et al., 2021). In CLIP, near-chance cross-modal attribute binding on CLEVR, PUG:SPAR, and PUG:SPARE was attributed not to missing unimodal information but to a cosine-based alignment that behaves like bag-of-words matching across modalities (Koishigarina et al., 5 Feb 2025).

A plausible implication is that BoW should be understood less as a single model than as a family of orderless aggregation schemes with different inductive biases. Some variants retain sparse explicit frequencies; others quantize geometric descriptors; others aggregate contextualized whole-word vectors; still others use BoW captions as training supervision. What unifies them is the replacement of structured sequences or local measurements by unordered collections of atomic units and the subsequent use of counts, weights, or set-level similarities over that collection (Hofstätter et al., 2022, Schmitt et al., 2016, Wilhelm et al., 16 May 2025).

The enduring limitation is unchanged: the representation does not, by itself, encode order. The word-subspace work explicitly identifies this as a limitation inherited from BoW and notes future work to incorporate word order and dynamic context changes (Shimomoto et al., 2018). openXBOW mitigates order loss with text $x \in \mathbb{R}^V$4-grams, character $x \in \mathbb{R}^V$5-grams, and temporal segmentation for numeric streams (Schmitt et al., 2016). ColBERTer restores interpretability at the whole-word level without reverting to raw counts (Hofstätter et al., 2022). LABCLIP modifies the cross-modal similarity function so that CLIP no longer behaves as a bag-of-whole-words matcher in attribute binding (Koishigarina et al., 5 Feb 2025). The continuing research pattern is therefore not abandonment of BoW, but systematic correction of its structural blind spots while retaining its efficiency, sparsity, and modularity.