Bag-of-Symbol Representation

• Bag-of-symbol representation is an order-invariant encoding that aggregates discrete symbols into histograms, capturing symbol identities and frequencies.
• Techniques such as k-means clustering and tf-idf weighting are employed to induce vocabularies and enable effective similarity comparisons across domains.
• Its simplicity and robustness afford computational efficiency, though the inherent loss of temporal or spatial structure remains a key limitation.

Bag-of-symbol representation denotes a family of order-invariant encodings in which an object is treated as an unordered multiset of discrete or quantized symbols, and the representation records only symbol identities and multiplicities, or a distribution induced by them. In text this yields bag-of-words and bag-of-concepts; in images, bag-of-features or bag-of-visual-words; in audio, bag-of-frames; in time series, histograms over codewords or symbolic bins; and in game analytics, bag-of-tokens over JSON paths. Across these settings, the common abstraction is that local observations are pooled into a compact summary that is simple, robust, and often effective, while deliberately suppressing temporal or spatial order (Lagrange et al., 2014, Amato et al., 2016, Wang et al., 2012, Lebret et al., 2014).

1. Conceptual and mathematical basis

The defining operation is aggregation over a finite vocabulary. In text, a document may be represented as a sparse count vector over a dictionary, with binary or term-frequency entries. In image retrieval, an image is mapped to a term vector over a visual vocabulary, often with tf-idf weighting. In time-series bag-of-words, local windows are assigned to codewords and accumulated into a histogram. In JSON-Bag, a trajectory is converted to a token count vector and then normalized to a probability distribution. These constructions all share the same structural principle: they encode what symbols occur and how often, but not where or when they occur (Amato et al., 2016, Wang et al., 2012, Nguyen et al., 1 Aug 2025, Lauly et al., 2014).

Common pooled forms include hard-assignment histograms,

$h_i = \sum_j \mathbf{1}[a(s_j)=i],$

normalized distributions,

$\tilde{h}_k = \frac{h_k}{\sum_j h_j},$

and, in audio bag-of-frames, parametric density models such as

$$p(f \mid \theta) = \sum_{k=1}^K \pi_k \mathcal{N}(f;\mu_k,\Sigma_k).$$

The choice among these forms depends on whether the application benefits more from discrete counts, normalized frequencies, or an explicit probabilistic model over local descriptors (Wang et al., 2012, Nguyen et al., 1 Aug 2025, Lagrange et al., 2014).

The loss of order is not incidental but constitutive. In environmental audio, the identity of a scene is assumed to be largely captured by the distribution of short-term spectral features rather than event order. In text autoencoding, sentence representations are formed by summing word embeddings, again discarding order. In symbolic time-series methods, aggregation across windows is intended to provide invariance to warping, temporal shifts, and localized noise. This order suppression is therefore both the source of the representation’s robustness and its central expressive limitation (Lagrange et al., 2014, Lauly et al., 2014, Garcia et al., 24 Sep 2025).

2. Symbol formation and vocabulary induction

Symbols may be given a priori or learned from data. In classical bag-of-words, the vocabulary is simply the set of lexical types in the corpus. In explicit concept-space methods, symbols are Wikipedia article titles or other knowledge-base concepts. In bag-of-concepts for document classification, symbols are induced by clustering continuous n-gram embeddings: unigrams, bigrams, and trigrams are embedded in a common semantic space, then partitioned by K-means into semantic concepts whose cluster indices serve as symbols (Shalaby et al., 2017, Lebret et al., 2014).

In vision, symbol induction is usually vector quantization of local appearance descriptors. A visual vocabulary

$V=\{w_1,\dots,w_M\}$

is learned from descriptors such as SIFT by k-means or related clustering, and each local descriptor is hard-assigned to its nearest centroid. The same logic appears in ancient-coin classification, where dense SIFT on reverse-side motifs is quantized into visual words, with subsequent spatial encoding adapted to coin geometry (Amato et al., 2016, Anwar et al., 2013).

Time-series applications instantiate the same idea with domain-specific local descriptors. Biomedical time-series bag-of-words extracts short, overlapping windows, z-normalizes them, transforms them with single-level DWT using db3 approximation coefficients, and learns a codebook by k-means; hard assignment then yields a histogram of codewords. In gait recognition, 1 s windows with 50% overlap are described by 15-dimensional statistics of thigh accelerometry, and K-means centroids define the vocabulary. In building-data analysis, ON cycles are symbolized by SAX after z-normalization and PAA, then converted into bags of words for hierarchical clustering (Wang et al., 2012, Bayat et al., 2022, Habib et al., 2016).

Alternative symbolization schemes avoid learned codebooks and instead discretize amplitudes or structures directly. Symbol-Temporal Consistency learning defines $K=64$ bins per channel using equi-width cut lines in $[-3\sigma,+3\sigma]$ plus two overflow bins, and aggregates the resulting symbols into a histogram over time and channels. JSON-Bag tokenizes JSON leaves by root-to-leaf path strings and atomic values, optionally preserving list indices in ordered mode and omitting them in unordered mode. This makes the symbolization procedure schema-driven rather than descriptor-driven (Garcia et al., 24 Sep 2025, Nguyen et al., 1 Aug 2025).

3. Pooling, comparison, and inference

Once symbols have been induced, bag-of-symbol systems differ mainly in weighting, normalization, and similarity measurement. Image retrieval classically uses tf-idf weighting and cosine similarity,

$$s(q,d)=\frac{h^{(q)}\cdot h^{(d)}}{\|h^{(q)}\|_2\|h^{(d)}\|_2},$$

while biomedical time-series classification compares histograms with Euclidean, Jensen–Shannon, histogram-intersection, or $\chi^2$ distances, with $\chi^2$ performing best overall in that study. Audio bag-of-frames instead compares recording-level GMMs with KL divergence, estimated by Monte Carlo because closed-form KL is intractable for Gaussian mixtures (Amato et al., 2016, Wang et al., 2012, Lagrange et al., 2014).

Prototype-based inference is a recurrent pattern. JSON-Bag forms a mean token distribution for each class and performs prototype-based nearest-neighbor search using Jensen–Shannon distance. In low-dimensional bag-of-concepts text classification, documents are represented in $K$-dimensional concept space and classified with a linear SVM. Ancient-coin symbol recognition uses RBF-kernel SVMs over concatenated regional BoVW histograms. These examples show that bag-of-symbol representations are not tied to a single decision rule; they support nearest-neighbor, margin-based, and tree-based classifiers as long as the pooled representation is compatible with the chosen geometry (Nguyen et al., 1 Aug 2025, Lebret et al., 2014, Anwar et al., 2013).

A notable refinement in text similarity is the probabilistic measure

$\tilde{h}_k = \frac{h_k}{\sum_j h_j},$0

where $\tilde{h}_k = \frac{h_k}{\sum_j h_j},$1 counts corpus documents whose frequency of term $\tilde{h}_k = \frac{h_k}{\sum_j h_j},$2 lies between the two documents’ term frequencies. Unlike TF–IDF cosine, this measure is query-dependent at the term level. In binary bag-of-words it reduces to an idf-like contribution for each shared term; in term-frequency bag-of-words it uses corpus frequency intervals rather than static weights. Empirically it yields better results in binary BoW and competitive, more consistent results in TF-based BoW (Aryal et al., 2019).

4. Cross-domain realizations

The family resemblance across modalities is clearest when symbol source and pooled form are viewed side by side.

Domain	Symbols	Pooled representation
Text	words, n-grams, concepts	counts, tf-idf, concept histograms
Images	quantized local descriptors	BoVW histograms, spatial pyramids
Audio and time series	frames, windows, SAX bins	histograms or recording-level distributions
Game trajectories	JSON path-value tokens	normalized token distributions

In text, bag-based methods span sparse lexical counts, clustered semantic concepts, prototype reconstructions, and explicit concept spaces. The n-gram clustering model reaches 84.00 on Pang & Lee and 88.55 on Maas et al. with $\tilde{h}_k = \frac{h_k}{\sum_j h_j},$3 features, approaching traditional BoW performance with far fewer dimensions. dCoT addresses over-sparsity by reconstructing frequent prototype terms from corrupted sparse bags, while concept-embedding densification replaces sparse bag-of-concepts vectors with weighted averages of learned concept embeddings (Lebret et al., 2014, Zhixiang et al., 2013, Shalaby et al., 2017).

In vision, bag-of-features became central because it supports inverted-file indexing and large-scale search. On Oxford Buildings 5k, idf and scale pruning achieve $\tilde{h}_k = \frac{h_k}{\sum_j h_j},$4 while reducing the average words per image from about 3,200 to about 800. In ancient-coin classification, circular tiling outperforms rectangular and log-polar tilings for smaller visual dictionaries, reflecting the rotational geometry of coins. In self-supervised learning, BoWNet constructs a visual-word target from quantized convnet features and predicts the original image’s BoVW from a perturbed view; on CIFAR-100, linear accuracy improves from 60.3 for RotNet to 71.5 for BoWNet (Amato et al., 2016, Anwar et al., 2013, Gidaris et al., 2020).

In audio and time series, the representation is often motivated by invariance to local distortions and repeated motifs. Biomedical time-series bag-of-words reaches 93.8% on EEG, 99.5% on ECG-40, and 100% on ECG-15 with 1-NN and $\tilde{h}_k = \frac{h_k}{\sum_j h_j},$5 distance. In gait recognition, the bag-of-words representation improves accuracy across all evaluated classifiers, with an average improvement of $\tilde{h}_k = \frac{h_k}{\sum_j h_j},$6 percentage points over the statistical-feature baseline. By contrast, the replication study on urban soundscapes finds that bag-of-frames does not outperform a one-point mean-vector baseline on more realistic datasets, with performance on Tardieu and QMUL settling near $\tilde{h}_k = \frac{h_k}{\sum_j h_j},$7 (Wang et al., 2012, Bayat et al., 2022, Lagrange et al., 2014).

Game-trajectory modeling shows that the abstraction remains viable even when the raw object is a structured state log rather than a signal or document. JSON-Bag tokenizes serialized game states, normalizes token counts to probability distributions, and compares them with Jensen–Shannon distance. On 7 Wonders seed classification, JSON-Bag with prototype-based nearest-neighbor search attains 0.942 versus 0.573 for hand-crafted features, and on Connect4 parameter classification it reaches 1.000. Across six games, prototype JSDs between agent classes correlate strongly with policy distances, with Pearson $\tilde{h}_k = \frac{h_k}{\sum_j h_j},$8 ranging from 0.6225 to 0.9688 (Nguyen et al., 1 Aug 2025).

5. Strengths, failure modes, and evaluation conditions

The strengths of bag-of-symbol representation are consistent across domains: simplicity, robustness, compactness, and favorable computational structure. It is effective when global statistics dominate over order, as in stationary textures, repeated local motifs, or coarse semantic categorization. Biomedical bag-of-words remains robust under additive white Gaussian noise from 10 dB down to 0 dB. Visual-word pruning preserves roughly 80% of retrieval effectiveness while retaining only 25% of the original words. JSON-Bag prototypes remain sample efficient, reaching 1.000 with as few as 3 shots for Connect4 parameter classification (Wang et al., 2012, Amato et al., 2016, Nguyen et al., 1 Aug 2025).

Its failure modes are equally recurrent. Sparse lexical bags suffer from over-sparsity, synonymy, and polysemy; sparse bag-of-concepts suffers from low concept overlap; discrete numeric tokenization can fragment semantically related values into unrelated symbols; and aggressive discretization can make assignments sensitive to noise near bin boundaries. The most consequential limitation, however, is the loss of structure. In soundscape analysis, bag-of-frames suppresses timing, duration, and co-occurrence patterns of acoustic events, and its apparent success on the original dataset is linked to low within-class variability and near-duplicate segmentation. When segmentation into 3-minute units is removed, precision drops to 71%, and on more varied datasets performance settles near 48%. Evaluation guidance arising from that study recommends avoiding segmentation that creates near-duplicates, enforcing cross-location or cross-device splits, analyzing class-wise variability, and comparing against simple baselines such as mean-vector nearest neighbor (Zhixiang et al., 2013, Shalaby et al., 2017, Nguyen et al., 1 Aug 2025, Garcia et al., 24 Sep 2025, Lagrange et al., 2014).

A common misconception is that strong performance on a constrained benchmark establishes the sufficiency of the bag abstraction itself. The cross-domain evidence is narrower. Bag-of-symbols is often a strong baseline, sometimes a competitive endpoint, and frequently an excellent indexing or retrieval substrate. This suggests that the decisive question is not whether order should be discarded universally, but under what data regimes the discarded structure is nonessential.

6. Densification, contextualization, and hybrid forms

Recent work increasingly uses bag-of-symbol objects not only as final descriptors but also as intermediate supervisory targets, auxiliary views, or dense priors. dCoT learns a dense representation by reconstructing a small set of frequent prototype terms from corrupted sparse inputs, with a marginalized closed-form solution

$\tilde{h}_k = \frac{h_k}{\sum_j h_j},$9

Efficient bag-of-concepts densification instead learns concept embeddings and computes a weighted average,

$$p(f \mid \theta) = \sum_{k=1}^K \pi_k \mathcal{N}(f;\mu_k,\Sigma_k).$$0

thereby reducing sparse-overlap problems while keeping aggregation linear in the number of nonzero concepts (Zhixiang et al., 2013, Shalaby et al., 2017).

In sequence models, the bag is often retained as a context summary rather than a terminal representation. Context-dependent neural machine translation computes a sentence-level nonlinear bag-of-words

$$p(f \mid \theta) = \sum_{k=1}^K \pi_k \mathcal{N}(f;\mu_k,\Sigma_k).$$1

then uses $$p(f \mid \theta) = \sum_{k=1}^K \pi_k \mathcal{N}(f;\mu_k,\Sigma_k).$$2 to gate source and target embeddings so that irrelevant embedding dimensions are suppressed. A multilingual bag-of-words autoencoder similarly learns aligned embeddings by reconstructing the source sentence’s bag from the encoding of its translation, without word-level alignments (Choi et al., 2016, Lauly et al., 2014).

In vision and time-series self-supervision, the bag becomes a target or auxiliary invariant view. BoWNet predicts the original image’s normalized bag-of-visual-words from a perturbed image using cross-entropy over visual-word distributions. Symbol-Temporal Consistency learning aligns raw temporal embeddings with bag-of-symbol histograms constructed from discretized time-series amplitudes, combining time-domain contrastive loss, symbol-domain contrastive loss, and a cross-view consistency term. In environmental audio, where plain BOF proves insufficient, the proposed alternative is explicitly hybrid: combine texture-like summaries with modeling of individual acoustic events and their temporal organization (Gidaris et al., 2020, Garcia et al., 24 Sep 2025, Lagrange et al., 2014).

This suggests a contemporary reinterpretation of bag-of-symbol representation. Rather than serving only as an end-state histogram, it now functions as an efficient invariant summary, a structured target for self-supervision, a dense semantic scaffold, and a complementary branch in hybrid models that reintroduce temporal, spatial, or event-level structure when order becomes indispensable.