byteSteady: Fast Byte-Level Classification

• byteSteady is a two-layer machine learning architecture that extracts byte-level n-grams and uses a hashing trick to build a compact embedding representation.
• It employs average pooling of embeddings before applying a linear classifier, achieving competitive results on multilingual text and gene classification tasks.
• The method supports operation on compressed byte streams, offering a novel accuracy-speed trade-off for resource-constrained and real-time applications.

byteSteady is a two-layer machine learning architecture for fast classification on arbitrary sequences of bytes. It represents inputs using averaged byte-level $n$-gram embeddings, enabling efficient and highly scalable application to a range of classification problems, including both natural language and non-language data. The key elements of byteSteady are: byte-level $n$-gram extraction, embedding table indexation via a hashing trick, pooling, and linear classification. The method supports direct operation on compressed sequences with minimal degradation of classification accuracy, presenting a novel accuracy-speed trade-off (Zhang et al., 2021).

1. Architecture and Input Representation

byteSteady operates directly on raw byte sequences $(b_1, b_2, \ldots, b_L)$. For each input, the model:

• Extracts all byte-level $n$-grams for a predefined set $N$ of $n$ values by sliding a window of length $n$ over the sequence.
• Hashes each extracted $n$-gram $g$ into an integer index using a function $h$, reduced modulo a fixed hash table size $n$0.
• Embeds each hashed $n$1-gram using a shared embedding matrix $n$2, with $n$3 the embedding dimension.
• Averages the embedding vectors across all $n$4-grams in the input to produce the representation $n$5.

Formally, for input $n$6 (the multiset of $n$7-grams),

$n$8

Alternatively, if $n$9 tallies normalized $(b_1, b_2, \ldots, b_L)$0-gram frequencies by hash index, then $(b_1, b_2, \ldots, b_L)$1.

This pooled representation $(b_1, b_2, \ldots, b_L)$2 is then passed through a linear classifier: $(b_1, b_2, \ldots, b_L)$3 where $(b_1, b_2, \ldots, b_L)$4 and $(b_1, b_2, \ldots, b_L)$5 is the number of classes.

2. Hashing Trick for Compact Embedding Tables

The hashing trick in byteSteady allows mapping a potentially vast set of byte-level $(b_1, b_2, \ldots, b_L)$6-grams into a fixed-size embedding table. Each $(b_1, b_2, \ldots, b_L)$7-gram $(b_1, b_2, \ldots, b_L)$8 is hashed by $(b_1, b_2, \ldots, b_L)$9 (e.g., FNV or CityHash), and the result is mapped into one of $n$0 slots via modulus. Collisions are allowed, but are rare for large $n$1 (typically $n$2).

Aspect	Mechanism	Significance
Hashing	$n$3-gram $n$4 integer $n$5	Reduces embedding table size; supports O(1) lookup
Embedding Table	$n$6	Parameter sharing; efficient memory usage
Collisions	Allowed, rare if $n$7 is large	Has negligible impact in practice

By fixing $n$8, memory complexity is decoupled from the exponential growth of possible byte-level $n$9-grams.

3. Classification Layer and Optimization

The final representation vector $N$0 is classified using a single linear transformation. For input $N$1 and label $N$2,

$N$3

where $N$4 is one-hot. Optimization is performed with asynchronous, single-sample SGD (Hogwild!), employing L2 weight decay. There are no intermediate nonlinearities, making the entire system a two-layer, purely linear model.

4. Training Regimen and Hyperparameter Selection

byteSteady uses a fixed embedding dimension ($N$5 shows saturation in ablation), and a hash-table size $N$6. The optimal set of $N$7 varies by domain:

• Text classification: $N$8.
• Gene classification: $N$9.

Weight decay is set to $n$0 for text and $n$1 for gene tasks. Learning proceeds with a fastText-style initial learning rate (approximately 0.1), decreased over epochs. Training is CPU-optimized using multithreaded Hogwild! and sparse updates.

5. Experimental Performance

byteSteady was benchmarked on 14 large-scale text datasets spanning English, Chinese, Japanese, and Korean, using standard train/test splits from Zhang & LeCun (2017). It achieved error rates competitive with or superior to word-level fastText and deep byte-level ConvNets. For instance, on the two-class Chinese Dianping dataset, byteSteady achieved an error of 22.61%, compared to 22.62% (fastText) and 23.17% (ConvNet).

In gene classification (PATRIC, 5.1 million bacterial genes, 6 classes), byteSteady outperformed baselines using k-mer features: 3.73% test error versus 12.89% and 13.88% for $n$2k and $n$3M k-mer features, respectively.

6. Classification from Compressed Byte Streams

Because byteSteady operates at the byte level, pre-processing with compression schemes such as Huffman coding is feasible. Byte-level and bit-level Huffman coding are supported:

• Bit-level: binary tree, symbols of length $n$4 bytes.
• Byte-level: 256-ary tree, symbols grouped by $n$5.
• Compression ratios on text (dev set): bit-level $n$6 yields $n$7 size; byte-level $n$8 yields $n$9.

Classification after light byte-level compression ($n$0) causes minimal accuracy loss. For text, error rises from 26.22% (uncompressed) to 26.49% (compressed); for gene data, from 7.03% to 7.76%. More aggressive compression introduces a tunable degradation in accuracy, offering a novel speed-accuracy trade-off. Preservation of byte boundaries in byte-level coding yields better performance than bit-level approaches at equivalent compression.

7. Applications, Implications, and Future Directions

byteSteady’s architecture is broadly applicable wherever data can be represented as byte sequences, including multilingual text classification, bioinformatics (e.g., gene function and phenotype prediction), malware or network packet inspection, and other domains involving raw byte signals. The minimal pre-processing requirement—no tokenization, alignment, or vocabulary construction—combined with speed and memory efficiency, differentiates the approach from more complex neural architectures.

A notable implication is that compression-aware classification, especially with pre-compressed input streams, offers flexible resource allocation for real-time or constrained environments—a paradigm not extensively explored previously in machine learning literature. Future research directions include unsupervised learning of byte-level $n$1-gram embeddings, extending to higher-order alphabets, and real-time compressed domain inference (Zhang et al., 2021).