Charagram: Character n-gram Embeddings

• Charagram is a character-based compositional model that replaces complex sequential or convolutional encoders with an exact-match n-gram summation followed by a nonlinear projection.
• It represents words and sentences as bags of character n-gram counts, effectively handling out-of-vocabulary issues and morphological variations.
• Empirical evaluations show Charagram converges faster and outperforms charCNN/charLSTM on semantic similarity tasks while remaining competitive on POS tagging.

Searching arXiv for the cited papers to ground the article. I’ll retrieve the arXiv entries for Charagram and the related IsalGraph paper. Charagram is a character-based compositional model for embedding words or sentences from their characters. Introduced in "Charagram: Embedding Words and Sentences via Character n-grams" (Wieting et al., 2016), it represents a textual sequence by a character $n$-gram count vector followed by a single nonlinear transformation that yields a low-dimensional embedding. The model was proposed against a background in which character LSTMs and character CNNs were prominent but comparatively complex, slower to train, and not always stronger empirically. Charagram therefore occupies a specific position in subword representation learning: it is neither a sequential recurrent encoder nor a convolutional feature extractor, but a sparse exact-match composition model over character $n$-grams.

1. Conceptual basis and model family

Charagram is defined as a simple character-based compositional model for learning embeddings of words or sentences from their characters. Its core design choice is to replace recurrent or convolutional processing with a bag of character $n$-grams plus a learned projection. A word or sentence is therefore represented through the exact character substrings it contains, rather than through hidden-state recurrence over positions or convolutional filter responses over local windows (Wieting et al., 2016).

This places Charagram in the family of subword-aware embedding methods, but with a notably different inductive bias from charLSTM and charCNN. A charLSTM reads characters one by one and uses the final hidden state as the representation. A charCNN applies filters over character $n$-grams and max-pools over the sequence. Charagram instead assigns each exact character $n$-gram its own embedding vector and sums all matching $n$-gram vectors before applying a nonlinearity. The paper emphasizes that this exact matching yields greater specificity: a learned vector can correspond to a particular $n$-gram with a particular meaning, whereas CNN filters may respond to many similar patterns. At the same time, the model preserves subword sharing and can generalize to rare words and morphological variants.

The model was explicitly motivated as a test of whether a much simpler architecture could match or outperform more elaborate character-level models, especially on semantic similarity tasks and on rare or out-of-vocabulary forms where subword information is central. In that sense, Charagram is best understood not merely as an engineering simplification, but as a methodological argument about the sufficiency of sparse character $n$-gram composition for a broad class of lexical and sentential embedding problems.

2. Formal representation and parameterization

For a character-based textual sequence $x=\langle x_1,x_2,\ldots,x_m\rangle$, the embedding is defined by

$g_{char}(x) = h\!\left({b} + \sum_{i=1}^{m+1} \sum_{j=1+i-k}^i I{x_j^i\in V} \charmat^{x_j^i}\right),$

where $n$0 is a nonlinear function, $n$1 is a bias vector, $n$2 is the maximum character $n$3-gram length, $n$4 is an indicator that is 1 when $n$5 is true, $n$6 is the fixed vocabulary of character $n$7-grams, and $n$8 is the learned vector for each $n$9-gram $n$0 (Wieting et al., 2016).

The paper states that this can be viewed equivalently as first populating a length-$n$1 vector with counts of character $n$2-grams and then applying a nonlinear transformation. The parameter count is $n$3. This formulation makes the model computationally simple: there is no recurrent state transition, no convolutional filter bank, and no pooling operator beyond the summed $n$4-gram contribution followed by a single nonlinearity.

A notable property of the representation is that it can embed any character sequence, because it is defined over character $n$5-grams rather than a fixed word vocabulary. The paper uses this property to explain the absence of an OOV problem in the model. A plausible implication is that Charagram trades explicit sequence dynamics for a form of compositional coverage that is especially useful when lexical sparsity is substantial.

3. Training objective and optimization regime

For similarity tasks, Charagram is trained on PPDB paraphrase pairs using a margin-based contrastive loss:

$n$6

Here $n$7 is the embedding function, $n$8 is the margin, $n$9 collects all parameters, $n$0 is the $n$1 regularization coefficient, and $n$2 are negative examples selected from the mini-batch (Wieting et al., 2016). The objective enforces that paraphrase pairs be more similar under cosine similarity than each phrase is to a hard or random negative by at least $n$3.

Negative sampling is performed with either MAX, which chooses the most similar non-paired phrase in the mini-batch, or MIX, which uses MAX with probability $n$4 and random negatives otherwise. This training setup was used for both word and sentence similarity settings, with task-specific data configurations. For word similarity, the training data is the lexical section of PPDB XXL, with 770,007 word pairs for tuning and 50 epochs for final training. The vocabulary $n$5 contains character $n$6-grams from the training data; the main model uses all character bigrams, trigrams, and 4-grams that occur in the data at least once or twice, tuned over $n$7. Adam with learning rate $n$8 is used.

For sentence similarity, the model is referred to as charagram-phrase because the inputs may span multiple words. Training proceeds by initial training on PPDB XL with 3,033,753 unique phrase pairs, validation on annotated PPDB phrase pairs from PPDB 2.0, and final training on the PPDB XXL phrasal section with 9,123,575 unique phrase pairs for 10 epochs. The paper notes that the character $n$9-gram vocabulary includes $n$0-grams spanning across word boundaries, enabling the model to capture some word order and co-occurrence information that a simple averaging baseline cannot.

For POS tagging, the setup follows Ling et al. (2015): a character-level word representation is fed into a second bidirectional LSTM tagger. The Penn Treebank WSJ split is sections 1–18 for training, 19–21 for development, and 22–24 for test. Character embeddings are 50-dimensional, induced word representations are 150-dimensional, optimization is stochastic gradient descent with mini-batch size 100, learning rate 0.2, momentum 0.95, and 50 epochs, and a capital-letter binary feature is added. ReLU is considered for Charagram and charCNN.

4. Empirical performance across evaluation settings

The paper evaluates Charagram on word similarity, sentence similarity, and part-of-speech tagging, using Spearman’s $n$1 for word similarity, Pearson’s $n$2 for sentence similarity, and tagging accuracy for POS. Across these settings, the principal empirical claim is that Charagram outperforms more complex character-level recurrent and convolutional baselines on the similarity tasks and remains highly competitive on tagging (Wieting et al., 2016).

Evaluation setting	Charagram result	Comparison stated in the paper
WS353	58.35	vs. 51.43 for charLSTM and 33.19 for charCNN
SL999	63.33	vs. 54.54 for charLSTM and 30.64 for charCNN
SL999, larger Charagram	70.6	state of the art in their setting
Stanford Rare Word	47.1	better than 41.8 from Soricut and competitive with 47.8 from GloVe
STS 2012	66.1	vs. 58.5 paragram-phrase, 56.5 charCNN, 40.1 charLSTM
STS 2013	57.2	vs. 57.7 paragram-phrase, 47.7 charCNN, 30.7 charLSTM
STS 2014	74.7	vs. 71.5 paragram-phrase, 64.7 charCNN, 46.8 charLSTM
STS 2015	76.1	vs. 75.7 paragram-phrase, 66.0 charCNN, 45.5 charLSTM
2014 SICK	70.0	vs. 72.0 paragram-phrase, 62.9 charCNN, 50.3 charLSTM
2015 Twitter	53.6	vs. 52.7 paragram-phrase, 48.6 charCNN, 39.9 charLSTM
Overall sentence average	68.7	vs. 66.2 paragram-phrase, 59.2 charCNN, 41.9 charLSTM
POS tagging	96.99	charCNN 97.02, charLSTM 96.90, Charagram (2-layer) 97.10

For word similarity, the evaluation datasets are WordSim-353, SimLex-999, and the Stanford Rare Word Similarity Dataset. On SL999, the paper also reports comparisons against prior work: 52 for Hill et al., 56 for Schwartz et al., 58 for Faruqui et al., and 66.7 for Wieting et al.; the larger Charagram model reaches 70.6. For sentence similarity, evaluation covers 22 datasets from SemEval STS 2012–2015, plus SemEval 2014 SICK and SemEval 2015 Twitter. The paper states that charagram-phrase outperforms charCNN and charLSTM on every task and beats paragram-phrase on 15 of the 22 datasets. It matches or exceeds the best task-tuned systems on 5 tasks and is within 0.003 on 2 more. It also beats FastSent’s reported average on overlapping STS data, with 74.7 versus 61.3 on the 2014 STS tasks.

For POS tagging, the paper presents a different picture. Performance is very close among the character-based methods: charCNN reaches 97.02, charLSTM 96.90, Charagram 96.99, and Charagram (2-layer) 97.10. The significance assigned in the paper is therefore not dramatic superiority on syntax-oriented tagging, but competitive accuracy with faster convergence.

5. Convergence, ablation structure, and qualitative behavior

A major empirical finding is that Charagram converges much faster than the more complex models. The paper plots performance against number of training examples and reports that Charagram reaches high word-similarity and tagging performance in far fewer epochs than charCNN or charLSTM (Wieting et al., 2016). The authors suggest that this may explain part of the performance advantage observed under a fixed computational budget: the more complex architectures may still be improving after 50 epochs but are slower to optimize on the available training sets.

The ablation study examines how the number and length of character $n$3-grams affect results. The paper tests $n$4 most frequent $n$5-grams for each order and compares using only bigrams, bigrams plus trigrams, up through 6-grams. The central trend is that semantic similarity requires substantially more $n$6-grams than POS tagging. With only 100 $n$7-grams per order, POS remains reasonably strong, but word and sentence similarity are poor. As the vocabulary grows to 1,000 and then 50,000 $n$8-grams, semantic performance rises sharply. The paper gives a concrete example for word similarity: the average score rises from 6.2 with 100 bigrams to 69.5 with 50,000 and $n$9. For sentence similarity, the best setting is 50,000 with $n$0, yielding 66.6 average Pearson $n$1.

The qualitative analysis is used to explain why the model works despite its simplicity. The paper shows that charagram-phrase handles unknown words better than the word-based paragram-phrase model because it has no OOV problem. On STS data binned by number of unknown words, charagram-phrase consistently outperforms paragram-phrase, and the gap widens when unknown words are present. The same analysis reports robustness across sentence lengths, with charagram-phrase outperforming paragram-phrase at every length bucket.

Nearest-neighbor inspection indicates that Charagram captures several forms of similarity simultaneously: spelling variation, morphology, synonymy, and some order-sensitive composition such as negation. The paper gives the example that “not capable” maps near “incapable” and “unable,” whereas paragram-phrase tends to overemphasize the word “not.” Other neighbors include misspellings and repeated letters, morphological variants such as “journeying” near “journey,” and semantically related words such as “vehicles,” “cars,” and “automobiles.” The paper also reports that nearest neighbors of learned $n$2-gram embeddings themselves form clusters with semantic themes such as death, food, speed, or language. This suggests that individual character $n$3-gram vectors can acquire meaningful latent semantics, although the model contains no explicit symbolic decomposition beyond the $n$4-gram inventory.

6. Significance, limitations, and relation to later sequence-based representations

The paper’s main message is that a sum of learned character $n$5-gram embeddings plus a nonlinearity can serve as a strong baseline for subword-aware representation learning (Wieting et al., 2016). Its strengths are simplicity, fast convergence, strong handling of rare and OOV words, and strong performance on semantic similarity. Its limitations are also explicit in the results: it depends on a sufficiently rich $n$6-gram vocabulary for semantic tasks, and on POS tagging it is competitive rather than clearly superior.

This model family is conceptually relevant beyond conventional lexical and sentence embedding. A later graph-representation paper, "Instruction set for the representation of graphs" (Lopez-Rubio et al., 11 Mar 2026), describes its string-based graph encoding as being closer in spirit to SMILES, SELFIES, or “Charagram-like” character-level composition ideas than to adjacency-matrix encodings. That paper does not discuss Charagram directly, but it identifies a similar philosophy: represent structured data as a string over a fixed alphabet, rely on sequence models or string metrics, and exploit subsequence overlap and local character edits. In that later framing, an IsalGraph string could be embedded by character $n$7-grams or character compositional pooling, allowing graph structure to be learned from raw instruction sequences without explicit graph parsing. This suggests that Charagram’s architectural principle is broader than lexical semantics alone: the same exact-match compositional mechanism can be interpreted as a generic interface between symbolic sequences and dense embeddings when a task benefits from small alphabets, local subsequence reuse, and OOV robustness.

A common misconception is that Charagram should be viewed merely as a lightweight approximation to recurrent or convolutional character models. The paper instead positions it as an alternative with its own strengths: exact $n$8-gram specificity, efficient optimization, and unexpectedly strong semantic behavior. Another potential misconception is that its simplicity implies low capacity. The ablation results argue against that reading: the model can be very weak with a small $n$9-gram inventory, but performance rises sharply when the character vocabulary is sufficiently rich. In that sense, Charagram is simple in architecture rather than necessarily small in representational resources.