TRIM: Token Reduction and Inference Modeling

• TRIM is a two-stage generative LLM wrapper that removes redundant function words via distillation before restoring the complete narrative using a reconstruction model.
• It achieves an average token saving of 20.58% by selectively omitting easily inferrable words, balancing efficiency with minimal degradation in evaluation metrics.
• The method leverages prompt engineering and cost-effective architectures, making it effective for fact-dense tasks like general-knowledge question answering.

TRIM, short for Token Reduction and Inference Modeling, is a two-stage wrapper for generative LLMs that seeks to reduce inference cost by exploiting redundancy in natural language (Ruiz et al., 2024). The method first prompts a generator to produce a shorter distilled answer in which a pre-selected set of easily inferrable function words is omitted, and then uses a much smaller reconstruction model to reinsert the missing terms and recover a full narrative. In the reported question-answering experiments, TRIM achieved an average token saving of 20.58% with only a small decrease in automated evaluation metrics, indicating a trade-off between token efficiency and reconstruction fidelity that is especially relevant for long-output generation (Ruiz et al., 2024).

1. Conceptual basis and formal definition

TRIM is defined as a two-stage pipeline around any generative LLM, whether closed or open (Ruiz et al., 2024). In the first stage, called Distillation, a generation model $M_G$ receives a user query $Q$ together with an omission instruction prompt $\mathcal{P}(S)$, where $S$ is a pre-selected set of easily inferrable function words such as “the, a, an, and, but…”. The model returns a distilled answer $A_D$ that omits the terms in $S$. In the second stage, called Reconstruction, a smaller encoder–decoder model $M_R$ takes $A_D$ as input and reinserts the missing terms from $S$, producing the final answer $A$ (Ruiz et al., 2024).

The pipeline is written as

$Q$0

The central efficiency quantity is the token-saving ratio, defined from the token counts of the original answer $Q$1 and the distilled answer $Q$2:

$Q$3

This formulation makes TRIM distinctively output-centric: the computational objective is not to reduce the parameter count of the primary generator, but to reduce the number of high-cost output tokens it must emit (Ruiz et al., 2024). A plausible implication is that TRIM is most relevant when output-token pricing or decode-time latency dominates total inference cost.

2. Omission set design and prompt engineering

The omission set $Q$4 is not chosen ad hoc. TRIM begins from 127 English function words and ranks them by “ease of inference” using a masked-language-model score gap $Q$5 (Ruiz et al., 2024). The paper evaluates 12 levels of top-$Q$6 words, with $Q$7 ranging from 5 to 60 in steps of 5, by fine-tuning T5 at each level and plotting saved-token percentage against metric degradation. The selected operating point is $Q$8, corresponding to ≈18.85% saved tokens on Wikipedia, because beyond that level the metric drop-off accelerates (Ruiz et al., 2024).

Prompting is central to the method’s behavior. In the reported GPT-4o runs, the baseline prompt was:

“Respond just in a paragraph. {knowledge_question}”

The distillation prompt was:

“Respond just in a paragraph in a distilled way removing the redundant and semantic-irrelevant terms of the language such as: $Q$9. That is to say, omit the terms of the ideal answer that do not affect directly the semantic meaning and can be reconstructed later on based on the context. Return only the distilled answer, nothing else. {knowledge_question}” (Ruiz et al., 2024)

The expected output is a single short paragraph $\mathcal{P}(S)$0 with all $\mathcal{P}(S)$1-words removed except when the LLM judges them non-inferrable. Empirically, approximately 0.3 words per 22-word answer were retained in this exception category (Ruiz et al., 2024). This indicates that the first stage is not a purely mechanical deletion procedure; it is a prompted generative transformation that preserves local exceptions when omission would damage recoverability.

A common misconception is that token removal in TRIM is equivalent to naive compression. The paper’s procedure is narrower: it targets a specific class of semantic-irrelevant or readily reconstructable terms, rather than arbitrary shortening of content. This suggests that the method depends on a structured redundancy hypothesis, not merely on brevity.

3. Reconstruction architecture and training regime

The reconstruction stage is implemented with comparatively small sequence-to-sequence models trained on (distilled $\mathcal{P}(S)$2 original) pairs (Ruiz et al., 2024). The architectures evaluated are BART (139 M), T5-base (223 M), Qwen2.5-0.5B, and Qwen2.5-1.5B. The training objective is standard sequence-to-sequence cross-entropy, and the loss is teacher-forced cross-entropy over the full vocabulary (Ruiz et al., 2024).

The training configurations reported in the supplementary summary are as follows:

• BART: 5 epochs, learning rate $\mathcal{P}(S)$3, batch size 8, gradient accumulation 8, weight decay 0.01, FP16 enabled, train time 1.5 h.
• T5: 5 epochs, learning rate $\mathcal{P}(S)$4, batch size 8, gradient accumulation 16, weight decay 0.01, FP16 enabled, train time 3 h.
• Qwen2.5-0.5B: 3 epochs, learning rate $\mathcal{P}(S)$5, batch size 4, gradient accumulation 32, weight decay 0.10, FP16 enabled, train time 7.5 h.
• Qwen2.5-1.5B: 2.5 epochs, learning rate $\mathcal{P}(S)$6, batch size 4, gradient accumulation 8, weight decay 0.10, FP16 enabled, train time 17 h (Ruiz et al., 2024).

All experiments were run on AWS g4dn.xlarge (NVIDIA T4, 16 GB) (Ruiz et al., 2024). The reconstruction model is therefore explicitly designed to be much cheaper at inference than the primary generator. The architecture choice introduces an important systems-level separation: the expensive model handles semantic content generation under omission constraints, while the cheaper model handles grammatical restoration.

4. Experimental corpora and evaluation methodology

TRIM is evaluated with two datasets (Ruiz et al., 2024). The first is a Wikipedia dump, used in multiple roles: approximately 100 k paragraphs for algorithmic selection of $\mathcal{P}(S)$7, 100 k distilled–original pairs for training $\mathcal{P}(S)$8, 5 k pairs per level for validation, and 9 k test paragraphs. The second is DKA (Distilled Knowledge Answers), consisting of 9 k LLM-generated question–answer pairs with both original and distilled answers (Ruiz et al., 2024).

Reconstruction fidelity is measured with both word-specific and full-text metrics. The paper introduces $\mathcal{P}(S)$9-metrics, which first align only the positions of $S$0-words via Needleman–Wunsch, then compute precision, recall, and F1 over correct insertions. It also reports standard n-gram and semantic metrics: BLEU, METEOR, ROUGE-1/L, cosine similarity using BERT embeddings, and perplexity using GPT-2 (Ruiz et al., 2024).

This evaluation design reflects two distinct questions. The $S$1-metrics isolate the specific subproblem TRIM is meant to solve—recovering omitted function words—whereas BLEU, METEOR, ROUGE, cosine similarity, and perplexity assess whether the resulting outputs remain globally close to the originals. A plausible implication is that TRIM can appear strong on semantic metrics even when restoration errors remain locally nontrivial, which is why the dedicated $S$2-family is methodologically important.

5. Quantitative results

The function-word selection study shows that with level 5 (25 words), token savings reach ≈18.85% on Wikipedia, and that beyond this point the decline in BLEU and ROUGE becomes steeper (Ruiz et al., 2024). This operating point is therefore used as the main trade-off between savings and reconstruction quality.

On DKA, where the distilled answers save 20.58% of tokens, the reported reconstruction results are as follows (Ruiz et al., 2024):

Model	Params (B)	Key results
BART	0.139	$S$3-F1 88.55, BLEU 85.09, Perp. 19.35
T5	0.223	$S$4-F1 89.97, BLEU 87.95, Perp. 17.04
Qwen2.5-0.5	0.500	$S$5-F1 88.44, BLEU 86.76, Perp. 18.42
Qwen2.5-1.5	1.500	$S$6-F1 91.36, BLEU 88.96, Perp. 17.59

The full metric table reports:

• BART: $S$7-P 91.50, $S$8-R 86.18, $S$9-F1 88.55, BLEU 85.09, METEOR 93.69, ROUGE-1 95.99, ROUGE-L 94.95, Cosine 99.99, Perp. 19.35, Orig Perp. 14.58, Saved 20.58.
• T5: $A_D$0-P 93.35, $A_D$1-R 87.14, $A_D$2-F1 89.97, BLEU 87.95, METEOR 94.65, ROUGE-1 96.66, ROUGE-L 95.85, Cosine 100.00, Perp. 17.04, Orig Perp. 14.58, Saved 20.58.
• Qwen2.5-0.5: $A_D$3-P 88.28, $A_D$4-R 88.93, $A_D$5-F1 88.44, BLEU 86.76, METEOR 94.66, ROUGE-1 96.00, ROUGE-L 94.95, Cosine 99.99, Perp. 18.42, Orig Perp. 14.58, Saved 20.58.
• Qwen2.5-1.5: $A_D$6-P 90.94, $A_D$7-R 92.04, $A_D$8-F1 91.36, BLEU 88.96, METEOR 95.59, ROUGE-1 96.64, ROUGE-L 95.94, Cosine 99.99, Perp. 17.59, Orig Perp. 14.58, Saved 20.58 (Ruiz et al., 2024).

The paper’s interpretation is that $A_D$9-F1 in the ≈89–91% range indicates good recovery of omitted words, that BLEU, METEOR, and ROUGE remain within 5% of the original, that cosine similarity is effectively ≈100%, and that perplexity rises only modestly from 14.58 to roughly 17–19 (Ruiz et al., 2024). Within the tested configurations, Qwen2.5-1.5B gives the highest $S$0-F1 and BLEU, while T5 achieves the lowest perplexity.

6. Cost model, applicability, and limitations

TRIM is not defined by token savings alone; it is justified through an explicit cost inequality (Ruiz et al., 2024). Let $S$1 denote extra prompt size, $S$2 the number of tokens saved, $S$3 the size of the distilled answer, $S$4 the size of the reconstructed final answer, and $S$5 the input/output token costs for the generation model and reconstruction model. TRIM is cost-effective when

$S$6

Using GPT-4o pricing of \$S$710 per million input and output tokens gives $S$8 (Ruiz et al., 2024). In the reported question-answering setting, the extra prompt length is $S$9 words, while the token saving is $M_R$0 of a typical ≈120-word answer, which the paper argues is sufficient to outweigh the overhead (Ruiz et al., 2024). This directly addresses a common misconception: fewer generator output tokens do not automatically imply lower total cost, because prompt expansion and reconstruction also contribute to inference expense.

The paper identifies several domain and methodological limits (Ruiz et al., 2024). It states that TRIM works best in fact-dense, redundancy-rich contexts, such as general-knowledge question answering. It further suggests that the approach should apply to other Zipfian languages, while noting that morphologically rich tongues, including Turkish, may be more challenging. Another limitation is an LLM generation penalty: omitting grammar can degrade continuation, and the discussion recommends a paragraph-by-paragraph restore strategy. Evaluation is entirely automated, and a human study is left as future work. Proposed future directions include dynamic $M_R$1 per query, adaptive omission thresholds, and joint distillation–reconstruction training (Ruiz et al., 2024).

Taken together, these features place TRIM within a broader line of work on inference-time efficiency, but with a specific emphasis on linguistic redundancy as a recoverable resource. The method does not claim lossless compression in a formal sense; rather, it operationalizes a learned omission-and-restoration process whose empirical viability depends on the redundancy structure of the domain and the quality of the reconstruction model (Ruiz et al., 2024).