Multiround Adaptive CoT Compression (MACC)

• The paper introduces MACC, a framework that progressively compresses chain-of-thought reasoning traces to reduce token count and inference latency.
• It employs a token elasticity phenomenon with a stop-on-regression rule to adaptively determine the optimal compression depth while managing accuracy trade-offs.
• Empirical evaluations on tasks like GSM8K, MATH-500, and AIME24 show that MACC achieves significant token reductions and efficiency gains compared to baseline methods.

Searching arXiv for the cited MACC and closely related CoT compression papers to ground the article. Multiround Adaptive Chain-of-Thought Compression (MACC) is a framework for shortening chain-of-thought (CoT) reasoning traces through progressive multiround refinement rather than one-shot pruning. It is introduced for the setting in which long CoTs improve reasoning performance but also increase attention cost, KV-cache growth, and autoregressive latency. The defining idea is to exploit a token elasticity phenomenon: when the prompt-specified token budget is pushed too aggressively, actual output length can rebound rather than continue to shrink. MACC therefore compresses CoTs iteratively, stopping when further shortening ceases to produce a shorter trace, and then fine-tunes the target model on both original and compressed rationales with a dedicated compression token (Yan et al., 26 Sep 2025).

1. Conceptual basis and problem setting

MACC is motivated by the observation that long CoTs are simultaneously useful and inefficient. The method is framed around three concrete costs of verbose reasoning: longer sequence length increases attention cost, transformer attention scales quadratically with context length, KV-cache memory usage grows with the number of generated tokens, and latency rises because the model must autoregressively decode more tokens. In test-time scaling settings, longer reasoning can improve accuracy, but the marginal gains come at a substantial computational price (Yan et al., 26 Sep 2025).

The framework departs from fixed-ratio or prompt-only shortening by treating compressibility as input-dependent. The broader CoT-compression literature provides a closely aligned rationale for this view. A systematic study of compressed reasoning prompts reports a universal tradeoff between reasoning length and accuracy and formalizes per-question “token complexity,” a minimal number of tokens required for successful problem-solving. In that formulation, correctness is governed by a threshold $\tau_i^\pi$ for question $i$ and model $\pi$, which makes uniform compression intrinsically brittle and highlights adaptive allocation of reasoning length as the central problem (Lee et al., 3 Mar 2025).

Within this landscape, MACC defines compression not as stylistic brevity but as controlled reduction of a reasoning trace while attempting to preserve the reasoning quality needed for downstream fine-tuning and inference. This places it between prompt-level concision methods and architectural memory-compression methods: it is neither a purely instruction-based shortening heuristic nor a permanent latent-memory replacement for all intermediate text.

2. Token elasticity and the multiround refinement rule

The key empirical observation in MACC is the token elasticity phenomenon. The paper describes a nonlinear budget-to-length relationship: moderate budget reduction yields shorter output, but excessive budget reduction can produce worse reasoning, more redundancy, or compensatory elaboration, so actual output length may increase again. This is the immediate reason for using multiround refinement rather than a single aggressive compression step (Yan et al., 26 Sep 2025).

The pipeline begins from an initial full CoT generated by the base model $\mathcal{S}$ from input question $x$ and prompt template $\mathcal{P}$:

$$r_0 = \mathcal{S}(x \mid \mathcal{P},\, \theta_{\mathcal{S}(D_{\text{train}})})$$

A separate compressor model $\mathcal{C}$ then produces progressively compressed traces:

$$r_i = f_i(r_{i-1} \mid \mathcal{P}_{\text{compress}}), \quad i=1,2,\ldots,N$$

The compression prompt instructs the compressor to read the question, read the original thought process, keep only the key reasoning needed to solve the problem, and avoid adding new information.

MACC does not fix compression depth in advance. Instead, it uses an instance-wise stopping rule based on token length. Compression continues only while the next round is strictly shorter than the previous round. The selected trace is

$$r^* = \arg\min_{r_j} |r_j|_{\text{tok}} \quad \text{subject to} \quad |r_j|_{\text{tok}} < |r_{j-1}|_{\text{tok}}.$$

Operationally, the paper summarizes this as progressive, adaptive, and stop-on-regression compression. The last strictly shorter chain is treated as the “maximally compressed yet valid” chain for that example.

For reasoning models, the authors also note that the chain is disentangled into a reasoning process and an answer process, each is compressed separately, and the results are concatenated. This suggests that MACC treats compression as a structure-preserving transformation over reasoning components, not merely as global truncation.

3. Training objective, data construction, and inference mode

After obtaining the compressed rationale $i$0, the target model is fine-tuned on both original and compressed CoTs. The model is taught to enter compressed-reasoning mode through a special token, written as <compress> or $i$1. Training examples are formatted as

$i$2

and optimized with the standard autoregressive objective

$i$3

The training mix also includes some original CoTs without the compression token so that the model does not lose its ability to reason normally (Yan et al., 26 Sep 2025).

The paper’s Algorithm 1 gives the operational procedure. For each training example $i$4, the system generates an original CoT $i$5, initializes $i$6, iteratively compresses up to a maximum round depth $i$7, stops if $i$8, otherwise updates $i$9, and stores $\pi$0 in the compressed training set. At inference time, the fine-tuned model is prompted with the compression token and generates a shorter reasoning trace plus the final answer autoregressively.

This training-and-inference design is narrower than a general online controller over arbitrary reasoning states. The adaptive component lies in multiround preprocessing and in the learned compressed-reasoning mode, whereas the stop rule itself is token-length based rather than a semantic verifier.

4. Empirical evaluation and reported performance

MACC is evaluated on GSM8K, MATH-500, and AIME24. The reported model families are LLaMA-3.1-8B-Instruct, Qwen2.5-3B-Instruct, Qwen2.5-7B-Instruct, DeepSeek-R1-distill-Qwen-1.5B, and DeepSeek-R1-distill-Qwen-7B. The main compressors studied include GPT-4o-mini, GPT-3.5-turbo, GPT-4.1-mini, GPT-4.1-nano, and GPT-4o. Baselines differ by backbone family: for the LLaMA/Qwen experiments they are Original, Prompt, TokenSkip, and TALE; for the DeepSeek-R1 experiments they are Original, O1-Pruner, TALE, and CoT-Valve. The reported metrics are Accuracy, Average CoT token count, Inference latency, and Token Efficiency, where

$\pi$1

The paper’s headline summary is an average accuracy improvement of 5.6 percent over state-of-the-art baselines, a 47-token average CoT length reduction, and significant latency reduction (Yan et al., 26 Sep 2025).

The detailed tables show that the comparison depends strongly on which baseline is used. Against the uncompressed Original setting, MACC often sacrifices some absolute accuracy in exchange for shorter CoTs and lower latency. For LLaMA-3.1-8B on GSM8K, Original reports 86.2 accuracy, 213.17 tokens, and 1.33 s, whereas MACC reports 81.1 accuracy, 88.57 tokens, and 0.75 s. On MATH-500 for the same model, Original reports 48.6 accuracy, 502.60 tokens, and 6.83 s, while MACC reports 44.0 accuracy, 198.04 tokens, and 2.05 s. For Qwen2.5-7B, Original on GSM8K is 91.4 accuracy, 297.83 tokens, and 1.96 s, while MACC is 86.2 accuracy, 148.76 tokens, and 0.87 s; on MATH-500, Original is 71.4 accuracy, 574.85 tokens, and 6.65 s, while MACC is 58.4 accuracy, 254.89 tokens, and 2.02 s (Yan et al., 26 Sep 2025).

Against compression baselines, the paper reports a more favorable accuracy-efficiency tradeoff. On LLaMA-3.1-8B / GSM8K, MACC achieves 81.1 accuracy versus 78.2 for TokenSkip, while using fewer tokens and lower latency. On LLaMA-3.1-8B / MATH-500, MACC achieves 44.0 versus 40.2 for TokenSkip, again with shorter CoTs and lower latency. Prompt-only shortening is reported as the weakest overall baseline, often sacrificing accuracy without competitive efficiency (Yan et al., 26 Sep 2025).

The DeepSeek-R1 results show that the method can also remain competitive on more difficult settings while cutting tokens substantially. For R1-Qwen-1.5B on GSM8K, Original reports 79.0 accuracy and 978 tokens, while MACC reports 79.3 accuracy and 471 tokens. For R1-Qwen-7B on GSM8K, Original reports 87.9 accuracy and 682 tokens, while MACC reports 90.1 accuracy and 361 tokens. On OOD tasks, the pattern shifts toward reduced token counts with some accuracy loss: for R1-Qwen-7B on MATH-500 OOD, Original is 90.2 accuracy and 3674 tokens, while MACC is 86.8 accuracy and 2039 tokens; on AIME24 OOD, Original is 53.5 accuracy and 10306 tokens, while MACC is 50.0 accuracy and 6144 tokens (Yan et al., 26 Sep 2025).

The round-by-round analysis reports that CoT length steadily decreases with more rounds, accuracy generally declines if compression becomes too aggressive, larger models are more robust to deep compression, and smaller models are more sensitive. This is one of the main empirical justifications for the stop rule.

5. Compressor selection, predictive modeling, and stated limitations

Beyond the compression procedure itself, MACC advances a “Performance Estimation Hypothesis” for forecasting the downstream effects of compression before repeated fine-tuning. The feature set includes compression rate, perplexity of the compressed CoT, original CoT length, accuracy of the original model or compressor on the training set, and related compression statistics. The target variables are downstream test-time accuracy and length, denoted $\pi$2, and the paper formalizes the prediction problem as

$\pi$3

The estimator is implemented with Bayesian regression, especially Bayesian Ridge Regression (Yan et al., 26 Sep 2025).

The reported held-out $\pi$4 values are high for both accuracy and length. For LLaMA3.1-8B, the paper reports $\pi$5 for accuracy and $\pi$6 for length. For Qwen2.5-7B, it reports $\pi$7 for accuracy and $\pi$8 for length. For Qwen2.5-3B, it reports $\pi$9 for accuracy and $\mathcal{S}$0 for length. Appendix correlations further indicate that compression rate correlates with compressed length, perplexity correlates negatively with accuracy, and original CoT length is often a strong predictor of final accuracy. Examples include, for LLaMA-3.1-8B, CR versus accuracy $\mathcal{S}$1, PPL versus accuracy $\mathcal{S}$2, and Len versus accuracy $\mathcal{S}$3; for Qwen2.5-7B, PPL versus accuracy $\mathcal{S}$4 and Len versus accuracy $\mathcal{S}$5.

The paper also studies compressor choice directly. Higher-capacity compressors such as GPT-4o and GPT-4o-mini preserve reasoning better, lower-capacity compressors such as GPT-3.5-turbo degrade accuracy more quickly, and the quality of the compressed rationale depends on compressor–student compatibility, not just on length reduction. The estimation module is presented as a complement to MACC: MACC decides how far to compress a given CoT, while the estimator helps decide which compressor or compression setting to use before expensive fine-tuning.

Several limitations are explicitly noted. MACC relies on API-based compressors such as GPT-4o-mini, which can introduce bias and reduce accessibility in low-resource settings. The multiround compression pipeline adds preprocessing latency before inference. The compression prompt is generic and may be suboptimal for domains with highly structured reasoning. The performance estimator may not generalize well to unseen model-task combinations (Yan et al., 26 Sep 2025).

6. Relation to precursor and neighboring methods

MACC sits within a broader family of CoT-compression methods, but its mechanism differs from both earlier precursors and later alternatives. A strong conceptual precursor is “Markov Chain of Thought” (MCoT), which reinterprets long mathematical CoT as a sequence of compressed state transitions: each derivation step produces a reduced question that becomes the next state, and the model can clear the KV cache after simplification. MCoT formalizes the reasoning step as depending only on the current question state, uses REACT-style <Text, [Code](https://www.emergentmind.com/topics/karpathy-agent-code), Observation> steps with a code interpreter for self-correction, and reports shorter derivation-stage prompts with favorable efficiency relative to standard multi-step reasoning. However, it is formulated for mathematical reasoning with stepwise reduced questions rather than as a general multiround adaptive compression framework under the MACC name (Yang et al., 2024).

A contrasting line is ConMax, which addresses CoT verbosity through a single-stage reinforcement-learning compressor. ConMax trains a compression policy $\mathcal{S}$6 to rewrite verbose traces into shorter traces by maximizing a reward that combines Answer Confidence and Thinking Confidence under a frozen auxiliary LRM. It reports, for Qwen2.5-7B, a reduction in average token length from 8,603 to 4,906, a 43.0% compression rate, and a 0.7-point average accuracy loss relative to the original verbose baseline. Yet its mechanism is explicitly not multiround and not adaptive at inference time: it is a one-shot offline compression policy rather than an iterative stop-on-regression procedure (Hu et al., 8 Jan 2026).

A more recent multiround relative is HybridThinker, which compresses each completed thought step into memory tokens but does not immediately discard the raw step. Instead, it temporarily retains raw thought-step KV caches for a sliding window of future rounds, while permanently retaining compressed memory tokens. To prevent the model from bypassing the compression pathway during training, HybridThinker introduces a hybrid attention scheme that mixes bottleneck-only and shortcut-access steps within the same sequence. Across GSM8K, MMLU, GPQA, and BBH, it reports that on Qwen2.5-7B the method matches the uncompressed baseline at 68.78 average accuracy and improves over LightThinker by 5.8 points, while reducing peak token usage by 61.7% versus Vanilla and reducing inference time by 20.3% (Liu et al., 2 Jun 2026).

These comparisons clarify what MACC is and is not. It is not merely prompt-only shortening, since the prompt baselines are explicitly separated and generally weaker. It is not a one-shot compressor like ConMax. It is also not a memory-token architecture in which raw thought steps are converted into persistent compressed states and then immediately evicted. Its distinguishing feature is progressive compression with an input-specific stopping criterion grounded in token-length behavior across rounds.

7. Interpretation and significance

MACC reframes CoT compression as an adaptive refinement problem. The method’s principal claim is that reasoning traces should be compressed only to the point at which they remain semantically stable enough to support downstream learning, and that over-compression can often be detected through token-length regression across compression rounds rather than by a fixed target ratio (Yan et al., 26 Sep 2025).

In relation to the token-complexity literature, this suggests a practical mechanism for approaching the per-question allocation problem: easier questions can terminate after shallow compression, while harder questions may admit only limited shortening before accuracy deteriorates. A plausible implication is that MACC operationalizes a coarse, data-driven proxy for question-specific token complexity without requiring explicit estimation of a threshold $\mathcal{S}$7 for each question (Lee et al., 3 Mar 2025).

The method’s significance therefore lies less in lossless compression than in structured tradeoff management. The empirical record presented in the paper does not support the claim that compressed reasoning universally dominates the original uncompressed CoT in raw accuracy. It does, however, support the narrower conclusion that multiround adaptive compression can substantially reduce CoT length and latency, outperform several compression baselines on the accuracy-efficiency frontier, and provide enough regularity for downstream performance to be forecast from interpretable training-set statistics (Yan et al., 26 Sep 2025).