---
title: 'Deco-G: Decoupling Task Solving & Formatting'
url: https://www.emergentmind.com/topics/deco-g
type: topic
---

# Deco-G: Decoupling Task Solving & Formatting

Searching arXiv for the Deco-G paper and closely related references.
Deco-G is a decoding framework for large language models that explicitly decouples task solving from output-format adherence. It was introduced in “Decoupling Task-Solving and Output Formatting in LLM Generation” and is motivated by the observation that instructive prompts often intertwine reasoning directives with rigid formatting requirements, creating competing goals for the model. Deco-G assigns task solving to the base instruction-tuned LLM and assigns format compliance to a separate tractable probabilistic model (TPM), combining their token-level probabilities at decoding time. The reported result is 1.0% to 6.0% relative gain over regular prompting practice with guaranteed format compliance across mathematical reasoning, LLM-as-a-Judge, and event argument extraction [2510.03595].

## 1. Problem setting and motivation

Deco-G is designed for settings in which an LLM must both solve a task and present the solution in a rigid target format. The motivating claim is that, as prompts grow more complex, models often struggle to adhere to all instructions, and that this difficulty is especially common when prompts mix directives about what the model should solve with directives about how the output must be presented [2510.03595].

The underlying rationale is stated in terms of goal interference. Empirical studies cited in the description—Long et al. 2025, Tam et al. 2024, and He et al. 2024—are described as showing that prompts asking LLMs both to “think step by step” and to “output exactly in this rigid template” can degrade performance on the underlying reasoning task. The explanation given is that the model’s capacity is split between following the reasoning directive and satisfying format constraints, which can conflict; the description notes, for example, that requiring exact tokens or JSON structure can interrupt chain-of-thought.

Deco-G addresses this by assigning each objective to a specialized component. The LLM receives only task instructions, while format compliance is enforced externally at decode time by the TPM. This separation is the central design principle: the framework treats reasoning competence and format compliance as distinct objectives rather than as co-equal prompt instructions.

## 2. Probabilistic formulation

The formal setting views generation as sampling a token sequence $y_{1:n}$ given an input $x$ and a desired output format $\alpha$ [2510.03595]. The base model provides a next-token distribution
$$
P_{\mathrm{LLM}}(y_t \mid y_{<t}, x),
$$
and the tractable probabilistic model provides an estimate
$$
P_{\mathrm{TPM}}(\alpha \mid y_{<t}, y_t),
$$
interpreted as the probability that the completed sequence will satisfy the format $\alpha$ after appending candidate token $y_t$.

Deco-G decodes from the posterior
$$
P_{\mathrm{Deco\mhyphen G}}\bigl(y_t \mid y_{<t}, x, \alpha\bigr)
\;\propto\;
P_{\mathrm{LLM}}\bigl(y_t \mid y_{<t}, x\bigr)\,
\bigl[P_{\mathrm{TPM}}(\alpha \mid y_{<t}, y_t)\bigr]^{\gamma},
$$
where $\gamma \ge 1$ is a control-strength hyperparameter and the default is $\gamma=1$.

For the special case $\gamma=1$, the framework is described as corresponding directly to a Bayes-style factorization. In implementation terms, this means that Deco-G performs element-wise fusion of two token-level signals: an LLM distribution specialized for task solving and a format-likelihood distribution specialized for compliance. The description presents this as a principled Bayesian-style fusion of two specialized modules rather than a modification of the underlying LLM weights.

A common misconception is that rigid output requirements are best handled by adding more prompt instructions. The reported formulation implies the opposite design choice: remove format instructions from the LLM prompt and impose them through a separate probabilistic controller at inference time.

## 3. Tractable probabilistic model and instruction-aware distillation

The TPM in Deco-G is implemented as a Hidden Markov Model (HMM) with hidden-state sequence $z_{1:n}$. Its joint likelihood is given by
$$
P_{\mathrm{HMM}}(y_{1:n}, z_{1:n})
= P(z_1)\,P(y_1 \mid z_1)\,\prod_{t=2}^n P(z_t \mid z_{t-1})\,P(y_t \mid z_t).
$$

A key innovation is instruction-aware distillation. Instead of fitting the HMM to unconditional text, the procedure samples one million continuations from the instruction-tuned LLM on natural-instruction prompts from Natural-Instructions-v2, and then applies the Baum–Welch algorithm (EM) to maximize
$$
\mathcal{L}_{\mathrm{HMM}}
=
\sum_{(x,y_{1:n}) \in \mathrm{data}}
\log\,P_{\mathrm{HMM}}(y_{1:n} \mid x).
$$
The description states that this is equivalent to minimizing $\mathrm{KL}(P_{\mathrm{LLM}} \| P_{\mathrm{HMM}})$ under the instruction-conditioned distribution [2510.03595].

The stated purpose of this distillation scheme is to make the HMM better match the LLM’s conditional, prompted behavior. This is significant because the TPM is not treated as a generic language model; it is trained to approximate the format-relevant behavior of an instruction-tuned model under task prompts. A plausible implication is that the TPM is intended to remain lightweight and tractable while still being aligned with the distributional regime in which the LLM is actually used.

## 4. Format compilation and constrained decoding

Deco-G handles many real-world templates that mix fixed pivots with wildcards. The description defines pivots as literal tokens such as “The final answer is” and wildcards as slots with allowed lengths between $\min$ and $\max$. The target specification $\alpha$ is compiled into a deterministic finite automaton (DFA) via a trie-building algorithm [2510.03595].

The provided `BuildFormatDFA` pseudocode takes a list of template segments
$$
S = [P_1, W_1, P_2, \ldots, P_k, W_k],
$$
where each $P_i$ is a fixed pivot token sequence and each $W_i$ is a wildcard with an allowed length interval $[\min_i,\max_i]$. The algorithm creates a root state, expands pivot segments token by token, adds wildcard transitions for lengths $1 \ldots \max$, marks states at lengths greater than or equal to $\min$ as wildcard-complete, and marks final states as accepting.

Once the DFA for $\alpha$ has been built, Deco-G runs the standard forward–backward algorithm on the product HMM $\times$ DFA to compute
$$
P_{\mathrm{TPM}}(\alpha \mid y_{<t}, y_t)
=
\frac{P_{\mathrm{HMM\times DFA}}\bigl(\text{accepted},\, y_{<t}, y_t\bigr)}
{P_{\mathrm{HMM}}(y_{<t}, y_t)}.
$$

This construction is what underwrites guaranteed format compliance. The guarantee is not attributed to prompt engineering or post hoc repair, but to token-by-token decoding under a format-acceptance model defined over the product of a probabilistic sequence model and a finite-state representation of the target format. In effect, format validity becomes part of the decoding distribution itself.

## 5. Architecture, hyperparameters, and efficiency

Deco-G is presented as a decoding wrapper around any off-the-shelf instruction-tuned LLM, including examples such as Llama-3 and Qwen-Instruct [2510.03595]. The prompting protocol is explicit: the original prompt $x$ is fed to the LLM with any format instructions omitted, so the model need only solve the task. In parallel, a Format Estimation Module (FEM) takes the desired format specification $\alpha$, compiles it to a DFA plus HMM, and returns $P_{\mathrm{TPM}}(\alpha \mid y_{<t}, y_t)$ at each step.

The decoding loop, whether greedy or sampling-based, multiplies the LLM probability vector and the FEM probability vector element-wise, applies the $\gamma$ exponentiation to the format term, renormalizes, and selects the next token. The reported hyperparameters are HMM hidden size $h=4096$, vocabulary size 128 K–152 K, pruning $k=200$, and steering strength $\gamma \in \{1,2\}$.

A central computational issue is the cost of HMM emission over a full vocabulary $\mathcal V$. The description states a per-step cost of $O(h\cdot |\mathcal V|)$ when $h=4096$ and $|\mathcal V| \approx 128\mathrm{k}$. The observation motivating pruning is that the posterior over the $h$ hidden states is very concentrated at each step. The proposed strategy keeps only the top $k$ states by posterior mass, renormalizes, and computes emissions only from them.

The complexity is reported as
- before pruning: $O(h\log h + h\,|\mathcal V|)$
- after pruning: $O(h\log h + k\,|\mathcal V|)$, with $k \ll h$

The empirical efficiency figures are specific. With $k=200$, corresponding to the top 5%, the retained mass exceeds 98%; the cost drops from approximately 1.08 GFLOPs to approximately 0.08 GFLOPs per step; and task accuracy degrades by at most 1%. At the system level, HMM distillation on 1 M examples takes approximately 56 GPU hr for sampling plus 1 GPU hr for training, and inference overhead is approximately 0.5% extra FLOPs versus the LLM.

## 6. Empirical evaluation across tasks

The reported evaluation spans mathematical reasoning, LLM-as-a-Judge, and generative event argument extraction, with the general claim that gains range from 1% to 6% relative over the best baseline and that format compliance is guaranteed [2510.03595].

### Mathematical reasoning (GSM8k)

The evaluated models are Llama-3.1-8B-Instruct, Qwen2.5-7B-Instruct, and Qwen3-8B. The metrics are format compliance (%) and answer accuracy (%).

For Llama-3.1-8B-Instruct, the reported numbers are:
- NL (free): 96.3% format, 82.3% accuracy
- JSON-S (structured): 100%, 75.7%
- Deco-G: 100% compliance, 85.2% accuracy, approximately +3.4% relative

For Qwen2.5-7B-Instruct, Deco-G with $\gamma=2$ yields 100% format compliance and 88.6% accuracy, compared with JSON-S at 79.0% accuracy. For Qwen3-8B, Deco-G with $\gamma=2$ yields 100% and 91.7%, compared with JSON-S at 90.6%.

These results directly address the central motivation. In the reported GSM8k setting, the structured prompt baseline achieves perfect formatting but can reduce answer accuracy relative to free-form prompting, whereas Deco-G preserves perfect compliance and improves task performance.

### LLM-as-a-Judge (SummEval)

The task is to rate summaries on Coherence, Consistency, Fluency, and Relevance. The evaluation metrics are Spearman $\rho$ and Kendall $\tau$ correlation with human judgments.

The reported average gain is +0.5–1.5% absolute correlation, and Deco-G is described as consistently best on 3/4 dimensions across models. The significance of this result is methodological: the framework is not limited to symbolic or extraction-style outputs, but also applies to evaluation settings in which the output format constrains rating structure.

### Generative event argument extraction (ACE05)

The metrics are F1 on AI, AC, AI+, and AC+. The description glosses these as argument-ID, class, and related variants.

For Llama, AI improves from 36.8 to 39.4 (+2.6 pt) and AC from 27.3 to 28.7 (+1.4). For Qwen2.5, AI improves from 32.6 to 35.2 (+2.6), with the description indicating further gains on related metrics. The reported aggregate characterization is that gains range from 1% to 6% relative over the best baseline.

## 7. Interpretation, scope, and related misconceptions

Deco-G should be understood as an inference-time framework rather than a new end-to-end architecture for task learning. It does not replace the instruction-tuned LLM, nor does it rely on prompt-only enforcement of schema constraints. Instead, it composes an LLM for reasoning with an HMM plus DFA for format compliance, and the composition occurs at each decoding step [2510.03595].

One common misconception is that guaranteed formatting necessarily requires sacrificing task quality. The reported results do not support that conclusion. On GSM8k, structured prompting with JSON-S reaches 100% format compliance but can reduce answer accuracy, while Deco-G reaches 100% compliance together with higher accuracy than both JSON-S and free-form NL in the reported Llama configuration. Another misconception is that format adherence is simply a matter of stronger instruction following. The framework is explicitly built on the premise that reasoning directives and formatting directives can become competing goals when placed in the same prompt.

The broader significance lies in specialization. The LLM is asked only to solve the task; the TPM is asked only to assess compliance with $\alpha$. This suggests a modular view of generation in which semantic problem solving and syntactic or structural validity are separable control axes. A plausible implication is that such modularity may be especially useful when output templates contain fixed pivots, bounded wildcards, or schema-like constraints that can be represented as finite-state structure.

In summary, Deco-G is a constrained decoding framework that combines an instruction-tuned LLM with an instruction-aware distilled HMM and a DFA-based format model. Its defining properties are explicit decoupling of task solving and output formatting, token-level posterior fusion with steering parameter $\gamma$, efficient hidden-state pruning, and guaranteed format compliance together with reported relative gains of 1.0% to 6.0% over regular prompting practice.

Source: https://www.emergentmind.com/topics/deco-g