FSA-Driven Constrained Decoding

• FSA-driven constrained decoding is a framework that restricts output generation to valid sequences defined by finite-state automata, ensuring structured adherence to lexical and syntactic rules.
• It integrates multi-stack beam search and shortest-string decoding algorithms over idempotent and non-idempotent semirings for optimal performance.
• Applications span neural machine translation, speech recognition, and schema-constrained language modeling with notable speedups and accuracy improvements.

A finite-state automaton (FSA)-driven constrained decoding framework enforces structured output generation by restricting hypotheses to those corresponding to valid paths through automata representing regular languages or other constraint families. Such approaches generalize well to neural, statistical, and hybrid models in tasks that require adherence to lexical, syntactic, schema, or rate constraints. In current state-of-the-art systems, FSA-driven decoding is crucial for applications including neural machine translation, language modeling with formal safety or shape guarantees, speech recognition with structured emission controls, and high-performance parallelization of weighted FST algorithms.

1. Formalism: FSAs, Semirings, and Constraint Encoding

The FSA, typically formalized as $A=(Q, \Sigma, \delta, q_0, F)$, represents a regular language or constraint set. $Q$ is a finite state set, $\Sigma$ the output alphabet (token or subword), $\delta$ a transition function, $q_0$ the initial state, and $F$ the accepting states. Transitions may be unweighted (acceptors) or carry weights from a semiring $(K, \oplus, \otimes, \mathbf{0}, \mathbf{1})$, supporting scored decoding—most critically for WFSA (weighted FSA) applications.

The semiring choice fundamentally affects both representational power and decoding algorithm applicability:

• Idempotent Semirings: Standard shortest-path algorithms (e.g., Dijkstra, A*) operate over semirings where $a \oplus a = a$ (e.g., tropical, max-plus). This guarantees a globally optimal single path corresponds to the best string.
• Non-idempotent (Monotonic Negative) Semirings: In plus-times $(a \oplus b = a + b, a \otimes b = ab)$ or log semirings, summing multiple path weights has cumulative effect. The notion of a "shortest path" is often undefined because two different paths' cumulative sum may be lower than either alone. Nevertheless, the "shortest string"—minimizing the sum over all accepting paths spelling a given output—remains well-defined and can be identified using backward distance heuristics and companion idempotent semirings (Gorman et al., 2022).

Constraints for decoding are encoded directly as FSAs (or FSTs for more complex supervision), frequently constructed to accept a language with required substrings (lexical constraints), valid schemas, or rate/length envelopes.

2. Core Decoding Algorithms and FSA Integration

2.1 Direct FSA Constrained Search

• Multi-stack Beam Search: Each FSA state defines a separate hypothesis stack. Extension of a partial hypothesis is allowed only when the appended token triggers a valid transition in the FSA. For $|Q|$ FSA states, one maintains up to $Q$0 hypotheses per time step ($Q$1 = per-stack beam size), although typical algorithms prune inactive stacks. Multi-word and synonym constraints expand the automaton but allow strict, explicit enforcement of complex user requirements (Hasler et al., 2018).
• FSA Masking with Determinized Product Automata: When dealing with tokenization mismatch (character-level constraints vs. subword output), the regular constraint automaton $Q$2 is composed with a de-tokenizer transducer $Q$3 to produce a token-level FSA. Masked decoding is then realized by tracking composite automaton states and applying the mask $Q$4 at every decoding step (Koo et al., 2024).

2.2 Weighted FSA Shortest-String Decoding

When the decoding problem is formulated in non-idempotent semirings, a variant of A* search is deployed over the deterministic companion semiring, using backward distance in the determinized automaton as a heuristic.

Pseudocode:

$Q$8

This is both sound and optimal under mild constraints; only a fraction of the determinized automaton need be traversed if on-the-fly determinization and tight admissible heuristics are employed (Gorman et al., 2022).

3. Parallelization and Hardware Acceleration

Efficient FSA-driven decoding requires scalable implementations:

• GPU Algorithms: Parallelization of weighted FSA/FST algorithms (Viterbi, forward-backward) exploits CUDA architectures by mapping traversal over transition structures in CSR/COO memory layouts and using atomic reductions for max-scores or log-sum-exp, with backpointer support for exact path reconstruction (Argueta et al., 2017).
• Constraint Composition: FSAs representing user or structural constraints are composed with model graphs either pre-runtime (host-side) for static constraints or lazily (GPU-side) for small, dynamically-specified constraints. Traversal operates directly over the cross-product state space, filtering transitions on-the-fly.

Parallel batched beam search is enabled by representing composite automaton state (including possible constraint state counters) with compact integer codes, facilitating on-the-fly pruning and top-k hypothesis selection.

• Practical Speedups: Proper data layout, memory coalescing, and algorithmic optimizations yield speedups of 5x to 6000x over serial and generic FSA toolkits for large-scale FSTs (Argueta et al., 2017).

4. Applications and Constraint Typologies

Application Domain	Constraint FSA Role	Key Technique
Neural Machine Translation (NMT)	Terminology phrase insertion	Multi-stack FSA decoding with attention-guided pruning (Hasler et al., 2018)
LLM Decoding	Syntax/schema enforcement	Composition of regex or JSON schema FSAs, detokenizer FST, mask-and-renormalize (Koo et al., 2024)
Non-Autoregressive Generation	Lexical/vocab/length control	WFSA intersection, DFS-Viterbi over constrained WFSA (Chen et al., 2024)
Speech Recognition (Transducer)	Emission count upper-bounds	FSA for per-frame emission control, GPU parallel beam search (Kang et al., 2022)
Weighted Decoding over LM scores	Non-idempotent semirings	A* with companion idempotent heuristic (Gorman et al., 2022)

Constraint automata accept languages defined by required substrings, token sets, regular expressions or more general context-free constructs; in practical settings, this includes application to schema-constrained JSON/YAML, API call signatures, and morphological acceptors in ASR.

5. Complexity and Scalability Considerations

• Automaton Size and State Explosion: The number of active states is typically $Q$5 for $Q$6 constraints but may be mitigated by (a) using attention to activate a single constraint at a time, reducing per-step complexity to $Q$7, and (b) dynamic pruning heuristics in automaton construction and traversal (live-state pruning, constraint grouping) (Hasler et al., 2018, Koo et al., 2024).
• Compile Time and Per-Step Overhead: FSA composition in optimized C++ (OpenFST) achieves dramatic reductions in compilation time for constraint automata compared to Python-level surrogate mapping approaches (milliseconds vs. seconds). Per-step mask construction is linear in the output degree of the automaton state, typically negligible relative to model computation (Koo et al., 2024).
• Weighted Decoding in Non-Idempotent Semirings: Determinization of acyclic NFAs may have exponential blow-up, but on-the-fly construction combined with tight heuristics drastically limits visited state space (Gorman et al., 2022).

6. Empirical Performance and Trade-Offs

Experimental results across domains demonstrate:

• Neural Machine Translation: Strictly constrained outputs (zero constraint violations) are achieved without degradation of BLEU (or improved BLEU with proper constraint design). Attention-guided FSA activation reduces computational overhead and repetition errors (Hasler et al., 2018).
• Non-Autoregressive Generation: Slot and OOV errors are eliminated, with small sacrifices in generation quality; latency remains well below autoregressive baselines due to FSA-based batching and WFSA algorithmic efficiencies (Chen et al., 2024).
• Speech Recognition: Enforcing emission limits per frame via FSA composition yields speedups of 3–18× (depending on hardware and configuration), with negligible increase in WER (Kang et al., 2022).
• Schema-constrained LM Decoding: Automata-based constraint mechanisms support millisecond-scale compilation and per-step overhead orders of magnitude below Python-level masking, making practical at-scale schema enforcement feasible (Koo et al., 2024).

7. Limitations and Future Directions

• Scalability: Constraints with many alternatives or very large vocabularies can cause state explosion; approximate intersection, masking via precomputed bitmaps, and on-the-fly composition are active areas to improve scalability.
• Extensibility: Automata-theoretic reformulations permit modular constraint replacement (new regex, tokenization schemas) and extension to context-free or pushdown constraints for more complex languages (Koo et al., 2024).
• Integration with Model Training: There is ongoing work exploring joint training of models with constraints, potentially with soft penalties or differentiable approximations to FSA acceptors (Chen et al., 2024).
• Deployment Efficiency: The gap between Python-based FSA libraries and C++ implementations (e.g., Pynini vs OpenFST, k2) points to engineering as a critical research direction to maximize real-time performance in large-scale or production environments (Kang et al., 2022, Chen et al., 2024).

FSA-driven constrained decoding frameworks constitute the most general, reliable, and efficient approach for controlled language generation and structured sequence modeling, with mathematical guarantees of constraint satisfaction and broad applicability across domains (Gorman et al., 2022, Hasler et al., 2018, Argueta et al., 2017, Koo et al., 2024, Chen et al., 2024, Kang et al., 2022).