LaSyn: Latent POS for NMT

• LaSyn is a latent-variable NMT model that incorporates explicit target-side POS sequences to guide syntactic word generation.
• It decouples dependencies among latent variables, allowing per-token exhaustive marginalization and maintaining efficient decoding.
• Empirical results on IWSLT and WMT tasks demonstrate improved BLEU scores and translation diversity through explicit syntactic regularization.

LeaSyn, written as LaSyn in the paper “Latent Part-of-Speech Sequences for Neural Machine Translation,” is a latent-variable neural machine translation model that injects target-side syntactic structure—specifically part-of-speech sequences—into a Transformer-based translation system while allowing effective and efficient inference over the latent space. Its central modeling decision is to decouple direct dependence between successive latent variables, so that target-side latent syntax can be marginalized exactly at each decoding step, with decoding speed proportional to the size of the latent variable vocabulary rather than exponential in sequence length (Yang et al., 2019).

1. Problem setting and motivation

LaSyn is situated in the line of work that treats target-side syntax as beneficial for neural machine translation. A standard Transformer NMT model parameterizes

$$p(\mathbf{y}\mid \mathbf{x})=\prod_{t=1}^{T} p(y_t\mid y_{<t},\mathbf{x}),$$

where $\mathbf{x}$ is the source sentence and $\mathbf{y}=(y_1,\dots,y_T)$ is the target sentence. In this formulation, syntax and semantics are learned implicitly in a dense representation, without an explicit target-side syntactic variable. LaSyn introduces such a variable and uses it to represent POS-like structure on the target side (Yang et al., 2019).

The motivation is twofold. First, explicit syntax can support better fluency, grammaticality, and interpretability. Second, prior latent-syntax NMT systems typically rely on greedy or approximate search over latent structures, because exact marginalization over a full latent sequence is intractable when the latent space scales as $|\mathcal{Z}|^T$. This severely limits exploration of syntactic alternatives. LaSyn addresses that limitation by using per-token latent syntax variables and a factorization that removes direct tag-to-tag dependence. This suggests a deliberate trade-off: the model gives up an explicit latent Markov structure in exchange for tractable exhaustive inference over target-side syntax.

2. Probabilistic formulation and latent POS sequences

LaSyn augments each target token $y_t$ with a discrete latent variable $z_t$, interpreted as a POS-like tag. The model defines a joint distribution over translation output and latent syntax,

$$p(\mathbf{y},\mathbf{z}\mid \mathbf{x})=\prod_{t=1}^{T} p(y_t,z_t\mid y_{<t},z_{<t},\mathbf{x}),$$

and then approximates each factor as

$$p(y_t,z_t\mid y_{<t},z_{<t},\mathbf{x}) \approx p(z_t\mid y_{<t},\mathbf{x})\,p(y_t\mid z_t,y_{<t},\mathbf{x}).$$

The crucial property is the absence of an explicit dependence $z_t\mid z_{<t}$. Under this factorization,

$$p(\mathbf{y},\mathbf{z}\mid \mathbf{x}) = \prod_{t=1}^{T} p(z_t\mid y_{<t},\mathbf{x})\,p(y_t\mid z_t,y_{<t},\mathbf{x}),$$

and the marginal over translations becomes

$\mathbf{x}$0

The latent summation therefore decomposes into a per-time-step sum rather than a sequence-level search. This is the mathematical basis for the paper’s claim of exhaustive search over latent syntactic choices at each step (Yang et al., 2019).

The latent symbols are drawn from a small vocabulary $\mathbf{x}$1. In the reported experiments, the POS vocabulary size is 32 on the IWSLT tasks and 16 on WMT14 English–German after merging similar POS tags. The decoder state $\mathbf{x}$2 predicts a distribution over latent tags,

$\mathbf{x}$3

and the chosen latent tag then conditions lexical generation,

$\mathbf{x}$4

In this design, $\mathbf{x}$5 carries semantic and contextual information, while $\mathbf{x}$6 steers word generation toward a syntactic category. The intended effect is syntax–semantics co-dependence without a syntactic transition model.

3. POS regularization and neural EM training

Although the latent syntax variables are unobserved in the NMT objective, LaSyn regularizes them with external POS information. For each target training sentence, Stanford Parser provides a POS sequence $\mathbf{x}$7, aligned to subword tokens. The model defines an auxiliary distribution $\mathbf{x}$8 and a POS regularization term

$\mathbf{x}$9

The full training loss is

$\mathbf{y}=(y_1,\dots,y_T)$0

with $\mathbf{y}=(y_1,\dots,y_T)$1 in the experiments. The POS labels therefore function as a soft guide rather than a hard constraint: the model is encouraged, but not forced, to align its latent states with parser-derived POS categories (Yang et al., 2019).

Training is described as a neural EM-style procedure. For each example $\mathbf{y}=(y_1,\dots,y_T)$2, the E-step computes posterior responsibilities over latent tags,

$\mathbf{y}=(y_1,\dots,y_T)$3

Because of the same factorization that makes decoding tractable, the posterior decomposes across positions: $\mathbf{y}=(y_1,\dots,y_T)$4 The M-step maximizes the expected complete-data log-likelihood under these responsibilities. In practice, the method computes soft responsibilities on the fly and uses them to weight gradients for the tag prediction network, the word-generation network, and the POS prediction network. A plausible implication is that LaSyn occupies a middle ground between purely unsupervised latent tagging and fully supervised syntax-conditioned NMT.

4. Decoder integration, exact per-step inference, and implementation

LaSyn is implemented by modifying a Transformer-base NMT system in Fairseq. The reported base architecture uses model dimension $\mathbf{y}=(y_1,\dots,y_T)$5, Adam with $\mathbf{y}=(y_1,\dots,y_T)$6 and $\mathbf{y}=(y_1,\dots,y_T)$7, learning rate $\mathbf{y}=(y_1,\dots,y_T)$8, 4000 warm-up steps, and a Vaswani-style schedule. Batch size is approximately 2000 tokens per GPU on IWSLT and approximately 800 tokens on WMT English–German. Inference uses beam search with beam size 5 (Yang et al., 2019).

At each decoder step, the model computes the usual Transformer decoder hidden state $\mathbf{y}=(y_1,\dots,y_T)$9, then predicts latent-tag logits

$|\mathcal{Z}|^T$0

For word generation, the syntax contribution can be represented either by a hard tag choice or by a soft mixture of tag embeddings,

$|\mathcal{Z}|^T$1

This is combined with the decoder state to produce the final prediction state $|\mathcal{Z}|^T$2, from which the word distribution is obtained.

The paper’s notion of “exhaustive” inference is local rather than globally structured. For each beam hypothesis $|\mathcal{Z}|^T$3, the next-token marginal is

$|\mathcal{Z}|^T$4

and the model sums over all latent tags $|\mathcal{Z}|^T$5. Because $|\mathcal{Z}|^T$6 is small and there is no dependence on $|\mathcal{Z}|^T$7, the additional complexity is linear in the latent vocabulary size relative to a standard Transformer step. This is the technical sense in which LaSyn enables exhaustive search through latent syntactic choices while keeping decoding efficient.

5. Experimental settings and empirical findings

The detailed experimental description specifies IWSLT’14 German–English, IWSLT’14 English–French, and WMT’14 English–German settings, while the abstract states that evaluation was conducted on four different MT tasks (Yang et al., 2019).

Task	Training data	POS vocabulary
IWSLT’14 German–English	~153K sentence pairs	32
IWSLT’14 English–French	~220K sentence pairs	32
WMT’14 English–German	4.5M sentence pairs	16

For IWSLT’14 German–English, the test set is the concatenation of dev2010, tst2010, tst2011, and tst2012; the data are lowercased and BPE tokenized. For IWSLT’14 English–French, validation is tst2014 and test is tst2015, with lowercasing. For WMT’14 English–German, development uses newstest2012 and newstest2013, test uses newstest2014, and the shared BPE vocabulary is approximately 40K.

The main baseline is a standard Transformer-base NMT system without syntax. Against this baseline, LaSyn improves BLEU on all three explicitly described tasks. On the IWSLT tasks, the reported gains are around approximately $|\mathcal{Z}|^T$8–$|\mathcal{Z}|^T$9 BLEU depending on the translation direction. On WMT’14 English–German, LaSyn also improves over the baseline with a modest but consistent gain. The paper further reports that incorporating target-side syntax provides an opportunity to improve diversity. The analysis mentions diversity metrics such as distinct-$y_t$0 and variation in POS patterns across beam candidates, and attributes the diversity effect to the model’s ability to assign non-negligible probability mass to multiple syntactic patterns.

The ablations are consistent with the paper’s design rationale. Removing the POS regularizer by setting $y_t$1 lowers performance, indicating that POS supervision is important for shaping the latent space. Reverting to the standard Transformer removes the latent syntax mechanism entirely and underperforms LaSyn. Variants in which tags do not feed into the word generator also underperform, suggesting that explicit conditioning of lexical choice on latent syntax is operationally important rather than merely auxiliary.

6. Interpretability, conceptual boundaries, and limitations

A notable property of LaSyn is that its learned latent tags are partially interpretable. The paper reports that many latent tags align with human POS categories such as nouns, verbs, prepositions, and determiners, and that latent sequences follow plausible POS patterns such as DET–NOUN–VERB–DET–NOUN. Tag embeddings also exhibit cluster structure under projections such as t-SNE, with functionally similar categories appearing close in embedding space. This suggests that the model is not merely using the latent variables as arbitrary discrete codes, but is organizing them in a linguistically meaningful way (Yang et al., 2019).

Two conceptual boundaries are especially important. First, LaSyn is not a tree-structured or globally constrained syntax model. Its tractability relies on removing direct dependencies among successive latent tags, so long-distance syntactic constraints are left to the decoder’s semantic representation rather than explicitly enforced in the latent structure. Second, “POS supervision” does not mean that POS tags are supplied as hard generation constraints; they act as a regularizer on a latent space that remains task-driven.

The limitations stated in the paper follow directly from those design choices. The method depends on an external POS tagger for regularization, and tagging noise may be problematic, especially in low-resource settings. The latent vocabulary is deliberately small—16 to 32 categories—which constrains expressiveness. The decoupling of tags across time simplifies inference but removes explicit latent modeling of phenomena such as subject–verb agreement across long distances. Domain transfer is also a concern because POS distributions vary across domains.

The name also requires disambiguation. In later speech-processing literature, LaSyn denotes “Latent Synthesis,” a framework for converting text into pseudo acoustic latent representations for end-to-end ASR and SLU, which is unrelated to the NMT model described here (Lu et al., 2023). In the translation literature, however, LaSyn refers specifically to the latent POS-sequence model introduced for Transformer-based NMT (Yang et al., 2019).