---
title: Intention in Neural Dialogue Models
url: https://www.emergentmind.com/topics/intention
type: topic
---

# Intention in Neural Dialogue Models

Searching arXiv for the target paper and closely related dialogue-model references.
“Attention with Intention for a Neural Network Conversation Model” introduces a neural conversation architecture in which **intention** and **attention** are treated as distinct but interacting processes in multi-turn dialogue [1510.08565]. The model consists of three recurrent networks: a word-level encoder for the current user utterance, a turn-level intention network that persists across dialogue turns, and a word-level decoder with attention that generates the system response. In this formulation, intention is a latent continuous vector representing the discourse-relevant purpose of the current turn, while attention is a content-based alignment over source-side words during decoding. The paper’s central claim is that separating these two levels allows the model to capture dialogue coherence over turns while still attending flexibly to local lexical content, and that the resulting system can be trained end-to-end without intention labels and can generate natural responses [1510.08565].

## 1. Conceptualization of intention in dialogue

The paper adopts the discourse theory of Grosz (1986), distinguishing **intentional structure** from **attentional state**. In the simplified formulation used here, intention is the high-level, turn-level structure that explains coherence across multiple utterances, such as “communicate the problem,” “resolve the issue,” or “acknowledge/close,” whereas attention is the low-level, word- or token-level focus inside an utterance [1510.08565]. This division is the defining conceptual move of the model.

Within the neural architecture, **intention** is modeled as a turn-level recurrent state that persists and evolves across dialogue turns, and **attention** is modeled as the familiar content-based alignment inside the decoder, focusing on particular encoder states when predicting each response token [1510.08565]. The interaction is asymmetric but tight: the decoder RNN is initialized from the current intention state, so every decoding step is conditioned on intention; attention then operates using decoder states that already embed intention, meaning that intention influences which source words become salient and how they are interpreted in context [1510.08565].

Compared to classical dialogue systems, the paper positions this intention mechanism as a latent continuous alternative to explicit symbolic representations such as dialogue acts, plans, goals, subgoals, hand-crafted rules, or POMDP state representations [1510.08565]. This implies a shift from manually specified discourse structure toward learned continuous state.

## 2. Three-network architecture

The overall architecture contains three recurrent components, unrolled over dialogue turns [1510.08565]. The first is an **encoder network**, a word-level RNN that encodes the current source utterance \(x_u^{(s)}\) into both a sequence of hidden states and a final summary vector. The second is an **intention network**, a turn-level recurrent model that maintains the dialogue intention state. The third is a **decoder network**, a word-level recurrent language model with attention that generates the response \(y_u^{(t)}\) conditioned on both the current intention state and the encoder states [1510.08565].

For a source utterance \(x = (x_t : t = 1,\dots,T)\), the encoder recurrence is given by
\[
h_t^{(enc)} = f\bigl(x_t, h_{t-1}^{(enc)}\bigr),
\]
with \(f\) implemented as a depth-gated LSTM [1510.08565]. The encoder state is coupled across turns: at the first word of the current turn, \(h_0^{(enc)}\) is set to the last hidden state of the decoder from the previous turn [1510.08565]. The encoder yields a fixed-length summary
\[
c^{(fix)} = h_T^{(enc)}
\]
for the intention network, together with the full set \(\{h_t^{(enc)}\}_{t=1}^T\) used for attention [1510.08565].

The intention network updates one hidden state per turn. At turn \(k\), the intention state depends on the current encoder summary, the last decoder hidden state from the previous turn, and the previous intention state:
\[
h_k^{(int)} = f^{(int)}\left(c_k^{(fix)},\; h_{k-1,\text{last}}^{(dec)},\; h_{k-1}^{(int)}\right).
\]
The paper also summarizes this schematically as \(h = f(c,h,h)\) [1510.08565]. In effect, the intention vector aggregates what the user just said, how the system last responded, and the history of previous intentions.

The decoder is a conditional language model:
\[
p(y \mid x) = \prod_{j=1}^{J} p\bigl(y_j \mid y_1,\dots,y_{j-1}, x\bigr),
\]
with hidden-state recurrence
\[
h_j^{(dec)} = f^{(dec)}(y_{j-1}, h_{j-1}^{(dec)}, c_j).
\]
The crucial design choice is that the decoder is initialized with the current intention state,
\[
h_0^{(dec)} = h_k^{(int)},
\]
so intention enters as an initial condition rather than an explicit per-step input vector [1510.08565].

## 3. Attention mechanism and decoder conditioning

At each decoder step \(j\), the model computes a context vector \(c_j\) through attention over encoder states:
\[
c_j = z\bigl(h_{j-1}^{(dec)}, \{h_t^{(enc)} : t = 1,\dots,T\}\bigr)
     = \sum_{t=1}^{T} \alpha_{jt} h_t^{(enc)}.
\]
The attention weights are
\[
\alpha_{jt} = \frac{\exp(e_{jt})}{\sum_{m=1}^{T} \exp(e_{jm})},
\]
with alignment scores
\[
e_{jt} = a\bigl(h_{j-1}^{(dec)}, h_t^{(enc)}\bigr)
       = v^\top \tanh\left( W_h h_{j-1}^{(dec)} + W_c h_t^{(enc)} \right),
\]
where \(W_h \in \mathbb{R}^{A \times H}\), \(W_c \in \mathbb{R}^{A \times H}\), and \(v \in \mathbb{R}^A\) [1510.08565].

Because the alignment model uses \(h_{j-1}^{(dec)}\), and because the decoder state sequence is initialized from \(h_k^{(int)}\), the current intention state affects attention indirectly but pervasively [1510.08565]. The model therefore does not merely combine a dialogue-state RNN and an attention mechanism in parallel; it uses intention-conditioned decoder dynamics to modulate token-level alignment.

The output distribution is described in standard sequence-to-sequence form as depending on the decoder hidden state and the attention context:
\[
p(y_j = w \mid y_{<j}, x) = \text{softmax}\bigl( W_o [h_j^{(dec)}; c_j] + b_o \bigr)_w.
\]
Here again, intention influences the output both through \(h_j^{(dec)}\) and through the context vector \(c_j\), since the latter is computed from intention-conditioned decoder states [1510.08565].

This separation of turn-level intention and token-level attention is the paper’s key architectural distinction. By contrast, standard single-turn encoder-decoder dialogue models typically map a single input utterance to a single response and do not explicitly model multi-turn dynamics [1506.05869], while context-RNN models use an additional recurrent layer over sentences or turns but generally do not combine it with decoder-side attention over encoder states [1506.06714].

## 4. Turn-level dynamics and discourse coherence

The intention network is designed to capture how the system’s goals and conversational stance evolve over turns [1510.08565]. The update sequence is structurally simple: the user utterance is encoded; the intention state is updated from the encoder summary, prior decoder state, and prior intention; the updated intention initializes the response decoder [1510.08565]. Graphically, the paper describes a three-layer unrolling over turns: encoder on top, intention chain in the middle, decoder with upward attention links at the bottom [1510.08565].

The intention state therefore acts as a turn-level summary of three information sources: the current user input, previous system behavior, and intention history [1510.08565]. The paper interprets this as enabling coherent shifts in dialogue phase, such as early turns devoted to problem description, middle turns devoted to diagnosis and resolution, and final turns devoted to closing or acknowledgments [1510.08565].

A plausible implication is that the architecture functions as a minimal hierarchical dialogue model: token-level sequence modeling is nested inside turn-level recurrent state propagation. Unlike hierarchical variants that treat utterance representations as fixed sentence vectors [1506.06714], this formulation preserves word-level attentional access within each turn while maintaining dialogue-level recurrence across turns.

## 5. Training and latent intention learning

The model is trained end-to-end as a conditional language model over response tokens, with no explicit intention annotations [1510.08565]. The objective is standard maximum likelihood, equivalently negative log-likelihood:
\[
\mathcal{L} = - \sum_{u=1}^{U} \sum_{j=1}^{J_u} \log p\bigl( y_j^{(u)} \mid y_{<j}^{(u)}, x^{(u)}, \text{history} \bigr).
\]
The paper states that minimizing this objective is equivalent to minimizing perplexity:
\[
\text{PPL} = \exp\left(\frac{\mathcal{L}}{\# \text{ tokens}}\right).
\]
No additional supervision, classification loss, reconstruction term, or intention-specific regularizer is imposed on the intention state [1510.08565].

The learning signal for intention is purely indirect. Gradients from response prediction backpropagate through the decoder, into the initial decoder state \(h_0^{(dec)} = h_k^{(int)}\), through the intention RNN across turns, and further into the encoder that produced \(c_k^{(fix)}\) [1510.08565]. This forces the latent intention states to encode whatever turn-level information is useful for predicting future responses.

The paper argues that the architecture itself encourages meaningful intention representations because every decoder time step depends indirectly on intention, and intention is the only channel carrying turn-level context from prior turns into the current decoder, aside from the previous-turn decoder state used to initialize the next encoder [1510.08565]. All three recurrent networks use depth-gated LSTMs, which the authors motivate as support for long-range dependencies across both tokens and turns [1510.08565].

## 6. Experimental evidence

The experiments use in-house helpdesk chat logs for computer-related issues [1510.08565]. The reported data split is 10,000 training dialogues comprising 96,913 turns, 1,000 development dialogues comprising 9,971 turns, and 500 test dialogues comprising 5,232 turns [1510.08565]. The training set contains 2,215,047 source tokens and 2,378,950 target tokens, with a shared source/target vocabulary of 9,085 types [1510.08565].

Training uses sentence-level SGD with initial learning rate 0.1, halving the learning rate when development-set perplexity increases [1510.08565]. Dialogues are shuffled per epoch, but turn order is preserved within each dialogue so that the intention RNN retains conversational context [1510.08565].

The paper reports perplexity for two hidden sizes of the Attention with Intention model:

| Hidden dim | PPL |
|---|---:|
| 50 | 30.8 |
| 200 | 22.1 |

These are internal model comparisons rather than ablations against no-intention or no-attention baselines [1510.08565]. The paper explicitly does not isolate the quantitative contribution of intention alone.

The qualitative examples are more central to the paper’s argument. The generated conversations show sustained troubleshooting context and a progression of responses such as problem acknowledgment, information gathering, diagnosis/recommendation, and closing/survey language [1510.08565]. The authors state that the “flow of intentions is clearly seen” in these examples, which suggests that the turn-level intention RNN is capturing something like dialogue phase or goal state rather than merely local lexical continuation [1510.08565].

## 7. Position within dialogue modeling and limitations

The paper situates itself against two nearby neural dialogue paradigms. Relative to standard single-turn encoder-decoder systems, it adds explicit multi-turn dynamics and a separate dialogue-state recurrence [1506.05869]. Relative to context-RNN models, it emphasizes the explicit separation between intention and attention, and the fact that intention-conditioned decoder states determine attention scores [1506.06714]. This yields two claimed advantages: better structural modeling of conversation through a distinct intention process, and end-to-end learning of discourse-theoretic notions without hand-labeled intentions [1510.08565].

At the same time, the limitations are clear in the paper’s own framing. There is no direct empirical ablation of intention versus no-intention, so the contribution of the intention state is not isolated quantitatively [1510.08565]. The intention vector is latent and not interpretable in a human-labeled sense [1510.08565]. The formulation is also deliberately simple: one intention state per turn, with no hierarchical sub-intentions, no explicit goal supervision, and no task-success signals [1510.08565].

The paper’s broader significance lies in making intention a first-class recurrent variable in neural dialogue generation rather than collapsing all contextual structure into either a fixed utterance embedding or a decoder hidden state. This suggests a line of development in which long-range conversational context is carried by a turn-level goal-oriented state, while word-level attention remains the mechanism for local linguistic grounding [1510.08565].

Source: https://www.emergentmind.com/topics/intention