---
title: Equation-Suffix Prediction in Event Sequence Modeling
url: https://www.emergentmind.com/topics/equation-suffix-prediction
type: topic
---

# Equation-Suffix Prediction in Event Sequence Modeling

As an *Editor’s term*, **“equation-suffix prediction”** may denote a suffix-prediction setting whose target is specified explicitly by optimization and loss equations. In the formulation given by Taymouri et al., the underlying task is **event suffix and remaining time prediction** for timestamped event sequences: given a prefix $X_{1:t} = (e_1,\tau_1),\ldots,(e_t,\tau_t)$, the objective is to obtain the most likely continuation $S$ maximizing $\hat S = \arg\max_S P(S \mid X_{1:t})$ and to predict the remaining time $R = \sum_{i=t+1}^T \tau_i$. The problem is cast as sequence-to-sequence learning over ordered events that carry at least two attributes, the event label and its timestamp, and is situated in applications including economics, digital health, business process management, and IT infrastructure monitoring [2102.07298].

## 1. Formal task definition

An event sequence is defined as
$$
X = (e_1,\tau_1),\ldots,(e_T,\tau_T),
$$
where $e_t \in A$ is a discrete label and $\tau_t \in \mathbb{R}_+$ is its timestamp or duration. For a prefix of length $t$,
$$
X_{1:t} = (e_1,\tau_1),\ldots,(e_t,\tau_t),
$$
the corresponding suffix is
$$
X_{t+1:T} = (e_{t+1},\tau_{t+1}),\ldots,(e_T,\tau_T).
$$
Suffix prediction seeks a continuation of the prefix, while remaining-time prediction estimates the time until the sequence finishes.

The paper treats these as coupled prediction targets rather than independent outputs. The suffix comprises the future labels $\langle e_{t+1},\ldots,e_T\rangle$, while the temporal target is the aggregate remaining time $R$. This pairing is important because the model is designed to capture the **joint temporal dynamics of events**, not merely the categorical continuation [2102.07298].

A central motivation is the train–test mismatch in prior deep learning approaches. The paper identifies a discrepancy between **closed-loop training**, in which the next event is conditioned on the ground truth of previous events, and **open-loop inference**, in which the next event is conditioned on previously predicted events. This mismatch is presented as a source of potentially large prediction errors.

## 2. Event representation and encoder–decoder design

Each event $e_i = (a_i,t_i)$ is represented as the concatenation of a one-hot embedding $a_i \in \{0,1\}^{|A|}$ and the scalar $t_i$. The encoder is a multi-layer LSTM that scans the prefix $X_{1:t}$ and returns its final hidden state $h_t$. The decoder is another LSTM seeded with $h_t$.

At each decoder step $k \ge t+1$, the decoder produces a hidden output $y^{(k)}$ that is passed through two parallel fully connected heads. The **label head** computes
$$
\pi^{(k)} = \operatorname{Softmax}(W_a y^{(k)} + b_a) \in [0,1]^{|A|},
$$
and the **time head** computes
$$
\hat{\tau}_k = \operatorname{ReLU}(W_t y^{(k)} + b_t) \in \mathbb{R}_+.
$$
The most probable label
$$
\hat e_k = \arg\max \pi^{(k)}
$$
or its continuous Gumbel-Softmax relaxation, together with $\hat{\tau}_k$, is concatenated and fed back as input to the next decoding step. Decoding continues until the special **[EOS]** token is produced [2102.07298].

This architecture instantiates a standard encoder–decoder decomposition but modifies its operational regime: the decoder is intended to run in the same feedback mode during training and inference. A plausible implication is that the architectural novelty lies less in the use of LSTMs per se than in the interaction between autoregressive decoding, open-loop operation, and adversarial supervision.

## 3. Adversarial formulation and optimization objective

The encoder–decoder network is treated as a generator
$$
G(X_{1:t};\theta_g) \to \hat X_{t+1:T},
$$
while a discriminator
$$
D(U;\theta_d) \to [0,1]
$$
is implemented as an LSTM plus fully connected network that scores a complete suffix $U$ as “real” or “fake.” The adversarial term is
$$
L_{\mathrm{GAN}}(G,D)
= \mathbb{E}_{X_{t+1:T}\sim p_{\mathrm{data}}}\bigl[-\log D(X_{t+1:T})\bigr]
+ \mathbb{E}_{X_{1:t}\sim p_{\mathrm{data}}}\bigl[-\log(1-D(G(X_{1:t})))\bigr].
$$
The generator minimizes this objective and the discriminator maximizes it.

The supervised component consists of a label loss and a time loss. The label loss is cross-entropy,
$$
L_{\mathrm{CE}}
= -\sum_{k=t+1}^T \sum_{i=1}^{|A|} a_{k,i}\log \pi_i^{(k)},
$$
and the time loss is mean squared error,
$$
L_{\mathrm{time}} = \sum_{k=t+1}^T (\tau_k - \hat\tau_k)^2.
$$
The full generator objective is
$$
L(\theta_g) = L_{\mathrm{CE}} + \lambda L_{\mathrm{time}} + \alpha L_{\mathrm{GAN}}(G,D).
$$
The discriminator is trained alternatingly with respect to $L_{\mathrm{GAN}}(G,D)$ [2102.07298].

Within this formulation, adversarial learning is not a replacement for supervised sequence modeling. It is an auxiliary mechanism added to the sequence loss in order to improve the realism of complete suffixes and, according to the paper, to boost prediction performance.

## 4. Open-loop training, differentiable decoding, and beam-search inference

The training regime is explicitly **open-loop (free-running)**. At each decoder step, the network feeds back its own prediction rather than the ground truth. The paper states that this simulates inference-time behavior and closes the train/test mismatch. A small **teacher-forcing probability (10%)** is occasionally used to speed up convergence.

Because label generation is discrete, the model uses **Gumbel-Softmax reparameterization** to back-propagate through $\pi^{(k)}$:
$$
\alpha_i = \frac{\exp((\log \pi_i + g_i)/\tau)}{\sum_j \exp((\log \pi_j + g_j)/\tau)},
$$
with $g_i \sim \mathrm{Gumbel}(0,1)$ and $\tau$ annealed to $0$. The implementation notes specify that the Gumbel-Softmax temperature is annealed from **$0.9 \to 0$**.

The training loop consists of four stages per iteration: generator forward pass with open-loop decoding and Gumbel-Softmax; discriminator update using real and generated suffixes; generator adversarial update; and generator supervised update using $L_{\mathrm{CE}} + \lambda L_{\mathrm{time}}$. After training, the discriminator is discarded, and inference uses the generator together with **beam search**. Beam search keeps the top-$n$ partial suffixes by accumulated log-probability, expands each candidate step by step until **[EOS]** or maximum length, and returns the $n$ best complete suffixes together with remaining-time predictions [2102.07298].

The reported implementation details are specific: **5-layer LSTMs with 200 units**, **RMSProp @ 5e−5**, and **gradient-clip=1**. The summary also reports a training-time difference of approximately **$\sim 1.5$ sec/iter (MLE) vs $\sim 4$ sec/iter (GAN)**. This indicates that the adversarial variant incurs additional optimization cost.

## 5. Evaluation protocol and reported empirical results

Suffix quality is evaluated using a normalized Damerau-Levenshtein-based similarity,
$$
\mathrm{SDL}(s_1,s_2)=1-\frac{DL(s_1,s_2)}{\max(|s_1|,|s_2|)},
$$
with $\mathrm{SDL}\in[0,1]$. Time quality is evaluated using
$$
\mathrm{MAE} = \text{average } |R-\hat R|,
$$
reported in days.

The experiments compare against **three strong LSTM-based baselines**—Tax et al. 2017, Lin et al. 2019, and Taymouri et al. 2020—on **four real logs**: **Helpdesk, BPI12(W), BPI12, and BPI17**. The reported results are **+4–11 pp improvement in SDL** and **2×–10× reduction in MAE**. One cited example is **MAE $\simeq 12$ days vs $34$ days on BPI12(W)**. The gains are reported as **statistically significant ($p<0.05$ via paired t-tests)** [2102.07298].

The paper also reports an ablation of adversarial training, contrasting **pure MLE** with **MLE+GAN**. For **beam $\ge 3$**, the **GAN-augmented model** yields significantly higher SDL and lower MAE. In the abstract, these improvements are described as **up to four times compared to the state of the art** in suffix and remaining time prediction of event sequences, specifically in the realm of business process executions.

These results situate the method primarily within business process execution logs, even though the task itself is introduced as relevant to a broader set of domains. A plausible implication is that the strongest empirical evidence currently concerns process-mining-style event data.

## 6. Conceptual significance and recurrent points of confusion

A common simplification is to treat suffix prediction as merely a **next-event prediction** problem. The paper’s formulation is broader: the decoder generates a full continuation until **[EOS]**, and the system jointly predicts the associated remaining time. The target is therefore a sequence suffix plus a scalar temporal aggregate, not a single-step categorical forecast [2102.07298].

Another recurring misunderstanding is to assume that the method is trained by standard teacher forcing throughout. In fact, the training regime is explicitly open-loop, with only a small teacher-forcing probability of **10%** used occasionally. This design is central to the paper’s critique of prior approaches and to its attempt to align training-time and inference-time dynamics.

A further point concerns the role of the discriminator at deployment. The discriminator is used only during training; at inference, it is discarded and the deployed system is **G + beam-search**. This matters for interpretation: the discriminator shapes the generator’s learning signal but does not participate in suffix generation at test time.

Finally, the reported evidence does not support unrestricted generalization claims. The experiments involve **four real-life datasets** and **three baselines**, and the strongest results are described for **business process executions**. This suggests that the contribution is best understood as a concrete, experimentally validated sequence-modeling method for timestamped event suffixes rather than a universal result for all suffix-prediction settings.

Source: https://www.emergentmind.com/topics/equation-suffix-prediction