Equation-Suffix Prediction in Event Sequence Modeling

• The paper demonstrates a novel joint prediction model with an LSTM-based encoder-decoder and GAN that forecasts event suffixes and remaining time.
• The method employs open-loop training and differentiable decoding with Gumbel-Softmax to accurately predict discrete event labels and time durations.
• Empirical results reveal significant gains in sequence similarity (SDL) and reduced timing errors (MAE) across business process logs compared to strong LSTM baselines.

As an Editor’s term, “equation-suffix prediction” may denote a suffix-prediction setting whose target is specified explicitly by optimization and loss equations. In the formulation given by Taymouri et al., the underlying task is event suffix and remaining time prediction for timestamped event sequences: given a prefix $X_{1:t} = (e_1,\tau_1),\ldots,(e_t,\tau_t)$, the objective is to obtain the most likely continuation $S$ maximizing $\hat S = \arg\max_S P(S \mid X_{1:t})$ and to predict the remaining time $R = \sum_{i=t+1}^T \tau_i$. The problem is cast as sequence-to-sequence learning over ordered events that carry at least two attributes, the event label and its timestamp, and is situated in applications including economics, digital health, business process management, and IT infrastructure monitoring (Taymouri et al., 2021).

1. Formal task definition

An event sequence is defined as

$X = (e_1,\tau_1),\ldots,(e_T,\tau_T),$

where $e_t \in A$ is a discrete label and $\tau_t \in \mathbb{R}_+$ is its timestamp or duration. For a prefix of length $t$,

$X_{1:t} = (e_1,\tau_1),\ldots,(e_t,\tau_t),$

the corresponding suffix is

$$X_{t+1:T} = (e_{t+1},\tau_{t+1}),\ldots,(e_T,\tau_T).$$

Suffix prediction seeks a continuation of the prefix, while remaining-time prediction estimates the time until the sequence finishes.

The paper treats these as coupled prediction targets rather than independent outputs. The suffix comprises the future labels $S$0, while the temporal target is the aggregate remaining time $S$1. This pairing is important because the model is designed to capture the joint temporal dynamics of events, not merely the categorical continuation (Taymouri et al., 2021).

A central motivation is the train–test mismatch in prior deep learning approaches. The paper identifies a discrepancy between closed-loop training, in which the next event is conditioned on the ground truth of previous events, and open-loop inference, in which the next event is conditioned on previously predicted events. This mismatch is presented as a source of potentially large prediction errors.

2. Event representation and encoder–decoder design

Each event $S$2 is represented as the concatenation of a one-hot embedding $S$3 and the scalar $S$4. The encoder is a multi-layer LSTM that scans the prefix $S$5 and returns its final hidden state $S$6. The decoder is another LSTM seeded with $S$7.

At each decoder step $S$8, the decoder produces a hidden output $S$9 that is passed through two parallel fully connected heads. The label head computes

$\hat S = \arg\max_S P(S \mid X_{1:t})$0

and the time head computes

$\hat S = \arg\max_S P(S \mid X_{1:t})$1

The most probable label

$\hat S = \arg\max_S P(S \mid X_{1:t})$2

or its continuous Gumbel-Softmax relaxation, together with $\hat S = \arg\max_S P(S \mid X_{1:t})$3, is concatenated and fed back as input to the next decoding step. Decoding continues until the special [EOS] token is produced (Taymouri et al., 2021).

This architecture instantiates a standard encoder–decoder decomposition but modifies its operational regime: the decoder is intended to run in the same feedback mode during training and inference. A plausible implication is that the architectural novelty lies less in the use of LSTMs per se than in the interaction between autoregressive decoding, open-loop operation, and adversarial supervision.

3. Adversarial formulation and optimization objective

The encoder–decoder network is treated as a generator

$\hat S = \arg\max_S P(S \mid X_{1:t})$4

while a discriminator

$\hat S = \arg\max_S P(S \mid X_{1:t})$5

is implemented as an LSTM plus fully connected network that scores a complete suffix $\hat S = \arg\max_S P(S \mid X_{1:t})$6 as “real” or “fake.” The adversarial term is

$\hat S = \arg\max_S P(S \mid X_{1:t})$7

The generator minimizes this objective and the discriminator maximizes it.

The supervised component consists of a label loss and a time loss. The label loss is cross-entropy,

$\hat S = \arg\max_S P(S \mid X_{1:t})$8

and the time loss is mean squared error,

$\hat S = \arg\max_S P(S \mid X_{1:t})$9

The full generator objective is

$R = \sum_{i=t+1}^T \tau_i$0

The discriminator is trained alternatingly with respect to $R = \sum_{i=t+1}^T \tau_i$1 (Taymouri et al., 2021).

Within this formulation, adversarial learning is not a replacement for supervised sequence modeling. It is an auxiliary mechanism added to the sequence loss in order to improve the realism of complete suffixes and, according to the paper, to boost prediction performance.

4. Open-loop training, differentiable decoding, and beam-search inference

The training regime is explicitly open-loop (free-running). At each decoder step, the network feeds back its own prediction rather than the ground truth. The paper states that this simulates inference-time behavior and closes the train/test mismatch. A small teacher-forcing probability (10%) is occasionally used to speed up convergence.

Because label generation is discrete, the model uses Gumbel-Softmax reparameterization to back-propagate through $R = \sum_{i=t+1}^T \tau_i$2:

$R = \sum_{i=t+1}^T \tau_i$3

with $R = \sum_{i=t+1}^T \tau_i$4 and $R = \sum_{i=t+1}^T \tau_i$5 annealed to $R = \sum_{i=t+1}^T \tau_i$6. The implementation notes specify that the Gumbel-Softmax temperature is annealed from $R = \sum_{i=t+1}^T \tau_i$7.

The training loop consists of four stages per iteration: generator forward pass with open-loop decoding and Gumbel-Softmax; discriminator update using real and generated suffixes; generator adversarial update; and generator supervised update using $R = \sum_{i=t+1}^T \tau_i$8. After training, the discriminator is discarded, and inference uses the generator together with beam search. Beam search keeps the top-$R = \sum_{i=t+1}^T \tau_i$9 partial suffixes by accumulated log-probability, expands each candidate step by step until [EOS] or maximum length, and returns the $X = (e_1,\tau_1),\ldots,(e_T,\tau_T),$0 best complete suffixes together with remaining-time predictions (Taymouri et al., 2021).

The reported implementation details are specific: 5-layer LSTMs with 200 units, RMSProp @ 5e−5, and gradient-clip=1. The summary also reports a training-time difference of approximately $X = (e_1,\tau_1),\ldots,(e_T,\tau_T),$1 sec/iter (MLE) vs $X = (e_1,\tau_1),\ldots,(e_T,\tau_T),$2 sec/iter (GAN). This indicates that the adversarial variant incurs additional optimization cost.

5. Evaluation protocol and reported empirical results

Suffix quality is evaluated using a normalized Damerau-Levenshtein-based similarity,

$X = (e_1,\tau_1),\ldots,(e_T,\tau_T),$3

with $X = (e_1,\tau_1),\ldots,(e_T,\tau_T),$4. Time quality is evaluated using

$X = (e_1,\tau_1),\ldots,(e_T,\tau_T),$5

reported in days.

The experiments compare against three strong LSTM-based baselines—Tax et al. 2017, Lin et al. 2019, and Taymouri et al. 2020—on four real logs: Helpdesk, BPI12(W), BPI12, and BPI17. The reported results are +4–11 pp improvement in SDL and 2×–10× reduction in MAE. One cited example is MAE $X = (e_1,\tau_1),\ldots,(e_T,\tau_T),$6 days vs $X = (e_1,\tau_1),\ldots,(e_T,\tau_T),$7 days on BPI12(W). The gains are reported as statistically significant ($X = (e_1,\tau_1),\ldots,(e_T,\tau_T),$8 via paired t-tests) (Taymouri et al., 2021).

The paper also reports an ablation of adversarial training, contrasting pure MLE with MLE+GAN. For beam $X = (e_1,\tau_1),\ldots,(e_T,\tau_T),$9, the GAN-augmented model yields significantly higher SDL and lower MAE. In the abstract, these improvements are described as up to four times compared to the state of the art in suffix and remaining time prediction of event sequences, specifically in the field of business process executions.

These results situate the method primarily within business process execution logs, even though the task itself is introduced as relevant to a broader set of domains. A plausible implication is that the strongest empirical evidence currently concerns process-mining-style event data.

6. Conceptual significance and recurrent points of confusion

A common simplification is to treat suffix prediction as merely a next-event prediction problem. The paper’s formulation is broader: the decoder generates a full continuation until [EOS], and the system jointly predicts the associated remaining time. The target is therefore a sequence suffix plus a scalar temporal aggregate, not a single-step categorical forecast (Taymouri et al., 2021).

Another recurring misunderstanding is to assume that the method is trained by standard teacher forcing throughout. In fact, the training regime is explicitly open-loop, with only a small teacher-forcing probability of 10% used occasionally. This design is central to the paper’s critique of prior approaches and to its attempt to align training-time and inference-time dynamics.

A further point concerns the role of the discriminator at deployment. The discriminator is used only during training; at inference, it is discarded and the deployed system is G + beam-search. This matters for interpretation: the discriminator shapes the generator’s learning signal but does not participate in suffix generation at test time.

Finally, the reported evidence does not support unrestricted generalization claims. The experiments involve four real-life datasets and three baselines, and the strongest results are described for business process executions. This suggests that the contribution is best understood as a concrete, experimentally validated sequence-modeling method for timestamped event suffixes rather than a universal result for all suffix-prediction settings.