ProcessGAN: Transformer-based Synthetic Data Model

• ProcessGAN is a GAN-based, Transformer-powered model that generates synthetic process logs while preserving key statistical and structural properties.
• It employs non-autoregressive generation using dual Transformer encoders and straight-through Gumbel–Softmax to handle discrete token outputs efficiently.
• Empirical evaluations show ProcessGAN excels on small, complex datasets by producing realistic, privacy-preserving sequences that mirror authentic workflows.

ProcessGAN is a GAN-based, Transformer-powered deep generative model for synthetic process data introduced in "Generating Privacy-Preserving Process Data with Deep Generative Models" (Li et al., 2022). In this setting, process data denotes event-log style sequential data in which each case/trace is a sequence of activities drawn from a finite vocabulary $V$, with no explicit modeling of timestamps. The model is designed for settings in which authentic logs contain confidential information and cannot be shared directly in public, particularly in healthcare, business, and education. Its central objective is to generate synthetic traces that preserve statistical and structural properties of authentic workflows while avoiding direct disclosure of individual-level sequences (Li et al., 2022).

1. Problem formulation and privacy motivation

ProcessGAN operates on event-log style sequential data where a trace is written as

$x = [a_1, a_2, \dots, a_n]$

with each $a_i$ an activity token from a finite vocabulary $V$. The vocabulary has size $N_y$ and contains all activity types plus a special $[\text{end}]$ token. The work focuses exclusively on ordered activity sequences and their structure; timestamps, durations, resources, and other event attributes are not modeled (Li et al., 2022).

The motivating domains include four pediatric trauma resuscitation workflows and two public event logs. The medical examples include the “Secondary survey” process, the “Intubation” process, emergency department procedures, and the “Maintaining airway” process. Typical activities include “Patient Arrival”, “Airway Assessment”, “BVM”, “NRB”, “Decision to Intubate”, “RSI Sedative Meds”, “RSI Paralytic Meds”, and “Laryngoscopy”. The public datasets comprise a sepsis treatment process and a loan application process. These logs exhibit parallel branches, loops, and long-range dependencies between activities (Li et al., 2022).

The privacy motivation is explicit. Process logs in healthcare, business, and education contain confidential information and are often linked to individuals. Even after removal of explicit identifiers, combinations of simple demographics and events can re-identify patients or customers. In the medical datasets considered here, traces are manually coded from trauma resuscitation videos, making the underlying data highly sensitive. The paper identifies three classes of risk in sharing raw logs: re-identification of individuals, exposure of sensitive medical details, and legal and ethical issues around data protection (Li et al., 2022).

The study contrasts this approach with traditional privacy methods. Encryption-based methods protect data in transit or at rest, but data must be decrypted for analysis, creating a point of vulnerability, and compromise of keys or encryption exposes the full raw logs. Traditional anonymization or masking is described as difficult to perform robustly for sequential, structured logs, liable to destroy process structure, and still vulnerable to re-identification via rare or unique activity patterns. The alternative proposed is to learn a generative model of the process and share only synthetic traces drawn from that model (Li et al., 2022).

2. Core architecture

ProcessGAN uses two Transformer encoders, one as the generator $G$ and one as the discriminator $D$. The model is explicitly non-autoregressive. The generator receives a random sequence

$Z = [a_1, \dots, a_l]$

of activity tokens drawn uniformly from the activity vocabulary, including $[\text{end}]$, padded to a fixed maximum length $x = [a_1, a_2, \dots, a_n]$0. For each position $x = [a_1, a_2, \dots, a_n]$1, the generator outputs a categorical distribution over activities, represented as a vector of length $x = [a_1, a_2, \dots, a_n]$2 (Li et al., 2022).

Each activity index in $x = [a_1, a_2, \dots, a_n]$3 is mapped to an embedding vector. The embedding size is given as

$x = [a_1, a_2, \dots, a_n]$4

After embedding, the sequence passes through $x = [a_1, a_2, \dots, a_n]$5 Transformer encoder blocks with multi-head self-attention and standard positional encodings. A linear output layer then produces, for each position $x = [a_1, a_2, \dots, a_n]$6, a vector

$x = [a_1, a_2, \dots, a_n]$7

which represents a probability distribution over activities after softmax. Collectively, $x = [a_1, a_2, \dots, a_n]$8 is the sequence of categorical distributions (Li et al., 2022).

To convert these outputs into discrete tokens while preserving differentiability, the model uses straight-through Gumbel–Softmax. In the forward pass, the generated token at position $x = [a_1, a_2, \dots, a_n]$9 is

$a_i$0

and the sequence given to the discriminator is

$a_i$1

This mechanism allows backpropagation through discrete activity tokens without reinforcement-learning-based sequence training (Li et al., 2022).

The discriminator receives both real authentic traces and synthetic sequences from the generator as sequences of one-hot vectors of length $a_i$2. For generated sequences, it finds the first $a_i$3 token and pads all subsequent positions so that the classifier focuses on the meaningful prefix. Its output is a scalar probability $a_i$4 that the input sequence is authentic. The use of Transformer encoders for both $a_i$5 and $a_i$6 is presented as a way to help avoid mode collapse that can arise from capacity or learning-rate mismatches between generator and discriminator (Li et al., 2022).

The paper motivates this architecture by contrasting it with two alternatives. Pure maximum-likelihood training on autoregressive RNNs or Transformers is said to suffer from exposure bias, because the model sees ground-truth prefixes during training but must condition on its own predictions during generation. Non-autoregressive Transformers trained with cross-entropy are described as assuming conditional independence between positions and tending to produce short, low-variance, structurally poor sequences. The adversarial signal from the discriminator is intended to encourage the generator to capture sequence-level realism, subtle dependencies, and global structure, especially when data is small and complex (Li et al., 2022).

3. Objective functions and training dynamics

The generator and discriminator are trained with the vanilla GAN objective. The generator loss is

$a_i$7

and the discriminator loss is

$a_i$8

Here, $a_i$9 denotes the empirical distribution of authentic sequences and $V$0 the distribution of random input sequences. The discriminator attempts to distinguish authentic from synthetic traces, while the generator attempts to maximize $V$1 so that synthetic traces appear authentic (Li et al., 2022).

Because pure adversarial training can converge slowly or poorly with limited data and a large search space, the model adds an auxiliary loss on activity distributions. For a mini-batch of size $V$2, $V$3 denotes the empirical activity frequency distribution over real sequences and $V$4 the empirical activity frequency distribution over generated sequences. Two variants are defined: $V$5 and

$V$6

The augmented generator loss is

$V$7

with

$V$8

The variants are named P-GAN-M for the MSE version, P-GAN-K for the KL version, and P-GAN for the plain GAN without the auxiliary term (Li et al., 2022).

Training uses padded authentic traces and random input sequences sampled uniformly from $V$9 to length $N_y$0. To prevent the discriminator from overpowering the generator early in optimization, the update schedule is intentionally unbalanced: for each cycle, the generator is optimized for $N_y$1 epochs or steps, and the discriminator for 1 epoch or step. The paper notes that $N_y$2 and $N_y$3 naturally oscillate because of the minimax game, and proposes a practical stopping criterion when discriminator accuracy on distinguishing real versus synthetic stabilizes around $N_y$4, meaning the generator fools the discriminator roughly half the time (Li et al., 2022).

The stabilization strategy is limited to three design choices: structurally similar Transformer encoders for both $N_y$5 and $N_y$6, the activity distribution regularizer $N_y$7, and the $N_y$8 update schedule for $N_y$9. The method does not use Wasserstein loss, explicit gradient penalty, or spectral normalization (Li et al., 2022).

4. Data domains and evaluation protocol

The study evaluates ProcessGAN on six real-world datasets, including four medical datasets from pediatric trauma resuscitation and two public datasets. The medical datasets were manually coded from video and collected at Children’s National Hospital under IRB approval. The public datasets are a sepsis process and a loan application process, with the loan dataset divided into short and long traces (Li et al., 2022).

Dataset	Cases / split	Mean length / activity types
D1: Secondary survey process	46 cases	46.03 activities; 44 activity types
D2: Intubation process	114 cases	12.35 activities; 23 activity types
D3: Emergency department process	46 cases	5.53 activities; 8 activity types
D4: Maintaining airway process	43 cases	31.72 activities; 38 activity types
D5: Sepsis process	502 cases	17.28 activities; 16 activity types
D6: Loan application process	2625 cases	19 activity types; split into D6_1 and D6_2

For D6, the split is defined as D6_1 with sequences of length $[\text{end}]$0, mean length $[\text{end}]$1, standard deviation $[\text{end}]$2, and D6_2 with sequences of length $[\text{end}]$3, mean length $[\text{end}]$4, standard deviation $[\text{end}]$5. The medical datasets are described as especially challenging because they combine a small number of cases, long and complex sequences, many activity types, long-range dependencies, and high privacy sensitivity (Li et al., 2022).

Evaluation is explicitly multi-faceted. The first dimension is statistical similarity between authentic and synthetic logs. Sequence length is assessed by comparing the mean and standard deviation of lengths. Activity type occurrence is measured by activity-frequency distributions $[\text{end}]$6 and $[\text{end}]$7, with similarity quantified by the L1 distance

$[\text{end}]$8

Sequence diversity is measured using sum of pairwise normalized edit distance (SPE): $[\text{end}]$9 where $G$0 is Levenshtein distance and $G$1 is the number of sampled sequences (Li et al., 2022).

The second dimension is a supervised learning score based on an independent binary classifier. Negative samples are created by starting from authentic sequences and randomly adding, deleting, or swapping tokens at a predefined ratio. The classifier uses a Transformer encoder and then concatenates the sequence representation with its activity distribution vector and its length, followed by two dense layers and a final sigmoid. It is trained until it achieves F1 > 0.8 on distinguishing authentic from negative sequences. Synthetic sequences are then evaluated using the false positive rate

$G$2

where higher FPR indicates that synthetic data looks more authentic to this external classifier (Li et al., 2022).

The third dimension is process mining / workflow discovery plus expert review. A trace alignment method is applied to derive a consensus sequence as the backbone workflow, side-branch activities are attached as branches, and infrequent activities are filtered. Workflow diagrams are reconstructed for authentic data and for synthetic data generated by GRU, ProcessGAN, and combined GRU+ProcessGAN. Pediatric trauma physicians then assess correct versus incorrect activity orders, spurious long-span dependencies, missing clinically essential activities, and new plausible but unobserved pathways (Li et al., 2022).

5. Empirical results and comparative behavior

The paper reports that ProcessGAN outperformed traditional sequential models when trained on small authentic datasets of complex processes and that it better represented the long-range dependencies between the activities, which is particularly important in medical processes. Conversely, traditional sequential models performed better when trained on large data of simple processes (Li et al., 2022).

For sequence lengths, the reported pattern is that on longer sequences and small datasets—specifically D1, D4, and D6_2—ProcessGAN variants (P-GAN, P-GAN-M, P-GAN-K) produced length distributions much closer to authentic data. In contrast, GRU, LSTM, and autoregressive Transformer (Trans-AR) tended to produce shorter sequences and/or larger standard deviations. On short sequences or large datasets—D2, D3, D5, and D6_1—autoregressive models achieved length statistics comparable to ProcessGAN. The non-autoregressive Transformer without GAN (Trans-NAR) consistently produced shorter sequences with low variance, which the paper attributes to the independence assumption in cross-entropy training (Li et al., 2022).

For activity type occurrence, GRU, LSTM, and Trans-AR typically matched activity-frequency distributions better than the vanilla ProcessGAN measured by lower L1 distance. However, adding the activity divergence loss in P-GAN-M and P-GAN-K brought the activity distributions much closer to those of the autoregressive models. The paper also notes that for complex processes with tiny data, especially D4, even autoregressive models struggled, implying that data sufficiency remained a strong determinant of performance (Li et al., 2022).

For sequence variance, all models generated reasonably diverse sequences, as reflected by non-zero SPE. On larger datasets such as D5 and D6_1, variance was similar across methods. On small/complex datasets, GAN-based models—especially P-GAN-M and P-GAN-K—produced synthetic variance closest to real data, suggesting better coverage of process variants (Li et al., 2022).

The external classifier FPR results reinforce this division. On most datasets—D1 through D4, D5, and D6_2—ProcessGAN variants achieved higher FPR than RNNs and Trans-AR, meaning that they produced sequences with more realistic local activity dependencies according to the external discriminator. D6_1 was the exception: for the shorter loan traces, autoregressive models had better FPR, while ProcessGAN lagged. When moving to the longer traces in D6_2, RNN FPR dropped but ProcessGAN remained strong or improved, which the paper interprets as evidence of advantage on more complex processes (Li et al., 2022).

A more specific case study concerns the Intubation dataset (D2). Both GRU-generated and ProcessGAN-generated workflows captured the major treatment steps and their order: “Patient Arrival” $G$3 pre-oxygenation steps $G$4 “Decision to Intubate” $G$5 “Critical Window” $G$6 “RSI Sedative Meds” $G$7 “RSI Paralytic Meds” $G$8 “Laryngoscopy”. The ProcessGAN workflow included more detailed pathways and longer traces, and introduced clinically valid but previously unobserved activities such as “Airway Assessment” in the correct position. It also introduced process errors, including “Laryngoscopy” and “Critical Window” spanning the entire diagram in approximately $G$9 of synthetic traces. The GRU workflow placed activities mostly correctly but tended to generate shorter traces and included an error where “RSI Sedative Meds” spanned from the start to “RSI Paralytic Meds”, which the experts deemed clinically impossible. When synthetic traces from both models were mixed, the combined GRU + ProcessGAN workflow most closely mirrored the authentic process, and the combined pathways were judged semantically valid by experts (Li et al., 2022).

This pattern is summarized explicitly in the paper. ProcessGAN outperforms RNNs and autoregressive Transformers when authentic datasets are small and the process is complex, when sequences are long, activity vocabularies are large, and long-range dependencies are important. Traditional models perform better when datasets are large and processes are simple or moderately complex, because teacher-forced training can accurately learn activity frequencies and local next-step conditional distributions. The failure mode of Trans-NAR is treated as evidence that adversarial learning is important for non-autoregressive generation of realistic process traces (Li et al., 2022).

6. Privacy claims, limitations, and position in the literature

The paper’s privacy rationale is qualitative rather than formal. It argues that only synthetic traces are shared, authentic traces remain within the secure institution, synthetic data is produced from random input sequences rather than explicit case-specific seeds, and the model is intended to imitate structural properties rather than copy individual traces. The presence of valid but previously unobserved pathways in expert review is presented as evidence that the model generalizes beyond exact replication of observed cases (Li et al., 2022).

At the same time, the paper explicitly states that there is no formal privacy guarantee such as $D$0-differential privacy, and there is no membership inference or reconstruction attack evaluation. A plausible implication is that the method should be regarded as privacy-preserving in a practical or procedural sense rather than in a formal cryptographic or differential-privacy sense. The study therefore frames reduced direct disclosure risk as a benefit, while leaving privacy leakage quantification to future work (Li et al., 2022).

Several limitations are identified or implied. First, GANs remain vulnerable to mode collapse / coverage issues, even though the auxiliary loss and symmetric Transformer architecture help stabilize training. Second, although Transformers can capture long-range dependencies better than RNNs, very long or hierarchical processes remain challenging, especially with small data. Third, the representation excludes timestamps, durations, roles, and resources, so temporal and multi-attribute events are outside the scope of the current model. Fourth, GAN training is sensitive to hyperparameters, update schedules, and convergence criteria, and the paper does not use more recent stabilization techniques such as Wasserstein loss or gradient penalty (Li et al., 2022).

Within the broader literature, ProcessGAN is positioned at the intersection of sequential generative models for process mining, GANs for text/sequences, and synthetic data for privacy. The paper contrasts its design with RNN-based models, LSTMs, GRUs, and Transformers used for next-event prediction and process prediction, as well as with sequence GANs such as SeqGAN and RelGAN that often rely on reinforcement learning to manage discrete outputs. Its stated distinctive contributions are: non-autoregressive Transformer encoders for both generator and discriminator, straight-through Gumbel–Softmax for differentiable discrete generation, GAN + activity-distribution regularization, and a process-mining–oriented evaluation combining statistical similarity, external-classifier FPR, workflow discovery, and expert clinical review (Li et al., 2022).

The paper concludes that ProcessGAN can generate a large amount of sharable synthetic process data indistinguishable from authentic data. In context, this conclusion is tied specifically to the reported statistical measures, supervised learning scores, and expert assessment of discovered workflows, rather than to any formal privacy or indistinguishability guarantee. A plausible implication is that ProcessGAN is most appropriately understood as a synthetic process data generator optimized for structural fidelity in small, complex, privacy-sensitive sequential domains, with privacy assurance remaining an open research problem (Li et al., 2022).