---
title: 'dedupT: Transformer Crash Deduplication'
url: https://www.emergentmind.com/topics/dedupt
type: topic
---

# dedupT: Transformer Crash Deduplication

Searching arXiv for the primary and related papers on dedupT and neighboring deduplication methods.
dedupT is a transformer-based approach to stack trace-based crash deduplication that groups automated crash reports by treating each crash’s stack trace or stack traces as a single “document,” rather than a bag of isolated frames [2508.19449]. In its most specific usage, the term denotes the pipeline introduced in “Stack Trace-Based Crash Deduplication with Transformer Adaptation” [2508.19449], where a pretrained language model is adapted to stack traces and paired with a lightweight ranking network for duplicate ranking and unique crash detection. Related literature uses closely allied deduplication ideas in adjacent settings, including stack-trace retrieval-and-reranking [2412.14802] and record deduplication for entity-distribution modeling in ASR transcripts [2306.06246], but dedupT as a named method is primarily associated with crash-report deduplication.

## 1. Problem setting and conceptual scope

Crash deduplication addresses the problem that automated crash reporting systems generate large volumes of duplicate reports, overwhelming issue-tracking systems and increasing developer workload [2508.19449]. In this setting, the object to be deduplicated is not arbitrary text but a crash report whose most informative artifact is usually one or more stack traces. dedupT models those traces holistically rather than as isolated frames, with the explicit goal of improving both duplicate ranking and unique crash detection [2508.19449].

The pipeline consists of four stages: stack-trace preprocessing, domain-specific adaptation of a pretrained language model (PLM) to produce stack-trace embeddings, feature-level aggregation of possibly multiple traces per crash report, and a lightweight, fully-connected ranking network that scores candidate crash pairs [2508.19449]. This organization places dedupT in a family of learned deduplication systems, but its defining characteristic is the combination of transformer adaptation and pairwise ranking over crash-report representations.

A useful contrast is provided by “Stack Trace Deduplication: Faster, More Accurately, and in More Realistic Scenarios” [2412.14802], which uses an embedding model with byte-pair encoding and approximate nearest neighbor search to retrieve candidates, followed by a reranker that takes repeated frames into account. This suggests that recent stack-trace deduplication research has increasingly emphasized representation learning and multi-stage ranking rather than only string similarity, rule-based heuristics, or alignment-based matching.

## 2. Stack-trace preprocessing and document construction

dedupT begins by normalizing raw stack traces into a form suitable for transformer encoding [2508.19449]. Consecutive frames with the same subroutine, such as recursive calls, are collapsed. This duplicate removal step reduces redundant local repetition before any embedding is computed.

The method then applies top-\(N\) sampling. Empirically, the top 10–15 frames carry the most diagnostic signal, and all later frames are dropped to respect the PLM’s maximum token length, typically 512 [2508.19449]. The retained trace is therefore a deliberately pruned representation of the crash, optimized for the length and inductive biases of sentence-transformer models.

Frame cleaning is language-aware. Language-specific syntax, including dots in Java packages and debugger prefixes in C/C++, is replaced by whitespace, and all tokens are lower-cased [2508.19449]. Each stack frame is treated as a “sentence,” and the entire pruned trace is treated as one passage. dedupT also prepends or appends special tokens to each frame to preserve its ordinal position in the call stack [2508.19449]. This positional encoding is central to the method’s claim to holistic modeling: frame identity is not used in isolation, but in sequence context.

The preprocessing choices are not merely implementation details. The reported ablations show that frame count matters: MRR peaked using the top 10–12 frames, while retaining too many (\(>20\)) or too few (\(<5\)) degraded performance [2508.19449]. Frame trimming was also dataset-dependent: no trimming was best on Netbeans, whereas aggressive trimming improved Eclipse [2508.19449]. These results indicate that representation quality depends on a balance between preserving diagnostic context and avoiding length-induced noise.

## 3. PLM adaptation and contrastive embedding learning

dedupT starts from a sentence-transformer PLM and, in the reported experiments, uses bge-base-en with 512-dimensional embeddings [2508.19449]. The model is fine-tuned on labeled crash data to better capture structural and semantic relationships in stack traces. Training examples are constructed from duplicate buckets: an anchor \(x_a\) is any stack trace from bucket \(B_i\), a positive \(x_p\) is sampled from the same bucket, and negatives \(\{x_n^j\}\) are sampled from other buckets \(B_j, j \neq i\) [2508.19449].

The embedding model is optimized with the Multiple Negatives Ranking loss:
$$
L_{\mathrm{MNR}} = - \frac{1}{N} \sum_{i=1}^{N} \log
\frac{\exp\!\left(\mathrm{CosSim}(f_\theta(x_a^i), f_\theta(x_p^i))/\tau\right)}
{\sum_{j=1}^{N} \exp\!\left(\mathrm{CosSim}(f_\theta(x_a^i), f_\theta(x_n^j))/\tau\right)}.
$$

Cosine similarity is defined as
$$
\mathrm{CosSim}(u,v) = \frac{\langle u,v\rangle}{\|u\|\,\|v\|}.
$$

Fine-tuning sharpens the embedding space so that duplicates become closer than non-duplicates [2508.19449]. The reported adaptation effect is substantial. On Netbeans, the cosine-Pearson correlation between model similarity scores and ground truth rose from \(0.779 \rightarrow 0.897\), Spearman rose from \(0.779 \rightarrow 0.844\), and retrieval MRR improved from \(0.687 \rightarrow 0.771\) [2508.19449]. Fine-tuning is done on the training split only, and once converged, all crash-report embeddings in train, validation, and test sets can be precomputed and cached for efficiency [2508.19449].

The paper also reports that after fine-tuning, different pretrained PLMs, including mpnet, distilroberta, and bge, performed within approximately \(1\)–\(5\%\) of each other, with sequence-length limits partly explaining the small differences [2508.19449]. This suggests that domain adaptation is more important than any single off-the-shelf encoder choice, provided the encoder can represent stack-trace sequences with adequate length budget.

## 4. Multi-trace aggregation and duplicate ranking

A crash report \(Q\) may contain multiple stack traces \(\{s_1^Q,\dots,s_n^Q\}\). dedupT first embeds each trace individually, \(f(s_i^Q)=f_\theta(s_i^Q)\), and then merges them using Parametric Max-Mean aggregation [2508.19449]. The method first identifies the most similar cross-report pair:
$$
(s_i^Q, s_j^S) = \arg\max_{i,j} \mathrm{CosSim}(f(s_i^Q), f(s_j^S)).
$$

It then computes mean-pooled report embeddings and defines the fused report representation as
$$
f_Q = \alpha \cdot f(s_i^Q) \,\|\, (1-\alpha)\cdot \mu(QE),
$$
where \(\alpha \in [0,1]\) is learnable and \(\|\) denotes concatenation [2508.19449]. In the reported ablation, Parametric Max-Mean outperformed max-only, mean-only, and multi-head-attention pooling on Netbeans, with MRR \(0.777\) versus \(0.761\), \(0.722\), and \(0.654\), respectively [2508.19449].

For pairwise scoring, dedupT constructs the joint feature vector
$$
f_{QS} = \left( |f_Q-f_S|,\; \frac{f_Q+f_S}{2},\; f_Q \odot f_S \right),
$$
that is, absolute difference, element-wise average, and element-wise product [2508.19449]. This vector is passed through a two-layer fully connected network with dropout \(p=0.1\):
$$
h = \mathrm{Dropout}(f_{QS}), \qquad
u = \mathrm{ReLU}(\Phi_1(h)), \qquad
\mathrm{DupScore} = \Phi_2(u).
$$

Training uses RankNet loss on triplets \((x_a,x_p,x_n)\):
$$
L_{\mathrm{RankNet}} = \log\!\left(1 + \exp\!\left(-\Delta f(x_a,x_p) + \Delta f(x_a,x_n)\right)\right).
$$

The classifier is trained with Adam using learning rate \(1 \times 10^{-4}\) and batch size \(25\), on an Intel Xeon Gold 6148 CPU and NVIDIA V100 16 GB GPU, with peak GPU memory below \(3\) GB [2508.19449]. Classification training converges in minutes to a few hours, and end-to-end embedding adaptation plus classifier training completes in under 3 hours per dataset [2508.19449].

## 5. Empirical performance, ablations, and operating characteristics

dedupT was evaluated on four public datasets: Netbeans and Eclipse for Java, and Gnome and Ubuntu for C/C++ [2508.19449]. The reported summary metrics are as follows.

| Dataset | MRR | ROC-AUC |
|---|---:|---:|
| Netbeans | 0.771 | 0.933 |
| Eclipse | 0.791 | 0.891 |
| Gnome | 0.742 | 0.905 |
| Ubuntu | 0.786 | 0.905 |

On duplicate retrieval, Netbeans achieved MRR \(=0.771\) versus best traditional \(0.705\) and best deep-learning baseline \(0.658\); Eclipse achieved \(0.791\) versus \(0.752\) and \(0.732\); Gnome achieved \(0.742\) versus \(0.710\) and \(0.565\); and Ubuntu achieved \(0.786\) versus \(0.752\) and \(0.675\) [2508.19449]. On unique-report detection, ROC-AUC reached \(0.933\) on Netbeans, \(0.891\) on Eclipse, \(0.905\) on Gnome, and \(0.905\) on Ubuntu [2508.19449]. The paper states that on four public datasets, dedupT improves Mean Reciprocal Rank often by over \(15\%\) compared to the best DL baseline and up to \(9\%\) over traditional methods, while achieving higher ROC-AUC in detecting unique crash reports [2508.19449].

The ablations clarify where these gains come from. Fine-tuning the PLM materially improves correlation and retrieval quality; Parametric Max-Mean is the strongest aggregation mechanism among those tested; and frame-count selection has a nontrivial optimum in the 10–12 range [2508.19449]. The paper also reports that transformer inference is heavier than purely IR methods, though parallelization and cached embeddings mitigate runtime [2508.19449]. This is a limitation rather than a contradiction of the method’s practicality, because dedupT shifts much of the computational burden into offline adaptation and embedding precomputation.

## 6. Relation to adjacent uses of the term and broader deduplication research

The label “dedupT” is not completely unique across the literature. In ASR, Huang et al. describe “dedupT” as a record-deduplication-based pipeline for modeling and biasing the true entity distribution in ASR transcripts, using blocking, pairwise similarity, and graph clustering rather than transformer adaptation over stack traces [2306.06246]. In ontology integration, the term appears as a descriptive name for a deduplication function inspired by OntoMerger, where mappings, source priority, and hierarchy reconnection determine canonical merges in knowledge graphs [2206.02238]. These usages are technically distinct from crash-report deduplication.

Within stack-trace deduplication more specifically, Shibaev et al. present a two-stage system composed of an embedding model with byte-pair encoding and approximate nearest neighbor search, plus a reranker that accounts for repeated frames [2412.14802]. Their evaluation includes Acc@1, ROC-AUC, and inference time, and on SlowOps the two-stage model reaches Acc@1 \(= 0.98\) and ROC-AUC \(= 0.96\) [2412.14802]. A plausible implication is that the field now contains two complementary design lines: candidate-retrieval pipelines with ANN indexing and reranking [2412.14802], and transformer adaptation with cached embeddings and pairwise ranking networks [2508.19449].

dedupT’s reported strengths are holistic modeling of stack traces as contextual sequences rather than isolated frames, domain adaptation via contrastive fine-tuning, a parametric aggregation mechanism that balances the most similar frame pair with global trace context, and significant gains in both ranking and unique-crash detection over traditional and deep-learning baselines [2508.19449]. Its reported limitations are the requirement for labeled duplicate buckets, sensitivity to stack-trace length and trimming policies, and heavier transformer inference than purely IR methods [2508.19449]. Taken together, these properties position dedupT as a technically specific instance of a broader shift toward representation-learning-based deduplication, while preserving the practical objective that motivates the field: reducing manual triage effort in large-scale issue-tracking systems [2508.19449].

Source: https://www.emergentmind.com/topics/dedupt