Bug-Report-Driven Fault Localization: Industrial Benchmarking and Lesson Learned at ABB Robotics

Abstract: Software quality assurance remains a major challenge in industrial environments, where large-scale and long-lived systems inevitably accumulate defects. Identifying the location of a fault is often time-consuming and costly, particularly during maintenance phases when developers must rely primarily on textual bug reports rather than complete runtime or code-level context. In this study, we investigated if artificial intelligence can support fault localization using only the natural-language content of bug reports. By relying only on textual information, our approach requires no access to source code, execution traces, or static analysis artifacts, making it directly deployable within existing industrial maintenance workflows. We framed fault localization as a supervised text classification problem and evaluated three traditional machine learning models (Logistic Regression, Support Vector Machine, and Random Forest) and two fine-tuned transformer-based LLMs (RoBERTa-Base and Distil-RoBERTa). Our evaluation used proprietary data from ABB Robotics in Sweden, comprising five years of resolved industrial bug reports, each linked to its verified code fix. This setting allowed us to assess model effectiveness under realistic industrial constraints. Our results showed that traditional models using term frequency-inverse document features consistently outperformed the fine-tuned LLMs on this dataset, while data augmentation improved Random Forest performance. These findings challenge the assumption that transformer-based models universally outperform classical approaches in industrial contexts with domain-specific data. We demonstrated that historical bug reports can be systematically used for text-based, artificial intelligence-assisted fault localization, providing a scalable, low-cost, and empirically grounded complement to common debugging practices in industry.

• The paper demonstrates that TF–IDF-based linear models outperform transformer models for bug-report driven fault localization in industrial settings.
• The methodology involves comprehensive data preprocessing, text augmentation, and supervised ranking with multiple evaluation metrics including Top-1 Accuracy and MRR.
• Results reveal practical insights for deploying computationally efficient, on-premise ML models that integrate seamlessly with industrial bug triage workflows.

Bug-Report-Driven Fault Localization in Industrial Contexts: Benchmarking and Insights at ABB Robotics

Problem Formulation and Practical Motivation

This study addresses the longstanding challenge of fault localization within industrial software maintenance, focusing on environments where diagnostic artifacts (traces, logs, call graphs) are routinely unavailable and developers are constrained to textual bug reports. The authors specifically frame fault localization as a multi-label, supervised ranking problem: given a bug report composed of title and description, predict a ranked shortlist of likely fault locations at the subfolder/component level using only natural language features. This contrasts with the majority of prior research that analyzes open-source settings with auxiliary artifacts and file-level granularity.

The practical motivation is clear: productive maintenance requires actionable recommendations that can be efficiently integrated into existing industrial bug triage workflows without requiring intrusive analysis or breach of intellectual property confidentiality. The reliance on proprietary data from ABB Robotics and the alignment with developer navigation at the subfolder level maximally respect these constraints.

Experimental Methodology

The pipeline comprises data collection, text preprocessing, feature extraction, model selection, training, and ranking-based evaluation. The proprietary dataset consists of 660 resolved bug reports over five years, each annotated with links to verified subfolder-level fixes. Crucial preprocessing steps include de-camelcasing, lemmatization, and normalization, ensuring the models are robust to domain-specific lexical noise.

The study benchmarks:

• Traditional ML classifiers: Logistic Regression (LR), Support Vector Machine (SVM), Random Forest (RF).
• Transformer-based LLMs: RoBERTa-Base and Distil-RoBERTa, fine-tuned on the industrial dataset.

Feature representations include TF–IDF and sentence embeddings for traditional models, while LLMs operate over tokenized input. Data augmentation is carefully deployed to address class imbalance and limited sample size, with both general and targeted strategies using WordNet synonym replacement and random swaps.

Evaluation metrics are grounded in developer-centric workflows: Top-$k$ Accuracy, Recall@$k$, Mean Average Precision (MAP), and Mean Reciprocal Rank (MRR), computed on a strictly held-out test split with multi-label stratification.

Empirical Results

Traditional Models vs Transformers

Key numerical findings:

• TF–IDF based LR (with random swap augmentation) achieves Top-1 Accuracy and MRR of 0.53 and 0.67, respectively, with Top-5 Accuracy approaching 0.86.
• SVM (unaugmented) achieves comparable or better Top-5 accuracy (0.86).
• RF benefits substantially from augmentation (MAP 0.62; Recall@1 0.43).
• Both RoBERTa-Base and Distil-RoBERTa underperform relative to the best traditional baselines, with Top-1 Accuracy around 0.46 and MRR peaking at 0.61.
• Data augmentation generally benefits transformers, but does not close the gap.

TF–IDF decisively outperforms general-purpose sentence embeddings and yields models that are both lightweight and interpretable relative to LLMs. Notably, the marginal benefits of complex models are not realized in this setting, in contradistinction to current expectations from literature based on large open-source corpora.

Statistical Significance and Model Robustness

Performance differences between the strongest models are modest yet consistent across metrics, and the augmentation strategies differentially affect model families: RF's ensemble structure is better suited to the diversity introduced via augmentation, while linear classifiers (LR/SVM) can see slight degradation due to dilution of rare, discriminative terms. The study employs leakage-safe splits and stratification to ensure the validity of comparisons.

Implications and Broader Impact

For Industrial Practice

The results advocate for deployment of TF–IDF based linear models as a primary ML-assisted tool for fault localization in data-constrained, confidentiality-bound industrial maintenance contexts. These methods are deployable on-premise, computationally inexpensive, and avoid the risks inherent to off-prem LLM-based solutions (e.g., leakage of proprietary data, regulatory constraints, or latency). The strong Top-1 and Top-5 metrics indicate these models can function as practical triage aids, narrowing the search space and reducing developer investigation time without automating root cause analysis.

Concretely, organizations can integrate such models within bug tracking or CI/CD systems, using historical bug-fix mappings to continuously retrain and monitor drift, optimizing Top-$k$/MAP for developer-facing workflows.

Theoretical Insights

The null result for transformer models—despite their strong out-of-domain language understanding—highlights the centrality of appropriate feature representations and data characteristics over sheer model complexity in fault localization. In highly specialized, jargon-laden industrial environments, domain-aligned lexical cues can supersede general semantic abstraction.

This finding contradicts the often-implicit hypothesis that transformer architectures universally dominate in text classification and ranking problems, and calls for re-examination of model selection strategies under realistic industrial constraints.

Limitations

The benchmarking is inherently limited by the single-system, single-domain scope, the specific granularity of component-level labels, and unavoidable restrictions on data sharing. The relatively small sample size (660 reports, 31 labels) and pronounced label imbalance further modulate the absolute performance and may inhibit transformers’ ability to learn deep representations. Replication in other industrial settings remains an open requirement for generalization.

Future Directions

The paper points to three salient research frontiers:

1. Multi-system, multi-domain replication to establish the validity of the observed model hierarchy across architectures, reporting conventions, and business domains.
2. Augmentation of text-only models with non-intrusive metadata (commit statistics, defect histories, etc.) under privacy-preserving integration.
3. Human-centered deployment studies quantifying workflow impact, search space reduction, and calibration of abstention/confidence thresholds for practical adoption.

The authors also stress the importance of varying label granularity (from file to architectural component or service) to analyze the precision–recall tradeoffs and potential gains in actionability.

Conclusion

This study presents rigorous evidence that, for bug-report-driven fault localization in industrial environments where only natural language input is available, traditional TF–IDF-based linear models (LR/SVM) are not only competitive but outperform state-of-the-art transformer-based solutions under practical data and deployment constraints. Data augmentation, especially under label imbalance, can yield additional gains for ensemble models. The research provides a reproducible methodology (excluding proprietary data), offering a scalable path for ML-assisted fault triage in high-assurance domains and contributing actionable guidance for industrial technology selection (2604.25700).

The separation between model complexity and deployment utility identified here is likely to persist as a key consideration in the evolution of AI-augmented maintenance tools for critical and long-lived software systems.