Why Do Discussion Comments Have Limited Impact on RCA Performance?

Investigate whether the limited impact of incorporating discussion comments from historical incident reports into the retrieval corpus is attributable to the three hypothesized factors: (1) comments predominantly report outcomes of diagnostic steps that models cannot replicate without access to the same diagnostic services; (2) sparsity of information in incident titles and descriptions hinders linking discussions to target incidents; and (3) discussion threads contain substantial administrative content that is not directly useful for RCA.

Background

The authors evaluate whether adding discussion comments from incident reports improves RCA performance for LLM-based approaches. Empirically, they observe mixed and generally small effects on lexical metrics and negligible changes in semantic similarity, suggesting minimal practical benefits.

They explicitly conjecture reasons for the limited impact, proposing three contributing factors related to the nature of discussion comments, the sparsity of incident titles and descriptions, and the presence of administrative content in discussions. Validating these factors remains necessary to understand and potentially leverage discussions more effectively.