Evaluating Active Learning Heuristics for Sequential Diagnosis

Abstract: Given a malfunctioning system, sequential diagnosis aims at identifying the root cause of the failure in terms of abnormally behaving system components. As initial system observations usually do not suffice to deterministically pin down just one explanation of the system's misbehavior, additional system measurements can help to differentiate between possible explanations. The goal is to restrict the space of explanations until there is only one (highly probable) explanation left. To achieve this with a minimal-cost set of measurements, various (active learning) heuristics for selecting the best next measurement have been proposed. We report preliminary results of extensive ongoing experiments with a set of selection heuristics on real-world diagnosis cases. In particular, we try to answer questions such as "Is some heuristic always superior to all others?", "On which factors does the (relative) performance of the particular heuristics depend?" or "Under which circumstances should I use which heuristic?"