HARP: Hint-Based Adaptive Prioritization

• HARP is a model-based test case prioritization technique that uses developer and manager hints to guide early fault detection in system-level testing.
• It combines human-provided risk signals with adaptive random selection to overcome the absence of historical execution data in MBT.
• Empirical studies confirm that consensual, high-quality hints significantly improve fault detection metrics like APFD and F-Measure compared to baseline ARP.

Hint-Based Adaptive Random Prioritization (HARP) is a model-based test case prioritization technique for system-level Model-Based Testing (MBT) that uses developer and manager knowledge about risky parts of a system to improve early fault detection. It was introduced to address a recurrent MBT condition: in initial and system testing, historical execution data are often unavailable, and purely structural or model-layout information is not a reliable prioritization signal. HARP therefore treats human-provided “hints” about jeopardized or error-prone functionality as a surrogate for fault-proneness and combines them with adaptive random prioritization to construct a test execution order that tends to start near suspicious regions while preserving exploration over the remaining suite (Ouriques et al., 2017).

1. Motivation and problem setting

Test Case Prioritization (TCP) seeks an execution order that favors a testing objective such as fault detection. In the setting considered by HARP, the relevant context is system-level MBT rather than code-based regression testing. The paper’s motivation is that MBT, particularly for initial or system testing, often lacks historical execution information, which prevents the direct use of techniques that depend on previous failures or earlier runs (Ouriques et al., 2017).

The method is also motivated by a negative result about alternative guidance sources: structural or model-layout information alone was found not to be a reliable guide. HARP therefore introduces a different source of signal, namely developer and manager knowledge about parts of the system that were hard to implement, subject to schedule reduction, dependent on risky external or untrusted resources, unstable, or already problematic during development. In this formulation, a hint is not a proof of defectiveness; it is an indication that a particular functionality or behavioral region is more likely to be error-prone.

This design places HARP between history-based TCP and purely structural MBT prioritization. It does not require prior failures, yet it does not discard domain knowledge already available inside the development team. A plausible implication is that HARP is particularly suited to early system-level testing campaigns, where execution history is sparse but implementation knowledge is still accessible.

2. Hints as test purposes

In HARP, a hint is an indication or suggestion about a jeopardized or error-prone portion of the system. The paper operationalizes such hints by translating them manually into test purposes that identify test cases matching the hinted behavior (Ouriques et al., 2017).

The adopted test-purpose notation is adapted from Cartaxo et al. A test purpose is represented as a sequence of strings separated by |, where each string is either a model label or the wildcard *. HARP omits the final accept/reject token from the original notation because the objective is only to identify error-prone regions. A representative example is

$* \;|\; \text{C - Invalid Login} \;|\; *$

which denotes any test case traversing the label C - Invalid Login. In effect, hints are converted into filters over the untreated test suite.

This representation is consequential because it makes the knowledge elicitation process compatible with existing MBT artifacts. Developers and managers need not provide executable test orders or complete fault models; they only identify suspicious use-case regions. The translation into test purposes then creates a formal bridge between informal risk knowledge and the prioritization algorithm.

3. Prioritization algorithm

HARP combines two mechanisms: hint filtering and adaptive random prioritization. Given an untreated test suite $U\_{TS}$ and a set of test purposes $TP$, the algorithm first filters the suite:

$filtTCs \gets filter(U\_{TS}, TP)$

This yields the subset of test cases related to the hints. The first prioritized test case is then selected randomly from that filtered subset:

$first \gets firstChoice(filtTCs)$

This initialization is central to HARP’s design: prioritization begins within the hinted region rather than in an arbitrary part of the model (Ouriques et al., 2017).

After the first choice, HARP iterates until all test cases are prioritized. At each iteration it generates a candidate set, evaluates resemblance between candidates and the already prioritized sequence, selects the next test case, appends it to the prioritized suite, and removes it from the untreated and filtered sets. The candidate-set generation strategy is a modified form of Adaptive Random Prioritization (ARP). A random variable is sampled as

$V \sim U(0,1)$

and, at each draw, if $v < 0.5$ a random test case is taken from the hint-filtered set; otherwise one is taken from the remaining non-hint-related test cases. The candidate set size is capped at 10, following prior ARP work. This mechanism biases exploration toward hint-related regions without collapsing the process into a deterministic hint-only ordering.

Among the candidates, HARP chooses the next test case using a similarity measure derived from Coutinho et al. The similarity between test cases $i$ and $j$ is

$$similarity(i, j) = \frac{nip(i, j) + |sit(i, j)|}{\left(\frac{|i| + |j| + |sdt(i)| + |sdt(j)|}{2}\right)}$$

where $U\_{TS}$0 is the number of identical transition pairs between the two test cases, $U\_{TS}$1 is the set of distinct transitions in test case $U\_{TS}$2, and $U\_{TS}$3 is the set of identical transitions shared by both. In the pseudocode, selection is performed by computing the candidate-to-prioritized similarity matrix, taking the maximum value,

$U\_{TS}$4

and retrieving the corresponding candidate,

$U\_{TS}$5

The resulting procedure starts from hinted behavior, then continues by adaptively spreading execution according to resemblance while keeping a persistent bias toward hint-related cases.

4. Hint collection workflow

Because hints are not assumed to be automatically derivable, the paper proposes a lightweight questionnaire-based collection process. Participants inspect a use case document, underline a step or flow they believe was error-prone, explain why they consider it risky, and have the time and perceived difficulty recorded (Ouriques et al., 2017).

The reasons sought in the questionnaire align with the motivating fault-proneness rationale: inherent complexity, schedule pressure, prior problems, and similar risk indicators. The collected indications are subsequently converted into test purposes and supplied to HARP. This yields a pipeline in which organizational knowledge is elicited in natural language, then formalized into model-level matching criteria.

The reported collection cost is low. In the questionnaire study, participants found the process easy, and the average completion time was about 4.17 minutes for SAFF and 3.67 minutes for TMA. The paper therefore treats hint acquisition as a lightweight activity rather than a major process overhead.

5. Empirical validation

The paper validates HARP through three studies, each addressing a different question about hint quality, feasibility, and effectiveness (Ouriques et al., 2017).

Study	Purpose	Metric
Controlled experiment	Effect of good vs. bad hints	APFD
Questionnaire study	Whether developers/managers can identify fault-related regions	Comparison with fault reports
Industrial case study	HARP versus baseline ARP	F-Measure

In the controlled experiment, the authors manually created two hint categories. Good hints produced filtered test cases that included faults, but not all filtered test cases failed; bad hints produced filtered test cases that did not reveal faults. They evaluated two resemblance functions, the paper’s similarity function and Jaccard distance, over industrial MBT suites, running each setup 1000 times and measuring Average Percentage of Fault Detection (APFD):

$U\_{TS}$6

where $U\_{TS}$7 is the position of the first test case that reveals the $U\_{TS}$8-th fault, $U\_{TS}$9 is the number of faults detected by the suite, and $TP$0 is the number of test cases. Higher APFD indicates earlier fault detection.

In the questionnaire study, participants from two industrial projects identified risky use-case regions, which were then compared with actual fault reports. In 3 out of 4 investigated use cases, a majority of participants indicated a region actually related to a fault. This finding supports the feasibility of collecting useful hints from the development team.

In the industrial case study, HARP was compared with baseline ARP using the questionnaire-derived hints. Each technique was run 1000 times, and the study used F-Measure rather than APFD because there was only one hint per use case and the objective was to emphasize time to first fault. The results favored HARP on both systems: on SAFF the improvement was present but small, whereas on TMA the effect favoring HARP was large.

6. Findings, interpretation, and caveats

The central empirical result is that hints help when they are good. The controlled experiment showed that good hints significantly improve HARP relative to bad hints, and the effect of hint quality was large (Ouriques et al., 2017). By contrast, the choice between the similarity function and Jaccard distance had only small or negligible impact under the same hint quality, suggesting that the dominant factor was the informativeness of the hint rather than the exact resemblance metric.

This finding clarifies the role of hints in the method. HARP is not merely an ARP variant with an extra filter; it is a technique whose value depends on the practical validity of its human guidance. The questionnaire and case study suggest that such guidance is often available and useful, but not uniformly so.

The paper therefore gives an explicit operational caveat: hints should be used only when there is consensus among development team members. A bad hint can mislead HARP and reduce performance. The recommended practice is to use a hint when a majority of developers or managers point to the same risky region, and to require deliberation by managers or testers when there is disagreement. This recommendation is not ancillary; it is a direct consequence of the method’s empirical sensitivity to hint quality.

Taken together, these results position HARP as a hint-driven MBT prioritization technique for settings where historical data are missing but expert knowledge exists. It filters test cases by hinted risky regions, starts execution from those regions, and then continues through similarity-based adaptive random selection. The evidence reported in the paper indicates that good, consensual hints improve fault detection over baseline ARP, while poor hints can degrade it.