Cluster-Aware Dual-Level Test Specification Generation for Large-Scale Automotive Software Requirements

Abstract: Generating test specifications that satisfy Automotive SPICE SWE.6 requirements becomes increasingly challenging and time-consuming as projects scale to thousands of requirements. Because this manual process often consumes weeks of engineering effort, automation becomes a critical necessity. However, standard LLM approaches struggle at scale: processing requirements individually discards vital inter-requirement dependencies, while feeding entire corpora at once exceeds context-window limits, leading to incomplete integration coverage and redundant test cases. This paper presents a novel "Cluster-then-Summarize" pipeline that addresses these limitations through three-stages. Requirements are embedded using sentence transformers and grouped using UMAP dimensionality reduction followed by HDBSCAN density-based clustering. This grouping utilizes an automatic minimum cluster size selection driven by a quality criterion combining normalized Silhouette and Calinski-Harabasz scores. A multi-level map-reduce summarization algorithm then distills each cluster into concise, domain-conformant descriptions while preserving quantitative thresholds and safety integrity levels. The pipeline exploits the derived cluster topology to generate test specifications at two levels: individual requirement verification and cluster-level integration tests that verify cross-requirement feature behavior. A nearby-cluster context mechanism provides bounded cross-feature awareness during each LLM call, and Retrieval-Augmented Generation grounds all outputs in ISO 26262 and ASPICE standards. Evaluation on automotive requirement datasets of varying scale demonstrates that the cluster-aware approach improves integration test coverage and maintains summarization fidelity compared to baseline methods while scaling efficiently to thousands of requirements.

• The paper introduces a five-stage UMAP-HDBSCAN and retrieval-augmented pipeline that uses adaptive clustering, map-reduce summaries, and dual-level test generation to address large-scale automotive requirements.
• Cluster-aware context improved semantic diversity by 4.21%, METEOR by 14.75%, specificity by 8.79%, and reduced overlap errors by 14.65% compared with isolated requirement prompting across seven datasets.
• The dual-level approach generated integration tests covering cross-requirement behaviors, while cluster-aware generation reached 89.59% overall faithfulness and integration tests achieved 91.29% average faithfulness, subject to heuristic evaluation limits.

Motivation and problem statement

Automotive SPICE process area SWE.6 requires that every software requirement be verified through traceable test specifications, and ISO 26262 additionally demands coverage that accounts for Automotive Safety Integrity Levels (ASILs) and inter-requirement dependencies. In practice, a single automotive project may contain 500 to over 5,000 requirements spanning powertrain, ADAS, and V2X domains, and producing complete test specifications remains a largely manual effort consuming weeks of engineering time. The paper by Ayman et al. identifies a specific failure mode in LLM-based automation of this task: processing requirements individually discards the inter-requirement context needed for integration testing, while feeding entire corpora at once exceeds context-window limits. The consequence is incomplete integration coverage and redundant test cases — a direct compliance risk under ASPICE SWE.6.

The authors' central hypothesis is that injecting cluster-derived semantic structure into LLM test generation increases integration coverage (H1), improves grounding (H2), and preserves summarization fidelity (H3). The claimed gap in prior work is precise: existing clustering approaches in requirements engineering organize or analyze requirements but do not exploit cluster topology to drive downstream test generation, and no prior work uses density-based clustering to produce both individual verification tests and cluster-level integration tests.

Pipeline architecture

The proposed "Cluster-then-Summarize" pipeline comprises five stages. Requirements are embedded using all-MiniLM-L6-v2 into 384-dimensional vectors; notably, only the description field is embedded, excluding identifiers and metadata so that adjacent requirement IDs cannot induce spurious lexical clustering. Embeddings are stored in a ChromaDB vector store alongside chunks from ASPICE SWE.6 and ISO 26262 to support retrieval-augmented generation (RAG).

Clustering proceeds in two phases: UMAP dimensionality reduction ($d=15$, $n_{\text{neighbors}}=15$, $\text{min\_dist}=0.0$, cosine input metric) followed by HDBSCAN with Euclidean distance. A distinctive contribution is automatic selection of HDBSCAN's min_cluster_size parameter $\mu$: candidates are drawn from fixed seeds and dataset-size-relative fractions, trial runs are scored with a weighted combination $Q(\mu) = 0.7\,\hat{s}(\mu) + 0.3\,\hat{h}(\mu)$ of min-max normalized Silhouette and Calinski–Harabasz indices, and $\mu^*$ is chosen by maximization, falling back to $\mu = 5$ if no candidate yields a valid multi-cluster partition. Noise points are not discarded but reassigned to the nearest cluster centroid by cosine similarity, yielding a complete partition without singleton inflation. For each cluster, the $\kappa=3$ nearest clusters by centroid distance provide bounded cross-feature context during generation.

Summarization uses a multi-level map-reduce scheme with batch size $B=10$ and merge factor $M=3$, requiring $n_{\text{neighbors}}=15$0 LLM calls per cluster and inheriting the maximum ASIL across cluster members. Test generation then operates at two levels: individual test specifications derived from a six-component context assembly (the requirement itself, its cluster, the cluster summary, nearby-cluster samples capped at $n_{\text{neighbors}}=15$1 requirements, and two RAG sources with a minimum cosine similarity threshold of 0.6), and cluster-level integration tests constrained such that each traces to at least two requirements. Two traceability matrices guarantee every requirement is covered by at least one individual test and, for multi-member clusters, at least one integration test.

Clustering quality across scales

Experiment 1 compares HDBSCAN against K-Means and DBSCAN on five datasets ranging from 33 to 521 requirements, all using UMAP-reduced embeddings for fairness. HDBSCAN achieves the highest Silhouette score on four of five datasets and the lowest noise ratio on four of five. Two results deserve emphasis. First, K-Means' apparent dominance on the largest dataset (Silhouette 0.9577) is an artifact of the elbow search hitting its upper bound of $n_{\text{neighbors}}=15$2, producing artificially tight clusters rather than semantically meaningful groupings — a caution against reading internal metrics uncritically. Second, DBSCAN's superior Calinski–Harabasz index on the larger datasets comes with severe over-fragmentation ($n_{\text{neighbors}}=15$3 and $n_{\text{neighbors}}=15$4), which the authors argue is impractical for downstream test generation. HDBSCAN produces stable, interpretable cluster counts with no manual tuning, supporting its role as the pipeline's structural backbone.

Summarization fidelity

Experiment 2 compares Single-Pass, Map-Reduce ($n_{\text{neighbors}}=15$5, $n_{\text{neighbors}}=15$6), and Recursive summarization against human references using ROUGE-L, BERTScore, and ratings from three ASPICE-experienced engineers (inter-rater ICC = 0.81). Map-Reduce dominates on all fidelity metrics:

Strategy	ROUGE-L	BERTScore	Completeness	Quant. preservation	Conciseness
Single-Pass	0.2934	0.8776	3.19	3.38	4.19
Map-Reduce	0.3793	0.8908	4.30	4.34	3.08
Recursive	0.3439	0.8875	3.98	3.82	3.23

Map-Reduce improves completeness by 1.11 points and quantitative preservation by 0.96 points over Single-Pass. The trade-off is explicit: Single-Pass is most concise, but conciseness costs content fidelity. This result directly supports H3 and addresses the standard objection that abstractive summarization hallucinates — although the evaluation relies on expert Likert judgments rather than automated factuality checks, which bounds the strength of the claim.

Effect of cluster context on test quality

Experiment 3 isolates the effect of cluster context by generating tests twice per requirement: once with the requirement alone (Condition A) and once with cluster summary, membership, and nearby-cluster context (Condition B). Across seven datasets, Condition B yields mean improvements of +4.21% semantic diversity, +14.75% METEOR, +8.79% specificity, and a 14.65% relative reduction in overlap error rate (tests whose similarity margin to the true requirement falls below a noise threshold). The gains are consistent but not uniform: specificity decreases slightly on datasets D5 and D6, which the authors attribute to dense clusters inducing more general wording. This is an honest concession that the context mechanism is not uniformly beneficial and depends on cluster density characteristics.

Dual-level coverage

Experiment 4 compares test counts between an individual-only baseline and the dual-level pipeline. Integration tests add between roughly 3% and 30% additional specifications depending on dataset, with the fraction of requirements mapped to cluster-level tests ranging widely — from 20% (D6) to 85% (D2). Qualitative examples from a Surround-View Automated Parking System dataset illustrate the mechanism: one generated integration test verifies simultaneous zone-level proximity warnings and collision-avoidance priority overriding an active parking maneuver, a compound behavior no single-requirement test can exercise; another verifies that a 200 ms HMI response constraint holds uniformly across all supported languages. These examples substantiate H1 more convincingly than raw counts alone, since they show cross-requirement interactions that isolated prompting structurally cannot capture.

Hallucination and faithfulness

Experiment 5 evaluates grounding with an automated detector covering four hallucination categories (numeric, entity, action-verb, condition), using hierarchical matching against linked requirements, cluster context, and dataset-wide references combined with SBERT thresholds and numeric tolerance checks. Cluster-aware generation reduces the average flag rate by 17.09 percentage points relative to the isolated baseline, reaching 89.59% overall faithfulness. Cluster-level integration tests achieve 91.29% average faithfulness with low hallucination intensity (0.089 items/test). An important caveat is that the detector itself is heuristic — SBERT similarity thresholds and lemmatized matching can both miss and over-flag hallucinations — so these figures should be read as estimates under a particular detection regime rather than ground truth. No human audit of flagged cases is reported.

Limitations and open questions

Several limitations are acknowledged or evident. Evaluation uses seven proprietary automotive datasets without public benchmarks such as PURE, limiting reproducibility and cross-domain comparison. Expert evaluation involves three raters from a single organization, and the hallucination detector's precision and recall are not independently validated. The specificity regressions on dense clusters indicate the nearby-cluster sampling strategy ($n_{\text{neighbors}}=15$7 random sample) may be suboptimal; adaptive context selection for very large clusters remains open. The pipeline also has not been validated through expert-in-the-loop review or integrated with industrial ALM toolchains, so end-to-end compliance claims under ASPICE SWE.6 rest on proxy metrics rather than assessor judgment.

Conclusion

This paper presents a five-stage pipeline that treats clustering as a structural scaffold rather than an analytical endpoint: UMAP+HDBSCAN grouping with adaptive min_cluster_size selection, map-reduce summarization preserving quantitative thresholds and ASIL levels, and dual-level test generation exploiting cluster topology with bounded nearby-cluster context and standards-grounded RAG. Empirical results across seven datasets show consistent improvements in summarization fidelity, test specificity and diversity, overlap-error reduction, and grounding faithfulness, with qualitative evidence that integration tests capture cross-requirement behaviors unreachable by isolated prompting. The work leaves open validation on public benchmarks, independent verification of the hallucination detector, and industrial deployment with continuous traceability updates.