Effectiveness of artificial intelligence in systems-engineering tasks

Establish the effectiveness of artificial intelligence in assisting systems-engineering tasks across relevant engineering contexts and use cases.

Background

The paper situates artificial intelligence for systems engineering within a rapidly expanding research area that includes virtual assistants, design evaluators, human–AI collaboration, and agentic-AI architectures. Despite these advances, the authors characterize the effectiveness of AI assistance for systems-engineering tasks as not yet resolved, indicating a broad open research problem rather than a question answered by the present benchmarking study.

This problem is broader than the paper’s specific evaluation of off-the-shelf LLMs for requirements quality assessment and concerns the reliability and usefulness of AI assistance across systems-engineering activities more generally.

References

Although these are promising advances, effectiveness of AI in assisting SE tasks remains mostly an open research avenue (Topcu et al., 2025; Zhang et al., 2021).

— Two Truths and A Lie? Benchmarking Off-the-Shelf LLMs for Requirements Quality Assessment: Performance, False Alarms, and Misses  (2609.03230 - Shefa et al., 3 Sep 2026) in Section 1, Introduction, p. 1

Collectively, these studies highlight that AI-based requirement quality assessment is an active and promising area of research. However, the literature also points to unresolved questions about how reliably shelf LLMs evaluate engineered-system requirements across established quality criteria.

— Two Truths and A Lie? Benchmarking Off-the-Shelf LLMs for Requirements Quality Assessment: Performance, False Alarms, and Misses  (2609.03230 - Shefa et al., 3 Sep 2026) in Section 2.3, Research Gap, p. 3