Retrieval Augmented (Knowledge Graph), and Large Language Model-Driven Design Structure Matrix (DSM) Generation of Cyber-Physical Systems

Abstract: We explore the potential of LLMs, Retrieval-Augmented Generation (RAG), and Graph-based RAG (GraphRAG) for generating Design Structure Matrices (DSMs). We test these methods on two distinct use cases -- a power screwdriver and a CubeSat with known architectural references -- evaluating their performance on two key tasks: determining relationships between predefined components, and the more complex challenge of identifying components and their subsequent relationships. We measure the performance by assessing each element of the DSM and overall architecture. Despite design and computational challenges, we identify opportunities for automated DSM generation, with all code publicly available for reproducibility and further feedback from the domain experts.

• The paper proposes automating DSM generation using LLMs, RAG, and GraphRAG, achieving high accuracy, with best-case F1-scores of 0.849–0.909, resulting in significant time and cost savings compared to manual methods.
• The methodology leverages different retrieval strategies and language models, demonstrating that architecture and training configurations often matter more than model size, as reflected in models like mixtral:8x22b and llama3.3:70b.
• Experiments on real-world CPS examples show promise but also highlight the need for careful reference curation and prompt design, with significant computational costs associated with large models.

Motivation and problem statement

The paper addresses the labor-intensive nature of constructing Design Structure Matrices (DSMs) for cyber-physical systems (CPS). Historical examples cited include four months of doctoral work to extract jet-engine interactions, 140 person-hours for an 84-component Xerox iGen3 DSM, and five months at NASA for a Mars Pathfinder technology risk assessment. The authors argue that these costs, combined with integration difficulties between system models and Product Lifecycle Management (PLM) workflows, motivate automated architectural generation. Their proposal is to use LLMs, Retrieval-Augmented Generation (RAG), and Graph-based RAG (GraphRAG) to generate DSMs directly from technical documentation, with two tasks of increasing difficulty: (i) determining relationships between predefined components, and (ii) identifying components themselves before determining their relationships.

Methodology

The pipeline has three steps. First, reference documents are collected from academic search engines and crowdsourced repositories and classified into three types: R1 (peer-reviewed literature), R2 (formal/normative sources such as textbooks and standards), and R3 (informal maker/hobbyist content). Second, one of three language-model configurations is applied: a baseline LLM relying on parametric knowledge only; RAG over chunked document corpora; or GraphRAG, which builds an entity knowledge graph via LLM-driven extraction, partitions it into communities using the Leiden algorithm, and answers queries through community-level map-reduce summarization. Third, results are evaluated against ground-truth architectures.

Evaluation is two-tiered. Local metrics—accuracy, precision, recall, and F1-score—are computed cell-by-cell on the binary DSM, with "unsure" cells excluded. Global structural metrics compare generated matrices to ground truth via edit distance ($d_{L1}$) and spectral distance ($d_{spec}$). For task (ii), component lists are aligned to ground truth using cosine-similarity filtering over name embeddings, and a validator prompt checks output format and component count in a chain-of-thought arrangement.

Five models spanning roughly 14.8B to 1.77T parameters were selected based on GraphRAG indexing capability benchmarked on A Christmas Carol: gpt-4-turbo-preview, gpt-4o, mixtral:8x22b, llama3.3:70b, and deepseek-r1:14b. Compute ranged from consumer laptops (8GB vRAM) to a six-GPU workstation (144GB vRAM) and OpenAI cloud infrastructure.

Use cases

Two systems with established architectural references serve as ground truth: Tilstra's High-Definition DSM of a Black & Decker power screwdriver (42 components, six subsystems, spatial-proximity relationships) and Friedenthal's SysML CubeSat architecture (standardized subsystems, whole-part relationships). The choice is deliberate: most complex CPS are iteratively designed from existing structures rather than de novo, so dissection of documented designs provides a realistic validation target. Each experiment was repeated five times; approximately 500 experiments cover both tasks across models, methods, and reference combinations.

Results

For task (i), model architecture frequently outweighed parameter count. On the power screwdriver's spatial reasoning, mixtral:8x22b achieved the best overall F1-score of 0.849 ± 0.074 despite being far smaller than the GPT-4 variants (~1.77T parameters), and deepseek-r1:14b (14.8B parameters) surpassed llama3.3:70b in accuracy. For the CubeSat's abstract whole-part relationships, llama3.3:70b reached accuracy 0.889 ± 0.000 and F1-score 0.909 ± 0.000, outperforming all other baseline configurations. The best structural alignment was also strong: edit distance of 4.0 ± 0.0 for llama3.3:70b on CubeSat and 16.0 ± 0.0 for mixtral:8x22b on the screwdriver.

A notable negative result concerns RAG: aggregating all three reference classes did not consistently improve performance, and some large models showed plateaued or diminished accuracy with R1-R2-R3 combinations. This non-monotonic relationship between retrieved context volume and classification accuracy indicates that retrieval pipelines require careful curation rather than maximal context. GraphRAG delivered gains in specific configurations—for example, gpt-4-turbo-preview with GraphRAG R2-R3 achieved accuracy 0.800 ± 0.011 on the power screwdriver, and deepseek-r1:14b with GraphRAG R2-R3 reached 0.794 ± 0.014—but not uniformly across models.

Task (ii) proved harder than task (i): top accuracies dropped substantially when models had to identify components first. The embedding-based alignment step improved raw accuracies, but the authors caution that part of this improvement comes from removing unmatched components during alignment rather than genuine relational correctness, so spectral distance and inspection of the aligned matrix are needed alongside headline metrics. The validator prompt generally confirmed outputs without correction, except for deepseek-r1:14b, whose occasional reasoning inconsistencies it remedied—suggesting validator utility may be model-specific.

Limitations and open questions

The paper concedes several constraints explicitly. Validation rests on only two use cases with a single relationship type each (spatial proximity; whole-part), leaving generalizability to functional, logical, or flow relationships untested. Performance is sensitive to prompt design—omitting relationship type or system identity caused significant degradation—and no method for automating prompt construction is provided here. Reference selection depends on domain expertise despite the semi-automated classification scheme, and the mechanism by which specific textual content shifts performance remains unidentified. Computational cost varies widely: cloud inference is fast but costly (e.g., $4.87 per GraphRAG indexing run for gpt-4-turbo-preview), while local GraphRAG indexing took up to 4 hours 11 minutes. Generated architectures are positioned as initial proposals requiring expert verification, not autonomous outputs. Open questions include whether content-level relevance scoring can replace coarse R1/R2/R3 categories, and whether agentic frameworks with programmatic xLM mechanisms can improve multi-granularity architectural generation.

Conclusion

The study demonstrates that LLMs, RAG, and GraphRAG can produce structurally faithful DSMs for documented CPS at moderate accuracy, with best-case F1-scores near 0.85–0.91 on relationship determination between known components. Its central empirical finding—that architecture and training choices can matter more than parameter count—is supported but acknowledged as requiring broader validation. By releasing code, configuration files, and logs, the work establishes a reproducible baseline for automated DSM generation while making clear that expert-in-the-loop refinement remains necessary.