SIMCODE: ns-3 Simulation Code Benchmark
- SIMCODE is a benchmark for natural-language to ns-3 simulation code generation, offering 400 tasks across introductory, intermediate, and advanced levels with verified solutions and unit tests.
- It fills a gap in evaluating LLMs by emphasizing functional correctness through metrics like execution accuracy, CodeBLEU, and CodeBERTScore, beyond mere textual similarity.
- The benchmark enables head-to-head comparisons of state-of-the-art models using multiple prompt-engineering strategies, while analyzing failure modes such as missing headers and API mismatches.
Searching arXiv for the SIMCODE paper and closely related benchmark context. SIMCODE is a benchmark for natural-language-to-ns-3 network simulation code generation that evaluates whether LLMs can synthesize executable ns-3 scripts directly from textual task descriptions. It was introduced as the first benchmark focused on this problem, with 400 tasks spanning introductory, intermediate, and advanced difficulty levels, each paired with reference solutions and test cases. The benchmark targets a gap between general-purpose code-generation evaluation and the requirements of discrete-event network simulation, where correctness depends not only on syntactic validity in C++ but also on domain-specific knowledge of telecom standards, event scheduling, helper classes, and tracing APIs (Ahmed et al., 15 Jul 2025).
1. Problem setting and motivation
SIMCODE addresses the problem of automatically constructing ns-3 simulation scripts from high-level natural-language descriptions. In modern networking research, this remains an open challenge because ns-3 programming requires domain expertise that is not adequately captured by generic code-generation benchmarks. Benchmarks such as HumanEval or MBPP emphasize general-purpose languages and typically evaluate surface-level similarity or unit-test correctness, whereas ns-3 tasks are sensitive to small but consequential implementation details such as missing #include headers, incorrect helper classes, and outdated APIs (Ahmed et al., 15 Jul 2025).
The benchmark was designed to fill a specific methodological gap. Prior tools, including GenOnet and LangChain-based pipelines, were described as demonstrating that LLMs can interactively construct 5G or O-RAN scenarios in ns-3, but they do not provide a static, publicly available dataset for reproducible evaluation. SIMCODE was therefore conceived to cover a broad spectrum of networking concepts, provide verified reference solutions and unit-test suites, and enable head-to-head comparison of multiple LLMs under controlled prompting conditions (Ahmed et al., 15 Jul 2025).
This positioning makes SIMCODE a domain-specific benchmark rather than a generic code benchmark with networking examples. A plausible implication is that it is intended to measure the “last mile” from plausible code text to executable simulation artifacts, not merely approximate textual similarity.
2. Benchmark composition and task taxonomy
SIMCODE comprises 400 individual simulation tasks organized into three levels of increasing complexity. The organization is intended to expose both foundational and specialized aspects of ns-3 programming (Ahmed et al., 15 Jul 2025).
| Level | Number of tasks | Characterization |
|---|---|---|
| Introductory | 111 | Point-to-point links, node instantiation, simple topologies, basic UDP/TCP flows |
| Intermediate | 101 | Multi-hop topologies, traffic flows, large-scale Wi-Fi or LTE setups, more intricate helper configurations |
| Advanced | 189 | Custom protocol implementation, performance optimization, CSMA/LAN/mesh, socket programming, MANET/Vanet, matrix or WAN topologies |
The introductory tasks have reference solutions averaging approximately 60 lines of C++ and 3–5 unit tests. Intermediate tasks involve approximately 110 lines and 5–8 tests. Advanced tasks require 150+ lines and 8–10 tests, and include topics such as custom callbacks and new helper classes (Ahmed et al., 15 Jul 2025).
Task classification was based on three stated criteria: solution length, conceptual depth, and origin. Conceptual depth included requirements such as new helper classes or custom callbacks. Origins included ns-3 tutorials, university coursework, and the authors’ novel scenarios. This taxonomy indicates that difficulty is not defined solely by code length, but also by the extent of ns-3-specific reasoning and API orchestration required.
3. Dataset construction and quality assurance
The dataset was built in three stages intended to ensure coverage and correctness. First, simulation tasks were collected from ns-3’s official tutorial repository, undergraduate and graduate networking courses, and bespoke scenarios. The resulting 400 prompts cover routing, scheduling, wireless protocols, congestion control, and related topics (Ahmed et al., 15 Jul 2025).
Second, each task was represented as a prompt–solution–test triplet. The prompts are unambiguous natural-language descriptions of approximately 250 tokens specifying topology, traffic patterns, and metrics to measure, including examples such as packet delivery ratio and throughput. The reference solutions are fully working C++ scripts for ns-3 version 3.41 or later, following best practices such as node creation, helper classes, and tracers, and were verified locally for compile and run correctness. The test cases, described as 5–7 per task, leverage ns-3 tracing and statistic-collection APIs to assert conditions such as “exactly N packets delivered” and “throughput ≥ X Mbps,” covering both normal and edge scenarios (Ahmed et al., 15 Jul 2025).
Third, quality assurance was conducted through dual independent reviews by networking experts. These reviews targeted prompt clarity, solution correctness and efficiency, and validity of the test harnesses. The explicit use of independent review is notable because it frames SIMCODE as a curated evaluation resource rather than a weakly filtered corpus.
4. Evaluation protocol
SIMCODE evaluates three state-of-the-art models and one task-adapted variant. The models are Gemini-2.0-flash, GPT-4.1, Qwen-3, and a fine-tuned GPT-4.1 configuration denoted GPT-4.1 (FT). The fine-tuning setup used 60% of SIMCODE for training, 10% for validation, and 30% for testing, with 2 epochs, batch size , and learning-rate multiplier (Ahmed et al., 15 Jul 2025).
Six prompt-engineering templates were applied to every task:
- Instruction
- Chain-of-Thought (CoT)
- Few-Shot
- ReAct
- Expert Prompt
- Self-Consistency
The benchmark specifies that all prompts cast the model as an ns-3 specialist, ban commentary and markdown, and require C++17 compliance (Ahmed et al., 15 Jul 2025).
Four metrics are used to assess output quality. Execution Accuracy is defined as the fraction of generated scripts that both compile and pass all unit tests. Pass@1 is the fraction of tasks whose first output sample passes all tests. CodeBLEU combines n-gram match, weighted n-grams, AST matching, and data-flow alignment, while CodeBERTScore measures semantic similarity using CodeBERT embeddings between generated code and the reference solution (Ahmed et al., 15 Jul 2025).
The metric design separates executable correctness from textual or structural similarity. This is significant because ns-3 scripts can appear close to a reference implementation while still failing due to API-level or build-level defects.
5. Empirical results
The reported results show that GPT-4.1 is the strongest model in the benchmarked setting. Under CoT prompting, it achieves 29.3% execution accuracy, and its average across all six prompt styles is 27.7%. Qwen-3 follows with an average execution accuracy of 24.3%, while Gemini-2.0 is approximately 15%. Fine-tuned GPT-4.1 reaches 30.6% execution accuracy under the plain instruction prompt, compared with 26.5% for the non-fine-tuned variant under the same prompt (Ahmed et al., 15 Jul 2025).
By difficulty under CoT prompting, the benchmark reports the following execution accuracies:
| Difficulty | GPT-4.1 | Qwen-3 | Gemini-2.0 |
|---|---|---|---|
| Introductory | 36.0% | 31.5% | 21.6% |
| Intermediate | 27.7% | 25.7% | 10.9% |
| Advanced | 25.9% | 23.3% | 14.3% |
The similarity metrics remain comparatively flat: CodeBLEU is reported at approximately $0.72$–$0.77$, and CodeBERTScore at approximately $0.926$–$0.940$. The benchmark therefore argues that execution accuracy reveals functional gaps that simple token overlap cannot capture (Ahmed et al., 15 Jul 2025).
Two broader findings are emphasized. First, model choice matters more than prompt style: CoT yields a modest gain of 2–3 percentage points over bare instructions, but the model-level differences are larger. Second, domain adaptation helps: lightweight fine-tuning yields a percentage-point improvement in execution accuracy. These findings suggest that the dominant bottlenecks are not reducible to prompt phrasing alone.
6. Failure modes and benchmark-specific error structure
The benchmark includes a fine-grained audit of GPT-4.1 failures. Five dominant error categories are reported: missing or outdated headers or modules (48%), API mismatches or renamed methods (21%), incorrect type or pointer usage (10%), access to private or undocumented APIs (7%), and runtime failures (5%). A residual miscellaneous category, including unused variables and CMake errors, accounts for 9% (Ahmed et al., 15 Jul 2025).
These results indicate that the principal failure modes are strongly coupled to the evolving and specialized ns-3 software stack. Advanced Wi-Fi and emerging 5G, NR, and mmWave tasks were especially prone to hallucinated or deprecated header names. By contrast, introductory tasks rarely failed for reasons beyond a missing #include (Ahmed et al., 15 Jul 2025).
This error structure distinguishes SIMCODE from generic code-generation benchmarks. In a general benchmark, an incorrect answer often reflects algorithmic failure or logic error. In SIMCODE, a large fraction of failures arise from build-system, header-resolution, API-versioning, and simulator-internal interface issues. This suggests that the benchmark measures a combination of program synthesis, domain grounding, and environment-specific software maintenance competence.
7. Research significance and future directions
SIMCODE is presented as a foundational step toward evaluating LLMs and research in domain-aware generative systems. Its principal contribution is not merely the dataset size, but the combination of 400 prompts, verified reference solutions, and test harnesses for controlled, reproducible evaluation of natural-language-to-ns-3 code generation (Ahmed et al., 15 Jul 2025).
The reported findings motivate several future directions explicitly identified in the benchmark. These include expansion to next-generation domains such as IoT, 6G, and mmWave; inclusion of other simulators such as OMNeT++ and OPNET and other programming languages; integration of additional metrics addressing performance and scalability; and development of retrieval-augmented and reinforcement-learning loops to correct API errors and missing headers (Ahmed et al., 15 Jul 2025).
The benchmark also points to interactive, compiler-in-the-loop feedback and retrieval of up-to-date API documentation as likely mechanisms for bridging the gap between syntactically plausible code and fully executable artifacts. This suggests that future progress may depend less on generic language-model scaling than on tighter coupling between generation, documentation retrieval, environment validation, and iterative repair.
In sum, SIMCODE defines a reproducible evaluation setting for a domain where textual similarity is an insufficient proxy for success. The benchmark’s results, including GPT-4.1’s 30.6% execution accuracy under fine-tuning and the persistence of header- and API-related failures, indicate both measurable capability and substantial unresolved difficulty in domain-specific code synthesis for network simulation (Ahmed et al., 15 Jul 2025).
