Neurosymbolic Auditing of Natural-Language Software Requirements

Abstract: Natural-language software requirements are often ambiguous, inconsistent, and underspecified; in safety-critical domains, these defects propagate into formal models that verify the wrong specification and into implementations that ship unsafe behavior. We show that LLMs, equipped with an SMT solver, can audit such requirements: translating them into formal logic, detecting ambiguity through stochastic variation in the generated formalization, and exposing inconsistency, vacuousness, and safety violations through solver queries on the resulting specification. We present VERIMED, a neurosymbolic pipeline that operationalizes this idea for medical-device software requirements, and report two findings. First, stochastic variation across independent formalizations is a signal of ambiguity: requirements that admit multiple plausible interpretations produce SMT-inequivalent formalizations, and bidirectional SMT equivalence checking turns this disagreement into a solver-checkable test. Second, the usefulness of symbolic feedback depends on its granularity: in counterexample-guided repair on a hemodialysis question-answering benchmark, concrete SMT counterexamples raise verified accuracy from 55.4% to 98.5%. Over an extensive experimental evaluation on open-source hemodialysis safety requirements, we show that the LLM-based approach in VERIMED successfully reduces ambiguity-sensitive requirements and enables rigorous auditing of software requirements through SMT-based queries.

• The paper introduces VeriMed, a neurosymbolic pipeline that integrates LLM autoformalization with solver-level audits to detect semantic defects in software requirements.
• The methodology employs repeated stochastic formalizations with SMT equivalence checks to resolve ambiguity, reducing detection uncertainty from 18.8% to zero.
• Evaluation on medical device requirements demonstrates robust fault detection with 98.4% defect identification and improved repair accuracy using concrete counterexamples.

Neurosymbolic Auditing of Natural-Language Software Requirements: An Expert Analysis

Motivation and Problem Statement

Natural-language software requirements, especially in safety-critical domains such as medical devices, are prone to ambiguity, inconsistency, and underspecification, which propagate into erroneous formal models and unsafe implementations. Traditional LLM-driven autoformalization methods often fail to detect such semantic defects, silently defaulting to statistically prevalent readings or omitting tacit constraints. The lack of ground-truth formalizations renders correctness verification non-trivial: syntactic validity alone does not guarantee that the specification faithfully implements intent.

VeriMed: Architecture and Methodological Contributions

VeriMed is introduced as a neurosymbolic pipeline that integrates LLM-powered autoformalization with solver-level audits. The framework operates through four stages:

1. Ambiguity Resolver: Screens requirements for ambiguous semantics by sampling multiple stochastic formalizations and comparing them with bidirectional SMT equivalence. Disagreement signals ambiguity, and solver-derived counterexample witnesses guide iterative clarification.
2. Autoformalizer: Translates the clarified requirements, in conjunction with a canonical typed schema and domain constraints, into an integrated SMT-LIB model, ensuring composability and joint analyzability.
3. Property Verifier: Executes four audit queries—global consistency, vacuousness, violatability, and redundancy—over the assembled SMT model to localize structural defects.
4. Query Verifier: Verifies scenario-based safety questions against the model, using solver feedback (symbolic and concrete counterexamples) to drive counterexample-guided repair (CEGR).

   Figure 1: Architecture of VeriMed, showing the ambiguity resolver, autoformalizer, and downstream audit and query modules.

Requirement-Level Auditing and Mutation Detection

VeriMed's formalization pipeline was evaluated on the hemodialysis machine case study (64 safety requirements), with additional generalizability testing on PCA pump requirements. Audit queries were implemented as SMT checks:

• Global Consistency: Detects conflicting subsets via unsat core extraction
• Vacuousness: Flags unreachable triggers by unsatisfiable guards
• Violatability: Identifies breakable requirements through satisfiable negations
• Redundancy: Localizes subsumed requirements by unsatisfiable negation checks under the reduced model

Parse success rate was 100%, round-trip equivalence held for all requirements (mean cosine similarity 96.2%), and audit detected 97.3% of single-requirement mutations, with detailed fault localization. Notably, two requirements were flagged as redundant, subsumed by weaker or more restrictive neighboring requirements. Generalizability experiments on PCA pump requirements similarly achieved full round-trip equivalence after bounded repair.

Figure 2: Mutation Detection outcomes by fault type, with near-complete coverage except for a schema-insensitive value mismatch.

Ambiguity Detection and Resolution

A novel contribution is ambiguity detection via repeated autoformalization: the stochastic sampling of N (=5) LLM-generated formalizations per requirement, followed by pairwise SMT-equivalence checking. Persistent disagreement indicates multiple plausible readings, and solver-provided witnesses identify concrete semantic divergences (e.g., differing boundary interpretations for numeric ranges).

Figure 3: Ambiguity-resolution loop for the dialysate-temperature requirement, illustrating boundary interpretation variations and iterative clarification driven by solver-generated witnesses.

Empirically, 18.8% of requirements were ambiguity-sensitive; SMT-guided clarification reduced this to zero, with full pairwise agreement achieved across all clarified cases. SMT disagreement robustly surfaced underspecified boundaries and overlapping numeric ranges, informing minimal requirement rewriting.

Figure 4: Ambiguity detection and reduction statistics, showing convergence to a single encoding after iterative clarification.

Counterexample-Guided Repair (CEGR) and Auditor Feedback

The CEGR experiment revealed that concrete SMT-generated counterexamples are a powerful repair signal. On a bespoke question-answering benchmark, full CEGR raised accuracy from 55.4% (baseline) to 98.5%; requirement-only symbolic feedback plateaued at 80.0%. The repair success rate gap (97.1% vs 59.4%) conclusively showed that only feedback containing concrete violating assignments enables precise localization and correction of errors.

Figure 5: Impact of verifier feedback type on iterative question answering and cumulative accuracy.

Fault Detection Beyond Consistency

End-to-end evaluation showed that 98.4% of manually injected requirement-level defects were detected, even when global consistency was preserved. Behavioral preservation, not consistency alone, was the distinguishing signal: requirement mutations that altered safety behavior, while remaining mutually satisfiable, were flagged through bidirectional encoding comparison.

Theoretical and Practical Implications

VeriMed demonstrates that:

• Requirement-level audits are effective for surfacing semantic defects and structural ambiguity in LLM-generated formalizations.
• SMT-guided clarification eliminates ambiguity and enables rigorous downstream verification.
• Concrete counterexamples drive highly effective repair, outperforming symbolic-only feedback signals.
• Consistency checks are insufficient; semantic agreement between independently formalized requirements exposes behavioral drift.

Practically, this enables rigorous auditing of natural-language software requirements prior to formal implementation or model checking. The approach is not domain-specific in principle, though evaluation primarily targets medical devices; extension to automated schema construction and richer temporal logics (e.g., temporal SMT or TLA+) is proposed for future work.

Limitations

• Generalizability: Empirical validation is limited to medical device domains (hemodialysis, PCA pump), and no claim is made beyond these classes.
• Faithfulness: Proxy signals (consistency, round-trip equivalence, mutation detection) are used but are not sufficient for verifying full intent alignment.
• Temporal Expressiveness: Only present-state properties are formalized; full temporal logic support remains an open challenge.
• Scalability: Industrial requirements documents are significantly larger; deployment in safety-critical contexts requires independent expert review.

Conclusion

VeriMed operationalizes neurosymbolic auditing for natural-language software requirements via LLM autoformalization, solver-level audits, and ambiguity detection through repeated formalization. SMT-based feedback, particularly concrete counterexamples, is shown to be critical for verifiable repair and requirement clarification. The pipeline achieves robust semantic auditability and ambiguity resolution, informing translation from informal natural-language requirements to verifiable formal specifications. Extension to richer formalisms and automated schema induction is a promising avenue for future research.