SpecDetect4AI: AI Code Smell Detector

• SpecDetect4AI is a specification-driven framework that defines AI-specific code smells using a high-level DSL and scalable static analysis.
• It achieves high detection performance with 88.66% precision and 88.89% recall by applying 22 smell rules across extensive AI codebases.
• The framework extends to LLM integration with SpecDetect4LLM, illustrating how targeted rules capture robustness and performance issues in LLM systems.

SpecDetect4AI is a tool-based approach for the specification and detection of AI-specific code smells at scale. It addresses the gap between generic static analysis tools and the semantic realities of AI software systems by combining a high-level declarative Domain-Specific Language (DSL) for rule specification with an extensible static analysis tool that interprets and detects these rules for AI-based systems (Mahmoudi et al., 24 Sep 2025). In the reported evaluation, SpecDetect4AI specified 22 AI-specific code smells and was applied to 826 AI-based systems comprising 20M lines of code, achieving a precision of 88.66% and a recall of 88.89%; a later extension, SpecDetect4LLM, adapted the framework to source-code-level code smells specific to LLM inference and reported that 60.50% of 200 analyzed open-source LLM systems contained at least one such smell, with an average detection precision of 86.06% (Mahmoudi et al., 24 Sep 2025, Mahmoudi et al., 19 Dec 2025).

1. Conceptual scope and problem setting

SpecDetect4AI targets AI-specific code smells, defined as recurring patterns in the code that may indicate deeper problems such as unreproducibility, silent failures, poor model generalization, performance inefficiency, and maintainability problems (Mahmoudi et al., 24 Sep 2025). The framework follows Fowler’s code-smell notion, but specializes it to AI-based systems, where important defects are often semantic and pipeline-aware rather than merely syntactic. The underlying claim is that general-purpose linters such as Pylint and flake8, and more specialized tools such as iSMELL, do not understand AI-specific intent, including whether a split occurs before preprocessing, whether random seeds are set, whether model evaluation is done in the right order, whether a training loop is missing early stopping, or whether a tensor operation uses inefficient manual tiling instead of broadcasting (Mahmoudi et al., 24 Sep 2025).

The framework is organized around a distinction between generic programming defects and AI-specific recurrent practices. In the 2025 formulation, the smell catalog is grouped by effect into correctness smells, efficiency smells, and maintainability smells (Mahmoudi et al., 24 Sep 2025). In the 2025 LLM extension, the same general philosophy is applied to LLM-integrating systems, where the relevant quality attributes are maintainability, reliability, performance, and robustness, and where the target is explicitly source-code-level code smells specific to LLM inference calls (Mahmoudi et al., 19 Dec 2025).

A plausible implication is that SpecDetect4AI occupies a software-engineering niche distinct from detectors of AI-generated text or images. The framework is concerned with source-code analysis of AI systems rather than forensics on AI outputs. This distinction is important because the name is close to other “SpecDetect” systems in the literature, including a training-free, frequency-domain detector for LLM-generated text (Luo et al., 15 Aug 2025).

2. Declarative design and static-analysis architecture

The central design of SpecDetect4AI is a three-step pipeline consisting of specification, matcher generation, and detection (Mahmoudi et al., 24 Sep 2025). In the specification step, a human author writes a smell as a declarative rule in the DSL. The rule is expressed as a first-order logical condition over reusable semantic predicates. The paper states that SpecDetect4AI provides 52 reusable semantic predicates that hide AST detail and expose AI-relevant concepts such as isMLMethodCall(call), hasExplicitHyperparameters(call), and usedBeforeTrainTestSplit(...) (Mahmoudi et al., 24 Sep 2025).

The DSL grammar is explicitly given as:

rule_definition: "rule" IDENTIFIER STRING ":" "condition:" cond_expr "action:" action_expr cond_expr: "exists" IDENTIFIER "in" "AST" ":" "(" cond_expr ")" | predicate_call | "not" cond_expr | cond_expr "and" cond_expr | cond_expr "or" cond_expr | "(" cond_expr ")" predicate_call: IDENTIFIER "(" argument_list? ")" action_expr: "report" STRING

A rule template is described as:

rule RULE_ID "Name of the smell": condition : <boolean expression over predicates> action : report <information and location of the smell>

The matcher-generation step parses the DSL rule using Lark and compiles it into a matcher file. The detection step then parses each Python file into an AST using Python’s built-in ast module, loads the generated matcher, and executes all rule matchers against the AST (Mahmoudi et al., 24 Sep 2025). The implementation is explicitly static-analysis-based and AST-oriented, but the authors emphasize that the framework detects undesirable coding patterns through declarative, domain-specific rules rather than by directly manipulating the AST (Mahmoudi et al., 19 Dec 2025).

A notable scalability detail is the execution model for matchers. Matchers are run in parallel on a cached AST, so analysis time is bounded by the slowest rule rather than the sum of all rules:

$$T = \max_i T_i \quad \text{vs.} \quad T = \sum_i T_i$$

This architectural choice is presented as one of the reasons the tool remains efficient on large corpora (Mahmoudi et al., 24 Sep 2025).

3. Rule language, semantic predicates, and smell catalog

SpecDetect4AI’s rule language is designed to let rule authors express intent rather than AST mechanics (Mahmoudi et al., 24 Sep 2025). The illustrative example in the paper is rule R5, “Hyperparameter Not Explicitly Set”:

rule R5 "Hyperparameter Not Explicitly Set": condition: exists call in AST: ( isMLMethodCall(call) and not hasExplicitHyperparameters(call) ) action: report "Hyperparameter not explicitly set at line{lineno}"

The generated matcher for such a rule is described as an executable AST traversal that checks the semantic predicates and appends a diagnosis to the problem list if the condition is met (Mahmoudi et al., 24 Sep 2025). This rule-compilation scheme is presented as a reason for maintainability and extensibility, especially when compared with analyzers that implement smell rules as handwritten AST logic.

The 2025 paper reports 22 AI-specific code smells, though the excerpt fully names only a subset (Mahmoudi et al., 24 Sep 2025). The named smells include:

• R1 Broadcasting Feature Not Used
• R2 Randomness Uncontrolled
• R4 Training / Evaluation Mode Improper Toggling
• R5 Hyperparameter Not Explicitly Set
• R6 Deterministic Algorithm Option Not Used
• R7 Missing the Mask of Invalid Value
• R9 Gradients Not Cleared before Backward Propagation
• R11 Data Leakage
• R12 Matrix Multiplication API Misused
• R13 Empty Column Misinitialization
• R14 DataFrame Conversion API Misuse
• R17 Unnecessary Iteration
• R18 NaN Equivalence Comparison Misused
• R19 Threshold Validation Metrics Count
• R20 Chain Indexing
• R21 Columns and DataType Not Explicitly Set
• R22 No Scaling Before Scale-Sensitive Operations

The paper further states that 15 of the 22 smells are generic rather than framework-specific, and that correctness smells dominate the empirical findings (Mahmoudi et al., 24 Sep 2025). Some smells are intrinsically more difficult for static analysis. The framework is reported to be strong on most rules, but R7 and R19 are described as difficult because they are more flow-sensitive and value-sensitive and may involve helper functions or dependencies beyond simple intra-file static reasoning (Mahmoudi et al., 24 Sep 2025).

The LLM extension introduced a separate catalog of five LLM code smells (Mahmoudi et al., 19 Dec 2025). These are:

Smell	Abbreviation	Effect categories
Unbounded Max Metrics	UMM	RO, P, M
No Model Version Pinning	NMVP	R, M
No System Message	NSM	R
No Structured Output	NSO	R
LLM Temperature Not Explicitly Set	TNES	R, M

The effect categories used in the LLM study are Robustness (RO), Performance (P), Maintainability (M), and Reliability (R) (Mahmoudi et al., 19 Dec 2025). Each smell is defined in terms of recurring source-code patterns around LLM inference, along with context, problem, and remediation advice.

4. Empirical evaluation on AI-based systems

The main evaluation corpus for SpecDetect4AI consists of 826 GitHub AI-based systems, 142,747 Python files, and 20,454,021 lines of code, with a total corpus size of 29.44 GB (Mahmoudi et al., 24 Sep 2025). The corpus was selected using metadata filters: tagged AI / ML / DL, updated since April 2023, at least 100 stars, at least 20 forks, and primarily Python; the authors also added 87 systems from prior literature (Mahmoudi et al., 24 Sep 2025).

Because exhaustive annotation was infeasible, the paper constructed a representative labeled sample around mlflow, described as a deep-labeled system with 643 Python files, 120K LOC, and 19.7K stars (Mahmoudi et al., 24 Sep 2025). From this system, 241 files and 47,561 LOC were sampled using stratification by directory or functionality, file-size quartile, and test versus non-test status. Annotation was carried out by three Master’s-level annotators with blind double labeling and consensus resolution, achieving Fleiss’ kappa:

$\kappa = 0.85$

This is interpreted in the paper as near-perfect agreement (Mahmoudi et al., 24 Sep 2025).

The headline detection results are as follows (Mahmoudi et al., 24 Sep 2025):

Metric	SpecDetect4AI
Precision	88.66%
Recall	88.89%
F1	88.77%
True positives	344
False positives	44
False negatives	43
Median time per file	0.22 s

On the full corpus, SpecDetect4AI detected 86,910 smell instances across 25,311 files (Mahmoudi et al., 24 Sep 2025). Additional descriptive statistics reported in the paper include an average of 3.45 smells per file, a median of 2, a maximum of 477, 54 smell-free systems, and one most-affected system with 5,350 smell instances (Mahmoudi et al., 24 Sep 2025). The Gini coefficient of smell concentration is reported as 0.78, and 85% of systems had at least two smell types, while only 7% were completely clean (Mahmoudi et al., 24 Sep 2025).

The category distribution is also reported (Mahmoudi et al., 24 Sep 2025):

• Correctness smells: 83.45% of all detections
• Maintainability smells: 8.39%
• Efficiency smells: 8.15%

The top three smells are R2 Randomness Uncontrolled, R11 Data Leakage, and R6 Deterministic Algorithm Option Not Used, together accounting for 68.38% of all detections (Mahmoudi et al., 24 Sep 2025). Smell count is positively correlated with system size, with LOC versus smell count reported as:

$r = 0.71,\quad p < 0.001$

This suggests that larger AI systems tend to accumulate more smells, although the paper does not claim that size alone explains the phenomenon (Mahmoudi et al., 24 Sep 2025).

5. Comparative performance, scalability, and extensibility

SpecDetect4AI is compared with CodeSmile, mlpylint, and three LLM-based baselines: deepseek-r1, llama3-70b-8192, and gpt-4.1-mini (Mahmoudi et al., 24 Sep 2025). CodeSmile covers 12 of the 22 smells; mlpylint covers 20 of the 22 smells. The LLMs were supplied with the smell name, canonical definition, examples, and target snippet in order to make the comparison reproducible (Mahmoudi et al., 24 Sep 2025).

The comparative results reported are:

Tool	Precision	Recall	F1	Median time per file
SpecDetect4AI	88.66%	88.89%	88.77%	0.22 s
mlpylint	91.03%	36.57%	52.17%	0.51 s
CodeSmile	82.13%	75.79%	78.83%	0.74 s
deepseek-r1	55.33%	48.32%	51.59%	17.54 s/file
gpt-4.1-mini	49.70%	42.12%	45.59%	28.37 s/file
llama3-70b-8192	41.20%	32.04%	36.05%	9.63 s/file

The paper describes mlpylint as conservative, with high precision but poor recall, and argues that SpecDetect4AI provides the best balance (Mahmoudi et al., 24 Sep 2025). A Wilcoxon signed-rank test on per-rule F1 is reported as:

$W = 0,\quad p = 0.011$

This is presented as indicating statistically significant superiority (Mahmoudi et al., 24 Sep 2025).

Scalability is reported in corpus-level runtime terms. SpecDetect4AI analyzed all 826 systems in 15,180.6 seconds, approximately 4 hours 13 minutes (Mahmoudi et al., 24 Sep 2025). Additional runtime statistics include an average per system of 18.37 s, a median per system of 3.78 s, a 90th percentile of 6.20 s, a 99th percentile of 14.5 s, and a mean time per 1,000 LOC of 0.19 s (Mahmoudi et al., 24 Sep 2025). The paper also reports a modest correlation between LOC and runtime:

$r = 0.30,\quad p < 10^{-16}$

This is used to argue that AST complexity and dynamic imports matter more than raw size (Mahmoudi et al., 24 Sep 2025).

Extensibility was studied through two scenarios: reuse of existing predicates and introduction of a new predicate (Mahmoudi et al., 24 Sep 2025). Five participants—three Master’s-level students and two industry experts, none with prior exposure to the DSL—were tasked with specifying smells such as “Index Column Not Explicitly Set in DataFrame Read” and “Early Stopping Not Used in Model.fit.” The reported conclusion is that the DSL roughly halves effort in the predicate-reuse scenario and preserves the advantage when new predicates must be introduced. The associated usability result is:

• SUS = 81.7/100

This is described as excellent usability (Mahmoudi et al., 24 Sep 2025).

6. Extension to LLM-integrating systems: SpecDetect4LLM

The paper “Specification and Detection of LLM Code Smells” extends the framework into a versioned module called SpecDetect4LLM, described as an extension of SpecDetect4AI that adds LLM-specific semantic predicates and detection rules (Mahmoudi et al., 19 Dec 2025). The authors selected SpecDetect4AI because it was already open source, accurate relative to prior baselines, extensible through specifications, and reproducible (Mahmoudi et al., 19 Dec 2025).

The extension preserves the specification-driven methodology:

1. SpecDetect4AI is the base tool.
2. It detects AI-specific smells using DSL-defined rules.
3. These rules rely on semantic predicates rather than direct AST manipulation.
4. The authors extended it by adding LLM-specific detection rules, introducing reusable semantic predicates for common LLM integration idioms, and packaging the result as SpecDetect4LLM with tests and documentation (Mahmoudi et al., 19 Dec 2025).

The validation dataset for this extension contains 200 open-source Python LLM-integrating systems (Mahmoudi et al., 19 Dec 2025). Of these, 100 were collected via the GitHub API using LLM keywords, Python filtering, popularity thresholds (stars > 20), and recency, with LLM integration verified by parsing dependency files such as requirements.txt; 100 additional systems were taken from Shao et al., then merged with overlap removal and metadata harmonization (Mahmoudi et al., 19 Dec 2025).

The empirical procedure had two parts: running SpecDetect4LLM on the full dataset to obtain prevalence counts, and manually auditing a stratified sample of detections to estimate precision (Mahmoudi et al., 19 Dec 2025). For each smell, the sampling continued until there were at least 20 detections spanning at least 5 distinct systems. The audit used a predefined decision sheet, labeled instances as TP or FP, and computed Cohen’s $\kappa$ for agreement, although the exact $\kappa$ value is not reproduced in the excerpt (Mahmoudi et al., 19 Dec 2025). Recall was explicitly not estimated.

The main results are:

• 6,337 alerts across 60.50% of systems
• 121/200 projects contained at least one LLM code smell
• 86.06% average detection precision across smells (Mahmoudi et al., 19 Dec 2025)

Per-smell prevalence and precision are reported as follows (Mahmoudi et al., 19 Dec 2025):

Smell	Systems affected	Prevalence	Precision
UMM	76/200	38.00%	86.36%
NMVP	72/200	36.00%	95.65%
NSM	69/200	34.50%	85.71%
NSO	81/200	40.50%	80.00%
TNES	73/200	36.50%	82.61%

The paper highlights three patterns: NMVP produced the most raw alerts, NSO and UMM affected the largest share of projects, and TNES and NSM reflected common integration habits (Mahmoudi et al., 19 Dec 2025). A plausible implication is that the extension demonstrates the portability of the base framework’s declarative architecture: once AI-specific source-code semantics are modeled as predicates, the same engine can be specialized to rapidly evolving subdomains such as LLM integration.

7. Limitations, interpretation, and relation to adjacent “SpecDetect” work

The main SpecDetect4AI paper frames the analyzer as “soundy,” prioritizing practical precision, speed, and extensibility over full soundness (Mahmoudi et al., 24 Sep 2025). Several limitations are noted. Internally, the ground truth depends heavily on a sample centered on mlflow, though this is mitigated through stratified sampling and high inter-annotator agreement (Mahmoudi et al., 24 Sep 2025). Externally, the evaluated corpus is large and heterogeneous but entirely Python-based; the tool is language-agnostic in principle, yet the implementation studied is Python-based (Mahmoudi et al., 24 Sep 2025). Construct-validity limits arise because some smells are difficult to detect statically, especially where cross-file, flow-sensitive, or value-sensitive reasoning is required (Mahmoudi et al., 24 Sep 2025).

The LLM extension adds further limitations: recall was not measured; precision estimates rely on a relatively small sample per smell; the study focuses on Python; and static-analysis limitations remain because execution-dependent cases may be missed (Mahmoudi et al., 19 Dec 2025). The catalog is also intentionally conservative, containing only smells with clear detection criteria, cross-source support, and actionable remediation (Mahmoudi et al., 19 Dec 2025).

Within the broader literature, SpecDetect4AI is situated against general code smells, ML-specific code smells, defect taxonomies for LLM-based autonomous agents, higher-level defect taxonomies for LLM-enabled and RAG systems, prompt smell taxonomies, LLM inference benchmarking anti-patterns, and defects in LLM-generated code (Mahmoudi et al., 19 Dec 2025). The distinctive claim is its focus on source-code-level recurrent practices in human-written AI integration code, together with a declarative rule-specification mechanism (Mahmoudi et al., 24 Sep 2025, Mahmoudi et al., 19 Dec 2025).

A frequent misconception is to conflate SpecDetect4AI with output-forensics systems for AI-generated text or images. The literature included here contains several nearby but distinct methods: “SpecDetect: Simple, Fast, and Training-Free Detection of LLM-Generated Text via Spectral Analysis” is a training-free detector built on DFT total energy of token log-probability signals (Luo et al., 15 Aug 2025), while other studies investigate AI-generated image detection through spectral learning or spectral-semantic fusion (Karageorgiou et al., 2024, Hossain et al., 25 Dec 2025). These works address authenticity detection of AI outputs, not AI-specific code smells in source code. The shared lexical root “SpecDetect” therefore names different research trajectories, and in the software-engineering context the term denotes a declarative static-analysis framework rather than a spectral detector.

Taken together, the available work defines SpecDetect4AI as a specification-driven, predicate-based static analysis framework for AI software engineering. Its core contribution is methodological: smell rules are expressed declaratively, compiled into executable matchers, and evaluated at scale over real AI codebases. The reported results suggest that this approach is both practically effective and extensible, while the SpecDetect4LLM module indicates that the framework can be specialized to emerging AI programming domains without abandoning the underlying rule-and-predicate architecture (Mahmoudi et al., 24 Sep 2025, Mahmoudi et al., 19 Dec 2025).