AI Codebase Maturity Model (ACMM)

• ACMM is a framework that defines AI codebase maturity by feedback loop topology rather than autonomy, emphasizing structured instruction files, tests, and metrics.
• The model classifies maturity into five levels—from Assisted to Self-Sustaining—delineating progressive feedback mechanisms that convert human judgment into automated policy.
• By integrating rigorous testing, CI/CD practices, and telemetry, ACMM drives measurable improvements, reduces regressions, and enhances operational reliability.

AI Codebase Maturity Model (ACMM) is a five-level framework that defines maturity not by autonomy, but by feedback loop topology: the specific mechanisms through which a codebase measures, adapts to, and governs the behavior of AI agents. Its unit of analysis is the codebase and its surrounding infrastructure—instruction files, tests, metrics, CI/CD, and telemetry—rather than the organization or the underlying model. ACMM was proposed to explain why many teams plateau at “prompt and review”: they obtain rapid code suggestions, yet still experience regressions, cascading failures, and inconsistent output. The central claim is that “the intelligence of an AI-driven development system resides not in the AI model itself, but in the infrastructure of instructions, tests, metrics, and feedback loops that surround it” (Anderson, 10 Apr 2026).

1. Conceptual orientation

ACMM is explicitly organized around feedback mechanisms. Prompt-and-review is classified as Level 1 because it is open-loop, session-ephemeral, and dependent on human judgment applied manually at integration time. Progression occurs when judgment is moved into durable artifacts: instruction files encode preferences, tests encode trust boundaries, metrics encode priorities, and workflows encode policy. At the highest level, the paper states that “the code is the policy,” and describes the codebase as the AI’s operating manual: every test is a trust constraint, every workflow is a policy, and every metric threshold is a prioritization decision (Anderson, 10 Apr 2026).

The model is sequential and explicitly non-skippable. “You cannot skip levels, and at each level, the thing that unlocks the next one is another feedback mechanism.” This places ACMM in the lineage of CMMI, but with a different organizing principle: feedback topology rather than generic process areas. The paper also positions ACMM in relation to DORA by arguing that deployment frequency, lead time, change failure rate, and MTTR are outcomes of the feedback infrastructure encoded in the codebase. It contrasts this with autonomy-centric models such as Shapiro’s Dark Factory, AIDMM, AI-SDLC, and AI-MM SET, arguing that autonomy is an output of maturity, not its definition. A related maturity-model literature similarly argues that “AI software is still software and needs to be approached from the software development perspective,” which situates ACMM within a broader software-engineering treatment of AI systems (Anderson, 10 Apr 2026, Vakkuri et al., 2021).

2. Five levels and progression logic

The five ACMM levels are defined by their loop structure and by the mechanisms that must exist before the next level becomes possible.

Level	Name	Feedback loop topology
1	Assisted	Open loop
2	Instructed	One-way (human $\rightarrow$ AI)
3	Measured	Two-way with human interpretation
4	Adaptive	Closed loop
5	Self-Sustaining	Multi-loop, self-improving

Level 1, Assisted: Prompt and Review, requires no mechanisms beyond source control and ad hoc prompts. Entry is simply the use of AI suggestions with manual review. Exit depends on repeated corrections and preferences being recognized as patterns worth encoding. The unlocking mechanism from Level 1 to Level 2 is the creation of persistent instruction artifacts such as CLAUDE.md and .github/copilot-instructions.md.

Level 2, Instructed: Encoded Preferences, requires instruction files, Copilot instruction files, PR templates with AI-readable checklists, and component or card development guides that capture common rejection reasons. Entry requires that AI tools read instruction files every session and that PR templates enforce conventions. Exit requires quantitative measurement and test gating. The paper gives minimum measurable gates for this transition: a code coverage gate on PRs with an 80% threshold, CI build and lint gates that run automatically, and at least a baseline of nightly or scheduled suites; the case study reports 32 nightly suites. The unlocking mechanism from Level 2 to Level 3 is therefore measurement plus objective gating.

Level 3, Measured: Feedback Becomes Visible, adds coverage reports and coverage gates on each PR, extensive deterministic test suites, acceptance-rate tracking, telemetry and error monitoring, NPS surveys, and flaky-test detection with weekly review. The cited quality dimensions include unit, integration, E2E, cross-browser nightly, performance, security, nil-safety, i18n, accessibility, and compliance. Exit from Level 3 requires that measured loops be sufficiently reliable and deterministic to support automation: coverage gating must be active on every PR, flaky tests must be driven down, tests must gate merges, and telemetry and acceptance metrics must be acted upon rather than becoming a “dashboard graveyard.” The unlocking mechanism from Level 3 to Level 4 is automated actuation: self-tuning configurations, auto-triage, and auto-repair.

Level 4, Adaptive: Feedback Loops Close Themselves, is the first fully closed-loop stage. Required mechanisms include self-tuning configurations that adjust rotations or weights based on acceptance rates, multi-repo triage loops every 15 minutes, PR monitors every 60 seconds, error recovery with exponential backoff, retry queues, concurrent AI sessions managed via worktrees, escalation workflows, Auto-QA with multi-layer quality checks, and telemetry-driven issue creation. The paper gives concrete examples: categories below 20% acceptance are blocked; accessibility at 62% acceptance was boosted to weight 0.93; the operator category at 8% acceptance was blocked to weight 0. Exit to Level 5 requires end-to-end issue-to-implementation pipelines operating with minimal human execution.

Level 5, Self-Sustaining: Code Is the Model, is described as multi-loop and self-improving. Required mechanisms include community-driven pipelines that turn user issues into implemented changes in under an hour, automated documentation and tutorial generation, multi-agent orchestration across repositories, and weekly self-improvement cycles that update guidance from merged PRs. Entry requires end-to-end closed loops that detect, implement, test, and prepare fixes around the clock without human execution. Human roles shift upward to governance: values, architecture, and priorities remain human responsibilities (Anderson, 10 Apr 2026).

3. Formal feedback-loop model and maturity assessment

ACMM formalizes the development system as a directed multigraph $G = (V, E)$. The node set includes developers $H$, AI assistants $A$, repository Repo, CI/CD pipelines $P$, test suites $T$, coverage gate $G_c$, performance gate $G_p$, reviewers $R$, runtime monitors and telemetry $M$, data store $G = (V, E)$0 for metrics and acceptance, auto-tuning config $G = (V, E)$1, triage queue $G = (V, E)$2, deployment $G = (V, E)$3, and end users $G = (V, E)$4. Edges are typed feedback channels such as review requests, test executions, coverage reports, telemetry alerts, triage assignments, auto-config writes, PR merges, and rollbacks (Anderson, 10 Apr 2026).

Each edge $G = (V, E)$5 has reliability $G = (V, E)$6, latency $G = (V, E)$7, and throughput $G = (V, E)$8. For any cycle $G = (V, E)$9, end-to-end loop reliability is defined as

$H$0

and loop latency as

$H$1

The model therefore treats maturity as a property of operationally effective cycles rather than static artifact presence alone.

The paper also states explicit gate conditions. The trust boundary for gating is $H$2, with a PR coverage gate of $H$3. The determinism threshold is $H$4; the paper highlights that a 15% flake rate is unacceptable in autonomous gating and gives $H$5 as a practical threshold before automating merges. Self-sustaining loop viability requires $H$6 and $H$7, with examples such as $H$8 in the 0.95–0.99 range and SLOs such as $H$9 minutes. A PR is merge-eligible iff all gating tests pass, $A$0, performance gates are satisfied, and the category is not blocked by auto-tuning.

Progression is represented as a strict partial order:

$A$1

Level $A$2 requires both the presence and enforcement of unlocking mechanism $A$3 and sustained effective feedback cycles satisfying the relevant reliability and latency bounds for the loops introduced at Level $A$4. The paper gives these mechanisms explicitly: $A$5 is instruction artifacts, $A$6 is the measurement stack, $A$7 is automated actuation, and $A$8 is the community-to-implementation pipeline plus self-improvement cycles. A common misconception is therefore corrected directly by the model: maturity is not primarily a matter of reducing human involvement, but of constructing reliable, low-latency, testable control loops (Anderson, 10 Apr 2026).

4. Testing, measurement, and adaptive operations

Testing is treated as the primary enabler of progression. The paper’s strongest operational claim is that “Testing—the volume of test cases, the coverage thresholds, and the reliability of test execution—proved to be the single most important investment in the entire journey.” The rationale is explicit: without coverage, changes cannot be gated; without gating, merges and repairs cannot be automated; and without deterministic tests, closed loops become unsafe because false positives and false negatives destabilize automation (Anderson, 10 Apr 2026).

The testing stack described for ACMM includes unit and property-based tests for invariants and edge cases, integration and contract tests for seams such as API handlers, databases, and message buses, end-to-end tests for critical user journeys, cross-browser nightly suites, performance and regression suites with explicit SLOs such as TTFI, security and static analysis such as CodeQL, and accessibility and i18n checks. At Level 3, the paper gives a PR coverage gate of $A$9; proposed heuristics in the same paper recommend $P$0 on PRs and system coverage $P$1 on main at Level 4, and $P$2 on PRs with system coverage $P$3 on main at Level 5. For test reliability, the paper recommends driving gating-test flake rates to $P$4 before merge automation and aiming for $P$5 on critical E2E paths at Level 4 and above.

Operationally, the model couples testing with automation and observability. The paper lists build and lint gates on every push; coverage and performance gates on every PR; E2E suites on PRs; cross-browser and expanded suites nightly; an hourly GA4 error monitor that creates issues on spikes; triage loops every 15 minutes; Auto-QA running four times daily with eight quality layers; PR monitors every 60 seconds; retry-based error recovery with exponential backoff; concurrency limits to prevent interference; and rollback or canary policies with auto-revert on failing post-deploy checks. The significance of these mechanisms is not merely procedural. They convert human judgment into machine-enforceable policy and create the closed-loop preconditions for adaptive and self-sustaining behavior (Anderson, 10 Apr 2026).

5. KubeStellar Console experience report

The empirical basis for ACMM is a four-month experience report on KubeStellar Console, an open-source Kubernetes multi-cluster dashboard built from scratch by a single maintainer using Claude Code (Opus) and GitHub Copilot. The stack included a Go backend, React/TypeScript frontend, Helm, and multi-repo worktrees. By the time of reporting, the system had 63 GitHub Actions workflows, of which 22 were AI-specific; 32 nightly test suites; cross-browser and E2E coverage with 12 parallel shards; 91% code coverage with an 80% PR gate; 33 distinct feedback loops across Levels 2 through 5; bug-to-fix time of roughly 30 minutes; feature-to-implementation time of roughly 60 minutes; a triage loop every 15 minutes across four repositories; a PR monitor every 60 seconds; GA4 error monitoring hourly; and Auto-QA four times daily with eight layers (Anderson, 10 Apr 2026).

The developmental trajectory is central to the model’s argument. Weeks 1–2 corresponded to Level 1: rapid feature creation, but also cascading failures and repeated architectural mistakes. Weeks 3–4 introduced Level 2 artifacts—CLAUDE.md, copilot-instructions.md, and a card development guide that captured approximately 90% of prior PR rejection reasons. Weeks 5–8 represent the transition to Level 3, with major investment in testing and measurement: 32 nightly suites spanning compliance, performance, dashboard health, nil safety, accessibility, i18n, and visual regression; 91% coverage; weekly flaky-test analysis; and acceptance-rate tracking via auto-qa-tuning.json. The paper highlights one reliability fix as decisive: a drag-and-drop E2E test that passed 85% of the time was made deterministic, supporting the claim that “you cannot have autonomous AI development without deterministic tests.”

Weeks 9–12 corresponded to Level 4, where loops were closed: eight-layer Auto-QA, PR-acceptance-driven self-tuning, category blocks and boosts, issue triage every 15 minutes, PR monitoring every 60 seconds, exponential-backoff recovery, retry queues, worktree-based concurrency limits, and GA4-driven issue creation. Weeks 13–16 corresponded to Level 5, in which issues filed overnight were routinely triaged, implemented, tested, and ready by morning; weekly self-improvement updated guidance with rules such as “keep changes under 50 lines”; and automated tutorials and documentation synchronization were generated from screenshots and merged PRs.

The paper also records concrete incidents: an autonomous bug fix for an external user in three hours involving 624 LOC across seven files; resolution of a user misunderstanding in 10 minutes by referencing encoded architecture; detection and closure of a low-quality AI PR in eight minutes; an Auto-QA-discovered i18n fix completed in 11 minutes; and automatic blocking of the operator category after an 8% acceptance rate. These examples are used not as isolated anecdotes, but as evidence that measured and enforced loops can shift the system from reactive assistance to persistent, policy-constrained adaptation (Anderson, 10 Apr 2026).

6. Relation to security, ethics, and enterprise ML maturity literature

ACMM sits within a larger family of AI and ML maturity frameworks, but its emphasis is unusually codebase-centric. For security, the closest directly relevant model in the supplied literature is the Machine Learning Security Maturity Model (MLSMM), a lightweight, domain- and model-agnostic, prescriptive security maturity model for ML-based software components. MLSMM follows the nine-stage ML workflow from Amershi et al.—model requirements, feature engineering, training, evaluation, deployment, monitoring, collection, cleaning, and labeling—and associates practices with lifecycle phases using MITRE ATLAS mitigations. It defines three maturity levels, plus an assessment notion of Level 0 indicating complete lack of activity. The supplied text provides concrete security practices only for the Training phase: Model Hardening (AML.M0003), progressing from best-effort hardening to standardized hardening to proactive, by-design hardening; and Use Ensemble Methods (AML.M0006), progressing from simple ensembles without a security rationale to threat-driven ensemble decisions to continuously shuffled ensembles that avoid information leakage. The same synthesis proposes that ACMM’s security dimension can adopt MLSMM’s levels and stage structure, while noting that the paper text does not enumerate practices for the other eight phases despite stating that 19 practices exist in the prototype (Jedrzejewski et al., 2023).

For ethics and broader quality governance, a separate maturity-model literature argues that current AI ethics guidance often answers “what” rather than “how,” and calls for a maturity model that covers “the entire sphere of technical and ethical quality requirements.” Its Figure 1 describes a progression from Ad hoc (“Actions are unpredictable & unreliable”) through Responsive approach, Proactive approach, and Self-measuring, to Automated (“Ethics are part of process; Actions are planned, optimized & automated”). The same work lists candidate domains such as Understanding stakeholders, Accountability, Data privacy, Fairness, Human Agency, Safety & Security, System Oversight, Transparency, and Wellbeing. This suggests that ACMM’s feedback-loop framing can be extended beyond coding assistance into ethics and governance domains, but that such an extension requires explicit domain metrics and operationalized checklists not yet formalized in that paper (Vakkuri et al., 2021).

A third line of work, on enterprise ML process maturity, reinterprets CMM for the ML model lifecycle and emphasizes process areas that ACMM treats more indirectly: model goal setting and offering management, content and data management, data pipeline, feature pipeline, train pipeline, test and benchmarking pipeline, deployment pipeline, AI operations and continuous improvement, plus cross-cutting fairness, trust, transparency, error analysis, and monitoring. It adds roles such as Offering Manager, Content/Data Manager, Training Lead, Test Lead, Deployment Lead, AI Operations Manager, and Quality Assurance Lead; artifacts such as annotation guidelines, lineage records, deployment runbooks, and model disclosures; and controls such as payload logging, drift detection, and misalignment monitoring. A plausible implication is that ACMM can be interpreted as the codebase-level control topology inside a broader enterprise ML lifecycle, especially when data lineage, retraining governance, and model-scope disclosure become necessary for production systems (Akkiraju et al., 2018).

7. Limitations and open questions

The ACMM paper states several scope limits directly. Its evidence base is a single-project, solo-maintainer case in a dashboard-heavy stack; it is not safety-critical; survivorship bias is possible; and the transferability of the framework to large teams, embedded systems, or safety-critical domains remains an open question. The paper therefore proposes future work in multi-case validation, threshold benchmarking across domains, standardization of instruction file formats and acceptance-tracking schemas, and economic analysis of testing investments relative to gains in MTTR and throughput (Anderson, 10 Apr 2026).

Adjacent maturity-model literature exhibits complementary limitations. MLSMM provides a prescriptive security structure and explicitly states that its initial model contains 19 practices, but the supplied text enumerates only two Training-stage practices and provides no quantitative scoring, weighting, roles, governance mechanisms, or required artifacts beyond questionnaire-based evidence. The ethics-oriented maturity paper provides a strong argument for PDCA, self-measurement, and process-integrated ethics, but does not provide quantitative metrics or formal definitions for fairness, robustness, privacy risk, or transparency. This suggests that ACMM is currently strongest as a formal and operational theory of feedback infrastructure for AI-driven development, and weaker as a complete, empirically validated cross-domain standard spanning security, ethics, enterprise governance, and domain-specific regulation (Jedrzejewski et al., 2023, Vakkuri et al., 2021).