ESMA: Evolution Strategy for Metacognitive Alignment

• ESMA is a family of frameworks and algorithms that optimize metacognitive alignment in LLMs using evolutionary computation and explicit feedback.
• It employs dual-prompt evaluation, prompt evolution, and rule-based activation steering to integrate factual accuracy with internal self-assessment.
• Experimental results demonstrate boosted type-2 sensitivity, significant reductions in jailbreak rates, and effective sparse parameter updates across various models.

Evolution Strategy for Metacognitive Alignment (ESMA) encompasses a family of frameworks and algorithms leveraging evolutionary computation to optimize metacognitive capabilities in LLMs and related systems. ESMA methods aim to align model behaviors with their internal knowledge or safety principles by iteratively evolving parameter vectors, prompt representations, or rule-based controls, all guided by explicit metacognitive feedback or dual-objective reward functions.

1. Core Concepts and Formal Definitions

Metacognition, defined here as the agent’s awareness or explicit knowledge of its own internal state, underpins ESMA methodologies. The formal foundation is often provided by signal-detection theory, where type-2 metrics quantify a model’s capacity to distinguish between situations where it does or does not know the answer. The canonical measure is the type-2 sensitivity index:

$$d'_{\rm type2} = \Phi^{-1}(\mathrm{Hit\;Rate}) - \Phi^{-1}(\mathrm{False\;Alarm\;Rate})$$

with Hit Rate $$= \Pr(\text{meta} = \text{“Yes”}|\,\text{answer is correct})$$ and False Alarm Rate $$= \Pr(\text{meta} = \text{“Yes”}|\,\text{answer is incorrect})$$, as detailed in (Park et al., 2 Feb 2026). This metric underlies ESMA’s reward structure and evaluation protocols.

The central ESMA objective is to maximize the alignment between a model’s direct outputs (e.g., factual answers) and its self-reported or regulatory signals (e.g., confidence or safety assessment), subject to accuracy and robustness constraints (Park et al., 2 Feb 2026, Qiu et al., 28 Jul 2025, Shan et al., 10 Nov 2025).

2. Algorithmic Frameworks

2.1 Dual-Prompt and Joint-Reward ESMA

In "Fine-Tuning LLMs to Know What They Know" (Park et al., 2 Feb 2026), ESMA is instantiated as a non-differentiable evolutionary algorithm optimizing LLM weights for metacognitive alignment. The process is structured as follows:

1. Dual-Prompt Evaluation: Each test item is issued in two forms:
   • Direct Question: Model produces an answer.
   • Meta Question: Model answers "Do you know...?" with "Yes" or "No".
2. Reward Function: $$R(C, A) = \begin{cases} 2, & C=1,\,A=1 \ 1, & C=1,\,A=0 \ 1, & C=0,\,A=1 \ 0, & C=0,\,A=0 \end{cases}$$ where $C$ encodes factual correctness, $A$ encodes meta-alignment.
3. Evolution Strategy:
   • At each generation $t$, sample $N$ perturbations $\epsilon_i \sim \mathcal N(0, I)$.
   • Generate candidate weights $\theta_i = \theta_t + \sigma \epsilon_i$.
   • Evaluate joint reward $F_i = R(C, A)$ over batches.
   • Standardize rewards, update weights: $$\theta_{t+1} \leftarrow \theta_t + \alpha \frac{1}{N}\sum_{i=1}^N \hat{S}_i \epsilon_i$$ where $\hat{S}_i = (F_i - \mu_F)/\sigma_F$.

Hyperparameters: $\sigma=1\times10^{-3}$, $\alpha=5\times10^{-4}$, $N=32$, $T=750$, batch size $n=256$ (Park et al., 2 Feb 2026).

2.2 Prompt Evolution and Metacognitive Feedback

The MeLA architecture reinterprets ESMA as evolution over prompt representations rather than model weights (Qiu et al., 28 Jul 2025):

• Prompt Genotype: Each prompt $p \in P$ is embedded $\theta = \phi(p)$; evolution operates on $\theta \in \mathbb{R}^d$.
• Fitness: $$H(p) = \mathbb{E}_{s \sim S}\left[\mathrm{Perf}(G(p,s))\right]$$.
• Operators: Gaussian mutation, intermediate recombination, metacognitive correction $e_i$ derived from error diagnosis.
• Update: $\theta_i \mapsto \theta_i + \eta e_i$.

A full $(\mu+\lambda)$–ES cycle is deployed with selection among parent and offspring populations.

2.3 Self-Evolution via Rule Graphs and Activation Steering

MENTOR applies ESMA principles to safety alignment using a continuous cycle that combines:

• Metacognitive Self-Assessment ($M_e$): Generates safety scores $S_i \in \{1,2,3,4,5\}$ and feedback $T_i$ per query-response (Shan et al., 10 Nov 2025).
• Rule Evolution: Dynamic rule graph $R_G$ is incrementally expanded using experience summarizer output.
• Activation Steering: Layer-wise steering vectors $v_\ell$ applied to internal activations: $$a'_\ell(q) = a_\ell(q) + \alpha_s v_{s,\ell} + \alpha_d v_{d,\ell}$$ to bias outputs toward validated regulatory compliance.
• Evaluation Cycle ("MetaLoop"): Repeated response-reflection-revision to minimize jailbreak rate and improve value alignment.

3. Objective Functions, Metrics, and Evaluation

The reward and fitness design in ESMA frameworks targets both direct task performance and explicit metacognitive consistency, preventing degenerate or trivial solutions (such as constant "I don't know" responses).

Key metrics include:

• Type-2 Sensitivity $d'_{\rm type2}$ (Park et al., 2 Feb 2026)
• Alignment Rate $P(A=1)$: Proportion with matching factual and meta responses
• AUC/ROC for confidence discrimination
• Domain-Specific Safety Metrics: e.g., jailbreak rate (fraction of “unsafe” outputs under adversarial testing) (Shan et al., 10 Nov 2025)
• Empirical Success Rate for automatically generated heuristics (Qiu et al., 28 Jul 2025)

Explicit ablation studies demonstrate that:

• Accuracy-only or meta-only rewards are insufficient — only a joint reward enforces both alignment and factual competence (Park et al., 2 Feb 2026).
• Incorporating metacognitive feedback or error diagnosis consistently improves both robustness and solution quality (Qiu et al., 28 Jul 2025).
• Rule evolution and metacognitive reflection lower semantic attack success rates more than static rules alone (Shan et al., 10 Nov 2025).

4. Experimental Results and Generalization

Open-source LLMs (e.g., Qwen2.5, Llama3.2, Gemma3) and closed-source models (GPT 5.2, Claude 4.5, Gemini 3 Flash) all exhibit significant gains in $d'_{\rm type2}$ from ESMA application: e.g., Qwen2.5 3B increases from 0.29 to 1.02 (Park et al., 2 Feb 2026). The calibrated confidence AUC rises to ≈0.75 post-ESMA.

Tables in (Qiu et al., 28 Jul 2025) show that the MeLA/ESMA approach outperforms state-of-the-art baselines (EoH, ReEvo) across four NP-hard domains in both success rate and solution quality.

MENTOR’s ESMA-driven self-evolution reduces average jailbreak rates from ~60.6% to 3.49% over 9,000 domain-specific risk queries, with ablations confirming dual contributions from dynamic rules and the metacognitive meta-loop (Shan et al., 10 Nov 2025). Metacognitive evaluation achieves 79.3% agreement with human labelers and discovers additional latent risks.

Generalization is demonstrated across:

• Formats (integration of “I don’t know” prompt variants)
• Datasets (FreebaseQA, NQ Open, WebQuestions)
• Languages (Chinese, Korean, Spanish, no additional fine-tuning) (Park et al., 2 Feb 2026)
• Unseen task domains (fictional knowledge, high-variance problem settings) (Qiu et al., 28 Jul 2025, Shan et al., 10 Nov 2025)

5. Parameter Efficiency and Sparse Alignment

Analysis of weight changes post-ESMA reveals that the bulk of metacognitive gain is attributable to a small subset of parameters. Reapplying only the top 10% of parameter deltas recovers ≈80% of the full $d'_{\rm type2}$ improvement; the bottom 50% contribute negligibly (Park et al., 2 Feb 2026). This suggests the potential for sparse or low-rank update strategies that target only high-impact subspaces.

In the MeLA context, the embedding update via metacognitive correction vectors $e_i$ effectively focuses search pressure where it is most influential, supporting stable recovery and repair of prompt policies (Qiu et al., 28 Jul 2025).

6. Relationship to Prior and Contemporary Approaches

ESMA contrasts with gradient-based supervised fine-tuning (SFT), which improves metacognitive discrimination only modestly and at the cost of factual accuracy (Park et al., 2 Feb 2026). Unlike SFT, ESMA’s evolutionary paradigm directly optimizes non-differentiable objectives such as cross-prompt coherence and regulatory alignment.

Related paradigms include:

• Metacognitive prompt-search and repair (MeLA) (Qiu et al., 28 Jul 2025)
• Metacognitive self-assessment and dynamic rule consolidation (MENTOR) (Shan et al., 10 Nov 2025)
• Activation steering for robust inference-time control

Ablation and comparative studies within these frameworks systematically attribute gains to the metacognitive alignment mechanisms specific to the evolutionary cycle, not merely to the expansion of solution search space or reward shaping.

7. Outlook and Implications

Empirical findings indicate that even base LLMs possess latent metacognitive structure, which can be robustly amplified via ESMA regimes (Park et al., 2 Feb 2026). The sparsity of required parameter changes and the adaptability of feedback-driven evolution point toward efficient, scalable strategies for integrating metacognition and value alignment in complex generative models.

Future directions involve refining sparse-update dynamics, extending ESMA mechanisms to dynamic rule induction and distributed regulatory frameworks, and further integrating activation steering for safety-critical deployments (Shan et al., 10 Nov 2025). A plausible implication is that ESMA’s architectural separation of metacognitive feedback from base learning may accommodate a wide spectrum of downstream regulatory and alignment tasks across both factual and ethical domains.