Self-Guidance: Internal Control in AI

• Self-guidance is a design pattern where models use endogenous signals—such as internal representations and self-generated artifacts—to self-direct learning and inference.
• It spans diverse applications from medical imaging segmentation with pseudo-reports to attention-driven control in diffusion and language agents.
• This approach improves efficiency and adaptability by reducing dependency on external guidance while maintaining or enhancing performance metrics.

Self-guidance denotes a class of mechanisms in which guidance is derived from a system’s own internal signals, self-generated artifacts, or reflective state rather than from an external annotator, reward model, classifier, or tutorial. In current arXiv usage, the term is notably heterogeneous. It can refer to auto-generated pseudo-reports that replace clinical text at inference time in chest X-ray segmentation, gradients computed from a diffusion model’s own attention maps or feature activations, self-generated internal reward in language agents, self-synthetic data for task-specific finetuning, or learner-controlled reflection supported by adaptive prompts in human-centered systems (Ye et al., 2024, Epstein et al., 2023, Wang et al., 3 Apr 2026, Zhao et al., 2024, Nussbaumer et al., 2014).

1. Conceptual scope and definitional variants

Across the literature, self-guidance is not a single algorithmic primitive but a recurring design pattern. The common thread is endogenous control: a model or learner uses information already present within its own representations, outputs, or interaction history to steer subsequent inference, optimization, or reflection. This suggests a family resemblance concept rather than a single canonical definition.

In the broadest formalization, Self-directed Machine Learning (SDML) treats self-guidance as a self-aware control loop over task selection, data selection, model selection, optimization strategy selection, and evaluation metric selection. Its controller is machine self-awareness, $MA=\{\mathbb{C},\mathbb{S},G_T,\mathbb{Q},G_K\}$, combining internal awareness—capacity, state, task graph, and strategy library—with external awareness in the form of commonsense and domain knowledge graphs (Zhu et al., 2022). In that formulation, self-guidance is inseparable from self-selection and self-evaluation.

A distinct human-centered meaning appears in HCI and learning technologies. Liu et al. operationalize self-guidance as self-experimentation: trial-and-error exploration, “click-around” interaction, and prompt refinement without linear tutorials. ExploreSelf and the ROLE framework instead emphasize user-directed reflective journeys supported by on-demand prompts, theme navigation, and recommendation widgets rather than fully automated adaptation (Liu et al., 11 May 2026, Song et al., 2024, Nussbaumer et al., 2014).

Domain	Self-guidance signal	Representative papers
Medical image segmentation	Auto-generated, location-focused pseudo-report	(Ye et al., 2024)
Diffusion, MGM, and flow generation	Internal attention maps, feature activations, sub-network outputs, or batch features	(Epstein et al., 2023, Qiu et al., 2024, Wang et al., 22 Mar 2025, Chen et al., 18 Aug 2025, Hur et al., 2024, Bhat et al., 25 Jun 2026)
LLM adaptation and agents	Self-synthetic data, verbal self-assessment, internal reward, guide-scored subproblems	(Zhao et al., 2024, Wang et al., 3 Apr 2026, Bailey et al., 22 Apr 2026)
Human learning and reflection	Self-experimentation, adaptive prompts, SRL nudges	(Liu et al., 11 May 2026, Song et al., 2024, Nussbaumer et al., 2014)

2. Recurrent architectural patterns

A first recurrent pattern is training–inference bridging. SGSeg is trained multimodally with image and ground-truth report, but at inference its Localization-Enhanced Report Generation (LERG) module generates a short report $\langle\hat r\rangle$ from the image alone, allowing the same Guided U-Net decoder to operate without human-written text. The total training objective is

$$\mathcal{L}_{total} = \mathcal{L}_{seg} +\alpha\,\mathcal{L}_{pseudo} +\beta\,\mathcal{L}_{report},$$

with $\mathcal{L}_{seg}$ defined by BCE and Dice terms and $\mathcal{L}_{pseudo}$ enforcing pseudo-label consistency (Ye et al., 2024).

A second pattern is internal-state alignment. In neural speech codecs, self-guidance is not an inference-time controller but a training augmentation that aligns the decoder manifold under continuous encoder latents $z_e$ and quantized latents $z_q$. If $h_e=D_{backbone}(z_e)$ and $h_q=D_{backbone}(z_q)$, the guidance loss is

$L_{guide}=\|sg(h_e)-h_q\|_2^2.$

This leaves inference unchanged, adds one extra decoder forward pass per batch, and is reported to incur only $\langle\hat r\rangle$0 total training time while improving reconstruction metrics and enabling a $\langle\hat r\rangle$1 codebook reduction without fidelity loss (Li et al., 11 Jun 2026).

A third pattern is analytic self-guidance from a joint generative model. In offline reinforcement learning, Self-Weighted Guidance (SWG) diffuses jointly over actions and weights $\langle\hat r\rangle$2, so that the score needed for weighted-policy sampling is recovered from the same diffusion model rather than an auxiliary critic or guidance network. The key observation is that, because the extraction $\langle\hat r\rangle$3 is linear, the target score decomposes into a learned diffusion score plus a guidance term obtained directly from the model’s own prediction (Tagle et al., 23 May 2025).

These patterns differ operationally, but each replaces an absent or inconvenient external signal with an endogenous surrogate that is compatible with the model’s original training dynamics.

3. Self-guidance in medical imaging and structured perception

In chest X-ray segmentation, self-guidance addresses a practical asymmetry between retrospective multimodal analysis and prospective decision support. Language-guided segmentation methods can exploit clinical reports as a “soft prior,” but requiring ground-truth reports at test time is clinically mismatched because radiologists typically inspect the image before writing the report. SGSeg is explicitly designed to close that gap by learning with image-report pairs and inferring from image alone (Ye et al., 2024).

The SGSeg architecture has two stages. During multi-modal training, a ConvNeXt-Tiny image encoder downsamples $\langle\hat r\rangle$4, a BERT text encoder pretrained on MIMIC reports produces contextual token embeddings, and a Guided U-Net decoder fuses image features $\langle\hat r\rangle$5 with text features $\langle\hat r\rangle$6 through a cross-modal attention block to produce $\langle\hat r\rangle$7. In parallel, LERG is trained to generate location-focused reports. During uni-modal inference, LERG produces $\langle\hat r\rangle$8, the same BERT encoder embeds it, and the decoder produces $\langle\hat r\rangle$9 without any human-written report.

LERG consists of three components. First, a location-aware pseudo-label extraction stage applies fine-tuned BERT to ground-truth reports, extracts embeddings of location-descriptive tokens such as “upper,” “middle,” “lower,” “left,” and “right,” and clusters them with HDBSCAN into six pseudo-labels corresponding to lung sub-regions. Second, an RT-DETR-based object detector predicts presence vectors $$\mathcal{L}_{total} = \mathcal{L}_{seg} +\alpha\,\mathcal{L}_{pseudo} +\beta\,\mathcal{L}_{report},$$0 for lung sub-regions, supervised by

$$\mathcal{L}_{total} = \mathcal{L}_{seg} +\alpha\,\mathcal{L}_{pseudo} +\beta\,\mathcal{L}_{report},$$1

Third, a location-based attention aggregator stacks object predictions into $$\mathcal{L}_{total} = \mathcal{L}_{seg} +\alpha\,\mathcal{L}_{pseudo} +\beta\,\mathcal{L}_{report},$$2, computes

$$\mathcal{L}_{total} = \mathcal{L}_{seg} +\alpha\,\mathcal{L}_{pseudo} +\beta\,\mathcal{L}_{report},$$3

and decodes $$\mathcal{L}_{total} = \mathcal{L}_{seg} +\alpha\,\mathcal{L}_{pseudo} +\beta\,\mathcal{L}_{report},$$4 into a short textual description $$\mathcal{L}_{total} = \mathcal{L}_{seg} +\alpha\,\mathcal{L}_{pseudo} +\beta\,\mathcal{L}_{report},$$5.

On QaTa-COV19, the reported Dice and Jaccard of SGSeg are $$\mathcal{L}_{total} = \mathcal{L}_{seg} +\alpha\,\mathcal{L}_{pseudo} +\beta\,\mathcal{L}_{report},$$6 and $$\mathcal{L}_{total} = \mathcal{L}_{seg} +\alpha\,\mathcal{L}_{pseudo} +\beta\,\mathcal{L}_{report},$$7, versus $$\mathcal{L}_{total} = \mathcal{L}_{seg} +\alpha\,\mathcal{L}_{pseudo} +\beta\,\mathcal{L}_{report},$$8 and $$\mathcal{L}_{total} = \mathcal{L}_{seg} +\alpha\,\mathcal{L}_{pseudo} +\beta\,\mathcal{L}_{report},$$9 for the best uni-modal baseline, Swin U-Net. LanGuideMedSeg with true text at inference reaches Dice $\mathcal{L}_{seg}$0 and Jaccard $\mathcal{L}_{seg}$1, while SGSeg reports accuracy $\mathcal{L}_{seg}$2 against $\mathcal{L}_{seg}$3 for the uni-modal baseline and $\mathcal{L}_{seg}$4 for the multi-modal setting. The ablation sequence—without text $\mathcal{L}_{seg}$5, CLIP only $\mathcal{L}_{seg}$6, simple self-guidance $\mathcal{L}_{seg}$7, weakly supervised LERG $\mathcal{L}_{seg}$8, and ground-truth text at inference $\mathcal{L}_{seg}$9—shows that self-guidance recovers most, but not all, of the gap to fully multimodal inference.

4. Generative models: attention-, feature-, and subnet-based self-guidance

In text-guided diffusion, self-guidance first appears as direct control over internal representations. “Diffusion Self-Guidance for Controllable Image Generation” extracts centroid, size, shape, and appearance signals from cross-attention maps $\mathcal{L}_{pseudo}$0 and feature activations $\mathcal{L}_{pseudo}$1, and modifies the denoiser as

$\mathcal{L}_{pseudo}$2

Because $\mathcal{L}_{pseudo}$3 is built from the pretrained model’s own internals, the method requires no auxiliary classifier or extra training. The reported use cases include moving or resizing objects, mixing appearance from one image with layout from another, composing objects from many images, and editing real images after forward diffusion (Epstein et al., 2023).

Subsequent work specializes this principle to failure modes of attention-based generation. Self-Cross Diffusion Guidance penalizes overlap between the cross-attention map of one subject token and the aggregated self-attention map of the other subject’s image patches, targeting subject mixing for similar-looking entities. On the Animal-Animal split, the reported “w/o M” score rises from $\mathcal{L}_{pseudo}$4 for SD1.4 to $\mathcal{L}_{pseudo}$5 with Self-Cross, and on the Similar Subjects Dataset the same metric rises from about $\mathcal{L}_{pseudo}$6 to $\mathcal{L}_{pseudo}$7. Self-Coherence Guidance adapts the idea to Transformer-based TGDMs by extracting masks from the previous denoising step and refining all cross-attention maps at the current step. Its reported BLIP-VQA scores are $\mathcal{L}_{pseudo}$8 for coarse-grained binding, $\mathcal{L}_{pseudo}$9 for fine-grained binding, and $z_e$0 for style binding, compared with $z_e$1, $z_e$2, and $z_e$3 for the transformer baseline (Qiu et al., 2024, Wang et al., 22 Mar 2025).

Other variants use internal model weakness rather than attention overlap. S$z_e$4-Guidance analyzes classifier-free guidance through Gaussian-mixture examples and then subtracts the output of a stochastic block-dropped sub-network: $z_e$5 On SD3/SD3.5 it reports HPSv2.1 $z_e$6 versus $z_e$7 for CFG, and on Wan-1.3B text-to-video it reports total score $z_e$8 versus $z_e$9, with an approximately $z_q$0 slowdown from the extra forward pass (Chen et al., 18 Aug 2025).

Masked generative models implement a discrete analogue. Self-Guidance for MGMs introduces a learned bottleneck $z_q$1 that performs semantic smoothing in VQ token space and combines normal and bottleneck logits during sampling. On ImageNet $z_q$2 with $z_q$3 steps, it reports FID $z_q$4, IS $z_q$5, Precision $z_q$6, and Recall $z_q$7, surpassing earlier MGM sampling methods while using efficient masked generation (Hur et al., 2024).

Flow models adopt yet another mechanism: feature dispersion during batch generation. Feature Self-Guidance minimizes pairwise cosine similarity through a dispersion energy $z_q$8, then projects dispersed features back toward the manifold by reusing the same block and interpolating with coefficient $z_q$9. On GenEval prompts with batch size $h_e=D_{backbone}(z_e)$0, DINOScore rises from $h_e=D_{backbone}(z_e)$1 to $h_e=D_{backbone}(z_e)$2, DreamSim from $h_e=D_{backbone}(z_e)$3 to $h_e=D_{backbone}(z_e)$4, while CLIPScore drops only from $h_e=D_{backbone}(z_e)$5 to $h_e=D_{backbone}(z_e)$6, at roughly $h_e=D_{backbone}(z_e)$7 wall-clock latency on FLUX.1-dev (Bhat et al., 25 Jun 2026).

5. LLMs, agents, and autonomous learning

In language agents, self-guidance often becomes a step-level control signal coupled to policy optimization. Self-Guide for language agents samples a verbal assessment

$h_e=D_{backbone}(z_e)$8

maps $h_e=D_{backbone}(z_e)$9 to internal reward $h_q=D_{backbone}(z_q)$0, and then conditions action generation on the same signal: $h_q=D_{backbone}(z_q)$1 Training combines sparse environment reward with the aggregated internal reward under a trapezoidal trust coefficient $h_q=D_{backbone}(z_q)$2, and uses GRPO. The reported gains are $h_q=D_{backbone}(z_q)$3–$h_q=D_{backbone}(z_q)$4 percentage points over ReAct/Reflexion on ALFWorld for inference-time self-guidance alone, and an average $h_q=D_{backbone}(z_q)$5 percentage-point improvement over GRPO across ALFWorld, ScienceWorld, and WebShop; one cited example is $h_q=D_{backbone}(z_q)$6 success on ALFWorld with Qwen3-4B (Wang et al., 3 Apr 2026).

A separate LLM meaning of self-guidance is self-synthetic specialization. SELF-GUIDE generates task-specific inputs from the student model itself, filters them by noise and length, annotates outputs with in-context learning, filters again, and finetunes the same student on the resulting synthetic dataset. On held-out Super-NaturalInstructions V2 tasks, the reported averages are $h_q=D_{backbone}(z_q)$7 prompting, $h_q=D_{backbone}(z_q)$8 few-shot finetuning, and $h_q=D_{backbone}(z_q)$9 SELF-GUIDE for classification Exact Match, and $L_{guide}=\|sg(h_e)-h_q\|_2^2.$0, $L_{guide}=\|sg(h_e)-h_q\|_2^2.$1, and $L_{guide}=\|sg(h_e)-h_q\|_2^2.$2 respectively for generation ROUGE-L. Filter ablations are nontrivial: removing the noise filter hurts classification by $L_{guide}=\|sg(h_e)-h_q\|_2^2.$3 points, and removing the length filter hurts generation by $L_{guide}=\|sg(h_e)-h_q\|_2^2.$4 points (Zhao et al., 2024).

Self-guided self-play extends the idea to curriculum construction. In SGS for Lean4 theorem proving, a common base LLM plays three roles: Solver $L_{guide}=\|sg(h_e)-h_q\|_2^2.$5, Conjecturer $L_{guide}=\|sg(h_e)-h_q\|_2^2.$6, and Guide $L_{guide}=\|sg(h_e)-h_q\|_2^2.$7. The Guide scores synthetic subproblems on relevance, redundancy, and complexity, producing $L_{guide}=\|sg(h_e)-h_q\|_2^2.$8, while a solve-rate bandpass suppresses subproblems that are too easy or unsolved. The combined reward $L_{guide}=\|sg(h_e)-h_q\|_2^2.$9 is used to train the Conjecturer. Reported results include a $\langle\hat r\rangle$00 percentage-point higher asymptotic solve rate than the REINFORCE baseline, illustrated as $\langle\hat r\rangle$01, and a $\langle\hat r\rangle$02B model that after $\langle\hat r\rangle$03 rounds of self-play solves more problems than a $\langle\hat r\rangle$04B model at pass@4 (Bailey et al., 22 Apr 2026).

At the most abstract level, SDML generalizes these mechanisms into a six-level nested optimization over self-awareness construction, task selection, data selection, model selection, optimizer selection, model training, and meta-parameter update. Its significance lies less in a benchmark than in its attempt to formalize self-guidance as autonomous selection of what to learn, how to learn it, and how to measure progress (Zhu et al., 2022).

6. Human-centered self-guidance, trade-offs, and persistent limitations

In HCI, self-guidance frequently denotes a preference for autonomy rather than an algorithmic guidance term. Liu et al. report that $\langle\hat r\rangle$05 of survey respondents preferred learning GenAI image tools by “experimenting independently,” and that confusing terminology was the top reason for avoiding tutorials at $\langle\hat r\rangle$06. In a comparative study with a research probe, structured guidance reduced confusion from $\langle\hat r\rangle$07 to $\langle\hat r\rangle$08 and increased satisfaction from $\langle\hat r\rangle$09 to $\langle\hat r\rangle$10, yet many participants still preferred self-experimentation because guidance could limit creativity. The authors’ typology distinguishes experiment-first learners, hybrid explorers, and reflection-triggered experimenters (Liu et al., 11 May 2026).

ExploreSelf presents another human-centered formulation: adaptive guidance that remains user-driven. The system generates themes, Socratic questions, keywords, comments, and summaries from a session log while allowing users to choose themes and questions rather than being pushed along a fixed path. In a $\langle\hat r\rangle$11-participant study, the average number of themes explored was $\langle\hat r\rangle$12, questions answered $\langle\hat r\rangle$13, and session duration $\langle\hat r\rangle$14 minutes; the Pathways subscale increased from $\langle\hat r\rangle$15 to $\langle\hat r\rangle$16, with $\langle\hat r\rangle$17, $\langle\hat r\rangle$18, and Cohen’s $\langle\hat r\rangle$19 (Song et al., 2024).

The ROLE framework situates self-guidance within self-regulated learning rather than LLM interaction. It operationalizes four phases—Planning, Preparing, Learning, Reflecting—supports them through SRL widgets and recommenders, and makes guidance available as “template nudges,” activity sequence suggestions, and reflection scaffolds. In a short-term lab study with $\langle\hat r\rangle$20, pre-test versus post-test scores increased from $\langle\hat r\rangle$21 to $\langle\hat r\rangle$22, with $\langle\hat r\rangle$23 and $\langle\hat r\rangle$24. Long-term sandbox analytics over two years recorded $\langle\hat r\rangle$25 million API calls, $\langle\hat r\rangle$26 distinct external users, $\langle\hat r\rangle$27 spaces, and $\langle\hat r\rangle$28 active users, with SRL-enabled spaces showing higher proportions of “Plan & Organize” and “Reflect & Evaluate” events than non-SRL spaces (Nussbaumer et al., 2014).

Several limitations recur across otherwise unrelated uses of the term. First, self-guidance is bottlenecked by the quality of the self-signal. SGSeg with self-guidance reaches Dice $\langle\hat r\rangle$29, but ground-truth text at inference still yields $\langle\hat r\rangle$30, indicating residual mismatch between generated and true reports (Ye et al., 2024). Second, stronger guidance can distort outputs: Self-Cross reports occasional blur or cartoonish outputs when guidance is too strong, SCG notes possible errors from inaccurate masks, and MGM self-guidance reports oversharpening and color saturation for large guidance scales (Qiu et al., 2024, Wang et al., 22 Mar 2025, Hur et al., 2024). Third, many methods introduce inference overhead: roughly $\langle\hat r\rangle$31 for S$\langle\hat r\rangle$32-Guidance, $\langle\hat r\rangle$33–$\langle\hat r\rangle$34 for Self-Cross, and about $\langle\hat r\rangle$35 for feature self-guidance in flows, whereas decoder-manifold alignment in neural codecs is unusual in preserving inference cost entirely (Chen et al., 18 Aug 2025, Bhat et al., 25 Jun 2026, Li et al., 11 Jun 2026). Fourth, in agent learning, internal reward shaping can bias the long-run optimum; Self-Guide explicitly flags non-potential shaping as a limitation and uses late annealing for that reason (Wang et al., 3 Apr 2026).

Taken together, the literature treats self-guidance less as a fixed technique than as a strategy for endogenous control. Whether the medium is pseudo-reports, attention maps, sub-network outputs, internal reward, synthetic examples, or reflective prompts, the central objective is to reduce dependence on external guidance while preserving, approximating, or operationalizing the behavior that external guidance would otherwise supply.