Perception-Focused Supervision

• Perception-focused supervision is a paradigm that applies training signals to perceptual variables rather than solely optimizing final outputs.
• It is used across applications such as vision-language post-training, autonomous driving, and surgical gaze analysis to directly target perceptual bottlenecks.
• By decoupling perception from reasoning, these methods improve credit assignment, reduce errors, and enhance robustness in diverse machine learning systems.

Perception-focused supervision denotes a family of methods that place supervisory pressure on perceptual variables, perceptual processes, or perceptually grounded intermediate representations rather than relying exclusively on final actions, sequence-level rewards, or high-level reasoning traces. In recent literature, the term has been used for supervisory aids that augment human access to command and attention signals, for vision-LLM post-training schemes that explicitly separate “perception” from “reasoning,” for preference and reward designs that target hallucinated visual claims, and for self-supervised or synthetic objectives built from gaze, relative speed, CLIP-aligned semantics, viewpoint motion, lane geometry, or partial image views (Gilbert et al., 2024, Jei et al., 25 Jan 2026, Wu et al., 28 May 2026, Min et al., 27 Apr 2026).

1. Conceptual scope and common structure

Across domains, the unifying move is to replace coarse supervision with signals that are closer to the system’s perceptual bottleneck. In one formulation, the output is explicitly factorized as $y=(p,r)$, where $p$ is a compact, canonical transcription of the image and $r$ is the step-by-step derivation; in another, supervision is applied to attended regions, hallucinated spans, gaze allocation, or command signals; in still others, lane centerlines, predicted motions of surrounding agents, or egocentric camera trajectories become the primary learning targets for downstream control (Wu et al., 28 May 2026, Zhang et al., 2 Jun 2026, Min et al., 27 Apr 2026, Huang et al., 11 Nov 2025).

Research setting	Perceptual signal	Supervisory form
Human supervisory control	$u(t)$, gaze on AOIs	haptic display, adaptive saliency cues
VLM post-training	$p$ before $r$; boxes $\mathcal B$; description segment $d$	loss re-weighting, alternating GRPO, image-only teacher
Hallucination and defense	attended crops, hallucinated spans, partial crops	DPO pairs, token-level penalties, weak-to-strong prompting
Representation learning	relative speed, appearance difference, CLIP semantics, gaze labels	triplet loss, InfoNCE, MS-CLIP loss, supervised saliency
Embodied control	camera motion, lane centerlines, predicted agent motion	joint action prediction, positive/negative self-supervision

A recurring pattern is that perception-focused supervision narrows credit assignment. Instead of treating the final answer or final trajectory as the only object of optimization, these methods identify where perception enters the pipeline and attach training or intervention signals there. This suggests a broader methodological principle: when performance is limited by what a system sees, transcribes, attends to, or localizes, supervision that directly addresses those stages can outperform supervision applied only after reasoning or control has already compounded the error.

2. Human supervisory systems and perceptual access

In manual control, an operator issues a command signal $u(t)$ and gains access to that signal via efference copy and proprioception, enabling predictive sensory filtering under the “reafference principle.” In supervisory control, by contrast, the human typically observes only the reference $r(t)$ and plant output $p$0 visually, lacking access to the command $p$1 that drove the system. “Embodied Supervision” tests whether supplying a copy of $p$2 improves supervisory inference, especially when delivered haptically through a passive joystick driven by the operator’s command (Gilbert et al., 2024).

The experimental system used two identical one-degree-of-freedom joysticks on low-friction linear slides, a pure integrator plant $p$3, and three reference signals

$p$4

Supervisors viewed all three trajectories and the white cursor $p$5 under three conditions: uOff, uVisual, and uHaptic. The supervisory estimate of the reference was modeled as $p$6 without $p$7 and $p$8 with $p$9. Simulation RMS errors versus the true reference were $r$0 and $r$1. With $r$2, Target Selection Accuracy was $r$3 in uOff, $r$4 in uVisual, and $r$5 in uHaptic; the one-way repeated-measures ANOVA gave $r$6, $r$7, generalized $r$8, and the uHaptic vs. uVisual difference was significant under Bonferroni correction. Target Selection Delay also decreased from $r$9 in uOff to $u(t)$0 in uHaptic, while participant surveys reported that haptic feedback increased confidence and that visual joystick motion tended to distract or be ignored (Gilbert et al., 2024).

The same perceptual emphasis appears in gaze-based decision support for UAV command-and-control. “Eyes on the Mission” defines perception-focused supervision as supervisory aids that monitor and intervene in an operator’s perceptual processes so that critical information is neither overlooked nor under-attended. Its adaptive DST used a Tobii Pro Nano eye-tracker at 60 Hz, defined fixations as gaze samples staying within a $u(t)$1 visual angle for at least 100 ms, and triggered an AOI-specific cue when $u(t)$2 with $u(t)$3. The cue combined a warning message, color-coding, and a $u(t)$4 blinking red outline, and was placed near the timeline following the proximity-compatibility principle (Jei et al., 25 Jan 2026).

With $u(t)$5 participants and $u(t)$6 complete intervention datasets, the main effect of Condition on the composite Measure of Performance was $u(t)$7, $u(t)$8, with Intervention exceeding Control by about 7 points on average. The Condition×Trial Order interaction was also significant, $u(t)$9, $p$0. On the Convoy Health AOI, average fixation duration fell from a Control median of $p$1 to an Intervention median of $p$2, Wilcoxon $p$3, $p$4, $p$5, and total fixation duration fell from $p$6 to $p$7, $p$8, $p$9, $r$0. Notably, 88% of participants did not consciously notice the adaptive cues, yet reported feeling more secure and planning more proactively when the DST was active (Jei et al., 25 Jan 2026).

Taken together, these studies define a human-centered branch of perception-focused supervision: the supervisor is not merely given more outcome data, but is given access to missing perceptual channels—command proprioception in one case, attention-guiding salience in another.

3. Decoupling perception and reasoning in vision-LLMs

A central result of recent VLM post-training work is that reasoning and perception do not improve symmetrically. “On Asymmetric Optimization of Reasoning and Perception in Vision-LLM Post-Training” imposes disentangled outputs $r$1, where $r$2 is the perception segment and $r$3 the reasoning segment, and evaluates perception accuracy as $r$4 and counterfactual reasoning accuracy as $r$5. On Graph Coloring and Sudoku, standard SFT allocates only a weak signal to perception because $r$6 occupies about $r$7 of tokens in GC and $r$8 in Sudoku, contributing only about $r$9 of the loss and about $\mathcal B$0 of the gradient norm. The paper decomposes the loss into $\mathcal B$1 and $\mathcal B$2, uses

$\mathcal B$3

and further applies NGDiff with weights based on inverse gradient norms. Manual $\mathcal B$4-reweighting improves end-to-end accuracy by 10.0–13.8 points, and NGDiff yields up to +18.2 points over standard SFT (Wu et al., 28 May 2026).

The same paper shows that RL exhibits a different asymmetry. Under Group-Relative PPO, end-to-end outcome rewards correlate much more strongly with reasoning correctness (Pearson’s $\mathcal B$5–$\mathcal B$6) than with perception accuracy ($\mathcal B$7–$\mathcal B$8). The proposed mixed reward

$\mathcal B$9

improves end-to-end accuracy by up to +6.0 points with oracle $d$0, while a teacher-model surrogate reward with correlation about $d$1 to $d$2 yields +3.2 points (Wu et al., 28 May 2026). A common misconception is therefore directly contradicted by these results: better post-training on reasoning does not by itself imply comparable improvement in perception.

“Perceive-to-Reason” generalizes the same separation into a two-stage inference pipeline. A shared VLM backbone first acts as a Perceiver, sampling $d$3 bounding boxes $d$4 from $d$5, then as a Reasoner, answering from the annotated image $d$6, the cropped evidence $d$7, and the original question $d$8. Training uses Perception-Reasoning Alternating GRPO, with binary reward $d$9 and alternating updates so that final-answer correctness becomes a role-specific signal for either the Perceiver or the Reasoner. On a 4B model, P2R improves V-Star from 81.7% to 93.2%, HR-Bench-4K from 73.8% to 81.9%, and HR-Bench-8K from 67.0% to 80.5% (Li et al., 1 Jul 2026).

“Seeing Before Reasoning” implements a closely related principle through on-policy self-distillation. ViGOS requires the student to generate $u(t)$0, where $u(t)$1 is a visual description, $u(t)$2 a reasoning segment, and $u(t)$3 the final answer. The description tokens are supervised by an image-only perception teacher $u(t)$4, yielding

$u(t)$5

On a 3B backbone, removing the perception loss reduces CV-Bench Pass@5 from 91.51 to 91.09 and ViLP Score from 69.84 to 67.58; the full model also reduces Privileged Answer Leakage Rate from 17.26% to 6.33% at 3B and from 26.01% to 7.56% at 7B (Wang et al., 17 Jun 2026).

These works establish a distinct research program: perception is treated as a separable stage with its own output space, loss, teacher, or RL role, and model quality is improved by preventing reasoning supervision from dominating the training signal.

4. Preference, process rewards, and robustness-oriented supervision

Another line of work applies perception-focused supervision through preference optimization, process reward modeling, or inference-time weak-to-strong guidance. “P²-DPO” argues that hallucination in LVLMs reflects not only perceptual failure in the encoder but also “last-mile” errors in Perceptual Processing. It constructs on-policy, vision-aware preference pairs in two forms. Focus-and-Enhance pairs use an explicit-focus prompt, extract an attention map, crop the salient region, form an enhanced input $u(t)$6 and a degraded input $u(t)$7, then train with a DPO objective plus a Calibration Loss based on perceptual confidence gain $u(t)$8. Visual-Robustness pairs contrast clean and noisy images. On TextVQA, Attention Region Fidelity rises from 14.73 to 18.71 and Processing Accuracy from 66.29 to 70.10; on POPE with Gaussian noise $u(t)$9, F1 rises from about 76.5 to about 82.7; on LLaVA-1.5-7B, POPE Pop improves from 81.80 to 84.53, HallusionBench accuracy from 48.16 to 55.62, MMHal-Bench Halluc decreases from 6.7 to 5.9, and AMBER $r(t)$0 rises from 62.4 to 70.9 (Zhang et al., 2 Jun 2026).

Perceval pushes supervision to the token level. It parses a candidate response into atomic image-text claims, returns the exact spans that fail grounding, and converts a sequence-level GRPO advantage $r(t)$1 into

$r(t)$2

where $r(t)$3 on hallucinated spans. This preserves the original advantage on grounded tokens while penalizing hallucinated ones. Relative to +GRPO at 3B, Perceval-guided RL improves $r(t)$4 from 69.73% to 72.37%, $r(t)$5 from 80.10% to 83.25%, MathVision from 23.36% to 26.32%, and ChartQA from 83.32% to 86.48%. At test time, with $r(t)$6 samples, Truncate improves $r(t)$7 from 85.86% under major-voting to 89.53%, and BLINK from 48.41% to 49.45%. The best reported penalty strength is $r(t)$8, whereas $r(t)$9 degrades $p$00 to 78.53% (Min et al., 27 Apr 2026).

“Defending LVLMs Against Vision Attacks through Partial-Perception Supervision” adopts a training-free weak-to-strong formulation. Cropped partial images $p$01 are sent to a “weak” LVLM to obtain distributions $p$02, these are summarized into a supervisory message $p$03, and a “strong” LVLM answers from $p$04. The method is explicitly interpreted through a composite objective

$p$05

with $p$06 written as a KL divergence or cross-entropy between weak and strong distributions. Empirically, DPS cuts the average attack success rate by 76.3% across six datasets on three popular models. On Qwen-VL-Plus, the average misleading-defense ASR drops to 0.31, compared with 0.98 for Protector and 0.91 for SmoothVLM, while standard performance on MM-Vet drops by less than 1%; SmoothVLM, by contrast, degrades clean performance by 10–20 points on certain tasks (Zhou et al., 2024).

These methods differ in mechanics—DPO, GRPO, process reward models, or prompt-only supervision—but converge on the same idea: the corrective signal should be anchored in what the model actually perceives, not only in whether the final answer happened to be correct.

5. Synthetic, self-supervised, and data-centric perceptual targets

Perception-focused supervision is not limited to post-training. It also appears in representation learning and data construction, where the supervisory labels themselves are engineered to isolate perceptual content. In RSPNet, relative playback-speed perception (RSP) asks which of two clips from the same video is faster, while appearance-difference perception (A-VID) asks whether two clips come from the same video, with playback speed randomized so that motion cues become uninformative. Motion embeddings are trained with the triplet loss

$p$07

and appearance embeddings with the InfoNCE loss

$p$08

On UCF101, RSPNet reports 93.7% top-1 without labeled-data pre-training, and on HMDB51 64.7%; in ablations on TSM-18, ResNet-18, and C3D, the joint RSP + A-VID variant achieves 61.2% / 32.2%, outperforming RSP only, A-VID only, absolute-speed supervision, and the no-speed-augmentation variant (Chen et al., 2020).

VisionFoundry constructs perception supervision synthetically from a task keyword $p$09. An LLM generates question-answer-prompt triples $p$10, a text-to-image model synthesizes the image, and Gemini-3-Pro verifies a declarative statement $p$11 derived from $p$12, accepting the example only if $p$13. The resulting VisionFoundry-10K contains 10 low-level perception tasks with 1,000 examples each, including Orientation & Direction, Viewpoint & Perspective, Spatial Relationship, Depth Order, and Relative Distance. Fine-tuning Qwen2.5-VL-3B on VisionFoundry-10K improves MMVP pair accuracy from 35.3% to 42.0%, MMVP single accuracy from 64.3% to 68.3%, and CV-Bench-3D from 66.0% to 76.5%; removing the verifier under a matched 1,000-sample budget underperforms the verified pipeline by up to 2–3 points on spatial benchmarks (Zhou et al., 10 Apr 2026).

In surgical perception, gaze itself becomes the supervisory target. “Data-centric Design of Learning-based Surgical Gaze Perception Models in Multi-Task Simulation” collects paired active and passive gaze on four SimNow drills and evaluates the substitutability of passive gaze for operative supervision. Fixation density maps are compared with SIM and CC, and models are trained with frame-level saliency losses. MSI-Net yields stable alignment, with CC about 0.23–0.54 and NSS about 2.1–6.1, whereas SalGAN is unstable and often poorly aligned. For intermediate active targets, IA→IA gives CC=0.540 and NSS=6.14; passive-to-active transfer IP→IA recovers CC=0.372 and NSS=3.58, about 69% and 58% of the active ceiling; novice passive labels on high-quality demonstrations reach NP→IP-I CC=0.426 and NSS=4.38, about 95% of the intermediate-passive model on those demonstrations (Li et al., 9 Feb 2026).

Perception-focused supervision also enters learned compression. UG-ICM uses a single bitstream $p$14 and a preference-conditioned decoder $p$15 to produce either $p$16 for human fidelity or $p$17 for machine analytics. Machine-oriented supervision is provided by a frozen CLIP encoder at global, local, and instance scales:

$p$18

With Bmshj2018, UG-ICM improves unseen downstream analytics by +1.7% Top-1, +1.7% detection-mAP, +3.2% segmentation-mIoU, and +1.2% instance-mAP; with MLIC++, the gains are +2.8%, +3.9%, +6.0%, and +3.2%, while PSNR, SSIM, and LPIPS remain on par with the original LIC (Yin et al., 8 Jan 2025).

A plausible implication of these results is that perception-focused supervision is often as much a data-design problem as a loss-design problem: the label source may be synthetic verification, relative perceptual comparison, passive gaze, or pretrained semantic similarity rather than conventional human annotation.

6. Active perception, causal control, and open questions

In embodied AI, perception-focused supervision increasingly treats viewpoint behavior itself as an action target. ActiveMimic recovers synchronized egocentric camera and wrist trajectories from human video, re-expresses them in a chunk-relative frame, and forms the 27-dimensional action

$p$19

A transformer policy is pretrained with a conditional flow-matching loss over this unified action space, so that camera motion is no longer treated as nuisance but as active perception. The pretraining set contains 2,561 Ego4D episodes, about 10 hours at 10 fps. On four real-robot tasks, ActiveMimic achieves the highest success rates: 90.1 on Restocking, 88.9 on Reaching, 91.7 on Finding, and 93.3 on Pouring, outperforming $p$20, MotoVLA, AM$p$21, and AM$p$22 (Lin et al., 4 Jun 2026).

Perception-guided self-supervision in autonomous driving shifts supervision from noisy expert trajectories toward structured perception outputs. PGS uses lane centerlines $p$23 and predicted trajectories of other agents as primary supervisory signals, defines the positive self-supervision loss

$p$24

and the negative self-supervision loss

$p$25

On Bench2Drive, PGS reduces open-loop L2 error from 0.91 m to 0.77 m, raises Driving Score from 42.35 to 78.08, and raises Success Rate from 15.00% to 48.64%. In scenario-specific evaluation, it reports a mean ability score of 53.40%, exceeding DriveTransformer at 38.60% and DiffAD at 38.79% (Huang et al., 11 Nov 2025).

Despite the breadth of applications, the literature also identifies clear constraints. “Embodied Supervision” notes the use of a human “wizard” operator, a single-DOF integrator plant, small sample size, and position-only coupling (Gilbert et al., 2024). The asymmetric VLM study is conducted in a controlled synthetic setting with Graph Coloring and Sudoku (Wu et al., 28 May 2026). DPS is slower than single-prompt defenses because it requires about 5–6 LVLM calls per sample, and it protects only against vision-based perturbations that cropping disrupts (Zhou et al., 2024). Surgical gaze transfer is explicitly asymmetric between active and passive targets (Li et al., 9 Feb 2026). VisionFoundry reports that OCR-only tasks can degrade slightly when no OCR supervision is included (Zhou et al., 10 Apr 2026).

Across these results, the central lesson is stable. Systems frequently fail not only because they reason poorly, but because supervision is applied too late—after perceptual omissions, weak visual grounding, or causal confusion have already entered the computation. Perception-focused supervision addresses that failure mode by supervising the seeing, attending, localizing, transcribing, or viewpoint-selection stage itself. This suggests an increasingly general design doctrine: when the bottleneck is perceptual, the supervisory signal should be perceptual as well.