Agent Visibility & Control (AVC)

• Agent Visibility and Control (AVC) is the structured coupling of visible state information with control mechanisms that guide agent behavior across diverse domains.
• It integrates geometric, graph-theoretic, and semantic visibility definitions with tailored control strategies to optimize safety, efficiency, and system integrity.
• Key applications include UAV sensing, distributed multi-agent consensus, and real-time oversight of autonomous agents, highlighting practical trade-offs in performance and auditability.

Agent Visibility and Control (AVC) denotes a family of problems in which an agent’s effective behavior depends on what is visible—geometrically, graph-theoretically, semantically, or institutionally—and on the mechanisms that use that visibility to constrain, guide, audit, or admit action. In the literature, AVC spans at least three recurring formulations: physical visibility in robotics and UAV sensing, mutual-visibility or access-limited interaction in multi-agent systems, and runtime observability plus governance for autonomous software and LLM agents (Papaioannou et al., 2023, Qi et al., 2022, Chan et al., 2024, Gupta, 19 Dec 2025, Fernandez, 19 Mar 2026). Across these settings, the common technical question is not simply whether an agent can act, but what state is exposed to the agent or overseer, how that state is represented, and what control structure is imposed on the resulting action space.

1. Conceptual scope

A governance-oriented definition appears in “Visibility into AI Agents” (Chan et al., 2024), which defines visibility as “information about where, why, how, and by whom AI agents are used.” The paper organizes visibility measures into three categories—agent identifiers, real-time monitoring, and activity logs—and treats them as accountability primitives rather than as performance tools. A closely related shift appears in “Verifiability-First Agents” (Gupta, 19 Dec 2025), which argues that evaluation should move from “how likely misalignment is” toward “how quickly and reliably misalignment can be detected and remediated.” In “A Vision for Access Control in LLM-based Agent Systems” (Li et al., 13 Oct 2025), the same theme is expressed as a move from binary permission checking to information-flow governance.

These formulations differ in substrate, but they share a common architectural pattern. Visibility is the exposure of relevant state; control is the mechanism that uses that state to shape behavior. In robotics, the exposed state may be a field of view, a line of sight, or an occlusion-aware signed distance. In distributed control, it may be a mutual visibility graph or an accessibility relation. In LLM systems, it may be an attested action receipt, a provenance log, or a formally exposed application state. This suggests that AVC is best understood as a cross-domain systems concept rather than a single algorithmic paradigm.

Setting	Visibility object	Control object
Robotics and UAVs	FoV, LoS, occluded visibility, ROI masks, smoke visibility	QP/CBF constraints, MIQP planning, ergodic replanning, skip-based encoding
Networked multi-agent systems	Mutual visibility graph, visibility graph, accessibility relation	Laplacian feedback, broadcast leader-follower control, assignment optimization
LLM and enterprise agents	Identifiers, attested actions, provenance logs, capabilities, exposed UI state	Audit agents, challenge-response, access-control reasoner, admission control, execution gateways

2. Geometric visibility in sensing, tracking, and coverage

In robotics and UAV work, AVC is usually literal: a point, target, or region must be visible through a camera or sensor, and control is the process of maintaining or exploiting that visibility. “Integrated Guidance and Gimbal Control for Coverage Planning With Visibility Constraints” (Papaioannou et al., 2023) formulates coverage path planning as a joint optimization over UAV mobility controls $u_t=[f_t(x),f_t(y)]^\top$ and gimbal controls $\theta_t,\xi_t$. The paper distinguishes the full FoV $\mathcal{S}_t(\theta_t,x_t^\text{pos},\xi_t)$ from the visible FoV $\mathcal{S}'_t(\theta_t,x_t^\text{pos},\xi_t)$, where visibility requires both FoV inclusion and ray-based first intersection with the target boundary. Coverage is then enforced by constraints such as

$$p \in \mathcal{S}^\prime_{1:T}, ~\forall p \in \mathcal{P},$$

and by binary encodings of visibility in an MIQP formulation. In the reported scenario, the $J_1$ objective yields full coverage at time step 7.

“Control Strategies for Pursuit-Evasion Under Occlusion Using Visibility and Safety Barrier Functions” (Zhou et al., 2024) recasts visibility maintenance as a nonsmooth control-barrier-function problem. The visibility barrier is

$h(t,x) = -d(y(t), F(x)),$

where $d(\cdot,\cdot)$ is the signed distance to the occluded FoV $F(x)$. Visibility is preserved by enforcing a generalized-gradient CBF constraint, while obstacle avoidance is imposed through a second SDF-based CBF. The method combines a sampling-based kinodynamic planner with a convex online tracking controller. In CARLA, the full method reports 98% time in FoV, 0 collisions, first detection in about 11 seconds, and re-establishment of tracking in 4.6 seconds after a visibility loss. Real-world Jackal experiments with a $30^\circ$, 2 m triangular FoV report time-in-FoV values around 76%–92% depending on scenario.

“Visibility Maximization Controller for Robotic Manipulation” (He et al., 2022) addresses self-occlusion in eye-to-hand manipulation by making line-of-sight visibility a soft constraint in a whole-body QP over base, arm, and camera/head velocities. The central inequality is a soft velocity damper on the distance $\theta_t,\xi_t$0 between a robot link and the camera-to-target line: $\theta_t,\xi_t$1 Because the constraint is soft, the controller can still approach the target when grasping requires temporary LoS intrusion. The reported results show a persistent trade-off: in fixed-base real-world grasping, VMC achieves 19.4% occlusion with 100% success, while the more conservative MoveIt+ baseline reaches 5.1% occlusion but only 80% success.

A related formulation appears in “Multi-Agent Ergodic Exploration under Smoke-Based, Time-Varying Sensor Visibility Constraints” (Wittemyer et al., 6 Mar 2025), where visibility is an environmental variable. Smoke density $\theta_t,\xi_t$2 induces a visibility coefficient

$\theta_t,\xi_t$3

which modulates the expected information distribution used by ergodic trajectory optimization. The planner repeatedly recomputes $\theta_t,\xi_t$4 from uncertainty and current visibility, so agents move toward regions that are both informative and actually observable at that time. The Shannon-entropy EID reduces iterations to ergodicity by 11.7% relative to baseline and 5.42% relative to smoke-mask EID for static targets, and by 21.7% and 16.9% respectively for moving targets; uncertainty reduction improves by 93.8% and 38.1% for static targets, and by 235% and 97.1% for moving targets.

In UAV aerial surveillance coding, visibility is coupled to selective transmission rather than motion. “Region of Interest (ROI) Coding for Aerial Surveillance Video using AVC & HEVC” (Meuel et al., 2018) uses on-board detection of New Areas (NA) and Moving Objects (MO), then externally controls the encoder so that ROI blocks are encoded normally while non-ROI blocks are forced into skip mode. Background is reconstructed through global motion compensation using a projective transform estimated from Harris corners, KLT tracking, and RANSAC. Replacing the modified AVC back-end with HEVC improves coding efficiency by 32% on average, with rates of 0.7–1.0 Mbit/s for full HDTV 1920×1080 at 30 fps and ROI PSNR greater than 37 dB. Here, visibility is operationalized as a detector-produced ROI mask, and control is the external enforcement of coding mode.

3. Visibility graphs, access relations, and distributed control

In networked multi-agent systems, AVC is often formalized through graph structure. “Dual Quaternion Matrices in Multi-Agent Formation Control” (Qi et al., 2022) models sensing and communication by an undirected mutual visibility graph $\theta_t,\xi_t$5, where $\theta_t,\xi_t$6 means agents $\theta_t,\xi_t$7 and $\theta_t,\xi_t$8 can sense each other. Relative rigid-body configurations are stored in the dual quaternion adjacency matrix

$\theta_t,\xi_t$9

with analogous logarithm and twist adjacency matrices. The paper proves that the relative configuration adjacency matrix and logarithm adjacency matrix are dual quaternion Hermitian, introduces dual quaternion Laplacians $\mathcal{S}_t(\theta_t,x_t^\text{pos},\xi_t)$0, and states that a dual quaternion Laplacian matrix is positive semidefinite Hermitian with nonnegative dual-number eigenvalues. The closed-loop distributed law $\mathcal{S}_t(\theta_t,x_t^\text{pos},\xi_t)$1 gives the standard consensus-like interpretation: only visible neighbors contribute, but the spectrum still supports formation stabilization.

“Stochastic Broadcast Control of Multi-Agent Swarms” (Segall et al., 2016) uses the visibility graph induced by a finite sensing radius $\mathcal{S}_t(\theta_t,x_t^\text{pos},\xi_t)$2, with $\mathcal{S}_t(\theta_t,x_t^\text{pos},\xi_t)$3 iff $\mathcal{S}_t(\theta_t,x_t^\text{pos},\xi_t)$4. A random subset of agents detects an exogenous broadcast velocity $\mathcal{S}_t(\theta_t,x_t^\text{pos},\xi_t)$5 and becomes ad-hoc leaders, while all agents continue running the same local gathering law. For uniform influence, the collective velocity is

$\mathcal{S}_t(\theta_t,x_t^\text{pos},\xi_t)$6

for scaled influence, it is

$\mathcal{S}_t(\theta_t,x_t^\text{pos},\xi_t)$7

The paper shows that, for connected visibility graphs and piecewise-constant parameters, the swarm asymptotically aligns on a line in the direction of $\mathcal{S}_t(\theta_t,x_t^\text{pos},\xi_t)$8 and all agents move with identical speed. The dependence of the scaled case on degree structure makes explicit that limited visibility is not only a sensing constraint; it changes the achievable collective motion.

“Multi-Agent Discrete Search with Limited Visibility” (Ding et al., 2017) represents visibility as an accessibility relation

$\mathcal{S}_t(\theta_t,x_t^\text{pos},\xi_t)$9

In the location-only detection model, the total detection probability after $\mathcal{S}'_t(\theta_t,x_t^\text{pos},\xi_t)$0 searches of location $\mathcal{S}'_t(\theta_t,x_t^\text{pos},\xi_t)$1 is

$\mathcal{S}'_t(\theta_t,x_t^\text{pos},\xi_t)$2

and the assignment problem is exactly reformulated as a minimum-cost network flow. The specialized primal-dual algorithm has worst-case complexity $\mathcal{S}'_t(\theta_t,x_t^\text{pos},\xi_t)$3. When detection depends on both agent and location, the problem becomes NP-hard and is reduced to monotone submodular maximization over a matroid, with a $\mathcal{S}'_t(\theta_t,x_t^\text{pos},\xi_t)$4-approximation by greedy and a $\mathcal{S}'_t(\theta_t,x_t^\text{pos},\xi_t)$5-approximation by continuous greedy with pipage rounding.

Taken together, these works make the graph-theoretic meaning of AVC precise. Visibility is an edge relation that determines which relative states can enter the controller; control is the distributed law, augmentation rule, or assignment algorithm that acts only on those visible edges or accessible pairs. This also clarifies a common misconception: in multi-agent control, “visibility” need not mean camera geometry at all; it often means locality of relative information.

4. Observability, auditability, and runtime oversight for autonomous software agents

For LLM-based agents, AVC is centered on observability and intervention. “Visibility into AI Agents” (Chan et al., 2024) treats agent identifiers, real-time monitoring, and activity logs as three complementary visibility mechanisms. Agent identifiers indicate whether and which AI agents are involved in an interaction; real-time monitoring aims to flag and potentially filter problematic behavior as it occurs; activity logs record certain inputs and outputs of an agent. The paper emphasizes deployment context, noting that centralized deployment makes these measures easiest to implement, while decentralized deployment shifts leverage toward compute providers and tool or service providers. It also foregrounds privacy and concentration-of-power risks.

“Verifiability-First Agents” (Gupta, 19 Dec 2025) makes these ideas operational through a four-layer architecture: Intent Specification (ISpec), Action Attestation Layer (AAL), Audit Agents, and Challenge-Response Attestation plus Controller & Remediator. Each consequential action generates an attestation receipt

$\mathcal{S}'_t(\theta_t,x_t^\text{pos},\xi_t)$6

which is stored in the Provenance Log. Audit Agents combine rule-based verification, statistical detection, and semantic consistency analysis into

$\mathcal{S}'_t(\theta_t,x_t^\text{pos},\xi_t)$7

If the score crosses a threshold, the verifier can trigger challenge-response or invoke the controller to pause, restrict, or switch the agent to safe mode. The paper reports, over 150 episodes, mean detection time $\mathcal{S}'_t(\theta_t,x_t^\text{pos},\xi_t)$8 s, remediation latency $\mathcal{S}'_t(\theta_t,x_t^\text{pos},\xi_t)$9 s, Attribution Confidence 0.85, false positive rate 0.09, and runtime overhead under 6.5% for VFA, compared with $$p \in \mathcal{S}^\prime_{1:T}, ~\forall p \in \mathcal{P},$$0 s and AC 0.73 for heuristic log monitoring.

“Breaking and Fixing Defenses Against Control-Flow Hijacking in Multi-Agent Systems” (Jha et al., 20 Oct 2025) argues that incomplete visibility is a structural weakness in multi-agent security. Alignment checks over inter-agent messages can be bypassed because the orchestrator sees only summaries and trusted agent outputs, not full internal provenance. The proposed ControlValve defense replaces global “related to” or “likely to further” judgments with a permitted control-flow graph and zero-shot contextual rules for each edge. The runtime question is not whether an invocation appears aligned in the abstract, but whether it is a permitted transition under the CFG and its local constraints. On CFH-Hard, the paper reports 80% ASR for default LlamaFirewall on coding attacks and 67% on computer-use attacks, whereas ControlValve reports 0% attack success rate across the evaluated IPI, original CFH, and CFH-Hard suites, while reaching 97% single-answer accuracy on coding tasks versus 93% for the undefended system and 100% on computer-use tasks versus 89% for the base system.

These systems treat visibility as evidence rather than introspection. The agent’s own account is not sufficient; what matters is signed provenance, explicit transition structure, or other externally checkable artifacts. Control then becomes online enforcement at the point of execution rather than post-hoc diagnosis alone.

5. Protocols and architectural layers for machine-actionable control

Several papers turn AVC into an explicit protocol layer. “A Vision for Access Control in LLM-based Agent Systems” (Li et al., 13 Oct 2025) proposes Agent Access Control (AAC), which consists of multi-dimensional contextual evaluation and adaptive response formulation. The first module evaluates identity and relationship, interaction scenario, task intent, and normative adherence; the second shapes disclosure through granularity control, content redaction and anonymization, and semantic paraphrasing. The paper’s central claim is that the core access-control problem in agentic systems is information-flow governance, not endpoint-style allow/deny alone. It therefore argues for a dedicated access-control reasoning engine, ideally a compact, verifiable neuro-symbolic reasoner.

“Agent Control Protocol: Admission Control for Agent Actions” (Fernandez, 19 Mar 2026) provides a formal admission-control layer between agent intent and system-state mutation. Its core invariant is

$$p \in \mathcal{S}^\prime_{1:T}, ~\forall p \in \mathcal{P},$$1

ACP defines cryptographic identity, capability-based authorization, deterministic risk evaluation, verifiable chained delegation, transitive revocation, execution tokens, and immutable auditing. Agent identity is derived as

$$p \in \mathcal{S}^\prime_{1:T}, ~\forall p \in \mathcal{P},$$2

and risk is scored deterministically by

$$p \in \mathcal{S}^\prime_{1:T}, ~\forall p \in \mathcal{P},$$3

The v1.13 specification comprises 36 technical documents across five conformance levels, 22 Go packages for L1–L4 capabilities, 51 signed conformance test vectors, and more than 62 verifiable requirements plus 12 prohibited behaviors. The paper is explicit that ACP complements RBAC and Zero Trust rather than replacing them.

“Visual Analytics Context Protocol” (Stähle et al., 31 Mar 2026) addresses a different but related problem: making visual analytics systems machine-visible and machine-actionable. It formalizes the interface state as

$$p \in \mathcal{S}^\prime_{1:T}, ~\forall p \in \mathcal{P},$$4

where $$p \in \mathcal{S}^\prime_{1:T}, ~\forall p \in \mathcal{P},$$5 is the set of visual components and $$p \in \mathcal{S}^\prime_{1:T}, ~\forall p \in \mathcal{P},$$6 the available interactions. VACP exposes application state, interaction capabilities, and a direct execution gateway through tools such as get_state(), get_capabilities(), and execute_interaction(). The paper also defines four knowledge-representation layers—pixel-based, DOM-based, declarative grammar, and semantic knowledge representation—and argues that current computer-vision and raw-DOM approaches are inadequate for dense interactive analytics.

“Sola-Visibility-ISPM” (Engelberg et al., 11 Jan 2026) applies a similar principle to enterprise identity security. The Sola AI Agent is schema-grounded: it identifies the relevant platform, retrieves schema and example query patterns, constrains itself to valid tables and fields, and returns evidence-backed answers. Full-path execution produces a structured step journal, while all evaluation instances include an evidence bundle with the input question, final answer, generated SQL, retrieved examples, and tool outputs. This turns AVC into a property of inspectable dataflow rather than only final-answer correctness.

Across these architectures, control is increasingly separated into layers: state exposure, capability exposure, admission or execution gating, and audit. A plausible implication is that mature AVC systems favor narrow, formally specified control points over informal reliance on the primary agent’s prompt-level self-restraint.

6. Benchmarks, metrics, trade-offs, and open problems

AVC research uses markedly different evaluation regimes depending on substrate. OPERA in “Verifiability-First Agents” (Gupta, 19 Dec 2025) measures Time-to-Detect $$p \in \mathcal{S}^\prime_{1:T}, ~\forall p \in \mathcal{P},$$7, Remediation Latency $$p \in \mathcal{S}^\prime_{1:T}, ~\forall p \in \mathcal{P},$$8, Attribution Confidence, False Positive Rate, and a composite VScore. “Sola-Visibility-ISPM” (Engelberg et al., 11 Jan 2026) evaluates 77 foundational ISPM visibility questions using expert Accuracy, Success Rate, and LLM-as-Judge measures such as Answer Relevancy, Faithfulness, HallucinationNoGT, AnswerCorrectnessNoGT, Reasoning Coherence, and SQL Semantic Appropriateness. Across all 77 questions, the benchmark reports AnswerCorrectnessNoGT 0.82, Expert Accuracy 0.84, and Expert Success Rate 0.77; AWS performance is strongest, with Expert Accuracy 0.95 and Expert Success Rate 0.90, while Okta strict success is lower at 0.50. In “VACP” (Stähle et al., 31 Mar 2026), the baseline UI+DOM setting yields 28%–51% success, whereas VACP-enabled scenarios S3 and S4 achieve near-perfect success, with reduced token consumption and execution time; annotation validation reports Cohen’s $$p \in \mathcal{S}^\prime_{1:T}, ~\forall p \in \mathcal{P},$$9.

Robotics papers evaluate visibility against task performance and safety rather than audit metrics. “Visibility Maximization Controller” (He et al., 2022) explicitly reports a trade-off between occlusion rate, task time, and success. “Integrated Guidance and Gimbal Control” (Papaioannou et al., 2023) shows that ignoring visibility constraints causes occluded points to be incorrectly treated as observed, whereas visibility-aware MIQP yields physically meaningful coverage plans. “Control Strategies for Pursuit-Evasion Under Occlusion” (Zhou et al., 2024) compares planner-only, controller-only, and combined methods, with the full method achieving the best balance of safety, visibility maintenance, and relocalization. “ROI Coding for Aerial Surveillance Video” (Meuel et al., 2018) similarly shows that externally enforced skip on non-ROI regions reduces bitrate and runtime, with HEVC-skip runtime reduced by 80–95% compared with unmodified HM.

Several limitations recur. “Visibility into AI Agents” (Chan et al., 2024) emphasizes privacy loss, surveillance abuse, and concentration of power as risks of identifiers, monitoring, and logs. “A Vision for Access Control in LLM-based Agent Systems” (Li et al., 13 Oct 2025) notes that semantic interactions permit indirect exfiltration through summarization, paraphrase, or aggregation even when object-level permissions appear satisfied. “ControlValve” (Jha et al., 20 Oct 2025) argues that safety and functionality are in tension because flexible recovery behavior also creates attack surface for control-flow hijacking. “Sola-Visibility-ISPM” (Engelberg et al., 11 Jan 2026) identifies domain-specific disagreement between expert judges and LLM judges, especially in Google Workspace. “VACP” (Stähle et al., 31 Mar 2026) notes that semantic exposure can discard visual nuance and that synchronization failures can produce stale or obsolete interaction state.

The literature therefore converges on a broad but technically consistent lesson. Visibility is not merely the ability to sense a scene or log an action; it is the structured exposure of the state that matters for a control decision. Control is not merely actuation or denial; it includes optimization under visibility constraints, graph-local feedback, execution gating, evidence-backed auditing, and context-sensitive transformation of information. This suggests that AVC is best characterized as a design discipline for coupling observability with enforceable action constraints across physical, cyber-physical, and software-agent systems.