Intention-Aware Semantic Agent Communications for AI Glasses

Abstract: Smart glasses are emerging as a promising interface between humans and AI agents, enabling first-person perception, contextual awareness, and real-time assistance. However, continuous offloading of visual data from wearable devices to cloud-based vision-LLMs (VLMs) is fundamentally constrained by limited wireless bandwidth and energy resources. This paper proposes an intention-aware semantic agent communication framework for AI glasses, where data transmission is guided by user intention rather than raw pixel fidelity. In the proposed architecture, AI glasses act as an edge semantic agent while a server-side VLM executes high-level cognition and reasoning. The user intention can be inferred by the server-side VLM through the current transmitted content and the historical prompts. Driven by specific user intentions, the glasses adaptively preserve textual content, document layout, or object semantics before transmission. We evaluate three representative scenarios with different lightweight preprocessing tools on the AI glasses. Simulation results demonstrate that intention-aware preprocessing significantly achieves more than 50% bandwidth reduction depending on the current task while maintaining task performance. Moreover, semantic transmission exhibits graceful degradation under low SNRs. The findings demonstrate that aligning communication resources with user intention is essential for robust and efficient wearable AI agent systems.

• The paper introduces a dynamic intention-aware communication framework that leverages cloud-side vision-language models to adaptively steer edge preprocessing.
• It employs lightweight on-device modules like OCR, Canny edge detection, and YOLO to minimize uplink load while preserving key semantic information.
• Empirical results demonstrate significant bandwidth savings and maintained or enhanced task accuracy across text, document, and scene understanding tasks.

Intention-Aware Semantic Agent Communications in AI Glasses

Overview and Motivation

This paper advances the state of wearable AI by introducing an intention-aware semantic communication framework tailored for AI glasses, targeting the critical bottleneck of bandwidth and energy constraints in first-person, real-time perception. The central tenet is to eschew traditional pixel-focused or task-static data streams in favor of edge-cloud cooperative communication explicitly steered by dynamic user intention, recognized via server-side large vision-LLMs (VLMs). The system adaptively selects semantic preprocessing pipelines on-device and leverages robust, resolution-agnostic encoder-decoders to minimize uplink load without sacrificing task performance. Crucially, the framework is shown empirically to halve bandwidth demands in representative scenarios, preserving or even enhancing downstream task accuracy compared to conventional baselines.

Related Work and Theoretical Context

Semantic communication diverges from the classical Shannon paradigm, by recasting wireless transmission as an exercise in goal-driven, semantic relevance rather than signal fidelity [gunduz2022beyond]. Progress in deep joint source-channel coding (JSCC) and foundation model-based generative systems have further refined this vision to allow meaning-centric, adaptive communication with graceful performance degradation under poor wireless conditions [xie2020deep, grassucci2023generative, li2024end]. However, extant solutions are largely static—optimized for fixed tasks, static distortion measures, and assuming stable user goals throughout the session.

Agent-oriented advances, especially with the integration of large multimodal models, hint at more flexible, context-driven frameworks [jiang2024wcm_agent6g, chen2024enabling], but current works often lack the necessary granularity and edge-adaptive flexibility required for wearable, energy-constrained agents and cannot dynamically react to intention changes or channel variability.

The theoretical underpinning for the proposed approach integrates rate-distortion and information bottleneck perspectives, focusing on minimizing transmitted information while maximizing predictive utility for downstream tasks [tishby2000information].

System Model and Architecture

The core system (see Figure 1) executes a split-agent paradigm:

• Edge Agent (AI Glasses): Captures first-person vision data and executes lightweight, intention-aligned preprocessing drawn from a toolbox (OCR, edge detection, object detection).
• Cloud Agent (Server VLM): Performs high-level intention inference, context-driven task selection, and semantic decoding plus reasoning.

Upon each transmission event, the system cycles through two stages:

1. Low-Resolution Probing: AI glasses intermittently send downsampled images to conserve bandwidth and enable fast intention recognition with limited uplink cost. The server VLM predicts or confirms the current user intention.
2. Task-Specific Uplink: Once intention stabilizes, edge-side applies intention-matched preprocessing (e.g., OCR extraction for reading, Canny detection for document structure, YOLO cropping for object scenes), followed by adaptive semantic encoding prior to uplink transmission.

Edge-cloud communication utilizes an OFDM wireless model, with semantic latent symbols mapped to subcarriers, and the exponential effective SNR mapping (EESM) provides a direct link between physical channel quality and downstream semantic reliability.

(Figure 1)

Figure 1: Wireless communication architecture linking edge-based AI glasses and a cloud-resident VLM, with intention-aware triggering modulating the data flow.

Intention-Aware Semantic Pipeline

The pipeline orchestrates explicit (voice-triggered) and implicit (visual-inferred) intention signals. The visual intention mode leverages periodic low-res imaging with server-side VLM analysis to minimize unnecessary data movement. Switching from intention detection to task-optimized transmission is modulated dynamically whenever the VLM detects a change in intent.

Lightweight on-device preprocessing is pivotal:

• OCR module: Used for text-centric tasks (e.g., translation, receipts).
• Canny edge detector: Extracts document geometric structure for layout-dependent reasoning.
• YOLO detector: Isolates relevant objects for scene or catalog tasks.

This approach strictly limits information entropy in the uplink, handling varying input resolutions and semantic scopes seamlessly.

(Figure 2)

Figure 2: Toolbox of preprocessing modules enabling dynamic, intention-conditioned selection for edge-side semantic compression.

Robust Resolution-Agnostic Encoder-Decoder

The encoder/decoder system operationalizes fully convolutional, multi-scale transforms robust to arbitrary input resolutions. Quantization is adaptively set by the compression ratio parameter, with per-latent quantization enabling reduction from 16 bits to 4 bits per element, directly modulating channel symbol expenditure. Packing strategies allow for bandwidth-efficient symbol mapping onto OFDM subcarriers, with the decoder restoring image or token representations to their task-required fidelity.

(Figure 3)

Figure 3: Architecture of the fully convolutional semantic encoder-decoder that is robust to input size and compression granularity.

Empirical Results

Case Studies and Evaluation

Three task archetypes were explored:

1. Text Reading and Answering (Receipt/Document VQA): Edge OCR drastically reduces uplink to single-string payloads (down to 0.13 KB per sample), giving near-perfect task accuracy on high-quality text-centric samples and exposing the limits of pure-text transmission for visually structured documents.
2. Document Reasoning: Canny-processed document ROI transmission balances a 2x–6x reduction in channel symbols with minimal accuracy reduction compared to raw image transmission, and, under low SNR, semantic methods provide smooth SR degradation, outperforming digital baselines at low channel quality.

(Figure 4)

Figure 4: Bandwidth and task success rate comparisons for different preprocessing and semantic transmission modalities in document reading/QA.

1. Scene Watching: YOLO-based object cropping allows precise object coverage maintenance while delivering a 3x–8x bandwidth saving, maintaining ≥65% object recall on COCO benchmarks, and with BERTScore for scene captioning matching or exceeding digital baselines even at 0 dB SNR.

   Figure 5: Examples of intention-aware preprocessing preserving the main semantic content in multi-object scenes.

Activation and Ablation

Comparison across intention-activation modes reveals that:

• Intention-aware pipeline with stored historical commands approaches the accuracy of explicit voice commands but with lower user interaction burden and up to 80% lower bandwidth than full-frame digital transmission.
• Direct user command yields the best accuracy-bandwidth tradeoff but is not scalable for continuous wearable use.

Failure Analyses

Failures occur when semantic preprocessing (e.g., OCR) omits layout or structural features critical for certain tasks, underlining the importance of intention-aligned preprocessing tool selection.

(Figure 6)

Figure 6: Failure example illustrating SR drop in OCR-only mode due to loss of context in low-resolution or poorly structured images.

Implications and Future Directions

The architecture demonstrates that aligning wireless communication with dynamically inferred user intent is a superior paradigm for wearable agent systems. By structuring the pipeline as an explicit semantic interface between edge and cloud, and leveraging intention-driven tool selection, the system responds flexibly to user needs and transmission constraints.

Key implications include:

• Uplink cost for wearable AI can scale with semantic, not spatial, content—paving the way for longer battery life and richer on-device interactivity.
• Agentic wireless systems can exploit foundation models for both intention detection and semantic abstraction, closing the loop between perception, communication, and cognition.

Research frontiers:

• Multi-user, multi-intention scaling with federated intention modeling.
• Expansion of preprocessing toolkits and integration of generative foundation models for on-device reasoning [li2024end].
• Meta-adaptation to evolving user routines and implicit intent signals (e.g., gaze, gesture).
• Secure, privacy-preserving intention recognition and transmission.

Conclusion

This paper establishes intention-aware semantic agent communication as a foundational mechanism for efficient wearable AI, demonstrating that dynamic, VLM-guided inference of user needs can unlock order-of-magnitude improvements in bandwidth efficiency while preserving or improving task success rates. Future work on embodied, multi-agent, and context-rich wearable AI stands to benefit from this architecture, with anticipated impacts on mobile health, assistance for the visually impaired, and pervasive human-AI collaboration.

---

References:

• [gunduz2022beyond] D. Gündüz et al., "Beyond Transmitting Bits: Context, Semantics, and Task-Oriented Communications," IEEE JSAC, 2023.
• [xie2020deep] H. Xie et al., "Deep learning enabled semantic communication systems," IEEE TSP, 2021.
• [grassucci2023generative] E. Grassucci et al., "Generative Semantic Communication: Diffusion Models Beyond Bit Recovery," (Grassucci et al., 2023).
• [li2024end] S. Li et al., "End-to-End Generative Semantic Communication Powered by Shared Semantic Knowledge Base," (Li et al., 2024).
• [jiang2024wcm_agent6g] F. Jiang et al., "LLM Enhanced Multi-Agent Systems for 6G Communications," IEEE Wireless Commun., 2024.
• [tishby2000information] N. Tishby et al., "The Information Bottleneck Method," Proc. Allerton Conf. Commun., Control, Comput., 1999.