Task-Oriented Multimodal Edge Intelligence via Integrated Sensing-Communication-Computation

Abstract: Integrated sensing, communication, and computation (ISCC) has recently emerged as a unified framework for enabling edge intelligence. However, existing ISCC designs predominantly rely on single-modal sensing, which is inherently vulnerable to occlusions, environmental uncertainties, and modality-specific failures, leading to degraded robustness in real-world deployments. This motivates the need for multi-modal ISCC, yet its design remains insufficiently explored. Compared with the single-modal case, multi-modal ISCC is more challenging because heterogeneous modalities enlarge data dimensionality and tighten communication/computation/energy budgets, while inter-modal correlations further complicate performance characterization. To address these challenges, we propose a task-oriented multi-modal ISCC framework that integrates device-side feature extraction with edge-side joint multi-modal inference. A central component of our approach is the maximal coding rate reduction (MCR^2) criterion, which enables each device to learn compact and discriminative task-relevant features, offering clear advantages over conventional cross-entropy-based extractors. We further leverage MCR^2 as a principled metric for edge-side sensing evaluation. On this basis, we formulate a sensing accuracy maximization problem under delay and resource constraints and develop an efficient block coordinate descent (BCD) algorithm after transforming the problem into a more tractable equivalent form. Focusing on a human activity recognition task, we conduct extensive experiments on publicly available datasets to evaluate the performance of the proposed ISCC framework. The results demonstrate that our approach consistently outperforms three baseline schemes under limited resource conditions.

• The paper introduces a task-driven multimodal ISCC system that uses MCR² to enhance local feature extraction and overall sensing accuracy.
• It employs a block coordinate descent algorithm to optimize quantization bits and transmission time under energy, latency, and bandwidth constraints.
• Experimental results demonstrate that the proposed framework significantly outperforms single-modal and baseline approaches in resource-constrained environments.

Task-Oriented Multimodal Edge Intelligence via Integrated Sensing-Communication-Computation

Introduction and Motivation

This work introduces a comprehensive framework for task-oriented edge intelligence built on Integrated Sensing, Communication, and Computation (ISCC) principles, extending traditional ISCC to a multimodal architecture. Existing ISCC methods predominantly focus on single-modal sensing, which suffers from degraded robustness in the presence of occlusions, noise, or failures specific to a modality. This motivates the pursuit of a multimodal paradigm, leveraging heterogeneous data sources—such as mmWave radar, RFID, and WiFi—to achieve greater resilience and improved inference performance. However, multimodal ISCC introduces complexities stemming from increased data volumes, intricate cross-modal dependencies, and stringent joint resource constraints.

Figure 1: An ISCC system with multi-modal sensing.

Framework Overview and Feature Extraction

The proposed multi-modal ISCC system comprises $K$ heterogeneous IoT devices, each equipped with distinct sensing modalities, communicating with a base station hosting an edge server. The task pipeline encompasses four stages: (1) device-side multi-modal sensing, (2) task-oriented local feature extraction, (3) transmission of compressed feature representations to the edge, and (4) edge-based multi-modal inference aggregation.

Crucially, the paper adopts the maximal coding rate reduction ($\mathrm{MCR}^2$) criterion for local feature learning. This information-theoretic objective promotes representations that are simultaneously compact and inter-class discriminative, directly targeting the needs of downstream classification without recourse to cumbersome cross-entropy-based supervision. $\mathrm{MCR}^2$ is further leveraged as a differentiable system-level sensing quality metric, enabling end-to-end resource optimization across the ISCC pipeline.

Optimization Problem Formulation

The central optimization objective is sensing accuracy maximization, equivalent to maximizing the coding rate reduction $\Delta R$ under system-level resource constraints. The three primary constraints are:

• Successful Transmission: Ensuring that the quantized features from each device can be transmitted given the available communication bandwidth, transmission power, and allocation time.
• Latency: Satisfying a global end-to-end delay bound, accounting for both sensing/feature extraction and all devices' TDMA-based transmission schedules.
• Energy: Constraining per-device total energy budgets, covering sensing, feature extraction, and communication.

The resultant problem involves non-convex objectives and constraints intricately coupling quantization bit allocation, communication time assignment, and transmit power. This is tractably reformulated—introducing auxiliary variables for quantization distortion and resource usage—resulting in a structure amenable to efficient numerical optimization.

Figure 3: Solution procedure of the proposed ISCC framework.

Block Coordinate Descent Optimization

To solve the transformed maximization problem, a block coordinate descent (BCD) algorithm is developed with alternating optimization of:

1. Quantization Bit Allocation: For fixed transmission schedules, optimally allocate per-device, per-feature quantization bits under system and device constraints. The problem is device-separable and convex, solved efficiently with standard convex programming tools.
2. Communication Time Allocation: For fixed quantization, minimize total transmission delay to meet feasibility, reallocating excess or recovering from shortfall across devices based on communication overhead needs. This leverages bisection and allocation weighting based on task-essential data volume.

A theoretical guarantee of monotonic improvement and global convergence is provided by exploiting convexity properties and information-theoretic monotonicity relations between transmission resources, quantization accuracy, and sensing performance.

Figure 5: Convergence behavior of the proposed algorithms. The left two subfigures show global objective convergence, while the right illustrates fast inner AO convergence for alternate iterations.

Experimental Results and Numerical Analysis

Dataset: Three modalities (WiFi, RFID, mmWave radar) used for human activity recognition on a public dataset with eight action classes and 4,800 samples.

Implementation: Each device runs a ResNet-18 feature extractor trained under the $\mathrm{MCR}^2$ objective, with SVM and MLP classifiers deployed at the edge. All resource optimization is simulated under realistic channel models (3GPP path loss, Rayleigh fading) and typical IoT latency/energy parameters.

Metric Validation: Empirically, $\mathrm{MCR}^2$ is shown to be a reliable proxy for classification accuracy, monotonic across quantization distortion levels for both SVM and MLP.

Figure 2: Sensing accuracy with different coding rate reduction $\Delta R$; accuracy improves monotonically as $\Delta R$ increases.

Algorithmic Convergence: The BCD-based solution converges in approximately 20 iterations, with each sub-block rapidly stabilizing. Estimated computation delays for practical deployment are under 4 ms for moderate feature dimensions, supporting real-time operation.

Comparative and Ablation Study

Baselines Considered:

• Semantic communication (deep JSCC) approach
• Device-level quantization (equal bits per feature per device)
• Uniform time allocation (equal transmission duration per device)
• Single-modality case (WiFi only)

Key Results:

• Superiority Under Constraints: Under tight communication or energy budgets, the proposed algorithm exhibits significantly higher accuracy compared to all baselines.
• Effect of Communication Parameters: Sensing accuracy is positively correlated with allowed transmission delay and bandwidth, but saturates as quantization errors are minimized.
• Dimensionality and Energy Trade-offs: There exists an optimal feature dimension per device; excessive increase leads to reduced accuracy due to overcompression or introduction of noise. Higher energy budgets systematically enhance performance.
• Task Complexity Scaling: Increasing the number of activity classes reduces accuracy in all schemes, but the multi-modal ISCC framework degrades more gracefully.
• Modal Correlation Effects: When all devices use the same modality, optimal resource allocation is dictated by channel quality. For heterogeneous modalities, more resources are directed to semantically informative modalities even with lower channel quality, validating the covariance-informed optimization.

Figure 8: Sensing accuracy versus allowed total transmission delay—performance scales monotonically with resource availability, with the multi-modal ISCC algorithm consistently outperforming baselines.

Figure 6: Sensing accuracy versus communication bandwidth—proposed method maintains a significant margin throughout.

Figure 4: Sensing accuracy versus feature dimension—a sweet spot emerges beyond which further increases hurt due to quantization distortion.

Figure 12: Sensing accuracy versus per-device energy budget.

Figure 7: Sensing accuracy versus the number of classes—multi-modal ISCC is more resilient to increased task complexity.

Theoretical and Practical Implications

This work demonstrates the tractability and utility of task-oriented multi-modal edge intelligence within a joint ISCC framework leveraging information-theoretic metrics. The pivotal role of $\mathrm{MCR}^2$ for both learning and system evaluation bridges local feature learning with global system-level optimization. The results substantiate that cross-modal resource allocation tailored by statistical interdependencies can yield superior robustness and accuracy over single-modal or naïve multi-modal strategies.

Practically, the architecture supports scalable deployment over real-world wireless networks with heterogeneously-equipped edge devices, establishing guidance for real-time, energy-constrained AI inference in smart environments, XR, healthcare, and similar domains.

Conclusion

The paper proposes a task-driven multi-modal ISCC architecture, formulating end-to-end system design as an information-theoretic resource allocation problem and introducing efficient solution methods. Extensive evaluations evidence clear performance gains over single-modality and standard baselines, particularly under stringent latency and energy constraints. Theoretical properties and empirical results highlight the advantages of unified cross-modal optimization and the application of coding rate-based metrics. Future work may extend to dynamic, context-aware adaptation, handling streaming data, and integrating federated learning or privacy-preserving mechanisms into such ISCC systems.

---

Citation: "Task-Oriented Multimodal Edge Intelligence via Integrated Sensing-Communication-Computation" (2607.03907)