OS-Sentinel: Safety & Remote Sensing Frameworks

• The paper presents a hybrid framework that combines rule-based verifiers with VLM-based judges to improve safety detection in mobile GUI agents by 10–30% in F1 scores.
• OS-Sentinel is a dual-framework system that addresses both safety-critical mobile automation and land cover mapping through risk-balanced fusion and ensemble modeling.
• The research demonstrates robust performance with statistically significant improvements, achieving high Cohen’s kappa in remote sensing and enhanced violation localization in mobile environments.

OS-Sentinel encompasses two distinct, rigorously developed frameworks prominent in (1) hybrid safety validation for mobile GUI agents and (2) multisource remote sensing data fusion for land cover mapping. Both lines of research demonstrate convergent principles of risk-balanced fusion, ensemble modeling, and operationally robust evaluation.

1. Hybrid Safety Validation for Mobile GUI Agents

OS-Sentinel, as introduced in (Sun et al., 28 Oct 2025), addresses safety-critical automation in mobile environments driven by vision-LLM (VLM) agents. The OS-Sentinel framework integrates a formal, rule-based verifier with a VLM-based contextual judge to detect explicit system violations and subtle, context-dependent risks in agent workflows.

1.1 Motivation and Systemic Risk Factors

Autonomous agents operating in mobile platforms interact with diverse apps and complex GUIs, managing dynamic permission models, system-level changes, and latent privacy threats. Risks extend beyond malicious instructions to include over-automation, agent misunderstanding, and context or state-dependent violations undetectable by static or purely formal checkers.

Rule-based verifiers reliably flag explicit system-level changes (e.g., unauthorized file modifications, permission escalations, or sensitive-pattern exfiltration), but lack intent discernment and semantic context. VLM-based models interpret GUI semantics and longer-horizon user intent (e.g., distinguishing access to contacts for email composition from exfiltration), but are susceptible to audit gaps for hidden changes and possible misclassifications. OS-Sentinel's hybridization mitigates these limitations, combining transparent, deterministic coverage with contextual reasoning.

1.2 MobileRisk-Live Benchmark

The experimental substrate for OS-Sentinel is MobileRisk-Live, a dynamic Android-based sandbox featuring 48 pre-installed apps, state replay, and stepwise logging of screenshots, accessibility trees, action traces, and system snapshots $\sigma_t$, including package, file, network, and permission states. The benchmark comprises 204 annotated agent trajectories, assigned binary safety labels, first-violation localizations, and a 10-category taxonomy (7 agent-side, 3 user-side) for granular risk classification. Categories include destructive operations, privacy violations, security evasion, UI manipulation, resource abuse, compliance violations, over-automation, malicious use, prompt injection, and harmful content.

2. OS-Sentinel Safety Architecture

2.1 Formal Verifier

OS-Sentinel's formal verifier encodes atomic predicates per timestep $t$:

• Integrity violation $I_t$: flags unexpected file-system changes by comparing SHA256 hashes of file metadata across steps.
• Keyword risk $K_t$: detects presence of sensitive keywords from a curated lexicon in visible GUI elements, with per-keyword weights.
• Pattern risk $P_t$: applies regular expressions for sensitive diagnosable patterns (email, SSN, credit card, phone) with tunable penalty weights.

Aggregate risk is computed as $R_t = \alpha I_t + \beta K_t + \gamma P_t$, triggering a violation if $R_t > \theta_F$.

2.2 VLM-Based Contextual Judge

The contextual judge leverages a fine-tuned VLM (e.g., Qwen2.5-VL, GPT-4o) as classifier $\mathcal{J}_{\theta}$. The classifier encodes GUI observations ($o_t$) and agent actions ($a_t$) into features $t$0, producing a risk probability $t$1 (step flagged if $t$2). Judgments aggregate via sliding windows ("consecutive" mode) or uniform temporal sampling.

2.3 Hybrid Validation and Decision Logic

Stepwise and trajectory-level guards are implemented by logical OR: $t$3, $t$4, where $t$5, $t$6 follows the contextual aggregation rule. The closed-form fusion is $t$7, typically with $t$8.

3. Quantitative Evaluation and Comparative Performance

On the MobileRisk-Live benchmark, trajectory-level F1 scores demonstrate that OS-Sentinel significantly outperforms both standalone rule-based baselines and pure VLM contextual judges across backbones (gpt-oss-120B, Qwen2.5-VL-7B, GPT-4o):

Approach	F1 (gpt-oss-120B)	F1 (Qwen2.5-VL-7B)	F1 (GPT-4o)
Rule-based	52.7%	–	–
VLM-as-Judge	56.3%	54.8%	56.0%
OS-Sentinel	65.3%	65.6%	66.1%

Relative improvements range from 10–30% in F1 and 8–15 absolute points over the best baselines. Paired t-tests confirm statistical significance ($t$9). Step-level localization of violations is comparably improved, with normalized timeliness improvements.

Balanced F1 scores are achieved across all 10 risk categories, unlike rule-based or pure VLM judges, which show high error rates in categories such as destructive operations or prompt injection.

4. Failure Modes, Design Implications, and Limitations

Incomplete context windows can obscure early signals in multi-app workflows; false positives are possible when legitimate agent access to sensitive APIs is benign. Current implementation depends on Android system hooks and is unsuited to closed platforms (e.g., iOS). Semantic limitations follow VLM context-length constraints.

OS-Sentinel's hybrid guard can be deployed in two modes: in-loop (step-level monitoring) for early warning, and post-hoc (trajectory-level) audits for compliance review. Prospective extensions include integration of additional system checks (network, IPC logs), adaptive decision thresholds, and user-in-the-loop override for ambiguous actions. Generalizing to other platforms would require new formalizer development.

5. Remote Sensing: OS-Sentinel for Land Cover Mapping

A distinct instantiation of OS-Sentinel, as presented in (Pande, 2023), provides a state-of-the-art pipeline for land use/land cover (LULC) classification by fusing Sentinel-1 synthetic aperture radar (SAR) and Sentinel-2 visible–near-infrared to shortwave-infrared (VNIR–SWIR) imagery.

5.1 Bayesian Fusion Framework

At the core is a decision-level (pixel-wise) Bayesian model integrating a multidimensional Sentinel-2 VNIR–SWIR vector $I_t$0 with a SAR-derived GLCM homogeneity texture scalar $I_t$1 (from the VH polarization). Assuming a flat prior and independent Gaussian noise models for each modality, the posterior for the fused radiometric vector $I_t$2 is

$I_t$3

with maximum-a-posteriori estimate

$I_t$4

with $I_t$5 for 60% SAR weight.

5.2 Preprocessing, Feature Extraction, and Texture

The pipeline (SNAP+ENVI+MATLAB) conducts:

• Sentinel-1A preprocessing: radiometric calibration, orthorectification, generation of derived bands.
• Sentinel-2B preprocessing: bands resampled to 10m, atmospheric correction with Sen2Cor, ratio indices (e.g., RDVI).
• Coregistration onto a common grid.
• Texture extraction on VH: GLCM (window $I_t$6) for Haralick "homogeneity," yielding a single texture map band.
• Bayesian fusion at each pixel for a unified 6-vector feature.

5.3 Rotated Random Forest Ensembles (RFEs)

OS-Sentinel utilizes three rotation-based forest ensemble methods:

• PCA-RFE: Block-diagonal rotation constructed from local PCA in feature subsets.
• SRP-RFE (Sparse Random Projection): Subsetwise independent Gaussian random matrices, $I_t$7-normalized per column.
• CRP-RFE (Complete Random Projection): Full $I_t$8 Gaussian rotation with column normalization.

Ensembles are tuned ($I_t$9–$K_t$0 for SAR, up to $K_t$1 for VNIR sets) and tree count per RF is fixed at 30.

5.4 Classification Protocol and Benchmarking

Datasets compared:

1. SAR bands only
2. SAR+texture
3. VNIR–SWIR only
4. VNIR–SWIR+texture
5. Bayesian fused VNIR–SWIR+texture

Accuracy is assessed using Overall Accuracy (OA) and Cohen's kappa ($K_t$2), averaged over 25 randomized splits at training fractions 2–20%.

Dataset	Best RFE Type	$K_t$3 (RFE)	$K_t$4 (RF)
SAR only	SRP	61.80%	60.61%
SAR+texture	SRP	68.18%	66.21%
VNIR–SWIR	CRP	95.99%	94.08%
VNIR–SWIR+texture	CRP	96.93%	95.48%
Fused VNIR+texture	CRP	96.30%	94.55%

Addition of GLCM-homogeneity texture results in a maximal $K_t$5 gain of 10% on SAR-only sets and $K_t$63.5% on VNIR–SWIR sets. CRP yields superior performance in spectrally rich VNIR feature spaces; SRP is advantageous on lower-dimensional SAR sets.

6. Comparative Significance and Outlook

Both OS-Sentinel frameworks establish new paradigms in their respective domains: hybrid audit/comprehension for agent safety (Sun et al., 28 Oct 2025) and cross-modality remote sensing fusion with ensemble classifiers (Pande, 2023). The hybrid validation approach is demonstrated to be practical (66 ms/detection reported) and achieves 10–30% F1 gains over single-mode detectors. In land cover mapping, Bayesian-fused texture-optical features with CRP-RFE deliver over 96% $K_t$7 on VNIR–SWIR mapping and nearly 70% $K_t$8 on SAR-texture, outperforming standard random forests.

Limitations persist in out-of-domain generalization, platform trace access, and context-length of VLM models; future work is motivated to extend formalizability to closed platforms, enhance semantic context capacity, and refine thresholding for reduced false-positives.

• "OS-Sentinel: Towards Safety-Enhanced Mobile GUI Agents via Hybrid Validation in Realistic Workflows" (Sun et al., 28 Oct 2025)
• "Land use/land cover classification of fused Sentinel-1 and Sentinel-2 imageries using ensembles of Random Forests" (Pande, 2023)