---
title: Sola AI Agent for Open-World ISPM
url: https://www.emergentmind.com/topics/sola-ai-agent
type: topic
---

# Sola AI Agent for Open-World ISPM

A Sola AI Agent denotes a class of autonomous, agentic AI systems designed and benchmarked primarily for open-world, continual decision-making and data-driven reasoning, with concrete instantiations in security posture management and self-initiated open-world learning. The most prominent instantiation is the Sola AI Agent for Identity Security Posture Management (ISPM), introduced in conjunction with the Sola-Visibility-ISPM Benchmark, where Sola denotes “Self-Initiated Open-World Learning Agent” principles applied to foundational enterprise security tasks. The design philosophy is rooted in data-grounded, tool-using architectures with explicit natural-language (NL) to executable data exploration capability, continuous schema grounding, and verifiable, evidence-backed outputs [2601.07880]. Theoretical frameworks for Sola agents emphasize autonomous detection of novelties, self-driven label acquisition, and incremental adaptation as outlined by the SOL/SOLA paradigms [2110.11385, 2203.08994].

## 1. Architectural Framework and Layered Design

A Sola AI Agent is structured as a five-layer stack reflecting a progression from natural-language interpretation to evidence-backed synthesis:

1. **Natural-Language Interpretation:** Parses user ISPM queries, classifying intent (e.g., inventory vs. hygiene checks) and target platforms (AWS, Okta, Google Workspace).
2. **Retrieval Layer:** Dynamically acquires platform schema definitions, examples of historical SQL/QA patterns, and scaffold queries, filtered according to live metadata.
3. **Reasoning Engine:** Selects between fast-path template adaptation and explicit multi-step, Tree-of-Thought–decomposition, maintaining a journal of intermediate steps and validations.
4. **Tool Execution:** Generates and executes SQL/API calls against AWS, Okta, and Google Workspace, leveraging normalized tabular representations via an embedded query engine (e.g., DuckDB).
5. **Evidence Synthesis:** Aggregates platform-specific results, producing natural-language answers with citations to exact queries and returned data rows [2601.07880].

This stack is tightly coupled with a schema-grounded orchestration layer—“Identity Data Connector”—that normalizes disparate API and platform outputs, supporting cross-platform data joins and consolidated identity views.

## 2. Natural-Language–to–Data-Exploration Pipeline

The agent’s data-grounded reasoning pipeline is algorithmically formalized as follows:

- **Query Parsing:** ISPM questions are parsed and classified, extracting target entities and ISPM dimension.
- **Schema and Example Retrieval:** The agent loads relevant platform schemas, filtering example SQL queries for high structural compatibility.
- **Execution Mode Selection:** Using an embedding-based schema-template similarity, mode selection defaults to fast-path when similarity exceeds a data-driven threshold ($\tau$); otherwise full-path decomposition is invoked.
- **Full-Path Decomposition:** The question is broken into sub-queries $S_1,\ldots,S_n$, each step generating an SQL statement, executing it, and evaluating explicit step-wise success criteria. Failed criteria induce further decomposition or plan refinement.
- **Aggregation and Synthesis:** All intermediate and final results are merged, with a natural-language answer generated that exposes supporting queries and rows. A full-path “step journal” captures the entire thought process and execution trace [2601.07880].

Pseudocode found in the primary source formalizes this pipeline, with algorithmic blocks for plan generation, SQL synthesis, tool execution, stepwise validation, and evidence consolidation.

## 3. Data Sources, Schema Management, and Cross-Platform Correlation

Sola AI Agents are engineered for multi-platform identity and security environments:

- **AWS IAM:** Tables include IAM users, roles, groups, customer- and AWS-managed policies, access keys, MFA settings, password policies, and CloudTrail-derived logs.
- **Okta:** Includes user/group inventories, sign-on/MFA policies, application assignments, and IdP (SAML) configuration.
- **Google Workspace:** Covers users, organizational units, Super-Admin flags, 2-Step Verification status, OAuth allow-lists, and Drive-sharing.
- **Global Identity Directory:** Enables correlation for entities (e.g., by email, SAML subject) across schemas, supporting queries such as “Which users lack MFA in any system?” via schema-agnostic linking [2601.07880].

Upon each query, the agent loads the latest schema metadata, ensures SQL generation conforms to platform constraints, and normalizes query results to enable evidence-backed multi-platform answers.

## 4. Benchmarking, Evaluation Metrics, and Quantitative Results

The Sola Visibility ISPM Benchmark consists of 77 natural-language tasks partitioned into Inventory and Hygiene (AWS, Okta, GWS) domains, each grounded in production-grade enterprise identity environments. Core metrics include:

- **Expert Accuracy:** Ordinal, per-question metric: $\text{accuracy} = (1/N) \sum_{i=1}^N \text{score}_i$, where $\text{score}_i \in \{0, 0.5, 1\}$ from five human experts.
- **Expert Success Rate:** Strict proportion of perfectly answered questions: $\text{success rate} = (\#\{\text{score}_i=1\})/N$.
- **LLM-as-Judge Correctness:** Parallel (min-pooled) scoring by large language model judges.
- **Sub-metrics:** In full-path, additional criteria include AnswerRelevancy, Faithfulness, ReasoningCoherence, SQLSemanticAppropriateness, and ExampleAdaptation [2601.07880].

Key results (Table 1, main text):

| Domain             | Expert Accuracy | Expert Success Rate |
|--------------------|----------------|--------------------|
| AWS Hygiene        | 0.95           | 0.90               |
| Google Workspace   | 0.75           | 0.71               |
| Okta               | 0.65           | 0.50               |
| Inventory          | 0.75           | 0.64               |
| **Overall (77 Qs)**| **0.84**       | **0.77**           |
| LLM Judgment       | 0.82           | —                  |

Performance is maximized in AWS hygiene tasks, reflecting schema regularity and retrieval strength. GWS and Okta hygiene require more robust example adaptation due to schema heterogeneity.

## 5. Core Algorithms: Reasoning, Planning, and Adaptation

Sola AI Agents leverage two principal algorithmic bases:

- **Embedding-Based Example Similarity:** Retrieval of near-example SQL queries is driven by vector similarity computed between NL query embeddings and historical patterns.
- **Tree-of-Thought–Style Planning:** For divergent or complex queries, multi-step reasoning iteratively decomposes questions, aligns SQL generation with schema semantics, and journals all steps—each checked against explicit success criteria before proceeding.
- **Evidence Aggregation and Verification:** The final answer synthesizes information from all platforms, with citations to each SQL and returned evidence row, supporting auditability [2601.07880].

Fast-path reasoning enables low-latency adaptation for frequent/homogeneous questions, while the full-path approach guarantees fine-grained interpretability and alignment in complex, multi-platform queries.

## 6. Limitations and Prospective Research Directions

While Sola AI Agents set a new standard for data-grounded agentic reasoning in ISPM, certain limitations are noted:

- **Current Scope:** Restricted to inventory and hygiene (visibility) questions; complex risk analytics, behavioral modeling, and active mitigation remain out of scope.
- **Schema Heterogeneity:** Variance in GWS and Okta schema adaptation can degrade performance, with a 0.32 ExampleAdaptation score under full-path for GWS.
- **Auditability Limitation:** Fast-path runs lack a stepwise reasoning trace.
- **Generality:** The deployment and evaluation are presently constrained by the platforms and schemas with robust API/data access [2601.07880].

Future work targets multi-tenant and advanced governance tasks, cross-system behavioral reasoning, RL-based mode selection (fast/full-path) under latency constraints, higher-quality example retrieval through fine-tuning, and extension to identity sources beyond AWS/Okta/GWS, such as Azure AD and SaaS applications.

## 7. Theoretical and Broader Agentic Context

The design and operation of Sola AI Agents align with the broader SOL (“Self-Initiated Open-World Learning”) and SOLA (“Self-Initiated Open-World Continual Learning and Adaptation”) paradigms [2110.11385, 2203.08994]:

- **Autonomy:** Sola agents self-initiate incremental learning cycles, continually detecting and characterizing novelty in non-i.i.d. environments.
- **Continual Adaptation:** Instead of batch retraining, agents employ interaction modules for on-the-fly ground-truth acquisition and incremental model updates (e.g., gradient-based with rehearsal or regularization).
- **Agentic Capabilities:** The architecture supports lifelong operation, open-world skill extension, and mixed-initiative interactions for label or clarification acquisition, with risk evaluation mechanisms to mediate exploratory adaptation in safety-critical contexts.

A plausible implication is that Sola AI agents represent a robust prototype for future agentic systems required to address high-stakes, open-world reasoning and data-driven exploration tasks, integrating security, safety, and continual adaptability. Sola’s deployment in ISPM constitutes a reference implementation, but the broader architectural and algorithmic template is suitable for domains including service robotics, autonomous navigation, and conversational agents [2110.11385, 2203.08994].

Source: https://www.emergentmind.com/topics/sola-ai-agent