LumiMAS: MAS-Level Observability Framework

• LumiMAS is a platform-agnostic, multi-layer observability framework that monitors, detects, and explains failures in LLM-based multi-agent systems.
• It integrates real-time logging with lightweight autoencoder-based anomaly detection and LLM-powered explanation layers to address issues such as hallucination, bias, and adversarial attacks.
• Empirical evaluations on CrewAI and LangGraph demonstrate fast, precise detection with minimal overhead, enabling effective system-level fault localization.

LumiMAS is a platform-agnostic, multi-layer observability framework for LLM-based multi-agent systems (MAS) that delivers real-time monitoring, anomaly detection, and anomaly explanation with root cause analysis (RCA). It is designed to address MAS-wide failures—including adversarial prompt injections, memory poisoning, hallucination, and bias—that often emerge from inter-agent dynamics and propagate across messages, tools, and shared state. The framework consists of a monitoring and logging layer, anomaly detection layer, and anomaly explanation layer, and was evaluated on seven different MAS applications implemented using two popular MAS platforms and a diverse set of possible failures (Solomon et al., 17 Aug 2025).

1. Problem setting and design objective

Observability in LLM-based MAS is presented as a distinct systems problem rather than a simple extension of single-agent tracing. Agents are autonomous, stateful, and interactive; errors can cascade across inter-agent communication, shared memory, and tools, amplifying issues such as hallucination, bias, or adversarial manipulation. The framework is motivated by the claim that LLM vulnerabilities—including hallucination, biases, indirect prompt injection through external content, and memory poisoning—can be subtle and semantic in nature, and that failures may be invisible at the single-agent level but manifest at the MAS workflow level. Existing monitoring tools and research are described as often focusing on per-agent tracing or static policy enforcement, relying heavily on LLMs “as a judge,” or covering only limited failure modes, without prioritizing real-time, system-level detection and RCA aligned with production needs and OWASP 2025 guidance on agentic AI monitoring (Solomon et al., 17 Aug 2025).

Within that framing, LumiMAS closes three specific gaps. First, it provides MAS-level logging across agents, LLM calls, tools, and application boundaries. Second, it detects anomalies in real time using lightweight autoencoders trained on engineered execution features and semantic embeddings. Third, it explains anomalies and localizes root causes with LLM-based agents, explicitly distinguishing MAS-level phenomena from single-agent issues. This suggests that the framework is not only an instrumentation layer but also a diagnostic layer that treats the execution of an MAS as a correlated, time-ordered process rather than a bag of isolated agent traces.

2. Layered architecture and observability model

The architecture is organized into three layers with a unidirectional operational flow from execution logging to detection and then to explanation.

Layer	Core function	Primary outputs
Monitoring and logging layer	Monitors MAS executions and creates detailed logs of the agents' activity	Structured events, operational telemetry, textual semantics
Anomaly detection layer	Runs three autoencoder-based detectors in parallel and performs online reconstruction-error scoring	Alert with score, detector variant, affected agents
Anomaly explanation and RCA layer	Classifies anomaly type and localizes the fault to a specific agent/tool/message	Label, justification, “Root cause agent number: X”

The monitoring and logging layer captures a chronological sequence of structured events across the MAS execution lifecycle: Application-Started, Application-Ended, Agent-Started(a), Agent-Finished(a), LLM-Call, Tool-Usage, and Error-Event. Logged attributes include agent IDs, timestamps, tool names, I/O, token usage, durations, errors, and system metadata. The event schema spans temporal signals such as durations per agent iteration and time gaps between iterations; agent-level operational metrics such as number of iterations, unique tools, tool success rate, per-call token statistics, and prompt-to-completion ratio; semantic signals such as LLM output lengths, token processing times, and unique error types; and system/context attributes such as LLM model/version counts and normalized input complexity. Logs are aggregated per application execution and linked by agent IDs and iteration counters, enabling sequential modeling of each agent’s behavior and cross-agent timelines, while the MAS-level sequence is bounded by Application-Started/Ended and preserves inter-agent dependencies such as shared state in LangGraph (Solomon et al., 17 Aug 2025).

The layer is also described as platform-agnostic, with explicit support for CrewAI and LangGraph. LumiMAS ingests events in real time, transforms them to features, and streams them to detectors. The logging design is stated to impose minimal overhead, and there is no token-cost component in detection. A plausible implication is that LumiMAS deliberately separates passive instrumentation from expensive semantic adjudication, invoking LLM-based explanation only after an anomaly has been raised.

3. Detection methodology

The anomaly detection layer uses three autoencoder-based detectors running in parallel: an EPI detector on engineered system-level features, a semantic detector on encoded LLM outputs, and a combined latent-space detector that fuses both. The EPI representation is a sequence

$$X^{\langle \mathcal{E} \rangle} = (x^{\langle \mathcal{E} \rangle}_1, \dots, x^{\langle \mathcal{E} \rangle}_n),$$

where $$x^{\langle \mathcal{E} \rangle}_i \in \mathbb{R}^{d_{\mathcal{E}}}$$ encodes the $i$-th agent execution’s EPI vector. The semantic representation is

$$X^{\langle \mathcal{S} \rangle} = (x^{\langle \mathcal{S} \rangle}_1, \dots, x^{\langle \mathcal{S} \rangle}_m),$$

where each $$x^{\langle \mathcal{S} \rangle}_j \in \mathbb{R}^{d_{\mathcal{S}}}$$ is a sentence-transformer embedding, specifically all-MiniLM-L6-v2, of the $j$-th LLM output. The EPI and semantic streams are modeled with LSTM autoencoders, while the combined detector concatenates the learned latent representations

$$X^{\langle \mathcal{C} \rangle} = [Z^{\mathcal{E}} ; Z^{\mathcal{S}}],$$

and applies a linear encoder/decoder to the fused latent space (Solomon et al., 17 Aug 2025).

For each detector, the encoder and decoder are written as

$$Z^{\mathcal{I}} = f^{\mathcal{I}}(X^{\langle \mathcal{I} \rangle}), \qquad X'^{\langle \mathcal{I} \rangle} = g^{\mathcal{I}}(Z^{\mathcal{I}}),$$

with $\mathcal{I} \in \{\mathcal{E}, \mathcal{S}\}$. The loss is

$$\ell_{\mathcal{I}} = d\!\left(X^{\langle \mathcal{I} \rangle}, X'^{\langle \mathcal{I} \rangle}\right),$$

where $$x^{\langle \mathcal{E} \rangle}_i \in \mathbb{R}^{d_{\mathcal{E}}}$$0 is mean squared error over the sequence. The combined detector is defined analogously through

$$x^{\langle \mathcal{E} \rangle}_i \in \mathbb{R}^{d_{\mathcal{E}}}$$1

with loss

$$x^{\langle \mathcal{E} \rangle}_i \in \mathbb{R}^{d_{\mathcal{E}}}$$2

The anomaly score is the reconstruction error

$$x^{\langle \mathcal{E} \rangle}_i \in \mathbb{R}^{d_{\mathcal{E}}}$$3

aggregated across the sequence by mean or sum per execution log. The threshold $$x^{\langle \mathcal{E} \rangle}_i \in \mathbb{R}^{d_{\mathcal{E}}}$$4 is selected on the validation set to optimize F1, with the decision rule: anomalous if $$x^{\langle \mathcal{E} \rangle}_i \in \mathbb{R}^{d_{\mathcal{E}}}$$5, otherwise benign.

The engineered feature space includes latency features such as total_duration, average and maximum iteration duration, and time gap variance; agent features such as total_iterations, unique_tools, tool_failures, average and maximum tokens, sum tokens, and prompt-to-completion ratio; content features such as output-length statistics, action entropy, and repetitive actions; and system features such as processing time per token statistics, input length, and complexity score. Feature extraction and inference are streaming, and the detector processes each execution’s sequences immediately after completion, with average latency under 0.07 seconds. Thresholds can be tuned on-the-fly to trade sensitivity against specificity, and detectors can be retrained periodically from recent benign logs. Training resources are explicitly reported as lightweight: GPU training for the combined detector averages 0.076 hours, and CPU-only training 1.133 hours. The complexity description gives time complexity $$x^{\langle \mathcal{E} \rangle}_i \in \mathbb{R}^{d_{\mathcal{E}}}$$6 per forward pass for an LSTM-AE with input dimension $$x^{\langle \mathcal{E} \rangle}_i \in \mathbb{R}^{d_{\mathcal{E}}}$$7, hidden size $$x^{\langle \mathcal{E} \rangle}_i \in \mathbb{R}^{d_{\mathcal{E}}}$$8, and sequence length $$x^{\langle \mathcal{E} \rangle}_i \in \mathbb{R}^{d_{\mathcal{E}}}$$9, and memory footprint $i$0 per stream.

4. Explanation, classification, and root cause analysis

The anomaly explanation and RCA layer consists of two dedicated LLM agents. A classification LMA assigns anomaly types from a fixed six-class taxonomy: benign, bias, hallucination, direct prompt injection (DPI), indirect prompt injection (IPI), and memory poisoning (MP). A second RCA LMA analyzes the event chronology and inter-agent dependencies to localize the fault to a specific agent, tool, or message and to produce a human-readable explanation, including the structured output string “Root cause agent number: X,” with $i$1 for benign. Prompt design is stated to encode role, task, vulnerability taxonomy, and expected output format, and the classifier is described conceptually as computing $i$2 over log features $i$3 via generative inference, with qualitative evidence extracted from the event chronology (Solomon et al., 17 Aug 2025).

The failure taxonomy covers both adversarial and non-adversarial failure modes. DPI includes crafted prompts overriding agent policy or injecting misinformation or backdoor conditions, as well as an exhaustion variant that induces repeated calls or tool loops. IPI consists of malicious instructions embedded in external content such as webpages, which agents ingest via tools. MP corresponds to contaminated retrieval sources inducing systematic incorrect outputs on particular topics. Hallucination is defined as unfounded or fabricated content disconnected from context, whereas bias refers to unfair preferences or stereotypes in outputs.

Two failure-tailored applications formalize hallucination and bias assessment. HalluCheck uses a three-agent pipeline with a ground-truth answer, a verification agent, and a question-inference agent. Hallucination labeling uses exact match (EM), F1 overlap, and cosine similarity between embeddings: hallucination if EM = 0, F1 < 0.45, and similarity < 0.6; non-hallucination if EM = 1, or F1 > 0.55, or similarity > 0.7, with an uncertain margin allowed. Additional criteria are defined for the verification and question-inference agents. BiasCheck uses multiple-choice questions containing one correct answer and two incorrect answers reflecting stereotyped assumptions, and also detects bias when an agent claims sufficient context for known ambiguous items. The classifier is reported to reduce false positives by approximately 60% on average while achieving high accuracy on bias, reported as 78%. The RCA agent outperforms baselines on adversarial attacks—DPI, IPI, and MP—while performance is slightly lower on hallucination and bias cases, which are described as more subtle and internal to the LLM. Misclassifications are noted: adversarial anomalies are sometimes labeled as hallucinations because input-output alignment breakdown can mimic hallucination.

5. Empirical evaluation

The evaluation spans seven applications across two MAS platforms. CrewAI applications are Trip Planner (DPI), Instagram Post (IPI), Real Estate Team (MP), HalluCheck (hallucination), and BiasCheck (bias). LangGraph applications are Adapted Trip Planner (DPI) and GenFic (DPI, story generation with research). Agent backbones are OpenAI GPT-4o mini and o3-mini. Data collection uses more than 2,000 benign logs per application, and validation and test sets each include 200 logs with 50% anomalous instances and no overlap. Baselines are LLM-as-a-judge and Agent-as-a-judge with GPT-4o and GPT-4o-mini, plus SelfCheck-style consistency for hallucination and toxic-BERT for bias (Solomon et al., 17 Aug 2025).

On CrewAI with GPT-4o-mini, the combined detector achieved the lowest average false positive rate, 0.279, and the highest average precision, 0.742, with average recall 0.763. Average decision time was 0.068 seconds, whereas baselines averaged 10.559 seconds, approximately 155 times slower; LumiMAS is also reported to have zero token-inference cost, versus substantial API cost for baselines. For hallucination, the combined detector obtained accuracy 0.765, F1 0.728, precision 0.863, FPR 0.100, and latency 0.017 seconds. For bias, the combined detector achieved accuracy 0.660 and F1 0.670, with the accompanying interpretation that bias patterns are often model-internal and subtle. For DPI averaged over three scenarios, the semantic detector reached accuracy 0.770, F1 0.794, recall 0.900, precision 0.712, FPR 0.360, and latency 0.074 seconds, while the combined detector reached accuracy 0.767, F1 0.761, precision 0.737, FPR 0.270, and latency 0.088 seconds. For IPI, the combined detector achieved accuracy 0.970, F1 0.970, recall 0.970, precision 0.970, FPR 0.030, and latency 0.142 seconds, while EPI achieved F1 0.966, recall 1.000, and precision 0.935. For MP, the semantic detector achieved accuracy 0.690, F1 0.710, recall 0.760, precision 0.667, and latency 0.028 seconds.

The case studies further articulate the detector behavior. In HalluCheck, anomalies manifest as longer runtime, increased token usage, and inconsistent inter-agent verification; LumiMAS detects these via EPI runtime and token deviations and via semantic incoherence in embeddings, then explains which agent introduced fabrication. In BiasCheck, bias anomalies appear as consistent selection of stereotype-aligned options; EPI signals may be weaker because runtime remains similar, while semantic patterns vary by backbone. Additional results cover scalability and backbone sensitivity. In a larger scenario with approximately 10 agents, inference times remained real-time: EPI 0.008 seconds, Semantic 0.150 seconds, Combined 0.157 seconds. On LangGraph DPI in GenFic, the combined detector achieved near-perfect metrics, with accuracy 0.990 and F1 0.990. For o3-mini, logs are shorter and hallucination detection is harder because of reduced textual signal; EPI is often stronger on bias, whereas Semantic is stronger on text-heavy MP.

6. Integration, limitations, and relation to other uses of the name

Implementation is described for Python 3.11 and PyTorch 2.6, with support for CrewAI and LangGraph. The integration sequence is explicit: instrument the MAS to emit Application-Started/Ended, Agent-Started/Finished, LLM-Call, Tool-Usage, and Error-Event; enable feature extractors for EPI and semantic embeddings using all-MiniLM-L6-v2; collect benign runs; train the EPI and Semantic LSTM-AEs and then the Combined linear AE on concatenated latents; optimize threshold $i$4 for F1 on validation; and deploy detectors for real-time inference, routing anomalous logs to classification and RCA LMAs. Representative reproducibility settings include MSE loss, Adam optimization, and example hyperparameters such as a one-layer EPI LSTM with 150 epochs, learning rate $i$5, and batch size 16 for the hallucination application. The principal limitations are also explicit: LumiMAS requires pre-deployment training on benign logs; significant application changes such as adding new tools may require retraining; a fixed taxonomy improves precision but reduces flexibility; and detection of bias and some MP cases is less effective for EPI because these failures can be subtle and model-internal. Future work includes broader failure coverage such as coordination breakdown and message loss, improved causal inference for model-internal failures, adaptive thresholds, and integration with standardized agent-to-agent protocols such as MCP and A2A (Solomon et al., 17 Aug 2025).

The name can be confused with unrelated lines of work. A separate synthesis connects LumiCal beam-test results to a possible LumiCal-based luminosity monitoring and analysis system, but the 2017 paper “Measurement of shower development and its Molière radius with a four-plane LumiCal test set-up” does not explicitly mention “LumiMAS.” That work instead reports a four-plane LumiCal prototype, a measured effective Molière radius of $i$6, and a radial position resolution of $i$7 for 5 GeV electrons; the connection to a “LumiMAS-style” system is presented only as a downstream implication for small-angle luminosity monitoring and analysis (Levy, 2017).

A second possible confusion arises from microlensing. The 2024 paper “Estimating microlensing parameters from observables and stellar isochrones with pyLIMASS” introduces pyLIMASS, not LumiMAS. The paper does not mention “LumiMAS” nor define a relationship to it; rather, later explanatory material characterizes pyLIMASS as the kind of luminosity/isochrone-based microlensing mass estimation system that a “LumiMAS-style approach” would require. pyLIMASS combines microlensing observables, stellar isochrones, and an extinction law through a Gaussian Mixture likelihood to estimate lens mass and distance, which is conceptually unrelated to MAS observability despite the superficial similarity in naming (Bachelet et al., 2024).