MicLog: Meta In-Context Log Parsing

• The paper presents MicLog, a log parsing framework that meta-trains small LLMs via a progressive in-context learning curriculum to generalize to unseen log templates.
• It integrates weighted DBSCAN sampling and enhanced BM25 demonstration selection to tackle semantic variation and data scarcity in semi-structured logs.
• A multi-level cache system reduces LLM query overhead by over 40%, yielding state-of-the-art accuracy on large-scale, real-world log datasets.

MicLog is a log parsing framework that implements progressive meta in-context learning (ProgMeta-ICL) with small, open-source LLMs, notably Qwen-2.5-3B, to convert semi-structured log messages into structured templates. Log parsing is foundational for system analysis, powering downstream processes such as anomaly detection and root cause diagnosis. MicLog introduces a unified solution that addresses two persistent barriers in LLM-based log parsing: the underutilization of in-context learning capabilities—especially in dynamic demonstration selection and cross-domain generalization—and the computational overhead and expense associated with repeated LLM invocations. The system integrates a meta-learning approach with an in-context learning curriculum, advanced sampling and demonstration strategies, and a multi-level cache, yielding state-of-the-art accuracy and substantial efficiency gains on large-scale, real-world log datasets (Yu et al., 11 Jan 2026).

1. Problem Formulation and Motivation

The central challenge in log parsing is to map each raw semi-structured message $l$ to a template $t$ of constant segments interleaved with wildcard symbols (“<*>”) that encapsulate variable parameters. Two core difficulties arise:

• Semantic Variation: Log messages that share the same structural template may express themselves lexically in diverse forms (e.g., “disk /dev/sda1 full” vs. “disk /dev/nvme0n1 at 95% capacity”), thwarting syntax-based approaches.
• Data Scarcity and Domain Shift: Each system or deployment may introduce novel templates for which supervised, data-driven methods lack coverage, while syntax-based methods are unable to generalize semantically.

MicLog’s architecture is predicated on the insight that meta-training an LLM to “learn how to in-context learn” (ProgMeta-ICL) promotes generalization to new, previously unseen templates from zero- or few-shot exemplars. Additionally, to mitigate the high computational cost of LLM queries, MicLog incorporates a lightweight, multi-level cache targeting the temporal locality and structural recurrence of log messages (Yu et al., 11 Jan 2026).

2. Framework Architecture

MicLog operationalizes its paradigm in three coordinated modules:

• Weighted DBSCAN Sampling: The method begins with density-based clustering (DBSCAN) in complexity space to sample two diverse subsets from a deduplicated log corpus $\mathcal{D}$: one for meta-training and one for inference candidate selection. The log complexity metric, $$\mathrm{complexity}(l) = (\# \text{tokens}_l)^{\# \text{tokens}_l} + \mathrm{length}_l$$, is normalized to prevent numerical overflow, and sampling from clusters is weighted by this complexity.
• Progressive Meta-In-Context Learning: The LLM is meta-trained across a curriculum of parsing tasks progressing from zero-shot to $k$-shot (e.g., $k=5$) exposures. The meta-loss objective encourages optimization across all shot levels:

$$\mathcal{L}_{\mathrm{ProgMeta\text{-}ICL}} = \mathbb{E}_{\mathcal{T}\sim\mathcal{P}(\mathcal{T})} \left[\mathcal{L}_{\mathrm{ICL}}(\mathcal{T}, S_T^{(\mathcal{T})})\right]$$

with sequential demonstration exposure within each task.

• MLCELI-Parser with Multi-Level Cache: Runtime inference utilizes a two-tiered cache: an exact-match Least Recently Used (LRU) cache and a pattern cache. Messages are first searched in the LRU; if not found, they undergo pattern cache matching via constant-segment validation before resorting to a full LLM query. Demonstrations for LLM prompting are dynamically selected from the candidate pool via enhanced BM25 ranking and are ordered by ascending similarity to maximize template diversity.

3. Progressive Meta In-Context Learning Methodology

The ProgMeta-ICL protocol refines LLM generalization for log parsing through meta-learning:

• Zero-Shot to $k$-Shot Task Schedule: The LLM is meta-trained not with a fixed demonstration count per task but through a curriculum of tasks containing 0 to $k$ demonstrations, formalized over a distribution $\mathcal{P}(\mathcal{T})$ of parsing tasks.
• Clustering-Based Candidate Selection: Weighted DBSCAN, applied over a log complexity metric, forms clusters that ensure diverse coverage of log types. Logs are sampled from clusters based on normalized complexity-derived weights for both meta-training and candidate demonstration pools.
• Demonstration Ranking via Enhanced BM25: At inference, top-$k$ demonstrations are selected using BM25 scoring, where IDF and TF metrics are tailored to the properties of log text. Unlike conventional approaches, demonstrations are ordered by ascending similarity so that prompts capture maximal structural and lexical diversity, which facilitates generalization.

4. Multi-Level Pre-Query Cache Mechanism

The cache system in MicLog is constructed to maximize efficiency without sacrificing coverage:

• LRU Cache: Stores exact $\langle l_i, t_i \rangle$ log-template pairs, supporting constant-time lookup and eviction by recency.
• Pattern Cache: Maintains a subset of normalized templates $t_p$. Upon an LRU miss, incoming messages are normalized and undergo segment-wise matching against each template’s constant segments to determine structural similarity. On pattern cache hit, the matched template is promoted into the LRU.
• LLM Query Fallback: Only if both caches miss is the LLM invoked using a new prompt constructed with freshly selected demonstrations; the output template and message are then saved in both caches.

This design exploits the bursty, repetitive nature of log traffic: repeated or structurally similar messages are intercepted at the cache layer, resulting in over 40% reduction in LLM query volume during high-throughput parsing (Yu et al., 11 Jan 2026).

5. Empirical Evaluation

MicLog was evaluated on Loghub-2.0, which comprises 14 real-world datasets (e.g., HDFS, Hadoop, Zookeeper) with over 50 million log messages and 3,488 ground-truth templates. The framework was compared against syntax-based (Drain, Brain), semantic-tuned (LogPPT), and contemporary LLM-based (LUNAR, LibreLog, LILAC, AdaParser) log parsers.

Performance Comparison:

Method	Parsing Accuracy (PA)	Precision Template Accuracy (PTA)	Recall Template Accuracy (RTA)
Drain	45.0	24.3	38.5
Brain	41.5	32.8	40.5
LogPPT	73.6	47.0	49.0
LUNAR	76.5	75.6	78.8
LibreLog	85.4	76.8	75.8
LILAC	85.8	79.1	81.0
AdaParser	88.5	84.6	85.3
MicLog	97.6	95.3	90.5

Compared to AdaParser, MicLog achieves improvements of +10.3 percentage points in PA, +12.6 points in PTA, and +6.1 in RTA. A one-sided Wilcoxon signed-rank test indicates $p<0.005$ for PA and PTA, and $p<0.05$ for RTA. Additionally, MicLog reduces total parsing time by 42.4% relative to AdaParser (as measured using on-premise LLM inference rather than a cloud service) (Yu et al., 11 Jan 2026).

6. Design Rationale and Limitations

Several factors distinguish MicLog's performance:

• Meta-Learning via ProgMeta-ICL: Enhances the LLM’s capacity for cross-template adaptation by meta-training across a gradient of demonstration exposures.
• Weighted Diversified Sampling: Clustering and weighted sampling ensure exposure to a wide variety of log structures and complexities, avoiding overfitting to common or trivial forms.
• Semantically-Rich Prompt Construction: Enhanced BM25 demonstration selection and ascending similarity ordering fortify the instructive value of in-context examples.
• Exploiting Log Locality: The two-tier cache leverages statistical properties of logging behavior, with high temporal locality yielding high cache hit rates and efficiency.

However, several constraints remain:

1. The small, open-source Qwen-2.5-3B model may not reach the accuracy of significantly larger or proprietary LLMs, albeit with much lower computing and monetary costs.
2. The cache strategy presupposes sufficient repetition in real-world logging; workloads with predominantly unique, nonrecurring logs will achieve diminished caching benefits.
3. Meta-training relies on the corpus family from which inference tasks are drawn; generalization across disparate logging ecosystems (such as from cloud to mobile logs) remains an open challenge.

This suggests that while MicLog provides robust performance for common enterprise and system logs, its transferability to novel or less-structured domains warrants further study.

7. Prospects and Future Directions

Anticipated research avenues for MicLog include:

• Expansion to Multimodal Telemetry: Integrating logs with other system measurements such as metrics and traces for multimodal parsing and cross-signal analysis.
• Adaptive Template Drift Handling: Incorporating online clustering (e.g., adaptive DBSCAN) for continuous template evolution and dynamic parsing adaptation.
• Template Drift Detection and Update Triggers: Mechanisms to automatically recognize and respond to shifts in log format, facilitating scheduled cache refreshes and meta-training updates.

A plausible implication is that MicLog’s mechanisms—meta-learned in-context adaptation, informed clustering, advanced information retrieval (IR) techniques, and multi-level caching—provide a blueprint for scalable, accurate log parsing in environments where log message formats, contents, and recurrence patterns continually evolve (Yu et al., 11 Jan 2026).