Mapping Smarter, Not Harder: A Test-Time Reinforcement Learning Agent That Improves Without Labels or Model Updates

Abstract: The Enterprise Intelligence Platform must integrate logs from numerous third-party vendors in order to perform various downstream tasks. However, vendor documentation is often unavailable at test time. It is either misplaced, mismatched, poorly formatted, or incomplete, which makes schema mapping challenging. We introduce a reinforcement learning agent that can self-improve without labeled examples or model weight updates. During inference, the agent: 1) Identifies ambiguous field-mapping attempts. 2) Generates targeted web-search queries to gather external evidence. 3) Applies a confidence-based reward to iteratively refine its mappings. To demonstrate this concept, we converted Microsoft Defender for Endpoint logs into a common schema. Our method increased mapping accuracy from 56.4\%(LLM-only) to 72.73\%(RAG) to 93.94\% over 100 iterations using GPT-4o. At the same time, it reduced the number of low-confidence mappings requiring expert review by 85\%. This new approach provides an evidence-driven, transparent method for solving future industry problems, paving the way for more robust, accountable, scalable, efficient, flexible, adaptable, and collaborative solutions.

• The paper presents a novel test-time RL agent that improves schema mapping by iteratively integrating external evidence and using confidence scores as proxy rewards.
• The methodology involves generating candidate mappings, detecting conflicts through multiple prompt variants, and refining mappings over 100 iterations without model weight updates.
• Experimental results show a jump in accuracy from 72.73% to 93.94% and an 85% reduction in low-confidence fields, demonstrating significant efficiency gains in real-world log integration.

Test-Time Reinforcement Learning for Schema Mapping Without Labels or Model Updates

Introduction

This paper addresses the persistent challenge of schema mapping in enterprise log integration, particularly in cybersecurity contexts where source schemas are often poorly documented and labeled data is unavailable. The authors propose a test-time reinforcement learning (RL) agent that iteratively improves schema mapping accuracy by leveraging confidence-based proxy rewards and external evidence collection, all without updating model weights or requiring ground-truth labels. The approach is motivated by the operational constraints of large-scale log ingestion pipelines, where retraining or fine-tuning LLMs is computationally expensive and often infeasible.

Positioning Within the Schema Mapping Pipeline

The proposed agent is situated downstream of raw log parsing and upstream of code generation and deployment in the schema integration pipeline. It extends beyond prior work by operating in scenarios where documentation is incomplete or missing, and where traditional retrieval-augmented or fine-tuning-based methods are ineffective due to the absence of labeled data or comprehensive knowledge bases.

Figure 1: Position of the test-time RL agent relative to prior work in the schema matching pipeline, highlighting its unique ability to handle unseen logs with incomplete documentation.

Methodology

Problem Formulation

Given a source schema $S = \{s_1, ..., s_n\}$ and a target schema $T = \{t_1, ..., t_m\}$, the goal is to learn a mapping $f: S \rightarrow T \cup \{\emptyset\}$ that maximizes semantic correctness. In the absence of ground-truth mappings, the agent uses a confidence score $C(f(s_i))$—defined as the consistency of predictions across multiple prompt variants—as a surrogate objective.

Test-Time RL Framework

The agent operates in an iterative loop:

1. Mapping Generation: For each iteration, the LLM generates candidate mappings using the current evidence context.
2. Conflict Detection: Multiple prompt variants are used to identify fields with inconsistent mappings.
3. Evidence Collection: For ambiguous fields, the agent formulates targeted web search queries and collects external evidence.
4. Confidence Evaluation: The agent computes the confidence score for each mapping before and after evidence integration.
5. Context Update: Evidence is retained only if it increases confidence; otherwise, it is discarded.

This process is repeated for a fixed number of iterations (default: 100), with the agent's context (evidence set) serving as a dynamic, non-parametric memory that guides future decisions.

RL Formalization

• State: Current mapping hypothesis and accumulated evidence.
• Action: Selection of a field to investigate and formulation of a search query.
• Reward: Change in confidence score ($r_t = C_{t+1} - C_t$).
• Policy: LLM-driven mapping conditioned on context and evidence.

No model weight updates are performed; all adaptation occurs via context enrichment and evidence accumulation.

Experimental Results

Setup

The evaluation uses Microsoft Defender for Endpoint logs (195 fields) mapped to a proprietary Common Schema (137 fields), with 66 manually verified ground-truth mappings for retrospective accuracy assessment. GPT-4o serves as the underlying LLM. Two baselines are established: (1) single-shot prompt with field name/value (56.36% accuracy), and (2) single-shot prompt with enhanced context (72.73% accuracy).

Performance Analysis

The RL agent achieves a final accuracy of 93.94% after 100 iterations, starting from the 72.73% baseline. The number of low-confidence fields requiring expert review is reduced by 85% (from 26 to 4). The agent accumulates 81 evidence tuples, with the most significant accuracy gains occurring in early iterations. Notably, in 19 out of 100 iterations, the agent rejects new evidence, indicating effective self-regulation.

Figure 2: Performance analysis over 100 iterations: accuracy increases from 72.73% to 93.94%, confidence approaches 1.0 (with persistent overconfidence), and conflict field count drops by 85%.

Confidence as a Proxy Reward and Overconfidence

The agent uses confidence (prediction consistency across prompt variants) as a reward signal. While confidence generally tracks accuracy, a persistent overconfidence gap is observed: confidence values saturate near 1.0 even when accuracy lags behind. Increasing the number of inference attempts per iteration (from 3 to 10) improves calibration, reducing the overconfidence gap at the cost of slightly lower accuracy and increased computation.

Figure 3: Relationship between confidence and accuracy, illustrating the overconfidence phenomenon and the need for improved calibration.

Evidence Collection and Reasoning

The agent's evidence collection is structured and targeted. For each conflict, it formulates a search query, retrieves evidence, and integrates it only if confidence improves. Evidence is categorized as: (1) clarifying ambiguous fields, (2) enabling self-correction, (3) providing authoritative definitions, and (4) revealing context-dependent mappings. A case study on direction-sensitive port mapping demonstrates that evidence-based reasoning enables the agent to infer conditional mapping rules that cannot be learned via fine-tuning alone.

Implementation Considerations

• Prompt Architecture: The system uses a layered prompt design: a session-level system prompt (defining expert reasoning), a per-request user prompt (with context and mapping task), and a search prompt (for query generation).
• Computational Cost: The iterative process incurs higher computational overhead than single-shot approaches due to repeated LLM calls and external searches. However, the reduction in expert review effort and the elimination of retraining offset these costs in operational settings.
• Domain Generalizability: While the evaluation focuses on cybersecurity, the methodology is general and can be extended to other domains with schema mapping challenges, provided that external evidence is accessible.
• Limitations: The approach depends on the quality and availability of external evidence, is currently limited to 1-to-1 mappings, and may require further adaptation for more complex mapping cardinalities.

Theoretical and Practical Implications

The use of confidence as a proxy reward for test-time RL in schema mapping introduces a new paradigm for self-improving agents in settings where labeled data is unavailable and model retraining is impractical. The approach demonstrates that verbal, memory-based adaptation—driven by evidence accumulation and confidence evaluation—can yield substantial accuracy gains and operational efficiency. The persistent overconfidence phenomenon highlights the need for better calibration methods, such as ensemble-based confidence estimation or explicit self-assessment prompts.

Future Directions

Potential avenues for further research include:

• Calibration Techniques: Developing more robust confidence estimation methods, including ensemble approaches and explicit self-assessment.
• Generalization to Complex Mappings: Extending the framework to handle 1-to-N and N-to-M mappings.
• Automated Evidence Quality Assessment: Integrating mechanisms to assess and filter the quality of external evidence.
• Cross-Domain Validation: Applying the methodology to other domains (e.g., healthcare, finance) to assess generalizability.

Conclusion

This work presents a practical and theoretically grounded approach to schema mapping in enterprise log integration, leveraging test-time RL with confidence-based proxy rewards and external evidence collection. The agent achieves substantial improvements in mapping accuracy and reduces expert review burden without requiring model updates or labeled data. The findings underscore the viability of context-driven, verbal RL for real-world AI deployment and highlight open challenges in confidence calibration and evidence quality assessment.