BCL: Bayesian In-Context Learning Framework for Information Extraction

Abstract: Existing information extraction (IE) tasks increasingly adopt in-context learning (ICL) with LLMs. However, current approaches either show inconsistent performance across model scales or lack systematic optimization and generalizability. Building on this, we propose BCL (Bayesian In-Context Learning Framework for Information Extraction), the first optimization framework that uses particle filtering with Bayesian updates to systematically refine label representations across IE tasks. Through four steps initialization, observation, weight update, and resampling, BCL generalizes to both sequence labeling and relation classification paradigms. Extensive experiments demonstrate substantial and consistent improvements over existing approaches.

• The paper introduces a Bayesian framework that optimizes in-context learning for IE using particle filtering and semantic decomposition.
• It demonstrates significant performance gains, with improvements up to 7.73 F1 on NER tasks and robust generalization across models.
• The approach achieves data efficiency and mitigates frequency imbalance, enabling effective IE on both lightweight and resource-constrained systems.

Bayesian In-Context Learning for Information Extraction: An Expert Synthesis

Motivation and Context

The information extraction (IE) landscape has been transformed by LLMs employing in-context learning (ICL). However, prior ICL strategies in IE—particularly task transfer and guideline-oriented methods—exhibit deficiencies. Task transfer methods such as ChatIE and CodeIE degrade markedly on smaller models and lack robustness across task types. Guideline-based approaches, exemplified by GuideNER, are restricted to specific tasks (notably NER), rely on heuristic rule selection, and lack generalizable optimization. These constraints hinder practical deployment on diverse models and datasets, motivating a theoretical and systematic optimization paradigm.

Methodological Innovations

"BCL: Bayesian In-Context Learning Framework for Information Extraction" (2606.18620) introduces BCL, a generative and optimization framework underpinned by control-theoretic and probabilistic principles. BCL reframes the context as discrete, controllable variables via semantic decomposition of IE labels into atomic subcategories. This enables systematic search and refinement of label representations using a particle filtering mechanism guided by Bayesian inference.

Formalization:

• The optimization objective is to maximize IE performance on a validation set by identifying an optimal rule configuration $R^*$, consisting of particles (subcategory patterns) with associated confidence weights.
• Each particle represents a granular semantic rule under a specific entity label, initialized via LLM-guided extraction and assigned prior probabilities based on linguistic naturalness (perplexity).
• Particle weights are iteratively updated by combining prior scores and observed IE performance, employing Bayesian posterior update.

Algorithmic Pipeline:

• Initialization: Rule extraction via LLM from training data, setting initial priors.
• Observation: IE inference with candidate rules, yielding log-probability scores representing task-specific confidence.
• Weight Update: Bayesian aggregation of priors and performance-driven likelihoods.
• Resampling: Two-tier process—performance-based filtering for efficiency, followed by LLM-guided mutation for diversity, ensuring exploration and preventing premature convergence.

This pipeline is agnostic to model scale and generalizes across sequence labeling (NER) and relation classification (RE), overcoming the paradigmatic limitations of previous works.

Empirical Evaluation

BCL is evaluated on six canonical IE benchmarks (CoNLL-2003, ACE 2005, GENIA for NER; NYT, CoNLL04, SciERC for RE), using a representative set of LLMs (Qwen2.5-3B/7B, Llama-3.1-8B, Pixtral-12B). The main results demonstrate:

• Consistency and Superiority: BCL achieves substantial improvement over all baselines. For instance, on CoNLL03 (Qwen-2.5-7B), BCL yields 72.83 F1 (+7.73 over GuideNER, +47.19 over ChatIE). On RE tasks, prior methods largely fail (near-zero F1), while BCL maintains robust performance (e.g., 42.46 F1 on CoNLL04).
• Robustness Across Models: Performance remains stable on lightweight models, making BCL suitable for resource-constrained settings.
• Generalization: Rules learned on smaller models transfer effectively to stronger closed-source models (GPT-3.5-turbo), indicating the extraction of generalizable reasoning patterns rather than model-specific heuristics.

Ablation Analyses establish the architectural significance:

• Bayesian weight updates and diversity mutations account for the majority of performance gains.
• Removing these mechanisms results in 9–11 F1 point drops, validating the foundational role of Bayesian filtering and particle diversity.
• Semantic decomposition is critical; breaking semantic alignment of subcategories leads to consistent degradation (−5 to −12 F1).

Data Efficiency:

BCL achieves convergence with 3–5% of training data. Performance saturates quickly, emphasizing its annotation efficiency and practical scalability.

Resilience to Frequency Imbalance:

Performance stratified by entity frequency reveals marginal gaps, confirming that BCL's Bayesian filtering mitigates overfitting to frequent patterns, preserving effectiveness for rare entities.

Theoretical and Practical Implications

BCL formalizes IE in LLMs as a control system, optimizing externally observable rules rather than intractable internal states. This approach resonates with sequential Monte Carlo methods and Bayesian optimization in black-box environments. Unlike prompt engineering or RLHF, BCL systematically adapts the demonstration set, leveraging Bayesian reasoning for principled belief updates and exploration/exploitation balance.

Practical Applications:

• High-performing IE systems on resource-constrained hardware, democratizing access for a wider range of practitioners.
• Unified framework for structured prediction encompassing NER and RE, facilitating consistent protocol across IE paradigms.

Future Directions:

• Direct extension to other structured prediction tasks (event extraction, entity linking).
• Enhanced particle utilization efficiency; further reductions in computational overhead during optimization.
• Integration with active learning to adaptively select optimal rule sets with minimal annotation and computational burden.

Conclusion

BCL represents a systematic and generalizable Bayesian optimization framework for in-context information extraction. By treating semantic rules as controllable variables and refining them via particle filtering and Bayesian inference, BCL delivers consistent, robust, and efficient IE capabilities across tasks and LLMs. The results underscore the importance of principled context utilization and semantic decomposition in the control and optimization of LLM-driven IE systems.