Ontology-constrained multi-LLM scoring of hypothesis support in the predictive processing literature

Abstract: Fragmentation is common in interdisciplinary fields with diverse methods and theoretical commitments. Predictive coding neuroscience is a clear example: its literature spans computational theory, electrophysiology, imaging, behavior, and modeling, creating a synthesis problem that conventional meta-analysis cannot easily resolve. Here, we describe a local multi-LLM pipeline for ontology-constrained literature synthesis. The pipeline reads papers, extracts evidence, incorporates figure descriptions, assembles constrained prompts, and validates outputs against an expert glossary. We manually defined a predictive-coding glossary of thirty-six concepts grouped into three hypotheses: predictive suppression, feedforward error propagation, and ubiquity. A council of ten local LLMs scored 31 studies according to their agreement or disagreement with each glossary factor across local and global oddball contexts. This enabled pairwise study-agreement analysis, cross-model comparison, and three-dimensional hypothesis-space mapping. Agreement was high for some hypotheses but weaker for others, revealing structured disagreement, particularly across local versus global oddball paradigms. We further define hypothesis-space temperature, a geometric dispersion metric measuring how compactly studies occupy the hypothesis space. Temperature was lower for local oddball contexts and higher for global oddball contexts, indicating greater dispersion in the latter. The scoring geometry also allowed us to estimate vectors of change between experimental contexts. These results demonstrate that local multi-LLM councils can produce auditable disagreement measurements that map heterogeneous literatures into quantitative evidence spaces. This framework may generalize to cross-study hypothesis mapping where conventional meta-analysis lacks a common comparison space.

• The paper introduces an ontology-constrained pipeline in which ten local language models score 1,860 evidence items from 31 studies across 36 factors and three predictive-processing hypotheses, with cited reasoning and validation controls.
• The analysis finds that feedforward error propagation receives the strongest local-oddball support (mean 0.51) but declines most sharply in global oddballs (−0.158, p < 0.01), indicating context-sensitive evidence for scale-invariant predictive coding.
• The framework quantifies agreement and theoretical volatility: model consistency is low (AC = 0.11), while hypothesis-space temperature is significantly higher for global than local oddballs, helping identify structured disagreement and influential outliers.

Overview and motivation

This paper presents a local, multi-LLM pipeline for ontology-constrained synthesis of the predictive processing literature (2606.05206). The authors—Nejat, Maier, Spencer-Smith, and Bastos—address a well-recognized problem: predictive coding neuroscience spans computational theory, electrophysiology, imaging, behavior, and modeling, so conventional meta-analysis lacks a common comparison space. Their response is to compress heterogeneous studies into a quantitative "hypothesis space" defined by an expert glossary, scored independently by a council of ten locally hosted LLMs.

The empirical testbed is the local-versus-global oddball distinction (LO/GO), chosen because it operationalizes a central tension in predictive processing: whether short-timescale sensory prediction errors and long-timescale abstract rule violations are generated by the same hierarchical machinery. The local condition defines deviance against immediate sensory context (e.g., xxxY), while the global condition requires working-memory maintenance of abstract sequence rules (e.g., xxxY xxxY xxxX). If predictive coding is scale-invariant, the generative logic of LO should extend to GO; the literature is divided on this point, making it a suitable benchmark.

Pipeline design

The pipeline proceeds through deterministic PDF ingestion, figure-aware processing via a local vision-LLM (gemma-3-9b-it) whose descriptions are interleaved into study markdown files, constrained prompt assembly, scoring by ten open-weight models run through mlx-lm on Apple silicon, and schema validation before downstream analysis. Sampling used temperature $\tau = 0.70$, top-p $p = 0.9$, min-p $\alpha = 0.1$, and a 131,072-token context window with MXFP8 KV-cache precision.

The ontology is a 36-factor glossary partitioned into three hypotheses: H1 (Predictive Suppression) covers subtractive/divisive inhibition, activity suppression, sharpening, and omission responses; H2 (Feedforward Error Propagation) covers supragranular activity, gamma-band signaling, directed connectivity measures, and ascending latency shifts; H3 (Ubiquity) covers cross-area, cross-modal, cross-species, and state-invariant presence of the mechanism. Each factor carries context-scope tags restricting applicability (e.g., habituation factors apply only to LO), preventing sensory adaptation from being scored as evidence for abstract rule learning.

The instruction layer enforces several constraints that distinguish this from unconstrained LLM summarization: scores range from −1 to +1; zero denotes explicitly neutral results while null denotes unaddressed factors; anti-guess rules forbid inventing factor keys or forcing irrelevant contexts; and every non-null score requires a cited section, figure, or table as justification, producing auditable reasoning logs. A validation tool audits syntactic integrity and factor nomenclature before outputs enter the score matrix.

Consistency metrics

Three distance-based benchmarks structure the analysis, all built on mean-square difference (MSD) over hypothesis-level scores across both contexts:

Metric	Definition	Interpretation
Agent Consistency (AC)	Pairwise MSD between models, averaged over 45 pairs	Model-to-model agreement
Literature Consistency (LC)	Pairwise MSD between studies, averaged over 465 pairs	Study-to-study agreement
Literature-Agent Consistency (LAC)	MSD between each model's score and the leave-one-out aggregate, averaged over studies and models	Combined variability

Permutation tests provide null references: full shuffle (MSD = 0.67) and hypothesis shuffle (MSD = 0.5).

Results: structured agreement and disagreement

Across 31 studies and 10 models (1,860 individual scores), mean hypothesis scores were positive throughout but nonuniform. In the local oddball context, H2 (Feedforward Error Propagation) scored highest (mean = 0.51 ± 0.34), followed by H1 (Predictive Suppression, 0.46 ± 0.42), then H3 (Ubiquity, 0.31 ± 0.39). All three hypotheses showed significant downward shifts in the global oddball context: −0.049 for H1 (p < 0.05), −0.158 for H2 (p < 0.01), and −0.032 for H3 (p < 0.05), via paired model-study t-tests. The lowest scores overall were H3 in the global context. This pattern implies that the council detected the strongest evidence for feedforward error signaling under local oddballs, and that support for feedforward propagation is the most context-sensitive of the three claims—an empirically meaningful result given that scale-invariance is precisely what the LO/GO paradigm tests.

Inter-model agreement was strong: mean AC was 0.11, far below the full-shuffle (0.67) and hypothesis-shuffle (0.5) baselines. Mistral-nemo-12b was the most divergent council member. At the study level, canonical theoretical papers (Bastos et al., 2012; Friston et al., 2010; Spratling, 2008) formed low-distance central reference points (LC ≈ 0.07 ± 0.01 SEM), while Westerberg et al. (2025)—a multi-area laminar spiking study in mice and monkeys—emerged as a clear outlier (mean MSD = 0.464). Notably, the outlier's divergence was itself context-dependent: its distance to the rest of the corpus was significantly larger for global oddballs (mean = 0.55, p < 0.01) than local oddballs (mean = 0.15). The benchmark therefore reproduced, quantitatively, the qualitative claim of Westerberg et al. that global oddball findings depart most strongly from predictive-coding expectations.

Hypothesis-space geometry and temperature

Embedding studies in three dimensions defined by their H1–H3 scores revealed a compact cluster near the (+1, +1, +1) octant for local oddballs, with greater dispersion—and multiple studies going negative on H2—for global oddballs. To quantify dispersion, the authors introduce hypothesis-space temperature, defined as the volume of the minimum-volume enclosing ellipsoid containing all study points divided by the number of points:

$T_H = c\,\frac{V_{\mathrm{env}}}{n}, \quad c = 1.0$

Temperature was significantly lower for local oddballs (0.00114 ± 0.00016 SEM, excluding the outlier) than global oddballs (0.00348 ± 0.00039, p < 0.01); including all 31 studies, the contrast was 0.00177 versus 0.00566. Because a new study falling inside the existing ellipsoid leaves volume approximately stable while increasing $n$ (lowering temperature), whereas an outlying study inflates $V$, this metric provides a running index of theoretical volatility as literatures evolve.

Vector analysis of each study's displacement from its local to global position showed a coherent mean direction pointing away from full agreement, and—importantly—the outlier Westerberg et al. had the largest vector magnitude but a direction aligned with the population-level shift. The authors interpret this as the outlier amplifying, rather than contradicting, an existing trend in the literature.

Discussion points

The paper argues that the ontology matters alongside the foundation models: without the constrained glossary and instruction layer, models default to autoregressive priors and narrative summaries that blur distinctions needed for hypothesis testing. Disagreement is treated as signal rather than noise—the AC/LC/LAC framework maps whether a result is stable across contexts or model-specific, with permutation shuffles supplying statistical reference distributions. An illustrative reasoning log excerpt shows Mistral-Nemo citing specific figures (Fig. 2b for local hierarchical progression at +0.8; Fig. 3 showing no temporal order globally at −0.6), demonstrating the intended evidence-grounded behavior.

Limitations and open questions

The authors are explicit about boundaries. The corpus of 31 studies, drawn largely from the comparison set of Westerberg et al., supports a focused cartography rather than exhaustive coverage of predictive coding. Results depend on the specific 36-factor ontology; an alternative glossary grounded in active inference or other PC variants could shift the geometric clustering, and the paper does not test sensitivity to ontological choice. Scores are nested within papers, models, factors, and contexts, but the reported analyses use nominal alpha thresholds and paired t-tests rather than hierarchical or mixed-effects models, leaving per-level uncertainty estimation to future work. Manual verification of reasoning logs covered only a random subset, so the rate of unsupported assertions across all 1,860 evaluations is not characterized. Whether the glossary itself can be proposed or refined automatically, and whether the framework captures literature dynamics over time, remain open questions the paper poses but does not answer.

Conclusion

This work demonstrates that a council of ten local LLMs, constrained by an expert-defined ontology and auditable instruction layer, can convert a fragmented neuroscience literature into a quantitative evidence space with measurable consensus, outliers, and geometric dispersion. Its substantive finding—that support for feedforward error propagation degrades most sharply from local to global oddball contexts, with elevated hypothesis-space temperature in the latter—constitutes a structured, replicable measurement of a live theoretical dispute. The framework's generalizability beyond this corpus and ontology remains to be established.