Perception-Knowledge Chains (PKC) Overview

• Perception-Knowledge Chains (PKC) are a design doctrine that link raw multimodal inputs with explicit intermediate states to enable structured and interpretable reasoning.
• PKC frameworks employ intermediate representations like OCR outputs, scene graphs, and typed knowledge graphs to ground perceptual evidence in actionable knowledge.
• PKC enhances performance in fields such as document understanding, ophthalmic surgery, and visual mathematics by using chain-sensitive training and evaluation techniques.

Searching arXiv for papers on “Perception-Knowledge Chains” and closely related formulations. Perception‑Knowledge Chains (PKC) denote an architectural and analytical pattern in which perceptual evidence is transformed into explicit intermediate knowledge structures and then into higher-level understanding, reasoning, or decisions. Across recent multimodal and knowledge-grounded systems, PKC refers less to a single canonical model than to a family of chain-structured formulations that make the linkage between “what is perceived” and “what is concluded” inspectable, trainable, and evaluable. In contemporary work, PKC appears in at least three closely related senses: as perception–cognition consistency in multimodal document understanding (Shao et al., 2024), as a perception→comprehension→reasoning hierarchy in domain benchmarks such as ophthalmic surgery (Wang et al., 19 Sep 2025), and as an explicit typed multi-hop path structure over visual–knowledge graphs for search agents (Jiao et al., 7 Jul 2026). A plausible implication is that PKC has evolved into a general design principle for systems that seek grounded reasoning, reliable search, and interpretable multimodal inference.

1. Conceptual scope and defining characteristics

PKC is unified by the requirement that downstream outputs remain linked to upstream percepts through explicit or at least testable intermediate states. In the document-understanding formulation, the relevant chain connects low-level perception, such as OCR over a document region, to higher-level question answering; the central question is whether cognition is consistent with perception (Shao et al., 2024). In visual mathematical reasoning, a closely related three-stage flow is formulated as perception$\Rightarrow$internalization$\Rightarrow$reasoning, where structured primitives extracted from diagrams are converted into a reasoning-ready representation before answer generation (Chen et al., 5 Jan 2026). In ophthalmic surgery, the corresponding hierarchy is Perception → Comprehension → Reasoning, where fine-grained attributes are aggregated into scene graphs and semantic memory graphs that support clinical reasoning (Wang et al., 19 Sep 2025).

These variants share several recurrent properties. First, PKC presupposes that raw multimodal input is insufficiently informative unless transformed into a more structured state. Second, the intermediate representation is not merely auxiliary; it is the principal mechanism by which grounding and explainability are imposed. Third, the quality of the chain is not reducible to end-task accuracy alone. Recent work instead evaluates internal coherence, such as whether a visual question answering response is supported by the model’s own OCR reading (Shao et al., 2024), or whether reasoning is faithful to internalized geometric primitives (Chen et al., 5 Jan 2026).

A further extension appears in multimodal search. SearchEyes defines PKC as constrained multi-hop paths over the visual–knowledge intersection of Wikidata5M, with hops typed as Perception or Knowledge and retained as supervision, environment structure, and reward anchors simultaneously (Jiao et al., 7 Jul 2026). This suggests that PKC can also be understood as an externalized chain in graph space rather than only an internal cognitive decomposition.

2. Formalizations across multimodal reasoning systems

The most explicit formalization of perception–cognition linkage in the provided literature is the document-understanding setting. There, an MLLM $f(\cdot)$ receives an image $x_I$ and two prompt types: a cognitive query $x_C$ and a perceptual query $x_P$, producing

$y_C = f(x_C, x_I), \quad y_P = f(x_P, x_I).$

The model’s internal knowledge is decomposed into global knowledge $\mathcal{K}$, cognitive knowledge $\mathcal{K}_C \subseteq \mathcal{K}$, and perceptual knowledge $\mathcal{K}_P \subseteq \mathcal{K}$. A Cognition and Perception conflict is defined when $\Rightarrow$0 and $\Rightarrow$1 disagree sufficiently that $\Rightarrow$2 and $\Rightarrow$3 are inconsistent, regardless of whether either matches ground truth (Shao et al., 2024). The resulting metric,

$\Rightarrow$4

uses

$\Rightarrow$5

so that the cognitive answer must be literally contained in the perceived text. This is a particularly strict PKC criterion because it operationalizes grounding as string inclusion rather than semantic compatibility (Shao et al., 2024).

CogFlow introduces a different but related chain. Perception outputs structured visual primitives such as points, lines, and circles in normalized parametric space; internalization converts these primitives into geometrically meaningful facts; reasoning then derives the final answer under a gated reinforcement-learning regime (Chen et al., 5 Jan 2026). The combined perception reward is

$\Rightarrow$6

where $\Rightarrow$7 measures parametric fidelity and $\Rightarrow$8 measures semantic consistency through rendered-image similarity (Chen et al., 5 Jan 2026). The total reward used in training combines perception, internalization, and answer-level components: $\Rightarrow$9 This formulation makes the middle segment of the chain—knowledge internalization—a first-class optimization target (Chen et al., 5 Jan 2026).

SearchEyes externalizes PKC into graph topology. A PKC path is

$f(\cdot)$0

with retained gold entity sequence $f(\cdot)$1, predicate sequence, and semantic domain sequence (Jiao et al., 7 Jul 2026). Hops are typed as Perception or Knowledge, the first hop is always P, no consecutive P-hops are allowed, and each path must include at least two P-hops (Jiao et al., 7 Jul 2026). In this setting, PKC is not merely a metaphor for cognition; it is the literal substrate from which questions, environment dynamics, and RL credit assignment are constructed.

3. Intermediate knowledge representations

A defining feature of PKC is the use of intermediate representational layers. These vary by domain, but they share the role of stabilizing the transition from raw input to reasoning.

In document understanding, the implicit intermediate state is the text the model reads from a localized document region. The system does not require a formally exposed OCR JSON or symbolic latent, but it queries the same model in a perceptual mode and demands that the cognitive output be supported by that reading (Shao et al., 2024). The representation is therefore prompt-induced rather than architecturally separated.

In EyePCR, the intermediate layer is explicitly graph-structured. Perception consists of 1,048 fine-grained attributes across seven semantic views and three levels; these attributes and subtitles are transformed into Structured Scene Graphs built from entity–action–target triplets such as $f(\cdot)$2, then aggregated over time into a Semantic Memory Graph, from which multi-hop reasoning paths are extracted (Wang et al., 19 Sep 2025). Here PKC is closest to a conventional symbolic pipeline: perceptual attributes, knowledge graphs, then reasoning tasks.

In CogFlow, the intermediate state is textual but highly structured. <WATCHING> encodes normalized primitives, while the early portion of <THINKING> internalizes those primitives into canonical geometric facts before later chain-of-thought reasoning proceeds (Chen et al., 5 Jan 2026). The paper identifies five internalization failure types: omitting or misbinding primitives, introducing nonexistent facts, contradicting geometric constraints, invoking external theorems inappropriately, and inconsistent reference to established elements (Chen et al., 5 Jan 2026). This taxonomy is notable because it localizes chain failure specifically in the transition from perception to usable knowledge.

SearchEyes uses typed knowledge graphs as the intermediate state and the environment itself. Every PKC node corresponds to an entity with a document and, for visual entities, an image. The world searched by the agent is therefore not external to the chain but induced by it (Jiao et al., 7 Jul 2026). A plausible implication is that SearchEyes treats PKC less as a latent reasoning scaffold and more as an executable world model.

Beyond multimodal reasoning, related formulations support the same pattern. In Knowledge-based Entity Prediction, raw perception is semantically lifted into a scene knowledge graph $f(\cdot)$3, and the missing-entity inference problem is formalized as

$f(\cdot)$4

where observed scene entities are extended by graph-based prediction (Wickramarachchi et al., 2022). This suggests a PKC in which perception is incomplete by design and knowledge completion supplies the missing bridge.

4. Evaluation criteria: consistency, grounding, and boundary awareness

PKC research replaces or supplements conventional end-task evaluation with chain-sensitive metrics. The most direct example is C&P consistency in document understanding. The reported pre-fine-tuning micro-average consistency values are 79.98% for Qwen‑VL‑Max, 68.60% for GPT‑4o, 19.41% for Qwen‑VL‑Chat‑7B, 16.87% for InternVL2‑8B, and 12.09% for InternVL2‑2B (Shao et al., 2024). These results show that even strong MLLMs can answer questions in ways not literally grounded in their own OCR outputs.

EyePCR evaluates separate stages rather than a single chain score. Its benchmark includes 213,883 VQAs: 102,827 for Perception, 24,693 for Comprehension, and 86,363 for Reasoning, distributed over 82,893 annotated video segments from 1,544 curated ophthalmic surgery videos (Wang et al., 19 Sep 2025). The chain is therefore assessed stagewise through perception MCQs and OEQs, comprehension OEQs grounded in knowledge graphs, and four dimensions of reasoning: Procedural Flow Understanding, Surgical Intent Inference, Intraoperative Decision Simulation, and Anomaly/Risk Awareness Assessment (Wang et al., 19 Sep 2025). This does not directly measure consistency between adjacent stages, but it does expose performance by level of abstraction.

CogFlow evaluates the quality of the chain more explicitly through stage-specific rewards and accuracy on visual mathematical reasoning benchmarks. The paper reports strong performance on FlowVerse, MathVerse, and MathVista, and attributes gains to the combination of SynVRs, IntlzR, and Visual-Gated Policy Optimization (Chen et al., 5 Jan 2026). The important point for PKC is methodological: chain quality is judged not only by final answer accuracy but also by primitive recovery, internalization fidelity, and gated reasoning quality.

SearchEyes measures PKC both as data structure and as RL substrate. Its PKC ablation shows average accuracy drops from 61.8 to 57.6 without P–K alternation, to 59.3 without treewidth≤2 constraint edges, to 54.1 without anti-shortcut filtering, to 60.0 without domain diversity balancing, and to 58.5 without cross-modal grounding filters (Jiao et al., 7 Jul 2026). The same work reports that SearchEyes‑27B improves over the strongest open-source baseline by 6.2 points on average across six multimodal knowledge-intensive benchmarks (Jiao et al., 7 Jul 2026). These findings indicate that PKC design choices materially affect downstream search reasoning.

A distinct but relevant evaluation axis is knowledge-boundary perception. In LVLMs, alignment between confidence and correctness is measured as

$f(\cdot)$5

with Overconfidence $f(\cdot)$6, Conservativeness $f(\cdot)$7, and Uncertain-Rate $f(\cdot)$8 (Ding et al., 26 Aug 2025). Although this paper does not use the PKC label, it examines whether a model can perceive the boundary of what it knows. That function is closely related to PKC reliability because any chain that cannot recognize its own uncertainty risks propagating unsupported inferences.

5. Training and optimization strategies

Training methods for PKC-like systems typically aim either to tighten the link between adjacent stages or to provide supervision at multiple chain levels.

The most direct chain-tightening recipe is Multimodal Knowledge Consistency Fine-Tuning. In document understanding, MKCFT uses three stages: Perception Consistency, Cognition Consistency, and a C&P Connector that explicitly links questions, answers, and boxes (Shao et al., 2024). The data sizes are approximately 2,189k samples for Stage 1, 176k for Stage 2, and 146k for Stage 3 (Shao et al., 2024). The method uses standard supervised language modeling loss rather than an explicit direct consistency penalty, and it freezes visual encoders while fine-tuning only the LLM (Shao et al., 2024). This is significant because it shows that PKC alignment can be induced through correlated multi-task supervision rather than architectural modification.

CogFlow instead adopts reinforcement learning with stage-specific reward models. Knowledge Internalization Reward is trained on 10k positive trajectories and 50k negatives using Softmax-DPO: $f(\cdot)$9 where negatives are synthesized by injecting the five internalization failure types (Chen et al., 5 Jan 2026). Reasoning is then optimized with Visual-Gated Policy Optimization, in which a visual gate selects acceptable perception trajectories before reasoning begins (Chen et al., 5 Jan 2026). This is an unusually explicit PKC optimization regime because it penalizes middle-stage drift even when answer-level outputs might appear plausible.

SearchEyes uses PKC not only to generate data but also to supervise RL. Its Hop-Anchored Policy Optimization computes step-level advantages from gold entity retrieval anchors derived directly from the retained PKC sequence $x_I$0 (Jiao et al., 7 Jul 2026). The final per-token advantage blends episode-level and hop-level signals: $x_I$1 This eliminates the need for a separately trained process reward model (Jiao et al., 7 Jul 2026). A plausible implication is that PKC can serve simultaneously as curriculum, supervision trace, and credit-assignment scaffold.

Progressive Self-Distillation for Ground-to-Aerial Perception Knowledge Transfer provides a different but structurally related case. It does not use the PKC term, but it constructs a chain of viewpoints $x_I$2 and models $x_I$3, where each model transfers perception knowledge to the next through nearest-neighbor pseudo-labeling, MixView, and progressive distillation (Hu et al., 2022). The paper reports 22.2% and 16.9% accuracy improvement for synthesized and real-world datasets respectively (Hu et al., 2022). This suggests a broader reading of PKC as staged transfer of perception knowledge across structured domain shifts.

6. Empirical domains and representative instantiations

The surveyed literature reveals that PKC is not domain-specific. Instead, it is repeatedly rediscovered where a task requires strong coupling between perception and abstract reasoning.

Representative instantiations

Domain	PKC form	Key mechanism
Document understanding	Perception–cognition consistency	OCR-grounded QA and MKCFT (Shao et al., 2024)
Ophthalmic surgery	Perception → Comprehension → Reasoning	Attributes, scene graphs, semantic memory graphs (Wang et al., 19 Sep 2025)
Visual mathematics	Perception → Internalization → Reasoning	Structured primitives, IntlzR, VGPO (Chen et al., 5 Jan 2026)
Multimodal search	Typed graph PKC	P/K-hop chains, search world, HaPO (Jiao et al., 7 Jul 2026)
Autonomous systems	Perception → scene KG → missing-entity prediction	Knowledge completion over scene graphs (Wickramarachchi et al., 2022)
Drone perception transfer	Chained viewpoint adaptation	Progressive pseudo-labeling and distillation (Hu et al., 2022)

Document understanding emphasizes internal consistency. Open-source MLLMs show large gains after MKCFT: Qwen‑VL‑Chat rises from 19.41% to 54.24%, InternVL2‑2B from 12.09% to 49.94%, and InternVL2‑8B from 16.87% to 60.03% in micro-averaged C&P consistency (Shao et al., 2024). These improvements are accompanied by major OCR gains and, in many cases, improved cognitive-task performance.

In ophthalmic surgery, EyePCR provides what the paper describes as a three-stage cognitive hierarchy with more than 210k VQAs, 1,048 fine-grained attributes, a medical knowledge graph with 25,567 triplets, 16,034 entities, 3,507 relations, and 11,462 logical paths (Wang et al., 19 Sep 2025). EyePCR‑MLLM, a domain-adapted Qwen2.5‑VL‑7B, raises perception MCQ accuracy from 0.3866 to 0.7412 and outperforms open-source models on Comprehension and Reasoning while rivalling commercial systems such as GPT‑4.1 (Wang et al., 19 Sep 2025).

In visual mathematical reasoning, CogFlow contributes MathCog with 121,730 problems, including 100k SFT examples, 10k RL examples, and 10k positive plus 50k negative examples for internalization-reward training (Chen et al., 5 Jan 2026). Here PKC is unusually explicit because each stage is tagged in the data as <WATCHING>, <THINKING>, and <ANSWER>.

In multimodal search, SearchEyes constructs PKC over a typed knowledge graph comprising entities that simultaneously possess triples, textual articles, and images. SearchEyes‑27B reaches an average accuracy of 68.1 over six benchmarks, and SearchEyes‑9B matches or exceeds 30B-scale open-source agents on average (Jiao et al., 7 Jul 2026). The benchmark VisSearch Bench, which is itself constructed directly from PKC, remains difficult even for proprietary models (Jiao et al., 7 Jul 2026).

7. Limitations, misconceptions, and open problems

A common misconception is that a consistent chain guarantees correctness. Document understanding provides a direct counterexample: C&P consistency can be satisfied even when cognition and perception are both wrong but aligned (Shao et al., 2024). CogFlow likewise distinguishes faithful internalization from final answer correctness by maintaining a separate internalization reward (Chen et al., 5 Jan 2026). Thus, PKC consistency is necessary for groundedness but not sufficient for truth.

A second misconception is that PKC requires modular architectures with explicit symbolic interfaces. Several papers show otherwise. MKCFT introduces no new architectural heads or cross-attention modules and fine-tunes only the LLM while freezing the visual encoder (Shao et al., 2024). EyePCR‑MLLM is a monolithic Qwen2.5‑VL‑7B adapted on a dataset whose structure encodes the chain, even though the inference pipeline does not explicitly pass scene graphs into the model (Wang et al., 19 Sep 2025). This suggests that PKC can be instantiated either structurally in data and objectives or explicitly in model modules.

A third misconception is that PKC is only relevant to multimodal reasoning with images. SearchEyes generalizes PKC to graph-mediated deep search over visual–textual worlds (Jiao et al., 7 Jul 2026), while Knowledge-based Entity Prediction uses knowledge completion to infer unrecognized scene entities in autonomous driving and smart manufacturing (Wickramarachchi et al., 2022). Progressive self-distillation for viewpoint transfer shows a one-dimensional perception-knowledge chain over flight heights (Hu et al., 2022). The commonality is not the modality but the staged dependency of later inference on earlier grounded evidence.

Several open problems recur. OCR quality constrains both training and evaluation in document PKC because many samples cannot be grounded if the relevant text is absent from OCR output (Shao et al., 2024). EyePCR’s knowledge graph is used to generate data rather than as a retrieval-time reasoning module, leaving open the question of how explicit graph retrieval or graph neural reasoning would alter chain behavior (Wang et al., 19 Sep 2025). CogFlow is domain-specific to geometry and incurs significant RL and reward-modeling cost (Chen et al., 5 Jan 2026). SearchEyes relies on the visual–knowledge intersection of Wikidata5M and Wikipedia, which limits entity coverage and relation richness (Jiao et al., 7 Jul 2026). Knowledge-boundary perception remains imperfect in LVLMs, and verbalized confidence often leads to overconfidence (Ding et al., 26 Aug 2025).

More broadly, the literature suggests two unresolved tensions. One is between chain explicitness and efficiency: stronger PKC supervision often requires more stages, more sampled trajectories, or more synthetic data. The other is between interpretability and scalability: explicit graphs and structured annotations improve inspection, but monolithic models trained on PKC-shaped corpora can still achieve strong performance without exposing their internal states.

PKC therefore occupies an intermediate position between symbolic pipelines and end-to-end multimodal learning. It is best understood as a design doctrine: preserve, supervise, and evaluate the pathway from perception to knowledge to reasoning, rather than treating accurate final answers as sufficient evidence that such a pathway exists.