Knowledge to Sight (K2Sight)

• Knowledge to Sight is a design principle that converts structured, explicit knowledge into visual representations, using methods like semantic decomposition and rule-grounded fusion across tasks such as medical imaging and assistive wearables.
• It leverages explicit structures—including knowledge graphs, attribute-driven prompts, and hybrid neuro‐symbolic pipelines—to overcome shortcomings of standard vision models like hallucination and weak causal reasoning.
• Empirical studies demonstrate that K2Sight frameworks achieve significant efficiency and performance gains in areas like abnormality grounding, gaze-based skill assessment, and diagnostic reasoning.

Knowledge to Sight (K2Sight) denotes the conversion of structured knowledge into visually grounded perception, reasoning, or generation. In its broadest formulation, it is the idea that knowledge about visual concepts, their relations, operations, and reasoning can be formalized and operationalized so that machines “see” better: perceive more accurately, reason more soundly, and generate more faithfully (Wang et al., 2024). In current literature, the term appears both as a general research agenda and as the name of specific frameworks, including clinical abnormality grounding through knowledge decomposition, gaze-only skill assessment distilled from multimodal smart-glasses input, knowledge-intensive text-to-image generation with explicit constraint checking, and biomarker-centric diagnostic reasoning over fundus images (Li et al., 6 Aug 2025, Wu et al., 24 Nov 2025, Zhao et al., 24 Apr 2026, Huang et al., 6 Jul 2026).

1. Conceptual foundations

K2Sight is rooted in the claim that visual knowledge is a distinct form of knowledge representation rather than a mere byproduct of learned features. “Visual knowledge” is defined as stable, explicit mental representations about visual objects and the commonalities in the inherent rules across tasks. In this formulation, prototypes and scopes define concepts; relations include geometric, temporal, semantic, functional, and causal structure; operations transform and compose visual items; and reasoning uses these elements to derive conclusions. The same account distinguishes visual knowledge from subsymbolic perception features such as CNN features or masks, from generic symbolic knowledge graphs, and from broad multimodal knowledge, while arguing that explicit visual knowledge is interpretable, editable, and traceable in ways that implicit parameterized knowledge is not (Wang et al., 2024).

This distinction matters because several limitations of large vision and vision-LLMs are described in the surveyed literature as failures of explicit grounding. The review on visual knowledge argues that opacity, hallucination, weak systematic generalization, and weak causal inference persist when knowledge remains hidden in parameters. K2Sight therefore treats knowledge as a first-class computational object: not only something learned from data, but something represented, queried, constrained, or distilled so that perception can be audited and corrected (Wang et al., 2024).

A recurring implication across the literature is that K2Sight is less a single algorithm than a design principle. In medical grounding, it appears as semantic decomposition into visual attributes; in smart-glasses skill assessment, as distillation of multimodal knowledge into gaze; in clinical analytics, as externalization of expert knowledge into an explicit store; and in knowledge-intensive generation, as the translation of domain constraints into visual specifications and verifiable checklists. This suggests that the common denominator is not modality or task, but the explicit mediation between knowledge and visual inference (Li et al., 6 Aug 2025, Wu et al., 24 Nov 2025, Wagner et al., 2017, Zhao et al., 24 Apr 2026).

2. Representational and pipeline structures

The formal vocabulary associated with K2Sight is heterogeneous but structured. The visual-knowledge review endorses prototype-and-scope concept representations, scene graphs $G=(V,E)$ with triplet relations $$\langle \text{subject}, \text{predicate}, \text{object}\rangle$$, hierarchical ontologies, temporal relations based on Allen’s interval algebra, and affordance or causal schemas. It also describes hybrid neuro-symbolic settings in which explicit structures are operated on by neural modules, as well as training objectives that combine task loss with consistency or logic regularization, such as $$L_{\text{total}} = L_{\text{task}} + \lambda L_{\text{consistency}} + \mu L_{\text{logic}}$$ (Wang et al., 2024).

At the systems level, K2Sight usually appears as a staged pipeline. The review proposes a blueprint with knowledge acquisition, representation, alignment and reasoning, memory and retrieval, and downstream application modules. The same staged logic appears in more specialized systems. In Newvision, the assistive headgear is organized as sensing, perception, decision, and feedback: cameras and ultrasonic sensors acquire the environment, a BLIP-inspired vision-LLM produces captions or answers, modular task logic selects scene explanation or obstacle-related functions, and voice assistants deliver spoken guidance (Bobba et al., 2023). In KAVAGait, the loop is load new patients, perform automated knowledge-based matching, enable clinician-driven exploration, persist category updates in an explicit knowledge store, and immediately refresh matching and visual summaries; this was explicitly described as a closed-loop process in which knowledge informs visualization and visualization generates new knowledge (Wagner et al., 2017).

These pipeline structures differ in their locus of explicitness. Some systems store knowledge as graphs, ontologies, category ranges, or rule sets. Others encode it in prompts, constraint sets, or distilled latent representations. Yet the operational role is similar: knowledge changes what is attended to, what is considered valid, how evidence is aggregated, and how outputs are communicated.

3. Mechanisms for translating knowledge into visual inference

A central K2Sight mechanism is semantic decomposition. In the medical abnormality-grounding framework explicitly titled “Knowledge to Sight,” each abnormality definition $d(a)$ is converted into a visually grounded instruction $k(a)$ through a mapping $\phi: d(a) \to k(a)$, with the training taxonomy focusing on shape, intensity, density, and anatomical location. The resulting prompts are inserted into compact VLMs so that localization is framed as autoregressive sequence generation over coordinate tokens, optimized by cross-entropy. The stated purpose is to replace coarse report-level supervision with structured semantic supervision that bridges clinical language and spatial evidence (Li et al., 6 Aug 2025).

A second mechanism is distillation from rich multimodal supervision into low-power “sight-only” inference. SkillSight uses a teacher that jointly models egocentric video and gaze, and a gaze-only student that receives feature-level distillation and an auxiliary action-recognition head. Its student objective is

$$L_{\text{student}} = L_{\text{CE}} + \lambda_{\text{dis}} L_{\text{dis}} + \lambda_{\text{act}} L_{\text{act}},$$

where the distilled representation transfers video-plus-gaze knowledge into gaze-only inference. In the paper’s own formulation, this operationalizes K2Sight by moving knowledge learned from multimodal evidence into a pure sight/gaze signal (Wu et al., 24 Nov 2025).

A third mechanism is rule-grounded fusion. GlaKG encodes structural biomarkers, pathologies, diagnoses, risk levels, and 11 clinically validated rules in a heterogeneous fundus knowledge graph. Activated rules are accumulated into a reasoning-chain score and normalized into a KG-derived signal, which is then fused with image-based probability through

$$p_{\text{final}}^{(i)} = (1-\alpha)\,p_{\text{img}}^{(i)} + \alpha\,s_{\text{KG}}^{(i)}.$$

The stated purpose is to keep knowledge-based reasoning separate from label information while exposing an explicit reasoning chain from biomarker evidence to diagnosis and risk (Huang et al., 6 Jul 2026).

A fourth mechanism is explicit constraint auditing. In knowledge-intensive text-to-image generation, KVBench and KE-Check formalize correctness as atomic constraints $\mathcal{C}(x)=\{c_1,\dots,c_N\}$ and violations as $$\mathcal{V}(I,\mathcal{C})=\{c_i \mid I \not\models c_i\}$$. Evaluation uses item-wise binary judgments with final score

$$\langle \text{subject}, \text{predicate}, \text{object}\rangle$$0

KE-Check first elaborates a brief prompt into a structured, knowledge-rich specification and then performs checklist-guided refinement through violation identification and targeted edits. Here K2Sight denotes faithful translation of facts, structures, rules, and symbols into visual form (Zhao et al., 24 Apr 2026).

Other systems instantiate the same principle through external lexical or commonsense knowledge. K-LITE augments text supervision with WordNet hypernym paths, WordNet definitions, and Wiktionary glosses at both training and evaluation time, thereby making image-text alignment more semantic and disambiguated (Shen et al., 2022). Know2Look integrates commonsense triples, visual detections, and contextual text in a mixture LLM for retrieval, using commonsense knowledge to bridge abstract query intent and noisy document features (Chowdhury et al., 2019).

4. Domains and representative systems

The applications associated with K2Sight span medical imaging, assistive wearables, multimodal learning, visualization, retrieval, and generation.

Domain	Representative system	K2Sight mechanism
Assistive wearables	Newvision (Bobba et al., 2023)	Multimodal sensing, BLIP-inspired perception, spoken guidance
Smart glasses skill assessment	SkillSight (Wu et al., 24 Nov 2025)	Multimodal teacher, gaze-only student, feature distillation
Medical abnormality grounding	K2Sight (Li et al., 6 Aug 2025)	Knowledge decomposition into attribute-grounded prompts
Clinical visual analytics	KAVAGait (Wagner et al., 2017)	Explicit knowledge store, matching, knowledge-driven encodings
Glaucoma diagnosis	GlaKG (Huang et al., 6 Jul 2026)	Biomarker-centric knowledge graph and rule chains
Knowledge-intensive image generation	KVBench / KE-Check (Zhao et al., 24 Apr 2026)	Prompt elaboration, checklists, violation-guided refinement

In assistive technology, Newvision frames K2Sight as environmental knowledge converted into spoken guidance. Cameras and ultrasonic sensors acquire the scene; a unified vision-LLM generates descriptions, answers questions, and reasons about visual content; modular decision logic selects among scene explanation, object detection, traffic assistance, currency detection, and navigation-related responses; and voice assistants present actionable audio output. The paper also specifies failsafe behavior, over-the-air updates, and a long-term goal of miniaturization toward spectacle-like form factor (Bobba et al., 2023).

In human performance analysis, SkillSight shows that K2Sight need not preserve RGB input at inference. Its teacher explicitly models action–gaze interaction, attended object sequences, and gaze dynamics, while the student keeps the RGB camera off and uses only gaze. The paper motivates this with cognitive-science observations about early fixation, “quiet eye,” look-ahead fixations, efficient scanpaths, and domain-specific viewing patterns among experts (Wu et al., 24 Nov 2025).

In medicine, the named K2Sight framework focuses on abnormality grounding in chest radiography, whereas GlaKG focuses on explainable glaucoma diagnosis and risk assessment through a biomarker-centric graph. The first translates clinical definitions into visual-attribute prompts; the second translates clinically validated rules into reasoning chains. Both treat domain knowledge as an explicit scaffold for spatial or diagnostic inference rather than as latent background context (Li et al., 6 Aug 2025, Huang et al., 6 Jul 2026).

In adjacent work, KAVAGait realizes an analogous principle without using the term K2Sight explicitly. It externalizes clinicians’ implicit knowledge into an explicit knowledge store containing gait categories, value ranges, patient assignments, and distribution statistics, and then uses that knowledge to drive visual encodings such as Graphical Summary, Matching bars, Interactive Twin Box Plots, and Hatching Range Slider interactions (Wagner et al., 2017). K-LITE and Know2Look extend the same logic to transfer learning and retrieval, while VKnowU and VideoKnow+ turn it into a benchmarked question: whether multimodal LLMs can convert physical and social world knowledge into sight-grounded judgments over video (Shen et al., 2022, Chowdhury et al., 2019, Jiang et al., 25 Nov 2025).

5. Empirical evidence

The empirical record for K2Sight is distributed across tasks rather than concentrated in a single benchmark. In medical abnormality grounding, compact models trained with knowledge decomposition use only $$\langle \text{subject}, \text{predicate}, \text{object}\rangle$$1 of the data required by state-of-the-art medical VLMs and nevertheless achieve performance on par with or better than 7B+ medical VLMs, with up to $$\langle \text{subject}, \text{predicate}, \text{object}\rangle$$2 improvement in $$\langle \text{subject}, \text{predicate}, \text{object}\rangle$$3. On VinDr-CXR, K2Sight-Lite reports $$\langle \text{subject}, \text{predicate}, \text{object}\rangle$$4, $$\langle \text{subject}, \text{predicate}, \text{object}\rangle$$5, and $$\langle \text{subject}, \text{predicate}, \text{object}\rangle$$6, while K2Sight-Base reports $$\langle \text{subject}, \text{predicate}, \text{object}\rangle$$7 and $$\langle \text{subject}, \text{predicate}, \text{object}\rangle$$8; zero-shot PadChest-GR results remain competitive with larger medical baselines (Li et al., 6 Aug 2025).

In smart-glasses skill assessment, SkillSight reports that the multimodal teacher achieves $$\langle \text{subject}, \text{predicate}, \text{object}\rangle$$9 overall accuracy on Ego-Exo4D, while the gaze-only student reaches $$L_{\text{total}} = L_{\text{task}} + \lambda L_{\text{consistency}} + \mu L_{\text{logic}}$$0 and uses $$L_{\text{total}} = L_{\text{task}} + \lambda L_{\text{consistency}} + \mu L_{\text{logic}}$$1 mW. The paper states that this yields $$L_{\text{total}} = L_{\text{task}} + \lambda L_{\text{consistency}} + \mu L_{\text{logic}}$$2–$$L_{\text{total}} = L_{\text{task}} + \lambda L_{\text{consistency}} + \mu L_{\text{logic}}$$3 energy savings relative to video-heavy baselines, including a comparison to TimeSformer at $$L_{\text{total}} = L_{\text{task}} + \lambda L_{\text{consistency}} + \mu L_{\text{logic}}$$4 mW, and reports an average single-sample inference time of $$L_{\text{total}} = L_{\text{task}} + \lambda L_{\text{consistency}} + \mu L_{\text{logic}}$$5 ms on one GPU. On Multi-Sense Badminton, the teacher achieves $$L_{\text{total}} = L_{\text{task}} + \lambda L_{\text{consistency}} + \mu L_{\text{logic}}$$6 and the gaze-only student $$L_{\text{total}} = L_{\text{task}} + \lambda L_{\text{consistency}} + \mu L_{\text{logic}}$$7; on Expert–Novice Soccer, the gaze-only student reaches $$L_{\text{total}} = L_{\text{task}} + \lambda L_{\text{consistency}} + \mu L_{\text{logic}}$$8 (Wu et al., 24 Nov 2025).

In glaucoma diagnosis, GlaKG reaches binary classification $$L_{\text{total}} = L_{\text{task}} + \lambda L_{\text{consistency}} + \mu L_{\text{logic}}$$9, accuracy $d(a)$0, precision $d(a)$1, recall $d(a)$2, and $d(a)$3, together with $d(a)$4 accuracy and $d(a)$5 weighted $d(a)$6 for four-class risk stratification. The paper also reports that KG-derived and biomarker features contribute near-equally, at $d(a)$7 versus $d(a)$8, and openly characterizes these figures as an upper bound attainable with clean structured biomarkers because the underlying annotations are highly label-correlated (Huang et al., 6 Jul 2026).

In knowledge-intensive generation, KVBench evaluates 14 text-to-image models over 1,800 bilingual prompts and 5,158 checklist items. The benchmark reports a clear closed-source advantage, but also shows that KE-Check substantially improves Qwen-Image: in the Brief Caption track, Chinese overall score rises from $d(a)$9 to $k(a)$0, with especially large gains in Biology and Chemistry. The paper further reports evaluator alignment with human experts at $k(a)$1 accuracy and Cohen’s $k(a)$2 (Zhao et al., 24 Apr 2026).

In video understanding, VKnowU reports human performance of $k(a)$3 overall, with $k(a)$4 on world-centric tasks and $k(a)$5 on human-centric tasks, whereas leading models remain well below that level. VideoKnow+, trained with a See–Think–Answer paradigm and a visual knowledge reward, improves its Qwen2.5-VL-7B-Instruct base from $k(a)$6 to $k(a)$7 on VKnowU and also reports gains on MVBench, Video-MME, and MMVU. Persistent weaknesses are especially pronounced in Intuitive Physics and Spatial Awareness (Jiang et al., 25 Nov 2025).

Evidence for the broader K2Sight principle also appears in adjacent systems. KAVAGait reports a mean System Usability Scale score of $k(a)$8, which the paper associates with “good” usability, and describes expert confirmation that its explicit-knowledge encodings support quick decision-making (Wagner et al., 2017). K-LITE reports zero-shot ImageNet-1K gains such as $k(a)$9 in one combined-pretraining setting and strong average improvements across 20 downstream classification datasets, while Know2Look reports average Precision@10 of $\phi: d(a) \to k(a)$0 versus $\phi: d(a) \to k(a)$1 for “Extended Google” and $\phi: d(a) \to k(a)$2 for “Vanilla Google” on its tourism search benchmark (Shen et al., 2022, Chowdhury et al., 2019).

6. Limitations, misconceptions, and future directions

A common misconception is that K2Sight denotes a single model family. The literature instead presents it as a broad methodological orientation encompassing explicit knowledge graphs, attribute-grounded prompts, commonsense-enhanced retrieval, teacher–student distillation, checklist-based auditing, and knowledge-assisted visual analytics. Taken together, these works indicate that K2Sight is defined more by the role of knowledge in shaping visual computation than by any fixed architecture (Wang et al., 2024, Li et al., 6 Aug 2025, Wu et al., 24 Nov 2025, Zhao et al., 24 Apr 2026).

Another recurring issue is that explicit knowledge improves interpretability but does not remove engineering or data bottlenecks. The visual-knowledge survey identifies scalability of explicit knowledge, interpretability–performance trade-offs, grounding and faithfulness, hallucination control, continual learning, uncertainty quantification, and long-horizon temporal knowledge as open problems (Wang et al., 2024). The named medical K2Sight framework relies on curated definitions rather than comprehensive ontologies such as RadLex or SNOMED CT, and its human-aligned prompt selection step introduces subjectivity (Li et al., 6 Aug 2025). SkillSight depends on stable gaze tracking and notes domain shift, personalization, and subtle motor skills as persistent challenges (Wu et al., 24 Nov 2025). KVBench remains limited to senior high-school subject matter and reports multilingual fragility in non-Latin script rendering and academic terminology (Zhao et al., 24 Apr 2026).

Several systems also expose task-specific gaps. Newvision does not specify sensor placements, camera types, radar usage details, compute hardware, navigation algorithms, real-time latency, power budgets, field robustness metrics, or user studies, despite outlining a comprehensive headgear concept (Bobba et al., 2023). GlaKG explicitly warns that its strong predictive numbers should not be interpreted as leakage-free image-only performance because biomarker annotations are highly label-correlated (Huang et al., 6 Jul 2026). VKnowU shows that even strong MLLMs still lag far behind humans on world-centric visual knowledge, particularly intuitive physics and spatial reasoning (Jiang et al., 25 Nov 2025). KAVAGait presently uses only vertical ground-reaction force and a limited set of spatio-temporal parameters, with future work proposed for additional force components, symmetry indices, and machine learning integration (Wagner et al., 2017).

The future directions proposed across the literature are correspondingly diverse but convergent. The visual-knowledge review advocates a knowledge-first architecture with explicit visual knowledge bases, retrieval APIs, neuro-symbolic cores, program-of-thought orchestration, and unified memory (Wang et al., 2024). Medical abnormality grounding suggests expanding the attribute taxonomy, adding segmentation heads, integrating EHR or temporal imaging, and automating prompt verification against ontologies (Li et al., 6 Aug 2025). SkillSight points toward adaptive or personalized gaze models, online and continual learning, hybrid lightweight sensing, and explicit fixation analytics (Wu et al., 24 Nov 2025). GlaKG proposes automatic biomarker extraction from raw images, temporal knowledge graphs, and cross-dataset validation (Huang et al., 6 Jul 2026). In aggregate, these proposals describe a research program in which knowledge is increasingly externalized, aligned, audited, and fed back into perception—an increasingly explicit realization of the original claim that knowledge can be turned into sight (Wang et al., 2024).