PEEK: Proxy Embeddings for Knowledge Estimation

• The paper shows that lightweight adaptation of sentence embeddings via a linear head can predict LLM knowledge with accuracies up to 91%.
• PEEK is defined as a framework that leverages proxy embeddings to quickly infer if LLMs recognize specific facts, enabling scalable, model-agnostic auditing.
• It offers practical applications in knowledge auditing, retrieval-augmented generation, and LLM risk analysis by reducing direct probing needs.

Proxy Embeddings to Estimate Knowledge (PEEK) refers to a class of frameworks and methodologies in which pre-trained embedding models—typically optimized on large text or graph corpora—are adapted or utilized as lightweight surrogates to estimate, audit, or control the knowledge held by LLMs, vision-LLMs (VLMs), or other neural agents. PEEK approaches are designed to provide computationally efficient, scalable alternatives to direct LLM probing, internal activation analysis, or corpus-based estimation, enabling fast assessment and (in some cases) modulation of knowledge boundaries, factual coverage, or behavior generalization.

1. Fundamental Motivation and Definition

The central motivation of PEEK lies in the recognition that direct interaction with LLMs to ascertain knowledge coverage—by means of repeated probing, querying, or activation analysis—is computationally demanding and not scalable, especially for large models or proprietary endpoints. Proxy embeddings offer a means to “stand in” for the LLM’s knowledge state. By adapting or analyzing these embeddings, one can infer whether a given fact (typically expressed as a sentence, structured triple, or entity) is “known” (i.e., can be correctly generated or recognized) by the target LLM. This approach leverages the dense semantic information already latent in high-quality embedding spaces and makes the process largely model-agnostic and black-box compatible.

2. Methodologies for Proxy Embedding Knowledge Estimation

PEEK methodology typically follows these stages:

1. Knowledge Label Acquisition:
   • Facts are sampled and labeled using LLM probing functions $P_M$, such as binary question-answering, confidence logit extraction, or internal hidden state classification. Various strategies are employed to determine ground-truth “known”/“unknown” labels for supervision, e.g.,:
     • Binarized yes/no generation (“Is fact X true?”)
     • Logit-based confidence
     • Activation-based prediction (Sharma et al., 8 Aug 2025)
2. Proxy Embedding Model Selection and Adaptation:
   • Candidate embedding models are selected:
     • Sentence Embeddings: e.g., MPNET, Linq, NVE2, GTE, MXBAI; these are typically obtained via contrastive or discriminative pre-training on large text corpora.
     • Graph Embeddings: e.g., ULTRA; trained over triples in knowledge graphs, often using GNN architectures (Sharma et al., 8 Aug 2025).

• The embeddings are adapted to LLM knowledge via supervised learning. Most commonly, a linear classification or regression head $W$ is fitted:

  $\hat{P}_\theta(f) = W^\top E(f) \approx P_M(f)$

  where $f$ is a fact, $E$ is the fixed embedding map, and $P_M(f)$ is the LLM-labeled knowledge state.

1. Proxy-Driven Knowledge Estimation:
   • Once trained, the adapted proxy embedding + head can be used to rapidly predict, for unseen facts, whether the LLM is likely to “know” the fact or not.
   • No further LLM querying or activation reading is required, enabling scalable knowledge auditing.
2. Evaluation and Comparison:
   • Accuracy, ROC-AUC, and MAE (for continuous labels) are computed on held-out fact/test sets to assess the method’s effectiveness in tracking LLM knowledge across multiple domains, models, and embedding types.

3. Empirical Findings and Model Selection

Extensive experiments demonstrate:

• Sentence embeddings (particularly Linq, NVE2, and GTE) routinely yield high accuracy and AUC (up to 91% and 88%, respectively) in predicting whether an LLM (including GPT-4o-mini and ChatGPT) will recognize/generate a fact; graph-based embeddings underperform relative to these baselines (Sharma et al., 8 Aug 2025).
• The adaptation from a pre-trained sentence embedding to an LLM’s knowledge state is typically linearly decodable, with low-rank adaptations (e.g., LoRA) offering negligible gain.
• Proxy-based knowledge estimation works in “black-box” settings where internal LLM activations are inaccessible. In contrast, methods relying on hidden state probes (e.g., Llama-Hid) require model internals and are non-transferable.
• Adding negative samples during proxy model training improves AUC but can create class imbalance. Sentence embeddings display robust performance even under varied dataset sampling percentages.

A summary of results for common model and proxy configurations:

LLM Target	Proxy Embedding	ACC / AUC	Superior to Graph Emb.?
GPT-4o-mini	Linq	91% / 88%	Yes
Llama-3.1-8B	Linq/NVE2/GTE	~68-85% / 0.85-0.90	Yes
FactScore LLMs	Linq, NVE2, GTE	max 86% / 90%	Yes

Graph embeddings (e.g., ULTRA) are less suitable proxies for LLM knowledge, indicating a strong inductive bias in LLMs for sentence/textual structure over graph-triple formats (Sharma et al., 8 Aug 2025). This suggests that LLMs' factual representations are more “textual” than “graph-structural,” reflecting training data and architecture.

4. Relationships to Broader Proxy-Based Knowledge Systems

The PEEK family of frameworks relates to earlier work on using embeddings as knowledge surrogates:

• Semantic Projection: Generalized geometric queries on word embeddings (e.g., projecting object vectors onto feature axes) can reliably recover human-like property judgments across domains (Grand et al., 2018). This demonstrates the potential of embeddings as flexible, high-dimensional proxies for detailed knowledge, with only minimal external labeling.
• Embedding-Informed ARAG: In adaptive retrieval-augmented generation, the necessity for retrieval can be estimated directly from the model's token/entity embeddings, demonstrating that these vectors encode the LLM’s knowledge boundaries without further model inference or access to pre-training data (Huang et al., 4 Apr 2024).
• Partial Knowledge Representation: Classical embeddings encode complete worlds, lacking the ability to represent “unknowns.” Ensemble and aggregate embedding schemes allow for partial or uncertain knowledge assignment within proxy embedding models (Guha, 2017). This capacity is important for PEEK applications that wish to estimate not only “known” or “unknown” but also “uncertain” or “partially known” knowledge scenarios.
• Proxy Models for Contextual Knowledge Steering: In the CSKS framework (Wang et al., 27 Aug 2025), proxy models are used to usher in a controlled, continuous steering of LLM sensitivity to context events, again leveraging the proxy distribution’s gradients for model-agnostic intervention.

5. Practical Applications and Implications

PEEK frameworks support several concrete applications:

• Scalable Knowledge Auditing: Large candidate fact sets (e.g., billion-scale Wikipedia-derived corpora) can be rapidly screened for LLM knowledge coverage, identifying gaps and weaknesses pre-deployment.
• Pre-deployment Risk Analysis: Efficiently flag hallucination or coverage risks in black-box or proprietary LLMs by proxy estimation, circumventing costly direct probing.
• Retrieval-augmented generation (RAG): Serve as a filter for which facts/entities truly require external retrieval, reducing unnecessary computation and improving efficiency (Huang et al., 4 Apr 2024).
• LLM Model Editing and Updating: By rapidly locating knowledge gaps or boundary issues, targeted interventions or retraining can be performed.
• Zero-shot Generalization in Robotics: In vision-language and robotic manipulation policies, PEEK-derived proxy embeddings (e.g., keypoints, masks, or path overlays) predict “what” and “where” cues to offload reasoning from the policy, boosting robustness and transfer across tasks (Zhang et al., 22 Sep 2025).

6. Limitations, Challenges, and Future Directions

Several open issues and limitations are highlighted in current PEEK approaches:

• Data Overlap and Generality: The degree of pre-training data overlap between embedding models and LLMs may artificially improve correlation; extending to specialized or adversarial domains requires further research.
• Proxy Model/Embedding Choice: No single embedding model is universally optimal; matching LLM architecture, domain, and probing task remains a key variable.
• Scope of Applicability: Current methods are most effective for entity-centric, atomic factual knowledge. Generalizing to document-centric QA, composite knowledge, or ambiguous concepts is non-trivial (Huang et al., 4 Apr 2024).
• Proxy as Imperfect Surrogate: Embedding proxies encode an imperfect, specifically trained snapshot of LLM knowledge. Unexpected failure modes may arise for novel or ambiguous fact structures.
• Partial Knowledge Tracking: Most current PEEK instantiations predict binary knowledge, though ensemble/aggregate approaches open a path to representing uncertainty (Guha, 2017).

A plausible implication is that future work could explore hybrid methods, leveraging aggregate embedding clouds for uncertainty estimation, or proxy combinations for more granular knowledge audits and context sensitivities.

7. Comparative Table: Key PEEK Papers and Dimensions

Paper/Framework	Target Domain	Proxy Model Type	Main Application	Notes
(Sharma et al., 8 Aug 2025)	LLM facts/QA	Sent. Emb + Linear Head	Factual knowledge auditing	ACC/AUC to 91/88%; black-box compatible
(Huang et al., 4 Apr 2024)	ARAG/entity QA	Token embeddings + classifier	Retrieval necessity estimation	No pretrain data; robust to fine-tuning
(Zhang et al., 22 Sep 2025)	Robot policies	VLMs; mask/path embeddings	Zero-shot and sim2real transfer	41.4× sim2real gain; policy-agnostic
(Grand et al., 2018)	Semantic mem./NLP	Distributional word embeddings	Human-like property projection	Context-dependent features; no supervision
(Guha, 2017)	KR methodology	Ensemble/Aggregate embeddings	Partial/uncertain knowledge	Theoretical; links to partiality constraints
(Wang et al., 27 Aug 2025)	LLM context ctrl	Small proxy LMs (contrast)	Continuous contextual sensitivity	Output distribution steering

• (Sharma et al., 8 Aug 2025) Efficient Knowledge Probing of LLMs by Adapting Pre-trained Embeddings
• (Huang et al., 4 Apr 2024) Embedding-Informed Adaptive Retrieval-Augmented Generation of LLMs
• (Zhang et al., 22 Sep 2025) PEEK: Guiding and Minimal Image Representations for Zero-Shot Generalization of Robot Manipulation Policies
• (Grand et al., 2018) Semantic projection: recovering human knowledge of multiple, distinct object features from word embeddings
• (Guha, 2017) Partial Knowledge In Embeddings
• (Wang et al., 27 Aug 2025) Continuously Steering LLMs Sensitivity to Contextual Knowledge with Proxy Models