Landmark-Guided Knowledge (LGK)

• LGK is a family of methods that explicitly uses structured landmarks (geometric, symbolic, linguistic) to inject prior knowledge into computational tasks.
• The approach augments data-driven processing with techniques like regularization, cross-attention, and hierarchical subgoal selection to enhance model performance.
• LGK bridges diverse applications—from vision segmentation to planning and navigation—by converting salient cues into actionable constraints.

Landmark-Guided Knowledge (LGK) denotes a family of methods in which landmarks serve as explicit carriers of prior structure for learning, inference, retrieval, generation, or control. In the cited literature, a landmark may be an input–output information granule, a facial keypoint, a noun-phrase landmark in an instruction, a room centroid paired with a room type, a cultural place name, or a planning fact or action that must occur in every valid plan. The common design choice is to augment purely data-driven processing with landmark-conditioned knowledge and then inject that knowledge through regularization, channel concatenation, cross-attention, hierarchical subgoal selection, or heuristic filtering (Dai et al., 31 Mar 2026, Yang et al., 30 Sep 2025, Pereira et al., 2019).

1. Conceptual scope

The literature applies the LGK label to multiple, non-identical constructions. In "Informed Machine Learning with Knowledge Landmarks," landmarks are a finite collection of input–output information granules,

$\mathsf{K}=\{(A_i,B_i)\}_{i=1}^{c},$

where $A_i$ is a fuzzy set in the input space and $B_i$ is the corresponding fuzzy granule in the output space; each pair summarizes that outputs for inputs in region $A_i$ lie approximately in the fuzzy range $B_i$ (Dai et al., 31 Mar 2026). In planning-based goal recognition, a landmark is a fact or action that must occur on every valid plan achieving a goal, often with a partial order $l_i\prec l_j$ (Pereira et al., 2019). In vision systems, landmarks are frequently geometric keypoints or their heatmaps; in embodied navigation they are instruction entities, room-level spatial anchors, or retrieved commonsense correlates (Jackson et al., 2016, Yang et al., 30 Sep 2025, Staroverov et al., 2021).

Setting	Landmark form	Operational role
KD-ML	$(A_i,B_i)$ fuzzy granules	granular regularizer
VLN / ObjectNav	noun phrases, room centers, co-occurrences	guidance for retrieval or subgoal choice
Vision / inpainting	68 facial keypoints or heatmaps	pose or structure prior
Planning	unavoidable facts or actions	goal-recognition heuristic and filtering

A common misconception is that LGK always refers to geographic or architectural landmarks. The cited work shows a broader usage: landmarks can be geometric, symbolic, linguistic, granular, or topological, provided that they encode compact, semantically meaningful constraints (Yang et al., 2019, Lin et al., 2024).

2. Landmark representations and encodings

LGK systems differ most sharply in how they encode landmarks. In the facial part segmentation cascade of Jackson et al., each of 68 detected keypoints is converted into a single-channel heatmap defined as a 2D Gaussian centered at the landmark location; these 68 heatmaps are stacked with the RGB image so that the segmentation FCN receives 71 input channels (Jackson et al., 2016). In LaFIn, a masked face is first mapped to 68 predicted landmarks, those points are rasterized into a single-channel structural map, and the map is concatenated with the masked RGB image to form a 4-channel input volume for the inpainting generator (Yang et al., 2019).

Language-grounded LGK adopts embedding-based encodings. The VLN method explicitly called LGK constructs a knowledge base of $630{,}000$ canonical language descriptions, stores a parallel noun-only list of $630{,}000$ entries, and precomputes CLIP-text embeddings for those noun entries; each panoramic view is split into $5$ sub-regions, which are encoded by CLIP’s image encoder for cosine-similarity retrieval (Yang et al., 30 Sep 2025). MARKY-mT5 instead detects up to three landmark boxes per panorama, converts each detected box back into a perspective patch, embeds it with MURAL-large, and interleaves the resulting landmark embeddings with action tokens before generation (Wang et al., 2021).

Other representations are more symbolic or semantically typed. HLPO represents landmarks as

$A_i$0

where $A_i$1 is the room-center coordinate and $A_i$2 its room type; only room centroids and room categories are stored, not obstacle maps or object placements (Staroverov et al., 2021). LLandMark’s Landmark Knowledge Agent uses a curated Vietnamese Cultural Landmarks knowledge base, replacing a landmark name with a textual description of its distinctive visual attributes, such as “twin square bell towers, dark gray stone, Gothic architecture, neo-Gothic façade” for St. Joseph’s Cathedral (Phung et al., 3 Mar 2026).

3. Granular, symbolic, and optimization-based formulations

The most explicit optimization-centric LGK formulation appears in KD-ML. A numeric model $A_i$3 is trained with the augmented loss

$A_i$4

The data term is computed on a local training set, while the knowledge term is a granular regularizer evaluated on points sampled over the full domain. For each sampled point and each landmark, the model computes a fuzzy consistency score $A_i$5 and penalizes the squared deviation $A_i$6 weighted by the input-granule activation $A_i$7. Because both $A_i$8 and $A_i$9 are smooth membership functions, the knowledge loss is differentiable and back-propagates gradients that push the model to remain inside the fuzzy output envelope whenever the input falls under the corresponding input granule (Dai et al., 31 Mar 2026).

Planning-based LGK is structurally different but similarly explicit. After extracting ordered fact landmarks from a relaxed planning graph, the system monitors an observation sequence and computes progress toward each candidate goal using the goal-completion heuristic

$B_i$0

where $B_i$1 denotes the landmarks relevant to sub-goal $B_i$2 and $B_i$3 the achieved subset. A second heuristic, $B_i$4, weights landmarks by their inverse frequency across candidate goals, with

$B_i$5

The same landmark structure is also used for filtering via strictly activating, unstable activating, and strictly terminal facts before invoking more expensive recognition machinery (Pereira et al., 2019).

These formulations show that LGK is not restricted to attention-based neural architectures. It also includes explicit regularization of differentiable models and symbolic reasoning over necessary conditions.

4. Navigation and embodied intelligence

In vision-and-language navigation, LGK has been formulated as an external-knowledge pipeline. The 2025 LGK method first retrieves top-$B_i$6 knowledge sentences by CLIP matching between subviews and the $B_i$7K-entry knowledge base, then applies Knowledge-Guided by Landmark (KGL), in which retrieved knowledge entries cross-attend to noun-phrase landmarks extracted from the instruction, and finally performs Knowledge-Guided Dynamic Augmentation (KGDA) to fuse instruction, knowledge, vision, and historical information. Local and global branches are combined through a learned scalar gate,

$B_i$8

On R2R Test-Unseen it reports Navigation Error $B_i$9, Success Rate $A_i$0, Oracle SR $A_i$1, and SPL $A_i$2; on REVERIE Test-Unseen it reports OSR $A_i$3, SR $A_i$4, SPL $A_i$5, RGS $A_i$6, and RGSPL $A_i$7 (Yang et al., 30 Sep 2025).

CONSOLE frames VLN as open-world sequential landmark discovery. It uses ChatGPT to extract ordered landmarks and generate landmark co-occurrence commonsense, CLIP to score candidate views, a learnable co-occurrence scoring module to correct noisy priors, and an observation enhancement strategy that fuses corrected landmark features with visual features before action prediction. Relative to a DUET baseline on R2R Test-Unseen, the reported gain is SR $A_i$8 points $A_i$9 and SPL $B_i$0 points $B_i$1; ablations show that removing learnable scoring or observation enhancement reduces SPL by $B_i$2–$B_i$3 points (Lin et al., 2024).

HLPO brings LGK into ObjectNav through hierarchical reinforcement learning. The global policy selects room landmarks from $B_i$4 using current GPS, goal category, and precomputed $B_i$5, while three low-level PPO skills—PointNav, Explore, and GoalReacher—execute navigation, room-wise exploration, and final approach. On 100 held-out Matterport3D test episodes, the reported performance is Success $B_i$6 and SPL $B_i$7 for HLPO, compared with Success $B_i$8 and SPL $B_i$9 for a two-skill RL baseline without landmarks, and Success $l_i\prec l_j$0 and SPL $l_i\prec l_j$1 for HLPO(Map) with a perfect obstacle map (Staroverov et al., 2021).

MARKY-mT5 addresses the inverse problem: generating grounded navigation instructions. It bootstraps approximately $l_i\prec l_j$2 grounded landmark annotations in English, Hindi, and Telugu, then conditions instruction generation on detected landmark patches rather than full panoramic features. On R2R Val-Unseen, human wayfinders achieve SR $l_i\prec l_j$3, NE $l_i\prec l_j$4, and SPL $l_i\prec l_j$5 when following MARKY-mT5 with predicted landmarks, versus SR $l_i\prec l_j$6, NE $l_i\prec l_j$7, and SPL $l_i\prec l_j$8 for human-written instructions; on RxR Val-Unseen, predicted landmarks give SR approximately $l_i\prec l_j$9 and silver landmarks approximately $(A_i,B_i)$0 (Wang et al., 2021).

5. Perception, generation, and retrieval

LGK is also used as a structural prior in dense prediction. Jackson et al. implement a two-stage cascade in which a VGG-16 FCN-8s first predicts 68 landmark heatmaps and a second FCN-8s performs part segmentation conditioned on those heatmaps. The guided network outputs seven semantic-part score maps: background, skin, eyebrows, eyes, nose, upper lip/inner mouth, and lower lip. Mean IoU improves from approximately $(A_i,B_i)$1 for the unguided FCN-8 to approximately $(A_i,B_i)$2 for the full cascade guided by detected landmarks, with an upper bound of approximately $(A_i,B_i)$3 when guided by ground-truth landmarks; the paper summarizes this as an approximately $(A_i,B_i)$4 percentage-point absolute gain over the unguided baseline (Jackson et al., 2016).

LaFIn uses landmarks to separate structure from appearance in face inpainting. A landmark-prediction subnet first regresses 68 facial landmarks from the masked image using

$(A_i,B_i)$5

and an inpainting subnet then conditions on the rasterized landmark map with pixel, perceptual, style, total variation, and adversarial losses. On CelebA-HQ with a center mask, the reported quantitative highlights are PSNR $(A_i,B_i)$6, SSIM $(A_i,B_i)$7, and FID $(A_i,B_i)$8, compared with $(A_i,B_i)$9 for EdgeConnect and $630{,}000$0 for ContextAttn/PIC as listed in the summary (Yang et al., 2019).

LLandMark extends landmark guidance to multimodal video retrieval. Its Landmark Knowledge Agent detects landmark names, reformulates them into descriptive visual prompts, and passes the enriched semantic query to a CLIP-based retriever; additional modules cover OCR/ASR retrieval, object filtering, an LLM-assisted image-to-image branch, and score fusion. In the HCMAIC 2025 qualifier round, the reported overall score is $630{,}000$1, ranking top $630{,}000$2 teams, with round scores of $630{,}000$3, $630{,}000$4, and $630{,}000$5 (Phung et al., 3 Mar 2026).

Taken together, these systems show that landmark guidance can serve as a pose prior, a geometric scaffold, or a semantic bridge from entity names to visual appearance.

6. Empirical profile, misconceptions, and open directions

Several recurring empirical patterns appear across the literature. In KD-ML, the optimal trade-off parameter $630{,}000$6 decreases as the local-data noise level $630{,}000$7 grows, and increases as landmark granules become broader under the ratio $630{,}000$8; on two physics-governed benchmarks, the KD model reduces total error $630{,}000$9 by $630{,}000$0 in the environmental pollutant dispersion setting and by up to $630{,}000$1 in the piston cycle-time setting, with consistent gains across multiple local windows (Dai et al., 31 Mar 2026). In guided segmentation, training-time corruption of landmark locations is critical because a model trained only on perfect heatmaps degrades when fed real detections at test time (Jackson et al., 2016). In grounded instruction generation, silver landmarks outperform predicted landmarks and are treated as an upper bound, indicating that improved detection could further close the gap to human instructions (Wang et al., 2021).

The same dependence on landmark quality appears in other domains. LaFIn explicitly notes that if the landmark predictor is very poor under extreme occlusions, structure may still degrade (Yang et al., 2019). CONSOLE notes that ChatGPT-based prior generation is time-consuming and expensive, and that priors may be noisy in highly specialized scenes (Lin et al., 2024). In planning, exact extraction of all landmarks and their orderings is PSPACE-complete, and very low observability can leave too few landmarks visible for strong recognition performance (Pereira et al., 2019).

A second misconception is that LGK is inherently tied to a single architectural template. The evidence instead spans additive regularization, symbolic heuristics, heatmap conditioning, retrieval augmentation, hierarchical RL, and multimodal generation. This suggests that LGK is better understood as a design pattern than as a fixed model class: isolate semantically salient or unavoidable structure, represent it explicitly, and use it to constrain or guide downstream computation. A plausible implication is that future work will focus less on inventing new landmark labels and more on calibrating landmark confidence, handling noisy or partial landmarks, and composing multiple landmark sources within unified systems.