Knowledge-Action Graphs: Integrating Knowledge & Action

• Knowledge-Action Graphs (KAGs) are graph-based frameworks that represent entities (humans, objects, actions) and their relations to predict, recognize, and control actions.
• They integrate static knowledge and dynamic observations through nodes and edges, enabling applications like human action prediction, robot skill transfer, and biomedical QA.
• KAGs employ diverse reasoning mechanisms—ranging from attention and diffusion convolution to transformer-based approaches—to enhance decision-making and explainability.

Knowledge-Action Graph (KAG) denotes a family of graph-centered representations in which knowledge-bearing structures and action-bearing structures are modeled in a common formalism, so that graph reasoning can predict, recognize, constrain, explain, or execute actions. Across the literature, the idea appears in human action prediction, video action recognition, long-term anticipation, natural-language reinforcement learning, robot skill transfer, biomedical question answering, and post-FAIR knowledge infrastructures. The common pattern is that graph nodes encode entities such as humans, objects, actions, affordances, tools, skills, or semantic units, while graph edges encode relations such as spatial proximity, temporal succession, causal dependence, semantic compatibility, or applicability conditions; inference over these graphs then guides downstream action-related decisions (Chen et al., 2019, Zhang et al., 2020, Ammanabrolu et al., 2020, Vogt, 2 May 2026).

1. Conceptual scope and terminological variants

No single canonical definition of KAG is shared across all papers. In "SCR-Graph: Spatial-Causal Relationships based Graph Reasoning Network for Human Action Prediction," the architecture is explicitly described as essentially building what can be thought of as a Knowledge-Action Graph for human action prediction, with humans, objects, actions, and causal or knowledge states connected by spatial, temporal, and causal relations (Chen et al., 2019). In "Knowledge Integration Networks for Action Recognition," the graph module is called an Action Knowledge Graph (AKG), but the paper states that this corresponds closely to what many would call a KAG: a graph whose nodes are action, human, and scene features, and whose edges encode how such knowledge relates to actions (Zhang et al., 2020). "Knowledge-Guided Short-Context Action Anticipation in Human-Centric Videos" is presented as almost a direct instantiation of a KAG, because it links symbolic object-affordance-tool knowledge to anticipated future actions through transformer attention (Bhagat et al., 2023).

The term also appears by analogy rather than by name. "All About Knowledge Graphs for Actions" studies graphs whose nodes are action classes and whose edges encode semantic relatedness for zero-shot and few-shot action recognition; this is a class-centric KAG in which knowledge transfer is mediated through action-node connectivity (Ghosh et al., 2020). "Knowledge and Skill Graph" proposes a graph integrating static factual knowledge with dynamic behavioral knowledge, including agents, environments, skills, pre-trained networks, and offline datasets, and is explicitly characterized as being almost exactly what a Knowledge-Action Graph would be in reinforcement learning and robotics (Zhao et al., 2022). "Actionable Understanding: Action Units for Bridging the Knowledge-Action Gap in Post-FAIR Knowledge Infrastructures" does not use the term KAG, but it effectively specifies a graph in which operations, applicability conditions, and objectives are first-class typed objects, thereby providing a formal basis for a graph-native knowledge-to-action architecture (Vogt, 2 May 2026).

A separate terminological ambiguity arises because "KAG" is also used as the acronym for "Knowledge Augmented Generation" in a professional-domain QA framework. That work is not a video- or robotics-centered action graph, but it is nonetheless conceptually adjacent because the knowledge graph and logical forms organize the actions of an LLM-based reasoning system (Liang et al., 2024). This suggests that KAG is better understood as a research motif than as a single standardized model family.

2. Structural primitives: nodes, edges, and layered organization

Across the cited systems, KAG-like designs differ primarily in what they choose as nodes, what relations they encode as edges, and whether the graph is derived statistically from data, symbolically from domain knowledge, or by hybrid construction. Representative realizations are summarized below (Chen et al., 2019, Zhang et al., 2020, Bhagat et al., 2023, Ammanabrolu et al., 2020, Zhao et al., 2022, Vogt, 2 May 2026).

Realization	Nodes and relations	Operational role
SCR-Graph	Humans, objects, action classes, shadow nodes; human-human, human-object, action-transition edges	Human action prediction
KINet AKG	Action, human, and scene segment nodes; action-centric masked edges	Action recognition
Knowledge-guided anticipation	Objects, affordances, tools; object-affordance-tool chains	Long-term action anticipation
KG-A2C	Rooms, objects, “you”; containment and navigation relations	Constrain natural-language action generation
KSG	Agents, environments, skills, facts; model, dataset, and video attributes	Skill retrieval and transfer
Action Units framework	Semantic units, action units, applicability, objectives	Graph-native decision support

In SCR-Graph, the spatial graph at frame $t$ is heterogeneous, $G_{\text{spatial}}^{(t)} = (V^{(t)}, E^{(t)})$, with human and object nodes; human-object edges are added when an object lies within a perception radius $Dis_{(id)} = \lambda \cdot l_{(id)}$, where $l_{(id)} = \sqrt{m_{(id)}^2 + n_{(id)}^2}$, and human-human edges form a complete graph. The temporal side is a directed action graph $G_{\text{action}} = (V_a, E_a, W)$, where $W_{ij}$ is obtained by counting action switches between adjacent 12-frame segments and normalizing each row by softmax (Chen et al., 2019).

KINet constructs a higher-level graph from branch outputs rather than from symbolic entities. After global average pooling, the segment-level vectors $$x_i^{\text{action}}, x_i^{\text{human}}, x_i^{\text{scene}}$$ become AKG nodes. The graph is not fully connected: a binary mask $I_{\text{mask}}$ permits only edges incident to action nodes, thereby enforcing an action-centered fusion prior (Zhang et al., 2020).

The short-context anticipation model uses a symbolic domain graph $\mathcal{K}$ whose nodes are objects, affordances, and tools, with edges such as object $\rightarrow$ affordance and affordance $G_{\text{spatial}}^{(t)} = (V^{(t)}, E^{(t)})$0 tool. The graph is activated from detected objects and then expanded by graph search and thresholded importance scoring, so the active subgraph is conditioned on the current video frame (Bhagat et al., 2023).

Other KAGs extend the ontology further. KG-A2C maintains a dynamic graph $G_{\text{spatial}}^{(t)} = (V^{(t)}, E^{(t)})$1 over rooms, objects, inventory, and the ego node “you”; KSG partitions nodes into entity nodes and attribute nodes, with agents, environments, skills, and facts on the entity side, and descriptions, skill displays, pre-trained networks, and offline datasets on the attribute side (Ammanabrolu et al., 2020, Zhao et al., 2022). The Action Units framework defines compound semantic objects in which an Action Unit has typed parts for input, output, plan specification, applicability conditions, and objective, thereby moving from relation-centric graphs toward explicitly operational graphs (Vogt, 2 May 2026).

3. Reasoning mechanisms over KAGs

A central distinction among KAG systems lies in how graph structure is operationalized. SCR-Graph combines heterogeneous spatial attention with temporal causal diffusion. In the spatial dimension, node features are first projected by type-specific transformations $G_{\text{spatial}}^{(t)} = (V^{(t)}, E^{(t)})$2, then aggregated by node-level and type-level attention. In the temporal dimension, a Diffusion Convolutional Recurrent Neural Network operates on the action graph. The action-transition matrix is defined by

$G_{\text{spatial}}^{(t)} = (V^{(t)}, E^{(t)})$3

and diffusion convolution uses powers of $G_{\text{spatial}}^{(t)} = (V^{(t)}, E^{(t)})$4 to model multi-step random walks. The resulting DCGRU hidden state is fused back into the heterogeneous spatial graph through a shadow node, with self-attention determining when causal knowledge should be activated (Chen et al., 2019).

KINet uses two-level knowledge integration. At the medium level, Cross Branch Integration modulates action-branch feature maps with human and scene branches. At the semantic level, the AKG applies a masked graph convolution

$G_{\text{spatial}}^{(t)} = (V^{(t)}, E^{(t)})$5

where $G_{\text{spatial}}^{(t)} = (V^{(t)}, E^{(t)})$6 stacks action, human, and scene node features and $G_{\text{spatial}}^{(t)} = (V^{(t)}, E^{(t)})$7 ensures that only action-centered interactions remain active. The updated action nodes alone are retained for classification, so graph reasoning is explicitly organized around action refinement rather than around generic message passing (Zhang et al., 2020).

The short-context anticipation model injects symbolic knowledge into transformer attention by computing rectification matrices from the activated knowledge graph and replacing standard attention with

$G_{\text{spatial}}^{(t)} = (V^{(t)}, E^{(t)})$8

Here $G_{\text{spatial}}^{(t)} = (V^{(t)}, E^{(t)})$9 and $Dis_{(id)} = \lambda \cdot l_{(id)}$0 are produced from KG-derived context embeddings by LSTM-based functions. This turns the symbolic graph into an attention prior that boosts or suppresses interactions among visual tokens according to object-affordance-tool structure (Bhagat et al., 2023).

KG-A2C offers a different mechanism: the graph is both state representation and action-space constraint. The action space is template based, but object arguments are restricted to a graph-derived mask $Dis_{(id)} = \lambda \cdot l_{(id)}$1, yielding

$Dis_{(id)} = \lambda \cdot l_{(id)}$2

The dynamic knowledge graph is embedded by a Graph Attention Network and concatenated with observation and score encodings to form the RL state. The same graph therefore encodes belief about the world and constrains admissible natural-language actions (Ammanabrolu et al., 2020).

Explainable knowledge-graph embeddings for robot actions add an interpretability layer rather than a new predictive backbone. A black-box KGE produces action-supporting inferences such as ObjInLoc or ObjUsedTo; a local surrogate model based on Subgraph Feature Extraction and decision trees then approximates those inferences and grounds them in relation paths that can be verbalized. This is a KAG reconciliation mechanism: it makes the knowledge-to-action dependency explicit enough for non-experts to inspect and correct (Daruna et al., 2022).

4. Action prediction, recognition, anticipation, and control

In action prediction and recognition, KAGs are primarily evaluated by whether explicit relational structure improves over purely visual or purely textual baselines. SCR-Graph treats future human action prediction as multi-label classification with a binary cross-entropy loss and reports that, on VIRAT/ActEV, it improves mAP by $Dis_{(id)} = \lambda \cdot l_{(id)}$3–$Dis_{(id)} = \lambda \cdot l_{(id)}$4 over the Next baseline across feature settings; with full features, Next reaches mAP $Dis_{(id)} = \lambda \cdot l_{(id)}$5 and SCR-Graph reaches mAP $Dis_{(id)} = \lambda \cdot l_{(id)}$6, a $Dis_{(id)} = \lambda \cdot l_{(id)}$7 relative improvement (Chen et al., 2019).

KINet shows the same pattern in recognition rather than anticipation. The complete three-branch system with CBI, AKG, and multi-task training reaches $Dis_{(id)} = \lambda \cdot l_{(id)}$8 top-1 on Kinetics-400 with a ResNet-50 backbone, compared with $Dis_{(id)} = \lambda \cdot l_{(id)}$9 for TSN-ResNet50, and the two-stream KINet obtains $l_{(id)} = \sqrt{m_{(id)}^2 + n_{(id)}^2}$0 top-1 and $l_{(id)} = \sqrt{m_{(id)}^2 + n_{(id)}^2}$1 top-5. Transfer to UCF-101 yields $l_{(id)} = \sqrt{m_{(id)}^2 + n_{(id)}^2}$2 top-1. The ablations further isolate the contribution of the graph component: AKG plus multi-task training reaches $l_{(id)} = \sqrt{m_{(id)}^2 + n_{(id)}^2}$3 top-1, indicating that representation-level graph reasoning contributes materially beyond branch sharing alone (Zhang et al., 2020).

For long-term anticipation from short context, the symbolic KAG over objects, affordances, and tools improves FUTR on Breakfast and 50Salads by up to $l_{(id)} = \sqrt{m_{(id)}^2 + n_{(id)}^2}$4 MoC. The reported gains are strongest on 50Salads, while Breakfast shows smaller but mostly positive gains. The model is particularly intended to anticipate long-horizon action sequences and durations from short observed prefixes, so the KAG functions as a procedural prior when visual context alone is insufficient (Bhagat et al., 2023).

In zero-shot and few-shot action recognition, KAGs are class-graph transfer mechanisms rather than scene graphs. "All About Knowledge Graphs for Actions" compares three graph constructions: KG1 from action-phrase embeddings, KG2 from verb and noun graphs, and KG3 from few-shot visual prototypes. On UCF101 zero-shot, KG1+KG2 reaches $l_{(id)} = \sqrt{m_{(id)}^2 + n_{(id)}^2}$5 class-wise mean accuracy; on HMDB51 it reaches $l_{(id)} = \sqrt{m_{(id)}^2 + n_{(id)}^2}$6; on Charades it reaches $l_{(id)} = \sqrt{m_{(id)}^2 + n_{(id)}^2}$7 mAP. In the 5-shot setting on UCF101, KG3+KG1+KG2 reaches $l_{(id)} = \sqrt{m_{(id)}^2 + n_{(id)}^2}$8, substantially above the nearest-neighbor baseline at $l_{(id)} = \sqrt{m_{(id)}^2 + n_{(id)}^2}$9. The study also shows that sentence-level action representations are crucial: Word2Vec-based KG1 achieves $G_{\text{action}} = (V_a, E_a, W)$0 on UCF101, while Sentence2Vec-based KG1 reaches $G_{\text{action}} = (V_a, E_a, W)$1 (Ghosh et al., 2020).

Not all action-centric graphs benefit from standard KG completion. "Knowledge Graph Completion for Action Prediction on Situational Graphs -- A Case Study on Household Tasks" builds disconnected situational graphs of household activities from the KIT Bimanual Actions Dataset and shows that link-prediction models such as TransE, TransR, RotatE, ComplEx, DistMult, and LiteralE variants fail to outperform simple baselines. For parent action prediction, the best KG model reaches only about $G_{\text{action}} = (V_a, E_a, W)$2–$G_{\text{action}} = (V_a, E_a, W)$3 Hits@5, while a simple object-conditioned frequency baseline reaches $G_{\text{action}} = (V_a, E_a, W)$4 Hits@1 and $G_{\text{action}} = (V_a, E_a, W)$5 Hits@3; for subsequent sub-action prediction, the same baseline reaches $G_{\text{action}} = (V_a, E_a, W)$6 Hits@1, while the best KG model, RotatE, reaches $G_{\text{action}} = (V_a, E_a, W)$7. This negative result is important because it shows that episodic, sparse, weakly connected situational KAGs violate assumptions underlying many standard KGC benchmarks (Arustashvili et al., 19 Aug 2025).

5. Agentic and operational KAGs

A second line of work treats KAGs as explicit knowledge-to-action infrastructures rather than as action-recognition backbones. KSG, built on CN-DBpedia, adds agent, environment, and skill nodes together with attribute nodes for descriptions, videos, pre-trained networks, and offline datasets. It is used to retrieve transferable skills for new DRL tasks. In the reported experiments, using KSG to select pre-training models for a quadruped walking on irregular terrain reduces nearly half of the training time compared with direct training from scratch, and the overall conclusion is that KSG boosts new skill learning efficiency (Zhao et al., 2022).

The robot-explanation framework for learned KG embeddings provides another operational view. A KGE produces facts that support robot actions, such as object-location or object-use relations; a local decision-tree surrogate then reconciles those inferences with grounded graph paths and natural-language explanations. The surrogate achieves $G_{\text{action}} = (V_a, E_a, W)$8 F1-fidelity against the black-box KGE, compared with $G_{\text{action}} = (V_a, E_a, W)$9 for XKE. More importantly, human correction of nonsensical beliefs increases link-prediction MRR from $W_{ij}$0 to $W_{ij}$1, and simulated robot task success improves from 187 to 250 successes, a $W_{ij}$2 relative improvement (Daruna et al., 2022).

KAG-like systems also appear in question answering and generation when graph structure determines the action policy of an LLM agent. KGARevion converts latent LLM knowledge into triplets, verifies them against a biomedical KG, iteratively revises incorrect triplets, and answers with the verified subgraph. It reports accuracy improvements of over $W_{ij}$3 over 15 models on medical QA benchmarks and $W_{ij}$4 on three newly curated medical QA datasets, showing that query-specific verified subgraphs can act as action-driving evidence structures (Su et al., 2024). KAQG similarly turns a domain KG into a multi-agent educational pipeline: PageRank over concept nodes selects core content, Bloom’s Taxonomy guides cognitive level, and Item Response Theory is used for difficulty control. In its ACT-style evaluation, generated low-, medium-, and high-difficulty items show differentiated $W_{ij}$5-values while maintaining reasonable discrimination indices, and the full system is organized explicitly as a graph-centered action pipeline for question generation and validation (Chen et al., 12 May 2025).

The most explicit formalization of an operational KAG is given by the Action Units framework. There, an Action Unit is a compound semantic unit with typed parts for inputs, outputs, plan specifications, applicability conditions, and objectives, and applicability is formalized by

$W_{ij}$6

Conditional Action Units are executable IF-THEN structures in which the IF-clause is a query-like question unit and the THEN-clause is a directive unit. This turns the graph into a decision-support and workflow-orchestration system rather than merely a storage structure (Vogt, 2 May 2026).

A related but distinct use of the acronym appears in "KAG: Boosting LLMs in Professional Domains via Knowledge Augmented Generation." That framework combines LLM-friendly knowledge representation, mutual indexing between KG and chunks, logical-form-guided hybrid reasoning, semantic knowledge alignment, and model enhancement. On multihop QA, it reports relative F1 improvements of $W_{ij}$7 on 2Wiki and $W_{ij}$8 on HotpotQA over prior methods. Although its primary object is professional QA rather than action prediction, its logical forms—Retrieval, Sort, Math, Deduce, and Output—function as explicit actions over structured knowledge, so it is directly relevant to broader KAG discourse (Liang et al., 2024).

6. Limitations, misconceptions, and open research problems

A recurring misconception is that KAG necessarily means a single graph type or a single implementation pattern. The literature instead shows several incompatible but related families: data-driven action graphs built from observed transitions, multi-branch representation graphs, symbolic affordance graphs, dynamic RL world graphs, graph-plus-tool orchestration layers, and formally typed action-unit infrastructures. This diversity is productive, but it also makes direct comparison difficult.

Several technical limitations recur. KINet notes dependence on specific teacher networks, a static and coarse graph with one node per segment per branch and a single GCN layer, dataset-specific semantics, and limited modality; it also reports that an object branch was unstable because of resolution, motion blur, and limited object categories (Zhang et al., 2020). KG-A2C depends on rule-based OpenIE, hand-crafted update rules, template-based language, and a deterministic graph under partial observability; uncertainty over nodes and edges is not represented (Ammanabrolu et al., 2020). KSG is still narrow in coverage, centered on a limited set of locomotion skills, and does not yet represent richer temporal relations among skills or more complex manipulation behaviors (Zhao et al., 2022).

Negative evidence is especially informative in situational graphs. The household action case study shows that sparse, disconnected, episode-level KAGs can render standard link-prediction algorithms “not fit for the job,” because such graphs lack the global shared structure on which embedding models rely (Arustashvili et al., 19 Aug 2025). This result cautions against assuming that generic KG completion transfers automatically to action prediction.

Agentic KAGs introduce different challenges. KGARevion must manage KG incompleteness, noisy LLM-generated triplets, and computational cost from repeated Generate–Review–Revise cycles (Su et al., 2024). Knowledge Augmented Generation identifies costly LLM calls, planning complexity, and noise in open information extraction as central bottlenecks (Liang et al., 2024). The Action Units framework, while highly explicit, is conceptual and acknowledges that implementation toolchains, bounded action spaces, and probabilistic applicability remain open engineering and modeling problems (Vogt, 2 May 2026).

The forward-looking directions are correspondingly diverse. Existing papers propose dynamic and multi-layer graphs, explicit temporal edges, richer node types such as objects and affordances, external knowledge bases, multimodal extensions, better planning models for logical-form generation, graph-guided hybrid retrieval, and more rigorous benchmarks for disconnected situational graphs (Bhagat et al., 2023, Liang et al., 2024, Arustashvili et al., 19 Aug 2025). A plausible implication is that future KAG research will continue along two complementary trajectories: one focused on graph-based relational priors for perception and action understanding, and another focused on graph-native operational systems in which knowledge representations are directly executable.