---
title: Learning from Interactions (LFI)
url: https://www.emergentmind.com/topics/learning-from-interactions-lfi
type: topic
---

# Learning from Interactions (LFI)

Learning from Interactions (LFI) denotes learning settings in which supervision is generated through interaction with an environment, a teacher, a user, or another model, rather than through only fully supervised learning mechanisms relying on highly curated datasets or passively consumed scaffolded content. Across the cited literature, interaction supplies image changes after physical actions, sentence and scalar reward feedback, expert interventions, multidimensional feedback vectors with no explicit reward information, implicit user follow-up signals, dialogical transcripts, and cross-modal attention patterns. This suggests that LFI is best treated as a family of interaction-centered formulations whose common objective is to ground representation learning, policy learning, or knowledge transfer in the consequences of acting and communicating [2006.09306][1705.09906][2106.04887][2410.13852][2509.18733].

## 1. Conceptual scope and contrast classes

In computer vision, LFI is positioned against “fully supervised learning mechanisms relying on highly curated datasets.” In LLM-based learning support, scaffolded content is described as being “primarily consumed passively,” which offers limited support for active learner engagement. In grounded language learning, pre-collected annotated corpora are characterized as “a static snapshot representation of the knowledge from the annotator,” limiting adaptiveness and generalization. In higher learning, Interactionalism contrasts “individual, linear, and content-broadcast” learning with “personalized, socialized, and conversational” learning. In visual foundation models, result-oriented paradigms are said to “neglect the underlying interaction processes,” creating a representational discrepancy with vision-language models [2006.09306][2603.07277][1705.09906][2501.00867][2509.18733].

These formulations differ in object, scale, and epistemic target. Some target object discovery and physical property estimation; some target grounded dialogue; some target policy grounding without explicit reward; some target tutoring, social reasoning, or adaptive concept transfer; and some target process-level knowledge transfer between pretrained model families. The common denominator is that the learner is not only exposed to data but also participates in a loop in which its own actions, questions, or intermediate computations affect the supervision it later receives. This suggests that LFI is not a single architecture or theorem, but a recurrent design principle spanning embodied AI, robotics, educational technology, and model adaptation.

## 2. Forms of interactive supervision

The literature instantiates interaction through multiple supervision channels.

| Interaction source | Supervision signal | Illustrative papers |
|---|---|---|
| Physical environment | Image differences, noisy masks, force feedback | [2006.09306] |
| Teacher dialogue | Sentence feedback and scalar reward | [1705.09906] |
| Cooperative partner behavior | Observable physical actions in a shared environment | [1906.10187] |
| Human oversight | Interventions, segmentation review, ASR correction, natural-language correction | [1912.02241], [2212.10787], [2309.04316] |
| User follow-up turns | Positive, neutral, or negative implicit feedback | [2410.13852] |
| Ungrounded interface signal | Multidimensional feedback vector with no explicit reward | [2106.04887] |
| Cross-modal model internals | Cross-modal attention patterns as interaction-based supervision | [2509.18733] |

In the embodied object-learning setting, the agent computes the difference between pre- and post-interaction images and then post-processes with superpixel segmentation to obtain a noisy mask indicating regions of change. In grounded language learning, the teacher provides both verbal feedback and scalar reward, so the learner receives sentence-level and non-verbal supervision in the same conversational turn. In retrospective LLM improvement, feedback need not be solicited at all: rephrasing, frustration, task pivoting, approval, and continuations are retrospectively decoded from the follow-up utterance as positive, neutral, or negative feedback [2006.09306][1705.09906][2410.13852].

Other systems shift the supervision boundary further. Interaction-Grounded Learning assumes that the learner only observes a context, takes an action, and receives a multidimensional feedback vector with no explicit reward information, so reward must be discovered as a latent property of interaction. ITES inserts the user into the recognition pipeline through preview and interactive correction of both segmentation and ASR output. The ARMAR-6 system closes the loop by feeding back human instructions, environment observations, and execution results to the LLM, and then storing improved interactions for later retrieval. In the VLM-to-VFM transfer literature, “interaction” refers not to external dialogue but to the internal cross-modal attention mechanisms of a pretrained VLM, which are treated as transferable process supervision [2106.04887][2212.10787][2309.04316][2509.18733].

## 3. Core algorithmic patterns

A recurring pattern is the joint learning of *where to act*, *how to interpret outcomes*, and *how to update future behavior*. In “Learning About Objects by Learning to Interact with Them,” the core model is a UNet-inspired CNN with RGB+D input and three outputs: interaction scores, force logits, and spatial embeddings. Training is driven by three distinct loss functions—Interaction Loss, Force Loss, and Embedding Loss—and a fixed-size memory bank with a custom importance score for prioritized replay. At inference, instance segmentation is obtained by clustering pixels whose embedding vectors are close, while mass is estimated through repeated trial and error over three quantized classes: light, medium, and heavy [2006.09306].

In grounded language learning, the principal technical move is to combine imitation and reinforcement. The learner predicts the next teacher sentence through imitation while also optimizing a sentence-generation policy through policy gradient. The paper gives the combined objective as
$$
\mathcal{L}_\theta = \mathcal{L}^{\rm I}_\theta + \mathcal{L}^{\rm R}_\theta,
$$
so “knowing how to speak” and “knowing what to say” are tied through shared parameters, a hierarchical RNN, a visual encoder with attention, and a controller network with a Gaussian policy module [1705.09906].

Interaction-Grounded Learning formalizes a different problem: the learner receives only \((x,a,y)\), where \(y\) is an ungrounded feedback vector, and must jointly learn a policy \(\pi\) and a reward decoder \(\psi\). The central objective compares candidate policies to a “bad” policy:
$$
\mathcal{L}(\pi, \psi) \equiv V(\pi, \psi) - V(\pi_{\text{bad}}, \psi),
$$
under the key assumption that \((x,a) \perp y \mid r\). The paper provides both a batch algorithm and an online Explore-Exploit Ground Learning algorithm, together with finite-sample and regret guarantees. It also proves that unsupervised clustering of feedback vectors is generally insufficient, because equivalent IGL tasks can induce identical feedback distributions under the random policy while differing in the optimal action [2106.04887].

ReSpect operationalizes retrospective learning from human-LLM interactions through a deployment-and-retraining loop. A feedback decoder, implemented as a prompted LLM, labels each action as positive, neutral, or negative from the subsequent human utterance. Retraining then proceeds with one of three paradigms: supervised fine-tuning on positive examples, offline REINFORCE with reward mapping \(1, 0, -0.1\) for positive, neutral, and negative feedback, or KTO. Neutral feedback is generally discarded, and negative examples are downsampled to keep a \(\sim 5{:}4\) positive:negative ratio where needed [2410.13852].

In the visual foundation model literature, the LFI framework introduces Interaction Queries and interaction-based supervision extracted from VLM cross-attention. The resulting VFM is trained with
$$
\mathcal{L}_{\text{total}} = \mathcal{L}_{\text{task}} + \mathcal{L}_{\text{align}},
$$
where \(\mathcal{L}_{\text{align}} = D_{KL}(C_{\text{AGT}} \parallel C_{\text{VLM}})\), and fusion between VFM-standard and VLM-guided interactions is controlled by a Gated Control Network. Here the interaction is not an external action loop but an explicitly modeled reasoning process transferred across architectures [2509.18733].

## 4. Embodied, robotic, and physically grounded systems

Embodied LFI often treats interaction as the primary route to discovering task structure. In AI2-THOR, a self-supervised agent is placed at random locations in rich, photorealistic scenes and learns object geometry and relative masses without external guidance. The evaluated scenarios include “Novel Spaces” and “Novel Objects.” The model produces notable instance segmentation masks even for novel object categories, outperforms adapted unsupervised/video-based segmentation and optical flow baselines, and achieves roughly 50–55% mean per-class accuracy for mass estimation on novel objects and spaces, compared with chance at 33%. Qualitatively, it handles clutter and multiple objects but struggles with small or very large/massive objects [2006.09306].

“Learning to Interactively Learn and Assist” studies interaction as coordination under information asymmetry. A principal knows the task, an assistant does not, and both act in a cooperative partially observable Markov game without a specialized communication channel. Using MAIDRQN and MADDRQN, the framework produces emergent information-sharing, information-seeking, and question-answering. In the human-agent setting, the trained assistant generalizes to a human user and achieves significantly better performance cooperatively with a human than a human performing the task alone [1906.10187].

ITES extends Learning-from-Observation into a multimodal and interactive system for household robots. Its “stop-and-go” demonstration protocol alternates atomic-action boundaries with verbal instructions, then performs hand-stop detection, speech recognition, user preview and interactive correction, NLP-based task recognition, and skill parameter extraction. The task recognizer is a Random Forest over BERT embeddings trained on 1340 segmented video+instruction samples across 12 task classes, with 10-fold cross-validation average accuracy of about 83%. Real-robot evaluations demonstrate successful teaching of multiple operations for several scenarios, including pick-carry-place, obstacle avoidance, throwing away a cup, and opening a fridge door [2212.10787].

“Learning Predictive Models From Observation and Interaction” addresses a complementary problem: learning action-conditioned predictive models from both interaction data and observation data with missing actions. The model introduces latent variables \(z_t\), separate ELBOs for interaction and observation data, Jensen-Shannon regularization to align latent action representations, and domain-dependent priors to accommodate embodiment differences. In robotic planning, the resulting POI model reaches a 40.0% \(\pm 8.9\%\) success rate, compared with 23.3% \(\pm 7.7\%\) for SAVP and 36.7% \(\pm 8.8\%\) for the oracle, and enables a robot to learn to use tools by observing humans without ever seeing a robotic video of tool use [1912.12773].

The ARMAR-6 system pushes interaction into incremental natural-language improvement. An LLM orchestrates behavior by generating Python statements in an interactive console to invoke perception and action, while a second LLM produces improved transcripts when the robot receives corrective human feedback. These improved interaction episodes are stored in episodic memory and retrieved on semantically similar future requests, yielding quantitatively and qualitatively demonstrated generalized incrementally-learned knowledge in simulation and real-world settings [2309.04316].

A neighboring but distinct line is Learning from Interventions. There, an expert overseer intervenes only when an unsafe action is about to be taken. The 2019 hierarchical-policy formulation adds backtracking to account for expert reaction delay and a two-level policy that predicts sub-goals and actions, thereby learning long-term behavior while remaining robust to delay. The paper reports faster training and better asymptotic performance than typical Learning from Demonstrations in autonomous driving tasks [1912.02241].

## 5. LLMs, tutoring, and dialogical learning

In educational technology, LFI often denotes the difference between passive content delivery and active engagement with scaffolded material. “From Passive Consumption to Active Interaction” studies a tutoring prototype in which scaffolded content is masked and can be revealed only through a scratch-off mouse-drag interaction. In a within-subjects laboratory study with \(N=8\), participants reported higher engagement and attentional focus in the interactive condition, with mean Likert scores \(>5.5/7\). Average quiz scores were 80.0% for interactive scaffolding and 77.5% for non-interactive scaffolding, SUS was approximately 82/100, and NASA-TLX was slightly lower for the interactive condition. The work frames this as a movement from passive to active engagement under the ICAP framework [2603.07277].

Interactionalism generalizes this argument into a design blueprint for higher learning in the GenAI era. It emphasizes co-agency with AI, dialogical learning, and “interactional intelligence” as a skill set composed of meta-cognitive and meta-emotional components. The proposed educational architecture shifts evaluation away from end-products alone and toward the transcript of the dialogical interaction, with “always-on” feedback and multiple-shot evaluation replacing one-shot exams. The paper explicitly states that Interactionalism is not advanced as a theory of learning, but as a blueprint for the practice of learning in coordination with GenAI [2501.00867].

INTERACT studies question-driven concept transfer between a student LLM and a teacher LLM across 1,347 contexts, including song lyrics, news articles, movie plots, academic papers, and images. It evaluates three scenarios—static student with lesson, dynamic student without lesson, and dynamic student with lesson—and reports that interactive learning consistently enhances performance, achieving up to a 25% improvement. “Cold-start” student models match static learning baselines in as few as five dialogue turns, and interactive setups can mitigate the disadvantages of weaker teachers. A salient negative result is that borrowed interaction transcripts do not significantly improve learning, indicating that active engagement is not reducible to passive exposure to someone else’s dialogue [2412.11388].

“Building Knowledge from Interactions” transfers these concerns into social and task-oriented HRI. Its architecture combines an LLM-based Interaction Manager with working memory and a Neo4j knowledge graph used as long-term episodic and semantic memory. In a preliminary HRI user study with 10 participants, the reported Interaction Success Rate was 100%, Average Robot Response Time was \(1.1 \pm 0.26\) s, and Architecture Success Rate was 60%, with failures attributed to auxiliary modules such as OpenAI API rate limits, TextToSpeech failure, and “Face not saved.” In synthetic-data evaluation, GraphCypherQA matched RAG and Naïve Graph on user-specific questions but outperformed both on general or multi-hop questions, with faithfulness scores of 0.73 versus 0.49 and 0.57 [2504.01588].

## 6. Generalization, evaluation, and open questions

Generalization is a central empirical criterion across the LFI literature. In embodied object learning, the agent generalizes not only to novel instances from seen categories but also to novel object categories. In interactive grounded language learning, the joint imitation-plus-reinforcement model attains more than 98% accuracy in novel situations and 97.5%/89.0% in knowledge transfer settings, whereas imitation-only and reinforcement-only baselines are substantially worse. ReSpect improves task completion from 31% to 82% after six rounds without any external annotation, while also increasing utterance-level exact match from 31% to 53% and reducing mean turns per completed interaction from 8.9 to 6.7. In the VFM-transfer formulation of LFI, the framework yields 3.3 and 1.6mAP/2.4AP absolute gains on TinyImageNet classification and COCO detection/segmentation, 2.4 and 9.3 zero-shot improvements on PACS and VLCS, and 2.7 times better semantic consistency metrics in human evaluation [2006.09306][1705.09906][2410.13852][2509.18733].

Several misconceptions are explicitly challenged. First, interaction is not equivalent to unsupervised learning: Interaction-Grounded Learning proves that unsupervised reward decoding by clustering feedback vectors generally fails, because interaction is what grounds the latent reward. Second, interaction is not equivalent to exposing a learner to more text: INTERACT finds that passive access to a transcript of successful dialogue does not reproduce the gains of active questioning. Third, negative feedback is not always straightforwardly helpful: in ReSpect, positive-only supervised fine-tuning outperformed offline RL and KTO, and the paper notes that negative signals, if not partnered with more provided context, could confuse rather than regularize learning [2106.04887][2412.11388][2410.13852].

The term itself remains heterogeneous. In one line it refers to self-supervised discovery of objects and physical properties through deliberate action; in another, to interaction-grounded reward discovery; in another, to tutoring interfaces that structure attention; in another, to continual LLM self-improvement from user follow-up; and in another, to transferring cross-modal reasoning processes into visual foundation models. This suggests a shared commitment to learning from consequences, corrections, and interaction patterns, but not yet a single formalism that covers all uses. The literature therefore presents LFI simultaneously as a methodological family, a design principle for educational and social systems, and, in some subfields, a specific named framework with its own losses, prompts, or architectural modules [2603.07277][2501.00867][2509.18733].

Source: https://www.emergentmind.com/topics/learning-from-interactions-lfi