TalkingBabies Framework: Language Learning

• TalkingBabies Framework is a closed-loop experimental setup that integrates multimodal perception and adaptive feedback to study language development in both infants and artificial agents.
• It employs precise measurement tools like eye-tracking, thermal imaging, and behavioral coding to quantify infant attention, arousal, and engagement.
• The framework extends to synthetic grammar learning in LLMs, highlighting challenges in multi-turn conversations and the induction of combinatorial language structures.

The TalkingBabies Framework is a closed-loop experimental system designed to study the mechanisms of early language acquisition and interaction in two contexts: (a) naturalistic, contingent exchange between human infants and embodied artificial agents, and (b) pattern induction and online grammatical learning in LLMs interacting with synthetic grammars. The approach is characterized by multimodal perception, adaptive dialogue management, and quantifiable behavioral benchmarks that isolate core features of human linguistic development and agent learning.

1. System Architecture and Components

In its original implementation, the TalkingBabies Framework features a multiparty agent-infant setup (Gilani et al., 2018). Two coordinated agents engage in face-to-face communication with an infant: a physically-embodied humanoid robot (Hello Robot Maki platform) capable of gaze, head movements, and facial gestures, and a 3D animated avatar rendered on a screen, supporting American Sign Language (ASL) signing and multimodal expressions. Sensory apparatus includes a Tobii Pro X3-120 eye-tracker (acquiring infant gaze and attention at 120 Hz), a FLIR A655sc thermal infrared camera (measuring facial temperature—an arousal and engagement proxy), and an RGB-D camera for potential posture/gesture parsing.

All modules interface via an ActiveMQ publish/subscribe system. The central dialogue manager aggregates “area of interest” (AOI), “readiness-to-learn” (attention × arousal), and coded “baby-behavior” signals to trigger agent actions and manage group interaction routines, establishing a dynamic and socially contingent feedback loop essential for real-time engagement.

2. Perception Modules and Social State Estimation

Attention is operationalized through continuous eye-tracking: the AOI variable (possible values: Robot, Avatar, Between, Outside) is updated every 500 ms via majority-voted gaze point, with subsequent smoothing via a hysteresis threshold (≥150 ms for shift validity). A normalized attention score over any sliding window ΔT is defined as:

$$\mathrm{Attention}(\Delta T) = \frac{1}{\Delta T} \int_{t-\Delta T}^t 1_{\mathrm{[AOI(\tau)=Avatar]}}\,d\tau$$

Thermal arousal is measured by filtering the raw infrared data from the nasal tip, subtracting a rolling baseline, and thresholding the residual ΔT(t) into three states: +1 (parasympathetic/engaged), 0 (neutral), or –1 (sympathetic/disengaged), with calibration set by empirical thresholds.

The framework's state space, $$s = (\mathrm{AOI}, \alpha, \mathrm{BB}, \mathrm{Flags})$$, allows the dialogue manager to condition action selection on real-time indicators of infant engagement, arousal, and coded behaviors (23 categories, including proto-signs, gestures, vocalizations, and affect signals).

3. Dialogue Management and Policy Dynamics

The action set includes primitive behaviors (nodding, gaze shift, signing) and complex, temporally extended routines (e.g., 6-step greeting sequence, nursery rhyme presentation, re-engagement strategies). Policy selection $\pi(a|s)$ is realized as a deterministic rule base: if rule $j$ fires for a region in $S$, its associated action $a$ is executed. A plausible extension employs a softmax over an engagement-weighted utility function:

$$U(s, a) = \alpha \cdot \mathrm{Attention}(s) + \beta \cdot \alpha(t) + \gamma \cdot \mathrm{ContingencyScore}(s)$$

$$\pi(a|s) = \frac{\exp[U(s, a)/\tau]}{\sum_{a'} \exp[U(s, a')/\tau]}$$

Engagement-gated transitions prioritize pedagogical episodes (nursery rhymes, turn-taking) when the infant displays readiness (α=+1, AOI=Avatar), uses social cues (LOOK-AT-ME) to recruit attention, and falls back to soothing or re-engagement actions under disengagement.

4. Social Contingency, Behavioral Quantification, and Evaluation

Behavioral detection is achieved via a real-time GUI for human annotation (with future goals for automation), capturing event-aligned infant actions. The framework computes contingency metrics as follows: following each agent utterance $U$ at time $t_0$, the presence of any valid infant behavior $BB_i$ in the response window $[t_0, t_0+\Delta]$ (e.g., $\Delta = 3\,\mathrm{s}$) is counted; response latency is $L_i = t_{BB_i} - t_0$.

Key measures:

• Contingency Rate (fraction of agent prompts eliciting a timely infant response): $$\mathrm{ContingencyRate} = \frac{\#\,BB_i\,\text{in window}}{\text{total \# opportunities}}$$
• Response Latency (mean $L$ across episodes).
• Statistical analysis via paired $t$-test or binomial test on observed latencies versus a chance model.

Empirical results (Gilani et al., 2018) indicate high engagement and contingency: all $N=8$ infants maintained joint attention for $1.5$–$5$ min (mean $3$ m $40$ s), with Contingency Rates ≈ 0.72 and mean response latencies ($L \approx 1.4$ s) significantly faster than chance. Infants attempted to imitate both manual and affective components of agent behavior.

5. Tinkatongue: Synthetic Grammar Learning Protocol for LLM Agents

A complementary “TalkingBabies” paradigm is realized in silico for LLMs, using a synthetic language “Tinkatongue” to model rapid, unlabeled grammatical induction (Swain et al., 9 Sep 2025). Tinkatongue comprises:

• Phonology: Lexicon of 100 bisyllabic, monomorphemic, lowercase strings.
• Syntax: Sentences are ordered triples of words from Lex, with legal sentences $$L_{\mathrm{sent}} = \{ (w_1, w_2, w_3) \mid w_i \in \mathrm{Lex} \}$$.
• Conversation: Four-turn dialogues, with adjacency (overlap of at least one word between consecutive sentences) enforced; only 25 valid quadruples.
• Feedback: The agent receives “koro” for valid outputs, or the reset signal $μ$ (“moko lira bani”) for invalid ones.

A session ends after either three complete conversations or $T_{max}=100$ turns.

6. Quantitative Evaluation Metrics and Model Performance

Empirical evaluation on GPT-4o-mini, Gemini-2.5-flash, and Claude-3.5-haiku used the following metrics:

• Turn Validity Rate (TVR): $\mathrm{TVR} = \frac{\text{valid\_turns}}{N}$
• Feedback Responsiveness (FR)
• Adjacency Compliance (AC)
• Time to First Positive Feedback (TTFK)

Model	TVR (mean ± std)	AC (mean)	FR	TTFK (mean ± std)
GPT-4o-mini	0.012 ± 0.017	~0.09	1	26.8 ± 12.4
Gemini-2.5-flash	0.061 ± 0.082	~0.08	1	17.2 ± 10.1
Claude-3.5-haiku	0.337 ± 0.220	~0.10	1	6.4 ± 8.1

FR was perfect for all agents: every negative feedback was followed immediately by a valid reply, but multi-turn conversation success was never achieved (no agent completed three full sessions within 100 turns). Observed strategies included babbling, overgeneralization (“soro soro soro…”), systematic word-order exploration, and verbatim imitation of feedback prompts.

7. Implications, Limitations, and Extensions

The TalkingBabies Framework demonstrates the necessity of social contingency, real-time multimodal perception, and adaptive feedback in driving engagement and learning in both infants and LLMs (Gilani et al., 2018, Swain et al., 9 Sep 2025). Empirical results show that human infants exhibit rapid, synchronous engagement and behavioral contingency in response to agent input, strongly suggesting that artificial agents can scaffold early communicative exchanges if properly tuned to infant state.

For LLMs, failure to acquire the combinatorial constraints of Tinkatongue by feedback alone, despite high immediate responsiveness, highlights architectural limitations. These findings suggest future models will require explicit memory or latent parsing components, integrable curriculum learning (progressing from simpler to more complex structural constraints), and possibly differentiable grammar induction (e.g. neural-PCFG) to facilitate multi-turn, feedback-driven learning.

Recommended extensions include automated behavior detection, adaptive/reinforcement policy learning, curriculum design for increasing linguistic complexity, graded (rather than binary) feedback, and transfer to multimodal or recursively structured artificial languages. A plausible implication is that scaffolding grounded, socially-contingent learning environments for both infants and artificial agents will be central to advancing true grammatical and pragmatic acquisition.