Bridging Talk and Thought: Understanding Dialogue Dynamics Across Collaborative Problem-Solving Contexts

Abstract: We present a conceptual framework for analyzing dialogue in collaborative problem-solving contexts, with an emphasis on the emerging dynamics of human-AI and multi-agent collaboration. As intelligent systems become active agents capable of autonomous reasoning and strategic cooperation, understanding the dialogic interaction during collaborative problem solving is increasingly important for optimizing and evaluating such partnerships. Our framework addresses key limitations in current analytical approaches through a hierarchical two-layer coding scheme that integrates cognitive and non-cognitive problem solving with metacognitive regulatory mechanisms. We demonstrate its effectiveness and generalizability across nine datasets spanning multiple domains, and provide insights into how humans and agents coordinate their knowledge, skills, and efforts to solve complex problems, showing in particular that metacognitive regulation can be an essential discriminator of deeper collaboration.

• The paper establishes a hierarchical coding scheme that distinguishes between metacognitive, cognitive, and non-cognitive dialogue behaviors in collaborative problem solving.
• It demonstrates that in human-AI interactions, humans predominantly drive self-regulation, while AI agents remain mainly reactive in planning, monitoring, and reflection.
• The robust framework, validated with both human and LLM annotations, positions metacognitive regulation as essential for true collaborative engagement.

Bridging Talk and Thought: Dialogue Dynamics in Collaborative Problem Solving

Conceptual Framework for CPS Dialogue

This paper establishes a rigorous framework for analyzing dialogue in collaborative problem-solving contexts (CPS), emphasizing both human-human and human-AI interactions. The authors delineate the interplay between cognitive (task-related reasoning), non-cognitive (socio-emotional), and metacognitive (regulatory) dimensions, positioning metacognitive regulation as a critical discriminator for genuine collaboration. Their conceptual model foregrounds the necessity for dynamic regulatory behaviors, especially in prolonged, iterative collaborative engagements where AI agents must transition from passive tools to active, strategic partners.

Figure 1: The proposed framework captures dynamic interactions between cognitive, non-cognitive, and metacognitive dimensions during collaborative problem solving.

Hierarchical Dialogue Coding Scheme

The core methodological contribution is a hierarchical two-layer coding scheme for dialogue annotation. The upper layer targets metacognitive processes (planning, monitoring, evaluation) and regulatory behaviors at three levels: self-regulated, co-regulated, and socially shared regulation. The lower layer encodes cognitive and non-cognitive behaviors at the utterance level, enabling granular assessment of dialogic contributions.

Figure 2: The multi-structure coding scheme organizes metacognitive processes and regulatory behaviors above, with cognitive and non-cognitive behaviors at the utterance level.

The coding scheme integrates established CPS models, decomposing regulatory functions into seven sequential processes (task understanding, planning, goal setting, monitoring, strategy use, task execution, reflection). The utterance-level labels span goal-oriented and social utterances, capturing the full range of collaborative exchange. Context-dependent annotation includes off-task behavior, recognizing its potential to modulate engagement and cohesion.

Empirical Analyses Across Human-Human and Human-AI Corpora

The framework is evaluated across nine diverse datasets, including educational, workplace, survival, and game-based collaboration scenarios, featuring both human-human and human-AI partnerships. Human annotation is complemented with LLM-based analysis (GPT-4o), yielding robust inter-rater reliability.

The analysis reveals marked asymmetries in regulatory behavior within structured environments, where metacognitive leadership often falls to one participant. In human-AI contexts, humans overwhelmingly drive self-regulation; AI agents are primarily reactive, rarely initiating higher-order regulatory processes. This is shown quantitatively in regulation process characterization:

Figure 3: Regulation process distributions in human-human vs. human-AI dialogues, highlighting the imbalance and reactive nature of current AI agents.

Balanced collaborative interactions are only observed in contexts with explicit negotiation or task symmetry. The findings confirm metacognitive regulation—particularly the distribution of self-, co-, and socially shared regulated speech—as a necessary determinant of deep collaboration, distinguishing it from mere cooperation or tool-use patterns.

Collaborative Role Characterization

The authors extend their analysis with unsupervised clustering of speaker roles based on annotated contributions across metacognitive, cognitive, and non-cognitive axes. Different task environments yield distinct clustering patterns: highly structured tasks produce rigid role separation; negotiation-driven contexts display more role overlap and symmetry.

Figure 4: Clustering reveals separation and overlap of collaborative roles, contingent on task structure and metacognitive features.

Ablation studies illustrate that removing regulation-type features from clustering dissolves the clarity of role assignment; thus, metacognitive regulation is a key latent variable for collaborative function modeling.

Strong Numerical Results and Contradictory Claims

• Imbalance in Human-AI Regulation: Across all human-AI datasets, the proportion of self-regulated utterances is over 70% human, while AI is nearly absent in planning, monitoring, and reflection stages.
• LLMs vs. Human Annotation: LLM models achieve substantial agreement ($\kappa = 0.69$–$0.75$) with human annotators, validating scalability for large-scale analysis.
• Role Clustering: Inclusion of regulation features produces up to 90% purity in cluster separation for explicit collaborative roles; without them, purity drops below 50%.

The authors challenge prevailing assumptions about human-AI collaboration, demonstrating that current paradigms seldom achieve authentic partnership—AI agents remain predominantly co-regulated and do not share regulatory initiative.

Practical Implications and Future Directions

For system design, these results imply the need for AI agents capable of initiating and participating in metacognitive regulation, not merely executing assigned tasks. Real-time detection of regulatory imbalance could be leveraged to trigger adaptive scaffolding. The framework's generalizability suggests its utility across domains for evaluating collaboration quality and guiding the development of synergetic human-AI systems.

Theoretically, the hierarchical model reframes CPS research, validating metacognitive regulation as a fundamental axis for modeling collaborative depth. As AI agents become more autonomous in dialogic contexts, shared regulatory behaviors must be engineered at the architectural and interaction level. Future work should develop proactive metacognitive support strategies and benchmark genuine collaboration in long-term, multi-session workflows.

Conclusion

This work operationalizes collaboration as the dynamic interaction of metacognitive, cognitive, and non-cognitive behaviors, introducing a scalable, domain-adaptive coding scheme. Empirical analysis confirms metacognitive regulation as the pivotal element distinguishing true collaborative partnership from cooperation or tool use, both in human-human and human-AI dialogue. The findings inform both theoretical models and practical system development, providing a foundational methodology for advancing collaborative intelligence across agents.