Human–LLM directiveness comparison

Determine whether human Project Manager facilitation turns and Llama 3.1 8B Instruct facilitation responses differ in directiveness, using a valid domain-validated measure.

Background

The study initially treats directiveness as an exploratory sixth behavioral dimension rather than as part of the five confirmatory measures. Directiveness is estimated with a DistilBERT classifier trained on 197 synthetic examples, but validation against 100 AMI utterances produced Cohen’s kappa of −0.0231, indicating chance-level agreement.

Because the directiveness measure lacks domain validity, the reported human–LLM comparison cannot support a conclusion about differences in directiveness. The authors therefore identify the issue as unresolved and state that no conclusion about human versus LLM directiveness is warranted. Resolving the problem requires a reliable, domain-validated operationalization of directiveness and a subsequent comparison of the two facilitation sources.

References

Directiveness remains unresolved ($\kappa = -0.0231$). No conclusion about human versus LLM directiveness is warranted.

— Human and LLM Facilitation in Collaborative Design Meetings: A Computational Behavioral Comparison  (2610.13082 - Khan et al., 9 Oct 2026) in Section "Implications for future LLM-facilitation research" in the Discussion