QUACK: Questioning, Understanding, and Auditing Communicated Knowledge in Multimodal Social Deduction Agents

Abstract: Social deduction games have become a popular testbed for probing reasoning, deception, coordination, and belief modeling in LLM agents. However, most environments are scored only by game outcomes such as win rates and largely remain to text-only interaction, making it difficult to tell whether an agent's language is actually grounded in what it perceived and did, or to identify the failure modes underlying its behavior. To address this gap, we introduce QUACK, an open-source environment and evaluation framework for auditing the grounding of agent language in multimodal social reasoning. QUACK evaluates agents at three levels: game outcomes, behavioral trajectories, and utterance-level consistency. Its core Statement Verification Pipeline reconstructs each agent's ground-truth trajectory from engine logs and checks every discussion claim against it, automatically flagging spatial hallucination, unsupported accusation, deception collapse, and language-action inconsistency. Evaluating three frontier VLMs in both homogeneous and cross-model adversarial settings, we find that even the strongest agent hallucinates 15.1% of its verifiable spatial claims and makes over half of its accusations without grounded evidence. We release the full engine, evaluation framework, toolkit, and logs at https://github.com/AAAAA-Academia-Attractions/QUACK.

• The paper proposes QUACK, a novel framework integrating outcome, behavioral, and utterance-grounded analysis to audit social deduction agents.
• It leverages a replayable environment with fine-grained logs to verify each claim against agents’ perceptual data, achieving ≈99.5% precision and ≈98.7% recall.
• Experimental findings reveal substantial grounding failures, including spatial hallucinations, unsupported accusations, and language-action inconsistencies even among high-performing agents.

Auditing Grounded Reasoning in Multimodal Social Deduction Agents: An Analysis of QUACK

Introduction and Motivation

LLMs and Vision-LLMs (VLMs) are increasingly deployed as interactive agents in partially observable, multi-agent environments. In such settings, the fidelity of an agent’s utterances with respect to its perceptions and actions—termed linguistic grounding—is critical but often inadequately measured. Existing social deduction benchmarks typically reduce agent evaluation to outcome metrics such as win rate or accuracy, with little insight into the provenance, groundedness, or consistency of agent-produced language.

The QUACK framework addresses this deficiency by operationalizing a comprehensive environment for evaluating multimodal agents in social deduction tasks. Importantly, QUACK offers: (1) a controlled, partially observable environment with replayable, fine-grained logs; (2) a three-tier evaluation protocol encompassing outcome, behavioral, and utterance-grounded analysis; and (3) an automated Statement Verification Pipeline which audits every utterance against the agent’s ground-truth perceptual and behavioral trajectory.

QUACK Environment Overview

The QUACK environment takes inspiration from social deduction games such as Goose Goose Duck and Among Us, but advances the state-of-the-art for research by supporting visual and structured textual observations, configurable graph-structured maps, hidden roles (Geese and Ducks), and adversarial incentives.

Agents interact in discrete time steps (ticks), operating under partial observability. Each agent, powered by a frontier VLM, receives two visual modalities:

• A global map showing spatial layout and own task locations (with no visibility into other players).
• A local observation consisting of players and bodies in the current room, plus structured event summaries.

Agents either perform actions (movement, tasks, kills, meetings) or communicate via free-form natural language during discussion phases. The engine produces deterministic, serialized logs, enabling faithful replay and analysis of both behavior and dialogue.

Figure 1: Omniscient and agent-level multimodal observations, illustrating the information separation and partial observability central to QUACK.

Multitiered Evaluation Framework

Tier 1: Game Outcomes

The baseline metrics capture standard social deduction outcomes: team win rates, ejection accuracy, survival, and task completion. While these supply necessary context, the authors emphasize that raw performance cannot distinguish fluent but ungrounded language production from genuinely reasoned deduction.

Tier 2: Behavioral Trajectories

QUACK reconstructs detailed agent trajectories, enabling trajectory-based metrics such as voting and task efficiency, post-kill displacement, kill rate, and reporting latency. These supplement outcomes with agent-level behavioral diagnostics, but do not yet expose failures of language groundedness.

Tier 3: Statement Verification Pipeline

The core innovation is an automated pipeline for extracting and verifying all checkable claims made during agent discussions. Utterances are parsed by an LLM into structured claim types (location, sighting, activity, accusation, defense), and each claim is cross-validated against the agent’s logged ground-truth trajectory.

Failures are partitioned into four operational categories:

• Spatial hallucination: Contradictions between claimed agent locations/observations and their true trajectory.
• Unsupported accusation: Accusations for which the accuser lacks observable evidence, disambiguated from mere outcome accuracy.
• Deception collapse: Ducks’ (impostors’) use of easily falsifiable lies as opposed to subtle, plausible deception.
• Language-action inconsistency: Stated activities or routes at odds with the agent’s actual actions.

This automated audit is highly reliable (≈99.5% precision, ≈98.7% recall), as verified by human annotation, and directly exposes inconsistencies that are undetectable by final-game metrics.

Experimental Findings

The authors benchmark three modern VLMs—GPT-5.5, Gemini-3.1-Pro, and Claude-Opus-4.7—across 270 games in both homogeneous teams and cross-model adversarial setups. Despite strong game-level outcomes (Geese win rates up to 93.3%, task completion near 60%), the utterance-level audit reveals systematic and model-agnostic grounding failures.

• Spatial Hallucination: Even the strongest agents hallucinate up to 20.8% of all spatial claims. These include false assertions about player presence, room transitions, and witnessed activities, reflecting failure to maintain accurate belief under long-horizon, partial observation.
• Unsupported Accusations: Over half (53.5%) of all accusations are ungrounded, i.e., made without any observable, reconstructable evidence, regardless of their correctness post-facto.
• Deception Collapse: Ducks produce outright falsehoods in approximately 22.1% of their verifiable claims. Critically, deception sophistication is negligible—nearly all lies are grossly falsifiable by log replay—demonstrating that current VLMs lack nuanced adversarial reasoning.
• Language-Action Inconsistency: Agents produce numerous statements about tasks, movements, or routes that are contradicted by their own recorded actions, with these inconsistencies prevalent even among winning teams.

Notably, when Ducks tell verifiable lies in meetings, Geese fail to leverage this information effectively: on average, only 75.2% of lying Ducks are correctly ejected in the subsequent vote, with even lower detection in some configurations. This points to a multi-agent failure mode where both language production and language interpretation remain insufficiently grounded.

Implications and Future Directions

QUACK demonstrates that surface-level metrics in social deduction tasks (win rate, accuracy) are highly insufficient as proxies for robust, grounded language use and long-horizon belief tracking. Systematic grounding failures pervade even state-of-the-art VLMs, and adversarial, multi-agent pressure tends to exacerbate error rates.

Practically, this calls into question the deployment of current VLM-based agents in any safety- or reliability-critical setting where linguistic fidelity to perception and action is paramount. Theoretically, QUACK establishes social deduction games as a uniquely diagnostic substrate, uniquely coupling adversarial incentives, recoverable world states, and a high density of verifiable claims. This should catalyze future research along several axes:

• Developing architectures, training objectives, or reinforcement protocols that explicitly optimize for language groundedness and belief consistency over long horizons.
• Richer adversarial environments—including multiple impostors, larger teams, or external perturbations—to analyze error scaling and team-based fault tolerance.
• Stronger and more nuanced claim extraction and verification pipelines, potentially incorporating uncertainty quantification or implicit reasoning.
• Cross-modal ablations to tease apart the contributions of vision, language memory, and symbolic reasoning to grounding performance.

Conclusion

The QUACK framework provides a rigorous, open-source foundation for auditing and diagnosing grounded language use in multimodal social deduction agents. The authors’ evaluation demonstrates substantial, quantifiable, and model-agnostic grounding failures, which conventional outcome metrics completely obscure. By operationalizing utterance-level claim verification and publicizing the environment, toolkit, and logs, QUACK enables the research community to systematically measure, compare, and improve the fidelity of agent communication in partially observed, adversarial settings. Addressing the identified failure modes remains a central open challenge for the continued deployment and development of trustworthy AI agents.