Evaluating Generative Agents with Actions Grounded in Socially Distributed Task Environments using Incognita

Abstract: Effective agency in social environments depends on when an agent seeks knowledge, when it acts, and whether its actions are justified by acquired information. Existing grounded benchmarks provide executable actions, persistent state, and verifiable outcomes, while social simulation environments provide rich interaction among language agents. We study an evaluation setting that combines these requirements. We define socially distributed task environments as interactive environments where task-relevant knowledge is partitioned across role-isolated participants and consequential actions are accessible only through them. Communication serves as exploration over role-partitioned knowledge, while grounded action serves as exploitation over environment state. We introduce Incognita, a Concordia-based framework that separates social interaction from grounded execution. The evaluated agent routes messages to a user or specialist entities; specialists mediate admissible operations; a deterministic sub-environment executes accepted operations over a canonical state; and an offline evaluator scores outcomes with inherited rewards. Incognita-Retail transforms tau-bench retail into a multi-entity environment while preserving final-state reward semantics. We evaluate three generative agent models on 18 tasks stratified by social breadth, with 540 trials. Progress appears in reward and behavior: success rises from 0 percent to 8.9 percent and 17.2 percent, while premature finalization falls from 100 percent to 87 percent and 58 percent. Stronger models elicit more hidden knowledge, contact more entities, and attempt more grounded writes, yet reliability remains low. These findings show that socially distributed task environments expose behavior before reliable success, including knowledge elicitation, source selection, grounded action attempts, and premature completion belief.

• The paper introduces a socially distributed task environment framework that compels generative agents to explore and integrate role-isolated knowledge before executing grounded actions.
• The study employs a POMDP model to separate dialogue-based exploration from validated task execution and measures agent performance using tau reward metrics.
• The evaluation reveals that while later agent models improve in deliberate exploration and action-grounding, challenges remain in achieving accurate belief-to-action alignment.

Evaluating Generative Agents in Socially Distributed Task Environments: An Analysis of Incognita

Formalization of Socially Distributed Task Environments

This work introduces the notion of a socially distributed task environment, wherein task-relevant knowledge is partitioned across role-isolated participants, compelling the agent to actively explore and integrate knowledge from multiple interacting entities before grounded action is admissible or effective. This structure is a significant departure from prior single-agent benchmarks that provide centralized or fully observable contexts for action selection. The communication protocol in such environments is inherently a knowledge-exploration process, while actionable operations are tightly scoped to specialist entities with non-overlapping remits.

Incognita operationalizes this framework by separating the layers of social interaction (agent-to-entity/user dialogue, observation gathering) from grounded execution (task-consequential state changes validated and executed by entities) and relegating reward computation to offline evaluation. This explicit modularization enables the direct measurement of the agent’s exploration policies, exploitation of knowledge, and termination (finalization) beliefs within a single controlled environment.

Figure 1: Incognita framework decomposing a centralized retail task into a role-isolated, socially distributed environment with separate layers for exploration and grounded execution.

Environment and Interaction Protocol Design

The proposed setup is modeled as a POMDP $\mathcal{M}_I$, with observations, actions, transitions, and rewards all structured to reflect the role-isolated and knowledge-distributed nature of the environment. The action space consists of routed messages to user or specialist entities, optionally flagged to finalize the episode. Observations are always mediated by the recipient, who responds based solely on disclosed or inherent role-specific knowledge, ensuring strict partial observability and enforcing exploration over both knowledge holders and knowledge content.

The agent never observes the complete task state; every communicative act is a potentially critical information-seeking move. Only mediated actions—operations validated by specialist entities and accepted by the grounded sub-environment—can alter the canonical state. The delineation between knowledge acquisition, exploitation, and premature finalization is made explicit and observable.

Redistributed Knowledge and Process Measurement

Incognita transforms existing centralized task benchmarks—here, tau-bench retail—by distributing both user intent and system capabilities across multiple entities, further gating initial agent access to all relevant information. Thus, the agent must formulate hypotheses about the distribution of knowledge and which roles/entities to interrogate.

Finalization (termination) is an overt declaration that the agent believes sufficient knowledge has been acquired and actions executed to fulfill the user’s goal, making mismatch between belief and environment state directly measurable.

Incognita’s design admits the following research questions:

1. Exploration adequacy: Does the agent identify and extract all necessary pieces of distributed knowledge?
2. Source selection: Are relevant entities contacted and role-isolated protocols followed?
3. Action grounding: Are system states altered only after sufficient knowledge is gathered?
4. Belief-commitment alignment: Does the agent’s finalization reflect actual task completion in the canonical environment state?

Empirical Evaluation and Behavioral Dynamics

The empirical evaluation utilizes three generations of agent models (gpt-5.2, gpt-5.4, gpt-5.5) on Incognita-Retail, covering 18 tasks stratified by social breadth (the count of distinct specialist entities required per the reference solution)—with 540 total trials. The principal outcome metric is inherited tau reward, capturing strict task-completion conditions from the original centralized benchmark.

Figure 2: Grounded reward attainment as a function of social breadth and model capability, highlighting sparse but affirmative performance gains only in the most recent models and broadening engagement with problem complexity.

Key findings are as follows:

• gpt-5.2: Yields zero reward, with all task attempts ending in premature finalization before meaningful exploration or action is undertaken.
• gpt-5.4/gpt-5.5: Show moderate task success (8.9% and 17.2% reward respectively), with notable reduction in immediate failure modes and increased interaction with the knowledge space. gpt-5.5 obtains nonzero reward across all but the most complex social breadth levels.

Figure 3: (a) Outcome classes showing a shift from universal premature finalization in gpt-5.2 to a meaningful mix of premature, environment-belief, and grounded-action failures in later models; (b) Process metrics capturing disclosure coverage, write activity, entity contact breadth, and non-premature trial proportions as a function of agent capability.

Process metrics reveal the mechanism behind these changes:

• Disclosure Coverage: Stronger models disclose a greater fraction of user-locked knowledge.
• Entity-Contact Breadth: Increases to saturation at intermediate model levels, demonstrating deliberate, breadth-oriented exploration.
• Grounded Write Rate: Grows in line with increased knowledge state confidence.
• Non-premature Finalization: Substantially improved, reflecting more discriminative termination policies.

Yet, overall reward remains low even in the strongest model, confirming persistent inefficacy in belief formation and action-grounding under complex knowledge partitioning.

Practical and Theoretical Implications

The results validate the thesis that reliable social reasoning and collaborative agency cannot be simply inferred from final reward or dialogue fluency alone. Incognita exposes developmental gradients in generative agent architectures: early models display “completion at first opportunity” bias; later models increasingly engage in deliberate hypothesis formulation, multi-entity interrogation, and delayed exploitation, but still struggle with the synthesis and grounding of distributed knowledge for robust action selection.

The framework further provides high-granularity process metrics, enabling the isolation of failure points—such as insufficient knowledge acquisition, incorrect source targeting, and state-belief mismatch at finalization—that are masked in holistic reward-only settings.

These findings have implications for:

• Evaluation methodology: Future work should adopt process-exposing benchmarks like Incognita to track agent improvements beyond final task success and to diagnose bottlenecks in multi-agent, knowledge-distributed settings.
• Model architecture and supervisory signals: There is substantial headroom for innovation in memory mechanisms, attention to role provenance, and confidence calibration for termination decisions.
• Agent training regimes: Direct supervision on belief-alignment, exploration completeness, and role-aware policy structuring is likely required for further gains.

Future Directions

The environment’s strict partitioning of capability and information naturally suggests adaptations for more complex workflows, increased numbers and heterogeneity of entities, temporally extended tasks, and adversarial participants. Further, the offline evaluation protocol could be extended to include partial credit for “critical partial completions," as well as modeling persistent memory across linked episodes.

Agent models could incorporate modular “theory of mind” sub-networks or train using curriculum regimes that explicitly train exploration, source selection, and termination more robustly. Integration of meta-Learning-based belief-update routines, leveraging the environment’s POMDP attributes, is a promising direction for augmenting robustness.

Conclusion

This work substantiates socially distributed task environments as high-fidelity test beds for quantifying and dissecting generative agent performance in interactive, partially observable contexts. Incognita reveals a spectrum of agent behaviors—spanning premature completion, intentional exploration, and actionable belief formation—unobservable in conventional, fully centralized or flat dialogue benchmarks. The observed trajectory of model improvement highlights both incremental gains and persistent structural gaps, mandating continued innovation in architecture, training, and process-aware evaluation to approach robust, collaborative agency in open-world settings.