Shachi: Modular LLM Agent Framework

• Shachi is a methodology that models LLM agents using a decomposed policy comprising Configuration, Memory, Tools, and an LLM reasoning engine.
• The framework standardizes the agent–environment interface to facilitate reproducible, cross-task experiments and rigorous causal analysis.
• It enables controlled experiments through modular design, including backend substitution and nuanced simulation of multi-agent dynamics and economic shocks.

Shachi is a methodology and open-source framework for agent-based modeling with LLM agents that is explicitly designed to be modular, reproducible, and scientifically analyzable rather than ad hoc. Its central move is to treat an LLM agent not as a prompt-engineering artifact but as a formally decomposed policy whose constituent parts can be swapped, ablated, and studied. In the formulation introduced in "Reimagining Agent-based Modeling with LLM Agents via Shachi," the policy is implemented through four components—Configuration, Memory, Tools, and an LLM reasoning engine—and is evaluated within a standardized agent–environment interface intended to support controlled experimentation, cross-task portability, and analysis of emergent collective behavior (Kuroki et al., 26 Sep 2025).

1. Formalization as agent-based modeling with decomposed policy

Shachi models the setting as a partially observable multi-agent decision process. At time step $t$, agent $i$ has an internal state $S_t^i$, interpreted as memory, while the environment has a global state $S_t^E$. The environment emits an observation to each agent according to

$O_t^i = f(S_t^E, i),$

where $f$ tailors what agent $i$ sees. The agent then samples an action from a policy conditioned on its observation, internal state, and configuration: $A_t^i \sim \pi(\cdot \mid O_t^i, S_t^i; C^i),$ where $A_t^i$ is the action of agent $i$ at time $i$0, $i$1 is the current observation, $i$2 is the agent’s internal state or memory, $i$3 is configuration, and $i$4 is the agent policy. The environment collects all actions as

$i$5

and updates according to

$i$6

with $i$7 denoting the environment dynamics (Kuroki et al., 26 Sep 2025).

The methodological emphasis falls on the separation between the agent policy $i$8 and the environment transition $i$9. That decoupling is presented as the basis for agent portability across tasks and for cross-task evaluation. The paper’s motivating diagnosis is that progress in LLM-based multi-agent systems has been slowed by incompatible agent–environment interfaces, inconsistent implementations of memory, tool use, and persona, and weak validation against real-world phenomena. Shachi addresses these issues by standardizing the interface and isolating cognitively meaningful variables that can be manipulated independently.

A further formal distinction is drawn between an action and a tool call. An action is an output from $S_t^i$0 that is passed to $S_t^i$1 and advances the simulation clock. A tool call, by contrast, is an intra-step operation used during deliberation; it returns immediate information to the agent but does not advance the global state from $S_t^i$2 to $S_t^i$3. This distinction prevents tool use from being conflated with environment transitions and permits tool access to function as a controlled independent variable. A plausible implication is that Shachi is designed not merely for task completion, but for causal attribution over architectural ingredients.

2. Cognitive architecture: Configuration, Memory, Tools, and LLM

The Shachi cognition stack implements policy $S_t^i$4 through four components: Configuration, Memory, Tools, and LLM. In the architecture described in the paper, observation $S_t^i$5 enters this stack and action $S_t^i$6 emerges from it, while agent instantiation remains separated from task-specific environment settings such as agent profiles (Kuroki et al., 26 Sep 2025).

Configuration defines the agent’s relatively static identity and constraints. It is the formal variable $S_t^i$7 in $S_t^i$8, and encodes intrinsic traits, roles, incentives, or system-level conditioning. Examples include investment style in StockAgent—Conservative, Aggressive, Balanced, Growth-Oriented—profile or persona in OASIS, strategic type in AuctionArena such as profit-first versus item-first, and pre-announcement tariff news inserted into the prompt in the tariff shock study. The paper notes that configurations can be realized by system prompts or prompting strategies, access restrictions to tools, and potentially weight adaptation such as dynamic LoRA loading. In this sense, Configuration is broader than persona prompting; it is the locus for stable heterogeneity.

Memory is the dynamic state $S_t^i$9. It stores history and supports longitudinal coherence, allowing decisions to depend on prior experiences rather than only current observation. The paper does not prescribe a universal memory-update formula, but operationally memory functions as the internal state in the policy and is updated over time by appending and retrieving past observations, interactions, and outcomes. Implementations mentioned include simple buffers, chat history windows, retrieval-based memory, embedding-based memory, and potentially learnable memory. Experimental instantiations include buffer memory of length 3 in StockAgent, chat-history memory with window 20 and token limit 10,000 in AuctionArena, all-history memory with token limit 16,000 in Sotopia, and chat-history memory with window 5 and token limit 100,000 in OASIS. Memory is also the enabling mechanism for the “carry memory to the next life” experiment, in which agents move between environments without clearing internal state.

Tools extend the agent with external capabilities. A tool is described as having a name, a parameter schema, and a callable function. Tool affordances are included in the observation $S_t^E$0, so the agent perceives not only environmental state but also available instrumented operations. Examples include a forum API for reading stock-market comments in StockAgent, a daily news-retrieval tool in the tariff study, and environment-exposed functions such as comment_to() for inter-agent communication in OASIS. The tool-use loop is iterative: the LLM receives the prompt and tool specifications, may emit tool calls, the framework executes them, returns tool outputs as messages, and repeats until the LLM emits a final answer. The example code allows up to five tool calls.

The LLM is the reasoning engine rather than the whole agent. It takes observation, relevant memory, configuration conditioning, and tool affordances, then generates either a final action or an intermediate tool call. The framework supports proprietary and open-source backends and uses asynchronous invocation so multiple agents can be simulated in parallel. The explicit claim is that scientific analysis requires the policy not to be monolithic: the LLM is only one component in a larger cognitive architecture.

3. Agent–environment interface, communication, and output reliability

Shachi adopts a Gym-like environment loop with reset() and step(). reset() returns initial observations, each agent’s step(observation) returns a response, and env.step(responses) advances the world and returns next observations. The code example shows multiple agents instantiated with model and temperature, asynchronous parallel collection of responses via asyncio.gather, and reward accumulation when relevant. This standardization assigns observations and transitions to environments, while agents are responsible for policy computation (Kuroki et al., 26 Sep 2025).

Within this orchestration, the interaction of components follows a fixed sequence. The environment creates observation $S_t^E$1, which may include world state, messages, available tools, and required output schema. The agent consults memory $S_t^E$2 and configuration $S_t^E$3; the LLM reasons over the combined context; if necessary, it calls tools; after zero or more tool-use iterations, it emits action $S_t^E$4; the environment updates via $S_t^E$5; and the agent updates memory for the next step. This decomposition is what permits controlled experiments in which one can hold the LLM fixed and vary memory, hold all else fixed and vary tool access, or change only configuration.

Inter-agent communication is mediated by the environment rather than by direct calls between agents. The paper specifies a structured message class: $f$3 This design supports dynamic communication graphs, in which agents choose recipients at runtime; static communication graphs, in which the environment constrains valid recipients; and broadcast by setting dst_agent_id = None. Communication thus becomes analyzable as part of the environment rather than an unobserved side channel.

Because LLM outputs can be malformed, Shachi uses a two-stage parsing strategy. First, the main model generates free-form text. Second, a parsing model converts that text into schema-conforming JSON, with instructions to preserve wording if possible and output JSON matching response_type.model_json_schema(). This is followed by schema validation after parsing. The paper presents this as a practical reliability mechanism: it improves action validity without forcing the main LLM into brittle overconstrained formatting.

4. Benchmark structure, reproducibility, and cross-task generalization

Shachi includes a 10-task benchmark organized into three levels of social complexity. The levels are intended to support analysis at increasing degrees of social coupling, from individual cognition to indirect interaction via the environment and then direct communication (Kuroki et al., 26 Sep 2025).

Level	Tasks
Level I: single-agent	PsychoBench; CoMPosT; CognitiveBiases; EmotionBench; EmergentAnalogies
Level II: non-communicative multi-agent	EconAgent; StockAgent; AuctionArena
Level III: communicative multi-agent	OASIS; Sotopia

The task set is heterogeneous in a technically deliberate way. PsychoBench uses 13 psychometric scales; CoMPosT targets caricature susceptibility; CognitiveBiases contains 30 classic cognitive biases; EmotionBench covers 8 emotions under situational prompts; and EmergentAnalogies evaluates zero-shot analogical reasoning. The multi-agent environments span macroeconomic simulation, stock trading simulation, dynamic auction strategy, large-scale social media simulation, and open-ended social interaction.

For reproducibility, the authors reimplemented eight tasks in Shachi and measured fidelity using mean absolute error (MAE) against original results. The reported values are as follows.

Task	Baseline MAE	Shachi MAE
PsychoBench	1.96	0.80
CoMPosT	0.23	0.06
CognitiveBiases	0.24	0.04
EmotionBench	13.82	3.37
EmergentAnalogies	0.64	0.05
StockAgent	9.07	2.63
AuctionArena	10.49	2.22
Sotopia	3.17	0.95

These results are presented as evidence that bespoke systems can be reproduced faithfully within the framework. The paper also reports qualitative replication of stock price trajectories in StockAgent and priority score heatmaps across bidding rounds in AuctionArena, indicating that reproduction is not limited to aggregate metrics but extends to temporal dynamics.

Cross-task generalization was tested by transferring agents between tasks while fixing the underlying LLM to GPT-4o and normalizing scores by in-domain performance. The reported findings are that minimal tasks like EmergentAnalogies can be solved by simple LLM-only agents; richer architectures transfer better to complex tasks; tool-bearing agents generalize best overall; agents without tools underperform when transferred to tool-demanding tasks like StockAgent; and Sotopia was less sensitive to explicit memory differences than expected, perhaps because context window sufficed. This suggests that the Shachi decomposition exposes architectural variables that matter not only for within-task performance but also for transfer behavior.

5. Experimental inquiries enabled by the framework

One class of inquiry concerns longitudinal state transfer. In the “carry memory to the next life” experiment, agents from OASIS and EconAgent were transferred into CognitiveBiases without clearing memory, using GPT-4o-mini. Three runs were conducted, and significance was assessed by paired $S_t^E$6-test with $S_t^E$7. The result was that carry-over memory changed bias scores in systematic ways: OASIS memory increased tendencies consistent with hyperbolic discounting and in-group bias, while EconAgent memory increased endowment effect and reduced loss aversion and survivorship bias (Kuroki et al., 26 Sep 2025). The interpretation offered in the paper is that experiences in one environment can causally shape later cognitive tendencies in another.

A second inquiry places the same agents in multiple environments. In the “living in multiple worlds” experiment, agents alternated between StockAgent and OASIS while carrying internal state across both. The protocol was to observe the stock market and trade, move to OASIS and post or respond on social media, and then repeat the cycle. The settings were Stock A as a 10-year chemical stock, Stock B as a 3-year tech stock, an OASIS topic of Amazon opening physical stores, and 3 independent trials. With OASIS present, stock prices rose less than in the stock-only simulation, yet agent-level indicators suggested stronger interest in tech stock $S_t^E$8. Relative to control, the reported treatment changes were Volume of +10.0% for A and +20.0% for B, #Buys of +8.4% for A and +6.1% for B, and #Sells of +6.6% for A and -8.5% for B. Agents also posted about Amazon stock on social media, and other agents commented on it, showing cross-domain propagation of financial reasoning into discourse. The key methodological takeaway is that agent-level intuitions do not always translate into system-level outcomes.

A third inquiry concerns backend substitution. In EconAgent, the backend LLM was swapped while the environment was held fixed at 100 agents, 240 months, and 20 annual tax or monetary cycles. The evaluation examined macro indicators together with the Phillips Curve and Okun’s Law. All backends qualitatively exhibited both regularities, but details differed: there were intercept shifts in the Phillips Curve, slope shifts in Okun’s Law, GPT-4.1 Nano gave higher unemployment, and GPT-4.1 gave stronger GDP growth. This supports the paper’s claim that backend modularity makes model substitution scientifically cleaner.

6. Tariff shock case study, limitations, and future directions

The paper’s principal external-validity demonstration uses StockAgent to simulate a 5-day trading period, April 1–5, 2025, around a U.S. tariff shock (Kuroki et al., 26 Sep 2025). Four architectural conditions were introduced as a cumulative ablation. The Base condition used standard StockAgent agents with no extra information. Base + Config added a pre-announcement news headline about imminent tariffs to the configuration prompt. Base + Config + Memory additionally supplied memory with a summary of academic research showing tariffs hurt markets. Base + Config + Memory + Tool further added a daily news-retrieval tool with updates on escalating trade tensions. There were five trials per setting, and reported values are means.

The inputs are specified in detail. The configuration input was the headline: “Trump's ‘Liberation Day’ Tariffs Loom; Treasury Yields Fall. President Trump has set a deadline of Wednesday to announce sweeping tariffs.” The memory input consisted of summarized content from Amiti et al. (2021), Trade protection, stock-market returns, and welfare, including findings that tariff announcements reduced U.S. stock prices by $S_t^E$9 cumulatively across 11 events, yields fell due to flight to safety, China-exposed firms had worse outcomes, and estimated U.S. welfare loss was $O_t^i = f(S_t^E, i),$0. The tool input provided daily retrieved news for Apr 2–4 about tariff announcements, market declines, and China’s retaliation. The main metric was the buy-to-sell ratio in the order book, interpreted as market enthusiasm.

Setting	Stock A	Stock B
#1 Base	0.99	0.73
#2 +Config	0.51	0.45
#3 +Memory	0.62	0.59
#4 +Tool	0.44	0.55

The reported changes were $O_t^i = f(S_t^E, i),$1 and $O_t^i = f(S_t^E, i),$2 from #1 to #2 for Stocks A and B respectively, $O_t^i = f(S_t^E, i),$3 and $O_t^i = f(S_t^E, i),$4 from #2 to #3, and $O_t^i = f(S_t^E, i),$5 and $O_t^i = f(S_t^E, i),$6 from #3 to #4. The interpretation given is that adding tariff awareness via configuration caused a broad sell-off, adding academic knowledge in memory moderated the panic, and adding daily news changed relative preference again so that Stock B became less negatively affected than Stock A.

To compare with reality, the simulated stock profiles were mapped to plausible real companies by first using ChatGPT to propose candidate companies and then manually verifying them. Stock A was defined as an established chemical company with a 10-year listing history, declining revenue, stable operations, and a proactive new CEO. Stock B was defined as a recently listed 3-year tech company with high growth potential, questionable data reliability, and past IPO disclosure issues. Real-world candidate returns from Apr 1 to Apr 7, 2025 were DOW $O_t^i = f(S_t^E, i),$7, EMN $O_t^i = f(S_t^E, i),$8, and LYB $O_t^i = f(S_t^E, i),$9 for the chemical-like group, and PLTR $f$0, HOOD $f$1, and PATH $f$2 for the tech-like group. Both classes declined, but the tech-like group generally declined less severely, qualitatively matching setting #4 where Stock B fell less than Stock A. The paper’s main conclusion is that alignment with observed market behavior emerged only when the architecture included the right combination of Config, Memory, and Tools.

The paper is explicit, however, that agent architecture is only one side of realism. Emergent behavior also depends heavily on environmental mechanics; stock-market outcomes, for example, depend on market-clearing rules and not only on agent cognition. Additional cautionary findings include the observation that in Sotopia explicit memory may not have mattered as much as expected, that in the “multiple worlds” study intuitive micro-level behavior did not map straightforwardly to macro-level prices, and that backend changes altered macro patterns even under the same environment. These points delimit the scientific scope of the framework: modular cognitive design supports cleaner experimentation, but does not by itself guarantee realistic simulation.

The paper identifies three core contributions: a structured methodology for LLM-based ABM through formal decomposition of agent policy into LLM, Configs, Memory, and Tools plus a standardized agent–environment interface; a multi-level benchmark suite covering ten tasks across single-agent, non-communicative multi-agent, and communicative multi-agent settings; and new scientific affordances such as memory transfer across environments, agents living in multiple worlds, and external validity tests against real-world economic shocks. Future directions suggested by the authors include adding a more persistent internal state such as a learnable value system or motivational model, moving beyond static prompting toward agents with evolving goals, and expanding to multimodal environments and interactions.