Operant Conditioning Benchmark

• Operant conditioning benchmark is a family of evaluation frameworks that tests an agent's ability to adapt its behavior based on trial–outcome contingencies within a few trials.
• These benchmarks employ diverse methodologies—from embodied Virtual Skinner Box tests to synthetic scheduling—focusing on credit assignment amid non-stationary and distracting inputs.
• They use precise metrics such as trials-to-criterion, sample efficiency, and first-try accuracy to advance adaptive reinforcement learning research.

An operant conditioning benchmark is an evaluation framework in which an agent is assessed by how its behavior changes under action-contingent consequences rather than by static prediction alone. In current arXiv usage, the term denotes several related but non-identical constructs: a proposed “Virtual Skinner Box Test Framework” for embodied agents that must learn a new behavior policy in fewer than ~10 trials (Park et al., 2022), a “simple benchmark for linear on-policy control” in which only a very small subset of input signals are relevant and the majority are non-stationary noise (Javed et al., 22 Jul 2025), and a task battery of simple discrimination, changing contingencies, and conditional discrimination for adaptive AGI systems (Johansson, 2024). Related work describes mathematically characterized synthetic environments, return-conditioned offline RL evaluations, cognitively grounded LLM protocols, and naturalistic animal-learning paradigms that can be reused as benchmark designs (Darmasubramanian et al., 24 Sep 2025, Nguyen et al., 2022, Qin et al., 9 Apr 2026, Nandi et al., 2 Jul 2026). This suggests that “operant conditioning benchmark” is best understood as a family of benchmarking principles centered on contingency learning, few-trial adaptation, and policy change under reinforcement or punishment.

1. Conceptual definition and scope

In the biological framing adopted by HICA, operant conditioning is learning a new behavior policy based on trial–outcome contingencies and doing so in a few trials, rather than through millions of steps of model-free reinforcement learning (Park et al., 2022). In the NARS-based “Machine Psychology” framework, the same idea is formalized through the Skinnerian three-term contingency: behavior occurs in a stimulus context and is altered by the consequence that follows it (Johansson, 2024). In the Swift-Sarsa benchmark, the distinction from classical conditioning is explicit: in classical conditioning the subject only learns to predict rewards, whereas in operant conditioning the subject’s actions influence the rate of rewards (Javed et al., 22 Jul 2025).

Across these formulations, the common object of measurement is not merely whether an agent can represent reward, but whether it can alter action selection in response to reinforcement or punishment. HICA states this as learning a new mapping from sensory state to action; Swift-Sarsa instantiates it as a stimulus–response mapping embedded in high-dimensional distractors; Machine Psychology encodes it as executable operations whose consequences are revised through feedback; and the LLM-oriented analysis in ImplicitMemBench argues that a genuinely operant benchmark must replace simple cue–outcome pairings with explicit action–outcome contingencies (Qin et al., 9 Apr 2026).

A second commonality is the emphasis on adaptation rather than exploration. The Swift-Sarsa benchmark is constructed so that even a random policy will occasionally reveal the structure; the challenge is assigning credit to the relevant features and ignoring the noisy ones (Javed et al., 22 Jul 2025). HICA similarly treats the core problem as few-trial policy change supported by predictive memory, reward-modulated plasticity, and internal simulation rather than brute-force search (Park et al., 2022). In this sense, operant conditioning benchmarks isolate a narrower and more specific capability than general reinforcement learning benchmarks.

2. Major benchmark families

The current literature contains several benchmark families that instantiate operant conditioning at different levels of abstraction.

Family	Core task structure	Typical readouts
Virtual Skinner Box	Phase 1 basic behavior pretraining, then Phase 2 novel contingencies such as lever + light	Trials-to-criterion, sample efficiency, robustness, transfer
Swift-Sarsa operant conditioning benchmark	Binary stimuli and binary actions with delayed reward, plus massive non-stationary distractors	Lifetime average reward
Machine Psychology task battery	Simple discrimination, changing contingencies, conditional discrimination	% correct per 12-trial block, truth-value frequency and confidence
Synthetic operant scheduling environment	Optimize ad timing by minimizing operant-conditioning loss $L(t)$ for fixed $n$	Reward gap, loss gap, timing distance
LLM operant design derived from ImplicitMemBench	Learning–Interfere–Test with action–outcome episodes and first-attempt scoring	First-Try Accuracy, subset analyses
Naturalistic dog sociability paradigm	Repeated positive or threatening cue exposures across 5 days plus Day 6 generalization	Approach proportion, approach latency, demeanor

The Virtual Skinner Box Test Framework is the most explicit embodied benchmark proposal. In Phase 1 – Pretraining (Instinct / Basic Behavior), an embodied agent such as a four-legged robot learns locomotion, picking up blue balls for energy, and avoiding predators using traditional RL. In Phase 2 – Operant Conditioning, new contingencies are introduced, such as a lever + light mechanism in which pulling the lever when the light is on yields food and pulling it when the light is off yields shock; the benchmark condition is that the agent learns the new behavior policy in fewer than ~10 trials (Park et al., 2022).

The Swift-Sarsa operant conditioning benchmark is much narrower and more diagnostic. It is a family of linear on-policy control problems in which the optimal policy is linear, only the first $m$ of $n$ observation components matter, and the remaining $n-m$ features are noisy distractors drawn from a non-stationary distribution (Javed et al., 22 Jul 2025). Its purpose is to stress-test credit assignment and robustness to irrelevant features rather than planning complexity.

The Machine Psychology benchmark family is closer to canonical operant experiments. It comprises simple discrimination, changing contingencies, and conditional discrimination tasks, each with baseline, training, and testing phases (Johansson, 2024). These tasks can be read as prototypes for architecture-agnostic AGI benchmarks because they specify stimuli, responses, consequences, and phase structure in a standardized way.

The synthetic scheduling formulation of operant conditioning takes a different route. “Ads that Stick” defines a continuous-time environment in which the central optimization object for fixed $n$ is the operant-conditioning loss

$L(t) = \sum_{j<i} \delta^{\,t_i - t_j},$

and argues that this is a natural foundation for an operant conditioning benchmark in algorithmic research (Darmasubramanian et al., 24 Sep 2025). Here the benchmark is analytically structured rather than behaviorally naturalistic.

ImplicitMemBench does not itself instantiate operant conditioning, but it provides a detailed guide for converting its Learning/Priming–Interfere–Test protocol into an operant benchmark for LLMs by replacing CS–US associations with explicit state, action, outcome episodes (Qin et al., 9 Apr 2026). The free-ranging dog study contributes yet another form: a benchmark paradigm for sociability learning under repeated positive versus threatening human cues, with explicit measurement of generalization to an unfamiliar experimenter (Nandi et al., 2 Jul 2026).

3. Task structure and formalization

Despite their diversity, operant conditioning benchmarks tend to share a small number of recurring structural motifs.

The first is phase separation. HICA uses a two-phase benchmark in which costly RL-based acquisition of “instinct-like” skills is confined to pretraining, and few-shot policy adaptation is isolated in a later operant-conditioning phase (Park et al., 2022). Machine Psychology uses Baseline, Training, and Testing blocks, where baseline and test omit feedback and training includes it (Johansson, 2024). ImplicitMemBench recommends Learning – Interfere – Test, with no explicit reminder at test time and scoring based on the first attempt (Qin et al., 9 Apr 2026). The dog sociability paradigm repeats the same contingency across Days 1–5 and then runs Day 6A with an unfamiliar experimenter followed by Day 6B with the familiar one (Nandi et al., 2 Jul 2026).

The second is explicit contingency encoding. In the Swift-Sarsa benchmark, observations are binary vectors $x_t \in \{0,1\}^n$ and actions are binary vectors $a_t \in \{0,1\}^d$. At special time steps, exactly one of the first $m$ observation components is 1, and reward is delivered after a delay $n$0 if the agent chooses the corresponding sparse action; on all other time steps, reward is 0 (Javed et al., 22 Jul 2025). In Machine Psychology, the contingency is encoded symbolically as temporal/procedural implications such as $m$9 with positive and negative feedback revising the implication’s truth value (Johansson, 2024).

The third is delayed or accumulated consequence structure. The Swift-Sarsa benchmark includes delayed reward by design (Javed et al., 22 Jul 2025). HICA makes delayed consequence handling central to the architecture itself: the hippocampus functions as a loop-structured simulator whose preplay and replay guide action selection, while reward-modulated learning rates selectively consolidate sequences associated with significant outcomes (Park et al., 2022). In the scheduling setting, delayed interaction appears in the pairwise temporal penalty $n$1, where dense schedules create future negative effects that decay exponentially over time (Darmasubramanian et al., 24 Sep 2025).

The fourth is generalization under interference or distractors. Swift-Sarsa uses tens of thousands of noisy, drifting features so that the operative difficulty is credit assignment under distractors rather than sparse exploration (Javed et al., 22 Jul 2025). ImplicitMemBench inserts unrelated turns between learning and test to force behavioral adaptation beyond short-term echoing (Qin et al., 9 Apr 2026). The dog study tests whether repeated positive or threatening cues generalize from a familiar to an unfamiliar human (Nandi et al., 2 Jul 2026).

These formalizations differ in representation—symbolic implications, binary vectors, continuous-time schedules, or conversational trajectories—but they all operationalize operant conditioning as a change in action selection driven by action-contingent consequences.

4. Evaluation criteria and metrics

A distinctive feature of operant conditioning benchmarks is that they often privilege adaptation metrics over raw asymptotic reward.

In HICA’s proposed Virtual Skinner Box, the named performance metrics are Trials-to-criterion, Sample efficiency, Robustness, and Transfer (Park et al., 2022). These choices reflect the paper’s central contrast between biological operant conditioning and sample-inefficient modern RL. In the Swift-Sarsa benchmark, performance is assessed by lifetime average reward,

$n$2

over a fixed lifetime $n$3 time steps (Javed et al., 22 Jul 2025).

Machine Psychology combines external and internal measures. Externally, it uses % correct per block, with blocks of 12 trials. Internally, it tracks NARS truth-value components

$n$4

so that changes in learned implications can be inspected directly rather than inferred only from behavior (Johansson, 2024). This makes the benchmark simultaneously behavioral and mechanistic.

The synthetic scheduling framework uses exact optimization metrics. For fixed $n$5, the benchmark compares schedules through $n$6 or total reward

$n$7

and proposes metrics such as relative performance gap,

$n$8

together with timing distance $n$9 (Darmasubramanian et al., 24 Sep 2025). Because the optimum is analytically characterized and approximable with exponentially small error, this family supports unusually precise benchmarking.

For LLM-style implicit-to-operant adaptations, the core metric is First-Try Accuracy (FTA),

$m$0

supplemented by subset analyses such as inhibition versus preference tasks (Qin et al., 9 Apr 2026). The same paper uses Priming Influence Score (PIS) for priming tasks, but for operant benchmark design its main contribution is the emphasis on first-attempt policy evaluation after interference.

The naturalistic dog paradigm uses three complementary readouts: approach proportion, approach latency, and demeanor (Nandi et al., 2 Jul 2026). Approach proportion is a group-level fraction, latency is analyzed with censoring, and demeanor is coded into Affiliative, Neutral, Anxious, and Aggressive categories. This combination demonstrates that operant conditioning can be benchmarked not only through correctness or reward but also through speed and qualitative behavioral style.

Offline RL work on return-conditioned behavioral cloning contributes a different reliability criterion: one should examine achieved return versus target return curves and ask whether performance degrades when the conditioning signal becomes out-of-distribution (Nguyen et al., 2022). This effectively treats the desired-return input as an operant-conditioning signal and turns robustness to high OOD conditioning into a benchmark property.

5. Representative systems and empirical findings

The literature contains both benchmark proposals and actual benchmark results.

HICA is primarily a candidate architecture for solving a few-trial operant conditioning benchmark rather than a completed benchmark result. Its core claim is that a heterarchical network of Modulated Heterarchical Prediction Memory (mHPM) modules, combined with hippocampal simulation and reward-modulated local learning, is explicitly designed to support few-shot policy change and would be expected to perform well on the proposed Virtual Skinner Box (Park et al., 2022). The paper does not provide full numerical experiments on that benchmark.

By contrast, Swift-Sarsa reports direct results on its operant conditioning benchmark. The environment sizes include $m$1 and $m$2 binary inputs with $m$3 relevant observation components and $m$4 action components mapped to four discrete actions (Javed et al., 22 Jul 2025). The approximate optimal lifetime reward is reported as

$m$5

Swift-Sarsa is reported to learn to assign credit to the relevant signals without prior knowledge of which features matter, to retain good performance across a wide range of meta-step-sizes and initial step sizes, and to benefit substantially from step-size decay, especially when initial step sizes are too large (Javed et al., 22 Jul 2025).

Machine Psychology reports clear acquisition, reversal, and conditional-discrimination performance. In simple discrimination, NARS reaches 100% correct by the 2nd block of training and maintains 100% correct across all test blocks. In changing contingencies, it adapts to reversal, reaching 75% correct in the last retraining block and 91.7% correct in the final test. In conditional discrimination, it exceeds 75% correct by the second block and reaches 100% correct during testing (Johansson, 2024). These results position the benchmark as a test of stable contingency learning, reversal flexibility, and conditional control.

The synthetic ad-scheduling formulation provides a different kind of finding: with fixed $m$6, the optimal schedule depends only on the operant-conditioning function $m$7, and the paper gives a quasi-linear-time algorithm that computes a near-optimal schedule with exponentially small timing error (Darmasubramanian et al., 24 Sep 2025). Empirically, the optimized schedule outperforms uniform spacing, corner, and random heuristics across representative regimes, while revealing that uniform spacing is competitive only when $m$8 is small and becomes suboptimal as long-memory interaction increases (Darmasubramanian et al., 24 Sep 2025).

ImplicitMemBench reports broad limitations in implicit behavioral adaptation for LLMs. Across 17 models, no model exceeds 66% overall; the top performers are DeepSeek-R1 (65.3%), Qwen3-32B (64.1%), and GPT-5 (63.0%) (Qin et al., 9 Apr 2026). Of particular relevance to operant-style design is the reported asymmetry between inhibition and preference: 17.6% vs. 75.0%. The paper interprets this as evidence that current models are much better at selecting a positively marked option than at suppressing a previously punished or default option, a property directly relevant to punishment-based operant benchmarks.

The offline RL study “Reliable Conditioning of Behavioral Cloning for Offline Reinforcement Learning” diagnoses a reliability problem in return-conditioned policies: achieved return can increase with target return up to the dataset maximum and then sharply degrade for higher, OOD targets (Nguyen et al., 2022). Its proposed ConserWeightive Behavioral Cloning (CWBC) uses trajectory weighting and conservative regularization to improve reliability, and it reports substantial gains for both RvS and Decision Transformer variants across D4RL, Atari, and AntMaze benchmarks (Nguyen et al., 2022). Although framed as offline RL, this work effectively treats return conditioning as an operant-style benchmark dimension.

The dog sociability study offers naturalistic benchmark evidence for asymmetric generalization under positive and threatening cues. Dogs exposed to positive cues show increased approach behavior, reduced approach latency over time, and increased affiliative demeanor, whereas dogs exposed to threatening cues show reduced approach behavior, increased approach latency, and a shift toward neutral and less affiliative responses (Nandi et al., 2 Jul 2026). Positive experiences partially generalize to an unfamiliar experimenter, while threatening experiences mainly generalize as heightened caution rather than reduced approach probability. This yields a rare field-based benchmark for operant learning in a human–animal ecology.

6. Limitations, controversies, and future directions

A recurrent limitation is the gap between benchmark proposal and benchmark standardization. HICA explicitly proposes a benchmark and a candidate architecture, but the paper acknowledges no large-scale quantitative evaluation yet, the absence of full Virtual Skinner Box experiments, and the use of manually programmed subcortical modules rather than learned ones (Park et al., 2022). This leaves open whether the proposed benchmark will function primarily as a neuroscience-inspired explanatory task or as a reproducible engineering test.

Another limitation is narrow diagnostic scope. The Swift-Sarsa benchmark is intentionally simple in the sense that the optimal policy is linear and exploration is not the main challenge (Javed et al., 22 Jul 2025). Its strength is isolation of credit assignment under massive distractors, but that also means it does not address complex planning or expressive function approximation. The authors themselves note that a more thorough evaluation on a wider range of control problems is needed (Javed et al., 22 Jul 2025).

The mathematically explicit scheduling benchmark has a different set of assumptions: stationarity, homogeneous ads, no user context, and deterministic rewards (Darmasubramanian et al., 24 Sep 2025). Those assumptions make the environment analytically tractable, but they limit its direct use for contextual or non-stationary decision systems. The paper therefore suggests extensions involving non-stationary or seasonal rewards, competition between advertisers, user context / personalization, and learning the model online (Darmasubramanian et al., 24 Sep 2025).

For LLMs, ImplicitMemBench argues that to be truly operant, a benchmark must make consequences contingent on actions, include explicit reward or punishment signals, support delayed outcomes and repeated decision cycles, and move beyond single textual cues to multi-dimensional state contingencies (Qin et al., 9 Apr 2026). Its reported results also caution against assuming that explicit memory modules solve the problem: memory-augmented agents with explicit storage and retrieval do not reliably improve implicit memory (Qin et al., 9 Apr 2026). A plausible implication is that operant conditioning benchmarks for LLM agents will need to test policy change, not merely retrieval fidelity.

The naturalistic dog paradigm is ecologically rich but not fully controlled. The paper notes uncontrolled prior human experience, imperfect full individual tracking, the fact that the threatening cue involves a stick display without physical punishment, and the use of only one unfamiliar experimenter for generalization (Nandi et al., 2 Jul 2026). These are limitations for causal inference, but they also show what an ecologically valid operant benchmark must confront: asymmetry between positive and negative generalization, individual variation, and trade-offs between risk and reward.

Machine Psychology points toward broader future benchmark suites involving variable ratio, variable interval, extinction, partial reinforcement, and more advanced conditional or relational tasks (Johansson, 2024). Taken together with the other benchmark families, this suggests that a mature operant conditioning benchmark ecosystem will likely require several layers: analytically solvable synthetic tasks, high-dimensional credit-assignment diagnostics, few-trial embodied adaptation tests, conversational first-attempt evaluations, and naturalistic generalization paradigms.