---
title: Market-Based Post-Training (MPT)
url: https://www.emergentmind.com/topics/market-based-post-training-mpt
type: topic
---

# Market-Based Post-Training (MPT)

Market-Based Post-Training (MPT) denotes post-training procedures in which a pretrained or conventionally trained model is adapted according to feedback generated by a deployment environment, hardware platform, market mechanism, or decentralized allocation process. The term has at least two technically distinct uses. In decentralized multi-agent systems, MPT is a framework in which agents jointly learn task actions and bids that determine participation and collective credit assignment [2610.22322]. In high-frequency trading, MPT refers to adversarial post-training of a limit-order-book generator through free-running interaction with a matching engine and an online critic [2610.25604]. Hardware-specific analog in-memory-computing calibration exhibits an MPT-like deployment adaptation, although its authors characterize it as hardware-aware post-training rather than market-based optimization [2401.09859].

## 1. Conceptual scope and terminology

MPT separates broad capability acquisition from environment-conditioned adaptation. Pre-training or ordinary fine-tuning establishes a model’s general structure, task competence, or numerical representation. Post-training then modifies a comparatively limited set of parameters or policies using signals that arise from the intended deployment context. These signals may include collective task rewards, bids, counterfactual outcomes, hardware currents, activation distributions, ADC saturation behavior, or critic scores over environment-generated trajectories.

The defining feature is therefore not a particular optimizer, architecture, or loss function. It is the use of deployment-relevant feedback after an initial training stage. In the decentralized-agent formulation, MPT jointly learns action behavior and self-orchestration behavior: an agent learns both what to do and when it should try to receive control [2610.22322]. In adversarial LOB simulation, the generator is optimized against rollout-level realism after its autoregressive pre-training, rather than solely against teacher-forced token likelihood [2610.25604].

The term should not be conflated with ordinary post-training quantization, generic reinforcement learning, or economic market optimization. The AIMC study does not formulate buyers, sellers, prices, bids, economic utility, or resource allocation. Its “market” interpretation is conceptual: a trained model is adapted to a particular hardware platform through deployment-specific range optimization [2401.09859]. Conversely, the decentralized-swarm formulation uses bids as an explicit coordination mechanism, but its objective is collective task performance rather than monetary exchange.

A useful distinction is between three forms of deployment conditioning:

| Form | Adaptation target | Deployment signal |
|---|---|---|
| Decentralized orchestration | Agent actions and bids | Task rewards and counterfactual branch returns |
| Environment-grounded generation | Low-rank generator parameters | Critic scores over simulator-mediated rollouts |
| Hardware adaptation | Input and conductance ranges | Activation statistics, currents, and saturation constraints |

Across these forms, MPT is characterized by post-training specialization to conditions that are difficult to represent adequately through the original training objective.

## 2. Decentralized market coordination in agent swarms

The decentralized-agent formulation uses a swarm of $N$ agents with separate parameters. Agent $i$ uses policy $\pi_i$, conditioned on local context or history $h_{i,t}$, to produce both a bid and, when selected, an environment action. At each decision, every agent independently submits a bid $b_i$. The highest bidder acts, ties are broken uniformly at random, and a turn is skipped if every agent bids zero. The implemented bid space is

$$
b_i \in \{0,0.2,0.4,0.6,0.8\}.
$$

There is no learned manager, dispatcher, centralized critic, or central coordinator at execution time. The auction performs decentralized turn allocation. A bid simultaneously determines which agent acts and how much downstream value is passed backward as training credit. A high bid increases the probability of acting but causes the acting agent to pass a larger portion of later value backward to the preceding participant.

The bid-weighted credit mechanism is defined recursively by

$$
v_t=\hat r_t+b_{t+1}v_{t+1},
\qquad
\tilde r_t=(1-b_t)v_t,
\qquad
v_{T+1}=0.
$$

Here, $\hat r_t$ is the immediate environmental reward, $v_t$ is accumulated value available at turn $t$, and $\tilde r_t$ is local training credit for the agent acting at that turn. A reward at turn $t+k$ contributes to the value at turn $t$ with weight

$$
\prod_{j=t+1}^{t+k}b_j.
$$

This mechanism is analogous to a bucket-brigade process: later outcomes can train earlier actions, but the strength of the signal depends on intervening bids. Only action tokens from turns on which an agent acted are used for that agent’s action update. MPT applies REINFORCE with an agent-specific mean-credit baseline:

$$
\nabla_{\theta_i}J =
\mathbb{E}_{t\in W_i}
\left[
(\tilde r_t-\bar r_i)
\nabla_{\theta_i}\log\pi_i(a_t\mid h_{i,t})
\right],
$$

where $W_i$ is the set of turns assigned to agent $i$ and $\bar r_i$ is that agent’s mean training credit over its collected turns.

Bidding is trained separately through counterfactual comparisons. A turn is contested when the runner-up has a nonzero bid. The original branch is compared with an alternative branch generated by restoring the environment and agent states, resampling the selected agent’s bid up to three times, and causing the runner-up to act if the selected bid falls below the runner-up’s bid. Comparisons with equal returns or unsuccessful resampling are discarded. The bidding policy is optimized with a Bradley–Terry preference loss:

$$
\mathcal{L}_{\mathrm{bid}}
=
-\log \sigma
\left(
\log \pi_i(y_w\mid h_{i,t})
-
\log \pi_i(y_l\mid h_{i,t})
\right),
$$

where $y_w$ and $y_l$ are the bid responses associated with the higher- and lower-return branches.

Action and bidding outputs are produced by the same agent model and optimize the same parameter set. The training schedule performs action training on four out of every five iterations and bidding training on every fifth iteration. Training uses centralized trajectory storage, backward credit calculation, and counterfactual rollouts, but execution remains decentralized.

## 3. Applications and empirical behavior of swarm MPT

MPT was evaluated on embodied crafting, robot navigation, and deep research. In embodied crafting, three Qwen3-VL-4B-Instruct agents have role-specific action spaces: Logger gathers wood, Miner digs and mines, and Smith crafts, equips, and smelts. The objective is to obtain an iron pickaxe from an empty inventory in Minecraft through MineStudio. In navigation, three NaVILA-8B agents partition actions among Turner, Mover, and Stopper. In deep research, three gemma-4-E2B-it agents specialize in search, page opening, and in-page lookup, with any agent able to issue the final answer.

The primary baseline is a monolithic post-training system, called “Single,” in which one model is trained over the combined action space and all role demonstrations using conventional REINFORCE. With matched training compute, MPT obtains the following held-out results:

| Task | Monolithic RL | MPT |
|---|---:|---:|
| Embodied crafting | $20.3\pm4.1$ | $41.5\pm5.9$ |
| Robot navigation | $0.52\pm0.14$ | $1.23\pm0.18$ |
| Deep research | $9.0\pm2.1\%$ | $21.0\pm4.1\%$ |

The held-out evaluation contains 32 unseen Minecraft world seeds, 88 R2R-CE validation-unseen episodes, and 50 held-out research questions, with each instance evaluated three times. In the crafting task, MPT also outperforms alternative inference-time coordinators when the same post-trained action policies are frozen: round-robin obtains $23.8\pm4.2$, AgentNet obtains $20.9\pm4.2$, Graph-of-Agents obtains $25.4\pm4.6$, and MPT bidding obtains $41.5\pm5.9$. Relative to Graph-of-Agents, the reported improvement is approximately $63.4\%$.

Ablations separate action learning from bidding learning. With both initial action and initial bidding policies, mean reward is $15.4\pm4.1$. Training bidding alone increases it to $21.5\pm4.7$, training action alone increases it to $27.2\pm4.8$, and training both reaches $41.5\pm5.9$. At approximately $5\times10^{18}$ training FLOPs, MPT reaches about 42 mean reward, compared with about 15 for round-robin and about 31 when bidding participation remains but bid values are replaced with independent samples from $U(0,1)$ for credit assignment. These results indicate that learned bids affect both participation and the informativeness of action-training signals.

Participation evolves during learning. In the Iron Pickaxe task, Logger initially dominates most decisions, while Miner and Smith participate more during later stages. At decision step 120, Logger’s participation falls from approximately 80% before reinforcement learning to 45% afterward; Miner’s share rises from approximately 10% to 40%, and Smith’s from approximately 10% to 15%. The resulting temporal role transitions are learned through bids rather than externally specified.

The learned orchestration also exhibits transfer, scaling, and dropout behavior. Continuing from Iron Pickaxe checkpoints, MPT reaches approximately 35 mean reward on Torch versus 17 for monolithic reinforcement learning and approximately 38 versus 19 on Stone Bricks at roughly $10^{18}$ adaptation FLOPs. Duplicating trained agents without additional training improves pass@3 on crafting milestones: Wooden Pickaxe rises from approximately 78% with three agents to 97% with twelve, Stone Pickaxe from approximately 69% to 94%, and Furnace from approximately 66% to 91%. Under agent dropout probability $p=0.9$, the three-agent population obtains approximately 19 mean reward while the twelve-agent population retains approximately 33.

## 4. Environment-grounded adversarial post-training

A second MPT formulation adapts autoregressive foundation models for high-frequency limit-order-book simulation. The starting model is a 78.5-million-parameter Mamba-3 state-space model pretrained on GOOG LOBSTER data from 2022–2025. It generates LOBSTER-format messages, which are executed by the true matching engine. The generator receives a 500-message context corresponding to 13,000 tokens, represents each message with 26 tokens from a vocabulary of size 2,112, and uses top-$k$ sampling with $k=50$ and syntactic logit masks.

Cross-entropy pre-training evaluates next-message likelihood under historical prefixes. During deployment, however, generated messages are fed back into the matching engine, changing the limit order book and the conditional distribution for subsequent messages. This produces a compounding process in which early sampling errors alter the state and induce later errors. Cross-entropy does not directly optimize long-horizon free-running behavior, spread, depth, volume, imbalance, order-flow, inter-arrival statistics, or book stability.

MPT addresses this mismatch through an online Wasserstein critic. The training loop is:

$$
\theta
\longrightarrow
\text{generated messages}
\longrightarrow
\text{matching engine and LOB state}
\longrightarrow
\text{rollout statistics and critic score}
\longrightarrow
\text{parameter update}.
$$

For each context, the generator produces free-running continuations. These are executed through the matching engine, and the resulting rollout windows are compared with real market streams. The preferred critic observes lossless decoded messages organized into channels for event type, direction, price, size, time difference, and four reference channels. Absolute prices, absolute times, and order IDs are omitted. The critic is an ensemble of spectral-normalized temporal convolutional networks with fixed random projections for ensemble diversity.

For critic ensemble members $D_{\phi_k}$, the generator receives a pessimistic score:

$$
R(\tilde{x})
=
\operatorname{mean}_{k}D_{\phi_k}(\tilde{x})
-
\operatorname{std}_{k}D_{\phi_k}(\tilde{x}).
$$

The generator objective also includes a KL trust-region penalty relative to the pretrained anchor $G_{\theta_0}$:

$$
J(\theta)
=
\mathbb{E}[\widehat R_\theta]
-
\lambda\operatorname{KL}
\left(
G_\theta\Vert G_{\theta_0}
\right),
\qquad
\lambda=0.1.
$$

Because token sampling, categorical decisions, order matching, and book-state transitions are not differentiable in a usable manner, the generator is not updated by backpropagation through the simulator. Instead, the method uses EGGROLL, an evolution-strategy optimizer, over rank-4 LoRA perturbations. For a matrix $W$,

$$
W'=W+\Delta W,
\qquad
\Delta W=BA,
$$

with rank $r=4$. Perturbations are applied to interior projection kernels in the SSM blocks, while the pretrained embeddings, decoder, and core dynamics remain frozen.

The headline configuration uses $G=16$ perturbed candidates, $Q=128$ shared contexts, 2,048 rollouts per step, five critic ensemble members, perturbation scale $\sigma=0.003$, and generator learning rate $5\times10^{-4}$. The critic is retrained online and uses an $R_1$ coefficient $\gamma=10$. The method runs for 30 steps because reward hacking appears after approximately 15–20 steps, and checkpoints are selected offline using WS-21 rather than critic reward.

Across ten seeds, adversarial post-training improves held-out LOB-Bench realism by 17.6% on WS-21 and 17.0% on Mean-$L_1$. The paired 95% confidence intervals are $[-19.7\%,-15.3\%]$ for WS-21 and $[-18.9\%,-15.8\%]$ for Mean-$L_1$. Every seed improves, with per-seed WS-21 improvements ranging from 11.3% to 23.9%.

The normalized position-wise divergence of the pretrained anchor grows by $3.1\times$ from the first to the last 25-message bin over a 500-message continuation. Post-training reduces the growth to approximately $2.3\times$, corresponding to 26% less accumulated excess divergence. Compute-matched continued pre-training produces approximately $+0.5\%$ WS-21 and $-0.4\%$ Mean-$L_1$, with confidence intervals spanning zero. Random perturbation controls worsen by $+5.3\%$ WS-21 and $+7.8\%$ Mean-$L_1$.

The refinement requires under 50 million training tokens and approximately 12 GPU-hours per seed, compared with an approximately 48-billion-token pre-training budget. These results distinguish rollout-grounded post-training from simply adding likelihood-training compute.

## 5. Hardware-conditioned post-training in AIMC

Analog-based in-memory computing accelerators perform vector-matrix multiplication inside memory arrays. In an ideal crossbar,

$$
\mathbf{Y}^{T}=\mathbf{X}^{T}\mathbf{W},
$$

whereas the physical array computes through conductances and voltages:

$$
\mathbf{I}=\mathbf{V}\mathbf{G}.
$$

Device, circuit, conversion, and quantization non-idealities make the physical computation differ from floating-point arithmetic. The AIMC study models programming and read noise, conductance drift, output noise, device variability, imperfect programming, differential mapping, IR drop, nonlinear DAC and ADC behavior, finite resolution, ADC saturation, limited dynamic range, crossbar utilization, and tiling effects [2401.09859].

The conventional response is hardware-aware training, in which a hardware model is included in each forward pass and network weights are optimized for expected hardware errors. This requires substantial hardware-specific information, crossbar-granularity simulation, and a more computationally expensive and hyperparameter-sensitive training process. The proposed alternative retains trained weights and optimizes hardware-facing ranges after training using representative activation data and hardware information.

The first procedure performs per-crossbar-column conductance-range optimization. For each column, it selects an effective maximum conductance $G_{\mathrm{BL,max}}$, potentially below the physical maximum $G_{\max}=25\,\mu\mathrm{S}$. With channel-wise scale $\eta\in(0,1]$,

$$
G_{\mathrm{BL,max}}=\eta G_{\max}.
$$

Representative activations are used to estimate column currents. The effective range is reduced when a high-percentile current is predicted to exceed the ADC saturation current $I_S$. The procedure operates per tile or crossbar and per bit-line or column, requiring nominal conductance bounds, ADC saturation current, a conductance-to-ADC relationship, and calibrated hardware measurements or simulation.

The second procedure performs per-crossbar input or DAC-range optimization. For activation samples $\mathbf X$ and percentile $K$, it sets

$$
X_r=\operatorname{percentile}(\mathbf X,K),
$$

then applies symmetric clipping,

$$
\widetilde X=\operatorname{clip}(X,-X_r,X_r).
$$

Reducing $X_r$ improves effective DAC/ADC resolution when the activation distribution occupies only a small part of the nominal range, but increases clipping. Unlike the conductance procedure, percentile calibration does not require detailed column-current modeling. The paper suggests that input-range recalibration may be repeated as activation statistics or hardware operating conditions change without reprogramming memory devices.

The AIMC experiments use base RoBERTa fine-tuned on eight GLUE tasks and mapped to 512-by-512 PCM crossbars. The hardware model includes 8-bit PWM input encoding, differential conductance mapping, 8-bit ADCs, conductance drift, output noise, and IR drop with 0.35-ohm wire resistance between adjacent cells. Results are averaged over ten hardware/noise instances and reported after one hour of device operation unless otherwise stated.

With weight clipping and I/O quantization during training, post-training range optimization recovers an average of 5.3 percentage points of GLUE score. When input and conductance ranges are learned during hardware-aware training, replacing or supplementing their learning with post-training optimization causes only a 0.1 percentage-point average decrease. Further post-training optimization of learned parameters improves the average score by 0.9 percentage points. The detailed results report an average GLUE score of 82.6 for the configuration combining learned and post-training optimized ranges, although another conclusion reports an increase from 76.3 to 81.6%; this is an internal inconsistency in the supplied paper text.

The method is MPT-like because a fixed trained model is adapted to a particular accelerator instance. Strictly, however, it is hardware-aware post-training calibration or deployment-specific range optimization, not market optimization.

## 6. Objectives, limitations, and distinctions from related methods

MPT systems differ in what they optimize after initial training. Decentralized swarm MPT jointly learns action policies and bidding policies. LOB MPT optimizes low-rank generator perturbations against a rollout-level adversarial critic under a KL trust region. AIMC post-training optimizes input and conductance ranges against proxy hardware objectives such as percentile coverage, current control, ADC saturation avoidance, effective resolution, and signal-to-noise ratio.

These objectives are not interchangeable. The swarm framework does not use a fixed task-level differentiable objective for bidding; it obtains preference labels from counterfactual branch returns. The LOB framework does not use token-level REINFORCE or token-level credit assignment; EGGROLL estimates an update in parameter space from scalar rollout scores. The AIMC framework does not directly optimize a task-loss objective over all hardware-facing parameters; it uses calibration and saturation proxies.

MPT also differs from ordinary post-training quantization. Standard post-training quantization generally maps trained weights or activations to fixed numerical precision and calibrates tensor-level scales, zero points, clipping thresholds, or activation statistics. AIMC post-training instead optimizes analog operating ranges linked to conductance programming, DAC/ADC utilization, column-level current accumulation, and ADC saturation.

Several limitations recur across the formulations. Environment-grounded methods depend on reliable feedback. Sparse, noisy, subjective, or difficult-to-verify rewards can impair both action and bidding learning. Counterfactual bidding requires restoring environment and agent states and is difficult for irreversible real-world actions. In adversarial LOB post-training, an online critic can be exploited; reward hacking begins after approximately 15–20 steps, requiring short runs, offline checkpoint selection, held-out evaluation, KL regularization, critic ensembles, and external metrics.

Hardware-conditioned post-training remains dependent on the fidelity of the hardware model. The AIMC study evaluates PCM assumptions, one accelerator configuration, RoBERTa, and GLUE tasks, without physical RoBERTa inference. It does not provide a measured wall-clock cost for the post-training procedures, a precise calibration-sample count for input-range optimization, memory overhead in bytes, a direct tensor-wise PTQ comparison, or a complete convergence curve.

The decentralized-swarm study is limited by its reliance on externally evaluated task rewards, counterfactual rollouts, contested turns with unequal returns, shared auction infrastructure, and the tested scale. It does not evaluate substantially larger models, much larger populations, or heterogeneous populations containing agents above 10 billion parameters. Duplicated agents at inference time are not equivalent to training a genuinely diverse large swarm.

The LOB study establishes improvements for GOOG LOBSTER data and a 28-day held-out panel, but does not establish universal performance across assets, exchanges, market regimes, or much longer horizons. It does not directly optimize trading profitability or market impact in the headline experiment. Its critic can remain blind to some joint book properties, including spread degradation, and checkpoint selection remains dependent on external evaluation.

Taken together, these studies support a broad interpretation of MPT as environment-conditioned post-training. The common principle is to reserve a later optimization stage for behaviors, interactions, hardware effects, and deployment constraints that are not adequately captured by the initial training objective. The term “market-based” is literal for bid-mediated decentralized orchestration, operational for market-simulator feedback, and metaphorical for AIMC hardware adaptation.

Source: https://www.emergentmind.com/topics/market-based-post-training-mpt