---
title: 'GPT-OSS-Puzzle-88B: Optimized MoE Model'
url: https://www.emergentmind.com/topics/gpt-oss-puzzle-88b
type: topic
---

# GPT-OSS-Puzzle-88B: Optimized MoE Model

to=arxiv_search.search  天天中彩票的  彩神争霸是 code  天天中彩票公众号json
{"query":"gpt-oss-puzzle-88B OR \"Extending Puzzle for Mixture-of-Experts Reasoning Models with Application to GPT-OSS Acceleration\" OR MultiZebraLogic OR AsymPuzl", "max_results": 10, "sort_by": "relevance"}
to=arxiv_search.search  手机天天彩票 code  天天中彩票上json
{"query":"\"Extending Puzzle for Mixture-of-Experts Reasoning Models with Application to GPT-OSS Acceleration\"", "max_results": 5, "sort_by": "relevance"}
to=arxiv_search.search ,最新高清无码专区 code ￣奇米json
{"query":"2511.03553 2109.02797 2512.03466 2602.11937", "max_results": 10, "sort_by": "relevance"}
gpt-oss-puzzle-88B denotes an 88-billion-parameter open-source GPT-style model situated at the intersection of post-training architecture optimization and puzzle-oriented reasoning research. In one usage, it is the deployment-optimized derivative of gpt-oss-120B produced by extending the Puzzle post-training neural architecture search (NAS) framework to mixture-of-experts (MoE) reasoning models [2602.11937]. In another, closely related usage, it names an initiative that scales the “unnatural language” puzzle-solving paradigm—treating mazes, Rubik’s Cube states, and Sudoku grids as plain-text token sequences—to an approximately 88 B-parameter open-source transformer [2109.02797]. Taken together, these usages identify a model family and research program concerned with reducing inference cost while preserving, and in some cases improving, performance on reasoning-heavy workloads and structured puzzle domains.

## 1. Scope and conceptual lineage

The designation combines two distinct senses of “Puzzle.” In the optimization literature, Puzzle is a post-training NAS method that shrinks an existing LLM under hard deployment constraints such as memory, latency, and throughput, without further pre-training [2602.11937]. In puzzle-solving work, by contrast, “puzzle” refers to domains such as mazes, Rubik’s Cube, Sudoku, zebra puzzles, and asymmetric cooperative symbolic tasks, all expressed or evaluated through textual or formal interfaces [2109.02797].

This dual lineage matters because the model’s identity is not exhausted by either strand alone. As an optimized derivative of gpt-oss-120B, gpt-oss-puzzle-88B is a systems artifact: it inherits the parent’s 18-layer base, preserves hidden size and head count, and modifies the MoE and attention subblocks to improve serving efficiency [2602.11937]. As an 88 B-scale puzzle-solving initiative, it is also a methodological vehicle for testing whether large decoder-only transformers can solve sparse-reward combinatorial problems directly from text archives, without search trees, explicit constraint checks, or human-crafted heuristics [2109.02797].

A common misconception is to read the model name as implying only a benchmark specialization in puzzles. The available sources instead describe a broader configuration: the word “Puzzle” refers both to the optimization framework that produced the 88 B derivative and to a family of evaluation environments in which such a model can be studied. This suggests that the model is best understood as a convergence point between inference-efficient reasoning-model design and structured puzzle evaluation.

## 2. Extension of Puzzle to mixture-of-experts reasoning models

In its vanilla form, Puzzle treats each transformer layer as a block with alternative implementations for attention and FFN subblocks, assigns each candidate block a replace-1-block score, solves a mixed-integer program (MIP) to select layerwise replacements under resource constraints, and then heals the heterogeneous student model through a short end-to-end knowledge-distillation pass [2602.11937]. For gpt-oss-puzzle-88B, this framework was extended to MoE LLMs.

The parent gpt-oss-120B uses a 128-expert MoE in each transformer layer. The extension therefore expands each MoE FFN slot into a library of subblocks that keep only $E$ experts out of the original 128, where $E \in \{8,16,32,64,96,128\}$. Each such MoE variant is scored by activation-based replace-1-block mean-squared error, and the resulting expert-count choices are incorporated into Puzzle’s MIP alongside attention alternatives [2602.11937].

The expert-scoring procedure is defined as

$$
\mathrm{score}_i \;=\;\mathbb{E}_{x\sim\mathcal D}\Bigl[\|\,f(x)\;-\;f^{(i)}(x)\|_2^2\Bigr],
$$

where $f$ is a parent-model MoE layer and $f^{(i)}$ is the same MoE with expert $i$ zeroed out. Experts are sorted by $\mathrm{score}_i$ from smallest to largest, and subblocks are then built by keeping the top-$E$ experts. This is the mechanism by which the search process determines which layers tolerate aggressive pruning and which must remain close to the parent [2602.11937].

The resulting student has 88 B parameters, or 73% of the parent’s 120 B. The source describes this as a reduction in per-token FFN FLOPs, weight bandwidth, and memory footprint, which in turn lifts batch sizes and throughput, particularly in the MoE-dominated short-context regime [2602.11937]. The architecture is therefore not a simple uniform downsizing; it is a heterogeneous student selected by constrained post-training search.

## 3. Optimization stack and architectural configuration

The final model is assembled from four complementary optimizations rather than a single compression step [2602.11937].

| Component | Design choice | Stated role |
|---|---|---|
| MoE expert pruning | Keep $E \in \{8,16,32,64,96,128\}$ experts per layer | Reduce FFN FLOPs, weight bandwidth, and memory footprint |
| Selective window attention | Convert 8 of 18 global layers to window attention with $W=8\mathrm K$ | Bound long-context compute and KV-cache cost |
| FP8 KV-cache quantization | Store KV in FP8 with calibrated per-layer scales | Halve KV-cache footprint and unlock faster attention microkernels |
| Post-training RL tuning | Average “High-only” and “Balanced” RL checkpoints | Match high-effort accuracy while retaining medium verbosity |

The layerwise expert counts are explicitly heterogeneous. Early layers remain at 128 experts, while later layers drop as low as 8, with the 18-layer profile given as

$$
128,128,128,96,96,64,64,32,32,16,16,8,8,8,8,8,8,8.
$$

This pattern reflects the MIP’s decision about which layers can be pruned with minimal impact [2602.11937].

Long-context optimization is handled through selective window attention rather than blanket replacement. Standard full attention is described as incurring

$$
\text{Compute}\sim O(L^2),\quad \text{KV-cache}\sim O(L),
$$

where $L$ is sequence length. Sliding-window attention with window size $W \ll L$ reduces these to

$$
\text{Compute}\sim O(L\,W),\quad \text{KV-cache}\sim O(W).
$$

However, converting all global layers to window attention breaks long-range dependencies, so the search procedure enumerates, for each original global-attention layer, a full-attention alternative and a window-attention alternative with $W=8\mathrm K$, scores window variants by their drop on AA-LCR, and jointly optimizes short- and long-context scenarios. The final selection converts exactly 8 of 18 global layers to window attention. The YaRN RoPE scaling factor is also tuned from 32 to 56 to stabilize phase wrapping and boost accuracy at 128 K [2602.11937].

The KV-cache is quantized to FP8 using per-layer scales obtained by max calibration over the LNPT-gpt-oss dataset of high-effort prompts. Each layer’s scale factor is rounded up to the nearest power of two. The stated rationale is to trade negligible rounding noise for the simplicity of bit-shifts while mitigating underflow [2602.11937].

Finally, an RL fine-tuning phase is applied with router and experts frozen, over three reasoning environments: math, code, and general. The reward combines task correctness and length regularization. Two variants are trained: “High-only,” which yields peak accuracy but inflates verbosity, and “Balanced,” which uses a uniform mix of high-, medium-, and low-effort episodes. The final policy is a weight average of these checkpoints and is reported to match the high-effort accuracy while retaining medium verbosity and restoring the effort-length ratio to within 10% of the teacher [2602.11937].

## 4. Efficiency metrics, throughput, and accuracy retention

The model’s performance analysis explicitly distinguishes per-token speed from request-level efficiency. Per-token throughput is defined as

$$
\text{Throughput }T = \frac{\text{tokens generated}}{\text{sec}},
$$

and latency per token as

$$
\text{Latency per token }L=\frac{1}{T}.
$$

On an 8×H100 node, the reported throughput rises from $29.6\,\mathrm K/s$ to $36.1\,\mathrm K/s$ in the $4\mathrm K/4\mathrm K$ setting, for a 1.22× speedup, and from $5.7\,\mathrm K/s$ to $9.3\,\mathrm K/s$ in the $64\mathrm K/64\mathrm K$ setting, for a 1.63× speedup [2602.11937]. On a single NVIDIA H100 GPU, the reported gains are 2.44× in the $4\mathrm K/4\mathrm K$ scenario and 2.82× in the $64\mathrm K/64\mathrm K$ scenario [2602.11937].

The paper stresses that token counts vary with reasoning effort and model variant, so tok/s and ms/token do not necessarily imply end-to-end speedups. A 2× throughput gain can be erased if traces grow 2×, and throughput gains can instead be spent on additional reasoning tokens to improve accuracy [2602.11937]. This is an important corrective to simplistic speed comparisons.

To address that issue, the request rate is defined as

$$
R \;=\;\frac{T}{\overline N},
$$

where $\overline N$ is the average tokens generated per request. Plotting average reasoning accuracy against relative request rate across low-, medium-, and high-effort settings yields an accuracy–speed frontier. Along that frontier, gpt-oss-puzzle-88B is reported to dominate gpt-oss-120B: at high effort on 8×H100 it delivers 2.08× higher normalized request rate while retaining 100.8% of accuracy, and at low effort it achieves 1.29× higher request rate with 108.2% accuracy retention [2602.11937].

Accuracy retention is summarized across eight reasoning benchmarks—MMLU-Pro, GPQA, HLE, AALCR, AIME-25, SciCode, IFBench, and RULER-128K—and three effort levels. The fully quantized student achieves suite-average retention of 100.8% at high effort, 103.9% at medium effort, and 108.2% at low effort [2602.11937]. The source interprets this as showing no quality sacrifice and small gains at medium and low effort.

## 5. Relation to text-native puzzle solving

The earlier “unnatural language” paradigm provides a complementary view of what an 88 B open-source GPT puzzle model might do when applied directly to symbolic problem text rather than architecture search [2109.02797]. In that framework, mazes, Rubik’s Cube states, and Sudoku grids are represented as raw ASCII or token strings. A 5×5 maze is flattened into text, a scrambled cube is encoded as a 54-character face string in URFDBL order followed by move symbols, and a Sudoku instance appears as a token sequence of the form `[WP]81-digit start [RESPONSE]81-digit solution` [2109.02797].

The model is then fine-tuned to translate unsolved token streams into solved ones without search trees, explicit constraint checks, or human-crafted heuristics. The training objective is standard maximum likelihood:

$$
L(\theta)=−\sum_{t=1}^T \log p_\theta(x_t|x_{<t}),
$$

with no reinforcement-learning or policy-gradient terms and no explicit reward shaping. Self-attention remains standard,

$$
\mathrm{Attention}(Q,K,V)=\mathrm{softmax}\!\left(\frac{QK^T}{\sqrt{d_k}}\right)V.
$$

The only modification described is the introduction of special tokens such as `<|startoftext|>`, `[WP]`, `[RESPONSE]`, and `<|endoftext|>` into the byte-pair encoding vocabulary [2109.02797].

The source presents concrete baseline results for smaller GPT-2-scale models. On Rubik’s Cube, using a 601-example test set, 576 outputs are valid but incorrect formulas, 11 are invalid outputs, and 14 are fully solved. On a random sample of 10,000 Sudoku instances, fully solved grids remain at 0%, with many near-complete solutions but frequent row, column, or sub-block errors. On 4×4 and 5×5 mazes, valid maze–solution pairings exceed 98%, and mean solution length is within 1 step of the optimal breadth-first result [2109.02797].

Within that framework, the 88 B “gpt-oss-puzzle-88B” is presented as a scaling hypothesis rather than as the already optimized MoE derivative of the later work. The proposed gains include increasing context length beyond 2,048 tokens, exploiting greater capacity to memorize rare long-distance patterns, introducing lightweight auxiliary heads or constrained decoding to enforce legality, and leveraging in-context learning for new puzzle variants [2109.02797]. This suggests a second research trajectory for the same model name: not only a cheaper reasoning model, but also a larger platform for search-free combinatorial problem solving in plain text.

## 6. Benchmarking environments: zebra logic and asymmetric cooperation

Two later benchmark frameworks give formally specified evaluation environments for an 88 B open-source GPT-style model, but they should not be conflated with published results for gpt-oss-puzzle-88B itself. MultiZebraLogic is a multilingual logical-reasoning benchmark built from zebra puzzles in nine Germanic languages, with datasets of 128 and 1024 puzzles for sizes 2×3 and 4×5 [2511.03553]. The benchmark varies language, theme, puzzle size, 14 real clue types, and up to 8 red-herring types. Its generation loop samples candidate real clues from the ground-truth solution, keeps only those that preserve a unique solution under `python-constraint`, then adds uninformative red herrings and shuffles all clues into final puzzle text [2511.03553].

The formal encoding uses Boolean variables $X_{o,c,v} \in \{0,1\}$, uniqueness constraints
$\forall o, \sum_v X_{o,c,v}=1$ and $\forall c,v, \sum_o X_{o,c,v}=1$, together with positional variables $\mathrm{pos}(o)$ and symbolic encodings for clue types such as `found_at`, `not_at`, `same_object`, `next_to`, and `between` [2511.03553]. Evaluation is defined by puzzle-level accuracy, where the entire solution matrix must match the reference, and cell-wise accuracy, defined as the number of correct cells divided by $O \times N_{\text{attributes}}$ [2511.03553]. The benchmark reports that adding 5 red herrings to 4×5 puzzles drops $A_{\text{puzzle}}$ by $0.15 \pm 0.07$, and it states that scores of o3-mini on 4×5 puzzles are not significantly affected by English versus Danish or by the common houses theme versus the country-specific smørrebrød theme [2511.03553]. Translation guidance prioritizes correctness, unambiguity, naturalness, ease of generation, consistency, and diversity, and extension paths include non-unique attributes, super-attributes, ordinal comparisons, and probabilistic “half-herring” clues [2511.03553].

AsymPuzl supplies a different kind of controlled evaluation: a two-agent asymmetric puzzle environment for studying communication under information asymmetry [2512.03466]. A puzzle state is a sequence $H=[(s^{(1)},c^{(1)}),\dots,(s^{(N)},c^{(N)})]$ over distinct shapes and colors, with each shape and each color appearing exactly once. Alice observes the ordered list of shapes with colors hidden, while Bob observes the unordered set of shape-color pairs with positions hidden. Each maintains a private working hypothesis updated through `replace(position=i, by=(shape, color))` actions, and communication proceeds in strings from the message space $M=\Sigma^*$ until either convergence or a maximum of $T=2N$ turns [2512.03466].

The evaluation metrics are success rate, average turns to solution, and average actions per position, all reported over $M=30$ seeds. Feedback modes include no feedback, own, own detailed, joint, both, and both detailed. Under the “Both” feedback mode on 5-piece puzzles, OSS-120B reaches $90.0\% \pm 8.6\%$ success with Wilson confidence interval $(74.4,96.5)$, while Llama 3.2-11B records $0.0\% \pm 0.0\%$. OSS-120B without feedback solves only 53.3%, but “Own detailed” feedback raises it to 100%, and “Both detailed” yields 96.7%. Communication analysis reports approximately 2.3 edits per position for OSS-120B on $N=5$, against more than 4 edits per position for Llama 11B, with failure modes including ignoring partner messages, over-correction, and miscommunication under “Both detailed” feedback [2512.03466]. The source recommends that an 88 B-parameter GPT-style model be evaluated under the same suite of feedback modes and puzzle sizes, with attention to success rate, edits per position, and turn-wise convergence curves [2512.03466].

Taken together, these frameworks show how gpt-oss-puzzle-88B can be situated within a broader empirical ecosystem. MultiZebraLogic probes multilingual logical deduction with formally encoded constraints and controllable distractors, while AsymPuzl probes cooperative communication under asymmetric observation. In that sense, the model’s significance lies not only in its post-training efficiency gains, but also in the range of structured reasoning regimes to which an 88 B open-source GPT architecture can be systematically subjected.

Source: https://www.emergentmind.com/topics/gpt-oss-puzzle-88b