PolychromaticLM: Multi-Level Heterogeneous Modeling

• PolychromaticLM is a design paradigm that employs explicit heterogeneity at token, segment, and neuron levels to facilitate specialized processing.
• It integrates multiple model outputs via color-coded token attribution and semantic segmentation to combine domain-specific and general-purpose knowledge.
• Experimental evaluations show reduced error rates in ASR and effective multi-LLM consensus, while routing techniques yield efficient activation specialization.

PolychromaticLM is a label used in recent arXiv literature for several non-identical mechanisms that impose explicit differentiation within language-model pipelines. In one usage, it denotes a “coloring” method for combining a general-purpose LLM with a domain-specific LLM by tagging each token with the model that should score it during decoding in specialized-domain ASR (Eitan et al., 2023). In another, it names a next-generation interactive system architecture, recast from LLMartini, in which outputs from multiple LLMs are segmented into semantically aligned units, fused by consensus rules, and displayed with color-coded provenance (Shi et al., 22 Oct 2025). In a third, it refers to a 597,153,888-parameter decoder-only transformer whose PolyGLU feed-forward blocks route each FFN neuron among four activation functions through a differentiable gating mechanism trained with Gumbel-Softmax (Medeiros, 7 Mar 2026).

1. Terminological scope and conceptual motif

Across these works, “polychromatic” does not denote a single standardized model family. Instead, it appears at three different levels of abstraction: token-level language-model selection, segment-level multi-LLM composition, and neuron-level activation routing. The common motif is explicit attribution of heterogeneous computation rather than a monolithic scoring or generation process.

In the specialized-domain ASR formulation, attribution is lexical: each output token $w_t$ receives a color label $c_t$ indicating whether it belongs to the general-purpose or domain-specific vocabulary. In the interactive multi-LLM formulation, attribution is provenance-oriented: response segments are marked according to whether they are consensus content, unique content, or partial overlaps. In the PolyGLU transformer formulation, attribution is internal to the network: each FFN neuron dynamically routes among ReLU, Tanh, SiLU, and GELU. This suggests that the term functions less as a canonical architecture name than as a design pattern for making heterogeneity explicit.

2. Token coloring for specialized-domain language-model combination

In “Combining LLMs For Specialized Domains: A Colorful Approach,” PolychromaticLM is the coloring approach for integrating a domain-specific or secondary LM into a general-purpose LM (Eitan et al., 2023). The formal setup assumes two vocabularies, $V_{\mathrm{gen}}$ and $V_{\mathrm{dom}}$, with each output token assigned a color label

$c_t \in \{G,D\},$

where $c_t=G$ iff $w_t \in V_{\mathrm{gen}}$ and $c_t=D$ iff $w_t \in V_{\mathrm{dom}}$. The joint vocabulary is $V = V_{\mathrm{gen}} \cup V_{\mathrm{dom}}$. The paper notes that the formulation generalizes to $c_t$0 vocabularies, written as $c_t$1, with

$c_t$2

The core probabilistic object is the joint probability of a word sequence $c_t$3 and a color sequence $c_t$4. Under a simple independence assumption on the coloring process, MAP decoding is written as

$c_t$5

where $c_t$6 is the LM selected by the label $c_t$7. The paper also gives the corresponding gating interpolation form

$c_t$8

with

$c_t$9

In practice, the prior $V_{\mathrm{gen}}$0 is fixed to a uniform constant, for example $V_{\mathrm{gen}}$1.

The labeling process has both training-time and inference-time components. The domain LM is trained on a domain-specific corpus such as medical transcripts or factory-inspection recordings. To create the joint lexicon, characters of the general LM form $V_{\mathrm{gen}}$2, while characters of the domain LM are re-colored into a disjoint set $V_{\mathrm{gen}}$3 that maps one-to-one back to $V_{\mathrm{gen}}$4. This prevents general and jargon n-grams from colliding at the character level. At inference time, when a new word begins, the decoder evaluates whether its first character belongs to the general or domain alphabet, locks in the color $V_{\mathrm{gen}}$5, and allows only continuations from the same alphabet for the remainder of that sub-word.

3. Decoding mechanics and ASR results

Exact joint decoding of $V_{\mathrm{gen}}$6 is intractable, so the coloring approach adapts standard CTC beam search by carrying the color assignment along each partial hypothesis (Eitan et al., 2023). At each time step $V_{\mathrm{gen}}$7, the acoustic model produces scores $V_{\mathrm{gen}}$8 for each character $V_{\mathrm{gen}}$9, including a blank. Each partial hypothesis $V_{\mathrm{dom}}$0 stores three probability buckets: $V_{\mathrm{dom}}$1 for ending in blank, $V_{\mathrm{dom}}$2 for ending in non-blank, and $V_{\mathrm{dom}}$3. When the decoder extends a hypothesis by a new character $V_{\mathrm{dom}}$4, it also chooses a color $V_{\mathrm{dom}}$5, multiplying in the acoustic score, the constant color prior, and the appropriate LM probability:

$V_{\mathrm{dom}}$6

Pruning retains the top-$V_{\mathrm{dom}}$7 beams by descending $V_{\mathrm{dom}}$8.

The experiments are ASR evaluations on four datasets: Industrial English, Industrial Thai, Medical 1, and Medical 2. The Industrial English data consist of 2.5 h of mixed general and factory jargon, Industrial Thai of 1 h in the same scenario, Medical 1 of 1.5 h from a 55 h simulated physician–patient corpus, and Medical 2 of 80 min of real doctor–patient dialogues. The general acoustic model is XLSR-wav2vec 2.0 pre-trained and fine-tuned on CommonVoice. The general LM is a 3-gram trained on LibriSpeech, Switchboard, and Fisher; the domain LM is a 3-gram trained on each domain’s transcripts. Metrics are Character Error Rate and Word Error Rate. Hyperparameters include beam width $V_{\mathrm{dom}}$9, language-model weight $c_t \in \{G,D\},$0 and length penalty $c_t \in \{G,D\},$1 over $c_t \in \{G,D\},$2, unknown-word and unk-subword penalties over $c_t \in \{G,D\},$3, and for linear or log-linear baselines also $c_t \in \{G,D\},$4.

Quantitatively, the clearest result is on Industrial English: General LM yields $c_t \in \{G,D\},$5, Jargon LM $c_t \in \{G,D\},$6, Linear interpolation $c_t \in \{G,D\},$7, Log-linear $c_t \in \{G,D\},$8, Bin-estimation $c_t \in \{G,D\},$9, Bayes interpolation $c_t=G$0, and Polychromatic $c_t=G$1. On Medical 1, General yields $c_t=G$2, Jargon $c_t=G$3, Linear $c_t=G$4, and Polychromatic $c_t=G$5. Medical 2 and Industrial Thai are reported to show the same pattern: domain-word errors are reduced dramatically while overall WER and CER are matched or improved. The paper’s limitations are explicit: the uniform prior $c_t=G$6 is a blunt instrument, the method requires an explicit domain lexicon, character-level masking assumes a one-to-one alphabet mapping, and the experiments are restricted to two-way combinations. Proposed extensions include a context-sensitive $c_t=G$7, neural gating $c_t=G$8, integration into end-to-end ASR or LAS training, and applications to MT or text generation by coloring tokens by register or style.

4. Segment-level comparison and composition of multiple LLM outputs

In the LLMartini line of work, the paper explicitly recasts its key mechanisms as building blocks for a next-generation “PolychromaticLM” system (Shi et al., 22 Oct 2025). Here the problem is not specialized-domain decoding but interactive comparison and fusion of multiple LLM responses. The system begins by decomposing each model output into semantically coherent units using a two-tier Chain-of-Thought prompt. Given a user query $c_t=G$9 and model outputs $w_t \in V_{\mathrm{gen}}$0, the segmentation prompt $w_t \in V_{\mathrm{gen}}$1 returns

$w_t \in V_{\mathrm{gen}}$2

where each $w_t \in V_{\mathrm{gen}}$3 is a sentence- or paragraph-level chunk keyed to the recognized task type, such as “Problem-Solving: solution steps” or “Information Retrieval: facts vs. references.”

Alignment is defined over these units. With a similarity function $w_t \in V_{\mathrm{gen}}$4, such as cosine similarity of embeddings or a specialized LLM-based comparator, two units are aligned if

$w_t \in V_{\mathrm{gen}}$5

with typical $w_t \in V_{\mathrm{gen}}$6. The alignment procedure clusters units into $w_t \in V_{\mathrm{gen}}$7, where each cluster contains units from different models that discuss the same semantic point, while singletons represent unique contributions.

Fusion is then governed by a consensus rule. For a cluster $w_t \in V_{\mathrm{gen}}$8 drawn from a subset of models $w_t \in V_{\mathrm{gen}}$9, the method sets $c_t=D$0 as the majority-consensus threshold. If $c_t=D$1, the system produces a merged unit

$c_t=D$2

distilling common content. If $c_t=D$3, the cluster is treated as a diverse alternative and all member units are kept in parallel. The fused document is assembled cluster by cluster, and in practice an “enhancer” LLM pass refines coherence, punctuation, and transitions.

The interface encodes this structure visually. Each model is assigned a unique hue; the examples given are GPT-4 = blue, Claude = green, Gemini = orange, and DeepSeek = purple. Consensus segments receive no highlight, using a white background to signal high reliability. Unique segments are tinted with the source model color at low opacity. Partial overlaps use a split-color border or gradient. For clusters tagged as alternatives, left and right chevrons allow users to toggle among phrasings. Additional controls include hover tooltips showing source model names, click-to-hide with restoration from a “hidden content” sidebar, double-click editing with tracked edits, and a Copy button that collects all visible and edited segments in order.

The quantitative evaluation is Study2, a within-subject comparison against a manual multi-tab baseline with $c_t=D$4. Each participant completed 24 tasks, with 12 using LLMartini and 12 using the baseline. Tasks covered four broad purposes—Content Generation, Content Editing, Information Retrieval, and Problem Solving—crossed with three topic-frequency tiers. In the baseline condition, participants manually switched among DeepSeek, ChatGPT, Doubao, and Gemini. Metrics were Task Completion Time in minutes excluding prompt-thinking and wait time, System Usability Scale, NASA-TLX Cognitive Load, and a post-session 7-point Likert rating for overall usability and willingness to reuse. Average completion time was 2.59 min for LLMartini and 3.40 min for the baseline, with $c_t=D$5. SUS was 75.56 versus 49.72. NASA-TLX was 40.58 versus 59.66, with $c_t=D$6 on Mental, Temporal, Effort, and Frustration and no difference on Performance. Willingness-to-use was 6.17 versus 4.06, with $c_t=D$7. The extracted design tenets are Intent-Aware Model Dispatch, Dual-Mode Comparison vs. Fusion, Semantic Unit Control, Visual Differentiation of Consensus vs. Novelty, and Scaffolded Surprise and Exploration. The recasting identifies a task-classifier plug-in, a pluggable alignment-and-fusion pipeline, a color-engine for multi-model provenance, and an interface that toggles between compare and compose.

5. Decoder-only transformer with PolyGLU activation routing

In “PolyGLU: State-Conditional Activation Routing in Transformer Feed-Forward Networks,” PolychromaticLM is the name of a decoder-only transformer trained with PolyGLU, a drop-in replacement for SwiGLU (Medeiros, 7 Mar 2026). The backbone has 28 layers, $c_t=D$8, $c_t=D$9, Grouped Query Attention with 16 query heads and 8 KV heads of head size 64, context length 4,096, vocabulary size 151,669 using the Qwen3 tokenizer, pre-norm RMSNorm with QK-Norm, and RoPE. Total parameter count is 597,153,888, of which the routing overhead is approximately 1.4 M parameters, or 0.23%.

Standard SwiGLU is written as

$w_t \in V_{\mathrm{dom}}$0

followed by $w_t \in V_{\mathrm{dom}}$1. PolyGLU replaces the single activation with a mixture over $w_t \in V_{\mathrm{dom}}$2 activations, specifically ReLU, Tanh, SiLU, and GELU:

$w_t \in V_{\mathrm{dom}}$3

Each FFN neuron $w_t \in V_{\mathrm{dom}}$4 has a static preference vector $w_t \in V_{\mathrm{dom}}$5, initialized to zero, and there is a per-activation scale vector $w_t \in V_{\mathrm{dom}}$6, initialized to ones. A lightweight MLP consumes the mean-pooled hidden state $w_t \in V_{\mathrm{dom}}$7:

$w_t \in V_{\mathrm{dom}}$8

Routing logits are

$w_t \in V_{\mathrm{dom}}$9

and the soft routing weights use the Gumbel-Softmax relaxation at temperature $V = V_{\mathrm{gen}} \cup V_{\mathrm{dom}}$0:

$V = V_{\mathrm{gen}} \cup V_{\mathrm{dom}}$1

The temperature schedule is linearly annealed as

$V = V_{\mathrm{gen}} \cup V_{\mathrm{dom}}$2

with $V = V_{\mathrm{gen}} \cup V_{\mathrm{dom}}$3 training steps, moving from $V = V_{\mathrm{gen}} \cup V_{\mathrm{dom}}$4 to $V = V_{\mathrm{gen}} \cup V_{\mathrm{dom}}$5.

The principal analytical result is emergent near-deterministic routing. Dynamic routing entropy per neuron is

$V = V_{\mathrm{gen}} \cup V_{\mathrm{dom}}$6

and at convergence the mean dynamic entropy across all layers is

$V = V_{\mathrm{gen}} \cup V_{\mathrm{dom}}$7

which is 0.030% of the maximum $V = V_{\mathrm{gen}} \cup V_{\mathrm{dom}}$8. Most layers collapse to $V = V_{\mathrm{gen}} \cup V_{\mathrm{dom}}$9, but layers 9, 16, and 17 maintain elevated entropy of $c_t$00, $c_t$01, and $c_t$02, respectively; layer 17’s entropy increases during the final phase. Static preferences also show depth-dependent specialization: early layers 0–2 have 35–40% GELU and 15–25% Tanh or SiLU, middle layers 3–14 show gradual SiLU growth, and deep layers 15–27 exhibit 50–65% Tanh dominance.

Pre-training uses approximately 10.24 B tokens with a domain mix of Math 70%, STEM 25%, and Code 5%, annealed to 85/10/5 in the final 20% of steps. Tokenization uses the Qwen3 tokenizer with blocks of approximately 100 M tokens and FlashAttention2 varlen masking. Training runs on a single NVIDIA A100 80 GB GPU with DeepSpeed ZeRO Stage 0, bfloat16, and gradient checkpointing, at approximately 11,800 tokens/s for approximately 12.5 days and approximately \$346. Optimization uses AdamW with $c_t$03, $c_t$04, $c_t$05, and weight decay 0.1 on 2D weights only; learning rate linearly warms up for 2,000 steps to $c_t$06 and then decays cosinely to 0; gradient clipping is max-norm 1.0; effective batch size is 16 sequences $c_t$07 4,096 tokens $c_t$08 8 accumulations $c_t$09 tokens.

The paper also reports robustness to supervised fine-tuning. The SFT setup uses an approximately 347 K-example math dataset for 1 epoch, 13,067 steps, learning rate $c_t$10, 100-step warmup, and $c_t$11 frozen at 0.1. Routing entropy is reported to remain constant at $c_t$12 throughout SFT, and the detailed summary states that static and dynamic entropy remain exactly at $c_t$13, meaning routing preferences $c_t$14 are untouched by the supervised objective. Zero-shot evaluation with the EleutherAI harness gives the following base results: HellaSwag 28.51 acc_norm, ARC-Easy 41.04 acc_norm, ARC-Challenge 22.27 acc_norm, PIQA 58.87 acc_norm, WinoGrande 52.17 acc, and BoolQ 61.13 acc. These correspond to 69%, 63%, 66%, 84%, 89%, and 88% of Qwen3-0.6B, respectively, yielding an aggregate range of 62–89% of Qwen3-0.6B performance despite training on 10 B rather than 36 T tokens. Held-out domain perplexities are Math 3.56, Code 7.08, and STEM 31.93. The implementation note is practical: once routing converges, one may freeze the one-hot decision per neuron at inference time to eliminate the runtime cost of computing all $c_t$15 activations.

6. Comparative interpretation and research directions

The three uses of PolychromaticLM are linked by an emphasis on explicit heterogeneity, but they operate on different objects. The 2023 coloring approach selects among LLMs for token scoring in ASR; the 2025 recasting from LLMartini selects, aligns, and composes semantic units across multiple LLM outputs; the 2026 PolyGLU model routes FFN neurons among activation functions (Eitan et al., 2023, Shi et al., 22 Oct 2025, Medeiros, 7 Mar 2026). A common misconception would be to treat them as interchangeable implementations. They are instead distinct mechanisms at lexical, interface, and architectural levels.

Their respective open problems are also different. For token coloring, the unresolved issues are context-sensitive color priors, OOV jargon, extension beyond character-level masking, and scaling from two-way to many-domain combinations. For the interactive multi-LLM system, the forward-looking components are intent-aware model dispatch, dual-mode comparison versus fusion, semantic unit control, provenance-centric visualization, and exploratory comparison. For PolyGLU, the main practical implication is that a small routing overhead can support depth-dependent specialization and potentially permit frozen one-hot routing at inference. Taken together, these works suggest a broader research direction in which provenance, specialization, and routing are made explicit rather than hidden inside uniform decoding or uniform network computation.