Dynamic Activation Composition (Dyn)

• Dynamic Activation Composition (Dyn) is a neural network technique that combines adaptive activation mixing and dynamic steering to enhance model adaptability and control.
• It utilizes trainable, normalized convex combinations of basis activations and contrastive steering vectors during decoding to optimize expressiveness and property-specific behavior.
• Empirical evaluations show that Dyn improves performance in image classification and language generation tasks through robust multi-property conditioning with minimal fluency loss.

Dynamic Activation Composition (Dyn) refers to a class of neural network interventions in which activation functions or steering interventions are constructed or applied dynamically with trainable coefficients, or with decoding-time adaptive control, in order to enhance model expressiveness, interpretability, or property-specific behavior. Dyn has emerged as a unifying term for two distinct but related methodologies: (1) dynamic learned mixtures of basis activation functions, and (2) dynamic composition of steering directions with adaptive intensities in LLMs.

1. Formal Definition and Motivation

Dynamic Activation Composition denotes two principal mechanisms:

1. Adaptive Activation Mixing: Each layer's nonlinearity is defined as a convex combination of several canonical activation functions, with learnable, normalized mixture weights per layer. Concretely, for base functions $\{f_j(x)\}_{j=1}^K$ and non-negative layer-wise weights $w_j$, the Dyn activation is

$$A(x) = \sum_{j=1}^K P_j\,f_j(x), \quad \text{where}\quad P_j = \frac{w_j}{\sum_{i=1}^K w_i}$$

The network learns both the usual feature weights and these activation mixture coefficients.

2. Dynamic Steering in LLMs: During autoregressive decoding, steering vectors $A_i^{(\mu)}$—computed via contrastive prompt pairs—are injected into attention head outputs with a stepwise scalar weight $\alpha_i^{(\mu)}$ for property $\mu$. The steering intensity is not fixed but set dynamically per token based on information-theoretic contrast between distributions:

$z_i' = z_i + \alpha_i^{(\mu)}A_i^{(\mu)}$

For multi-property steering, Dyn composes:

$$z_i' = z_i + \sum_{\mu=1}^M \alpha_i^{(\mu)}A_i^{(\mu)}$$

The primary motivations are heightened adaptability to input distributions (activation mixing), and robust, minimally disruptive conditioning of model outputs (dynamic steering).

2. Mathematical Frameworks

2.1 Basis Activation Mixtures

Let $K$ candidate base activations, e.g., $\mathrm{ReLU}$, $\tanh$, $\sin$. For each layer $\ell$:

• Parameter vector: $\mathbf w^\ell = (w_1^\ell,\dots,w_K^\ell)$
• Normalized coefficients: $P_j^\ell = w_j^\ell / (\sum_i w_i^\ell)$
• Layer activation: $A^\ell(x) = \sum_{j=1}^K P_j^\ell f_j(x)$

Parameterization strategies:

• $w_j^\ell = \exp(s_j^\ell)$ (unconstrained $s_j^\ell$)
• Initialization: uniform weights or small random noise $w_j^\ell \sim U(0.9, 1.1)$

Joint optimization alternates between network weight updates and activation weight updates, typically using an Adam-based three-phase freezing/unfreezing schedule.

2.2 Dynamic Activation Steering (LLMs)

Activation steering in LLMs operates by adding steering vectors to intermediate activations. For step $i$ and property $\mu$, the steering intensity is inferred via KL-divergence between unsteered and strongly-steered next-token distributions, nucleus-filtered to top-$p$ tokens:

• KL-guided weighting:

$$\alpha_i^{(\mu)} = \min\big\{\mathrm{KL}(\tilde p_i\|\tilde p^{(\mu),\mathrm{strong}}_i), \alpha_{\max}\big\}$$

where $\tilde p_i$ and $\tilde p^{(\mu),\mathrm{strong}}_i$ are renormalized probabilities over the nucleus set $Q_i$.

• Steering vector extraction:

$$A_i^{(\mu)} = v_i^+ - v_i^-, \quad v_i^+ = \frac{1}{K}\sum_{k=1}^K f(P_k^+, y_{<i}), \quad v_i^- = \frac{1}{K}\sum_{k=1}^K f(P_k^-, y_{<i})$$

The procedure enables both single- and multi-property steering.

3. Decoding and Optimization Procedures

3.1 Basis Activation Learning

Training schedule:

• Epochs 1–10: Freeze activation weights $\{w_j\}$; optimize network weights $\Theta$.
• Epochs 11–20: Freeze $\Theta$; optimize activation weights $\{w_j\}$.
• Epochs 21–30: Freeze $\{w_j\}$; optimize $\Theta$. No explicit $\ell_1$ or $\ell_2$ regularization on $w$, beyond non-negativity and normalization constraints.

Pseudocode

for epoch = 1 to 10: freeze activation weights unfreeze network weights train network weights for epoch = 11 to 20: freeze network weights unfreeze activation weights train activation weights for epoch = 21 to 30: freeze activation weights unfreeze network weights train network weights

3.2 Dynamic Steering in LLMs

Dyn decoding pseudocode (high-level):

procedure DynDecode(prompt q, max_length M): y ← [] for i in 1…M: logits = f.forward(q, y; injection=0) p_un = softmax(logits) for μ in 1…M: logits_strongμ = f.forward(q, y; injection=α_max·A_i^(μ)) p_strongμ = softmax(logits_strongμ) Q_i = top_p_tokens(p_un, p_top) ∪ top_p_tokens(p_strongμ, p_top) p̄_unμ = renormalize(p_un over Q_i) p̄_strongμ = renormalize(p_strongμ over Q_i) α_i^(μ) = min(KL(p̄_unμ ‖ p̄_strongμ), α_max) Δ_i = ∑_{μ=1}^M α_i^(μ) · A_i^(μ) logits = f.forward(q, y; injection=Δ_i) p_final = softmax(logits) t_next = argmax p_final y.append(t_next) return y

4. Empirical Evaluations

4.1 Activation Mixing (Image Classification)

Empirical results (MNIST, FashionMNIST, KMNIST) reveal learned layer-wise preference for base activations:

• First layer: $P_1$ (ReLU) $\approx 0.48-0.56$, network exhibits classical ReLU-like behavior.
• Second/Third layer: Increasing reliance on tanh/sin, $P_3$ (sin) becomes dominant at depth, up to $\approx 0.92$.

Behavior by input range: For small $x$, mixtures mimic LeakyReLU; for large $x$, ReLU dominates due to unbounded growth.

4.2 Dynamic Steering in LLMs (Conditioned Generation)

Datasets: Alpaca QA (multi-lingual), BeaverTails (safety), GYAFC/XFORMAL (formality) Model: Mistral-7B-Instruct v0.2 Baselines: Start, Fixed, Dim schedules; In-Context Learning; noICL

Metrics:

• Conditioning strength: property-specific confidence (langdetect, LLama Guard 2 8B, XLM-R classifier)
• Fluency: $\Delta$ perplexity vs. ICL baseline

Results:

• Dyn matches or outperforms static baselines, securing strong multi-property accuracy with minimal perplexity increase.
• For multi-property prompts (“Italian + Unsafe”, “French + Informal”), Dyn delivers robust conditioning and fluency preservation, outperforming fixed and decaying schedules.
• $\alpha$-schedules spike at segment starts, decay as property is established.

5. Constraints, Limitations, and Extensions

Constraints:

• Non-negativity and normalization of mixture weights.
• KL divergence is capped at $\alpha_{\max}$; nucleus filtering constrains contrast computation.

Limitations:

• Activation mixing with only three basis functions omits newer forms (Swish, GELU).
• LLM steering vector extraction hinges on synthetic/MT-generated parallel data; human-labeled corpora may offer refinement.
• Only Mistral-7B was studied; broader architecture validation is pending.
• Conditioning and fluency metrics are automatic and proxy-based; human-grade evaluation is desirable.
• Training schedules for activation mixing introduce complexity.
• Hyperparameters ($p_{\text{top}}$, $\alpha_{\max}$) are coarsely tuned.

Potential Extensions:

• Activation mixing over larger dictionaries (Mish, ELU, SELU).
• End-to-end, joint weight optimization with regularization on mixture entropy.
• Investigate alternative steering vector extraction methods (probing-classifiers, PCA).
• Adaptive, per-property or per-layer steering schedules.

6. Significance and Impact

Dynamic Activation Composition unifies advances in neural function expressiveness (layerwise activation blending) and controlled LLM output conditioning (robust multi-property steering), yielding models capable of task- or property-specific adaptation without manual schedule tuning or loss of output fluency. The Dyn paradigm demonstrates empirical gains—layerwise specialization in image tasks and robust multi-property conditioning in LLMs—using relatively modest architectural or computational changes, with promising prospects for generalization to richer activation sets, steering vectors, and broader model classes.

7. Related Areas and Prospective Research

Dynamic activation mixing and steering are thematically allied with dynamic network parameterization, meta-learning, and neural network interpretability. Research parallels exist in dynamic composition in tree-structured models, wherein meta-networks synthesize composition functions at each parse node (Liu et al., 2017), further highlighting the utility of adaptive mixtures and dynamic, context-sensitive interventions. A plausible implication is that further exploration into both Dyn-style activation mixtures and dynamic steering may yield new directions for universal neural adaptation mechanisms, bridging fine-grained representation, robust output control, and minimal disruption to underlying model fluency.