---
title: Dynamic Activation Composition (Dyn)
url: https://www.emergentmind.com/topics/dynamic-activation-composition-dyn
type: topic
---

# Dynamic Activation Composition (Dyn)

Dynamic Activation Composition (Dyn) refers to a class of neural network interventions in which activation functions or steering interventions are constructed or applied dynamically with trainable coefficients, or with decoding-time adaptive control, in order to enhance model expressiveness, interpretability, or property-specific behavior. Dyn has emerged as a unifying term for two distinct but related methodologies: (1) dynamic learned mixtures of basis activation functions, and (2) dynamic composition of steering directions with adaptive intensities in large language models (LLMs).

## 1. Formal Definition and Motivation

Dynamic Activation Composition denotes two principal mechanisms:

1. **Adaptive Activation Mixing:** Each layer's nonlinearity is defined as a convex combination of several canonical activation functions, with learnable, normalized mixture weights per layer. Concretely, for base functions $\{f_j(x)\}_{j=1}^K$ and non-negative layer-wise weights $w_j$, the Dyn activation is
   $$
   A(x) = \sum_{j=1}^K P_j\,f_j(x), \quad \text{where}\quad P_j = \frac{w_j}{\sum_{i=1}^K w_i}
   $$
   The network learns both the usual feature weights and these activation mixture coefficients.

2. **Dynamic Steering in LLMs:** During autoregressive decoding, steering vectors $A_i^{(\mu)}$—computed via contrastive prompt pairs—are injected into attention head outputs with a stepwise scalar weight $\alpha_i^{(\mu)}$ for property $\mu$. The steering intensity is not fixed but set dynamically per token based on information-theoretic contrast between distributions:
   $$
   z_i' = z_i + \alpha_i^{(\mu)}A_i^{(\mu)}
   $$
   For multi-property steering, Dyn composes:
   $$
   z_i' = z_i + \sum_{\mu=1}^M \alpha_i^{(\mu)}A_i^{(\mu)}
   $$

The primary motivations are heightened adaptability to input distributions (activation mixing), and robust, minimally disruptive conditioning of model outputs (dynamic steering).

## 2. Mathematical Frameworks

### 2.1 Basis Activation Mixtures

Let $K$ candidate base activations, e.g., $\mathrm{ReLU}$, $\tanh$, $\sin$. For each layer $\ell$:
* Parameter vector: $\mathbf w^\ell = (w_1^\ell,\dots,w_K^\ell)$
* Normalized coefficients: $P_j^\ell = w_j^\ell / (\sum_i w_i^\ell)$
* Layer activation: $A^\ell(x) = \sum_{j=1}^K P_j^\ell f_j(x)$

Parameterization strategies:
- $w_j^\ell = \exp(s_j^\ell)$ (unconstrained $s_j^\ell$)
- Initialization: uniform weights or small random noise $w_j^\ell \sim U(0.9, 1.1)$

Joint optimization alternates between network weight updates and activation weight updates, typically using an Adam-based three-phase freezing/unfreezing schedule.

### 2.2 Dynamic Activation Steering (LLMs)

Activation steering in LLMs operates by adding steering vectors to intermediate activations. For step $i$ and property $\mu$, the steering intensity is inferred via KL-divergence between unsteered and strongly-steered next-token distributions, nucleus-filtered to top-$p$ tokens:

* **KL-guided weighting:**
  $$
  \alpha_i^{(\mu)} = \min\big\{\mathrm{KL}(\tilde p_i\|\tilde p^{(\mu),\mathrm{strong}}_i), \alpha_{\max}\big\}
  $$
  where $\tilde p_i$ and $\tilde p^{(\mu),\mathrm{strong}}_i$ are renormalized probabilities over the nucleus set $Q_i$.

* **Steering vector extraction:**
  $$
  A_i^{(\mu)} = v_i^+ - v_i^-, \quad v_i^+ = \frac{1}{K}\sum_{k=1}^K f(P_k^+, y_{<i}), \quad v_i^- = \frac{1}{K}\sum_{k=1}^K f(P_k^-, y_{<i})
  $$

The procedure enables both single- and multi-property steering.

## 3. Decoding and Optimization Procedures

### 3.1 Basis Activation Learning

**Training schedule:**
- Epochs 1–10: Freeze activation weights $\{w_j\}$; optimize network weights $\Theta$.
- Epochs 11–20: Freeze $\Theta$; optimize activation weights $\{w_j\}$.
- Epochs 21–30: Freeze $\{w_j\}$; optimize $\Theta$.
No explicit $\ell_1$ or $\ell_2$ regularization on $w$, beyond non-negativity and normalization constraints.

**Pseudocode**
```python
for epoch = 1 to 10:
    freeze activation weights
    unfreeze network weights
    train network weights

for epoch = 11 to 20:
    freeze network weights
    unfreeze activation weights
    train activation weights

for epoch = 21 to 30:
    freeze activation weights
    unfreeze network weights
    train network weights
```

### 3.2 Dynamic Steering in LLMs

**Dyn decoding pseudocode (high-level):**
```python
procedure DynDecode(prompt q, max_length M):
    y ← []
    for i in 1…M:
        logits = f.forward(q, y; injection=0)
        p_un = softmax(logits)
        for μ in 1…M:
            logits_strongμ = f.forward(q, y; injection=α_max·A_i^(μ))
            p_strongμ = softmax(logits_strongμ)
            Q_i = top_p_tokens(p_un, p_top) ∪ top_p_tokens(p_strongμ, p_top)
            p̄_unμ = renormalize(p_un over Q_i)
            p̄_strongμ = renormalize(p_strongμ over Q_i)
            α_i^(μ) = min(KL(p̄_unμ ‖ p̄_strongμ), α_max)
        Δ_i = ∑_{μ=1}^M α_i^(μ) · A_i^(μ)
        logits = f.forward(q, y; injection=Δ_i)
        p_final = softmax(logits)
        t_next = argmax p_final
        y.append(t_next)
    return y
```

## 4. Empirical Evaluations

### 4.1 Activation Mixing (Image Classification)

Empirical results (MNIST, FashionMNIST, KMNIST) reveal learned layer-wise preference for base activations:
- **First layer:** $P_1$ (ReLU) $\approx 0.48-0.56$, network exhibits classical ReLU-like behavior.
- **Second/Third layer:** Increasing reliance on tanh/sin, $P_3$ (sin) becomes dominant at depth, up to $\approx 0.92$.

*Behavior by input range*: For small $x$, mixtures mimic LeakyReLU; for large $x$, ReLU dominates due to unbounded growth.

### 4.2 Dynamic Steering in LLMs (Conditioned Generation)

**Datasets:** Alpaca QA (multi-lingual), BeaverTails (safety), GYAFC/XFORMAL (formality)
**Model:** Mistral-7B-Instruct v0.2
**Baselines:** Start, Fixed, Dim schedules; In-Context Learning; noICL

**Metrics:**
- Conditioning strength: property-specific confidence (langdetect, LLama Guard 2 8B, XLM-R classifier)
- Fluency: $\Delta$ perplexity vs. ICL baseline

**Results:**
- Dyn matches or outperforms static baselines, securing strong multi-property accuracy with minimal perplexity increase.
- For multi-property prompts (“Italian + Unsafe”, “French + Informal”), Dyn delivers robust conditioning and fluency preservation, outperforming fixed and decaying schedules.
- $\alpha$-schedules spike at segment starts, decay as property is established.

## 5. Constraints, Limitations, and Extensions

**Constraints:**
- Non-negativity and normalization of mixture weights.
- KL divergence is capped at $\alpha_{\max}$; nucleus filtering constrains contrast computation.

**Limitations:**
- Activation mixing with only three basis functions omits newer forms (Swish, GELU).
- LLM steering vector extraction hinges on synthetic/MT-generated parallel data; human-labeled corpora may offer refinement.
- Only Mistral-7B was studied; broader architecture validation is pending.
- Conditioning and fluency metrics are automatic and proxy-based; human-grade evaluation is desirable.
- Training schedules for activation mixing introduce complexity.
- Hyperparameters ($p_{\text{top}}$, $\alpha_{\max}$) are coarsely tuned.

**Potential Extensions:**
- Activation mixing over larger dictionaries (Mish, ELU, SELU).
- End-to-end, joint weight optimization with regularization on mixture entropy.
- Investigate alternative steering vector extraction methods (probing-classifiers, PCA).
- Adaptive, per-property or per-layer steering schedules.

## 6. Significance and Impact

Dynamic Activation Composition unifies advances in neural function expressiveness (layerwise activation blending) and controlled LLM output conditioning (robust multi-property steering), yielding models capable of task- or property-specific adaptation without manual schedule tuning or loss of output fluency. The Dyn paradigm demonstrates empirical gains—layerwise specialization in image tasks and robust multi-property conditioning in LLMs—using relatively modest architectural or computational changes, with promising prospects for generalization to richer activation sets, steering vectors, and broader model classes.

## 7. Related Areas and Prospective Research

Dynamic activation mixing and steering are thematically allied with dynamic network parameterization, meta-learning, and neural network interpretability. Research parallels exist in dynamic composition in tree-structured models, wherein meta-networks synthesize composition functions at each parse node [1705.04153], further highlighting the utility of adaptive mixtures and dynamic, context-sensitive interventions. A plausible implication is that further exploration into both Dyn-style activation mixtures and dynamic steering may yield new directions for universal neural adaptation mechanisms, bridging fine-grained representation, robust output control, and minimal disruption to underlying model fluency.

Source: https://www.emergentmind.com/topics/dynamic-activation-composition-dyn