Adaptive Linear Feature Setpoint (LFS)

• Adaptive Linear Feature Setpoint (LFS) is a framework that dynamically determines optimal affine projections in feature spaces to guide both symbolic regression and LLM alignment.
• It employs unsynchronized LCF nodes with gradient-based or hybrid weight updates in symbolic regression and LQR feedback control for semantic steering in LLMs.
• Empirical evaluations reveal significant performance gains in regression fit and language model safety, supported by rigorous theoretical guarantees on tracking errors.

Adaptive Linear Feature Setpoint (LFS) is a principle and methodology that enables the dynamic determination and pursuit of optimal affine projections in feature spaces for learning systems. The framework is employed in both symbolic regression (via linear-combination-of-features nodes) and LLMs for closed-loop behavioral alignment. Adaptive LFS specifies, at each relevant decision point, the target “setpoint” along a data-driven linear feature combination. This target is updated adaptively as learning or inference unfolds, providing a semantically interpretable control axis for model behavior. Recent advances rigorously cast Adaptive LFS within model-based control theory for LLMs and evolutionary approaches for symbolic regression, with substantive empirical and theoretical development (Žegklitz et al., 2017, Skifstad et al., 21 Apr 2026).

1. Conceptual Foundations and Formal Definition

The Adaptive Linear Feature Setpoint paradigm is instantiated as an explicit mechanism to “set” or track an optimal affine transformation in the feature space. In symbolic regression, this corresponds to nodes realizing $f_{lcf}(\mathbf{x}) = w_0 + \sum_{i=1}^d w_i x_i$, where weights $w_i$ and bias $w_0$ are tuned to project the input $\mathbf{x}$ along problem-adaptive directions (Žegklitz et al., 2017). In LLM alignment, LFS generalizes to an adaptive semantic setpoint $s_k$ at each transformer layer $k$, derived from local linearizations of model dynamics and grounded in user-specified concept directions (Skifstad et al., 21 Apr 2026).

The general principle is to extract, at each inference or optimization step, target values in low-dimensional, interpretable subspaces—either as fixed affine projections or dynamically updated semantic axes—then optimize the system to remain close to those setpoints.

2. Implementation in Symbolic Regression

Within symbolic regression, Adaptative LFS is concretely realized through the introduction of linear-combination-of-features (LCF) nodes in genetic programming (GP). Each LCF node computes an affine transformation $f_{lcf}(\mathbf{x})$ as above. LCFs can operate in three modes:

• Unsynchronized (U): Each LCF leaf maintains independent weights, allowing maximal flexibility.
• Synchronized (S): LCFs are grouped by index within an individual, forcing all in the same group to share weights—gradients or mutations are aggregated.
• Globally Synchronized (G): Index-based weight sharing extends across the entire population.

Weight evolution uses either (a) stochastic mutation (Gaussian perturbation), (b) gradient-based backpropagation calculating derivatives $\partial E/\partial w$ through the GP tree, or (c) hybrid updates (few backprop steps plus rare mutation) (Žegklitz et al., 2017). Experimental results demonstrate that only the unsynchronized mode with backpropagation (UB) or hybrid updates (UC) consistently and significantly outperforms the baseline—especially for problems requiring oblique projections (rotated sigmoid), where UB/UC achieves $R^2 \approx 1$. Mutational methods or aggressive synchronization confer no reliable benefit and may conflict with learning dynamics.

3. Model-Based Control and LFS in LLM Activation Steering

In LLMs, Adaptive LFS formalizes the semantic steering of hidden activations via linear time-varying (LTV) control. The autoregressive transformer block at layer $k$ evolves via

$w_i$0

where $w_i$1 is the activation, $w_i$2 is nonlinear, and $w_i$3 is the additive control input. Local linearization about a nominal trajectory yields

$w_i$4

with $w_i$5 the layerwise Jacobian and $w_i$6. Steering toward a setpoint $w_i$7 is posed as an LQR tracking problem in this linearized model, with cost

$w_i$8

subject to the linearized dynamics. Feedback gain matrices $w_i$9 are computed by standard Riccati recursions. The closed-loop control law,

$w_0$0

requires only per-layer Jacobians precomputed at a nominal activation, and the online cost is dominated by matrix–vector products (Skifstad et al., 21 Apr 2026).

4. Adaptive Semantic Feature Setpoint Determination

A core component is the adaptive computation of semantic setpoints $w_0$1 for each layer. Given a contrastive dataset $w_0$2 reflecting the target concept (e.g., non-toxic vs. toxic language), the procedure is:

• Compute mean activations $w_0$3 and $w_0$4.
• Form the direction $w_0$5.
• At inference, compute $w_0$6 (current feature strength), set the desired strength $w_0$7 for hyperparameter $w_0$8.
• The minimal setpoint shift is $w_0$9.

This provides a closed-form, adaptive, and interpretable target (the setpoint) for the feedback law at each layer, requiring only one hyperparameter per concept (Skifstad et al., 21 Apr 2026).

5. Theoretical Performance Guarantees

The tracking error when using LQR with linearized model A_k subject to nonlinear dynamics is analytically bounded. Under a C² assumption and local Lipschitz bound $\mathbf{x}$0, the deviation satisfies

$\mathbf{x}$1

where $\mathbf{x}$2 is a closed-loop transition and $\mathbf{x}$3 is the trajectory mismatch. The error in semantic feature tracking $\mathbf{x}$4 is similarly bounded. These results provide formal guarantees that, provided sufficient contraction ($\mathbf{x}$5), the steering error remains controlled in the presence of local nonlinearity (Skifstad et al., 21 Apr 2026).

6. Empirical Evaluation Across Domains

Experimental analysis substantiates Adaptive LFS benefits:

• Symbolic Regression (MGGP): On rotated and unrotated toy benchmarks, as well as nine real datasets, unsynchronized LCFs with gradient-based tuning achieved statistically significant R² gains over baseline in 5 out of 9 test problems, with no degradation in most others (Žegklitz et al., 2017).
• LLM Alignment: Adaptive LFS via LQR feedback achieves robust, fine-grained behavioral control for arbitrary concepts, semantic safety (toxicity reduction by 20–50× compared to 8–10× for baselines), truthfulness (10–20% improvement in informativeness), and resistance to refusal/jailbreaking, with minimal compute overhead (<5% perplexity increase, and ≈10–20% token-generation slowdowns on modern GPUs) (Skifstad et al., 21 Apr 2026).

7. Practical Use and Recommendations

Adaptive LFS is most beneficial when the data or objective exhibits structure that can be captured by affine projections (e.g., rotated features, semantic axes). In symbolic regression, this warrants using unsynchronized LCF nodes with gradient-based or hybrid weight evolution. For LLM alignment, Adaptive LFS with offline-computed LQR feedback controllers, steering to adaptively derived semantic setpoints, provides a performant, theoretically justified alternative to open-loop or non-anticipative steering. Global synchronization or mutation-only updates are discouraged except in highly constrained circumstances. For low-dimensional, smooth tasks, Adaptive LFS dramatically reduces model complexity. For tasks with high noise or requiring explicit division, benefits may not manifest (Žegklitz et al., 2017, Skifstad et al., 21 Apr 2026).