Affinity-Based Reinforcement Learning

• Affinity-based RL is a method that integrates explicit affinity metrics, such as binding strength and task similarity, to steer policy learning through reward shaping and regularization.
• It is applied in diverse fields including molecular and antibody design, multi-objective RL, and resource placement to enhance performance and control in complex tasks.
• Recent algorithmic techniques include reward-based affinity steering, policy regularization with affinity priors, and the use of affinity matrices for efficient task clustering and interpretability.

Affinity-based reinforcement learning (ABRL) refers to a family of methodologies in which the reinforcement learning objective, architecture, or reward function is directly structured around some notion of “affinity.” Here, “affinity” may denote binding strength in molecular design, pairwise task affinity in multi-task RL, semantically or hardware-imposed constraints in resource placement, or domain-specific priors over action distributions reflecting intrinsic or extrinsic behavioral preferences. The ABRL paradigm is characterized by the explicit inclusion of affinity metrics—computed, predicted, or imposed—as algorithmic drivers, integrated via reward shaping, regularization, architectural bias, or clustering of objectives. The approach is pervasive in molecular design tasks (where binding affinity is optimized); multi-task RL (where task affinity informs policy modularity); resource allocation (where hardware and semantic affinity shape mappings); and interpretable RL, where intrinsic affinity priors enable control and explainability.

1. Mathematical Formulations of Affinity in RL

Affinity can enter the RL framework at multiple levels. The three most prevalent are:

• Reward-based affinity steering: The reward $R(s,a)$ directly encodes a scalar affinity, e.g., a predicted negative binding free energy in antibody design $$R = -\mathrm{FreeEnergy}(\text{seq}|\text{antigen})$$ (Vogt et al., 2024), a DTA-predicted $pK_d$ in de novo drug generation (Li et al., 2022), or a functional matching score in graph alignment (Liu et al., 2020).
• Policy regularization toward affinity priors: A regularization term is added to the RL objective, penalizing deviation of the learned policy $\pi_\theta(a|s)$ from a predefined affinity-prior action distribution $\alpha$. For example,

$$J_{\text{affinity}}(\theta) = J_{\mathrm{ext}}(\theta) - \lambda\,\Omega_{\mathrm{affinity}}(\pi_\theta; \alpha)$$

where $\Omega_{\rm affinity}$ is typically an $\ell_2$ or $\mathrm{KL}$ metric between marginal action frequencies and $\alpha$ (Maree et al., 2022, Maree et al., 2022).

• Affinity matrices for multi-objective RL: In multi-task or meta-RL, task-to-task affinity is encoded in a symmetric matrix $A$, whose elements $A_{ij}$ reflect the expected improvement or compatibility between objective pairs. PolicyGradEx constructs $A$ via surrogate post-adaptation performance across sampled objective subsets, enabling clustering via convex relaxation (Zhang et al., 16 Nov 2025).

Affinity may additionally be implemented as hard constraints or action masking based on semantic or hardware requirements, as in semantic-aware edge-agentic placement (Zheng et al., 5 Jan 2026).

2. Domain-Specific Applications

A. Molecular and Antibody Design

Affinity-based RL is the canonical solution for generative biological sequence and molecular design, where the sought property is binding affinity between a candidate ligand/antibody and a target protein. Representative systems include:

• Diffusion + RL for Antibody CDRH3 Design: BetterBodies leverages a VAE to encode amino acids, guides a conditional diffusion policy with offline Q-learning, and employs affinity as a sparse episodic reward via the Absolut! simulator (Vogt et al., 2024). An optional Q-filter further selects high-affinity outputs post-generation.
• Protein Sequence-based RL for Small Molecule Design: Li et al. integrate a SMILES RNN with a Siamese CNN DTA predictor; at each RL episode, the binding score $$R_{\mathrm{aff}}(S) = f_{\rm aff}(f_{\rm seq}(p), f_{\rm smi}(S))$$ is the dominant reward component (Li et al., 2022).
• Graph-based Topological RL: GraphTRL constructs state representations using MWCG and persistent homology features, optimizing an external affinity predictor as primary reward; this yields superior binding scores and diversity against strong baselines (Zhang, 2024).

B. Multi-Objective and Meta RL

• Task Clustering via Affinity Estimation: PolicyGradEx builds a task affinity matrix $A$ using a first-order Taylor-based surrogate for loss improvement upon finite adaptation, partitioning objectives into groups to maximize intra-cluster affinity. This yields substantial efficiency and generalization gains on robotics benchmarks (Zhang et al., 16 Nov 2025).

C. Semantic and Resource Placement

• Affinity-Aware Service Placement: AgentVNE applies LLM-based semantic extraction to impose hard affinity constraints (e.g., node hardware requirements) and augments RL resource graphs accordingly. Affinity-driven resource biasing is coupled to a similarity-based GNN and PPO to optimize mapping of virtual agentic workflows to edge resources under strict affinity/dependency constraints (Zheng et al., 5 Jan 2026).

D. Policy Interpretability and Human-aligned RL

• Affinity-regularized Policy Learning: RL agents are regularized toward interpretable global (state-independent) action affinities, either prototypical (e.g., personality traits in personalized finance) or user-specific. Such regularization ensures solution transparency and enables construction of symbolic Markov surrogates for post hoc explanation (Maree et al., 2022, Maree et al., 2022).

E. Graph Matching with Robustness

• Affinity Regularization for Outlier-Resistant Matching: RGM applies a quadratic regularization of Lawler QAP–style affinity in sequential graph matching RL, penalizing growth beyond estimated inlier set size and enhancing both accuracy and outlier robustness (Liu et al., 2020).

3. Algorithmic Techniques and Architectures

• Offline Q-Learning with Affinity Shaping: In biological sequence design, double Q-learning is used to steer latent diffusion or generative models toward high-affinity end states, employing delayed target networks, behavior cloning, and Q-value filtering (Vogt et al., 2024).
• Policy Gradient with Affinity-Driven Rewards: The REINFORCE algorithm is adapted to use predicted binding affinity and secondary molecular metrics as reward for generative molecular policies (Li et al., 2022).
• Affinity-based RL Regularization: DDPG and related actor-critic methods are augmented with global affinity regularizers computed as MSE or KL divergence over marginal action frequencies (Maree et al., 2022, Maree et al., 2022).
• Surrogate-based Task Affinity Estimation: Meta-policy gradients are linearized around initialization, and adaptation loss is used to construct a pairwise affinity matrix efficiently for clustering in large $n$-objective settings (Zhang et al., 16 Nov 2025).
• LLM-Augmented RL for Constraint Extraction: Pre-trained LLMs parse structured (graph) and unstructured (text) descriptions to infer affinity constraints, which are injected as resource augmentations or action-weighting factors in resource placement RL (Zheng et al., 5 Jan 2026).
• Graph and Topology-Aware RL: GraphTRL’s state design incorporates MWCG and persistent homology to encode both chemical interaction and global shape, with reward directly tied to affinity (Zhang, 2024).

4. Empirical Findings and Quantitative Results

Application Domain	Affinity Mechanism	Main Result Highlights	Reference
Antibody design	Offline Q, diffusion RL	$-128.2\pm0.3$ kcal/mol free energy (BetterBodies-CF)	(Vogt et al., 2024)
Drug design	Policy gradient RL	Up to $47.8\%$ active, improved docking (<$-10$ kcal/mol, CDK20)	(Li et al., 2022)
Graph matching	Quadratic affinity reg.	$60.2\%$ F1 on Pascal VOC, +1-2% F1 over NGM-v2	(Liu et al., 2020)
Task clustering (Meta-RL)	Loss-based affinity	NMI $>0.73$, $+21\%$ absolute success on MT10	(Zhang et al., 16 Nov 2025)
Resource placement	Semantic/hardware affinity	$97\%$ acceptance, $<40\%$ hops vs baselines	(Zheng et al., 5 Jan 2026)
Interpretable policy	Intrinsic affinity reg.	$>90\%$ Markov surrogate fidelity, increased entropy	(Maree et al., 2022)

Empirical evidence indicates that affinity-based RL substantially improves both domain-specific objectives (e.g., affinity, robustness, acceptance rate) and the interpretability or structural fidelity of learned solutions.

5. Interpretability, Regularization, and Symbolic Surrogates

ABRL enables explicit control over agent strategy character via prescribed action affinity priors, often resulting in more interpretable or transparent policies. For example, regularizing the marginal action distribution toward a prior $\alpha$ assures that RL solutions reflect desired personality/prototypical profiles in finance or other domains (Maree et al., 2022). Symbolic Markov models extracted from trained policies can reproduce and explain long-term spending/investment patterns, achieving $>90\%$ fidelity in test environments (Maree et al., 2022). These surrogates facilitate pathway tracing through discretized state-action spaces and confer a degree of white-box verifiability to otherwise black-box RL agents.

6. Limitations, Hyperparameter Sensitivity, and Future Directions

Recognized challenges include the risk of collapsing exploration/diversity under strongly affinity-driven reward shaping (as seen in molecular tasks (Vogt et al., 2024, Li et al., 2022)), difficulties in reward model generalization beyond affinity-training domains, and trade-offs between fidelity and interpretability in symbolic surrogates (Maree et al., 2022). In robustness-oriented scenarios, the calibration of affinity regularization (e.g., Lawler QAP regularizer parameters (Liu et al., 2020)) and the choice of clustering algorithms or surrogate spaces (e.g., first-order accuracy in meta-RL (Zhang et al., 16 Nov 2025)) can materially affect solution quality.

Potential extensions span (i) adaptive affinity tuning or curriculum scheduling, (ii) hybridization with other reward models (e.g., curiosity, diversity, and feasibility constraints), (iii) broader application to domains with cross-modal affinity structures (e.g., robotics, combinatorial design, semantic-aware resource management), and (iv) deeper integration of LLMs for affinity extraction from heterogeneous specifications (Zheng et al., 5 Jan 2026).

A plausible implication is that, across domains, explicit affinity structuring increases not only task objective attainment but also the controllability and explainability of RL policies; the continued evolution of ABRL methodologies is anticipated to further lower deployment barriers in real-world scientific, engineering, and decision-making contexts.