ICM-CA: Curiosity & Cross-Attention in RL

• The paper introduces ICM-CA by integrating intrinsic curiosity with cross-attention to enhance exploration and improve training efficiency in reinforcement learning.
• It leverages a forward dynamics model where intrinsic rewards stem from prediction error adjustments and compression progress, mitigating noisy learning signals.
• Empirical results show up to 3× faster convergence and reduced information leakage in multi-hop split learning, emphasizing practical trade-offs in design.

Searching arXiv for the cited papers and closely related terms to ground the article in current arXiv records. Intrinsic Curiosity Module and Cross-Attention (ICM-CA) denotes a family of reinforcement-learning designs that combine an Intrinsic Curiosity Module (ICM) with a cross-attention mechanism. In the supplied literature, the term is used explicitly for a soft actor-critic framework for deceptive signal-assisted private multi-hop split learning (Wei et al., 9 Jul 2025). Its conceptual basis is older: curiosity is formalized as reward for compression progress rather than raw novelty (0812.4360); attention is later inserted into actor–critic and ICM pipelines as a probabilistic weighting mechanism, although not as transformer-style cross-attention (Reizinger et al., 2019); and more recent work proposes a cross-attention-based realization aligned with in-context learning, while proving sharp limits on what update-free predictive interfaces can estimate in general temporal settings (Elmoznino et al., 17 Jun 2026).

1. Conceptual lineage and scope

The central idea behind ICM-CA is the conjunction of two components. The first is curiosity-driven exploration, where intrinsic reward supplements sparse or insufficient extrinsic feedback. The second is attention over historical context, used to emphasize state-action information judged relevant for control or prediction. In the most direct use of the term, a centralized agent employs an ICM module to encourage exploration of novel actions and states, and a CA module to determine the importance of each historical state-action pair, thereby improving training efficiency (Wei et al., 9 Jul 2025).

The curiosity component inherits a longstanding distinction between prediction error and learning progress. The compression-progress formulation prescribes intrinsic reward when a continually improving predictor or compressor achieves measurable compression progress on the agent’s growing data history. Actions should be chosen to maximize expected future compression progress, not raw novelty. The same source explicitly warns that rewarding prediction error itself is inappropriate, because it makes agents seek randomness or regions where the model cannot improve (0812.4360).

Later ICM formulations operationalize curiosity through a forward dynamics model in latent feature space. Attention-based extensions then weight features or loss terms so that exploration becomes selective rather than uniform. However, the 2019 attention paper explicitly states that it does not introduce cross-attention or transformer-style attention; its attention modules are learned probability distributions over features or loss components, not query-key-value cross-attention blocks (Reizinger et al., 2019). This distinction is essential for interpreting the label “ICM-CA”: in the literature here, it refers either to a specific SAC-based architecture with genuine cross-attention (Wei et al., 9 Jul 2025) or to a proposed cross-attention design for ICL-driven curiosity (Elmoznino et al., 17 Jun 2026), not to all attention-augmented ICM variants.

2. Formal foundations of curiosity

The compression-progress account defines an agent with history $h(\le t)$ of inputs $x$, actions $y$, and rewards $r$, together with a compressor or predictor $p(t)$ evaluated by a computable metric $C(p(t), h(\le t))$. Beauty is identified with compressibility by setting subjective beauty to the negative description length, $B_t(D) = -L_t(D)$. Interestingness is the first derivative of subjective beauty, so in discrete time

$$I_t(D) = B_t(D) - B_{t-1}(D) = L_{t-1}(D) - L_t(D).$$

The canonical intrinsic reward is therefore compression progress on the same data before and after learning:

$r_t^{int} = L_{t-1}(h(\le t)) - L_t(h(\le t)).$

The RL controller should maximize the expected sum of future intrinsic rewards,

$$u(t) = E_{\mu} \left[ \sum_{\tau=t+1}^{T} r_{\tau}^{int} \,\big|\, h(\le t) \right].$$

This framework rejects Boltzmann/Shannon surprise as a sufficient curiosity signal: both constant darkness and white noise are boring because neither yields further compression progress (0812.4360).

By contrast, standard ICM-style formulations instantiate intrinsic reward from forward-model prediction error. In the attention-based ICM description, the forward loss is

$x$0

and that forward error is utilized as an intrinsic reward (Reizinger et al., 2019). In the MHSL-specific ICM-CA framework, the intrinsic reward is likewise the forward-model feature prediction error,

$x$1

This establishes a persistent theoretical tension within the literature: ICM-CA may be implemented with instantaneous forward error, yet the compression-progress theory implies that an ideal curiosity signal should track improvement in prediction or compression rather than error magnitude itself (Wei et al., 9 Jul 2025).

3. Attention inside and around the Intrinsic Curiosity Module

The attention-based curiosity literature inserts attention at two distinct loci. In the actor–critic pathway, separate attention modules are used so that policy and value estimation can emphasize different parts of the feature space:

$x$2

Inside ICM, attention is applied to the inputs of the forward and inverse dynamics models. In the single-attention variant, attention operates on concatenated inputs; in the double-attention variant, features and actions are weighted separately before concatenation. The paper also proposes “rational curiosity,” where the forward loss is reweighted by an attention mechanism controlled by the next-state feature, yielding

$x$3

The stated aim is to encourage curiosity only where it is useful for the task, rather than uniformly across the state space (Reizinger et al., 2019).

A common misconception is to equate these attention modules with cross-attention. The paper explicitly disallows that reading: there is no explicit $x$4 definition, no multi-head formulation, and no queries from one modality attending to keys and values from another. Attention is instead described as a probabilistic weighting mechanism that emphasizes features or loss contributions (Reizinger et al., 2019). In this sense, the 2019 methods are attention-augmented ICM architectures, but not ICM-CA in the stricter cross-attentional sense.

The stricter notion appears when cross-attention is used to query historical state-action memory. In the MHSL framework, the CA module augments the actor’s input with a relevance-weighted summary of the last $x$5 state-action pairs:

$x$6

where $x$7 and

$x$8

Here the actor uses $x$9, while the critic uses the current state $y$0 (Wei et al., 9 Jul 2025).

4. In-context learning, Bayesian information gain, and cross-attention designs

A more recent line of work reformulates curiosity around an in-context learner (ICL) that emits exact Bayesian posterior predictives:

$y$1

Within this Bayes-Adaptive MDP formalization, the preferred intrinsic objective is Bayesian information gain (BIG),

$y$2

Three ICL-derived rewards are analyzed: surprisal $y$3, description-length reduction $y$4, and the sum-of-future-improvements reward

$y$5

The main theoretical result is negative for general BAMDPs: for every reward implementable as a finite function of ICL posterior-predictive likelihoods on conditioning subsets of the actual trajectory, there exists a BAMDP in which the reward is not an unbiased estimator of BIG. Surprisal carries an additive aleatoric-entropy term, and $y$6 carries an abductive term and a residual term. Only a double limit involving unbounded block size and infinite gap recovers unbiased BIG, which the paper deems practically infeasible in standard ICL (Elmoznino et al., 17 Jun 2026).

The positive result holds in non-temporal Bayesian Experimental Design (BED), where observations are conditionally independent given $y$7 and the action sequence. In BED, $y$8 is asymptotically unbiased from below as $y$9, while $r$0 is implementable and asymptotically unbiased from above up to a non-negative Jensen gap that vanishes as $r$1. The paper therefore recommends $r$2 or $r$3 in BED-like domains and states that both avoid the “noisy TV” pathology that afflicts surprisal (Elmoznino et al., 17 Jun 2026).

Within this framework, ICM-CA is proposed as a cross-attention-based design. A context encoder uses self-attention over the history $r$4 to produce a context memory $r$5, and cross-attention query heads serve three roles: a predictive head for $r$6, a masked-context head for computing suffix predictives with $r$7 replaced by a MASK token, and a counterfactual head for implementing the BED description-length reward. The paper is explicit, however, that cross-attention cannot mitigate the nuisance terms in temporal MDPs in general; the impossibility theorem depends on environment dynamics, not on the attention mechanism. Cross-attention improves engineering—especially masked queries and counterfactual commitments—but does not remove the fundamental bias (Elmoznino et al., 17 Jun 2026).

5. The MHSL-specific ICM-CA framework

The most concrete use of the term “ICM-CA” appears in deceptive signal-assisted private multi-hop split learning. The system contains a set of $r$8 edge devices and one server that collaboratively train a global model $r$9 using multi-hop split learning while $p(t)$0 eavesdroppers attempt to intercept transmitted intermediate features and gradients. Some devices transmit deceptive signals to confuse eavesdroppers. The centralized agent decides the number of sub-models $p(t)$1, selects the $p(t)$2 training devices, partitions and assigns the global model sequentially into $p(t)$3, chooses deceptive-signal devices per hop, and controls transmit powers. The optimization objective is to minimize expected leakage subject to model-training energy-consumption and delay constraints (Wei et al., 9 Jul 2025).

The RL formulation uses a structured state and action space. The state includes remaining energy and time, the unassigned model portion, sub-model-to-device mapping, the current transmitter index, and vectors of distances from the current transmitter to eavesdroppers and to other devices. The action selects the next training device, the assigned sub-model, the deceptive set, transmit powers, and, in the backward phase, the gradient-receiver index. The extrinsic reward penalizes leakage and energy/time violations, and the total reward is

$p(t)$4

The actor is trained with CA-enhanced input $p(t)$5, whereas the critic follows a maximum-entropy SAC-style value formulation without an explicit twin-$p(t)$6 description in the paper (Wei et al., 9 Jul 2025).

The reported empirical outcomes are specific. Simulation results state that the proposed method improves the convergence rate by up to $p(t)$7 and reduces the information leaked to eavesdroppers by up to $p(t)$8 compared to the traditional SAC algorithm. The detailed results further report up to $p(t)$9 higher cumulative reward versus the variant without ICM, up to $C(p(t), h(\le t))$0 reward gain from adding CA, up to $C(p(t), h(\le t))$1 lower leakage versus PPO as monitoring probability increases, and up to $C(p(t), h(\le t))$2 and $C(p(t), h(\le t))$3 lower leakage versus SAC and PPO, respectively, at $C(p(t), h(\le t))$4. The framework also explores $C(p(t), h(\le t))$5 more states within 20 epochs than vanilla SAC, and when eavesdropper locations are unknown, cumulative reward decreases by approximately $C(p(t), h(\le t))$6 at 25 epochs while convergence remains similar. The policy behavior described in the paper assigns larger sub-models to devices farther from eavesdroppers, selects nearby devices as deceptive transmitters, and balances hop lengths and safety to reduce leakage and latency (Wei et al., 9 Jul 2025).

6. Limitations, misconceptions, and research directions

Several limitations are explicit across the literature. First, rewarding prediction error alone remains theoretically fragile. The compression-progress account states that intrinsic reward should reflect improvements in prediction or compression, not error magnitude, because otherwise the agent is attracted to irreducible randomness or model limitations (0812.4360). This critique applies directly to ICM-style forward-error rewards unless an additional mechanism separates epistemic from aleatoric uncertainty.

Second, cross-attention should not be treated as a universal remedy. In the ICL formalism, any finite-horizon estimator formed from posterior-predictive queries remains biased for BIG in general BAMDPs, regardless of whether the architecture uses self-attention, cross-attention, or other predictive heads. The positive guarantees are confined to BED-like settings; in temporal settings, the abductive term persists and the residual disappears only asymptotically (Elmoznino et al., 17 Jun 2026).

Third, not every attention-augmented curiosity method is an instance of ICM-CA. The 2019 attention-based curiosity paper neither defines nor evaluates cross-attention, and it explicitly frames its attention layers as learned weighting mechanisms rather than transformer-style cross-attention blocks (Reizinger et al., 2019). This suggests that the term “ICM-CA” is best reserved for architectures in which cross-attention is part of the control or predictive interface, rather than for all attention-enhanced ICM variants.

The future directions are correspondingly domain-specific. In the MHSL setting, proposed extensions include twin-$C(p(t), h(\le t))$7 critics and target networks, automatic temperature tuning for $C(p(t), h(\le t))$8, prioritized replay with CA-weighted sampling, more precise modeling of $C(p(t), h(\le t))$9 such as MI-based leakage per layer, adaptive eavesdroppers, asynchronous MHSL, and heterogeneous spectrum (Wei et al., 9 Jul 2025). In the ICL-based setting, open problems include exposing and manipulating latent posteriors $B_t(D) = -L_t(D)$0, jointly training $B_t(D) = -L_t(D)$1 and $B_t(D) = -L_t(D)$2 to mitigate covariate shift, and scaling to online RL with bounded context windows (Elmoznino et al., 17 Jun 2026). Taken together, these directions indicate that ICM-CA is not a single settled algorithm but a design space organized around three recurring questions: what intrinsic signal should be rewarded, how historical context should be queried, and under what environment assumptions those two choices are theoretically aligned.