In-Context Weight Prediction Task

• The task involves using contextual evidence to predict latent weights such as model parameters, example coefficients, or physiological trajectories.
• It encompasses distinct settings including prompt-conditioned inference, demonstration reweighting, and body weight forecasting, each applying specialized mechanisms like linear regression and attention reweighting.
• Evaluation protocols focus on phase transitions, bias correction, and mechanistic separability, highlighting trade-offs in accuracy, generalization, and practical deployment.

“In-context weight prediction task” denotes a family of problems in which a model uses information available in a context window to infer a latent weighting object needed for prediction. In recent arXiv usage, the phrase covers at least two technically distinct settings: prompt-conditioned inference of task parameters, example weights, or indicator weights in in-context learning, and temporal forecasting of human body weight or weight-objective outcomes from multimodal history (Daniels et al., 2 Jul 2025, Gui et al., 2024). The unifying idea is that prediction depends on contextual identification of a latent object that is not directly given at test time: an unknown system matrix, a regression weight vector, a prompt-specific reweighting over demonstrations, a task-model-specific indicator weight vector, or a future physiological weight trajectory.

1. Scope and terminology

A useful way to organize the literature is to distinguish between what may be called “parameter-centric ICWP” (Editor’s term) and body-weight forecasting tasks. In the first case, the model predicts weights in the sense of model parameters, per-example coefficients, or multi-indicator scores; in the second, it predicts human weight or weight-objective attainment from contextual history. The sources use the same phrase for both, but the underlying mathematical objects differ substantially. This suggests terminological polysemy rather than a single unified benchmark.

Usage	Predicted quantity	Representative papers
Prompt-conditioned parameter inference	System matrix $U$, task vector $w$, or effective predictor parameters	(Daniels et al., 2 Jul 2025, Chang et al., 3 Mar 2025, Lu et al., 17 May 2025, Abedsoltan et al., 2024, Li et al., 2023)
Demonstration or data weighting	Example weights $w_i$ or indicator weights $w_j$	(Yang et al., 2023, Chu et al., 2023, Song et al., 10 May 2026)
Physiological weight forecasting	Future body weight or success at a stated weight objective	(Gui et al., 2024, Veličković et al., 2017)

Within parameter-centric ICWP, the common pattern is that a frozen or pretrained model must infer a task-specific object from contextual evidence and then immediately use it. In linear-regression-style formulations, the target is a latent weight vector $w$; in dynamical-system formulations, the target is a latent transition matrix $U$; in reweighting formulations, the target is a vector over prompt examples or data indicators. By contrast, in diet and wearable forecasting, “weight prediction” refers literally to human weight measurements and related health outcomes (Chang et al., 3 Mar 2025, Gui et al., 2024).

2. Prompt-conditioned inference of latent parameters

A central formulation appears in interleaved linear dynamical systems. “Decomposing Prediction Mechanisms for In-Context Recall” trains a transformer on symbolically labeled segments from multiple independent linear deterministic dynamical systems, each defined by an unknown orthogonal matrix $U \in \mathbb{R}^{5\times 5}$ drawn iid from the Haar measure over $O(5)$ and evolving according to $x_{i+1} = U x_i = U^{i+1}x_0$. In evaluation, a haystack of systems is followed by a query open symbol, and the model must identify which system to resume and then predict future states. For these orthogonal systems, exact identification is possible after 6 positions using

$$U = [x_1\ x_2\ x_3\ x_4\ x_5][x_0\ x_1\ x_2\ x_3\ x_4]^{-1}.$$

The training loss is next-output prediction with MSE on the 5-D payload coordinates, with symbolic-token losses masked to zero (Daniels et al., 2 Jul 2025).

A closely related formalization arises in multi-task linear regression with one-layer linear attention. “Provable Benefits of Task-Specific Prompts for In-context Learning” treats ICWP as predicting $w$0 for an unseen task after inferring the task weight vector $w$1 from contextual examples. Under a reduced “PGD-like” predictor,

$w$2

task-specific prompt $w$3 carries information about the conditional mean $w$4, while the learned preconditioner $w$5 captures variance structure. The paper’s covariance–mean decomposition states that prompt-tuning explains the conditional mean and attention weights explain the variance; adding task-specific heads yields a fully decoupled loss depending on the debiased moment $w$6 rather than $w$7 (Chang et al., 3 Mar 2025).

“Inverse linear regression” makes the weight-prediction interpretation explicit. “Transformer learns the cross-task prior and regularization for in-context learning” studies the rank-deficient regime $w$8, where the context is too short to identify $w$9 without regularization. The model is trained across tasks by reconstructing labels,

$w_i$0

and learns both a cross-task prior $w_i$1 and an effective regularization strategy. The learned estimator empirically matches the oracle Bayesian posterior mean and obeys the same scaling laws with noise level, task-to-context ratio, and conditioning of $w_i$2, with a necessary condition for success given as low task dimensionality relative to context length, $w_i$3 (Lu et al., 17 May 2025).

The same theme appears in simplified attention models. “Context-Scaling versus Task-Scaling in In-Context Learning” shows that a one-block simplified transformer computes a data-dependent feature map $w_i$4, with $w_i$5 giving

$w_i$6

which corresponds to one step of gradient descent from $w_i$7 in linear regression. The feature map alone context-scales but does not task-scale, while concatenating $w_i$8 with vectorized data enables both context-scaling and task-scaling (Abedsoltan et al., 2024). “The Closeness of In-Context Learning and Weight Shifting for Softmax Regression” extends the analogy to softmax regression, showing that one self-attention layer and one GD step induce bounded shifts of the same form, with $w_i$9 controlling the change in the effective target under both parameter updates and attention-induced data transformations (Li et al., 2023).

3. Mechanisms, circuits, and emergence dynamics

Mechanistic work shows that parameter-centric ICWP is often not implemented by a single algorithm. In the interleaved dynamical-system task, the model uses two distinct mechanisms: a label-based associative recall mechanism for the first state after the query open label, and a label-agnostic “Bayesian-style” continuation mechanism for subsequent tokens. Their learning dynamics differ sharply. For $w_j$0 haystack, the first-token MSE drops sharply near $w_j$1 examples; for $w_j$2, the transition occurs closer to $w_j$3. Continuation tokens improve earlier and more gradually, and at the end of training first-token performance is good and relatively independent of $w_j$4, at $w_j$5 median MSE across haystack sizes, whereas later-token performance degrades as $w_j$6 increases. Edge pruning isolates a 200-edge “1-after” circuit and a 40-edge “2-after” circuit with 0% edge overlap, indicating mechanistic separability rather than a shared computation graph (Daniels et al., 2 Jul 2025).

Gated recurrent attention yields an explicit weighting interpretation. “Gating is Weighting: Understanding Gated Linear Attention through In-context Learning” shows that Gated Linear Attention implements Weighted Preconditioned Gradient Descent. With scalar gating, the effective token weight is

$w_j$7

and one-layer GLA implements one-step WPGD; multi-layer GLA implements multi-step WPGD-like updates. The paper also proves that, under mild spectral-gap conditions, the WPGD risk has a unique global minimum up to scaling, and that vector gating spans the full WPGD class and can be provably better than vanilla linear attention when scalar monotone block-wise weighting is insufficient (Li et al., 6 Apr 2025).

A different mechanistic lens is statistical physics. “Spin glass model of in-context learning” maps a single-layer transformer with linear attention to a real-valued spin glass in which the flattened parameter matrix $w_j$8 becomes the spin vector $w_j$9, the features $w$0 induce dense couplings $w$1, and inference is described by a Gibbs distribution $w$2. Approximate message passing yields posterior means $w$3 and variances $w$4, and task diversity causes the Gibbs measure to concentrate on the unique solution with $w$5 and $w$6. The reported phase diagram shows a smooth transition toward perfect in-context generalization as $w$7 increases (Li et al., 2024).

4. Reweighting demonstrations and indicator scores

A second major branch of the literature uses “weight prediction” to mean prompt-specific or task-specific reweighting. “Not All Demonstration Examples are Equally Beneficial: Reweighting Demonstration Examples for In-Context Learning” defines the task as predicting a weight vector $w$8 over the $w$9 demonstrations in an ICL prompt. Since validation labels are unavailable in true few-shot settings, the paper introduces the Masked Self-Prediction score,

$U$0

where each demonstration label is masked in turn. Beam search over a discretized weight set is then combined with attention-level reweighting through Scaling Key Matrix or Scaling Attention Weights. MSP shows a Pearson correlation of $U$1 with final accuracy on MR, and WICL-SKM consistently improves over uniform-weight ICL; for example, in 16-shot settings GPT-13B improves from $U$2 to $U$3 average accuracy (Yang et al., 2023).

“Fine-tune LLMs to Approximate Unbiased In-context Learning” casts reweighting as bias correction. Given a biased prompt and an unbiased validation set, the method learns a diagonal reweighting matrix $U$4 and bias $U$5 applied after the embedding layer,

$U$6

RICL optimizes validation loss through the frozen transformer, while LARICL replaces the full model with a linear approximation in which

$U$7

The paper proves monotone decrease of the validation objective under a stepsize condition and gives the convergence rate $U$8 in $U$9 steps (Chu et al., 2023).

“Learning Multi-Indicator Weights for Data Selection” shifts from prompt examples to instruction-data indicators. Each candidate sample $U \in \mathbb{R}^{5\times 5}$0 receives normalized scores $U \in \mathbb{R}^{5\times 5}$1, and the learned weight vector $U \in \mathbb{R}^{5\times 5}$2 defines

$U \in \mathbb{R}^{5\times 5}$3

The selected subset is evaluated not through full fine-tuning but through the proxy

$U \in \mathbb{R}^{5\times 5}$4

where $U \in \mathbb{R}^{5\times 5}$5 is an IRT-constructed tiny-validation set and Bayesian optimization searches over $U \in \mathbb{R}^{5\times 5}$6. On GSM8K with 30% of the training data, TMAP reports Qwen2.5-7B Full $U \in \mathbb{R}^{5\times 5}$7 versus TMAP $U \in \mathbb{R}^{5\times 5}$8, and Llama3-8B Full $U \in \mathbb{R}^{5\times 5}$9 versus TMAP $O(5)$0. The paper also reports that adding explicit diversity filtering can hurt reasoning performance, revealing a trade-off between semantic diversity and logical complexity (Song et al., 10 May 2026).

5. In-context versus in-weight learning

The distinction between prompt-conditioned inference and knowledge stored in parameters is itself a central object of study. “Toward Understanding In-context vs. In-weight Learning” models prediction as a gated mixture

$O(5)$1

where $O(5)$2 is an in-weight predictor and $O(5)$3 is an in-context similarity-weighted predictor. The paper proves that in-weight learning improves with the per-query sample size $O(5)$4 at an $O(5)$5 rate, while in-context learning faces an error floor that increases with distractors and similarity noise. The resulting theory predicts emergence of ICL on rare or long-tail classes and its eventual disappearance as repeated exposures make IWL more reliable. Synthetic experiments, Omniglot few-shot classification, and Gemini Nano fine-tuning corroborate the predicted shift from ICL to IWL with continued training (Chan et al., 2024).

The interleaved dynamical-system work gives a concrete instance of the same distinction. Using OLMo-2 7B checkpoints on few-shot translation, the paper reports that purely symbolic labels “X:” and “Y:” produce a late phase transition in first-token performance, while the second token improves earlier and more smoothly. When the labels are semantically meaningful—“Spanish:” and “English:” rather than purely symbolic markers—the first-token gap disappears, and an unseen label “Z:” leaves second-token accuracy unchanged. The result separates task competence already present in weights from label-based associative recall that must be learned in context (Daniels et al., 2 Jul 2025).

In this sense, many ICWP formulations are not purely in-context or purely in-weight. Task-specific prompts can encode prior means, while attention or continuation mechanisms estimate variance or dynamics from local evidence; conversely, continued training can shift apparent in-context behavior into parameter memory. The literature therefore treats source selection—prompt versus weights—not as a philosophical distinction but as a measurable component of predictive behavior (Chan et al., 2024).

6. Physiological weight forecasting from historical context

A separate usage applies the phrase to body-weight prediction from contextual health signals. “Navigating Weight Prediction with Diet Diary” defines a multivariate time-series problem in which the history consists of daily body weights $O(5)$6 and meal sets $O(5)$7. Unified Meal Representation Learning produces meal embeddings $O(5)$8, and the multivariate sequence

$O(5)$9

is passed to a forecasting backbone $x_{i+1} = U x_i = U^{i+1}x_0$0. The paper introduces a diet-aware objective based on daily weight change $x_{i+1} = U x_i = U^{i+1}x_0$1,

$x_{i+1} = U x_i = U^{i+1}x_0$2

and releases the DietDiary dataset with 611 users, over 5,000 daily records, nearly 30,000 food images, and over 15,000 ingredient annotations. Using NLinear, the baseline 7–7 setting has MSE $x_{i+1} = U x_i = U^{i+1}x_0$3 and MAE $x_{i+1} = U x_i = U^{i+1}x_0$4, while the image-based framework reports $x_{i+1} = U x_i = U^{i+1}x_0$5 and $x_{i+1} = U x_i = U^{i+1}x_0$6; with iTransformer, several settings also improve under the proposed UMRL plus diet-aware loss (Gui et al., 2024).

An earlier multimodal formulation predicts whether a user will achieve a stated weight objective rather than the future weight sequence itself. “Cross-modal Recurrent Models for Weight Objective Prediction from Multimodal Time-series Data” uses daily weight, sleep, and steps histories together with static covariates and objective information. The proposed X-LSTM processes modalities in separate streams with recurrent cross-connections, then fuses them at the final timestep. On 18,036 sequences, the best X-LSTM variant reports AUC $x_{i+1} = U x_i = U^{i+1}x_0$7, Accuracy $x_{i+1} = U x_i = U^{i+1}x_0$8, F1 $x_{i+1} = U x_i = U^{i+1}x_0$9, and MCC $$U = [x_1\ x_2\ x_3\ x_4\ x_5][x_0\ x_1\ x_2\ x_3\ x_4]^{-1}.$$0, improving over a baseline LSTM with AUC $$U = [x_1\ x_2\ x_3\ x_4\ x_5][x_0\ x_1\ x_2\ x_3\ x_4]^{-1}.$$1 and over SH-LSTM with AUC $$U = [x_1\ x_2\ x_3\ x_4\ x_5][x_0\ x_1\ x_2\ x_3\ x_4]^{-1}.$$2 while using a comparable parameter budget (Veličković et al., 2017).

Although this physiological branch is mathematically and operationally distinct from parameter-centric ICWP, it shares the same contextual structure: prediction depends on extracting latent contributions from temporally organized evidence rather than from a single static input (Gui et al., 2024).

7. Evaluation protocols, limitations, and open directions

Across parameter-centric formulations, evaluation must separate initiation from continuation and prompt-conditioned inference from stored competence. In the interleaved dynamical-system task, the most informative probes are first-task-token versus second-task-token performance, held-out pretraining loss thresholds, and out-of-distribution interventions such as incorrect labels, unseen labels, synchronized states, and misdirections to seen labels. The paper explicitly recommends tracking emergence timelines by both training steps and held-out pretraining loss thresholds, with first-token transition roughly when held-out pretraining MSE reaches $$U = [x_1\ x_2\ x_3\ x_4\ x_5][x_0\ x_1\ x_2\ x_3\ x_4]^{-1}.$$3 and second-token transition at $$U = [x_1\ x_2\ x_3\ x_4\ x_5][x_0\ x_1\ x_2\ x_3\ x_4]^{-1}.$$4, and using edge pruning or attribution tools to isolate disjoint circuits (Daniels et al., 2 Jul 2025).

Theoretical guarantees remain conditional on narrow regimes. The inverse linear regression results require low intrinsic task dimension relative to context length, $$U = [x_1\ x_2\ x_3\ x_4\ x_5][x_0\ x_1\ x_2\ x_3\ x_4]^{-1}.$$5, and are derived under Gaussian priors, Gaussian features, and linear observation models (Lu et al., 17 May 2025). The covariance–mean decoupling theory is specific to one-layer linear attention with task-aware prompts and, in the main results, zero-mean Gaussian features and linear regression labels (Chang et al., 3 Mar 2025). The simplified-transformer account shows that $$U = [x_1\ x_2\ x_3\ x_4\ x_5][x_0\ x_1\ x_2\ x_3\ x_4]^{-1}.$$6-only feature maps can context-scale without task-scaling, which limits claims that context aggregation alone suffices for full in-context learning (Abedsoltan et al., 2024). Reweighting methods such as WICL also require access to model internals because SKM and SAW modify attention computations directly (Yang et al., 2023).

Application-side limitations are different but equally important. DietDiary does not model activity levels, hydration, sleep, medication, menstrual cycles, or illness, and the paper notes error accumulation in long-horizon autoregression; user-provided ingredient labels are sparse, and FoodLMM-generated annotations remain imperfect (Gui et al., 2024). The weight-objective prediction setting depends on future logging behavior and requires at least 10 contiguous days of measurements, which makes censoring and adherence relevant to interpretation (Veličković et al., 2017). These constraints suggest that “in-context weight prediction task” should be read as a family resemblance term: a shared emphasis on contextual identification of latent weighting structure, but not a single problem with a single benchmark or a single mechanism.