AFAContext: Synthetic Benchmark for Lookahead AFA

• AFAContext is a synthetic classification dataset with 30 features and 8 classes, designed to test non-myopic feature acquisition strategies.
• It uses a contextual switch with two one-hot encoded features to determine which downstream feature block is informative, challenging greedy selection methods.
• Evaluated under a fixed budget within AFABench, AFAContext highlights the necessity of lookahead policies for effective sequential feature acquisition.

Searching arXiv for the topic and closely related papers.

AFAContext is a synthetic classification dataset introduced in AFABench, a benchmark framework for Active Feature Acquisition (AFA), to test a specific failure mode of sequential feature selection: the inability of greedy policies to value features whose importance is primarily downstream rather than immediate. Its defining property is a contextual switch in which two one-hot context features determine which later block of features is informative, so the dataset functions as a controlled benchmark for lookahead and non-myopic acquisition rather than merely instance-specific feature selection (Schütz et al., 20 Aug 2025).

1. Position within active feature acquisition

In the AFABench formulation, AFA is a sequential decision problem in which the learner does not observe all features of a test example upfront. Instead, an acquisition policy $\pi$ reveals features one at a time under a hard budget $b$, and a predictor $f$ makes the final class prediction from the acquired subset. The benchmark objective is written as

$$\min_{f,\pi}\; \mathbb{E}_{p(\mathbf{x},\mathbf{y})}\!\left[ \ell\!\bigl(f(\{\mathbf{x}_{a_t}\}_{t=1}^{b}),\,\mathbf{y}\bigr) \right],$$

where $a_n=\pi(\{\mathbf{x}_{a_t}\}_{t=1}^{n-1})$ for $n=1,\dots,b$. AFABench assumes equal acquisition cost for all features and a fixed budget $b$ for all examples, so exactly $b$ features are acquired per instance (Schütz et al., 20 Aug 2025).

This setting clarifies the scientific role of AFAContext. Standard synthetic benchmarks such as CUBE are useful for testing whether a method performs instance-specific dynamic feature selection, but the AFABench authors argue that they are poor tests of whether a method can perform non-greedy, lookahead acquisition. AFAContext was introduced specifically to fill that gap. It asks when a policy that optimizes only immediate predictive gain fails, and when policies that optimize for future value do better.

The contrast with greedy acquisition is explicit in the benchmark. An ideal greedy policy is written as

$$\pi(x_S) = \argmax_{i \in [n]} I(\mathbf{y}; \mathbf{x}_i \mid x_S),$$

with

$I(\mathbf{y}; \mathbf{x}_i \mid x_S) = D_{\text{KL} \big(p(\mathbf{x}_i, \mathbf{y} \mid x_S) \,\big\|\, p(\mathbf{x}_i \mid x_S)\, p(\mathbf{y} \mid x_S)\big).$

AFAContext is designed so that this one-step criterion systematically undervalues strategically important context features.

2. Construction and formal specification

AFAContext is a synthetic classification dataset with 30 features and 8 classes. Its reported split sizes are 700 training, 150 validation, and 150 test examples (Schütz et al., 20 Aug 2025).

Property	Value
Train / Validation / Test	700 / 150 / 150
Number of features	30
Number of classes	8

The dataset is defined compositionally relative to CUBE. In the AFABench description, CUBE consists of 20-dimensional real-valued vectors divided into 8 classes; for each class, three informative features are drawn from a Gaussian distribution $b$0 with class-specific means $b$1, while the remaining features are drawn from $b$2 as noise. AFAContext inherits this structure but adds a contextual switch (Schütz et al., 20 Aug 2025).

The paper specifies the following feature groups:

• Context features: $b$3, described as two one-hot encoded context features.
• First candidate informative block: when $b$4, the ten features $b$5 are informative and follow the same structure as the first ten features of CUBE.
• Second candidate informative block: when $b$6, the ten features $b$7 are informative and mirror the structure of CUBE.
• Inactive block: whichever of the two ten-feature blocks is not selected by context contains no information and is random noise (Schütz et al., 20 Aug 2025).

This gives AFAContext its essential dependency pattern: the label is predicted from informative features located in one of two candidate blocks, but which block is informative depends on the one-hot context. The role of features $b$8 is not explicitly specified in the paper’s main text. A cautious reading is therefore that the semantics of 22 of the 30 features are explicitly defined, while the remaining 8 features are left unspecified. It is reasonable to infer that these are additional noise features, but that inference is not stated as part of the formal description.

The paper also does not provide a full symbolic generative process for AFAContext. In particular, it does not state the exact class prior, the exact distribution of the context variables $b$9, or the class-conditional means $f$0 used inside the contextual construction. Instead, the dataset is defined by the contextual reuse of CUBE-like structure.

3. Sequential dependency and non-myopic structure

The central structural idea of AFAContext is that some features are useful mainly because they alter the value of later acquisitions. The paper describes the dependency chain as

$f$1

This is not a temporal dependency in data generation, but it induces a sequential dependency in the acquisition problem: the best next action depends on whether the policy has already identified the context (Schütz et al., 20 Aug 2025).

Accordingly, the paper states that an optimal policy should first acquire the context features and then exploit the correct block:

“An optimal acquisition policy should therefore first select the context features $f$2 and $f$3 to identify the relevant context group, and then acquire the corresponding informative features.”

This is the point at which greedy acquisition fails. The context features “do not offer predictive value on their own and are therefore overlooked by greedy methods that focus solely on maximizing short-term performance.” In other words, a context feature can have little or negligible immediate $f$4 while still having high multi-step value because it determines which later features become worth acquiring (Schütz et al., 20 Aug 2025).

AFAContext therefore operationalizes three properties simultaneously. Immediate predictive gain is small for some strategically critical actions. Future gains depend on first resolving context. One-step optimality is therefore not identical to sequential optimality. The paper does not provide a symbolic proof of greedy suboptimality, nor a hand-worked numerical trajectory, but its conceptual argument is explicit: by construction, the dataset contains an “initially uninformative” feature whose value lies in what it unlocks later.

4. Benchmark protocol and MDP formulation

AFAContext is evaluated under the same AFABench protocol as the other benchmark datasets. At test time, the policy begins with all feature values unobserved, sequentially acquires features one at a time, and stops only when exactly $f$5 features have been acquired. Because AFABench assumes equal feature cost and excludes a stop action in the main benchmark protocol, cost is controlled entirely by the hard budget $f$6. Performance is reported as a function of budget, with accuracy as the metric (Schütz et al., 20 Aug 2025).

For reinforcement-learning methods, AFABench also gives an MDP formulation: $f$7 Here $f$8 is the set of observed feature indices, $f$9 the observed values, and the reward definition depends on the method (Schütz et al., 20 Aug 2025).

The paper lists three reward constructions used by representative RL baselines:

• JAFA-MFRL: sparse terminal reward,

$$\min_{f,\pi}\; \mathbb{E}_{p(\mathbf{x},\mathbf{y})}\!\left[ \ell\!\bigl(f(\{\mathbf{x}_{a_t}\}_{t=1}^{b}),\,\mathbf{y}\bigr) \right],$$0

• ODIN-MBRL and the ODIN model-free variant: dense reward,

$$\min_{f,\pi}\; \mathbb{E}_{p(\mathbf{x},\mathbf{y})}\!\left[ \ell\!\bigl(f(\{\mathbf{x}_{a_t}\}_{t=1}^{b}),\,\mathbf{y}\bigr) \right],$$1

• OL-MFRL: certainty-change reward,

$$\min_{f,\pi}\; \mathbb{E}_{p(\mathbf{x},\mathbf{y})}\!\left[ \ell\!\bigl(f(\{\mathbf{x}_{a_t}\}_{t=1}^{b}),\,\mathbf{y}\bigr) \right],$$2

AFAContext was reported in the main figure with Budget = 10 using the shared classifier, and additional appendix figures report budgets 3, 5, and 10, as well as analogous builtin-classifier evaluations. This makes the dataset a controlled way to examine how much budget is needed before a policy can benefit from context resolution.

5. Empirical behavior within AFABench

AFAContext is one of the few datasets in AFABench where a non-greedy structure is known in advance. The results are therefore interpreted diagnostically rather than only comparatively. In the paper’s textual summary, ODIN-MFRL and OL-MFRL “significantly outperform all others” on AFAContext, and this is taken as evidence that reinforcement-learning approaches are well suited to capturing the dataset’s non-greedy structure (Schütz et al., 20 Aug 2025).

The same summary states that “all greedy methods perform poorly as expected.” This is the empirical counterpart of the construction principle: features with high future value but low one-step utility are systematically missed by greedy selection. At the same time, the dataset does not imply that every non-greedy method will succeed. The paper notes that some RL methods also perform poorly, “probably due to convergence challenges,” and that the oracle-based non-greedy method AACO also performs poorly, likely because the heuristic used to adapt it to one-at-a-time acquisition “reduces the method’s non-greedy behavior.”

The comparison between ODIN-MFRL and ODIN-MBRL is also notable. The paper reports that ODIN-MFRL outperforms ODIN-MBRL on AFAContext and interprets this as evidence that model-based RL may not have a clear advantage when enough data are available (Schütz et al., 20 Aug 2025).

The broader benchmark context sharpens the significance of these findings. AFABench reports that on most real-world datasets, discriminative greedy methods such as GDFS-DG and DIME-DG are among the strongest performers. AFAContext does not overturn that broader observation. Instead, it shows that the apparent sufficiency of greedy methods on many benchmark datasets may reflect the absence of strong sequential dependencies in those datasets, rather than the irrelevance of lookahead as a principle.

6. Reproducibility, underspecification, and implementation details

The paper provides a substantial but incomplete specification of AFAContext. Explicitly stated details include: synthetic dataset, 30 features, 8 classes, 700/150/150 train/validation/test split, evaluation with accuracy, equal feature costs, hard budget setting, and reported budgets 3, 5, and 10 (Schütz et al., 20 Aug 2025).

Several benchmark-wide settings also apply to AFAContext. The shared classifier is an MLP with dropout 0.1, learning rate $$\min_{f,\pi}\; \mathbb{E}_{p(\mathbf{x},\mathbf{y})}\!\left[ \ell\!\bigl(f(\{\mathbf{x}_{a_t}\}_{t=1}^{b}),\,\mathbf{y}\bigr) \right],$$3, batch size 128, and two hidden layers of 128 ReLU units. For non-MNIST datasets, masking probability during pretraining is sampled from $$\min_{f,\pi}\; \mathbb{E}_{p(\mathbf{x},\mathbf{y})}\!\left[ \ell\!\bigl(f(\{\mathbf{x}_{a_t}\}_{t=1}^{b}),\,\mathbf{y}\bigr) \right],$$4. RL methods use $$\min_{f,\pi}\; \mathbb{E}_{p(\mathbf{x},\mathbf{y})}\!\left[ \ell\!\bigl(f(\{\mathbf{x}_{a_t}\}_{t=1}^{b}),\,\mathbf{y}\bigr) \right],$$5, batch size 512, 128 parallel agents, and 10,000 batches for training, except on FashionMNIST. Reported results are averaged over multiple data splits and random seeds, with standard deviations shown as error bars (Schütz et al., 20 Aug 2025).

AFAContext also required dataset-specific modifications for ODIN. When training the PVAE in ODIN-MBRL, “the scaling factor for the KL loss term” was set to 0.01 for AFAContext instead of the usual 0.1, because the default value 1 caused posterior collapse. For AFAContext comparisons, the PPO policy/value networks in ODIN were changed from single linear layers to hidden layers of size $$\min_{f,\pi}\; \mathbb{E}_{p(\mathbf{x},\mathbf{y})}\!\left[ \ell\!\bigl(f(\{\mathbf{x}_{a_t}\}_{t=1}^{b}),\,\mathbf{y}\bigr) \right],$$6 because the original architecture could underfit (Schütz et al., 20 Aug 2025).

The principal limitation is that exact dataset recreation from the paper text alone is unlikely. The paper does not specify the exact class prior distribution, the exact means $$\min_{f,\pi}\; \mathbb{E}_{p(\mathbf{x},\mathbf{y})}\!\left[ \ell\!\bigl(f(\{\mathbf{x}_{a_t}\}_{t=1}^{b}),\,\mathbf{y}\bigr) \right],$$7, the exact distribution of the context variables $$\min_{f,\pi}\; \mathbb{E}_{p(\mathbf{x},\mathbf{y})}\!\left[ \ell\!\bigl(f(\{\mathbf{x}_{a_t}\}_{t=1}^{b}),\,\mathbf{y}\bigr) \right],$$8, the precise role of features $$\min_{f,\pi}\; \mathbb{E}_{p(\mathbf{x},\mathbf{y})}\!\left[ \ell\!\bigl(f(\{\mathbf{x}_{a_t}\}_{t=1}^{b}),\,\mathbf{y}\bigr) \right],$$9, or a full probabilistic definition $a_n=\pi(\{\mathbf{x}_{a_t}\}_{t=1}^{n-1})$0. This suggests that the released code is necessary for exact replication, even though the benchmark intent and high-level construction are clear.

7. Terminological scope

Within the arXiv literature, the primary and explicit meaning of AFAContext is the AFABench synthetic dataset described above (Schütz et al., 20 Aug 2025). By contrast, in several unrelated papers the exact string is not used literally; rather, it has been used interpretively to describe a context-aware variant of a method whose acronym is also AFA. Examples include Adaptive Feature Alignment for adversarial training, where the latent “context” is unknown attack strength and the model predicts fusion weights between two BN-based feature domains (Wang et al., 2021), and Adaptive Friend Agent, where context is partitioned by user identity through speaker identification, identity-aware routing, and per-user memory stores (Al-Ratrout et al., 27 Apr 2026).

These latter uses are not standardized technical definitions of “AFAContext.” They indicate only that the term can be read descriptively as “AFA with an explicit context mechanism.” In strict usage, however, AFAContext denotes AFABench’s deliberately constructed lookahead test for active feature acquisition: a synthetic 8-class, 30-feature dataset in which two one-hot context features determine which downstream feature block is informative, thereby exposing the limitations of greedy sequential acquisition (Schütz et al., 20 Aug 2025).