Macro Set Identification Problem

• Macro Set Identification Problem is the principled process of selecting structured action sequences or interventions to boost learning and causal inference efficiency.
• It employs systematic methodologies such as LZW-based compression, do-calculus, and Monte Carlo confidence set estimation to evaluate macro utility and identifiability.
• Empirical and theoretical evidence shows that using tailored macros expedites convergence in MDPs, clarifies causal effects, and refines parameter estimation in econometrics.

The Macro Set Identification Problem concerns the principled selection or identification of a set of “macros”—structured objects such as open-loop action sequences, interventions on clusters in graphical models, or sets of admissible parameter values—that enable maximal efficiency or identifiability in the context of learning, causal inference, or statistical estimation across a range of related tasks or models. Different fields instantiate this problem with precise methodologies, including reinforcement learning, causal graphical modeling, and econometric identification theory. Below, key frameworks from reinforcement learning and causal inference are detailed, alongside algorithmic, theoretical, and applied dimensions of the Macro Set Identification Problem.

1. Formal Definitions and Problem Setting

In reinforcement learning, the Macro Set Identification Problem is the process of selecting a set of open-loop action sequences (“macros”) that, when added to an agent’s primitive action set, maximally accelerate policy learning across a family of Markov decision processes (MDPs) sharing state and action spaces but differing in dynamics and rewards (Garcia et al., 2017). Formally, macros are finite sequences:

$$m = (a_1, a_2, \ldots, a_l), \quad a_i \in \mathcal{A},$$

with execution from a state $s$ resulting in a stochastic state trajectory according to the environment transitions. The target is to select $\mathcal{M}^*$ maximizing the expected performance gain across training MDPs:

$$\mathcal{M}^* \in \underset{\mathcal{M}}{\operatorname{argmax}}\;\frac{1}{|\mathcal{C}'|}\sum_{c\in\mathcal{C}'} \rho(\mathcal{M},c),$$

where $\rho(\mathcal{M},c)$ quantifies the mean reward using an action set augmented by candidate macros.

In causal inference over cluster-directed mixed graphs (C-DMGs), the analogous problem is to identify which sets of macro interventions (on variable clusters) yield identifiable macro causal effects under partial model knowledge (Ferreira et al., 2 Apr 2025). Here, “macro effects” denote the causal effects of entire variable clusters upon others within partially specified graphical models, and the aim is to establish identifiability conditions and algorithms over cluster groupings and interventions.

In econometrics, the Macro Set Identification Problem links to constructing confidence sets for the identified parameter set in partially identified models, where the objective is to select the subset of parameters consistent with model-imposed constraints, typically determined by level sets of criterion functions (Chen et al., 2016).

2. Macro Generation and Candidate Construction

Macro generation in reinforcement learning operates via data-driven compression of optimal policy trajectories. Given near-optimal behaviors for a sample of tasks, an LZW-style dictionary-building algorithm encodes highly recurring action subsequences as macro candidates. The procedure iteratively builds a codebook by scanning action-only trajectories, appending novel sequences (not previously in the dictionary) as new macros:

• Initialize codebook $\Sigma\gets\mathcal{A}$.
• For each trajectory $h_a$, accumulate subsequence $m$, add $m{+}a$ to $\Sigma$ upon encountering novelty, and reset $m$.
• The set of candidate macros is $\mathcal{M}=\Sigma\setminus\mathcal{A}$ (Garcia et al., 2017).

In causal macro-effect identification, candidate “macro sets” are determined by the clustering of micro-variables and analysis of possible interventions and their effects, given partially specified graph structure. These clusters are defined formally over the underlying ADMG by partitioning the node set, and macro interventions correspond to simultaneous interventions on all micro-variables within a cluster (Ferreira et al., 2 Apr 2025).

Statistical macro set identification in partially identified models involves constructing sets of parameter vectors $\Theta_I$ that maximize a population criterion or satisfy the model-imposed optimality (e.g., likelihood maximizer, moment equality). These candidates are generated by inverting sample criterion level sets (Chen et al., 2016).

3. Macro Evaluation: Utility, Identifiability, and Criterion Functions

Macro evaluation in the learning context relies on assigning scores to each candidate using a closed-form utility function. The expected utility of a macro $m$ is given by

$$U_{\mathcal{C}}^\pi(m) = \mathbb{E}_{c\sim Unif(\mathcal{C}),\, S\sim d_\pi}\left[Q_c^\pi(S, m)\right],$$

where $Q_c^\pi(s,m)$ is computable without rollouts given access to the primitive Q-values and model transitions, as established by Theorem 1 in (Garcia et al., 2017). This closed-form expression enables cost-effective prioritization based on predicted performance impact.

In C-DMGs, macro evaluation is conducted via identifiability analysis using the do-calculus. The macro effect $P(\mathbf{C}_Y \mid \mathrm{do}(\mathbf{C}_X))$ is called identifiable if it is expressible in terms of the observed data distribution in all compatible graphical models. The rules of do-calculus are sound and complete at the macro level under mild cluster size assumptions (Ferreira et al., 2 Apr 2025).

For identified sets in statistics, macro evaluation corresponds to the calculation of quasi-likelihood ratios or GMM criteria that indicate how strongly candidate parameter vectors are supported by the sample. The identified set is characterized by parameters within a contour of the sample criterion attaining values close to the optimum, as in

$\Theta_I = \{\theta : Q(\theta) = 0\},$

with $Q_n(\theta)$ or its profiled version serving as evaluative metrics (Chen et al., 2016).

4. Macro Selection and Diversity

Macro selection aims to balance utility with diversity to avoid redundancy and action set bloat. In reinforcement learning, macros are greedily selected in decreasing order of utility, imposing a minimum Kullback-Leibler divergence in end-state distributions relative to already chosen macros:

• For each macro $m$, calculate $D_{KL}(p_m \,\|\, p_{m'})$ for existing macros $m'$ in the selected set.
• Accept $m$ if $$\min_{m'\in\mathcal{M}'} D_{KL}(p_m \| p_{m'}) > \delta$$ for a chosen threshold $\delta$ (Garcia et al., 2017).

In C-DMG-based causal inference, macro selection is restricted by identifiability: selection is terminated if a graphical non-identifiability criterion, the SC-hedge, is encountered in the strongly connected projection of the cluster graph. The selection procedure thus interleaves do-calculus rule application with combinatorial SC-hedge-checking to ensure only identifiable macro effects are pursued (Ferreira et al., 2 Apr 2025).

In econometric inference, selection of the macro (parameter) set is operationalized by Monte Carlo thresholding of the profile quasi-likelihood process, forming confidence sets for $\Theta_I$ or for marginalized parameter subvectors via quantiles from the simulated criterion distribution (Chen et al., 2016).

5. Soundness, Completeness, and Theoretical Guarantees

In C-DMGs, macro effect identifiability is characterized rigorously:

• Soundness: Every application of the three do-calculus rules at the cluster (macro) level is valid in all compatible ADMGs.
• Completeness: If a macro effect cannot be reduced to an observational formula by repeated rule applications, then non-identifiability is guaranteed, and a “macro hedge” exists witnessing this obstruction. The SC-hedge criterion provides necessary and sufficient conditions for non-identifiability (Ferreira et al., 2 Apr 2025).

For Monte Carlo confidence sets on identified sets, theoretical guarantees include exact asymptotic frequentist coverage for both full parameters and subvectors in regular models, extensions to models with singularities, and uniform validity under drifting data generating processes (Chen et al., 2016).

6. Algorithmic Approaches and Complexity

The algorithmic structure of macro set identification varies by setting:

Field	Generation	Evaluation	Selection Criteria
Reinforcement Learning	LZW trajectory compression	U-function (closed-form Q)	Greedy, KL diversity
Causal Inference (C-DMG)	Clustering, SC-projection	Do-calculus	SC-hedge check, completeness
Econometric ID Sets	Criterion inversion	QLR/profile QLR	Monte Carlo CS quantiles

Macro candidate construction may be exponential in the length or number of primitives, while selection steps (KL-divergence or SC-hedge checking) may require evaluating combinatorial subsets, with no polynomial-time polynomial-time guarantee for the full identification procedure in general (Garcia et al., 2017, Ferreira et al., 2 Apr 2025).

7. Illustrative Examples and Empirical Significance

In reinforcement learning, empirical results show that agents augmented with compressed macros achieve more reliable exploration, faster convergence, and improved learning performance on novel but related MDPs compared to agents restricted to primitive actions or using other option-discovery methods (Garcia et al., 2017).

In causal inference, macro effects mediated by a “front-door” structure are identifiable despite latent confounding between the exposure and outcome clusters, as shown by C-DMG do-calculus derivation. In contrast, direct exposure-outcome cycles with latent confounding and no mediating clusters yield SC-hedges that preclude identifiability (Ferreira et al., 2 Apr 2025).

Monte Carlo confidence set methods for partially identified parameter sets demonstrate high frequentist coverage even in modest sample regimes, outperforming naive percentiles, and offer practical utility in diverse economic modeling contexts, such as discrete games with multiple equilibria or incomplete models for trade flows (Chen et al., 2016).

---

In summary, the Macro Set Identification Problem synthesizes algorithmic compression, criterion-based selection, and graphical identifiability analysis to enable principled discovery of macro-level structural elements—be they action sequences, causal interventions, or parameter sets—with broad applicability in machine learning, causal modeling, and econometric theory.