Graph Structured Combinatorial Semi-Bandit with Nonlinear Reward Associations through Separable Signals

Abstract: The identification of optimal structures within vast arrays of interconnected data necessitates significant sampling- and computational effort. Learning and leveraging underlying signal dependencies can improve efficiency and predictive capabilities considerably, but the ubiquity of nonlinear statistical relations amplifies the complexity of such undertakings. In this paper, we develop novel generic and adaptive strategies equipped with routines for graph-based causal reward modeling, analytic reproducing kernel methods, and Taylor approximation of functional processes. We establish theoretical performance guarantees sublinear in time and linear in data volume over time. Our analyses cover robustness to a multitude of uncertainties arising from noise interference, gradual model convergence, and solution space mismatch. The framework's general appeal is substantiated by a minimalistic set of conditions or reliance on prior estimates, while various outlined modifications address specific or extended settings. To demonstrate practical effectiveness, we conduct numerical experiments using both benchmarked synthetic and real-world transportation datasets.

• The paper introduces the SSEM-UCB framework that leverages causal SEM modeling to capture nonlinear reward dependencies in graph-structured bandit scenarios.
• It employs kernel-based RKHS optimization and dual UCB exploration strategies to achieve provable sublinear regret bounds despite noisy, complex signal feedback.
• Empirical results on synthetic and real-world datasets demonstrate the method’s scalability and effectiveness in settings like delay propagation and resource allocation.

Graph-Structured Combinatorial Semi-Bandits with Nonlinear Reward Associations

Problem Formulation and Motivation

The paper addresses the problem of sequential decision-making in combinatorial semi-bandit settings where rewards are generated as nonlinear functions of selected base-arm signals, and the dependencies among arms follow a sparse, directed graph structure. Specifically, the authors generalize prior work on combinatorial semi-bandits with linearly related rewards by introducing reward functions defined via analytic, additively separable structural equation models (SEMs). Here, the agent chooses super arms (subsets of arms up to cardinality $s$) at each round, observes both the exogenous (base-arm realizations) and endogenous (graph-propagated rewards) signals, and aims to maximize expected cumulative reward (or, equivalently, minimize regret) over an unknown time horizon.

This setting subsumes classical and combinatorial bandit models, advancing them by capturing arbitrary nonlinear and causal reward dependencies present in practical domains, such as biological or transportation networks, resource allocation, or neural architecture search. The framework motivates both theoretical and practical questions regarding sample-efficient identification of latent graph structures, reliable nonlinear reward estimation, and robust adaptive data-driven decision making under functional and structural uncertainties.

Methodological Contributions

The proposed framework and algorithms, collectively termed SSEM-UCB, are underpinned by three key methodological advances:

1. Explicit SEM-Based Causal Modeling: Rewards are determined by a SEM whose graph structure and analytic, additively separable edge functions are unknown and learned online. Base-arm selections are encoded as interventions on the exogenous components, while observed endogenous variables propagate as per the learned SEM.
2. Nonparametric Graph Function Estimation via RKHS Optimization: The functional graph weights are estimated using kernel-based methods in reproducing kernel Hilbert spaces (RKHSs), tailored for analytic functions. The authors formulate both joint and node-wise regularized least-squares programs to recover the SEM operator from noisy, partial feedback.
3. Combined Internal/External UCB Exploration: The learning agent applies an Upper Confidence Bound (UCB) principle in two layers: internally, for distributional uncertainty over exogenous signals (tracked via moments and their empirical confidence intervals, supporting Taylor-approximate reward expansions), and externally, to incentivize functional exploration and structural learning in the SEM parameter space. The selection policy adaptively balances these exploration bonuses according to the current model's estimates and uncertainties.

Algorithmically, two principal variants are advanced: SSEM-UCB (node-wise structure learning in acyclic graphs) and SSEM-UCB-JO (joint optimization over all arms, supporting cyclic and general graphs). Both share a phased initialization (ensuring first observations for each base-arm), online kernel-based SEM training, and an adaptive UCB-driven action selection mechanism.

Theoretical Guarantees

A substantial technical component of the work is the derivation of finite-time, high-probability regret bounds for the proposed algorithms. Key results include:

• Regret Guarantees (Theorem 1/Proposition 1): For the acyclic setting with proper analytic function spaces and sufficient moment/Taylor expansion degree, the expected regret is upper-bounded by $\mathcal{O}(N\ln^2 T)$ for SSEM-UCB and $\mathcal{O}(N\sqrt{T}\ln T)$ for SSEM-UCB-JO, where $N$ is the number of base arms and $T$ is the time horizon. Both regret rates are linear in $N$ and sublinear (or nearly so) in $T$. The bounds account for approximation errors arising from Taylor truncation, model misspecification, and noise contamination in observed signals.
• Robustness: The analysis is free from restrictive assumptions on feedback noise, signal distributions (beyond boundedness and absolute continuity), or precise function classes, and accommodates (bounded) observation and model noise by direct translation in the bounds.
• Comparison to Prior Art: The regret rates for SSEM-UCB align favorably with those for previously studied special cases—linear SEMs, monotone/submodular reward combinations, and kernelized bandits—without requiring prior knowledge of the reward graph or functions, see (2606.14650) (Table in Appendix).

Numerical Evaluation

The algorithms are empirically validated on both synthetic and real-world datasets. Key findings include:

• Synthetic Graph/Reward Systems: On randomly generated DAGs with a mix of nonlinear (periodic, quadratic) and linear endogenous functions, SSEM-UCB-JO achieves consistently lower and more stable time-averaged regret than strong kernel or neural bandit baselines. When increasing the Taylor expansion degree (i.e., capturing higher-order nonlinear effects), the empirical regret approaches optimality as measured relative to the best possible action, with diminishing statistical variation across trials.
• Real-World Application (Train Delay Prediction): On a temporal dataset of train station delays, SSEM-UCB-JO identifies clusters of infrastructure nodes where compounded delays are causally significant, outperforming benchmarks in subnetwork detection. The learned SEMs reflect realistic propagation of stochastic delays across the network, and dynamic adaptation to observed system changes.
• Scalability and Extensions: Practical recommendations are offered for efficient computation, including dynamic graph sparsification, regularization, and trade-offs in the Taylor expansion degree. The architecture is amenable to parallelization, block inversion, and context (feature) augmentation, thus supporting high-dimensional settings.

Implications and Future Directions

The paper establishes that causal, nonlinear reward modeling via analytic SEMs enables sample-efficient global optimization in combinatorial bandits where reward independence and linearity assumptions are untenable. The theoretical guarantees highlight that incorporation of causal structure learning does not asymptotically worsen regret rates versus linear or fully observed settings, provided sufficient regularization and function approximation quality can be maintained.

Practically, this framework can be extended to:

• Contextual Bandits/Generalization: Incorporation of side-information (contexts) via exogenous operator extensions;
• Thompson Sampling and Bayesian Policies: Replacement of UCB with Thompson-style sampling for improved empirical trade-offs under uncertainty;
• Reinforcement Learning Generalization: Extension to controlled Markov dynamics via recurrent graph models, potentially with time-lagged adjacencies;
• Model Selection/Regularization: Automated adaptation of functional bases and action pruning for computational tractability in dense or continuous-action regimes.

Limitations remain in terms of worst-case computational complexity for very large, densely connected graphs, and potential challenges in non-analytic or non-separable nonlinearities. The paper's analysis and algorithms are directly extensible to various other uncertainty models, delayed feedback, and stationary/nonstationary environment variations as well.

Conclusion

The study provides a comprehensive methodology for online combinatorial optimization under unknown, nonlinear, graph-structured reward mappings, combining robust causal inference and nonparametric functional regression with tailored exploration-exploitation strategies. The demonstrated theoretical and empirical results affirm the viability of causal SEM-based approaches for scalable, interpretable, and performant sequential learning in semibandit and related decision-making domains (2606.14650).