The on-line shortest path problem under partial monitoring (0704.1020v1)

Abstract: The on-line shortest path problem is considered under various models of partial monitoring. Given a weighted directed acyclic graph whose edge weights can change in an arbitrary (adversarial) way, a decision maker has to choose in each round of a game a path between two distinguished vertices such that the loss of the chosen path (defined as the sum of the weights of its composing edges) be as small as possible. In a setting generalizing the multi-armed bandit problem, after choosing a path, the decision maker learns only the weights of those edges that belong to the chosen path. For this problem, an algorithm is given whose average cumulative loss in n rounds exceeds that of the best path, matched off-line to the entire sequence of the edge weights, by a quantity that is proportional to 1/\sqrt{n} and depends only polynomially on the number of edges of the graph. The algorithm can be implemented with linear complexity in the number of rounds n and in the number of edges. An extension to the so-called label efficient setting is also given, in which the decision maker is informed about the weights of the edges corresponding to the chosen path at a total of m << n time instances. Another extension is shown where the decision maker competes against a time-varying path, a generalization of the problem of tracking the best expert. A version of the multi-armed bandit setting for shortest path is also discussed where the decision maker learns only the total weight of the chosen path but not the weights of the individual edges on the path. Applications to routing in packet switched networks along with simulation results are also presented.

• The paper presents an efficient online learning algorithm that achieves an optimal O(1/√n) regret while scaling polynomially with the graph size.
• The paper extends the approach to a label-efficient setting, balancing exploration with feedback costs in dynamic, limited-information environments.
• The paper adapts follow-the-perturbed-leader strategies to track time-varying paths, effectively handling adversarial changes in network conditions.

Overview of the On-Line Shortest Path Problem Under Partial Monitoring

The paper "The On-line Shortest Path Problem under Partial Monitoring" explores various online learning algorithms for the shortest path problem in a dynamic environment where the edge weights can change adversarially. This paper generalizes the multi-armed bandit problem to scenarios where the decision maker only learns the weights of the edges belonging to the chosen path, rather than the entire graph. The authors present algorithms with linear complexity relative to both the number of rounds and the number of edges, offering solutions for situations with partial feedback and adapting to more complex graph structures.

At the core of the paper lies the challenge of minimizing the average cumulative loss over rounds, measured against an optimal path that is determined offline. The decision making occurs in a weighted directed acyclic graph where the decision maker faces a loss defined by the cumulative weight of a chosen path. The paper provides competitive algorithms whose regret bounds are polynomial with respect to the number of edges, achieving convergence rates matching the theoretical bounds of $O(1/\sqrt{n})$, where $n$ is the number of rounds.

Key Contributions

1. Algorithm for Multi-Armed Bandit Setting: The paper introduces an algorithm for the multi-armed bandit version of the shortest path problem with exploration-exploitation strategies informed by exponentially weighted aggregation of past performance. This algorithm maintains the optimal $O(1/\sqrt{n})$ regret bound while scaling polynomially with the graph's size.
2. Label Efficient Extension: The research extends to a label-efficient setting where feedback is costly, and only a subset of losses can be queried. The authors develop a methodology that balances the tradeoff between exploration and packet transmission cost, enhancing the decision maker's efficiency.
3. Tracking Time-Varying Paths: An adaptive approach is proposed to compete against dynamic paths that may change over time, using variations of follow-the-perturbed-leader strategies. These strategies are tailored to handle path variations while keeping computational costs manageable.
4. Restricted Bandit Problem Algorithm: A significant challenge addressed is when only path losses (not individual edge losses) are observed. Here, the authors adapt techniques from previous works to create a refined algorithm exhibiting a $O(n^{-1/3})$ regret bound, retaining efficacy in this limited feedback environment.

Implications and Future Work

This work carries important implications for routing in dynamic network environments such as packet-switched networks where feedback may be limited due to cost or technical limitations. The algorithms presented are capable of adapting to changing network conditions, enhancing robustness against adversarial actions like denial-of-service attacks in secure network contexts.

Future research could focus on refining these algorithms to further minimize regret in restricted bandit settings, as the current paper highlights that the optimal $O(1/\sqrt{n})$ regret is yet to be achieved without exponential increases concerning graph size. Moreover, practical implementations in real-world networks would benefit from additional research into scalability and deployment optimizations in environments with rapidly changing topologies.

By bridging online learning with dynamic path optimization, this paper provides a foundational framework that is pivotal for advancing adaptive decision-making algorithms in various applications where complete information is inaccessible.