Learning to Place Guards by Reinforcement: A Geo-Free Neural Policy for the Vertex-Guard Art Gallery Problem

Abstract: Neural combinatorial optimization (NCO) has shown that policies trained by reinforcement can construct strong solutions to NP-hard problems directly from raw instances. What such a policy actually learns, as opposed to what its decoder expresses, remains much less clear. We study this distinction on the vertex-guard Art Gallery Problem, the NP-hard task of choosing polygon vertices from which to observe an entire region. A pointer-network policy is trained from a coverage-aware reward over its own rollouts under the constraint we call geo-free inference: at test time it sees only vertex coordinates, with no visibility computation and no geometric oracle. The policy places guards economically but leaves a tail of under-covered polygons that widens far beyond the training range. To locate the cause, we freeze the trained encoder and read its embeddings with a small single-shot classifier, still geo-free at inference. The classifier closes most of the feasibility gap, in and out of distribution and at up to roughly five times the training range, cutting under-covered polygons by about an order of magnitude at an explicitly reported cost in guard count. We read this as evidence that the reinforcement-trained representation already encodes the geometry required for feasibility, and that residual failures reflect decoder calibration rather than missing knowledge. Probing a frozen encoder thus offers a practical way to ask what a neural combinatorial solver has internalized.

• The paper presents a reinforcement learning approach with a pointer-network LSTM trained solely on vertex coordinates to internalize geometric structure for the vertex-guard Art Gallery Problem.
• Empirical results show competitive coverage and a near-optimal guard ratio, with a supervised encoder probe significantly reducing infeasibility in out-of-distribution cases.
• Representation probing reveals that the encoder effectively captures non-trivial geometric invariants, offering a diagnostic tool for practical combinatorial optimization under hard feasibility constraints.

Reinforcement Learning and Representation Probing for the Vertex-Guard Art Gallery Problem

Introduction

This paper presents a rigorous study of whether a neural policy can learn the geometric and combinatorial structure underlying the vertex-guard Art Gallery Problem (AGPVG) using only vertex coordinates, trained solely from a coverage-aware reward with no explicit visibility computation available at inference. The approach is rooted in neural combinatorial optimization (NCO) and leverages preference optimization (PO) with Bradley–Terry losses in a pointer-network LSTM backbone. The central methodological innovation is the introduction of a strict geo-free protocol at inference: only coordinates and features derived from the policy’s internal representations are read, isolating the information internalized by the model from any direct geometric oracle access.

Of particular significance is the analysis of what the reinforcement-trained encoder genuinely captures versus what is merely an artifact of decoder calibration. This is operationalized by freezing the encoder after policy training and probing its embeddings with a supervised, single-shot per-vertex classifier. This probe (noted as the ) is evaluated both in-distribution and substantially out-of-distribution, assessing generalization to polygons with many more vertices than seen during training. Comprehensive controls, including ablations and capacity-constrained probes, clarify where geometric feasibility is internalized.

Problem Setting and Methodology

The AGPVG is formalized as an NP-hard discrete set cover problem: given a simple polygon with $n$ vertices, find a minimum-cardinality subset of these vertices (guards) such that every point in the polygon is visible to at least one guard. All feasible placements must be vertex guards, preserving combinatorial hardness and non-uniform visibility geometry.

The policy architecture is a pointer-network LSTM with an autoregressive decoder. Input consists of normalized vertex coordinates only; no visibility matrix or geometric oracle is available, either for per-vertex coverage or as auxiliary features. Training proceeds via PO over the policy’s own rollouts, ranking and updating trajectories based on a utility that heavily penalizes infeasibility and, subject to coverage, minimizes guard cardinality.

Policy training uses discrete-visibility Monte Carlo samples to efficiently estimate coverage; exact evaluation via the CGAL library is reserved for reporting and supervised probe target construction. Preference optimization replaces fragile REINFORCE baselines, yielding stable convergence and higher-quality solutions as measured by coverage and guard efficiency.

Policy Behavior: Empirical Results

Empirically, the learned policy achieves competitive mean coverage and mean normalized guard count ($|S|/n$) compared to the classical greedy set-cover heuristic, attaining $|S|/OPT \approx 1.09$ in distribution. However, a significant left tail of infeasibility remains: a non-trivial fraction of test polygons (19% in the main test split) are under-covered even when the greedy decode is used, and this fraction expands for larger, out-of-distribution polygons ($n$ up to 5–11x the largest in training). This tail represents instances where the neural policy's decoder cannot guarantee feasibility, even as mean metrics suggest strong performance.

Representation Probing: Encoder vs Decoder

The core contribution is the diagnostic use of a supervised probe to infer what is actually learned by the encoder:

• The probe reads only frozen encoder embeddings, vertex coordinates, and a binary indicator for whether the policy placed a guard at a given vertex.
• It is formulated as a small Transformer with a per-vertex sigmoid head, trained to predict membership in a reference guard set obtained by running a visibility-aware local search initialized at the policy output.
• At inference, only one forward pass is used; iterative application shows empirical fixed-point behavior.

Key empirical findings:

• The probe, even at modest thresholds ($t = 0.20$), eliminates the in-distribution feasibility tail entirely and reduces the OOD infeasibility rate by an order of magnitude.
• Out-of-distribution (e.g., $n >$ 5x maximum seen in training), feasibility rises from 85% (policy alone) to 99% (probe at the same threshold), and for the largest instances considered ($n$ up to 2250), worst-case coverage is dramatically improved (e.g., minimum coverage rises from 0.09 with no encoder to 0.61 with the encoder, averaged over seeds).
• Encoder ablation (masked embeddings) reveals this improvement is not attainable with coordinates or seed indicator alone; the encoder uniquely carries geometric information supporting feasibility.
• A zero-capacity linear probe over the frozen encoder already achieves ROC-AUC ≈ 0.84 for guard prediction, indicating the learned geometric structure is linearly accessible.

However, recovering strict feasibility universally requires paying a cost in guard count vs. optimum ($|S|/OPT$ increases to ≈2.5–2.6x as the threshold is lowered for full feasibility). This tradeoff is thoroughly reported.

Protocol and Alternative Architectures

The geo-free constraint on inference strictly disallows use of any visibility computations, either as per-vertex features or for rollout evaluation during decoding. This contrasts with nearly all prior NCO work, where feasibility masking is achieved by consulting an oracle at decode-time.

The authors test (and find ineffective) several natural alternatives:

• Decoding via best-of-$K$ stochastic sampling from the policy distribution does not recover the lost feasibility tail.
• Learned iterative editors and supervised sequence models for guard set improvement suffer from exposure bias and error accumulation, and underperform both the local search baseline and the single-shot probe.
• The probe design thus isolates representation from procedural decoding and is validated as a practical fixed-point diagnostic.

Implications and Theoretical Significance

Theoretical Implications

The decoupling of representation (encoder) and expression (decoder output) provides concrete evidence that reinforcement-trained neural networks can internalize non-trivial geometric structure even from sparse, black-box reward signals absent explicit supervisions. Feasibility failures are not due to a lack of geometric knowledge in the policy’s learned representation, but rather to limitations in how the decoder utilizes those representations.

This has broader implications for combinatorial optimization with hard feasibility constraints. Where classical concentration is on route problems with unconstrained feasibility, the AGPVG forces the learned model “to know” a geometry that cannot be certified without oracle queries. Encoder probing offers a general diagnostic for what is internalized in such black-box reinforcement learners.

Practical and Future Directions

Practically, the geo-free-inference protocol is significant for deployment in settings where explicit visibility or coverage computations are computationally burdensome, e.g., online sensor placement or robotics applications. The probe allows geometric information learned during training to be extracted and utilized efficiently at deployment.

Future avenues include:

• Upgrading the regime to Transformer policies and encoders (as already validated for the probe).
• Extending the protocol to problems where coverage or feasibility is even more expensive to compute (terrain guarding, watchtower placement, etc.).
• Employing per-instance, learned thresholds for adaptively achieving feasible solution size–coverage tradeoffs without geometric oracle input.
• Investigating policy and probe generalization with more explicit invariance to rotation, reflection, or vertex indexing (possibly via attention-based architectures).

Conclusion

This work rigorously characterizes the separation between what neural combinatorial optimization policies internalize through reinforcement and what they express in their decoded solutions. By imposing geo-free inference and carefully dissecting decoder and encoder contributions, the authors establish that substantial geometric insight is internalized by the encoder, accessible with negligible downstream supervision, but frequently under-expressed by conventional decoders. The representation probing methodology provides a template for interpretability and practical improvement in NCO settings where direct feasibility evaluation is costly or unavailable. Looking ahead, these findings suggest comparable approaches are likely to be transferable and illuminating for a broad class of geometric set-cover and sensor placement problems.