Strategic Classification with Non-Linear Classifiers

Abstract: In strategic classification, the standard supervised learning setting is extended to support the notion of strategic user behavior in the form of costly feature manipulations made in response to a classifier. While standard learning supports a broad range of model classes, the study of strategic classification has, so far, been dedicated mostly to linear classifiers. This work aims to expand the horizon by exploring how strategic behavior manifests under non-linear classifiers and what this implies for learning. We take a bottom-up approach showing how non-linearity affects decision boundary points, classifier expressivity, and model classes complexity. A key finding is that universal approximators (e.g., neural nets) are no longer universal once the environment is strategic. We demonstrate empirically how this can create performance gaps even on an unrestricted model class.

• The paper demonstrates that strategic behavior transforms non-linear decision boundaries via effects like wipeout, expansion, and collision.
• It quantifies how user adaptations alter curvature, influencing both the structure and complexity (VC dimension) of effective classifiers.
• Notably, universal approximator models lose their universal properties in strategic settings, inherently capping achievable accuracy.

This paper, "Strategic Classification with Non-Linear Classifiers" (2505.23443), investigates how strategic user behavior—where users modify their features to achieve a positive classification—impacts non-linear classifiers. The research moves beyond the common focus on linear models in strategic classification, providing a foundational understanding of how non-linearity alters decision boundaries, classifier expressivity, and model class complexity. A key takeaway is that universal approximators like neural networks lose their universal approximation capabilities in strategic environments.

The authors adopt a bottom-up approach, analyzing the transformation from non-strategic to strategic settings at three levels: individual points, entire classifiers, and model classes.

1. Point-Level Analysis:

When users respond strategically to a non-linear classifier $h$ (with $\ell_2$ cost for feature modification, parameter $\alpha$), the mapping of points from the original decision boundary of $h$ to the effective decision boundary of $h^\Delta$ (the classifier accounting for strategic responses) is not always straightforward. Four cases are identified:

• One-to-one: A typical case where a smooth point $x$ on $h$'s boundary maps to a single point $\hat{x} = x - \alpha n_x$ on $h^\Delta$'s boundary, where $n_x$ is the unit normal at $x$.
• Wipeout: Points on $h$'s boundary may have no corresponding point on $h^\Delta$'s boundary. This can be:
  • Direct wipeout: Occurs if the signed curvature at $x$ is too negative (intuitively, $h$ curves too sharply away from the positive region).
  • Indirect wipeout: Occurs if the set of points that would map to $x$ are "closer" (in terms of strategic cost) to another part of $h$'s decision boundary.
• Expansion: Non-smooth points (kinks/corners) on $h$'s boundary that are locally convex towards the positive region can map to a continuous arc on $h^\Delta$'s boundary.
• Collision: Multiple distinct points on $h$'s boundary can map to a single point on $h^\Delta$'s boundary, often creating non-smooth kinks in $h^\Delta$ even if $h$ was smooth.

These point mappings are illustrated in Figure 1 of the paper:

Original h boundary | Strategic movement | Effective h_delta boundary --------------------|--------------------|-------------------------- Smooth | Normal shift | Smooth (shifted) Sharp negative curve| Movement "overruns"| Point disappears (wipeout) Sharp positive kink | Movement from kink | Arc forms (expansion) - | Multiple points aim | Kink forms (collision) | for same target |

2. Classifier-Level Analysis:

The aggregation of point-level effects alters the entire decision boundary.

• Curvature Changes: The paper characterizes how signed curvature $\kappa$ of $h$ at point $x$ maps to the effective curvature $\kappa^\Delta$ of $h^\Delta$ at the corresponding point $\hat{x}$ (Proposition 1):

  $\kappa^\Delta = \kappa / (1 + \alpha \kappa)$

  This implies:

  • The sign of curvature is preserved.
  • Positive curvature decreases ($\kappa^\Delta < \kappa$ for $\kappa > 0$).
  • Negative curvature magnitude increases ($|\kappa^\Delta| > |\kappa|$ for $\kappa < 0$).
  • Effective positive curvature $\kappa^\Delta$ is upper-bounded by $1/\alpha$. No $h^\Delta$ can have positive curvature greater than this.
  • If original negative curvature $\kappa < -1/\alpha$, the point is wiped out.
  • Points with $\kappa = -1/\alpha$ can map to non-smooth kinks on $h^\Delta$.
• Containment Effects:
  • Small negative regions in $h$ can completely disappear in $h^\Delta$ if they are contained within a ball of radius $\alpha$ from some point.
  • Strategic behavior can merge distinct positive regions or split negative regions, changing the topology of the decision boundary.
• Impossible Effective Boundaries: Due to these effects, certain decision boundary shapes are impossible for $h^\Delta$. Propositions 2 and 3 provide conditions for when a candidate classifier $g$ cannot be an effective classifier. Examples of impossible $h^\Delta$ (illustrated in Figure 2) include:
  1. Classifiers with a small positive region (enclosed by a ball of radius $\alpha$).
  2. Classifiers with a narrow positive strip (where points are $\alpha$-close to the negative region on both sides).
  3. Classifiers with positive signed curvature greater than $1/\alpha$.
  4. Classifiers with locally convex piecewise linear segments pointing towards the positive region. The mapping $h \mapsto h^\Delta$ is not bijective; many $h$ can map to the same $h^\Delta$, and many potential $h^\Delta$ shapes have no corresponding $h$.

3. Model Class-Level Analysis:

The non-bijective nature of $h \mapsto h^\Delta$ impacts the complexity (e.g., VC dimension) of the effective model class $H^\Delta = \{h^\Delta : h \in H\}$.

• Complexity Reduction:

  • VC dimension can decrease. For instance, for classes $Q^k_s$ of negative-inside $k$-vertex polytopes bounded by radius $s$, $VC(Q^k_s) \ge VC((Q^k_s)^\Delta)$ (Proposition 4).
  • Drastic reductions are possible: Theorem 1 shows classes $H$ exist where $VC(H)=\infty$ but $VC(H^\Delta)=0$. This means a non-learnable class in the standard setting might become learnable strategically (Corollary 1). An example is a class of classifiers with many tiny, disjoint negative regions; strategically, these all become positive, making all effective classifiers identical (always positive).
• Complexity Increase:
  • VC dimension can also increase. An example is provided where non-overlapping classifiers in $H$ result in overlapping effective classifiers in $H^\Delta$, increasing shattering capacity.
  • Theorem 2 states that if $H$ is closed under input scaling (e.g., polynomials, many neural networks) and $VC(H) < \infty$, then $VC(H) \le VC(H^\Delta)$.
  • For polytope classes $P^k$ (positive-inside) under $\ell_1, \ell_2, \ell_\infty$ costs, $VC(H^\Delta) = O(kd^2 \log k)$, which is the same order as $VC(P^k)$, suggesting increases might be bounded (Theorem 3).
  • The authors conjecture that if $H$ is learnable, $H^\Delta$ is also learnable.
• Universality and Approximation:
  • A crucial finding is that universal approximator classes (e.g., neural networks, RBF kernels) are no longer universal under strategic behavior (Corollary 2). This is a direct consequence of "impossible effective boundaries."
  • This implies an approximation gap: even with an unrestricted hypothesis class, the maximum achievable strategic accuracy might be less than 1, even if the data is perfectly separable in a non-strategic setting. Figure 3 (right) illustrates a dataset where strategic responses inherently limit accuracy.
  • Over-parameterized models that interpolate data in standard settings may not be as effective strategically, as intricate decision boundaries might be "smoothed out" or become impossible.
  • Proposition 5 shows distributions exist that are realizable in the standard setting but strategic accuracy degrades to the majority class rate.
  • Proposition 6 gives an example where maximum strategic accuracy approaches 0.5 as $\alpha$ increases.

Implementation Considerations and Practical Implications:

• Model Selection: The choice of non-linear model class becomes more critical. Knowing that certain boundary shapes are impossible for $h^\Delta$ can guide model selection. For instance, if the problem inherently requires a sharp positive curvature, it might be unattainable strategically.
• Loss Functions and Optimization: Standard loss functions and optimization procedures might need adaptation. The paper primarily analyzes the structure of the effective classifier, but optimizing for $h$ to achieve good $h^\Delta$ is a challenge.
• Expected Accuracy: Practitioners should be aware that strategic behavior can inherently limit the maximum achievable accuracy, regardless of model complexity or data size.
• Cost Parameter $\alpha$: The value of $\alpha$ (user's budget for manipulation) is critical. It determines the scale of boundary deformation, the $1/\alpha$ limit on positive curvature, and the size of features that can be "wiped out." Estimating or designing systems robust to a range of $\alpha$ is important.
• Feature Engineering: Understanding how features might be manipulated and how this interacts with non-linear boundaries can inform feature engineering to create more robust systems.
• Deployment: Models deployed in environments where users can adapt strategically (e.g., loan applications, spam filtering, college admissions) must account for these non-linear effects. The common practice of using powerful non-linear models needs to be re-evaluated in light of their reduced expressiveness in strategic settings.

Experiments:

Two sets of experiments on synthetic data illustrate the theoretical findings:

1. Expressivity of Polynomial Classifiers:
   • They took random polynomial classifiers $h$ of degree $k$ and found the polynomial degree $k_\Delta$ that best approximates the resulting $h^\Delta$.
   • For small $\alpha$, $k_\Delta \approx k$.
   • For moderate $\alpha$, $k_\Delta > k$, suggesting $h^\Delta$ can become more complex (e.g., due to new cusps from point collisions).
   • For large $\alpha$ and high $k$, $k_\Delta$ can drop, as large parts of the original boundary are wiped out, simplifying $h^\Delta$.
2. Approximation Limits:
   • They generated data from class-conditional Gaussians with varying separation $\mu$.
   • They compared the maximum strategic accuracy of an unrestricted effective class ($H^*_\Delta$) against standard accuracy ($H^*$, always 1 for separable data) and linear strategic accuracy ($H_{lin}^\Delta$).
   • As class separation $\mu$ decreased (harder problem), the gap between $H^*$ and $H^*_\Delta$ increased, especially for larger $\alpha$. This shows the performance degradation due to strategic behavior.
   • $H^*_\Delta$ still outperformed $H_{lin}^\Delta$, indicating non-linear models can still offer benefits over linear ones, even if their capabilities are curtailed.

In summary, this research provides a comprehensive analysis of how non-linear classifiers behave in strategic settings. It highlights fundamental limitations, such as the loss of universality for powerful model classes, and provides mechanisms (curvature changes, wipeouts, expansions) to understand how decision boundaries are reshaped. These insights are crucial for developers aiming to build robust and fair AI systems in real-world applications where users may adapt to the deployed models.