Input Space Mode Connectivity

• Input space mode connectivity is defined as the existence of continuous, low-loss pathways between distinct input configurations in high-dimensional systems such as deep neural networks and optical devices.
• Empirical methods like Bézier path optimization in neural networks demonstrate near-100% adversarial success, outperforming linear interpolation by effectively exploiting the manifold structure.
• This connectivity insight informs practical adversarial defenses and enhances model interpretability while guiding the design and analysis of efficient optical systems through singular value decomposition.

Input space mode connectivity refers to the existence and characterization of continuous, often low-loss or functionally invariant, paths between distinct input modes (i.e., input configurations or perturbations) within the input domain of high-dimensional systems such as deep neural networks or linear optical devices. The concept generalizes the study of mode connectivity from parameter space to input space, revealing geometric structures and manifold connectivity that inform both theoretical understanding and practical algorithms for search, optimization, and robustness analysis.

1. Formal Definitions and Theoretical Foundations

Input-space mode connectivity is most rigorously defined in the context of decision functions or operators acting on input Hilbert spaces. For deep neural networks, let $X = \mathbb{R}^{d_X}$ denote input space, $Y \subseteq \mathbb{R}^{d_y}$ the output space, and $f(\cdot;\theta): X \to Y$ a fixed model. An input-space mode for class $y_i$ is defined as any minimizer $$x_i = \arg\min_{x \in X} \mathcal{L}(f(x;\theta), y_i)$$, for a given loss $\mathcal{L}$ such as cross-entropy.

Two such modes $x_a$, $x_b$ are $\delta$-connected if there exists a continuous path $\gamma: [0,1] \to X$ with $Y \subseteq \mathbb{R}^{d_y}$0, $Y \subseteq \mathbb{R}^{d_y}$1, such that $Y \subseteq \mathbb{R}^{d_y}$2 for all $Y \subseteq \mathbb{R}^{d_y}$3. In adversarial robustness, mode connectivity is formulated in terms of adversarial perturbations $Y \subseteq \mathbb{R}^{d_y}$4, $Y \subseteq \mathbb{R}^{d_y}$5 of a given input $Y \subseteq \mathbb{R}^{d_y}$6: they are said to be connected if there exists a continuous path $Y \subseteq \mathbb{R}^{d_y}$7 with $Y \subseteq \mathbb{R}^{d_y}$8, $Y \subseteq \mathbb{R}^{d_y}$9, such that $f(\cdot;\theta): X \to Y$0 and $f(\cdot;\theta): X \to Y$1 for all $f(\cdot;\theta): X \to Y$2 (Kim et al., 18 May 2026, Vrabel et al., 2024).

In linear optical systems, the device operator $f(\cdot;\theta): X \to Y$3 between input Hilbert space $f(\cdot;\theta): X \to Y$4 and output $f(\cdot;\theta): X \to Y$5 is defined such that $f(\cdot;\theta): X \to Y$6, with mode connectivity formalized via the singular value decomposition (SVD) of $f(\cdot;\theta): X \to Y$7. An orthonormal set of input modes $f(\cdot;\theta): X \to Y$8 is mapped to corresponding output modes $f(\cdot;\theta): X \to Y$9 with coupling strengths given by singular values $y_i$0 (Miller, 2012).

2. Empirical and Algorithmic Approaches

Neural Networks and Adversarial Examples

Empirical studies demonstrate that input-space modes—both natural and synthetic—are typically connected by continuous paths with low loss. For real inputs on ImageNet and CIFAR, linear interpolants between $y_i$1 and $y_i$2 typically reveal a barrier (local loss increase), quantified by the maximum gap along $y_i$3. Optimizing barriers perpendicular to the linear direction yields piecewise-linear, low-loss paths (Vrabel et al., 2024).

In adversarial search, evolutionary algorithms can exploit mode connectivity by replacing discrete crossover with continuous Bézier paths. Mode Connectivity Evolutionary Attack (MoCo-EA) constructs Bézier curves between parent perturbations and optimizes intermediate control points to maximize the expected adversarial loss along the curve, ensuring all sampled offspring are valid adversarial examples (Kim et al., 18 May 2026).

Linear Optical Devices

For linear optical devices, mode connectivity is operationalized through the SVD of the device operator $y_i$4, which enables the direct association of input and output modes, and quantifies the transmission or conversion efficiency for each mode through the singular values $y_i$5 (Miller, 2012).

3. Geometric and High-Dimensional Phenomena

Theoretical analysis suggest that mode connectivity in input space is strongly influenced by high-dimensional geometry and percolation theory. Specifically, for random networks under mild Lipschitz continuity, the probability that two modes are $y_i$6-connected approaches unity exponentially fast with increasing input dimension $y_i$7 (Vrabel et al., 2024). This percolation-based argument implies that most real and synthetic input modes are joined by continuous, near-linear, low-loss pathways in high-dimensional spaces.

In adversarial contexts, quadratic Bézier curves are shown to support high attack success rates (ASR). For instance, MoCo-EA achieves $y_i$8100% ASR for image-wise and class-wise mode pairs, and 97–99% for cross-class pairs under various $y_i$9 norms. Linear interpolation, by contrast, yields much lower ASR (12–37%) in challenging settings, confirming that true adversarial connectivity requires path optimization beyond linear blending (Kim et al., 18 May 2026).

4. Practical Implications and Applications

Adversarial Attack and Defense

Exploitation of input-space mode connectivity enables more efficient, transferable, and reliable black-box and white-box adversarial attacks. Intermediate points along optimized continuous paths often have higher transferability than the endpoint attacks, and adversarial evolutionary search with Bézier crossover dramatically reduces generations and queries compared to traditional genetic algorithms (e.g., 1.7 vs 367.9 generations; 628 vs 12,329 queries on CIFAR-10) while achieving 100% success (Kim et al., 18 May 2026).

For defense, the geometric structure of adversarial manifolds necessitates robust methods that disrupt connectivity—by smoothing decision boundaries, inserting barriers, or otherwise collapsing high-loss tunnels in the input space. Detection algorithms exploiting barrier statistics along paths between candidate inputs and class-representative modes can distinguish adversarial from natural examples, outperforming previous feature-based methods (AUC of 98.3% on C&W adversarial detection on CIFAR-10) (Vrabel et al., 2024).

Interpretability and Visualization

Mode connectivity facilitates mapping and exploration of the “optimal input manifold” for each class, illuminating smooth interpolations between real and synthetic prototypes. This reveals the feature structures relied upon by the model and enables interpretability analyses of deep networks (Vrabel et al., 2024).

Linear Devices: Design and Analysis

In optics, diagonalization of the device operator via the SVD provides direct access to efficient channels, guides the shaping of index profiles or geometries, and quantifies misalignment tolerance—e.g., coupling efficiency under spatial displacement is given by the squared overlap of displaced and ideal input modes (Miller, 2012).

5. Representative Results and Metrics

Empirical, theoretical, and application-relevant statistics for input-space mode connectivity are summarized as follows:

Domain	Connectivity Metric	Key Statistics and Outcomes
Neural Networks	Barrier gap (real–real vs real–adv)	Real–real median ≈0.47; real–adv median ≈5.30 (Vrabel et al., 2024)
Adversarial Examples	ASR on Bézier vs linear path (classwise/cross-class)	Bézier: 97–99%; Linear: 12–37% (Kim et al., 18 May 2026)
Adversarial Detection	Accuracy/AUC on CIFAR-10 (DeepFool/C&W)	≈93.7%/98.3% (outperforms LID, Mahalanobis, MFS) (Vrabel et al., 2024)
Optical Devices	Singular value $$x_i = \arg\min_{x \in X} \mathcal{L}(f(x;\theta), y_i)$$0 (channel efficiency)	$$x_i = \arg\min_{x \in X} \mathcal{L}(f(x;\theta), y_i)$$1: efficient; $$x_i = \arg\min_{x \in X} \mathcal{L}(f(x;\theta), y_i)$$2: lossy (Miller, 2012)

Efficient manipulation of these metrics in both attack and defense pipelines is enabled directly by algorithms leveraging mode connectivity.

6. Extensions, Limitations, and Future Directions

Input-space mode connectivity is not restricted to vision tasks. Extensions to text, graph, and non-Euclidean input domains are anticipated, with open questions regarding the impact of architecture, training, and input domain geometry. In optical systems, while lossless combining of orthogonal input modes into one output mode is impossible for linear devices, the singular-value framework provides quantitative tools for multiplexing and tolerance analysis (Miller, 2012).

A plausible implication is that, in high-dimensional models, input-space connectivity phenomena are generic rather than accidental, and may pose a challenge for traditional robustness and attack paradigms. Higher-order curve-based crossovers, manifold-aware sampling, and geometric adversarial defenses represent promising avenues for exploiting and mitigating mode connectivity across modalities (Kim et al., 18 May 2026, Vrabel et al., 2024).

7. Summary and Conceptual Impact

Input-space mode connectivity reveals that, contrary to traditional views of isolated input optima or adversarial points, the input space of high-dimensional models is structured as a set of richly connected manifolds. This connectivity has concrete implications: it enables efficient search and adversarial attack, informs model interpretability, underpins new detection mechanisms, and provides design principles for engineered systems. The shift from viewing input optima as isolated to viewing them as manifold-connected transforms both practical methodology and theoretical understanding across domains (Kim et al., 18 May 2026, Vrabel et al., 2024, Miller, 2012).