First-Order Influence Objective

• First-order influence objective is a framework that uses linear approximations to quantify the marginal effect of individual components on global outcomes in systems like machine learning, networks, and physics.
• Methodologies include gradient inner products (TracIn/IP), subcritical path enumeration in cascading networks, and Sobol variance-based indices, enabling efficient and scalable influence estimation.
• Practical applications span from noisy label detection and adversarial robustness in deep learning to seed selection in network cascades and tuning vibrational modes in materials, with promising extensions toward higher-order interactions.

The term "first-order influence objective" encompasses several rigorous mathematical constructs and practical algorithms used to quantify the marginal effect of a system component—such as a data sample, node, or external field—on a collective property or global functional, under linear (first-order) approximations. This article surveys the main research threads across machine learning, network science, and materials physics, emphasizing formal definitions, computational methodologies, theoretical properties, and empirical validations.

1. Formal Definitions and Mathematical Foundations

The first-order influence objective arises from using first-order expansions (linearizations) to approximate the change in a target metric due to a perturbation of an individual element:

• Model Influence (Machine Learning): For a loss function $\ell(z;\theta)$ with optimal parameters $\hat\theta$, and a global functional $f(\theta)$ (e.g., validation loss), the classical influence function measures the effect of upweighting or removing a sample $z_i$:

$$I^{util}(-z_i) \approx -\nabla_\theta f^{util}(\hat\theta)^\top \hat{H}^{-1} \nabla_\theta \ell(z_i; \hat\theta)$$

where $\hat{H}$ is the Hessian of the average training loss. The first-order (Hessian-free) objective simplifies this by taking $\hat{H}^{-1} \approx I_p$, yielding the "Inner Product" (IP) or TracIn score:

$$I_{IP}^{util}(z_i) = -\nabla_\theta f^{util}(\hat\theta)^\top \nabla_\theta \ell(z_i; \hat\theta)$$

with efficient extensions for checkpoint ensembles and specific target samples (Yang et al., 2024).

• Collective Influence in Cascading Networks: In threshold-based cascade processes (e.g., Linear Threshold Model), the first-order objective quantifies the marginal contribution of a "seed" node to global activation through enumeration of "subcritical paths"—chains of nodes exactly one below their threshold:

$$CI\text{--}TM_L(i) = \text{number of subcritical paths of length } \leq L \text{ emanating from } i$$

and for a seed set $S$,

$$CI\text{--}TM_L(S) = \sum_{i\in S} CI\text{--}TM_L(i)$$

which directly approximates the linearized impact of each seed on cascade size (Pei et al., 2016).

• Variance-Based First-Order Indices (Influence Maximization): In stochastic settings, the first-order effect of a binary variable $X_i$ (e.g., seed inclusion) on a function $f(X)$ (e.g., expected spread) is given by the first-order Sobol index:

$$S_i = \frac{\mathrm{Var}_{X_i}\big[\,\mathbb{E}_{X_{\setminus i}}(f(X)|X_i)\,\big]}{\mathrm{Var}(f(X))}$$

isolating its average effect independent of interactions with other variables (Zhang et al., 2022).

• First-Order Physical Influence (Phonons in Materials): In lattice dynamics, the response of vibrational modes (phonons) to an external field is captured by the first-order term in the dispersion relation. For SWNTs under axial load:

$$\omega^2(k) = \frac{F}{\mu} k^2 + \frac{EI}{\mu} k^4 + O(k^6)$$

with the $k^2$ term providing a direct, first-order "influence" of force $F$ on the TA mode spectrum (Aghaei et al., 2012).

2. Methodological Approaches

Various computational strategies exist for realizing first-order influence objectives:

• Gradient Inner Products (TracIn/IP): The TracIn protocol computes summed inner products of gradients between training and test samples, optionally accumulated over multiple checkpoints:

$$I_{\mathrm{TracIn}}(z_i, z_t) = -\sum_{k=0}^{T-1} \eta_k \nabla_\theta \ell(z_i;\theta_k)^\top \nabla_\theta \ell(z_t;\theta_k)$$

enabling efficient Hessian-free influence attribution in deep learning (Yang et al., 2024).

• Subcritical Path Enumeration: The CI–TM objective in threshold models is efficiently optimized using a greedy O($N\log N$) algorithm that prioritizes seeds with maximal numbers of short subcritical paths, combining tree-based message passing with heap updates (Pei et al., 2016).
• Sensitivity Decomposition: For small-scale stochastic models, all $2^k$ possible configurations of $k$ potential seeds are simulated to estimate the first-order Sobol indices, decomposing the variance of influence spread into additive and interaction components (Zhang et al., 2022).
• Continuum and Lattice Calculations: In physics settings, explicit expansion of the relevant energy functional or lattice-dynamics equations yields closed-form expressions for first-order effects (e.g., axial stress on phonon frequencies), validated by symmetry-adapted or group-theoretic computational techniques (Aghaei et al., 2012).

3. Extensions and Theoretical Properties

First-order influence objectives share key theoretical properties governed by the validity of linear (first-order) expansion and the control of higher-order terms:

• Order-Consistency and Robustness: In convex loss landscapes, the identity approximation to the Hessian in TracIn/IP perturbs only the orientation of the validation gradient by a small angle, preserving the sign (beneficial/detrimental ordering) for all but nearly orthogonal gradients. The error is bounded by the spectral norm $\|\hat{H}^{-1} - I\|$, which is small near convex optima (Yang et al., 2024).
• Interplay with Higher-Order Effects: First-order objectives ignore interactions between components (e.g., overlapping seed influence), leading to potential overestimation of total effect. The full ANOVA or Sobol decomposition partitions these effects, with higher-order terms quantifying synergy or redundancy. For influence maximization, ranking nodes by first-order indices gives a pure but sometimes misleading ordering (Zhang et al., 2022).
• Physical First-Order Terms: In SWNTs, the term $F(w')^2$ in the deformation energy functional directly encodes the linear influence of axial force on the TA dispersion curve, shifting it from quartic ($k^4$) to quadratic ($k^2$) behavior, with profound consequences for density of states and transport (Aghaei et al., 2012).

4. Practical Applications and Empirical Evaluations

First-order influence objectives have demonstrated significant practical value in multiple domains:

Application Domain	First-Order Objective	Key Outcome/Metric
Deep Learning	TracIn/IP	Noisy label removal, adversarial defense, fairness
Network Cascades	CI–TM (subcritical paths)	Seed selection for information/behavioral cascades
Social Influence	Sobol index $S_i$	Diagnostic ranking, influence maximization (seed set)
Materials Physics	$F$-term in phonon dispersion	Tunable vibrational modes in SWNTs

• Noisy Label Detection: Removing the most detrimental samples using TracIn/IP Ensemble improves accuracy on noisy datasets (e.g., CIFAR-10N), with Ensemble IP reaching 82.93% versus 80.90% for vanilla cross-entropy and outperforming Hessian-based methods at >100× speed-up (Yang et al., 2024).
• Adversarial Robustness: First-order influence-guided sample removal or reweighting is effective for defending against adversarial attacks, yielding the highest average post-attack accuracy on several classification benchmarks (Yang et al., 2024).
• Fairness-Critical Curation: In fine-tuning LLMs, first-order influence objectives enable selection of data subsets that achieve simultaneous improvements in both accuracy and demographic parity fairness, a property not observed for other influence estimators (Yang et al., 2024).
• Influence Maximization: CI–TM achieves lower critical seed fractions $q_c$ for global cascades in synthetic and large-scale real networks, outperforming degree, PageRank, and other scalability-focused heuristics (Pei et al., 2016).
• Physical Modulation: In SWNTs, varying the axial load continuously tunes the low-frequency phonon dispersion and density of states, with quadratic-to-linear transitions observed experimentally and reproduced by symmetry-adapted phonon calculations (Aghaei et al., 2012).

5. Variants, Generalizations, and Limitations

Researchers have developed several ensemble and extension strategies to improve or adapt first-order objectives:

• Checkpoint and Dropout Ensembles: Averaging TracIn/IP scores across multiple training checkpoints or submodels (with dropout in parameters) increases robustness and mitigates instability in nonconvex regimes, with $M=5$ ensemble members sufficing for consistent gains (Yang et al., 2024).
• Generalized Impact Functions: By redefining the global metric $f$ (e.g., setting $f^{fair}$ for demographic parity, $f^{robust}$ for adversarial robustness), the underlying first-order machinery directly quantifies sample influence on fairness or robustness objectives, enabling multi-objective selection (Yang et al., 2024).
• Path Length Truncation in Cascades: In CI–TM, the practical depth $L$ for subcritical path enumeration can be truncated to a small constant for scalability, without significant loss in accuracy for large graphs (Pei et al., 2016).
• First-Order vs. Higher-Order Indices: The limitation of first-order objectives is their neglect of interactions: higher-order Sobol indices capture multi-seed synergy and redundancy. For small seed budgets, exact enumeration enables computation of all variance components, but for large $k$ only first-order indices are scalable (Zhang et al., 2022).

6. Significance and Outlook

First-order influence objectives unify a wide range of influence estimation, sensitivity analysis, and control tasks across machine learning, network science, and materials physics. Their computational efficiency, direct interpretability, and theoretical guarantees (under weak assumptions) have led to adoption in large-scale practical settings—often enabling analyses previously infeasible with classical second-order (Hessian-based) methods.

Recent ensemble and multi-metric extensions further expand the scope of these methods, making them central tools for robust, fair, and efficient data selection, model interpretation, and intervention design. Ongoing work aims to refine their accuracy in highly interactive or non-linear regimes by systematically incorporating higher-order corrections or leveraging randomized estimators for partial higher-order information.

References:

• "Revisit, Extend, and Enhance Hessian-Free Influence Functions" (Yang et al., 2024)
• "Anomalous phonon behavior of carbon nanotubes: First-order influence of external load" (Aghaei et al., 2012)
• "Efficient collective influence maximization in cascading processes with first-order transitions" (Pei et al., 2016)
• "Understanding Influence Maximization via Higher-Order Decomposition" (Zhang et al., 2022)