Fitting Problem: Inference and Model Selection

• Fitting Problem is a class of inference tasks that constructs admissible models matching observations under defined error, regularization, and structural criteria.
• Methodologies include variational formulations, parameterized error minimization, and logical conditions, addressing existence, uniqueness, and ill-conditioning in solutions.
• Applications span diverse domains such as cosmology, electrostatics, image processing, and database theory, illustrating both theoretical challenges and practical optimization strategies.

The fitting problem denotes a broad class of inference tasks in which one constructs an admissible object that matches observations, examples, or measurements under an explicit criterion. In recent work, the fitted object may be a Sobolev function, an ultrametric, a geometric model, a conjunctive query, an ontology, a set of tuple-generating dependencies, a point-charge distribution, or an idealized FLRW background. Across these settings, the recurring technical issues are existence, uniqueness, conditioning, approximation error, regularization, and the relation between the fitted object and the semantics of the underlying model class (Dachiraju, 2018, Cate et al., 2022, Yao et al., 2023, Deledicque, 2022).

1. Variational, algebraic, and logical formulations

One canonical formulation is variational. In "A Function Fitting Method" the domain is the open unit cube $\Omega=(0,1)^m$ with periodic boundary conditions, and the fitting functional is

$C(f)=\sum_{i=1}^n (f(p_i)-a_i)^2+\|f\|_{T^k}^2,$

where $$\|f\|_{T^k}^2=\|f\|_{L^2(\Omega)}^2+\lambda\|\nabla^k f\|_{L^2(\Omega)}^2$$, $k>m/2$, and the admissible class is $S=M\cap H^k(\Omega)$. Because $C$ is strictly convex and coercive on $S$, it has a unique minimizer $f_\lambda\in S$, and the Euler–Lagrange condition yields a linear, weak partial differential equation with point-mass terms (Dachiraju, 2018).

A second canonical formulation is error minimization over a parameterized fitting map. In the generalized $L_1$-norm error fitting model, the objective is to minimize $\|B-f(X,A)\|_{1,1}$ subject to $C(f)=\sum_{i=1}^n (f(p_i)-a_i)^2+\|f\|_{T^k}^2,$0, under the associativity assumption $C(f)=\sum_{i=1}^n (f(p_i)-a_i)^2+\|f\|_{T^k}^2,$1. This encompasses ordinary $C(f)=\sum_{i=1}^n (f(p_i)-a_i)^2+\|f\|_{T^k}^2,$2 regression, $C(f)=\sum_{i=1}^n (f(p_i)-a_i)^2+\|f\|_{T^k}^2,$3 regression subset-selection, $C(f)=\sum_{i=1}^n (f(p_i)-a_i)^2+\|f\|_{T^k}^2,$4 regression over a sphere, orthogonal Procrustes, and several $C(f)=\sum_{i=1}^n (f(p_i)-a_i)^2+\|f\|_{T^k}^2,$5-PCA variants (Park, 2017). This suggests a generic template: a fitting problem is determined by a model class, an admissibility constraint, and either an error functional or a separation criterion.

A third formulation is logical rather than metric. For conjunctive queries, a CQ $C(f)=\sum_{i=1}^n (f(p_i)-a_i)^2+\|f\|_{T^k}^2,$6 fits $C(f)=\sum_{i=1}^n (f(p_i)-a_i)^2+\|f\|_{T^k}^2,$7 when every positive example $C(f)=\sum_{i=1}^n (f(p_i)-a_i)^2+\|f\|_{T^k}^2,$8 satisfies $C(f)=\sum_{i=1}^n (f(p_i)-a_i)^2+\|f\|_{T^k}^2,$9 and every negative example $$\|f\|_{T^k}^2=\|f\|_{L^2(\Omega)}^2+\lambda\|\nabla^k f\|_{L^2(\Omega)}^2$$0 satisfies $$\|f\|_{T^k}^2=\|f\|_{L^2(\Omega)}^2+\lambda\|\nabla^k f\|_{L^2(\Omega)}^2$$1. By the Chandra–Merlin theorem, this is equivalent to the homomorphism conditions

$$\|f\|_{T^k}^2=\|f\|_{L^2(\Omega)}^2+\lambda\|\nabla^k f\|_{L^2(\Omega)}^2$$2

When a fitting CQ exists, it is in general not unique, which motivates the refined notions of most-general fitting CQ, most-specific fitting CQ, and unique fitting CQ (Cate et al., 2022).

2. Ill-conditioning, sloppiness, and the geometry of fit landscapes

A central difficulty is that poor numerical behavior often reflects the geometry of the fitting problem rather than a defect of a particular optimizer. In covariance-weighted fitting, the objective

$$\|f\|_{T^k}^2=\|f\|_{L^2(\Omega)}^2+\lambda\|\nabla^k f\|_{L^2(\Omega)}^2$$3

can be rewritten as

$$\|f\|_{T^k}^2=\|f\|_{L^2(\Omega)}^2+\lambda\|\nabla^k f\|_{L^2(\Omega)}^2$$4

after diagonalizing $$\|f\|_{T^k}^2=\|f\|_{L^2(\Omega)}^2+\lambda\|\nabla^k f\|_{L^2(\Omega)}^2$$5. If one direction has $$\|f\|_{T^k}^2=\|f\|_{L^2(\Omega)}^2+\lambda\|\nabla^k f\|_{L^2(\Omega)}^2$$6, even a tiny projection onto that eigenvector produces a large contribution to $$\|f\|_{T^k}^2=\|f\|_{L^2(\Omega)}^2+\lambda\|\nabla^k f\|_{L^2(\Omega)}^2$$7. The basic pathology is not simply noisy data: the paper states that the essence of the problem is that we do not have an accurate fitting function enough to fit extremely precise data (Yoon et al., 2011).

The same phenomenon appears in inverse problems. In electrostatic point-charge fitting, least-squares estimation of charges $$\|f\|_{T^k}^2=\|f\|_{L^2(\Omega)}^2+\lambda\|\nabla^k f\|_{L^2(\Omega)}^2$$8 solves $$\|f\|_{T^k}^2=\|f\|_{L^2(\Omega)}^2+\lambda\|\nabla^k f\|_{L^2(\Omega)}^2$$9, with Hessian $k>m/2$0. The ill-conditioning is traced to the decay of the curvatures corresponding to the eigenvectors of the least-squares Hessian matrix. In the analytically soluble two-sphere model, the singular values associated with multipole order $k>m/2$1 decay geometrically, so high-order multipole directions are intrinsically soft. For atom-centered point-charge models, the association between eigenvectors and multipole moments breaks down beyond the first few modes, and the curvature spectrum spreads even more widely (Ivanov et al., 2015).

Nonlinear least-squares fitting exhibits an analogous geometric structure. The model manifold in data space has a hierarchy of widths, is tightly bounded, and typically has low extrinsic curvature. The induced metric is $k>m/2$2, so the same stiff-versus-sloppy decomposition appears in the eigenvalues of $k>m/2$3. The consequence is that algorithms move toward manifold boundaries where the model becomes unresponsive to parameter changes. In this setting, adding geodesic acceleration to the Levenberg–Marquardt step improved convergence, raising the success rate to $k>m/2$4 in the reported test problem while cutting the average Jacobian calls by almost an order of magnitude (Transtrum et al., 2009).

3. Regularization, projection, and optimization strategies

Because conditioning is often the governing issue, many fitting methods are defined as much by their stabilization strategy as by their nominal objective. For Sobolev-space fitting, the weak-PDE formulation of $k>m/2$5 admits both a representer form

$k>m/2$6

and a linear system $k>m/2$7, as well as a gradient-descent scheme in the Fourier domain. The paper also proposes choosing the regularization parameter by

$k>m/2$8

As $k>m/2$9, the minimizer interpolates the data but collapses to zero almost everywhere; as $S=M\cap H^k(\Omega)$0, the solution is trivial (Dachiraju, 2018).

In covariance fitting of highly correlated data, the classical responses are the diagonal approximation, the cut-off method, the modified covariance-matrix method, and the Bayesian augmented-$S=M\cap H^k(\Omega)$1 approach. The proposed Eigenmode-Shift method leaves the covariance matrix untouched and instead augments the theory by shifts along the troublesome eigenmodes:

$S=M\cap H^k(\Omega)$2

with priors on the $S=M\cap H^k(\Omega)$3. In the pedagogical example, diagonal and cut-off methods fail in fitting manifestly, but the ES method and the Bayesian approach work well (Yoon et al., 2011).

Other stabilization strategies replace constraints by projections or aggregation. Ultrametric fitting converts the constrained problem over ultrametrics into an unconstrained optimization by injecting the operator $S=M\cap H^k(\Omega)$4 directly into the cost and then applying gradient descent, in practice with AMSGrad; the paper emphasizes that there are no global-optimality guarantees, but reports good empirical performance (Chierchia et al., 2019). The Aggregate-and-Iterative-Disaggregate algorithm for generalized $S=M\cap H^k(\Omega)$5 fitting clusters observations, solves aggregated subproblems, and selectively splits clusters until a sign-consistency optimality criterion is met, with monotonic convergence to a global optimum under the stated assumption on $S=M\cap H^k(\Omega)$6 (Park, 2017). In dynamic volatility fitting, variants of DDPG and Soft Actor Critic are reported to achieve at least as good as standard fitting algorithms, with the reinforcement-learning formulation motivated by complex objective functions and online learning (Gnabeyeu et al., 2024).

4. Structured geometric fitting

A large subliterature treats fitting as geometric estimation under explicit structural priors. The problem of fitting concentric ellipses is described as vital in image processing, pattern recognition, and astronomy. "Fitting Concentric Elliptical Shapes Under General Model" studies the problem under a more general setting, proposes two estimators for estimating the parameters, investigates several numerical schemes because both estimators are obtained iterative fashion, determines the best initial guess, derives the constraint Cramér–Rao lower bound, compares it with the variance of each estimator, and validates the theory on both real and synthetic data (Al-Sharadqah et al., 2023).

Manifold fitting addresses a different geometric object class. In "Manifold Fitting", the observations are $S=M\cap H^k(\Omega)$7 in ambient space $S=M\cap H^k(\Omega)$8, where the latent variables are sampled from an unknown compact $S=M\cap H^k(\Omega)$9-dimensional $C$0 submanifold. The estimator is produced by a two-step mapping approach $C$1 and $C$2, and the paper proves that, under certain mild assumptions and with sample size $C$3, the estimators are true $C$4-dimensional smooth manifolds whose Hausdorff error is bounded by $C$5 with high probability (Yao et al., 2023).

Multi-structure geometric fitting often replaces direct parameter optimization by higher-level combinatorial or learned representations. Hypergraph Modelling for Geometric Model Fitting formulates the problem as a hypergraph partition problem in which vertices represent data points and hyperedges denote model hypotheses, with large and data-determined degrees of hyperedges. The method is designed to estimate the number of, and the parameters of, model instances in multi-structural data heavily corrupted with outliers simultaneously (Xiao et al., 2016). By contrast, "Learning for Multi-Model and Multi-Type Fitting" formulates the problem as learning deep feature embedding that is clustering-friendly, followed by K-means in the learned feature space and model selection by analyzing the K-means residuals (Xu et al., 2019).

A related example is cylindrical regression in three-dimensional Euclidean space. The minimum root mean square deflection formulation has no analytic solution, but replacing root mean square averaging by a certain biquadratic averaging yields an almost analytic solution in coordinate-free form (Ageev et al., 2019). This suggests a recurring pattern in geometric fitting: exact structural constraints are often preserved by altering the objective, the parameterization, or the representation rather than by solving the original formulation directly.

5. Extremal, relational, and algebraic notions of fitting

In database theory and knowledge representation, fitting is primarily a separability notion. For conjunctive queries, structural characterizations relate refined fitting notions to homomorphism dualities, frontiers, and direct products. The core of the direct product of positive examples yields a most-specific fitting, the core of the union of negative examples yields a most-general fitting, and unique fitting is characterized by frontier existence. The associated existence and verification problems are highly intractable in general: the classical fitting problem for CQs is CoNExpTime-complete, while several refined problems are NExpTime-complete or CoNExpTime-complete, although tree CQs admit polynomial-time algorithms and frontier constructions of only linear blow-up (Cate et al., 2022).

The same theme extends to ontologies and constraints over relational structures. "Fitting Ontologies and Constraints to Relational Structures" studies description logics $C$6 and $C$7 and several TGD classes, pinpoints the exact computational complexity, designs algorithms, and analyzes the size of fitting ontologies and TGDs. Finite bases exist for $C$8, $C$9, guarded TGDs, and inclusion dependencies, but in general do not exist for full, frontier-guarded, and frontier-one TGDs. The complexity spectrum ranges from $S$0-complete for inclusion dependencies to coThreeNExpTime for unrestricted TGDs (Hosemann et al., 11 Aug 2025).

An algebraic variant appears in projective geometry. Given a finite set of points $S$1 in $S$2 not all contained in a hyperplane, the fitting problem asks for the maximum number $S$3 of these points that can fit in some hyperplane and for the equation(s) of such hyperplane(s). Under the $S$4-generic condition, the main theorem states

$S$5

where $S$6 is the index of nilpotency of the ideal $S$7 constructed from products of dual linear forms. In $S$8, this becomes “maximum number of collinear points = $S$9” (Tohaneanu, 2012). The result is exact, not approximate, and exemplifies a fitting problem solved by primary decomposition rather than numerical optimization.

6. Scientific applications and interpretive consequences

In several scientific domains, the principal difficulty is not merely estimating parameters but deciding what it means for a model to fit the data. The lattice-QCD covariance-fitting literature is explicit on this point: when the covariance matrix has very small eigenvalues, even a tiny error of fitting function can spoil the fitting procedure. The Eigenmode-Shift method therefore fine-tunes the fitting function while keeping the covariance matrix untouched, rather than redefining the statistical structure of the data (Yoon et al., 2011).

In molecular electrostatics, the point-charge fitting problem is treated as a discrete inverse problem. The analysis shows that the numerical problems are due to the decay of the curvatures of the Hessian eigenmodes and that a Lebedev-grid model is naturally suited for the inverse electrostatic problem. Basis truncation, quadrature choice, and total-charge correction by projection onto the leading eigenvector are presented as ways to alleviate ill-conditioning without introducing external restraints and/or constraints (Ivanov et al., 2015).

Cosmology supplies a stronger semantic challenge. "Dark energy and the fitting problem" argues that the process that fits the idealized FLRW manifold to the real universe is not uniquely defined, may depend on the cosmic probe that has been used for the measurements, and leads to different observed temporal evolutions of the scale factor. The paper establishes the fitting processes for the SNIa, BAO and CMB cosmic probes, derives related averaged Einstein equations, and demonstrates that the way these fittings have been applied in practice lead to an apparent dark energy effect; it also highlights conceptual differences between SNIa/BAO and CMB fitting and then shows how the so-called Hubble tension can be explained (Deledicque, 2022).

A complementary toy model appears in the lattice-Universe setting. There, the spacetime is kinematically equivalent to the FLRW solution with the same averaged energy density, so there is no kinematical backreaction at second order. Yet observables along the past lightcone agree perturbatively with the smoothed FLRW model only if the compacity criterion

$f_\lambda\in S$0

is satisfied. If this bound is violated, the problem must be addressed fully non perturbatively along the past lightcone, even though the spacetime geometry can be described perturbatively (Larena, 2012). This sharply separates kinematical equivalence from observational equivalence.

Taken together, these works show that the fitting problem is not a single algorithmic task but a family of model-selection and approximation problems whose mathematical content depends on the admissible object class, the error or separability criterion, and the semantics assigned to the fitted object. In some settings the central theorems concern existence, uniqueness, and complexity; in others they concern lower bounds, conditioning, projection operators, or observational interpretation. The common thread is that fitting is never just parameter estimation: it is the problem of choosing what structure is allowed to explain the data, and under what notion of explanation.