Dual-Encoder Contrastive Objectives

• Dual-encoder contrastive objectives are a framework that trains two parallel networks to encode inputs so that related pairs have closer embeddings while unrelated pairs are separated.
• They are widely applied in cross-modal retrieval and recommendation systems, leveraging paired comparisons to improve semantic alignment.
• Practical implementations include temperature scaling, momentum updates, and efficient negative sampling to optimize the balance between bias and variance in learning.

The term ABC-parametrization denotes two distinct, context-dependent statistical parameterization strategies that leverage auxiliary constraints for analytic or computational tractability. The first arises in latent variable model estimation, specifically in hidden Markov models (HMMs) with analytically intractable emission likelihoods; here, it refers to the introduction of approximate Bayesian computation (ABC) kernels and a pseudo-likelihood controlled by an $\epsilon$-parameter. The second usage appears in regression models with categorical covariates and interactions, where abundance-based constraints (ABC) define a reparametrization of categorical effects to enable efficient, interpretable, and equitable estimation. The following sections delineate the formal machinery, methodological workflow, theoretical guarantees, and empirical properties of both forms.

1. ABC-Parametrization in Hidden Markov Models

When the emission density $g_\theta(y_n|x_n)$ of an HMM cannot be evaluated in closed form but is simulable, an auxiliary-likelihood construction enables parameter estimation via ABC (Ehrlich et al., 2012). The standard HMM comprises a state space $X\subset\mathbb{R}^n$, observation space $Y\subset\mathbb{R}^m$, and static parameter $\theta\in\Theta\subset\mathbb{R}^d$. The emission process admits samples $u\sim g_\theta(\cdot|x_n)$ for any $x_n$, yet the density $g_\theta(y_n|x_n)$ is not directly available.

The ABC-parametrization replaces the intractable likelihood component $g_\theta(y_n|x_n)$ in the joint smoothing density with the ABC surrogate:

$$g_{\theta,\epsilon}(y_k|x_k) \equiv \frac{1}{C_\epsilon} \int K_\epsilon(y_k|u) g_\theta(u|x_k) du,$$

where $g_\theta(y_n|x_n)$0 is a kernel function (e.g., uniform, Gaussian) centered at $g_\theta(y_n|x_n)$1 with tolerance $g_\theta(y_n|x_n)$2, and $g_\theta(y_n|x_n)$3 normalizes the perturbation. The overall ABC-approximated marginal likelihood is:

$g_\theta(y_n|x_n)$4

where the one-step predictive densities marginalize over the intractable emission via kernel-weighted simulation.

The key theoretical result is an $g_\theta(y_n|x_n)$5 upper bound on the log-likelihood and gradient bias between the true and ABC marginal likelihood, assuming Lipschitz continuity and boundedness conditions for transition and emission densities and their parameter gradients. This ensures that, for moderate $g_\theta(y_n|x_n)$6 and $g_\theta(y_n|x_n)$7, ABC-induced error remains computationally and statistically negligible relative to particle filter Monte Carlo error.

2. Particle Filter Implementation and Parameter Estimation

Efficient computation under the ABC-parameterization is achieved via a sequential Monte Carlo (SMC) particle filter using $g_\theta(y_n|x_n)$8 particles and pseudo-observation draws:

1. Initialization: $g_\theta(y_n|x_n)$9, weights $X\subset\mathbb{R}^n$0.
2. Resampling: If the effective sample size of weights is low, resample ancestors.
3. Propagation: Propose $X\subset\mathbb{R}^n$1 and sample $X\subset\mathbb{R}^n$2.
4. Weighting: Compute $X\subset\mathbb{R}^n$3 and normalize.
5. Marginal-Likelihood Estimation: The estimated marginal contribution is $X\subset\mathbb{R}^n$4; the overall marginal likelihood estimate is the product $X\subset\mathbb{R}^n$5. A second-order bias correction may be applied to the log-likelihood.

Parameter updates are performed online using simultaneous perturbation stochastic approximation (SPSA). Two SMC filters are run at $X\subset\mathbb{R}^n$6 and $X\subset\mathbb{R}^n$7, where $X\subset\mathbb{R}^n$8 is a vector of independent Rademacher random variables. The gradient estimate for component $X\subset\mathbb{R}^n$9 is $Y\subset\mathbb{R}^m$0; parameter updates proceed as $Y\subset\mathbb{R}^m$1 with suitable diminishing step sizes $Y\subset\mathbb{R}^m$2.

3. Bias–Variance Trade-offs and Numerical Properties

Empirical studies demonstrate fundamental tradeoffs in the ABC-parameter $Y\subset\mathbb{R}^m$3 and the Monte Carlo sample size $Y\subset\mathbb{R}^m$4, as well as pseudo-observation replicate number $Y\subset\mathbb{R}^m$5:

• Bias in the marginal likelihood and parameter gradients is bounded by $Y\subset\mathbb{R}^m$6; variance increases as $Y\subset\mathbb{R}^m$7 due to weight degeneracy.
• For fixed $Y\subset\mathbb{R}^m$8, increasing $Y\subset\mathbb{R}^m$9 stabilizes particle weights but increases estimator bias; reducing $\theta\in\Theta\subset\mathbb{R}^d$0 shrinks bias but amplifies variance.
• Variance of estimates typically scales as $\theta\in\Theta\subset\mathbb{R}^d$1, with improvements for larger $\theta\in\Theta\subset\mathbb{R}^d$2.
• In practical scenarios (e.g., Lorenz '63 model), empirically optimal $\theta\in\Theta\subset\mathbb{R}^d$3 is suggested to balance bias and variance, with bias nearly linear in $\theta\in\Theta\subset\mathbb{R}^d$4 and variance inversely proportional.

A summary of empirical findings:

Setting	Bias Behavior	Variance Behavior	Notes
$\theta\in\Theta\subset\mathbb{R}^d$5	$\theta\in\Theta\subset\mathbb{R}^d$6	$\theta\in\Theta\subset\mathbb{R}^d$7	Bias–variance trade-off, “sweet-spot” for midrange $\theta\in\Theta\subset\mathbb{R}^d$8
$\theta\in\Theta\subset\mathbb{R}^d$9 (particles)	Const. bias	$u\sim g_\theta(\cdot|x_n)$0	Increasing $u\sim g_\theta(\cdot|x_n)$1 reduces estimator variance
$u\sim g_\theta(\cdot|x_n)$2 (replicates)	Stable bias for $u\sim g_\theta(\cdot|x_n)$3	Var. decreases	Redundant samples reduce Monte Carlo error

4. ABC-Parametrization for Regression with Categorical Interactions

The abundance-based constraints (ABC) parametrization for categorical-modified regression models addresses challenges inherent in traditional codings (e.g., reference-group or sum-to-zero constraints) when modeling main and interaction effects of categorical covariates (Kowal, 2024).

Given data $u\sim g_\theta(\cdot|x_n)$4 with categorical variables $u\sim g_\theta(\cdot|x_n)$5 of $u\sim g_\theta(\cdot|x_n)$6 levels, the cat-modified linear model includes main effects, categorical–continuous, and categorical–categorical interactions:

$u\sim g_\theta(\cdot|x_n)$7

ABC constraints impose that category-level effects are centered by their empirical proportions:

$u\sim g_\theta(\cdot|x_n)$8

and for categorical–categorical interactions,

$u\sim g_\theta(\cdot|x_n)$9

where $x_n$0 is the empirical proportion of group $x_n$1.

5. Estimation Invariance, Power, and Interpretation Advantages

Main effect estimates—continuous slopes and categorical effects—are preserved under ABCs, even when categorical modifiers (interactions) are added. Under equal variance (or covariance) of covariates within groups, analytic results guarantee:

• Invariance: Estimators for intercept and main effects are identical across models with and without interactions (e.g., in ANCOVA and two-way ANOVA).
• Efficient Standard Errors: Addition of interaction terms under ABCs does not inflate, and often reduces, the standard errors (SEs) of main effect estimates. This reflects reduction in the model residual sum-of-squares $x_n$2.
• Interpretability: Main effects under ABC parametrization coincide with abundance-weighted population or group averages; interaction coefficients represent group deviations.

Contrast with traditional codings:

Coding Type	Main Effect Interpretation	SE Behavior on Interaction Inclusion	Reference Group Bias
Reference-Group	Effect in reference group	SEs may inflate/change	Yes
Sum-to-Zero	Average effect (unweighted)	No special invariance	No
ABC (abundance)	Abundance-weighted group averages	SEs never increase; often decrease	No

6. Implementation, Theoretical Conditions, and Examples

Implementation of ABC regression requires:

• Construction of the full overparametrized design matrix,
• Computation of categorical and joint categorical proportions,
• Formation of the constraint matrix $x_n$3,
• QR decomposition to obtain a reduced-basis $x_n$4 for reparametrization,
• Solving the unconstrained regression in the lower-dimensional space.

Theoretical validity depends on centering of covariates and, for strongest invariance results, on homogeneity of group covariance matrices for covariates. Empirical studies show that near-invariance holds under mild deviations from these conditions.

Illustrative examples clarify the behavior of estimates and SEs in main-only versus cat-modified models, as well as confirm empirically that ABC-based estimates remain unaltered while SEs do not increase, in contrast to non-ABC encodings.

ABC-parametrization thus provides consistent, interpretable, and equitable estimation procedures in both intractable latent variable models (via kernel-ABC approximation) and categorical regression settings (via data-driven centering constraints), underpinned by rigorous theoretical safeguards and verified by simulation and real data analysis (Ehrlich et al., 2012, Kowal, 2024).