TWICE Methods: Dual Penalization & Wage Inference

• TWICE is a term for two distinct dual-structured methods that leverage complementary penalties to stabilize estimation in asymmetric datasets and wage modeling.
• The twice-penalized P-spline approach integrates a traditional roughness penalty with a marginal alignment penalty to accurately model overlapping asymmetric datasets.
• The tree-based wage inference framework uses observable clustering of workers and firms to decompose wage variance and overcome limitations of standard fixed-effects models.

TWICE denotes two distinct methodological constructions in recent arXiv literature. In one usage, TWICE is the acronym for Tree-based Wage Inference with Clustering and Estimation, a framework for modeling conditional wage functions from observables with gradient-boosted trees, observable-anchored worker and firm partitions, and a variance decomposition that includes sorting and non-additive interactions (Bakirov et al., 2 Jan 2026). In another usage, “twice” refers to a twice-penalized P-spline approach for handling overlapping asymmetric datasets, where a flexible regression on a smaller cohort is regularized both by a customary P-spline roughness penalty and by a second penalty that aligns horizontal-cohort marginals with information learned from a larger cohort (McTeer et al., 2023). The two methods are unrelated in application domain, but both are organized around a dual-regularization or dual-structure idea: one combines tree ensembles with clustering and estimation, while the other combines smoothing with marginal alignment.

1. Terminological scope and problem classes

The two uses of TWICE arise from different statistical problems. McTeer et al. study overlapping asymmetric datasets, motivated by healthcare settings in which “a small amount of information” may be available for a larger number of patients, while “a small number of patients may have had extensive further testing” (McTeer et al., 2023). Their objective is to model the smaller cohort against a response while considering the larger cohort, without relying on missing data imputation when the smaller cohort is “significantly different in scale to the larger sample.”

Bakirov, Del Prato, and Zacchia define TWICE explicitly as Tree-based Wage Inference with Clustering and Estimation (Bakirov et al., 2 Jan 2026). Their setting is matched worker–firm panel data with log wages $Y_{it}=\ln w_{it}$, worker covariates $Z_{it}$, and firm covariates $X_j$. The framework is proposed against a background in which standard latent fixed-effects approaches depend on worker mobility, enforce additivity, and may produce decompositions with limited interpretability.

These two constructions therefore address different inferential objects. The twice-penalized P-spline targets a predictive function on an asymmetric, partially overlapping data structure. TWICE in wage inference targets the conditional-expectation function $m_0(X_{J(i,t)},Z_{it})$ and then projects that function onto observable worker and firm partitions. A plausible implication is that the commonality in naming is methodological rather than substantive: both use a “two-part” architecture to stabilize estimation under structural asymmetry.

2. Twice-penalized P-spline methodology for overlapping asymmetric datasets

The twice-penalized P-spline approach is formulated around a B-spline basis and a penalized objective on the smaller “horizontal” cohort $H$ (McTeer et al., 2023). Let $x$ and $z$ lie in $[a,b]$, with an ordered knot sequence $t_1\le t_2\le\cdots\le t_{m+q}$, where $q$ is the B-spline order and $Z_{it}$0 is the number of interior segments. The Cox–de Boor recursion defines B-splines $Z_{it}$1, and the design matrix $Z_{it}$2 is constructed either additively, with separate bases in $Z_{it}$3 and $Z_{it}$4, or as a tensor-product basis $Z_{it}$5 when interactions are allowed.

The central objective is to estimate $Z_{it}$6 so that $Z_{it}$7 fits $Z_{it}$8 in the small cohort while enforcing two regularization criteria. The first uses the customary P-spline roughness matrix $Z_{it}$9, for example $X_j$0 with a finite-difference operator of order $X_j$1. The second uses a matrix $X_j$2 that maps $X_j$3 to estimated horizontal-cohort marginals on a grid $X_j$4, together with $X_j$5, the vector of marginal fits from the large “vertical” cohort $X_j$6. The criterion is

$X_j$7

For continuous responses, the estimator has the closed form

$X_j$8

with $X_j$9. For binary responses, the method maximizes a penalized log-likelihood with $m_0(X_{J(i,t)},Z_{it})$0 and fits the model by penalized iteratively weighted least squares, specifically Newton–Raphson or Fisher scoring. The gradient is

$m_0(X_{J(i,t)},Z_{it})$1

and the Hessian includes the usual GLM curvature term together with the two penalty contributions.

The second penalty is the distinctive component. It is created “through discrepancies in the marginal value of covariates that exist in both the smaller and larger cohorts” (McTeer et al., 2023). This makes the approach different from a once-penalized P-spline: the estimator is not only smoothed, but also pulled toward consistency with a marginal structure learned from the larger cohort.

3. Tuning, identifiability, and empirical behavior of the twice-penalized P-spline

The two tuning parameters have distinct roles. $m_0(X_{J(i,t)},Z_{it})$2 is chosen by K-fold cross-validation in the horizontal cohort $m_0(X_{J(i,t)},Z_{it})$3, cycling through a grid of $m_0(X_{J(i,t)},Z_{it})$4 and selecting the value that optimizes one of several held-out criteria: sum of squares, penalized log-likelihood, or AUC for binary $m_0(X_{J(i,t)},Z_{it})$5 (McTeer et al., 2023). $m_0(X_{J(i,t)},Z_{it})$6 is then selected over a modest grid $m_0(X_{J(i,t)},Z_{it})$7 by requiring either a prescribed reduction in the marginal discrepancy $m_0(X_{J(i,t)},Z_{it})$8, such as 50%, or the absolute best marginal fit. In simulated or real NAFLD data, the paper reports $m_0(X_{J(i,t)},Z_{it})$9 when $H$0.

Theoretical properties are stated in terms of the bias–variance trade-off, identifiability, and convergence. $H$1 controls smoothness of $H$2, while $H$3 controls bias–variance in the marginal alignment. Two separate penalties “ensure one does not collapse into the other,” and $H$4 is full rank after adding a small ridge if necessary. Convergence is immediate in the continuous case because of the closed-form estimator, while local convergence for the binary case is guaranteed under standard GLM regularity conditions.

The empirical results reported in the paper are substantial. In simulations with $H$5, noise $H$6, with or without $H$7–$H$8 interaction, and $H$9 knots, the twice-penalized fit (“Fit 2”) reduced the sum of squares of the marginal curve by 40–90% over the single-penalty P-spline (“Fit 1”) and by even more relative to an unpenalized B-spline linear model (“Fit 0”) (McTeer et al., 2023). Applied to NAFLD registry data with $x$0 patients and $x$1 patients, a binary “At-Risk NASH” outcome, 19 routine covariates in $x$2, and 37 genomic covariates in $x$3, the method achieved $x$4 reduction in marginal-SS across linear predictor, PCA, and tSNE dimension-reduction schemes, and $x$5 improvement in AUC versus standard P-splines. The authors characterize this as a flexible and numerically stable way to borrow “vertical” cohort accuracy into a smoother “horizontal” cohort fit, without imputation.

4. TWICE as Tree-based Wage Inference with Clustering and Estimation

In the wage-inequality setting, TWICE begins from the conditional-expectation model

$x$6

and estimates $x$7 with a gradient-boosted tree ensemble (Bakirov et al., 2 Jan 2026). The fitted predictor is

$x$8

where each tree partitions the feature space into leaves and assigns each leaf a constant score. Estimation minimizes a regularized squared-error objective

$x$9

with

$z$0

Here $z$1 penalizes the number of leaves and $z$2 shrinks leaf weights toward zero; early stopping, learning-rate control, maximum depth, and minimum leaf-size constraints are used for regularization.

The framework then constructs observable-anchored partitions rather than latent worker and firm effects. For firms, TWICE computes each firm-year’s mean log wage $z$3 and fits a single regression tree of depth $z$4 to predict $z$5 from $z$6. The $z$7 terminal leaves define firm-cells $z$8. For workers, it aggregates each worker to mean log wage $z$9 and mean covariates $[a,b]$0, then fits a depth-$[a,b]$1 tree to predict $[a,b]$2 from $[a,b]$3. The $[a,b]$4 leaves define worker-cells $[a,b]$5. At each observation $[a,b]$6, assignment uses the period-specific $[a,b]$7, so workers may move between leaves over time.

With $[a,b]$8 and $[a,b]$9 fixed, the final predictor is estimated via LightGBM using raw covariates $t_1\le t_2\le\cdots\le t_{m+q}$0 together with one-hot indicators of worker-cell and firm-cell membership. The procedure is therefore a hybrid of nonparametric prediction and structured discretization: tree ensembles recover nonlinearities in the conditional wage function, while the worker and firm partitions create an interpretable representation of two-sided heterogeneity.

5. Cross-fitting, decomposition, and relation to AKM in TWICE

A defining estimation feature of TWICE is two-way ID-blocked cross-fitting (Bakirov et al., 2 Jan 2026). Worker IDs are partitioned into $t_1\le t_2\le\cdots\le t_{m+q}$1 blocks $t_1\le t_2\le\cdots\le t_{m+q}$2, firm IDs into $t_1\le t_2\le\cdots\le t_{m+q}$3 blocks $t_1\le t_2\le\cdots\le t_{m+q}$4, and for each pair $t_1\le t_2\le\cdots\le t_{m+q}$5 the validation set is

$t_1\le t_2\le\cdots\le t_{m+q}$6

Training excludes any data with the same worker or firm as those in $t_1\le t_2\le\cdots\le t_{m+q}$7, and the blocked MSE

$t_1\le t_2\le\cdots\le t_{m+q}$8

is minimized over a grid in $t_1\le t_2\le\cdots\le t_{m+q}$9. The final model is retrained on all training data with the selected $q$0 and evaluated on an external firm-level holdout set.

The fitted ensemble induces predicted cell means

$q$1

which are decomposed as

$q$2

The pair $q$3 solves a weighted two-way least-squares projection using cell probabilities $q$4, and the interaction term $q$5 is orthogonal to both additive components by construction. Defining

$q$6

TWICE yields the exact variance decomposition

$q$7

The comparison benchmark is the canonical AKM model

$q$8

Bakirov, Del Prato, and Zacchia emphasize two AKM limitations: additivity, which rules out complementarities, and limited-mobility bias, under which sparse job-switching networks inflate $q$9 and attenuate estimated sorting $Z_{it}$00 (Bakirov et al., 2 Jan 2026). TWICE addresses these by replacing latent effects with observable partitions, allowing explicit non-additivity $Z_{it}$01, and accepting a trade-off: it gives up some ability to capture purely idiosyncratic unobservables in exchange for robustness to sampling noise and out-of-sample portability.

6. Empirical results and interpretability diagnostics in TWICE

The empirical application uses Portuguese administrative data, combining Quadros de Pessoal and SCIE over 2012–2019, restricted to full-time workers aged 20–65 at firms with at least five employees and to the largest mobility-connected set (Bakirov et al., 2 Jan 2026). The sample contains approximately 3.4 million worker-years, 750,000 workers, and 96,000 firms. On an external firm-holdout test, TWICE with two-way cross-fitted LightGBM and worker-plus-firm cells achieves Test MSE $Z_{it}$02 and Test $Z_{it}$03, whereas a flexible OLS benchmark with age polynomials attains Test $Z_{it}$04.

The variance decomposition shares are reported as follows:

Component	TWICE share	Comparison values
Worker	27.6%	AKM: 59%; Bonhomme and Manresa (2019): 50%
Firm	8.7%	AKM: 20%; Bonhomme and Manresa (2019): 5%
Sorting	11.6%	AKM: 7%; Bonhomme and Manresa (2019): 20%
Interaction	7.3%	AKM: none; Bonhomme and Manresa (2019): none
Residual	44.8%	AKM: 14%; Bonhomme and Manresa (2019): 25%

These figures support the paper’s central substantive claim that sorting is quantitatively important and that non-additive interactions are nonzero (Bakirov et al., 2 Jan 2026). The interaction share is “modest (7.3%)” but present, while sorting is larger than in AKM on the same sample.

Interpretability is provided through Partial Dependence Plots and Accumulated Local Effects. The paper reports that age–wage PDPs by worker qualification and education show standard concave profiles, gender gaps, and steeper slopes for managers; tenure ALEs reveal a brief probationary dip at very low tenure followed by sustained firm-specific returns; and firm-size PDPs conditional on productivity are flat, suggesting that the canonical size premium largely reflects sorting on productivity and workforce composition rather than size per se (Bakirov et al., 2 Jan 2026). The framework also compares observable partitions to AKM fixed effects using $Z_{it}$05, the fraction of AKM effect-variance explained by TWICE cell membership, finding $Z_{it}$06 for workers and $Z_{it}$07 for firms on an observation-weighted basis.

7. Comparative significance and future directions

The two TWICE-related methods occupy different positions in contemporary statistical methodology. The twice-penalized P-spline approach is framed as, to the authors’ knowledge, “the first work to propose additional marginal penalties in a flexible regression” for asymmetric datasets, and it is explicitly motivated by avoiding missing data imputation while incorporating information from a larger cohort (McTeer et al., 2023). Its future work includes adaptation “to not require dimensionality reduction” and extension to “parametric modelling methods.” This suggests a broader agenda in which the second penalty might be embedded in richer semiparametric or multivariate structures.

TWICE in wage inference is positioned as an alternative to latent fixed-effects decompositions. Its contribution is to model the wage CEF directly from observables, to cluster workers and firms into interpretable cells, and to decompose wage dispersion into worker, firm, sorting, interaction, and residual components (Bakirov et al., 2 Jan 2026). The authors emphasize that this replaces latent effects with observable-anchored partitions and thereby trades off the ability to capture idiosyncratic unobservables for robustness to sampling noise and out-of-sample generalization.

Taken together, these two usages illustrate different meanings of “TWICE” in current quantitative research. In one case, the term denotes a dual-penalty smoothing architecture for overlapping asymmetric datasets. In the other, it denotes a tree-based inferential framework for wage inequality decomposition. The shared conceptual thread is the deliberate introduction of a second structural layer—marginal alignment in one case, clustered observable heterogeneity in the other—to address limitations of simpler one-stage estimators.