Firth-Type Penalized Likelihood (FT-PL)

• Firth-type penalized likelihood (FT‑PL) is a bias-reducing method that modifies the log-likelihood by adding a term based on the Fisher information to remove first-order bias.
• FT‑PL is widely used in logistic and binomial regression to address issues like separation, rare events, and high-dimensional nuisance structures, ensuring finite and stable estimates.
• Various reformulations and approximations of FT‑PL enhance computational efficiency, extend its applications to few-shot learning, network models, and mixture-cure estimations, and provide robust performance in challenging settings.

Firth-type penalized likelihood (FT‑PL) denotes a class of bias-reducing penalized likelihood methods in which the ordinary log-likelihood $\ell(\theta)$ is modified by an information-based penalty, most commonly

$$\ell^*(\theta)=\ell(\theta)+\frac12\log|I(\theta)|,$$

so that the resulting estimator can be interpreted as a posterior mode under Jeffreys’ prior and the leading $O(n^{-1})$ term in the asymptotic bias of the maximum likelihood estimator is removed in regular models. In current research, FT‑PL functions both as a foundational device in binomial and logistic regression—especially under separation, rare events, and high-dimensional nuisance structures—and as a transferable construction for few-shot classifiers, network models, and mixture-cure estimation, with further generalization to differential-geometric bias correction for generic estimands (Rigon et al., 2022, Ogawa et al., 2023, Hirose et al., 2020, Zietkiewicz et al., 2023).

1. Foundational definition and statistical rationale

Firth’s original idea is to modify the score equations so that the first-order term in the asymptotic bias of the maximum likelihood estimator disappears. In the FT‑PL representation, this becomes maximization of a penalized log-likelihood

$$\ell_{\mathrm F}(\beta)=\ell(\beta)+\tfrac12\log|I(\beta)|,$$

where $I(\beta)$ is the expected Fisher information. For regular models, $|I(\beta)|^{1/2}$ is proportional to Jeffreys’ prior, so FT‑PL is simultaneously a bias-reducing frequentist device and a maximum a posteriori estimator under an invariant default prior. In logistic regression, this is the canonical example of FT‑PL (Rigon et al., 2022).

The same principle admits a broader formulation. A differential-geometric treatment develops penalized likelihoods

$l^*(\xi;x)=l(\xi;x)+\tilde l(\xi)$

for generic estimands $f(\xi)$, not only for the parameter vector itself. In that framework, the penalty is characterized by a quasi-linear first-order PDE,

$$\langle \operatorname{grad} f,\operatorname{grad}\tilde l\rangle+\tfrac12\Delta^{(-1)}f=0,$$

so that the plug-in estimator $f(\hat \xi)$ becomes second-order asymptotically unbiased. Firth’s classical Jeffreys-prior penalty is recovered as the special case in which $$\ell^*(\theta)=\ell(\theta)+\frac12\log|I(\theta)|,$$0 is a coordinate component of the parameter (Hirose et al., 2020).

This suggests that FT‑PL is not merely an ad hoc regularizer. Its defining feature is that the penalty is derived from statistical curvature—usually through a log-determinant of an information matrix—and is chosen to remove a specific asymptotic bias term. A plausible implication is that FT‑PL should be distinguished from penalties whose main purpose is sparsity, variance control, or degeneracy prevention but which are not information-based.

2. Logistic and binomial regression: modified scores, separation, and existence

For binomial logistic regression with independent responses

$$\ell^*(\theta)=\ell(\theta)+\frac12\log|I(\theta)|,$$1

the ordinary log-likelihood is

$$\ell^*(\theta)=\ell(\theta)+\frac12\log|I(\theta)|,$$2

FT‑PL replaces it by

$$\ell^*(\theta)=\ell(\theta)+\frac12\log|I(\theta)|,$$3

with $$\ell^*(\theta)=\ell(\theta)+\frac12\log|I(\theta)|,$$4. Equivalently, the estimator solves modified score equations. In one widely used form, the Firth scores are

$$\ell^*(\theta)=\ell(\theta)+\frac12\log|I(\theta)|,$$5

where $$\ell^*(\theta)=\ell(\theta)+\frac12\log|I(\theta)|,$$6 are the diagonal elements of the hat matrix

$$\ell^*(\theta)=\ell(\theta)+\frac12\log|I(\theta)|,$$7

The second term is the bias-reducing adjustment, and its shrinkage effect pulls coefficients toward $$\ell^*(\theta)=\ell(\theta)+\frac12\log|I(\theta)|,$$8, or equivalently probabilities away from $$\ell^*(\theta)=\ell(\theta)+\frac12\log|I(\theta)|,$$9 and $O(n^{-1})$0 (Rigon et al., 2022).

The practical importance of this construction is most evident under complete or quasi-complete separation. In such cases, the ordinary MLE can fail to exist in $O(n^{-1})$1; some coefficients diverge to $O(n^{-1})$2, fitted probabilities become $O(n^{-1})$3 or $O(n^{-1})$4, and standard errors and Wald procedures become pathological. A central existence theorem establishes that, if the design matrix $O(n^{-1})$5 has full column rank, then a maximizer of

$O(n^{-1})$6

exists in $O(n^{-1})$7, and the set of maximizers is bounded. The same paper proves analogous existence results for probit and complementary log-log binomial regression with Jeffreys-prior penalization, while noting that for non-canonical links these are FT‑PL estimators in the same spirit, though not exactly Firth’s original bias-reducing estimator in the score-modification sense (Ogawa et al., 2023).

This existence theory formalizes a fact long exploited in applications: FT‑PL is not only a small-sample bias correction, but also a remedy for ill-posed likelihood geometry in binomial-response models. It regularizes the problem by driving the penalized objective to $O(n^{-1})$8 as information degenerates along separating directions.

3. Reformulations and approximations of FT‑PL

In multinomial logistic and cosine classifiers for few-shot image classification, the Firth-type penalty admits an explicit simplification. Starting from a Jeffreys-type log-determinant penalty based on the Fisher information, the authors show that for multinomial logistic regression

$O(n^{-1})$9

which is equivalent, up to constants, to penalizing cross-entropy by

$$\ell_{\mathrm F}(\beta)=\ell(\beta)+\tfrac12\log|I(\beta)|,$$0

The resulting minimization objective is

$$\ell_{\mathrm F}(\beta)=\ell(\beta)+\tfrac12\log|I(\beta)|,$$1

In this setting, FT‑PL simplifies to encouraging uniform class assignment probabilities. The same effective penalty carries over to cosine classifiers after separating direction and scale, so the practical loss remains cross-entropy plus a KL term from the uniform distribution (Ghaffari et al., 2021).

A different reformulation arises in logistic regression through a conjugate-prior approximation. There, the penalized log-likelihood

$$\ell_{\mathrm F}(\beta)=\ell(\beta)+\tfrac12\log|I(\beta)|,$$2

approximates the Firth score by replacing leverage weights $$\ell_{\mathrm F}(\beta)=\ell(\beta)+\tfrac12\log|I(\beta)|,$$3 with trial weights $$\ell_{\mathrm F}(\beta)=\ell(\beta)+\tfrac12\log|I(\beta)|,$$4. The resulting estimator is not exactly FT‑PL except in special cases, but it is a first-order approximation with a pseudo-count interpretation: $$\ell_{\mathrm F}(\beta)=\ell(\beta)+\tfrac12\log|I(\beta)|,$$5 This converts penalized estimation into ordinary logistic regression on modified responses, preserves existence and uniqueness under full-rank $$\ell_{\mathrm F}(\beta)=\ell(\beta)+\tfrac12\log|I(\beta)|,$$6, and is practically close to Firth’s estimator when leverages are roughly constant (Rigon et al., 2022).

These reformulations illustrate two recurring themes. First, FT‑PL often becomes more transparent after rewriting the penalty in probability space or pseudo-data space. Second, exact Jeffreys-prior penalization and tractable approximations can be close in bias behavior while differing in computational cost and in the way they weight observations.

4. Computation and scalable estimation

Direct FT‑PL computation can be expensive because the penalty depends on the Fisher information and, in logistic regression, on leverage quantities that must be updated iteratively. In classical implementations, this requires repeated construction of $$\ell_{\mathrm F}(\beta)=\ell(\beta)+\tfrac12\log|I(\beta)|,$$7, evaluation of hat diagonals, and quasi-Fisher scoring. The conjugate-prior approximation sidesteps this by turning the problem into a standard GLM fit on pseudo-counts, so that any ordinary logistic regression routine can be used. In the reported high-dimensional simulation, for $$\ell_{\mathrm F}(\beta)=\ell(\beta)+\tfrac12\log|I(\beta)|,$$8 the proposed method required about $$\ell_{\mathrm F}(\beta)=\ell(\beta)+\tfrac12\log|I(\beta)|,$$9 ms per fit, whereas direct Firth fitting via brglm2 required about $I(\beta)$0 seconds; for $I(\beta)$1, the direct Firth method was practically infeasible within hours while the approximation remained tractable (Rigon et al., 2022).

A separate line of work develops bounded-memory estimation for generalized linear models with adjusted score equations and maximum Jeffreys-prior penalized likelihood. The penalized objective is

$I(\beta)$2

where $I(\beta)$3 is the standard Jeffreys prior and any $I(\beta)$4 yields a Firth-type penalized likelihood. Two new IWLS variants eliminate the need to store $I(\beta)$5 quantities in memory by using incremental QR decompositions on chunks of predetermined size. One variant implements the exact adjusted IWLS step through two passes over the data; the other uses a one-pass approximation per iteration based on previous leverages but has the same stationary point. Both procedures can operate with data sets that exceed computer memory or even hard drive capacity, and both can be adapted to settings where distinct parts of the data are stored across different sites and cannot be fully transferred because of privacy constraints (Zietkiewicz et al., 2023).

This computational literature changes the scope of FT‑PL. The method is no longer confined to small or moderate data sets where direct leverage calculations are feasible. It can be embedded in chunked, streaming, or distributed workflows while preserving the same statistical target.

5. Empirical behavior and major application domains

In few-shot image classification, FT‑PL is reported to yield statistically significant gains in almost all tested settings across mini-ImageNet, CIFAR-FS, tiered-ImageNet, and CUB. The improvements are typically in the range $I(\beta)$6 to $I(\beta)$7 absolute accuracy, with gains above $I(\beta)$8 in harder high-way tasks such as tiered-ImageNet $I(\beta)$9-shot $|I(\beta)|^{1/2}$0-way classification. The effect is robust across feature backbones, numbers of shots, numbers of classes, and imbalanced class distributions, and it persists even when FT‑PL is added on top of Distribution Calibration (Ghaffari et al., 2021).

In rare-event network meta-analysis, FT‑PL is implemented as a common-effect logistic NMA with the Jeffreys-prior penalty applied to the study-intercept and treatment-contrast design matrix. This “penalized likelihood NMA” yields finite treatment effects and standard errors even when studies report zero events in all treatment arms, thereby avoiding continuity corrections and study exclusion. Simulation results show that the penalized method most often results in smaller bias than inverse-variance NMA, Mantel-Haenszel NMA, or non-central hypergeometric NMA, while profile-likelihood intervals improve coverage relative to Wald intervals (Evrenoglou et al., 2021).

In Weibull proportional-hazards mixture cure models with missing covariates, FT‑PL is applied to the full cure-model likelihood

$|I(\beta)|^{1/2}$1

and combined with profile-likelihood inference and CLIP after multiple imputation. The reported simulations indicate that, compared with complete-case analysis, multiple imputation with penalization reduces estimation bias and improves coverage. At lower event rates, the exact conditional-distribution imputation model generates smaller bias and higher coverage than the approximate alternatives, while FT‑PL stabilizes estimation in the low-event, imbalanced-covariate regimes where ordinary ML can fail (Xu et al., 22 Jul 2025).

In high-dimensional logistic regression, the maximum Jeffreys-prior penalized likelihood estimator remains finite above the phase-transition region where the MLE diverges. Large simulation studies suggest that below the phase transition the estimator is nearly aggregate-unbiased, while above it the slope coefficients are over-shrunk by a scalar factor $|I(\beta)|^{1/2}$2. A conjectured rescaling

$|I(\beta)|^{1/2}$3

is reported to yield almost zero aggregate bias over a wide range of designs, including settings where ML and AMP-based corrections do not exist (Kosmidis et al., 2023).

In dyadic network formation models with sender and receiver fixed effects, a Firth-type log-determinant penalty built from the block-diagonal fixed-effects Hessian guarantees finite-sample existence and removes the leading incidental-parameter bias in common parameters. In the global textiles trade application, the unpenalized fixed-effects MLE required trimming countries with zero in-degree and then trimming additional countries created by the induced boundary degrees, whereas the penalized estimator was computable on the full network and avoided the resulting selection bias (Yan et al., 1 May 2026).

6. Limitations, distinctions, and recurrent misunderstandings

A recurring misunderstanding is to treat FT‑PL as interchangeable with generic regularization. The few-shot classification study explicitly argues otherwise: $|I(\beta)|^{1/2}$4 regularization mainly controls parameter magnitude, whereas the Firth penalty acts in probability space and targets the first-order bias of the estimator. Likewise, confidence penalties $|I(\beta)|^{1/2}$5, unigram penalties $|I(\beta)|^{1/2}$6, and original label smoothing can have similar algebraic forms in multinomial logistic models, but the paper reports that the specific Firth direction $|I(\beta)|^{1/2}$7 and its Jeffreys-based derivation are crucial, especially under imbalance and low-shot bias (Ghaffari et al., 2021).

A second limitation is that coefficient bias reduction does not automatically imply calibrated prediction. In rare-event logistic regression, FT‑PL pushes predicted probabilities toward $|I(\beta)|^{1/2}$8, so the average predicted probability no longer equals the observed event fraction. The paper on rare events shows that this bias can be substantial and proposes two modifications: FLIC, which performs a post-hoc intercept correction while leaving slopes unchanged, and FLAC, which augments the data with an indicator distinguishing original and pseudo-observations. Both restore unbiased average predicted probabilities while preserving the separation-handling advantages of Firth-type estimation (Puhr et al., 2021).

A third distinction concerns exact FT‑PL versus approximations. The Diaconis–Ylvisaker conjugate-prior method is a close approximation to the Firth score and often has very similar bias, but it is not exactly Firth’s FT‑PL except in special cases such as $|I(\beta)|^{1/2}$9 or some saturated balanced settings. When leverages vary widely across observations, replacing $l^*(\xi;x)=l(\xi;x)+\tilde l(\xi)$0 by $l^*(\xi;x)=l(\xi;x)+\tilde l(\xi)$1 can be a poor approximation, and exact Firth penalization may yield smaller bias (Rigon et al., 2022).

The boundary of the concept is also important. Not every penalized likelihood is Firth-type. In mixtures of von Mises–Fisher distributions, the penalty used to prevent likelihood degeneracy is additive in the concentration parameters, for example $l^*(\xi;x)=l(\xi;x)+\tilde l(\xi)$2, and corresponds to an exponential prior on $l^*(\xi;x)=l(\xi;x)+\tilde l(\xi)$3. That construction is explicitly described as not being a Jeffreys-prior or information-based penalty and therefore not an FT‑PL in the Firth sense, even though it is still a penalized likelihood (Ng, 2020).

Finally, several current directions remain only partially resolved. In few-shot learning, the theory-correct Firth coefficient would be $l^*(\xi;x)=l(\xi;x)+\tilde l(\xi)$4, but the practical loss treats $l^*(\xi;x)=l(\xi;x)+\tilde l(\xi)$5 as a hyperparameter and does not establish that fixed $l^*(\xi;x)=l(\xi;x)+\tilde l(\xi)$6 is always optimal in deep-learning pipelines (Ghaffari et al., 2021). In high-dimensional logistic regression, the aggregate-bias rescaling for Jeffreys-prior penalization is supported by extensive simulation evidence but remains a conjecture rather than a proved limit law (Kosmidis et al., 2023). These points do not undermine FT‑PL’s established role in bias reduction and existence theory; they indicate that the modern literature is extending the method beyond the classical fixed-$l^*(\xi;x)=l(\xi;x)+\tilde l(\xi)$7 regime in which its original guarantees were derived.