---
title: Horvitz–Thompson Estimator Overview
url: https://www.emergentmind.com/topics/horvitz-thompson-estimator
type: topic
---

# Horvitz–Thompson Estimator Overview

The Horvitz–Thompson estimator is a foundational tool in survey sampling, importance weighting, and related fields, providing design-unbiased estimation for population totals and means under arbitrary, potentially unequal probability sampling designs. Its versatility has led to widespread application and extensive theoretical generalization, with rigorous developments addressing its bias-variance properties, extensions to complex and adaptive designs, model-assisted improvements, and modern applications in causal and network inference, machine learning, and statistical genomics.

## 1. Mathematical Formulation and Principle

Let \( U = \{1, \dots, N\} \) be a finite population, with each unit \( i \) associated to a value \( y_i \). Suppose a sample \( S \subset U \) is drawn such that for each unit, the (first-order) inclusion probability is \( \pi_i = P(i \in S) \), with \( 0 < \pi_i \leq 1 \) and known for all \( i \). The Horvitz–Thompson (HT) estimator for the population total \( T = \sum_{i \in U} y_i \) is

\[
\hat{T}_{HT} = \sum_{i \in S} \frac{y_i}{\pi_i}
\]

and for the mean, \( \hat{\mu}_{HT} = \frac{1}{N} \sum_{i \in S} y_i / \pi_i \) [2204.14121].

The essential attribute is design-unbiasedness: \( \mathbb{E}_p[\hat{T}_{HT}] = T \), regardless of the sampling scheme, as long as inclusion probabilities are strictly positive and correctly specified.

The variance is given by

\[
\mathrm{Var}(\hat{T}_{HT}) = \sum_{i=1}^N \sum_{j=1}^N \frac{\Delta_{ij}}{\pi_i \pi_j} y_i y_j
\]

where \( \Delta_{ij} = \pi_{ij} - \pi_i \pi_j \) and \( \pi_{ij} = P(i \in S, j \in S) \) [2204.14121].

## 2. Extensions, Efficiency, and Alternative Normalization

HT’s design-unbiasedness often comes at the cost of high variance, particularly when \( \pi_i \) are highly heterogeneous. The estimator ignores the randomness of the realized sample size \( \hat{n} \), motivating the use of alternative normalization and shrinkage strategies to control variance.

**Self-Normalized (Hajek) Estimator.** Normalizes by the HT-estimated sample size:

\[
\hat{\mu}_H = \frac{\sum_{k=1}^n Y_k I_k / p_k}{\sum_{k=1}^n I_k / p_k}
\]

This introduces correlation between numerator and denominator, often reducing variance at the cost of slight bias [2106.07695].

**Adaptive Normalization.** The Trotter–Tukey family of IPW estimators interpolates between pure HT (\( \lambda=0 \)) and Hajek (\( \lambda=1 \)), with the adaptively normalized estimator (AN) minimizing asymptotic variance:

\[
\hat{\mu}_{AN} = \frac{\hat{S}}{n} + \frac{\hat{T}}{\hat{\pi}} (1 - \hat{n}/n)
\]

where

\[
\hat{S} = \sum_{k=1}^n \frac{Y_k I_k}{p_k}, \;\;\;
\hat{T} = \frac{1}{n} \sum \frac{1-p_k}{p_k} Y_k \frac{I_k}{p_k}, \;\;\;
\hat{\pi} = \frac{1}{n} \sum \frac{1-p_k}{p_k} \frac{I_k}{p_k}
\]

The AN estimator always achieves variance smaller than or equal to HT and Hajek except in degenerate cases [2106.07695]. It is also optimal in the bias–variance trade-off within the class of affine-normalized IPW estimators.

**Hard-Thresholding and Variance Regularization.** To further reduce variance, especially when some \( \pi_i \) are extremely small, hard-thresholding replaces \( \pi_i \) below a threshold \( a \) by \( a \):

\[
\pi_i' = \max(\pi_i, a), \qquad
\hat{T}_{IHT} = \sum_{i \in S} \frac{y_i}{\pi_i'}
\]

This “improved Horvitz–Thompson estimator” introduces small bias (order \( O(n^{-1}) \)) but yields strictly reduced MSE under mild regularity [1804.04255].

## 3. Advanced Designs: Two-Stage, Cross-Classified, and Network Sampling

### Two-Stage Sampling

When sampling is performed hierarchically (e.g., clusters then individuals within clusters), the two-stage HT estimator is

\[
\hat{Y}_{HT} = \sum_{i\in S_I} \frac{1}{\pi_{Ii}} \sum_{k\in S_i} \frac{y_{ik}}{\pi_{k|i}}
\]

Under mild regularity, this estimator is consistent and asymptotically normal, with unbiased estimators available for the full variance (which decomposes into between- and within-cluster components) [1808.09758].

### Cross-Classified Sampling

For settings where units are sampled independently along two dimensions (e.g., time × location), the cross-classified HT estimator is

\[
\hat{\tau}_{HT} = \sum_{i\in S_M} \sum_{k\in S_D} \frac{Y_{ik}}{\pi_i^M \pi_k^D}
\]

This estimator is unbiased, but often less efficient than two-stage sampling for the same allocated sample size; variance estimation requires careful handling of joint inclusion probabilities [1511.00507].

### Network and Interference Settings

Networked data, particularly under interference (where outcomes for one unit depend on others' treatments or exposures), require extended HT forms using exposure mappings:

\[
\hat{\tau} = \frac{1}{n}\sum_{i=1}^n \frac{\mathbf{1}\{Z_i=1, e_i \in \mathcal{R}_1\}}{\pi_i^1} Y_i - \frac{1}{n} \sum_{i=1}^n \frac{\mathbf{1}\{Z_i=0, e_i \in \mathcal{R}_0\}}{\pi_i^0} Y_i
\]

Here, \(\pi_i^1\) is the probability unit \(i\) receives treatment and is in exposure arm \(\mathcal{R}_1\); similar for controls [2405.15887]. The estimator remains unbiased when the exposure mapping is correct, but becomes inadmissible relative to MSE among linear estimators under most random-exposure designs [2312.01234].

## 4. Efficiency, Bayesian Remedies, and Adaptive Estimation

HT estimation is not, in general, semiparametrically efficient: it is uniformly minimum-variance among design-unbiased estimators but can be dominated in variance by biased or model-assisted estimators [2208.06039].

**Semiparametric Efficiency.** Adaptive estimators leveraging auxiliary models for the sampling weights, outcome distributions, or both, can achieve the efficiency bound representing the lowest possible asymptotic variance among regular asymptotically linear estimators. Construction typically proceeds via efficient influence function calculations and can use double machine learning to nonparametrically estimate required conditional moments [2208.06039].

**Bayesian Smoothing and Shrinkage.** In the presence of small or highly variable inclusion probabilities, classical HT estimators are susceptible to “weak paradoxes,” yielding estimates with extreme variance or even erratic behavior (e.g., Basu’s or Wasserman’s examples). Bayesian fix approaches, such as conjugate hierarchical models or binned shrinkage, regularize the HT estimator to achieve smaller MSE at minor cost in bias, without requiring a hard minimum bound on \(\pi_i\) [2204.14121].

## 5. Model-Assisted and Nonresponse-Adjusted Extensions

When auxiliary information is available, model-assisted estimators combine HT’s unbiasedness with the predictive power of a working model. The generalized regression estimator (GREG)

\[
\hat{Y}_{GREG} = \sum_{i \in U} \hat{m}(x_i) + \sum_{i \in s} \frac{y_i - \hat{m}(x_i)}{\pi_i}
\]

is asymptotically unbiased and attains smaller variance than pure HT if the model is approximately correct [2208.04621, 1101.0831]. Calibration methods adjust the HT weights to exactly match known auxiliary totals.

For nonresponse, double expansion applies: first for the sampling design (\(1/\pi_i\)), then for the response probability (\(1/p_i\)), giving

\[
\hat{Y}_{2HT} = \sum_{i \in s_r} \frac{y_i}{\pi_i p_i}
\]

In practice, \(p_i\) is estimated, and model-assisted adjustments can further improve variance properties, provided missingness is at random and the model for response is well specified [2208.04621].

## 6. Practical Advancements and Applications

Horvitz–Thompson methodology has been deployed in a broad range of modern problems:

- **Prediction error estimation in complex samples:** By extending Efron's optimism correction with HT weights, estimators of generalization error are unbiased for the true target population, and are algebraically equivalent to design-based AIC in canonical GLMs [1711.04877].
- **Rare subgroup and network interference causal inference:** HT estimators remain unbiased for subgroup means and contrasts under complex two-stage and interference designs, whereas “natural” or normalized estimators become undefined or biased for rare subgroups [2001.02719].
- **Graph statistics from subgraph samples:** For network summary quantities that can be expressed as sums over sampled units or edges (e.g., Dirichlet energy, homophily), the HT estimator affords unbiased and consistent estimates even when the total network is only partially observed, so long as the sampling design and edge-inclusion probabilities are known [2512.17084].

Adaptive, model-assisted, and robust HT-like estimators have been developed and studied for settings with partially observed probabilities, finite response efforts (requiring nonparametric deconvolution for response probabilities), and unknown detection in spatial sampling, with established consistency and MSE guarantees under regularity [1309.2136, 1910.14647].

## 7. Limitations, Optimality, and Open Directions

While the Horvitz–Thompson estimator is indispensable for design-unbiased inference under arbitrary unequal probability sampling, its limitations are well-documented:

- **High Sampling-Weight Variance:** Small or highly heterogeneous \(\pi_i\) can yield estimators with extremely high or even infinite variance, motivating shrinkage or regularization [1804.04255, 2106.07695].
- **Inadmissibility under Complex Dependence:** In experiments with network interference, the HT estimator is inadmissible in MSE among all linear estimators except under fixed-exposure designs. Shrinkage, model-based, and design-restricted alternatives are under active study [2312.01234].
- **Sensitivity to Design and Weight Specification:** Misspecification of inclusion probabilities, response models, or exposure mapping can lead to breakdown of unbiasedness, requiring data-adaptive, model-assisted, or semiparametric approaches for robust inference [2405.15887].

Continued research addresses semiparametric optimality with flexible models for response or inclusion, robustification to misspecification and informative sampling, theoretical guarantees in high-dimensional or network-dependent settings, and practical computation in large-scale or streaming data regimes.

---

**References:**

- [1804.04255] Improved Horvitz-Thompson Estimator in Survey Sampling
- [1808.09758] Inference for two-stage sampling designs with application to a panel for urban policy
- [1711.04877] Estimating prediction error for complex samples
- [2106.07695] Adaptive normalization for IPW estimation
- [2405.15887] Data-adaptive exposure thresholds for the Horvitz-Thompson estimator of the ATE in experiments with network interference
- [1910.14647] Horvitz-Thompson-like estimation with distance-based detection probabilities for circular plot sampling of forests
- [2312.01234] On the admissibility of Horvitz-Thompson estimator for estimating causal effects under network interference
- [1309.2136] Deconvolution with application to estimation of sampling probabilities and the Horvitz-Thompson estimator
- [2208.04621] Model-Assisted Estimators under Nonresponse in Sample Surveys
- [1101.0831] Nonparametric Additive Model-assisted Estimation for Survey Data
- [2208.06039] Semiparametric adaptive estimation under informative sampling
- [2204.14121] Inverse Probability Weighting: from Survey Sampling to Evidence Estimation
- [2512.17084] Dirichlet Meets Horvitz and Thompson: Estimating Homophily in Large Networks via Sampling
- [1511.00507] Estimation under cross-classified sampling with application to a childhood survey
- [1209.6503] Variance estimation and asymptotic confidence bands for the mean estimator of sampled functional data with high entropy unequal probability sampling designs
- [2103.05915] A cautionary note on the Hanurav-Vijayan sampling algorithm
- [2001.02719] A note on Horvitz-Thompson estimators for rare subgroup analysis in the presence of interference

Source: https://www.emergentmind.com/topics/horvitz-thompson-estimator