Sequential Testing Framework

• Sequential Testing Framework is a dynamic statistical method that determines sample size based on accumulating data and adapts testing procedures in real time.
• It employs techniques like SPRT and self-tuning generalized likelihood ratios to enforce precise error control with calibrated stopping rules.
• The framework achieves asymptotic optimality by minimizing expected sample size and supports adaptive designs, including computerized adaptive testing.

Sequential Testing Framework

A sequential testing framework provides statistical decision procedures in which the sample size is not fixed in advance but determined dynamically based on the incoming data and, optionally, adaptive experiment selection. This approach underlies classical sequential probability ratio tests (SPRT), modern generalized likelihood ratio (GLR) procedures, and their extensions to adaptive designs, non-parametric models, and real-time applications such as computerized adaptive testing (@@@@1@@@@). Contemporary sequential frameworks optimize expected sample size subject to rigorous control of type I and II error probabilities across both fixed-length and open-ended settings, and adaptively focus sampling on critical regions of uncertainty.

1. Fundamental Model and GLR Construction

Let $X_1, X_2, \ldots$ be a sequence of observations under an exponential-family model, with densities

$$f_\theta(x) = \exp\{ x\,T(x) - \psi(\theta) \},\;\;\theta\in \Theta\subset \mathbb{R}.$$

Observations may be i.i.d., or, in adaptive designs, generated according to item-specific models (e.g., in CAT, each item $j$ has $f_{\theta, j}$ and a corresponding Kullback-Leibler information $I_j(\theta, \theta')$).

The sequential test considers composite hypotheses defined via cut-points for "mastery":

$$H_0: \theta \geq \theta_+ \qquad\text{vs.}\qquad H_1: \theta \leq \theta_-,$$

with an "indifference region" $(\theta_-, \theta_+)$.

The classical SPRT utilizes the fixed-point likelihood ratio $L_k(\theta_-)/L_k(\theta_+)$. Modern frameworks generalize this to the self-tuning generalized likelihood ratio (GLR):

$$\Lambda_k = \frac{L_k(\hat\theta_k)}{L_k(\theta_{\text{ref}})},$$

where $$\hat \theta_k = \arg \max_{\theta\in \Theta} L_k(\theta)$$ is the MLE after $k$ observations, and $\theta_{\text{ref}}$ is a context-specific reference (typically $\theta_+$ or $\theta_-$).

2. Stopping Rules and Error Control via Modified Haybittle–Peto Procedure

Sequential frameworks enforce a maximum sample size $N$ and control type-I ($\alpha$) and type-II ($\beta$) error probabilities. The modified Haybittle–Peto procedure is defined as follows, with a burn-in period $pN$ and tuning parameter $\epsilon$:

• For $k$ with $pN \leq k < N$, compute

$$Z_k = \log[L_k(\hat\theta_k)/L_k(\theta_+)],\quad W_k = \log[L_k(\hat\theta_k)/L_k(\theta(N))].$$

• Decision boundaries:
  • Reject $H_0$ ("mastery") if $\hat\theta_k < \theta_+$ and $Z_k \geq A$,
  • Accept $H_0$ ("non-mastery") if $\hat\theta_k > \theta(N)$ and $W_k \geq B$.
• At $k = N$, declare mastery if $\log[L_N(\theta_+)/L_N(\theta(N))] \geq C$.

Thresholds $(A, B, C)$ are calibrated so that

$$P_{\theta(N)}\{\text{accept } H_0 \text{ before } N\} = \beta,$$

$$P_{\theta_+}\{\text{reject } H_0 \text{ before } N \} = \alpha (1-\epsilon)/2,$$

$$P_{\theta_+}\{\text{reject } H_0 \text{ at } N \} = \alpha (1+\epsilon)/2,$$

achieving exact overall error rates.

Threshold calibration is performed via Monte Carlo simulation, normal-approximation recursions, or Siegmund’s closed-form formulas.

3. Asymptotic Optimality and Theory of Sequential Experiment Selection

Define $M$ as the random stopping time. Among all tests $T$ that stop in $[pN, N]$ and satisfy error constraints, the modified Haybittle–Peto test achieves

$$E_\theta[M] \sim \inf_{T \in \mathcal{T}_{\alpha, \beta, N}} E_\theta[T] \quad \text{as } \alpha, \beta \to 0, \log \alpha \sim \log \beta,$$

meaning no other test in this class can asymptotically achieve a lower expected sample size at any parameter value $\theta$.

Extensions to adaptive experiment selection (e.g., CAT):

• At each stage, select item $j_i$ informed by past data, observe $X_i \sim f_{\theta, j_i}$.
• Provided long-run item frequencies $v_j$ exist and all $I_j(\theta, \theta')$ satisfy a uniform convexity bound, the modHP procedure remains asymptotically optimal in the adapted setting.
• If items fall into $K$ classes with common response models and only limiting class-frequencies $D_k$ need control, optimality persists.

Proofs rely on Hoeffding-type lower bounds for expected sample size and martingale CLT for GLR increments.

4. Sequential CAT Algorithmic Realization

For item pools with parameters $(a_j, b_j, c_j)$ under 3PL models,

$$p_j(\theta) = c_j + (1 - c_j) / [1 + e^{-a_j(\theta - b_j)}],$$

the algorithm selects at each step $i$ the unused item $j_i$ maximizing chosen information index at the current ability estimate $\hat\theta_{i-1}$:

• Fisher information $I_j(\hat\theta_{i-1})$,
• KL information $I_j(\hat\theta_{i-1}, \theta_{\text{ref}})$.

After observing response $u_i \in \{0, 1\}$, update the log-likelihood, recompute the MLE

$$\hat\theta_i = \arg \max_\theta \prod_{\ell \leq i} p_{j_\ell}(\theta)^{u_\ell} [1 - p_{j_\ell}(\theta)]^{1 - u_\ell},$$

and check stopping-rule conditions.

5. Real-Time Adaptive Mastery Testing and Performance Benchmarking

The sequential testing protocol enables:

• Early stopping for clear mastery ($\theta \gg \theta_+$) or clear non-mastery ($\theta \ll \theta_-$),
• Prolonged testing within the indifference region $(\theta_-, \theta_+)$.

The self-tuning GLR statistic $$\Lambda_k = L_k(\hat\theta_k)/L_k(\theta_{\text{ref}})$$ dynamically concentrates statistical information on the hardest to classify examinees.

Empirical comparison using a large test-item pool (ETS Chauncey data, 1136 items) reveals:

• Classical truncated SPRT yields inflated type-I error (≈16%, target 5%) and longer average test length.
• Modified Haybittle–Peto test (modHP) achieves error rates (α, β) exactly, and reduces average test length by 40–50% compared to fixed-length and TSPRT designs, without exceeding the maximum allowed N.
• Exposure-control and content-balancing overlays can be applied without compromising statistical validity as long as item selection remains outcome-adaptive and limiting frequencies exist.

6. Calibration, Implementation, and Robustness Considerations

Calibration of thresholds $(A, B, C)$ is accomplished via:

• Monte Carlo routines: estimation of implied alternatives $\theta(N)$ for fixed-N tests and subsequent simulation to resolve target error rates.
• Normal-approximation formulas: use the signed-root statistic

$$S_k := \text{sign}(\hat\theta_k - \theta) \sqrt{2 k \log \left[ L_k(\hat\theta_k)/L_k(\theta) \right] } \approx N(0, k),$$

enabling efficient computation via recursion.

• Empirical choices for the burn-in $pN \in [N/3, N/2]$ and $\epsilon \in [1/3, 1/2]$ deliver robust practical performance.

Exposure-control/content-balancing layers can be safely added when item-selection protocols satisfy long-run frequency existence.

7. Summary of Theoretical and Practical Advances

By deploying self-tuning GLR thresholds in modified Haybittle–Peto boundaries, rigorously calibrated via simulation or analytic approximations, the modern sequential testing framework for CAT and related domains:

• Enforces exact type-I/type-II error control at pre-specified levels $(\alpha, \beta)$,
• Guarantees not to exceed user-chosen maximum test length $N$,
• Adapts in real time to individual subject ability,
• Achieves asymptotic optimality in expected sample size among all procedures meeting the constraints,
• Demonstrates in simulation 30–50% reduction in mean sample size compared to classical and fixed-length sequential approaches, with robust empirical and analytic validation (Bartroff et al., 2011).