urlhealth: Python Toolkit for URL Verification

• urlhealth is an open-source Python toolkit that verifies citation URLs by distinguishing between stale (link rot) and hallucinated links.
• It combines real-time HTTP liveness checking with Wayback Machine archival queries to accurately classify URL statuses.
• Empirical benchmarks show that integrating urlhealth can reduce non-resolving URL rates significantly, enhancing the reliability of generated citations.

urlhealth is a lightweight, open-source Python toolkit designed to systematically detect and classify broken ("non-resolving") and hallucinated URLs in outputs from LLMs and deep research agents. By combining HTTP liveness checking and longitudinal web archiving queries, urlhealth enables scalable, automated validation of citation URLs—differentiating transient link rot from model-fabricated references that never existed. The toolkit standardizes this verification step for both research and deployment settings, underpinned by empirical findings from large-scale benchmarks such as DRBench and ExpertQA (Rao et al., 3 Apr 2026).

1. Motivation: Citation URL Failures in LLMs and Agents

As LLMs and agentic research models increasingly generate answers with citation URLs, two predominant failure modes have emerged:

• Link Rot (Stale URLs): References that once pointed to valid online content, but now return HTTP client/server errors (status 4xx/5xx). These URLs may have been valid during model training or browsing but have since become inaccessible.
• URL Hallucinations: Citations that were fabricated by the model, with no evidence of prior existence in web archives. These represent pure generative hallucination and cannot be linked to any real web resource.

Empirical studies on DRBench (ten models, 53,090 URLs) and ExpertQA (three models, 168,021 URLs) found that 5–18% of LLM-cited URLs are non-resolving, and 3–13% are hallucinated (absent from the Wayback Machine). Domain-specific effects are notable, with non-resolving rates of 5.4% in Business and 11.4% in Theology. Deep research agents (agent-style, multi-hop models) emit more citations per query and exhibit higher hallucination rates (up to 13.3%) compared to single-pass, search-augmented LLMs (3–9%). This prevalence of invalid citations undermines user trust and introduces risks, especially in high-stakes contexts such as medicine, law, and academic publishing (Rao et al., 3 Apr 2026).

2. System Architecture and Operational Workflow

urlhealth's architecture comprises three modular components, enabling robust classification of citation URL validity.

1. HTTP Liveness Checker: Issues HTTP HEAD requests to the candidate URL, with GET fallbacks on specific response codes (405, 403, 501). URLs returning status 200 are marked as LIVE; 4xx/5xx statuses or timeouts are considered candidates for further investigation. Special handling is applied for HTTP 403 (bot-blocking), designating these as UNKNOWN.
2. Wayback Machine Interface: For URLs failing the initial liveness check, urlhealth queries the Wayback Machine API for historical snapshots. Existence of any archive indicates that the URL was once live.
3. Classification Module: Integrates contemporary liveness with archival data to assign each URL one of four statuses: LIVE, STALE (link rot), LIKELY_HALLUCINATED (no archival record), or UNKNOWN (e.g., persistent bot-block).

The per-URL workflow is structured as follows:

1. Issue HEAD request.
   • Status 200 → LIVE.
   • Status 403 → UNKNOWN.
   • Status 4xx/5xx/timeout → Proceed to archival check.
2. Query Wayback Machine.
   • Archive available → STALE.
   • No archive record → LIKELY_HALLUCINATED.

Optionally, UNKNOWN cases can be audited using a headless browser; empirical auditing suggests that 89% of UNKNOWN URLs are either live or actively blocked (not dead) (Rao et al., 3 Apr 2026).

3. Classification Formalism and Evaluation Metrics

The urlhealth classification algorithm uses the following definitions and partitions on the universal set $U$ of tested URLs:

• $$\text{live} = \{u \in U \mid \text{http\_status}(u) = 200\}$$
• $N = |U|$
• $$\text{NR} = \{u \in U \mid \text{http\_status}(u) \in \{4xx, 5xx, \text{timeout}\} \land u \notin \text{live}\}$$ (non-resolving)
• $S = \{u \in \text{NR} \mid$ archive.exists$(u) \}$ (stale)
• $H = \text{NR} \setminus S$ (hallucinated)

Key evaluation metrics are:

• $\text{NonResolvingRate} = \frac{|\text{NR}|}{N}$
• $\text{HallucinationRate} = \frac{|H|}{N}$
• $$\text{LinkRotFraction} = \frac{|S|}{|S| + |\text{live}|}$$
• $$\text{live} = \{u \in U \mid \text{http\_status}(u) = 200\}$$0

For models engaged in self-correction, the Improvement Factor is defined as

$$\text{live} = \{u \in U \mid \text{http\_status}(u) = 200\}$$1

This enables quantitative assessment during post-processing or model-in-the-loop correction (Rao et al., 3 Apr 2026).

4. Implementation, Usage, and Extensibility

urlhealth is implemented as a pure-Python package (≈ 83 lines), with minimal dependencies:

• requests (mandatory HTTP client)
• backoff (optional, for retries)
• tqdm (optional, for progress bars)

The core API exposes primitives for programmatic and bulk use:

$$\text{live} = \{u \in U \mid \text{http\_status}(u) = 200\}$$2

• status.code reports the HTTP result.
• status.wayback_archived reflects archival snapshot existence.
• status.label is in {LIVE, STALE, HALLUCINATED, UNKNOWN}.

Batch processing supports parallel workers and pandas DataFrames. Command-line usage is available via urlhealth check --input urls.txt --output report.csv --workers 50.

The toolkit is extensible: archival backends may be swapped by subclassing ArchiveChecker, accommodating alternate or complementary web archives beyond the Wayback Machine. Headless browser auditing can be enabled via the [headless] install extra, which leverages Playwright to further interrogate ambiguous (UNKNOWN) URLs.

5. Experimental Results: Benchmarks and Correction Efficacy

Experiments on DRBench (100 multilingual queries, 53,090 URLs, 10 models) and ExpertQA (2,177 expert questions, 168,021 URLs, 32 domains, 3 models) yielded the following results:

Dataset	Model	Non-resolving (%)	Hallucinated (%)	Domain extremes (%)
DRBench	gpt-4.1	5.4	---	---
DRBench	gemini-2.5-pro-deepresearch	18.5	13.3	---
DRBench	claude-3-5	---	3.0	---
ExpertQA	gemini-2.5-pro	4.20	---	Business: 5.4
ExpertQA	claude-4.5	9.38	---	Theology: 11.4

Self-correction experiments on an ExpertQA subset (435 questions) demonstrated that integrating urlhealth with agentic post-processing reduced non-resolving URL rates to below 1%:

Model	Pre-correction NR%	Post-correction NR%	Improvement Factor (IF)
Gemini-2.5-pro	6.1	0.1	79×
GPT-5.1	16.0	0.6	26×
Claude-4.5	4.9	0.8	6.4×

This suggests that tool-enabled self-correction can virtually eliminate non-resolving URL citations, with efficacy dependent on the model's competence in tool use and citation replacement (Rao et al., 3 Apr 2026).

6. Integration with Agentic Self-Correction and Pipeline Positioning

urlhealth serves as a callable verification tool in agentic research workflows. Agent loops typically operate as follows:

1. Enumerate answer citations.
2. Run urlhealth to identify broken or hallucinated URLs.
3. Query search or archives to replace broken links.
4. Update the answer and repeat if necessary.

Empirical findings confirm that integrating urlhealth in this manner reduces non-resolving citation rates by a factor of 6–79, with all evaluated top-tier commercial models achieving sub-1% error rates post-correction.

urlhealth’s design complements approaches that restrict generation to previously-visited URLs (e.g., WebGPT), but its primary use case is post-hoc validation and correction, making it agnostic to upstream model architecture.

7. Limitations and Opportunities for Extension

Several limitations are observed:

• Incomplete archival coverage: The Wayback Machine may fail to archive legitimate URLs, risking misclassification of hallucinations as stales and inflating the stale fraction via archived soft-404s.
• Ambiguous failures: HTTP 403 or 429 (bot-blocking or rate-limits) are marked as UNKNOWN; browser-based auditing reveals most are not genuinely dead.
• Temporal liveness: urlhealth's assessments are point-in-time; URL accessibility may fluctuate.
• Scope: The tool does not verify bibliographic metadata, content correctness, or snippet accuracy. Future directions include integration with CrossRef, Semantic Scholar, and headless browsers for content-based checks.
• Applicability: Architectures that generate citations strictly from visited resources naturally reduce hallucinated URLs but do not obviate the need for external post-processing checks.

By offering HTTP and archival checks, evidence-based classification logic, and a lightweight interface, urlhealth establishes a practical methodology for measuring and mitigating URL hallucinations at scale within LLM and agent pipelines, supporting both research evaluation and real-world deployment workflows (Rao et al., 3 Apr 2026).