Efficient Crawling for Scalable Web Data Acquisition (Extended Version)

Abstract: Journalistic fact-checking, as well as social or economic research, require analyzing high-quality statistics datasets (SDs, in short). However, retrieving SD corpora at scale may be hard, inefficient, or impossible, depending on how they are published online. To improve open statistics data accessibility, we present a focused Web crawling algorithm that retrieves as many targets, i.e., resources of certain types, as possible, from a given website, in an efficient and scalable way, by crawling (much) less than the full website. We show that optimally solving this problem is intractable, and propose an approach based on reinforcement learning, namely using sleeping bandits. We propose SB-CLASSIFIER, a crawler that efficiently learns which hyperlinks lead to pages that link to many targets, based on the paths leading to the links in their enclosing webpages. Our experiments on websites with millions of webpages show that our crawler is highly efficient, delivering high fractions of a site's targets while crawling only a small part.

• The paper formulates focused web crawling as an NP-complete minimum-cost coverage problem and introduces SB-CLASSIFIER, which learns target-rich DOM tag-path clusters using a sleeping-bandit strategy.
• Across 18 multilingual websites totaling 22.2 million pages, SB-CLASSIFIER often retrieved 90% of target files after accessing about 20% of webpages and typically under 36% of non-target data volume.
• The crawler combines online URL classification with early stopping, saving up to 82.6% of requests on some sites while highlighting open challenges in recrawling, deep-Web access, and scalable validation of statistics datasets.

Motivation and problem

The paper addresses the acquisition of statistics datasets (SDs) — numeric, multidimensional aggregates published by national and international bodies as CSV, spreadsheets, SDMX/XML, JSON, or PDF files — from large institutional websites. Existing access paths are inadequate: per-site APIs are brittle; search engines index only a small fraction of available files (e.g., Google returns 302 PDFs on justice.gouv.fr where more than 9k exist, 641 results on ilo.org instead of over 49k, and does not recognize TSV at all despite 11,097 such files); and exhaustive crawling is prohibitively slow under crawling-ethics constraints (a 1-second politeness delay alone takes 11 days for a million-page site). The authors therefore formulate a focused crawling problem: retrieve as many targets (data files of user-specified MIME types) as possible from a single website while minimizing HTTP requests or transferred volume, without prior knowledge of the site's structure.

Problem formalization and complexity

A website is modeled as a rooted, node-weighted, edge-labeled directed graph $G=(V,E,r,\omega,\lambda)$, where nodes are pages, edges are hyperlinks, $\omega$ assigns retrieval cost, and $\lambda$ labels each edge. A crawl is an $r$-rooted subtree; the goal is a minimum-cost crawl covering a target set $V^*$. The decision variant is shown NP-complete by reduction from set cover, with hardness holding even for constant cost functions and target nodes without outgoing links. This holds even assuming the full graph is known in advance, which it is not during an actual crawl — so only heuristic methods are viable. The website boundary is defined pragmatically via hostname subdomain containment, since no agreed definition exists.

The choice of edge labeling is central: each hyperlink's label is its tag path, the full DOM path from the page root to the anchor tag, including class and id attributes. The method rests on two hypotheses: links appearing under similar tag paths lead to similar content types, and this structure can be learned online to predict which paths lead toward targets.

The SB-CLASSIFIER crawler

The crawler uses reinforcement learning with a deliberately simple single-state model, avoiding multi-state MDPs whose state definitions (topic-relevance summaries) have no natural equivalent for non-topical target retrieval. Actions are clusters of similar tag paths rather than individual links:

• Tag paths are encoded as bag-of-$n$-grams vectors over a dynamically growing vocabulary, projected to fixed dimension $D=2^m$ via a multiplicative hash function, with collisions resolved by averaging.
• Clusters are maintained as centroids in an HNSW approximate nearest-neighbor index; a new path joins the nearest centroid if cosine similarity exceeds a threshold $\theta$, otherwise it forms a new action.
• Action selection follows the Awake Upper-Estimated Reward (AUER) sleeping-bandit algorithm, a UCB variant that handles actions becoming empty once all their URLs are visited. The score combines mean reward with an exploration term weighted by $\alpha = 2\sqrt{2}$, the value optimal under standard assumptions that do not hold here (rewards are unbounded counts, roughly power-law distributed), but which works well empirically across all sites studied.
• Rewards count newly discovered links to targets in the crawled page, encouraging novelty.

Because computing rewards requires knowing MIME types before following links, the crawler includes an online URL classifier: a logistic regression over character-level 2-gram bag-of-words of URLs, trained incrementally via SGD. It is trained on only two classes ("HTML" and "Target"), deliberately never predicting "Neither", because misclassifying HTML or Target URLs as Neither would prune large parts of the site, whereas the reverse error costs only one wasted request. Labels come free after an initial batch from HTTP HEAD responses, since every GET itself yields ground truth. Ablations show richer features (anchor text, DOM path, surrounding text) and heavier models (SVM, Naive Bayes, passive-aggressive) yield no consistent gain over URL-only logistic regression.

An early-stopping mechanism monitors the slope of cumulative target discovery through an exponential moving average and halts when discovery stalls; on some sites it saves up to 82.6% of requests while losing essentially no targets.

Experimental evaluation

Experiments cover 18 websites totaling 22.2M distinct pages, spanning French government ministries, international organizations (ILO, WHO, World Bank, OECD), and national statistical agencies, in over 20 languages, with target densities from 2.5% to 66.8% and mean target depths from about 3 to nearly 90. Baselines include RANDOM, BFS, DFS, an omniscient upper bound, the early focused-crawler approach (FOCUSED), the offline tag-path-based ACEBot adapted to target retrieval (TP-OFF), and TRES, a topical RL crawler given three deliberate unfair advantages (hand-picked keywords, pre-trained positive examples, and a free oracle for URL typing).

The headline result: on several very large websites, SB-CLASSIFIER retrieves 90% of targets while accessing only around 20% of webpages. On il it needs 14.2% of requests to reach 90% of targets versus $+\infty$ for FOCUSED, TP-OFF, DFS, and RANDOM; on wo it needs 18.6% versus 92% for BFS. In volume terms it typically retrieves 90% of target bytes after fetching under 36% of non-target volume, against 80–98%+ for baselines. On wo, linear extrapolation suggests the best baseline would need three months to match the crawler's yield, and on wh up to six years for TP-OFF. The classifier-based crawler closely tracks the unrealistic oracle variant, indicating the learned URL classifier contributes little overhead relative to perfect knowledge.

Topical crawlers fare poorly even when favored: TRES fails on 9 of 10 fully evaluated sites and cannot scale beyond small ones due to exhaustive feature computation during tree expansion, supporting the claim that topical locality assumptions do not hold for target retrieval. Analysis of learned action rewards shows the top tag-path group averages 258 new targets per crawl versus a site-wide non-zero-reward mean often below 3, confirming strong concentration of targets along specific structural patterns — sometimes human-interpretable (e.g., download-link CSS classes), sometimes not, which also demonstrates language independence.

On a manual sample of 280 retrieved targets across seven sites, 35–93% contained at least one statistics table, with 1.4–9.1 SDs per collected target, giving an empirical estimate of end-to-end SD yield.

Limitations and open questions

Several limitations are stated plainly. The optimality guarantee for $\omega$0 does not apply because reward distributions are unknown and non-normal; the choice is pragmatic. High similarity thresholds can be catastrophic ($\omega$1 caused an out-of-memory failure on ed by creating one action per page). Evaluation on partially crawled sites relies on comparisons restricted to the smallest achieved crawl size, and repeated crawls were infeasible for ethical and time reasons, so some results rest on single runs. Detecting whether retrieved targets actually contain statistics tables remains costly at scale (roughly one second per PDF page with state-of-the-art extraction), so SD-yield figures are based on small manual samples. The crawler performs a single-shot acquisition: it does not handle updates or newly published resources, a significant gap for statistical journalism, and it does not address deep-Web content behind forms. Whether more complex reward functions than raw counts of reachable targets would improve performance is left open.

Conclusion

The paper formalizes target-focused crawling within a single website, proves the optimization problem NP-complete, and delivers a practical solution combining sleeping-bandits over tag-path clusters with an online URL classifier. Across 18 heterogeneous, multilingual websites of up to a million pages, it consistently outperforms classical, topical, and RL-based baselines, frequently retrieving 90% of targets with a small fraction of the requests and bandwidth an exhaustive crawl would require. Its main open problems are incremental re-crawling for freshness, deep-Web access, and scalable validation that acquired targets genuinely contain statistics tables.