Beyond Classification Accuracy: An Exploration-Range Evaluation of Adaptive Crawling for Fake Shopping Sites

Abstract: In recent years, fake shopping sites targeting Japanese users have appeared in the top results of search engines through SEO poisoning, causing increasing damage. Conventional collection methods rely on fixed keywords and cannot keep up with evolving attack campaigns, delaying the discovery of new sites. We propose a closed-loop crawler that incorporates the page-level outputs of a fake-site classifier (fastText+LightGBM) into the search queries of the next cycle. Search queries are generated by a seed-compound strategy that combines characteristic words extracted from positive pages with seed words from the fake-shopping context (e.g., deep discount,''official''). To complement evaluations that tend to focus on classifier accuracy, we also introduce per-cycle new-host counts and cumulative unique-host counts as exploration-range metrics. In a comparative experiment ($n=3$ for the proposed method, $n=2$ for the baseline), the fixed-keyword baseline yielded zero new-host acquisition from cycle 2 onward, indicating complete stagnation, whereas the proposed method continued to discover new hosts and, at cycle 3, achieved a cumulative unique-host count approximately 7.6 times that of the baseline on average.

• The paper presents a novel closed-loop crawler that uses classifier outputs to dynamically generate search queries.
• It integrates modules for search, feature extraction, classification, and automatic keyword expansion to enhance site discovery.
• Empirical results show a 6.7–8.3 times increase in new-host acquisition compared to static keyword methods, despite some local saturation.

Closed-Loop Crawling and Exploration-Range Evaluation for Fake Shopping Site Discovery

Motivation and Context

The rapid evolution of fake shopping sites targeting Japanese users, exacerbated by SEO poisoning tactics, poses persistent detection and containment challenges. Traditional collection approaches—primarily reliant on static keyword searches—exhibit early stagnation, missing emergent sites as attack strategies and SEO patterns shift. Karasawa et al. address both the technical inadequacies of conventional crawlers and the shortcomings of classifier-centric evaluation, proposing an adaptive collection methodology rooted in closed-loop crawling, where page-level classifier outputs dynamically drive the generation of new search queries.

System Architecture and Methodology

The architecture is modular, comprising search/collection, feature extraction, classification (fastText+LightGBM), automatic keyword generation, and persistent storage. The closed-loop design is central: after each crawl-classify cycle, classifier outputs from positively-predicted pages are morphologically analyzed to extract characteristic nouns, which are then combined with seed contextual words ("deep discount", "official") via a seed-compound strategy. These composite queries, filtered to exclude generic or template words, form the next cycle's search input.

This automatic keyword expansion is empirically scored via a frequency scheme, currently favoring the top-$K$ (20) frequent nouns across positive pages for subsequent search query generation. Each cycle's unique host acquisition is recorded, facilitating exploration-range metrics.

Evaluation Metrics and Comparative Framework

Recognizing the inadequacy of classifier accuracy as a sole metric, the study incorporates both per-cycle new-host counts (new_hosts) and cumulative unique-host counts (cum_unique_hosts) as primary exploration-range measures. This holistic approach quantifies not just detection efficacy but the breadth and freshness of site discovery.

Two comparative conditions are executed: Condition A uses fixed keywords across cycles; Condition B employs the closed-loop keyword expansion. Multiple independent runs account for the stochasticity and temporal variability inherent in search API responses.

Empirical Findings

Results demonstrate pronounced stagnation in the fixed-keyword baseline: from cycle 2 onward, new_hosts acquisition is zero, while cum_unique_hosts plateaus at 30.0±1.4. By contrast, the closed-loop method (Condition B) achieves continuous expansion: at cycle 3, the average cum_unique_hosts is 229.0±18.4—approximately 7.6 times that of the baseline, substantiated across all independent runs (range: 6.7—8.3).

The closed-loop approach maintains a significantly higher positive rate (0.33—0.38 versus baseline 0.20—0.23), signifying that adaptive query generation not only expands reach but also improves the classifier's yield of fake candidates. New-reach rates remain high (0.74—0.78) in Condition B, whereas the baseline quickly converges to existing sites, failing to uncover new hosts.

However, local saturation is observed: new_hosts decelerates in later cycles (cycle 3 value drops to ~29.0), highlighting that simple frequency-based query generation eventually leads to diminishing marginal returns as exploration clusters overlap.

Analysis of Keyword Generation and Operational Considerations

Morphologically derived candidate words, especially when combined with seed context, robustly converge toward the fake-shopping domain across all runs. Yet, the inclusion of generic e-commerce terms (e.g., "favorites", "shipping", "order") does not always translate to efficient discovery, and the system can drift into known clusters, as shown by increased known revisits in later cycles.

Operational variability in total collected pages is noted, especially in cycle 3, due to API constraints and candidate word duplication. The study underscores the need for improved candidate word scoring (TF-IDF, log-likelihood ratio), phrase extraction, pred_score-weighted prioritization, and diversity mechanisms (e.g., MMR) to mitigate query genericization and prolong exploration-range expansion.

Browser-based fetching and screenshotting partially address cloaking and dynamic content, though full coverage is constrained by cost. WHOIS data was unreliable in this experiment; its potential for characterizing domain churn and registrar/country bias is acknowledged as a future extension.

Theoretical Implications and Future Directions

The work advances exploration-focused collection strategies, complementing prevailing detection-centric paradigms. The closed-loop crawler demonstrates that adaptive feedback from classifier outputs into search-query generation is essential for sustained exploration in adversarial, dynamic web environments.

The study's exploration-range evaluation offers a more rigorous means to quantify the effectiveness of crawling strategies under real-world stochastic and adversarial conditions, advocating for standardizing such metrics in future work.

Further research should prioritize:

• Robust classifier ground-truth validation and resilience to distribution shift.
• Sophisticated query scoring and diversity to counter local saturation.
• Mechanisms for efficient, interactive fetching to counter cloaking and dynamic page generation.
• Improved WHOIS retrieval for domain profiling.

Conclusion

Karasawa et al. establish that closed-loop crawling—where classifier outputs drive dynamic search-term expansion—achieves substantially higher exploration range in discovering fake shopping sites compared to fixed-keyword baselines. The empirical effect size (6.7—8.3 times the baseline cum_unique_hosts) is robustly observed. While operational and architectural refinements are required, this work marks a significant step toward adaptive, exploration-driven collection frameworks in the context of evolving web attacks. Exploration-range metrics should be treated as primary criteria for crawler efficacy in dynamic search environments.