2000 character limit reached
Exploratory Analysis of a Terabyte Scale Web Corpus (1409.5443v2)
Published 18 Sep 2014 in cs.IR
Abstract: In this paper we present a preliminary analysis over the largest publicly accessible web dataset: the Common Crawl Corpus. We measure nine web characteristics from two levels of granularity using MapReduce and we comment on the initial observations over a fraction of it. To the best of our knowledge two of the characteristics, the language distribution and the HTML version of pages have not been analyzed in previous work, while the specific dataset has been only analyzed on page level.
Collections
Sign up for free to add this paper to one or more collections.
Paper Prompts
Sign up for free to create and run prompts on this paper using GPT-5.