|Type of business||501(c)(3) non-profit|
|Headquarters||San Francisco, California; Los Angeles, California, United States|
|Key people||Peter Norvig, Nova Spivack, Carl Malamud, Kurt Bollacker Joi Ito|
Common Crawl is a nonprofit 501(c)(3) organization that crawls the web and freely provides its archives and datasets to the public. Common Crawl's web archive consists of petabytes of data collected since 2011. It completes crawls generally every month.
Common Crawl was founded by Gil Elbaz. Advisors to the non-profit include Peter Norvig and Joi Ito. The organization's crawlers respect nofollow and robots.txt policies. Open source code for processing Common Crawl's data set is publicly available.
In December 2012, blekko donated to Common Crawl search engine metadata blekko gathered from crawls it conducted from February to October 2012. The donated data helped Common Crawl "improve its crawl while avoiding spam, porn and the influence of excessive SEO."
In 2013, Common Crawl began using Apache Software Foundation's Nutch webcrawler instead of a custom crawler. Common Crawl switched from using .arc files to .warc files with its November 2013 crawl.
A filtered version of Common Crawl was used to train OpenAI's GPT-3 language model, announced in 2020. One challenge of using Common Crawl data is that despite the vast amount of documented web data, individual pieces of crawled websites could be better documented. This can create challenges when trying to diagnose problems in projects that use the Common Crawl data. A solution proposed by Timnit Gebru, et al., in 2020 to an industry-wide documentation shortfall is that that every dataset should be accompanied with a datasheet that documents its motivation, composition, collection process, and recommended uses.
History of Common Crawl data
The following data have been collected from the official Common Crawl Blog.
|Crawl date||Size in TiB||Billions of pages||Comments|
|March 2014||223||2.8||First Nutch crawl|
|January 2014||148||2.3||Crawls performed monthly|
|November 2013||102||2||Data in Warc file format|
|July 2012||—||—||Data in Arc file format|
|January 2012||—||—||Public Data Set of Amazon Web Services|
|November 2011||40||5||First availability on Amazon|
Norvig Web Data Science Award
In corroboration with SURFsara, Common Crawl sponsors the Norvig Web Data Science Award, a competition open to students and researchers in Benelux. The award is named for Peter Norvig who also chairs the judging committee for the award.
- Rosanna Xia (February 5, 2012). "Tech entrepreneur Gil Elbaz made it big in L.A." Los Angeles Times. Retrieved July 31, 2014.
- "Gil Elbaz and Common Crawl". NBC News. April 4, 2013. Retrieved July 31, 2014.
- "So you're ready to get started". Retrieved 2018-06-02.
- Lisa Green (January 8, 2014). "Winter 2013 Crawl Data Now Available". Retrieved June 2, 2018.
- "Startups - Gil Elbaz and Nova Spivack of Common Crawl - TWiST #222". This Week In Startups. January 10, 2012.
- Tom Simonite (January 23, 2013). "A Free Database of the Entire Web May Spawn the Next Google". MIT Technology Review. Retrieved July 31, 2014.
- Jennifer Zaino (March 13, 2012). "Common Crawl To Add New Data In Amazon Web Services Bucket". Semantic Web. Archived from the original on July 1, 2014. Retrieved July 31, 2014.
- Jennifer Zaino (July 16, 2012). "Common Crawl Corpus Update Makes Web Crawl Data More Efficient, Approachable For Users To Explore". Semantic Web. Archived from the original on August 12, 2014. Retrieved July 31, 2014.
- Jennifer Zaino (December 18, 2012). "Blekko Data Donation Is A Big Benefit To Common Crawl". Semantic Web. Archived from the original on August 12, 2014. Retrieved July 31, 2014.
- Jordan Mendelson (February 20, 2014). "Common Crawl's Move to Nutch". Common Crawl. Retrieved July 31, 2014.
- Jordan Mendelson (November 27, 2013). "New Crawl Data Available!". Common Crawl. Retrieved July 31, 2014.
- Brown, Tom; Mann, Benjamin; Ryder, Nick; Subbiah, Melanie; Kaplan, Jared; Dhariwal, Prafulla; Neelakantan, Arvind; Shyam, Pranav; Sastry, Girish; Askell, Amanda; Agarwal, Sandhini (2020-06-01). "Language Models are Few-Shot Learners". p. 14. arXiv:2005.14165 [cs.CL].
the majority of our data is derived from raw Common Crawl with only quality-based filtering.
- Gebru, Timnit; Morgenstern, Jamie; Vecchione, Briana; Wortman Vaughan, Jennifer; Wallach, Hanna; Daumé III, Hal; Crawford, Kate (March 19, 2020), Datasheets for Datasets, arXiv.org, pp. working paper, retrieved June 6, 2021
- Lisa Green (November 15, 2012). "The Norvig Web Data Science Award". Common Crawl. Retrieved July 31, 2014.
- "Norvig Web Data Science Award 2014". Dutch Techcentre for Life Sciences. Archived from the original on August 15, 2014. Retrieved July 31, 2014.