|Developer(s)||Zyte (formerly Scrapinghub)|
|Initial release||26 June 2008|
2.6.2 / 25 July 2022
|Operating system||Windows, macOS, Linux|
Scrapy (// SKRAY-peye) is a free and open-source web-crawling framework written in Python and developed in Cambuslang. Originally designed for web scraping, it can also be used to extract data using APIs or as a general-purpose web crawler. It is currently maintained by Zyte (formerly Scrapinghub), a web-scraping development and services company.
Scrapy project architecture is built around "spiders", which are self-contained crawlers that are given a set of instructions. Following the spirit of other don't repeat yourself frameworks, such as Django, it makes it easier to build and scale large crawling projects by allowing developers to reuse their code.
The Scrapy framework provides you with powerful features such as auto-throttle, rotating proxies and user-agents, allowing you scrape virtually undetected across the net. Scrapy also provides a web-crawling shell, which can be used by developers to test their assumptions on a site’s behavior.
Scrapy was born at London-based web-aggregation and e-commerce company Mydeco, where it was developed and maintained by employees of Mydeco and Insophia (a web-consulting company based in Montevideo, Uruguay). The first public release was in August 2008 under the BSD license, with a milestone 1.0 release happening in June 2015. In 2011, Zyte (formerly Scrapinghub) became the new official maintainer.
- https://github.com/scrapy/scrapy/releases/tag/2.6.2; publication date: 25 July 2022; retrieved: 3 September 2022.
- Commit 975f150
- Scrapy at a glance.
- "Frequently Asked Questions". Retrieved 28 July 2015.
- "Scrapy shell". Retrieved 28 July 2015.
- Bell, Eddie; Heusser, Jonathan. "Scalable Scraping Using Machine Learning". Archived from the original on 4 June 2016. Retrieved 28 July 2015.
- Scrapy | Companies using Scrapy
- Montalenti, Andrew. "Web Crawling & Metadata Extraction in Python".
- "Scrapy Companies". Scrapy website.
- Hyphe v0.0.0: the first release of our new webcrawler is out!
- Ben Firshman [@bfirsh] (21 January 2010). "World Govt Data site uses Django, Solr, Haystack, Scrapy and other exciting buzzwords bit.ly/5jU3La #opendata #datastore" (Tweet) – via Twitter.
- Medina, Julia (19 June 2015). "Scrapy 1.0 official release out!". scrapy-users (Mailing list).
- Pablo Hoffman (2013). List of the primary authors & contributors. Retrieved 18 November 2013.
- Interview Scraping Hub.