Scrapy is a fast high-level web crawling and web scraping framework, used to crawl websites and extract structured data from their pages. It can be used for a wide range of purposes, from data mining to monitoring and automated testing.

For more information including a list of features check the Scrapy homepage at: http://scrapy.org

Code Quality Rank: L4
Programming language: Python
License: BSD 3-clause "New" or "Revised" License
Tags: HTTP     Web Crawling     Application Frameworks     Internet     WWW     Scrapy    

Scrapy alternatives and related packages

Based on the "Web Crawling" category

Do you think we are missing an alternative of Scrapy or a related project?

Add another 'Web Crawling' Package

Scrapy Recommendations

There are no recommendations yet. Be the first to promote Scrapy!

Have you used Scrapy? Share your experience. Write a short recommendation and Scrapy, you and your project will be promoted on Awesome Python.
Recommend Scrapy

Recently added Scrapy resources

  • Creating a web crawler in Scrapy and scrape data
    Deploying Spider to ScrapingHub
    Logging into Websites with Scrapy
    Running Scrapy as a Standalone Script
    Using Scrapy with Selenium in Special Cases, e.g. to Scrape JavaScript Driven Web Pages
    Tutorial  Added by GoTrained // www.udemy.com // almost 2 years ago
  • Scrapy Tutorial Getting Started
    Craigslist Scrapy Spider #1 – Titles
    Craigslist Scrapy Spider #2 – One Page
    Craigslist Scrapy Spider #3 – Multiple Pages
    Craigslist Scrapy Spider #4 – Job Descriptions
    Tutorial  Added by GoTrained // python.gotrained.com // almost 2 years ago