Popularity

9.9

Stable

Activity

9.7

Growing

Stars 50,763

Watchers 1,776

Forks 10,317

Last Commit 7 days ago

Description

Scrapy is a fast high-level web crawling and web scraping framework, used to crawl websites and extract structured data from their pages. It can be used for a wide range of purposes, from data mining to monitoring and automated testing.

For more information including a list of features check the Scrapy homepage at: http://scrapy.org

Code Quality Rank: L4

Programming language: Python

License: BSD 3-clause "New" or "Revised" License

Tags: HTTP Web Crawling Application Frameworks Internet WWW Scrapy

Latest version: v2.4.1

Scrapy alternatives and similar packages

Based on the "Web Crawling" category.
Alternatively, view Scrapy alternatives based on common mentions on social networks and blogs.

pyspider

9.5 0.0 L3 Scrapy VS pyspider

A Powerful Spider(Web Crawler) System in Python.
requests-html

9.2 0.0 Scrapy VS requests-html

Pythonic HTML Parsing for Humans™

WorkOS - The modern identity platform for B2B SaaS

The APIs are flexible and easy-to-use, supporting authentication, user identity, and complex enterprise features like SSO and SCIM provisioning.

Promo workos.com

portia

9.0 0.0 L2 Scrapy VS portia

Visual scraping for Scrapy
MechanicalSoup

7.7 5.9 L4 Scrapy VS MechanicalSoup

A Python library for automating interaction with websites.
RoboBrowser

7.4 0.0 L4 Scrapy VS RoboBrowser

A simple, Pythonic library for browsing the web without a standalone web browser.
PSpider

6.5 0.0 Scrapy VS PSpider

简单易用的Python爬虫框架，QQ交流群：597510560
Grab

6.5 3.0 L3 Scrapy VS Grab

Web Scraping Framework
cola

6.4 0.0 L3 Scrapy VS cola

A high-level distributed crawling framework.
Scrapely

6.3 0.0 Scrapy VS Scrapely

A pure-python HTML screen-scraping library
gain

6.1 0.0 Scrapy VS gain

Web crawling framework based on asyncio.
feedparser

6.1 7.7 L3 Scrapy VS feedparser

Parse feeds in Python
Sukhoi

4.2 0.0 Scrapy VS Sukhoi

Minimalist and powerful Web Crawler.
MSpider

4.0 0.0 Scrapy VS MSpider

Spider
Google Search Results in Python

3.7 4.5 Scrapy VS Google Search Results in Python

Google Search Results via SERP API pip Python Package
spidy Web Crawler

3.2 0.0 Scrapy VS spidy Web Crawler

The simple, easy to use command line web crawler.
reader

3.1 9.2 Scrapy VS reader

A Python feed reader library.
Crawley

2.6 0.0 Scrapy VS Crawley

Pythonic Crawling / Scraping Framework based on Non Blocking I/O operations.
brownant

2.6 0.0 Scrapy VS brownant

Brownant is a web data extracting framework.
Demiurge

2.0 0.0 L5 Scrapy VS Demiurge

PyQuery-based scraping micro-framework.
Pomp

1.6 0.0 L5 Scrapy VS Pomp

Screen scraping and web crawling framework
FastImage

1.0 0.0 L4 Scrapy VS FastImage

Python library that finds the size / type of an image given its URI by fetching as little as needed
Mariner

0.4 0.0 Scrapy VS Mariner

This a is mirror of Gitlab repository. Open your issues and pull requests there.