Popularity

9.9

Stable

Activity

9.5

Declining

Stars 55,206

Watchers 1,771

Forks 10,800

Last Commit 5 days ago

Description

Scrapy is a fast high-level web crawling and web scraping framework, used to crawl websites and extract structured data from their pages. It can be used for a wide range of purposes, from data mining to monitoring and automated testing.

For more information including a list of features check the Scrapy homepage at: http://scrapy.org

Code Quality Rank: L4

Programming language: Python

License: BSD 3-clause "New" or "Revised" License

Tags: HTTP Web Crawling Application Frameworks Internet WWW Scrapy

Latest version: v2.4.1

Scrapy alternatives and similar packages

Based on the "Web Crawling" category.
Alternatively, view Scrapy alternatives based on common mentions on social networks and blogs.

pyspider

9.5 0.0 L3 Scrapy VS pyspider

DISCONTINUED. A Powerful Spider(Web Crawler) System in Python.
requests-html

9.1 0.0 Scrapy VS requests-html

Pythonic HTML Parsing for Humans™

InfluxDB – Built for High-Performance Time Series Workloads

InfluxDB 3 OSS is now GA. Transform, enrich, and act on time series data directly in the database. Automate critical tasks and eliminate the need to move data externally. Download now.

Promo www.influxdata.com

portia

8.9 0.0 L2 Scrapy VS portia

Visual scraping for Scrapy
MechanicalSoup

7.7 4.6 L4 Scrapy VS MechanicalSoup

A Python library for automating interaction with websites.
RoboBrowser

7.2 0.0 L4 Scrapy VS RoboBrowser

A simple, Pythonic library for browsing the web without a standalone web browser.
PSpider

6.4 0.0 Scrapy VS PSpider

简单易用的Python爬虫框架，QQ交流群：597510560
cola

6.4 0.0 L3 Scrapy VS cola

A high-level distributed crawling framework.
Grab

6.4 3.0 L3 Scrapy VS Grab

Web Scraping Framework
feedparser

6.3 7.5 L3 Scrapy VS feedparser

Parse feeds in Python
Scrapely

6.1 0.0 Scrapy VS Scrapely

A pure-python HTML screen-scraping library
gain

6.0 0.0 Scrapy VS gain

Web crawling framework based on asyncio.
Sukhoi

4.2 0.0 Scrapy VS Sukhoi

Minimalist and powerful Web Crawler.
Google Search Results in Python

4.1 5.0 Scrapy VS Google Search Results in Python

Google Search Results via SERP API pip Python Package
MSpider

4.0 0.0 Scrapy VS MSpider

Spider
reader

3.4 9.1 Scrapy VS reader

A Python feed reader library.
spidy Web Crawler

3.3 0.0 Scrapy VS spidy Web Crawler

The simple, easy to use command line web crawler.
Crawley

2.7 0.0 Scrapy VS Crawley

Pythonic Crawling / Scraping Framework based on Non Blocking I/O operations.
brownant

2.6 0.0 Scrapy VS brownant

Brownant is a web data extracting framework.
Demiurge

2.1 0.0 L5 Scrapy VS Demiurge

PyQuery-based scraping micro-framework.
Pomp

1.6 0.0 L5 Scrapy VS Pomp

Screen scraping and web crawling framework
FastImage

1.0 0.0 L4 Scrapy VS FastImage

Python library that finds the size / type of an image given its URI by fetching as little as needed
Mariner

0.4 0.0 Scrapy VS Mariner

This a is mirror of Gitlab repository. Open your issues and pull requests there.