Popularity

9.9

Stable

Activity

9.7

Growing

Stars 46,621

Watchers 1,781

Forks 9,903

Last Commit 1 day ago

Description

Scrapy is a fast high-level web crawling and web scraping framework, used to crawl websites and extract structured data from their pages. It can be used for a wide range of purposes, from data mining to monitoring and automated testing.

For more information including a list of features check the Scrapy homepage at: http://scrapy.org

Code Quality Rank: L4

Programming language: Python

License: BSD 3-clause "New" or "Revised" License

Tags: HTTP Web Crawling Application Frameworks Internet WWW Scrapy

Latest version: v2.4.1

Scrapy alternatives and similar packages

Based on the "Web Crawling" category.
Alternatively, view Scrapy alternatives based on common mentions on social networks and blogs.

pyspider

9.5 0.0 L3 Scrapy VS pyspider

A Powerful Spider(Web Crawler) System in Python.
requests-html

9.3 0.0 Scrapy VS requests-html

Pythonic HTML Parsing for Humans™

Access the most powerful time series database as a service

Ingest, store, & analyze all types of time series data in a fully-managed, purpose-built database. Keep data forever with low-cost storage and superior data compression.

Promo www.influxdata.com

portia

9.1 0.0 L2 Scrapy VS portia

Visual scraping for Scrapy
MechanicalSoup

7.8 5.4 L4 Scrapy VS MechanicalSoup

A Python library for automating interaction with websites.
RoboBrowser

7.5 0.0 L4 Scrapy VS RoboBrowser

A simple, Pythonic library for browsing the web without a standalone web browser.
PSpider

6.5 3.7 Scrapy VS PSpider

简单易用的Python爬虫框架，QQ交流群：597510560
cola

6.5 0.0 L3 Scrapy VS cola

A high-level distributed crawling framework.
Grab

6.5 0.0 L3 Scrapy VS Grab

Web Scraping Framework
Scrapely

6.2 0.0 Scrapy VS Scrapely

A pure-python HTML screen-scraping library
gain

6.1 0.0 Scrapy VS gain

Web crawling framework based on asyncio.
feedparser

5.9 4.7 L3 Scrapy VS feedparser

Parse feeds in Python
Sukhoi

4.3 0.0 Scrapy VS Sukhoi

Minimalist and powerful Web Crawler.
MSpider

4.1 0.0 Scrapy VS MSpider

Spider
spidy Web Crawler

3.2 1.7 Scrapy VS spidy Web Crawler

The simple, easy to use command line web crawler.
Google Search Results in Python

3.0 2.6 Scrapy VS Google Search Results in Python

Google Search Results via SERP API pip Python Package
reader

2.7 9.0 Scrapy VS reader

A Python feed reader library.
Crawley

2.5 0.0 Scrapy VS Crawley

Pythonic Crawling / Scraping Framework based on Non Blocking I/O operations.
brownant

2.5 0.0 Scrapy VS brownant

Brownant is a web data extracting framework.
Demiurge

2.0 0.0 L5 Scrapy VS Demiurge

PyQuery-based scraping micro-framework.
Pomp

1.5 0.0 L5 Scrapy VS Pomp

Screen scraping and web crawling framework
FastImage

1.0 0.0 L4 Scrapy VS FastImage

Python library that finds the size / type of an image given its URI by fetching as little as needed
Mariner

0.4 0.0 Scrapy VS Mariner

This a is mirror of Gitlab repository. Open your issues and pull requests there.