Python Web Crawling

9.9

8.7

Scrapy

9.7

3.3

pyspider

9.2

7.7

requests-html

9.2

4.8

portia

7.9

0.0

RoboBrowser

7.7

8.4

MechanicalSoup

21 Web Crawling packages and projects

Scrapy

9.9 8.7 L4 Python

A fast high-level screen scraping and web crawling framework.
pyspider

9.7 3.3 L3 Python

A powerful spider system.
requests-html

9.2 7.7 HTML

Pythonic HTML Parsing for Humans™
portia

9.2 4.8 L2 JavaScript

Visual scraping for Scrapy.
RoboBrowser

7.9 0.0 L4 Python

A simple, Pythonic library for browsing the web without a standalone web browser.
MechanicalSoup

7.7 8.4 L4 Python

A Python library for automating interaction with websites.
cola

7.1 0.0 L3 Python

A distributed crawling framework.
gain

6.7 0.1 Python

Web crawling framework based on asyncio.
Grab

6.6 6.9 L3 Python

Site scraping framework.
Scrapely

6.6 0.0 HTML

A pure-python HTML screen-scraping library
PSpider

6.4 6.9 Python

A simple web spider frame written by Python, which needs Python3.5+
feedparser

5.0 1.7 L3 Python

Universal feed parser.
Sukhoi

4.5 0.0 Python

Minimalist and powerful Web Crawler.
MSpider

4.3 0.4 Python

Spider
spidy Web Crawler

2.6 1.3 Python

The simple, easy to use command line web crawler.
brownant

2.5 0.0 Python

Brownant is a web data extracting framework.
Crawley

2.4 0.0 Python

Pythonic Crawling / Scraping Framework based on Non Blocking I/O operations.
Demiurge

1.9 0.0 L5 Python

PyQuery-based scraping micro-framework.
Pomp

1.4 0.0 L5 Python

Web crawling framework inspired by and similar to Scrapy
FastImage

0.9 0.0 L4 Python

Python library that finds the size / type of an image given its URI by fetching as little as needed
Mariner

0.6 8.1 Python

Command line torrent searcher.

* Code Quality Rankings and insights are calculated and provided by Lumnify.
They vary from L1 to L5 with "L5" being the highest. Visit our partner's website for more details.

Popular Comparisons

Add another 'Web Crawling' Package