Python Web Crawling

9.9

9.1

Scrapy

9.7

6.6

pyspider

9.2

8.3

portia

8.0

0.0

RoboBrowser

7.4

0.0

cola

7.0

8.9

Grab

Know what errors lurk in your code? Automatically know what’s broken and why with Rollbar. Rollbar instantly detects when code breaks and provides the source code and state to help you defeat impactful errors. LibHunt users get 100,000 errors monitored for free.

» Learn more from our partner.

11 Web Crawling packages and projects

Scrapy

9.9 9.1 L4 Python

A fast high-level screen scraping and web crawling framework.
pyspider

9.7 6.6 L3 Python

A powerful spider system.
portia

9.2 8.3 L2 JavaScript

Visual scraping for Scrapy.
RoboBrowser

8.0 0.0 L4 Python

A simple, Pythonic library for browsing the web without a standalone web browser.
cola

7.4 0.0 L3 Python

A distributed crawling framework.
Grab

7.0 8.9 L3 Python

Site scraping framework.
MechanicalSoup

6.7 7.4 L4 Python

A Python library for automating interaction with websites.
feedparser

4.7 0.0 L3 Python

Universal feed parser.
Demiurge

1.4 0.6 L5 Python

PyQuery-based scraping micro-framework.
Pomp

1.4 5.4 L5 Python

Web crawling framework inspired by and similar to Scrapy
FastImage

0.7 1.8 L4 Python

Python library that finds the size / type of an image given its URI by fetching as little as needed

* Code Quality Rankings and insights are calculated and provided by Lumnify.
They vary from L1 to L5 with "L5" being the highest. Visit our partner's website for more details.

Popular Comparisons

Add another 'Web Crawling' Package