All Versions
11
Latest Version
Avg Release Cycle
63 days
Latest Release
2132 days ago

Changelog History
Page 1

  • v0.6.41 Changes

    June 24, 2018

    🔄 Changed

    • 👀 Restrict pycurl version to <7.43.0.1 (see #354)

  • v0.6.40 Changes

    May 13, 2018

    🛠 Fixed

    +- 🛠 Fix #346: spider does not process initial_urls
    +- 🛠 Fix #344: raise GrabInvalidUrl for pycurl error 3

  • v0.6.39 Changes

    May 09, 2018

    🛠 Fixed

    • 🛠 Fix bug: task generator works incorrectly
    • 🛠 Fix bug: pypi package misses http api html file
    • 🛠 Fix bug: dictionary changed size during iteration in stat logging
    • 🛠 Fix bug: multiple errors in urllib3 transport and threaded network service
    • 🛠 Fix short names of errors in stat logging
    • 👌 Improve error handling in urrllib3 transport
    • 🛠 Fix #299: multi-added errors
    • 🛠 Fix bug: pypi package misses http api html file
    • 🛠 Fix #285: pyquery extension parses html incorrectly
    • 🛠 Fix #267: normalize handling of too many redirect error
    • 🛠 Fix #268: fix processing of utf cookies
    • 🛠 Fix #241: form_fields() fails on some HTML forms
    • 🛠 Fix normalize_unicode issue in debug post method
    • 🛠 Fix #323: urllib3 transport fails with UnicodeError on some invalid URLs
    • 🛠 Fix #31: support for multivalue form inputs
    • 🛠 Fix #328, fix #67: remove hard link between document and grab
    • 🛠 Fix #284: option headers affects content of common_headers
    • 🛠 Fix #293: processing non-latin chars in Location header
    • 🛠 Fix #324: refactor response header processing

    🔄 Changed

    • 🔨 Refactor Spider into set of async. services
    • ➕ Add certifi dependency into grab[full] setup target
    • 🛠 Fix #315: use psycopg2-binary package for postgres cache
    • Related to #206: do not use connection_reuse=False for proxy connections in spider

    ✂ Removed

    • ✂ Remove cache timeout option
    • ✂ Remove structured extension

  • v0.6.38 Changes

    May 17, 2017

    🛠 Fixed

    • 🛠 Fix "error:None" in spider rps logging
    • 🛠 Fix race condition bug in task generator

    ➕ Added

    • ➕ Add original_exc attribute to GrabNetworkError (and subclasses) that points to original exception

    🔄 Changed

    • ✂ Remove IOError from the ancestors of GrabNetworkError
    • ➕ Add default values to --spider-transport and --grab-transport options of crawl script

  • v0.6.37 Changes

    May 13, 2017

    ➕ Added

    • ➕ Add --spider-transport and --grab-transport options to crawl script
    • ➕ Add SOCKS5 proxy support in urllib3 transport

    🛠 Fixed

    • 🛠 Fix #237: urllib3 transport fails without pycurl installed
    • 🛠 Fix bug: incorrect spider request logging when cache is enabled
    • 🛠 Fix bug: crawl script fails while trying to process a lock key
    • 🛠 Fix bug: urllib3 transport fails while trying to throw GrabConnectionError exception
    • 🛠 Fix bug: Spider add_task method fails while trying to log invalid URL error

    ✂ Removed

    • Remove obsoleted hammer_mode and hammer_timeout config options

  • v0.6.36 Changes

    February 12, 2017

    ➕ Added

    • ➕ Add pylint to default test set

    🛠 Fixed

    • 🛠 Fix #229: using deprecated response object inside Grab

    ✂ Removed

    • ✂ Remove spider project template and start_project script

  • v0.6.35 Changes

    February 06, 2017

    🛠 Fixed

    • 🛠 Fix bug in deprecated grab.choose_form method
    • ➕ Add default project templates files to the distribution, by @rushter
    • 🛠 Fix #222: debug_post option fails with big post data
    • 🛠 Fix #148: pycurl ignores sigint signal

  • v0.6.34 Changes

    February 04, 2017

    ➕ Added

    • ✅ Start running Grab tests in OSX environment on travis CI

    🔄 Changed

    • 📜 Use defusedxml library to parse HTML and XML, by @kevinlondon
    • Put selection, lxml and pycurl libs back to required dependencies in setup.py
    • 📚 Update installation documentation

  • v0.6.33 Changes

    January 28, 2017

    ➕ Added

    • ➕ Add API documentation about few grab modules, by @rushter
    • 🏁 Start running Grab tests in Windows enviroment on appveyor CI
    • 🆕 New spider transport based on threads that allows to use Spider with any Grab network backend e.g. urllib3
    • Add remove_from_post option to grab.doc.submit method
    • ➕ Add random option to grab.change_proxy method
    • 👌 Support for deprecated attributes Spider.items and Spider.counters
    • If Spider handler raises ResponseNotValid exception, then that task goes back to task queue until task.task_try_count reaches the spider.task_try_limit

    🔄 Changed

    • 🔨 Refactor management of internal threads, fix random test failures related to cache sub-module
    • 0️⃣ Disable default logging to files while running spider by run crawl command
    • Multiple improvements in urllib3 transport
    • 0️⃣ Set default spider network & try limits to 3 (was 10)

    🛠 Fixed

    • Different bugs in urllib3 transport
    • Different bugs

    ✂ Removed

    • Remove grab.use_next_proxy method
    • ✂ Remove grab.dump method
    • ✂ Remove deprecated Spider methods and attributes

  • v0.6.32 Changes

    December 31, 2017

    🛠 Fixed

    • 🛠 Fix setup.py