pyspider v0.3.10 Release Notes
Release Date: 2018-04-18 // about 7 years ago-
๐ New features:
- โ add phantomjs proxy support #692 @volvofixthis
- ๐ support redis 3.x in cluster mode for message queue @hackty
๐ Fix several bugs:
- ๐ Improve the performance of counter.to_dict
- ๐ Fixed issue of counter changed during read
- ๐ Fix tornado version dependency in setup.py
Previous changes from v0.3.9
-
๐ New features:
- ๐ Support for Python 3.6.
- โฑ Auto Pause: the project will be paused for
scheduler.PAUSE_TIME
(default: 5min) when lastscheduler.FAIL_PAUSE_NUM
(default: 10) task failed, and dispatchscheduler.UNPAUSE_CHECK_NUM
(default: 3) tasks afterscheduler.PAUSE_TIME
. Project will resume if any one of lastscheduler.UNPAUSE_CHECK_NUM
tasks success. - 0๏ธโฃ Each callback now have a default 30s process time limit. (Platform support required) @beader
- ๐ New Javascript render engine - Splash support: Enabled by fetch argument
--splash-endpoint=http://splash:8050/execute
- ๐ Python3 webdav support.
- ๐ Python3
from projects import project
support. - A link to corresponding task is added to webui debug page when debugging a exists task in webui.
- ๐ New
user_agent
parameter inself.crawl
, you can set user-agent by headers though.
๐ Fix several bugs:
- ๐ New webui dashboard frontend framework - vue.js, improved the performance when having large number of tasks (e.g. http://demo.pyspider.org/)
- ๐ Fix crawl_config doesn't work in webui while debugging a script issue.
- ๐ Fix CSS Selector Helper doesn't work issue. @ackalker
- ๐ Fix
connection_timeout
not working issue. - ๐ FIx
need_auth
option not applied on webdav issue. - ๐ Fix "fix can't dump counter to file: scheduler.all" error.
- ๐ Some other fixes