pyspider v0.3.4 Release Notes
Release Date: 2015-04-21 // almost 9 years ago-
Global
- ๐ New message queue support: beanstalkd by @tiancheng91
- ๐ New global argument:
--logging-config
to specify a customization logging config (to disable werkzeug logs for instance). You can get a sample config from pyspider/logging.conf). - ๐ฆ Project
group
info is added to task package now. - ๐ Change docker base image to cmfatih/phantomjs, you can use phantomjs with same docker image now.
- 0๏ธโฃ Auto restart phantomjs if crash, only enabled in all mode by default.
WebUI
- ๐ Show next
exetime
of a task in task page. - ๐ Show fetch time and process time in tasks page.
- ๐ Show average fetch time and process time in 5min in dashboard page.
- ๐ Show message queue status in dashboard page.
- ๐
limit
andoffset
parameter support in result dump. - ๐ Fix frontend bug when crawling pages with dataurl.
Other
- ๐ Fix support for phantomjs 2.0.
- ๐ Fix scheduler project update inform not work, and use md5sum of script as another signal.
- โฑ Scheduler: periodic counter report in log.
- Fetcher: fix for legacy version of pycurl