pyspider v0.3.4 Release Notes

Release Date: 2015-04-21 // almost 9 years ago
  • Global

    • ๐Ÿ†• New message queue support: beanstalkd by @tiancheng91
    • ๐Ÿ†• New global argument: --logging-config to specify a customization logging config (to disable werkzeug logs for instance). You can get a sample config from pyspider/logging.conf).
    • ๐Ÿ“ฆ Project group info is added to task package now.
    • ๐Ÿ”„ Change docker base image to cmfatih/phantomjs, you can use phantomjs with same docker image now.
    • 0๏ธโƒฃ Auto restart phantomjs if crash, only enabled in all mode by default.

    WebUI

    • ๐Ÿ‘‰ Show next exetime of a task in task page.
    • ๐Ÿ‘‰ Show fetch time and process time in tasks page.
    • ๐Ÿ‘‰ Show average fetch time and process time in 5min in dashboard page.
    • ๐Ÿ‘‰ Show message queue status in dashboard page.
    • ๐Ÿ‘ limit and offset parameter support in result dump.
    • ๐Ÿ›  Fix frontend bug when crawling pages with dataurl.

    Other

    • ๐Ÿ›  Fix support for phantomjs 2.0.
    • ๐Ÿ›  Fix scheduler project update inform not work, and use md5sum of script as another signal.
    • โฑ Scheduler: periodic counter report in log.
    • Fetcher: fix for legacy version of pycurl