pyspider v0.3.7 Release Notes

Release Date: 2016-04-20 // over 5 years ago
  • retry_delay is a dict to specify retry intervals. The items in the dict
    are {retried: seconds}, and a special key: '' (empty string) is used to
    0️⃣ > specify the default retry delay if not specified.

    • πŸ”€ dict parameters in crawl_config, @config will be merged (e.g. headers), thanks to @ihipop
    • βž• add parameter max_redirects in self.crawl to control maximum redirect numbers when doing the fetch, thanks to @AtaLuZiK
    • βž• add parameter validate_cert in self.crawl to ignore the error of server’s certificate.
    • πŸ†• new property etree for Response, etree is a cached lxml.html.HtmlElement object, thanks to @waveyeung
    • πŸ’» you can now pass arguments to phantomjs from command line or config file.
    • πŸ‘Œ support for pymongo 3.0
    • local.projectdb now accept a glob path (e.g. script/*.py) to load multiple projects from local filesystem.
    • queue size in the dashboard is not working for osx, thanks to @xyb
    • counters in dashboard will shown for stopped projects
    • other bug fix