All Versions
Latest Version
Avg Release Cycle
Latest Release

Changelog History

  • v0.3.0 Changes

    ๐Ÿš€ Does not released yet

    • ๐Ÿ‘ท BaseDownloadWorker.get_one(self, request) -> BaseDownloadWorker.process(self, request)
    • BaseDownloader.get(self, requests) -> BaseDownloader.process(self, crawler, request)
    • non BaseRequest instances are items
    • BaseHttpResponse.get_request() instead of BaseHttpResponse.request property
    • ๐Ÿ‘ป BaseMiddleware.process_(request|response|exception) now may be awaitable
    • BaseCralwer.(process|next_requests|on_processing_done) now may be awaitable
    • BaseDownloader.(start|stop) now may be awaitable
    • BaseDownloader.prepare(self) renamed to BaseDownloader.start(self,crawler)
    • BasePipeline.(start|stop|process) now may be awaitable
    • ๐Ÿ–จ process Future objects without wrapping it with Planned object
    • asynciotools with async/await syntax
    • removed RUNNING, CANCELLED_AND_NOTIFIED, exception method from Planned class
  • v0.2.1 Changes

    ๐Ÿš€ Released at 2016-09-12

    • ๐Ÿ›  fix pipe start/stop/process exception processing
    • fix: AioPomp ensure_future for process_requests in main loop
    • ๐Ÿ–จ BaseCrawler.on_processing_done when request, middlewares, response, extract, pipelines processing was done
    • count param for BaseQueue.get_requests - number allowed concurrent requests for current downloader, may be None
  • v0.2.0 Changes

    ๐Ÿš€ Released at 2016-03-01

    • ๐Ÿšš middlewares logic moved to engine and BaseMiddleware.process_(request|response|exception) with crawler and downloader params
    • Downloader does not have any more middlewares param
    • BaseDownloaderMiddleware renamed to BaseMiddleware
    • BaseDownloderException renamed to BaseCrawlException
    • ๐Ÿ‘ AioConcurrentCrawler with asyncio and concurrent futures support
    • ๐Ÿšš pomp.core.item moved to pomp.contrib.item without backward compatibility
    • ๐Ÿ‘ AioPomp with asyncio support
    • queue semaphore to prevent fetching more requests than downloader can process now
    • ๐Ÿ”ง bfo and dfo orders now configured via Pomp constructor
    • no recursion, internal queue is used
    • twsited is not a part of pomp
    • defer object replaced to Planned object (like Future object)
  • v0.1.3 Changes

    ๐Ÿš€ Will not be released

    • crawler extract_items method can yield next requests
    • process exception on extract_items and next_requests by downloader middlewares
    • ๐Ÿ‘ป store result of calling sys.exc_info in exception instance
  • v0.1.2 Changes

    ๐Ÿš€ Released at 2015-05-08

    • โฑ twistedtools on request timeout can raise ResponseNeverReceived or others (see _newcleint twisted api)...
    • utils.DeferredList works with deferred and not-deferred objects
    • ๐Ÿ›  fixed Item object behavior (get item field value)
  • v0.1.1 Changes

    ๐Ÿš€ Released on December 12nd 2013

    • urllib and twisted downloader on get method yield result
    • ๐Ÿ›  bugfix depth first method
    • concurrenttools change try/finally to generator behavior
    • ๐Ÿ–จ processing requests through queue
    • ๐Ÿ‘ better generator usage
  • v0.1.0 Changes

    ๐Ÿš€ Released on June 14nd 2013

    • ๐Ÿ“‡ rename SimpleDownloader to UrllibDownloader
    • urllib code now in contrib/
    • next_url renamed to next_requests
    • ๐Ÿ‘ async support
    • ๐Ÿ‘ Twisted support
    • ๐Ÿ‘ concurrent future support
    • pipelines accept crawler on start/stop/process calls
    • ๐Ÿ‘ป downloader middleware also process exception
  • v0.0.2 Changes

    ๐Ÿš€ First public preview release.