All Versions
30
Latest Version
Avg Release Cycle
44 days
Latest Release
-

Changelog History
Page 1

  • v1.2.1 Changes

    • --precision and --recall arguments added to the CLI
    • 👍 better text cleaning: paywalls and comments
    • 👌 improvements for Chinese websites (with @glacierck & @immortal-autumn): #186, #187, #188
    • 🛠 further bugs fixed: #189, #192 (with @felipehertzer), #200
    • 🖨 efficiency: faster module loading and improved RAM footprint

  • v1.2.0 Changes

    • efficiency: replaced module readability-lxml by trimmed fork
    • 🐛 bug fixed: (#179, #180, #183, #184)
    • 👌 improved baseline extraction
    • 📇 cleaner metadata (with @felipehertzer)

  • v1.1.0 Changes

    • 👍 encodings: better detection, output NFC-normalized Unicode
    • 🐎 maintenance and performance: more efficient code
    • 🐛 bugs fixed (#119, #136, #147, #160, #161, #162, #164, #167 and others)
    • prepare compatibility with upcoming Python 3.11
    • 🔄 changed default settings
    • 📚 extended documentation

  • v1.0.0 Changes

    • compress HTML backup files & seamlessly open .gz files
    • 👌 support JSON web feeds
    • 📦 graphical user interface integrated into main package
    • faster downloads: reviewed backoff, compressed data
    • optional modules: downloads with pycurl, language identification with py3langid
    • 🐛 bugs fixed (#111, #125, #132, #136, #140)
    • 🛠 minor optimizations and fixes by @vbarbaresi in #124 & #130
    • 🛠 fixed array with single or multiples entries on json extractor by @felipehertzer in #143
    • 🔨 code base refactored with @sourcery-ai #121, improved and optimized for Python 3.6+
    • ⬇️ drop support for Python 3.5

  • v0.9.3 Changes

    • 👍 better, faster encoding detection: replaced chardet with charset_normalizer
    • ⚡️ faster execution: updated justext to 3.0
    • 👍 better extraction of sub-elements in tables (#78, #90)
    • 📜 more robust web feed parsing
    • further defined precision- and recall-oriented settings
    • license extraction in footers (#118)

  • v0.9.2 Changes

    • first precision- and recall-oriented presets defined
    • 👌 improvements in authorship extraction (thanks @felipehertzer)
    • requesting TXT output with formatting now results in Markdown format
    • 🐛 bugs fixed: notably extraction robustness and consistency (#109, #111, #113)
    • setting for cookies in request headers (thanks @muellermartin)
    • 👍 better date extraction thanks to htmldate update

  • v0.9.1 Changes

    • 👌 improved author extraction (thanks @felipehertzer!)
    • 🐛 bugs fixed: HTML element handling, HTML meta attributes, spider, CLI, ...
    • 📄 docs updated and extended
    • 🗄 CLI: option names normalized (heed deprecation warnings), new option explore

  • v0.9.0 Changes

    • focused crawling functions including politeness rules
    • more efficient multi-threaded downloads + use as Python functions
    • 📚 documentation extended
    • 🐛 bugs fixed: extraction and URL handling
    • ✂ removed support for Python 3.4

  • v0.8.2 Changes

    • 👍 better handling of formatting, links and images, title type as attribute in XML formats
    • more robust sitemaps and feeds processing
    • more accurate extraction
    • 🛠 further consolidation: code simplified and bugs fixed

  • v0.8.1 Changes

    • 👍 extraction trade-off: slightly better recall
    • 🔧 code robustness: requests, configuration and navigation
    • 🛠 bugfixes: image data extraction