Charset Normalizer v3.0.1 Release Notes

Release Date: 2022-11-18 // over 1 year ago
  • ๐Ÿ›  Fixed

    • Multi-bytes cutter/chunk generator did not always cut correctly (PR #233)

    ๐Ÿ”„ Changed

    • Speedup provided by mypy/c 0.990 on Python >= 3.7

Previous changes from v3.0.0

  • โž• Added

    • ๐ŸŒฒ Extend the capability of explain=True when cp_isolation contains at most two entries (min one), will log in details of the Mess-detector results
    • ๐Ÿ‘Œ Support for alternative language frequency set in charset_normalizer.assets.FREQUENCIES
    • Add parameter language_threshold in from_bytes, from_path and from_fp to adjust the minimum expected coherence ratio
    • normalizer --version now specify if current version provide extra speedup (meaning mypyc compilation whl)

    ๐Ÿ”„ Changed

    • ๐Ÿ“‡ Build with static metadata using 'build' frontend
    • ๐Ÿ‘‰ Make the language detection stricter
    • Optional: Module can be compiled using Mypyc to provide an extra speedup up to 4x faster than v2.1

    ๐Ÿ›  Fixed

    • CLI with opt --normalize fail when using full path for files
    • ๐Ÿ”Œ TooManyAccentuatedPlugin induce false positive on the mess detection when too few alpha character have been fed to it
    • ๐Ÿ“š Sphinx warnings when generating the documentation

    โœ‚ Removed

    • Coherence detector no longer return 'Simple English' instead return 'English'
    • Coherence detector no longer return 'Classical Chinese' instead return 'Chinese'
    • ๐Ÿ’ฅ Breaking: Method first() and best() from CharsetMatch
    • UTF-7 will no longer appear as "detected" without a recognized SIG/mark (is unreliable/conflict with ASCII)
    • ๐Ÿ’ฅ Breaking: Class aliases CharsetDetector, CharsetDoctor, CharsetNormalizerMatch and CharsetNormalizerMatches
    • ๐Ÿ’ฅ Breaking: Top-level function normalize
    • Breaking: Properties chaos_secondary_pass, coherence_non_latin and w_counter from CharsetMatch
    • ๐Ÿ‘Œ Support for the backport unicodedata2