Changelog History
Page 1
-
v2.0.2 Changes
September 29, 20202.0.2 (Sep 29, 2020)
Consumer
- KIP-54: Implement sticky partition assignment strategy (aynroot / PR #2057)
- ๐ Fix consumer deadlock when heartbeat thread request timeout (huangcuiyang / PR #2064)
Compatibility
- ๐ Python 3.8 support (Photonios / PR #2088)
Cleanups
- โฌ๏ธ Bump dev requirements (jeffwidman / PR #2129)
- ๐ Fix crc32c deprecation warning (crc32c==2.1) (jeffwidman / PR #2128)
- ๐ Lint cleanup (jeffwidman / PR #2126)
- ๐ Fix initialization order in KafkaClient (pecalleja / PR #2119)
- ๐ Allow installing crc32c via extras (mishas / PR #2069)
- โ Remove unused imports (jameslamb / PR #2046)
Admin Client
- Merge _find_coordinator_id methods (jeffwidman / PR #2127)
- ๐ Feature: delete consumergroups (swenzel / PR #2040)
- ๐ Allow configurable timeouts in admin client check version (sunnyakaxd / PR #2107)
- โจ Enhancement for Kafka Admin Client's "Describe Consumer Group" (Apurva007 / PR #2035)
Protocol
- โ Add support for zstd compression (gabriel-tincu / PR #2021)
- โ Add protocol support for brokers 1.1.0 - 2.5.0 (gabriel-tincu / PR #2038)
- โ Add ProduceRequest/ProduceResponse v6/v7/v8 (gabriel-tincu / PR #2020)
- ๐ Fix parsing NULL header values (kvfi / PR #2024)
โ Tests
- โ Add 2.5.0 to automated CI tests (gabriel-tincu / PR #2038)
- โ Add 2.1.1 to build_integration (gabriel-tincu / PR #2019)
๐ Documentation / Logging / Errors
- ๐ฒ Disable logging during producer object gc (gioele / PR #2043)
- โก๏ธ Update example.py; use threading instead of multiprocessing (Mostafa-Elmenbawy / PR #2081)
- ๐ Fix typo in exception message (haracejacob / PR #2096)
- โ Add kafka.structs docstrings (Mostafa-Elmenbawy / PR #2080)
- ๐ Fix broken compatibility page link (anuragrana / PR #2045)
- ๐ Rename README to README.md (qhzxc0015 / PR #2055)
- ๐ Fix docs by adding SASL mention (jeffwidman / #1990)
-
v2.0.1 Changes
February 19, 2020Admin Client
- KAFKA-8962: Use least_loaded_node() for AdminClient.describe_topics() (jeffwidman / PR #2000)
- ๐ Fix AdminClient topic error parsing in MetadataResponse (jtribble / PR #1997)
-
v2.0.0 Changes
February 10, 20202.0.0 (Feb 10, 2020)
๐ This release includes breaking changes for any application code that has not
๐ migrated from older Simple-style classes to newer Kafka-style classes.๐ Deprecation
- โ Remove deprecated SimpleClient, Producer, Consumer, Unittest (jeffwidman / PR #1196)
Admin Client
- ๐ Use the controller for topic metadata requests (TylerLubeck / PR #1995)
- Implement list_topics, describe_topics, and describe_cluster (TylerLubeck / PR #1993)
- Implement eq and hash for ACL objects (TylerLubeck / PR #1955)
- ๐ Fixes KafkaAdminClient returning
IncompatibleBrokerVersion
when passing anapi_version
(ian28223 / PR #1953) - โก๏ธ Admin protocol updates (TylerLubeck / PR #1948)
- ๐ Fix describe config for multi-broker clusters (jlandersen / PR #1869)
๐ Miscellaneous Bugfixes / Improvements
- Enable SCRAM-SHA-256 and SCRAM-SHA-512 for sasl (swenzel / PR #1918)
- ๐ Fix slots usage and use more slots (carsonip / PR #1987)
- ๐ Optionally return OffsetAndMetadata from consumer.committed(tp) (dpkp / PR #1979)
- ๐ป Reset conn configs on exception in conn.check_version() (dpkp / PR #1977)
- โฑ Do not block on sender thread join after timeout in producer.close() (dpkp / PR #1974)
- Implement methods to convert a Struct object to a pythonic object (TylerLubeck / PR #1951)
๐ Test Infrastructure / Documentation / Maintenance
- โก๏ธ Update 2.4.0 resource files for sasl integration (dpkp)
- โ Add kafka 2.4.0 to CI testing (vvuibert / PR #1972)
- convert test_admin_integration to pytest (ulrikjohansson / PR #1923)
- xfail test_describe_configs_topic_resource_returns_configs (dpkp / Issue #1929)
- โ Add crc32c to README and docs (dpkp)
- Improve docs for reconnect_backoff_max_ms (dpkp / PR #1976)
- ๐ Fix simple typo: managementment -> management (timgates42 / PR #1966)
- ๐ Fix typos (carsonip / PR #1938)
- ๐ Fix doc import paths (jeffwidman / PR #1933)
- โก๏ธ Update docstring to match conn.py's (dabcoder / PR #1921)
- ๐ Do not log topic-specific errors in full metadata fetch (dpkp / PR #1980)
- ๐ Raise AssertionError if consumer closed in poll() (dpkp / PR #1978)
- ๐ฒ Log retriable coordinator NodeNotReady, TooManyInFlightRequests as debug not error (dpkp / PR #1975)
- โ Remove unused import (jeffwidman)
- โ Remove some dead code (jeffwidman)
- ๐ Fix a benchmark to Use print() function in both Python 2 and Python 3 (cclauss / PR #1983)
- ๐ Fix a test to use ==/!= to compare str, bytes, and int literals (cclauss / PR #1984)
- ๐ Fix benchmarks to use pyperf (carsonip / PR #1986)
- โ Remove unused/empty .gitsubmodules file (jeffwidman / PR #1928)
- โ Remove deprecated
ConnectionError
(jeffwidman / PR #1816)
-
v1.4.7 Changes
September 30, 20191.4.7 (Sep 30, 2019)
๐ This release is focused on KafkaConsumer performance, Admin Client
๐ improvements, and Client concurrency. The KafkaConsumer iterator implementation
๐ has been greatly simplified so that it just wraps consumer.poll(). The prior
๐ implementation will remain available for a few more releases using the optional
KafkaConsumer config:legacy_iterator=True
. This is expected to improve
consumer throughput substantially and help reduce heartbeat failures / group
rebalancing.Major thanks to @carsonip @Baisang @iv-m @davidheitman @cardy31 @ulrikjohansson @iAnomaly @Wayde2014 @ossdev07 @commanderdishwasher @justecorruptio @melor @rustyrothwurt @sachiin @jacky15 and @rikonen for submitting PRs; thanks as well to everyone that
submitted bug reports and issues, and to @jeffwidman and @tvoinarovskyi for code
โ reviews, comments, testing, debugging, and helping to maintain kafka-python!Client
- Send socket data via non-blocking IO with send buffer (@dpkp / PR #1912)
- Rely on socket selector to detect completed connection attempts (@dpkp / PR #1909)
- ๐ Improve connection lock handling; always use context manager (@melor @dpkp / PR #1895)
- โฌ๏ธ Reduce client poll timeout when there are no in-flight requests (@dpkp / PR #1823)
KafkaConsumer
- Do not use wakeup when sending fetch requests from consumer (@dpkp / PR #1911)
- ๐ Wrap
consumer.poll()
for KafkaConsumer iteration (@dpkp / PR #1902) - ๐ Allow the coordinator to auto-commit on old brokers (@justecorruptio / PR #1832)
- โฌ๏ธ Reduce internal client poll timeout for (legacy) consumer iterator interface (@dpkp / PR #1824)
- ๐ Use dedicated connection for group coordinator (@dpkp / PR #1822)
- ๐ Change coordinator lock acquisition order (@dpkp / PR #1821)
- Make
partitions_for_topic
a read-through cache (@Baisang / PR #1781,#1809) - ๐ Fix consumer hanging indefinitely on topic deletion while rebalancing (@commanderdishwasher / PR #1782)
๐ Miscellaneous Bugfixes / Improvements
- ๐ Fix crc32c avilability on non-intel architectures (@ossdev07 / PR #1904)
- 0๏ธโฃ Load system default SSL CAs if
ssl_cafile
is not provided (@iAnomaly / PR #1883) - Catch py3 TimeoutError in BrokerConnection send/recv (@dpkp / PR #1820)
- โ Added a function to determine if bootstrap is successfully connected (@Wayde2014 / PR #1876)
Admin Client
- โ Add ACL api support to KafkaAdminClient (@ulrikjohansson / PR #1833)
- Add
sasl_kerberos_domain_name
config to KafkaAdminClient (@jeffwidman / PR #1852) - ๐ Update
security_protocol
config documentation for KafkaAdminClient (@cardy31 / PR #1849) - Break FindCoordinator into request/response methods in KafkaAdminClient (@jeffwidman / PR #1871)
- Break consumer operations into request / response methods in KafkaAdminClient (@jeffwidman / PR #1845)
- Parallelize calls to
_send_request_to_node()
in KafkaAdminClient (@davidheitman / PR #1807)
๐ Test Infrastructure / Documentation / Maintenance
- โ Add Kafka 2.3.0 to test matrix and compatibility docs (dpkp / PR #1915)
- โ
Convert remaining
KafkaConsumer
tests topytest
(@jeffwidman / PR #1886) - โฌ๏ธ Bump integration tests to 0.10.2.2 and 0.11.0.3 (@jeffwidman / #1890)
- โ
Cleanup handling of
KAFKA_VERSION
env var in tests (@jeffwidman / PR #1887) - โ Minor test cleanup (@jeffwidman / PR #1885)
- โ
Use
socket.SOCK_STREAM
in test assertions (@iv-m / PR #1879) - Sanity test for
consumer.topics()
andconsumer.partitions_for_topic()
(@Baisang / PR #1829) - ๐ Cleanup seconds conversion in client poll timeout calculation (@jeffwidman / PR #1825)
- โ Remove unused imports (@jeffwidman / PR #1808)
- Cleanup python nits in RangePartitionAssignor (@jeffwidman / PR #1805)
- โก๏ธ Update links to kafka consumer config docs (@jeffwidman)
- ๐ Fix minor documentation typos (@carsonip / PR #1865)
- โ Remove unused/weird comment line (@jeffwidman / PR #1813)
- ๐ Update docs for
api_version_auto_timeout_ms
(@jeffwidman / PR #1812)
-
v1.4.6 Changes
April 02, 20191.4.6 (Apr 2, 2019)
๐ This is a patch release primarily focused on bugs related to concurrency,
โ SSL connections and testing, and SASL authentication.
Major thanks to @pt2pham , @isamaru , @braedon , @gingercookiemage , for submitting
PRs to help fix many of these issues. And major thanks to everyone that
submitted bug reports and issues. And thanks always to @jeffwidman and @tvoinarovskyi for code
โ reviews, comments, testing, debugging, and helping to maintain this project!Client Concurrency Issues (Race Conditions / Deadlocks)
- ๐ Fix race condition in
protocol.send_bytes
(isamaru / PR #1752) - ๐ Do not call
state_change_callback
with lock (dpkp / PR #1775) - โ Additional BrokerConnection locks to synchronize protocol/IFR state (dpkp / PR #1768)
- Send pending requests before waiting for responses (dpkp / PR #1762)
- Avoid race condition on
client._conns
in send() (dpkp / PR #1772) - ๐ Hold lock during
client.check_version
(dpkp / PR #1771)
Producer Wakeup / TimeoutError
- Dont wakeup during
maybe_refresh_metadata
-- it is only called by poll() (dpkp / PR #1769) - Dont do client wakeup when sending from sender thread (dpkp / PR #1761)
โ SSL - Python3.7 Support / Bootstrap Hostname Verification / Testing
- Wrap SSL sockets after connecting for python3.7 compatibility (dpkp / PR #1754)
- ๐ Allow configuration of SSL Ciphers (dpkp / PR #1755)
- ๐ Maintain shadow cluster metadata for bootstrapping (dpkp / PR #1753)
- โ Generate SSL certificates for local testing (dpkp / PR #1756)
- ๐ Rename ssl.keystore.location and ssl.truststore.location config files (dpkp)
- Reset reconnect backoff on SSL connection (dpkp / PR #1777)
๐ SASL - OAuthBearer support / api version bugfix
- ๐ Fix 0.8.2 protocol quick detection / fix SASL version check (dpkp / PR #1763)
- โก๏ธ Update sasl configuration docstrings to include supported mechanisms (dpkp)
- ๐ Support SASL OAuthBearer Authentication (pt2pham / PR #1750)
๐ Miscellaneous Bugfixes
- ๐ Dont force metadata refresh when closing unneeded bootstrap connections (dpkp / PR #1773)
- Fix possible AttributeError during conn._close_socket (dpkp / PR #1776)
- Return connection state explicitly after close in connect() (dpkp / PR #1778)
- ๐ Fix flaky conn tests that use time.time (dpkp / PR #1758)
- โ Add py to requirements-dev (dpkp)
- ๐ Fixups to benchmark scripts for py3 / new KafkaFixture interface (dpkp)
- ๐ Fix race condition in
-
v1.4.5 Changes
March 14, 20191.4.5 (Mar 14, 2019)
๐ This release is primarily focused on addressing lock contention
and other coordination issues between the KafkaConsumer and the
๐ background heartbeat thread that was introduced in the 1.4 release.Consumer
- connections_max_idle_ms must be larger than request_timeout_ms (jeffwidman / PR #1688)
- Avoid race condition during close() / join heartbeat thread (dpkp / PR #1735)
- ๐ Use last offset from fetch v4 if available to avoid getting stuck in compacted topic (keithks / PR #1724)
- ๐ Synchronize puts to KafkaConsumer protocol buffer during async sends (dpkp / PR #1733)
- ๐ Improve KafkaConsumer join group / only enable Heartbeat Thread during stable group (dpkp / PR #1695)
- Remove unused
skip_double_compressed_messages
(jeffwidman / PR #1677) - Fix commit_offsets_async() callback (Faqa / PR #1712)
Client
- Retry bootstrapping after backoff when necessary (dpkp / PR #1736)
- ๐ Recheck connecting nodes sooner when refreshing metadata (dpkp / PR #1737)
- Avoid probing broker versions twice on newer brokers (dpkp / PR #1738)
- ๐ Move all network connections and writes to KafkaClient.poll() (dpkp / PR #1729)
- ๐ Do not require client lock for read-only operations (dpkp / PR #1730)
- โฑ Timeout all unconnected conns (incl SSL) after request_timeout_ms (dpkp / PR #1696)
Admin Client
- ๐ Fix AttributeError in response topic error codes checking (jeffwidman)
- Fix response error checking in KafkaAdminClient send_to_controller (jeffwidman)
- ๐ Fix NotControllerError check (jeffwidman)
Core/Protocol
- ๐ Fix default protocol parser version / 0.8.2 version probe (dpkp / PR #1740)
- ๐ Make NotEnoughReplicasError/NotEnoughReplicasAfterAppendError retriable (le-linh / PR #1722)
๐ Bugfixes
- ๐ Use copy() in metrics() to avoid thread safety issues (emeric254 / PR #1682)
โ Test Infrastructure
- โ Mock dns lookups in test_conn (dpkp / PR #1739)
- โ Use test.fixtures.version not test.conftest.version to avoid warnings (dpkp / PR #1731)
- Fix test_legacy_correct_metadata_response on x86 arch (stanislavlevin / PR #1718)
- ๐ Travis CI: 'sudo' tag is now deprecated in Travis (cclauss / PR #1698)
- ๐ Use Popen.communicate() instead of Popen.wait() (Baisang / PR #1689)
Compatibility
- Catch thrown OSError by python 3.7 when creating a connection (danjo133 / PR #1694)
- โก๏ธ Update travis test coverage: 2.7, 3.4, 3.7, pypy2.7 (jeffwidman, dpkp / PR #1614)
- โฌ๏ธ Drop dependency on sphinxcontrib-napoleon (stanislavlevin / PR #1715)
- โ Remove unused import from kafka/producer/record_accumulator.py (jeffwidman / PR #1705)
- ๐ Fix SSL connection testing in Python 3.7 (seanthegeek, silentben / PR #1669)
-
v1.4.4 Changes
November 20, 2018๐ Bugfixes
- (Attempt to) Fix deadlock between consumer and heartbeat (zhgjun / dpkp #1628)
- ๐ Fix Metrics dict memory leak (kishorenc #1569)
Client
- ๐ Support Kafka record headers (hnousiainen #1574)
- โฑ Set socket timeout for the write-side of wake socketpair (Fleurer #1577)
- โ Add kerberos domain name config for gssapi sasl mechanism handshake (the-sea #1542)
- ๐ Support smaller topic metadata fetch during bootstrap (andyxning #1541)
- โฑ Use TypeError for invalid timeout type (jeffwidman #1636)
- ๐ Break poll if closed (dpkp)
Admin Client
- โ Add KafkaAdminClient class (llamahunter #1540)
- Fix list_consumer_groups() to query all brokers (jeffwidman #1635)
- Stop using broker-errors for client-side problems (jeffwidman #1639)
- ๐ Fix send to controller (jeffwidman #1640)
- โ Add group coordinator lookup (jeffwidman #1641)
- ๐ Fix describe_groups (jeffwidman #1642)
- Add list_consumer_group_offsets() (jeffwidman #1643)
- โ Remove support for api versions as strings from KafkaAdminClient (jeffwidman #1644)
- Set a clear default value for
validate_only
/include_synonyms
(jeffwidman #1645) - Bugfix: Always set this_groups_coordinator_id (jeffwidman #1650)
Consumer
- ๐ Fix linter warning on import of ConsumerRebalanceListener (ben-harack #1591)
- โ Remove ConsumerTimeout (emord #1587)
- Return future from commit_offsets_async() (ekimekim #1560)
Core / Protocol
- โ Add protocol structs for {Describe,Create,Delete} Acls (ulrikjohansson #1646/partial)
- Pre-compile pack/unpack function calls (billyevans / jeffwidman #1619)
- Don't use
kafka.common
internally (jeffwidman #1509) - Be explicit with tuples for %s formatting (jeffwidman #1634)
๐ Documentation
- Document connections_max_idle_ms (jeffwidman #1531)
- ๐ Fix sphinx url (jeffwidman #1610)
- โก๏ธ Update remote urls: snappy, https, etc (jeffwidman #1603)
- โ Minor cleanup of testing doc (jeffwidman #1613)
- ๐ Various docstring / pep8 / code hygiene cleanups (jeffwidman #1647)
โ Test Infrastructure
- ๐ Stop pinning
pylint
(jeffwidman #1611) - โ
(partial) Migrate from
Unittest
topytest
(jeffwidman #1620) - โ Minor aesthetic cleanup of partitioner tests (jeffwidman #1618)
- Cleanup fixture imports (jeffwidman #1616)
- ๐ Fix typo in test file name (jeffwidman)
- โ Remove unused ivy_root variable (jeffwidman)
- โ Add test fixtures for kafka versions 1.0.2 -> 2.0.1 (dpkp)
- โฌ๏ธ Bump travis test for 1.x brokers to 1.1.1 (dpkp)
๐ฒ Logging / Error Messages
- ๐ฆ raising logging level on messages signalling data loss (sibiryakov #1553)
- ๐ Stop using deprecated log.warn() (jeffwidman #1615)
- ๐ Fix typo in logging message (jeffwidman)
Compatibility
-
v1.4.3 Changes
May 26, 2018Compatibility
- ๐ Fix for python 3.7 support: remove 'async' keyword from SimpleProducer (dpkp #1454)
Client
- ๐ Improve BrokerConnection initialization time (romulorosa #1475)
- ๐ Ignore MetadataResponses with empty broker list (dpkp #1506)
- ๐ Improve connection handling when bootstrap list is invalid (dpkp #1507)
Consumer
- Check for immediate failure when looking up coordinator in heartbeat thread (dpkp #1457)
Core / Protocol
- ๐ Always acquire client lock before coordinator lock to avoid deadlocks (dpkp #1464)
- โ Added AlterConfigs and DescribeConfigs apis (StephenSorriaux #1472)
- ๐ Fix CreatePartitionsRequest_v0 (StephenSorriaux #1469)
- โ Add codec validators to record parser and builder for all formats (tvoinarovskyi #1447)
- ๐ Fix MemoryRecord bugs re error handling and add test coverage (tvoinarovskyi #1448)
- ๐ฎ Force lz4 to disable Kafka-unsupported block linking when encoding (mnito #1476)
- Stop shadowing
ConnectionError
(jeffwidman #1492)
๐ Documentation
- Document methods that return None (jeffwidman #1504)
- Minor doc capitalization cleanup (jeffwidman)
- Adds add_callback/add_errback example to docs (Berkodev #1441)
- โฑ Fix KafkaConsumer docstring for request_timeout_ms default (dpkp #1459)
โ Test Infrastructure
- โ Skip flakey SimpleProducer test (dpkp)
- ๐ Fix skipped integration tests if KAFKA_VERSION unset (dpkp #1453)
๐ฒ Logging / Error Messages
- ๐ Stop using deprecated log.warn() (jeffwidman)
- ๐ Change levels for some heartbeat thread logging (dpkp #1456)
- ๐ฒ Log Heartbeat thread start / close for debugging (dpkp)
-
v1.4.2 Changes
March 10, 2018๐ Bugfixes
- Close leaked selector in version check (dpkp #1425)
- ๐ Fix
BrokerConnection.connection_delay()
to return milliseconds (dpkp #1414) - Use local copies in
Fetcher._fetchable_partitions
to avoid mutation errors (dpkp #1400) - ๐ Fix error var name in
_unpack
(j2gg0s #1403) - ๐ Fix KafkaConsumer compacted offset handling (dpkp #1397)
- ๐ Fix byte size estimation with kafka producer (blakeembrey #1393)
- ๐ Fix coordinator timeout in consumer poll interface (braedon #1384)
Client
- โ Add
BrokerConnection.connect_blocking()
to improve bootstrap to multi-address hostnames (dpkp #1411) - Short-circuit
BrokerConnection.close()
if already disconnected (dpkp #1424) - Only increase reconnect backoff if all addrinfos have been tried (dpkp #1423)
- ๐ Make BrokerConnection .host / .port / .afi immutable to avoid incorrect 'metadata changed' checks (dpkp #1422)
- ๐ Connect with sockaddrs to support non-zero ipv6 scope ids (dpkp #1433)
- โฑ Check timeout type in KafkaClient constructor (asdaraujo #1293)
- โก๏ธ Update string representation of SimpleClient (asdaraujo #1293)
- Do not validate
api_version
against known versions (dpkp #1434)
Consumer
- ๐ Avoid tight poll loop in consumer when brokers are down (dpkp #1415)
- ๐ Validate
max_records
in KafkaConsumer.poll (dpkp #1398) - ๐ KAFKA-5512: Awake heartbeat thread when it is time to poll (dpkp #1439)
Producer
- Validate that serializers generate bytes-like (or None) data (dpkp #1420)
Core / Protocol
- ๐ Support alternative lz4 package: lz4framed (everpcpc #1395)
- ๐ Use hardware accelerated CRC32C function if available (tvoinarovskyi #1389)
- โ Add Admin CreatePartitions API call (alexef #1386)
โ Test Infrastructure
- โ Close KafkaConsumer instances during tests (dpkp #1410)
- โ Introduce new fixtures to prepare for migration to pytest (asdaraujo #1293)
- โ Removed pytest-catchlog dependency (asdaraujo #1380)
- ๐ Fixes racing condition when message is sent to broker before topic logs are created (asdaraujo #1293)
- โ Add kafka 1.0.1 release to test fixtures (dpkp #1437)
๐ฒ Logging / Error Messages
-
v1.4.1 Changes
February 09, 2018