dpark v0.5.0 Release Notes
Release Date: 2018-07-27 // about 5 years ago-
API change
- โ Remove module-level api like
dpark.textFile
. - ๐ Support Streaming shuffle and Disk shuffle (Experimental, compatible).
๐ Fixes
- ๐ Bug when parsing mfs chunk info.
๐ Improvement
- ๐ Better broadcast impl using shared memory for tasks on the same slave to reduce memory cost.
- ๐ Better offer-matching logic for MesosScheduler which remember bad slaves.
- ๐ Refactor: style and layout.
๐ New Feature
- Multi segment dump to save memory.
- Gather statics for stage.
- ๐ Support run tests/test_rdd on mesos.
- โ Add colorful progress bar for dpark.
- ๐ Support mesos role.
- ๐ Support multi named mesos master in conf.
- Loghub for admin.
- โ Remove module-level api like
Previous changes from v0.4.2
-
- ๐ Support Python3 & PyPy
- ๐ Support MooseFS 3.x & refactor on file-system interface