dpark v0.5.0 Release Notes

Release Date: 2018-07-27 // over 3 years ago
  • API change

    • โœ‚ Remove module-level api like dpark.textFile.
    • ๐Ÿ‘Œ Support Streaming shuffle and Disk shuffle (Experimental, compatible).

    ๐Ÿ›  Fixes

    • ๐Ÿ› Bug when parsing mfs chunk info.

    ๐Ÿ‘Œ Improvement

    • ๐Ÿ‘ Better broadcast impl using shared memory for tasks on the same slave to reduce memory cost.
    • ๐Ÿ‘ Better offer-matching logic for MesosScheduler which remember bad slaves.
    • ๐Ÿ’… Refactor: style and layout.

    ๐Ÿ†• New Feature

    • Multi segment dump to save memory.
    • Gather statics for stage.
    • ๐Ÿ‘Œ Support run tests/test_rdd on mesos.
    • โž• Add colorful progress bar for dpark.
    • ๐Ÿ‘Œ Support mesos role.
    • ๐Ÿ‘Œ Support multi named mesos master in conf.
    • Loghub for admin.

Previous changes from v0.4.2

    • ๐Ÿ‘Œ Support Python3 & PyPy
    • ๐Ÿ‘Œ Support MooseFS 3.x & refactor on file-system interface