bcolz provides columnar, chunked data containers that can be compressed either in-memory and on-disk. Column storage allows for efficiently querying tables, as well as for cheap column addition and removal. It is based on NumPy, and uses it as the standard data container to communicate with bcolz objects, but it also comes with support for import/export facilities to/from HDF5/PyTables tables and pandas dataframes.

bcolz objects are compressed by default not only for reducing memory/disk storage, but also to improve I/O speed. The compression process is carried out internally by Blosc, a high-performance, multithreaded meta-compressor that is optimized for binary data (although it works with text data just fine too).

bcolz can also use numexpr internally (it does that by default if it detects numexpr installed) so as to accelerate many vector and query operations (although it can use pure NumPy for doing so too). numexpr can optimize the memory usage and use multithreading for doing the computations, so it is blazing fast. This, in combination with carray/ctable disk-based, compressed containers, can be used for performing out-of-core computations efficiently, but most importantly transparently.

Just to whet your appetite, here it is an example with real data, where bcolz is already fulfilling the promise of accelerating memory I/O by using compression:

Programming language: Jupyter Notebook
License: BSD 3-clause "New" or "Revised" License

bcolz alternatives and related packages

Based on the "Science and Data Analysis" category

Do you think we are missing an alternative of bcolz or a related project?

Add another 'Science and Data Analysis' Package

bcolz Recommendations

There are no recommendations yet. Be the first to promote bcolz!

Have you used bcolz? Share your experience. Write a short recommendation and bcolz, you and your project will be promoted on Awesome Python.
Recommend bcolz

Recently added bcolz resources

Do you know of a usefull tutorial, book or news relevant to bcolz?
Be the first to add one!