HDF5

A data format I need to know about

June 17, 2020 — June 8, 2022

computers are awful
data sets
standards
Figure 1

HDF5: (Python/R/Java/C/Fortran/MATLAB/whatever) An optimised and flexible data format from the Big Science applications of the 90s. Inbuilt compression and indexing support for things too big for memory. A good default with wide support and good performance. For numerical data, very simple.

The table definition when writing structured data is boring and it mangles text encodings if you aren’t careful, which means a surprising amount of time can be lost writing schemas if there is highly structured data to store.

Built-in lossless compression in HDF5 is not impressive on floats, and their lossy compression is bad. Recent HDF5 supports filters providing fancier methods.

I am currently doing a lot of heavy HDF5 processing, so this is scruffy.

1 Compression

I have been trying to use compression from HDF5, but the built-in options are not great. Extended lossy compression is available via HDF5.

Untidy notes

2 Parallel access

HDF5 with multiple processes is complicated; may be worthwhile.

Useful tools: visualisers.

To install hdf5py using the homebrew HDF5 libraries (useful for Apple Silicon or other weird architectures) try this tip:

brew install hdf5
HDF5_DIR="$(brew --prefix hdf5)" pip install --no-binary=h5py h5py
#or
HDF5_DIR="$(brew --prefix hdf5)" pip install --no-build-isolation h5py

3 Virtual datasets

a.k.a. arrays-made-up-of-multiple-arrays-across-multiple-files.

I wonder how this interacts with parallelism? How about when writing? It sounds like that is a supported means of getting multiple writers if we are careful.

4 References

Diffenderfer, Fox, Hittinger, et al. 2019. “Error Analysis of ZFP Compression for Floating-Point Data.” SIAM Journal on Scientific Computing.
Di, Tao, Liang, et al. 2018. “Efficient Lossy Compression for Scientific Data Based on Pointwise Relative Error Bound.” IEEE Transactions on Parallel and Distributed Systems.
Dube, Tian, Di, et al. 2022. “SIMD Lossy Compression for Scientific Data.” arXiv:2201.04614 [Cs].
Liang, Di, Tao, et al. 2018. “Error-Controlled Lossy Compression Optimized for High Compression Ratios of Scientific Datasets.” In 2018 IEEE International Conference on Big Data (Big Data).
Zhao, Di, Dmitriev, et al. 2021. “Optimizing Error-Bounded Lossy Compression for Scientific Data by Dynamic Spline Interpolation.” In 2021 IEEE 37th International Conference on Data Engineering (ICDE).
Zhao, Di, Liang, et al. 2020. “Significantly Improving Lossy Compression for HPC Datasets with Second-Order Prediction and Parameter Optimization.” In Proceedings of the 29th International Symposium on High-Performance Parallel and Distributed Computing. HPDC ’20.