Replacing HDF5 interface

[pedrocamargo]

We have now had issues with OMX-Python in a few instances due to the slow development of PyTables.

Would replacing PyTables with H5Py be reasonable/acceptable?

In case positive, I would be happy to contribute that code.

[billyc]

Hi, yes!

In the original development of omx (years ago now!), h5py was lagging behind the pytables implementation. Specifically it didn't support any fast compression filters, which is why we went with the slow gzip compression: that was supported by all HDF5 libraries and all languages. I haven't been paying attention at all to the development of the libraries since then.

If pytables is now causing problems, and h5py has feature-parity or better, I'd gladly support the switch. OMX is used in many different contexts now, so testing the change is important: the switch can't break existing workflows. It would also be great to finally support a better/faster compression algorithm; Simwrapper struggles to decompress large OMX files in real time!

Final note - in addition to your suggestion, there are conversations about adding a completely different NON-HDF5 based matrix format to OMX which would be more flexible with modern tooling. This is just an idea right now but @jpn-- and others have raised it several times. Just wanted to put that here in case we want to have a multifaceted OMX upgrade (perhaps OMX 2) in the near future.

[pedrocamargo]

That's great, @billyc

I'll put together a PR with the switch and some much-needed software engineering improvements (modern packaging, integrated CI through GitHub actions, etc.).

I am working through the proposed code in this PR (pedrocamargo#1) on my fork and will squash commits before submitting the PR. I am not able to test it beyond Simwrapper, AequilibraE, and POLARIS, however.

On @jpn--'s suggestion, I agree that alternative formats could help quite a bit, but even sparse formats based on standard sparse matrices (e.g. csr/coo) would already be a huge improvement. We have implemented that support in a private fork and would be happy to add that as an additional PR if of interest.

[billyc]

Well, these all sound like very nice features to have but we need to remember that OMX files which match the specification need to be readable in all of our supported languages, not just Python. Changing from pytables to h5py won't change the file format but rather just the tech used to read/write the file.

Updating the OMX spec with better compression, sparse matrix support, or other container formats needs to work with all languages we are using (Python, Java, R, Javascript (via WebAssembly) .. others?). And I don't know what the commercial vendors are using internally in their tools. If no one is using other implementations I imagine we could trim the list a bit but that's another conversation.

I know you know this, I'm just writing it here as a placeholder in case we do want to add these features. 😺

[e-lo]

SInce replacing the hdf5 python package is an independent and short-term feature that provides immediate upgrade in experience, let's separate that from the serialized data format discussion, which I just added as an actual issue rather than existing in the middle of lots of other discussions as issue is main OMX repo: osPlanning/omx#37

[pedrocamargo]

Of course, @e-lo and @billyc .

We will add the proposal for sparse matrices in the appropriate issue