Allow using a `{rhdf5}` file handle instead of a path

[lazappi]

Would it be possible to enable this?:

h5file <- rhdf5::H5Fopen("path/to/my.hdf5")
HDF5Array::HDF5Array(h5file, "my_array")

At the moment we internally pass around the file handle and starting to use {HDF5Array} would require refactoring to pass the path instead and open/close the file every time we read/write something.

[hpages]

Hi Luke,

Short answer: Can you take a look at ?H5File in the h5mread package and see if you can use that? There are some issues with storing a naked file handle in an HDF5Array object so I'd like to avoid adding support for that. One nice thing about using an H5File object is that the object keeps track of the file path so path(<HDF5Array>) still works 😉

Long answer:

Files handles and connection ids are fragile i.e. they tend to become invalid in some contexts.

EXAMPLE 1

For example, they break in the context of parallel processing:

library(rhdf5)
path <- system.file(package="HDF5Array", "extdata",  "toy.h5")

library(BiocParallel)

## Works with the path:
bplapply(1:5, function(i) h5read(path, "M1", list(1:4, 1:5)), BPPARAM=SnowParam(workers=3))

## Doesn't work with the file handle:
h5file <- rhdf5::H5Fopen(path)
bplapply(1:5, function(i) h5read(h5file, "M1", list(1:4, 1:5)), BPPARAM=SnowParam(workers=3))
# Error: BiocParallel errors
#   3 remote errors, element index: 1, 3, 5
#   2 unevaluated and other errors
#   first remote error:
# Error: Error in h5checktypeOrOpenLoc(). H5Identifier not valid.

That's because the SnowParam backend uses serialization/deserialization to communicate with the workers, and serialization breaks files handles and connection ids.

EXAMPLE 2

Here's another reason why it would not be a good idea to store the naked file handle returned by rhdf5::H5Fopen() in an HDF5Array object.

The HDF5Array package depends on the rhdf5 and h5mread packages which are both linked to the static hdf5 lib (libhdf5.a) provided by the Rhdf5lib package. Because the linking is static, the two packages are linked to separate copies of the library so file handles and other resources originating from one copy are not valid in the other copy. Concretely this means that if we store the file handle returned by rhdf5::H5Fopen() in the HDF5Array object, any operation on the object that triggers reading data from disk won't work. That's because these operations use h5mread::h5mread() internally to read data from disk but h5mread::h5mread() cannot use a file handle originating from rhdf5.

This is the reason why I came up with the H5File hack solution a few years ago. An H5File object stores two file handles, one that originates from the static hdf5 lib linked to rhdf5, and one that originates from the static hdf5 lib linked to h5mread. Yes, you can call this a dirty hack 😉

Also note that it doesn't solve the issue with parallel processing (there's a BIG NOTE in ?H5File about that). In other words, be aware that if you pass an HDF5File to the HDF5Array() constructor, you (or your users) won't be able to use the SnowParam backend if you ever need to process the returned HDF5Array object in parallel.

Let me know if that works for you,

H.

[lazappi]

Thanks for the explanation! That explains some things.

I had a look at H5File but I couldn't work out how/if it can be used with regular {rhdf5}. Is there a way to do that? We need/want to use a combination of {rhdf5} and {HDF5Array} but obviously having two open file handles is a bad idea and there are potential issues with opening/closing all the time.

[hpages]

To use the H5File object with rhdf5, coerce it to H5IdComponent:

library(HDF5Array)

path <- system.file(package="HDF5Array", "extdata",  "toy.h5")
h5file <- H5File(path)

M1 <- HDF5Array(h5file, "M1")
M1[1:4, 1:5]
# <4 x 5> DelayedMatrix object of type "double":
#            [,1]       [,2]       [,3]       [,4]       [,5]
# [1,] 0.44015009 0.02303051 0.11688992 0.54876625 0.52518001
# [2,] 0.18883376 0.84313244 0.12153315 0.45807701 0.75956282
# [3,] 0.99232698 0.22924928 0.20998143 0.03086302 0.80789361
# [4,] 0.52631346 0.15438076 0.43283016 0.54889746 0.97223445

rhdf5::h5read(as(h5file, "H5IdComponent"), "M1", list(1:4, 1:5))
#           [,1]       [,2]      [,3]       [,4]      [,5]
# [1,] 0.4401501 0.02303051 0.1168899 0.54876625 0.5251800
# [2,] 0.1888338 0.84313244 0.1215332 0.45807701 0.7595628
# [3,] 0.9923270 0.22924928 0.2099814 0.03086302 0.8078936
# [4,] 0.5263135 0.15438076 0.4328302 0.54889746 0.9722344

I realize this is not documented, sorry. Will add it to H5File's man page.

H.

[hpages]

Done in h5mread 1.2.1 and 1.3.1: Bioconductor/h5mread@6ead8d5

[lazappi]

Thanks! I tried a few things but didn't think of that. I'll give this a go, I'm fairly confident it should work but I'll let you know if I run into issues.

[lazappi]

I gave this a go but I ran into another roadblock which I hadn't thought of. We currently open the HDF5 file in different modes depending on what the user wants to do by setting the flags argument in rhdf5::H5Fopen(). At the moment H5File only has a read-only mode. I tried modifying it to add a readonly argument but I kept getting errors if one of the IDs was opened in read/write mode.

Without a write option it's a bit hard to use H5File as the file handle. Maybe there's another way to handle this I haven't thought of though?

[hpages]

Are you saying that you need to write to other parts of the HDF5 file and leave the on-disk data of the HDF5Array object alone? In that case it's fine.

Just to clarify: An HDF5Array object is meant to point at read-only data, that is, the on-disk data should not be touched during the lifespan of the object, otherwise bad things will happen. For this reason I'm reluctant to allow passing a read/write file handle to the HDF5Array() constructor. This was actually another reason for the early design decision to take the path to the HDF5 file and store it inside the object instead of a file handle.

Another reason for not passing an open handle to the HDF5Array() is because of how fragile HDF5 file handles are. Take for example the following:

library(rhdf5)
path <- system.file(package="HDF5Array", "extdata",  "toy.h5")

## If we open in read-write mode first, then we can open other handles on the same file in any mode:
h5file1 <- rhdf5::H5Fopen(path, flags="H5F_ACC_RDWR")
h5file2 <- rhdf5::H5Fopen(path, flags="H5F_ACC_RDONLY")
h5file3 <- rhdf5::H5Fopen(path, flags="H5F_ACC_RDWR")
h5file4 <- rhdf5::H5Fopen(path, flags="H5F_ACC_RDONLY")

rhdf5::H5Fclose(h5file1)
rhdf5::H5Fclose(h5file2)
rhdf5::H5Fclose(h5file3)
rhdf5::H5Fclose(h5file4)

## However, if we open in read-only first, then the only other handles that we can open on the same
## file must also be in read-only mode!
h5file5 <- rhdf5::H5Fopen(path, flags="H5F_ACC_RDONLY")
h5file6 <- rhdf5::H5Fopen(path, flags="H5F_ACC_RDWR")  # Unable to open file!
h5file7 <- rhdf5::H5Fopen(path, flags="H5F_ACC_RDONLY")

Doesn't make any sense to me!

Anyways, what I've learned is that with HDF5 you really want to avoid keeping open file handles around because then chances are high that this will block other applications -- or even your own code -- from accessing the file. In other words, the best/safer thing to do really is to create the handle when you actually need to access the file and release it as soon as you're done.

Obtaining/releasing a handle takes only 0.0001 sec:

system.time({for (i in 1:10000) h5file <- rhdf5::H5Fopen(path, flags="H5F_ACC_RDONLY"); rhdf5::H5Fclose(h5file)})
#    user  system elapsed 
#   1.002   0.043   1.046 

system.time({for (i in 1:10000) h5file <- rhdf5::H5Fopen(path, flags="H5F_ACC_RDWR"); rhdf5::H5Fclose(h5file)})
#    user  system elapsed 
#   0.992   0.041   1.033 

Is this really an issue for your use case?

[lazappi]

I think you are right, that's probably the way to go. I can't remember exactly why we decided to store the handle rather than the path but it was probably more about simplifying the code than because of any overhead. For example, we only have to check the mode once when the object is created rather than whenever the user does something.

That said, it was kind of a feature that opening a file multiple times was blocked. Less important than having support for DelayedArray though.