Testing open_virtual_mfdataset on GES DISC Cloud OPeNDAP collections

[battistowx]

I've started to test out the functionality of open_virtual_mfdataset on a few NASA GES DISC collections, to prepare for upcoming tutorials. I wanted to record my testing experience, along with a few thoughts and questions in this thread. For my testing, I am using our cloud OPeNDAP-enabled MERRA-2 collection, which are all NetCDF files, and have sizes of roughly 300-400MB per file. As of today, our cloud OPeNDAP-enabled collections consist only of NLDAS and MERRA-2, which are all in NetCDF.

Errors:

1. KeyError: 0 triggered when running access="direct" outside of us-west-2.
2. TypeError: Union[arg, ...]: each arg must be a type. Got {} triggered when h5py is not installed.
3. NotImplementedError: The ManifestArray class cannot concatenate arrays which were stored using different codecs, But found codecs Codec(compressor=None, filters=[{'id': 'shuffle', 'elementsize': 4}, {'id': 'zlib', 'level': 2}]) vs Codec(compressor=None, filters=[{'id': 'shuffle', 'elementsize': 4}, {'id': 'zlib', 'level': 6}]) .See https://github.com/zarr-developers/zarr-specs/issues/288 raised when opening certain granules which contain different zlib compression levels for some reason.

More about the codec error:

This one is particularly interesting. When accessing the first 10 granules of M2T1NXSLV, which are from 1980-01-01 to 1980-01-10, no errors appear and the ManifestArrays are all created. However, once I hit that 11th granule, I get the codec error saying that the compression level is different. Indeed, when opening that 11th granule and analyzing its attributes, I see that the level is 6, instead of 2. When picking other random date ranges, such as in 1990, no errors and the compression levels are all the same. I will report this back to our internal staff, and will also try other MERRA-2 collections to see if this is a trend.

As I test, I'll continue to add posts to this thread.
Great work on this new functionality, everyone!

[ayushnag]

Hey @battistowx thanks for reporting the bugs and opening this discussion! Here is my understanding on the codecs error. Older dmrpp versions do not contain zlib (aka "deflate") compression level (relevant issue here) so the dmrpp parser has to choose the recommended default zlib compression level 6. Newer dmrpp versions contain the deflateLevel attribute which means the field should match the source netcdf. So most likely, 2 is the correct deflate level and updating those dmrpp files will fix this error.

[battistowx]

I will look into it more with our staff, thank you!

[battistowx]

Testing M2TMNXSLV (Monthly)

For this one, I have success when opening 1980-2010. When I approach 1980-2020, however, I get this same but slightly different error:

The ManifestArray class cannot concatenate arrays which were stored using different codecs, But found codecs Codec(compressor=None, filters=[{'id': 'shuffle', 'elementsize': 4}, {'id': 'zlib', 'level': 2}]) vs Codec(compressor=None, filters=None) .See https://github.com/zarr-developers/zarr-specs/issues/288

When looking at these three dmrpp files throughout the extent, it looks like deflateLevel is also missing in the later granules. Not sure why this one didn't get assigned a default value of 6, though:

• MERRA2_400.tavgM_2d_slv_Nx.202402.nc4.dmrpp (missing deflateLevel, also roughly half the size of the other dmrpp files)
• MERRA2_300.tavgM_2d_slv_Nx.201001.nc4.dmrpp (working)
• MERRA2_100.tavgM_2d_slv_Nx.198001.nc4.dmrpp (working)

[ayushnag]

Great! I will take a look into the dmrpp files that return no compression.