Parent groups automatically created in Zarr-Python 3

[y4n9squared]

In zarr-python==3, it appears that parent groups (all the way up to the root) are automatically created if they don't already exist.

import zarr

store = "/tmp/foo.zarr"
shape = (64, 64, 4)

za = zarr.create_array(
    store,
    shape=shape,
    dtype=float,
    zarr_format=3,
    name="foo/bar",
)

initializes the directory tree

drwxrwxr-x   - yang.yang 27 Jan 13:48  /tmp/foo.zarr
drwxrwxr-x   - yang.yang 27 Jan 13:48 ├──  foo
drwxrwxr-x   - yang.yang 27 Jan 13:48 │   ├──  bar
.rw-rw-r-- 657 yang.yang 27 Jan 13:48 │   │   └──  zarr.json
.rw-rw-r--  99 yang.yang 27 Jan 13:48 │   └──  zarr.json
.rw-rw-r--  99 yang.yang 27 Jan 13:48 └──  zarr.json

This is a departure from writing V3 format in zarr-python==2, where only the child array is written:

drwxrwxr-x   - yang.yang 27 Jan 13:45  /tmp/foo.zarr
drwxrwxr-x   - yang.yang 27 Jan 13:45 ├──  meta
drwxrwxr-x   - yang.yang 27 Jan 13:45 │   └──  root
drwxrwxr-x   - yang.yang 27 Jan 13:45 │       └──  foo
.rw-rw-r-- 575 yang.yang 27 Jan 13:45 │           └──  bar.array.json
.rw-rw-r-- 200 yang.yang 27 Jan 13:45 └──  zarr.json

Is this an intended change in behavior? This subtlety is important in the case of unsynchronized, concurrent writers. It used to be the case that if two writers were writing to different Zarr groups, there was a guarantee that they would not touch the same files. Now, if one writer wrote foo/bar/a and another wrote foo/bar/b, then

• foo/bar/zarr.json
• foo/zarr.json

would be written to by both writers. While the write is idempotent, we have a use-case where, like Icechunk, we have an abstraction layer that handles write consistency, and part of that requires us to detect whether two writes mutate the same files.

[d-v-b]

thanks for this report. the story here is kind of complicated, because the zarr specs don't actually contain a complete definition of what a zarr hierarchy is. The v2 spec says this:

If the user requests a group to be created under some logical path, then groups MUST also be created at all ancestor paths. E.g., if the user requests group creation at path "foo/bar" then groups MUST be created at path "foo" and the root of the store, if they don't already exist.

If the user requests an array to be created under some logical path, then groups MUST also be created at all ancestor paths. E.g., if the user requests array creation at path "foo/bar/baz" then groups must be created at path "foo/bar", path "foo", and the root of the store, if they don't already exist.

Similarly, the v3 spec says:

To create a group at hierarchy path P, perform
set(meta_key(P), value), where value is the
serialization of a valid group metadata document, and
ensure the existence of groups at all ancestor paths of P.

So although we are consistent with the specs, think the specs are on weak ground here, and I think we should provide APIs for users to create "isolated" arrays and groups if they need this.

• In the case of create_array, we could have a create_parents flag that defaults to True. When True, ancestor groups will be created / ensured. When False, this would not happen.
• We should still ensure that a Zarr group always creates a single valid hierarchy, so Group.create_array should not take an ensure_parents argument (it will be set to True).

You might also be interested in some of the functions i'm adding over in #2665. The goal is to make creating groups and arrays much more functional and explicit. There is a create_nodes function that just creates arrays / groups, with no checking or overhead. This might be useful for users like yourself who want to be sure that IO can be run concurrently.

[d-v-b]

If I were iterating on the text in the specs, I would phrase the same idea differently. Instead of placing requirements on how applications create arrays or groups, I would define a "connectedness" relation between two nodes located at different paths in the same hierarchy. Nodes a and b are "connected" iff you can connect a to b via nodes that are all Zarr groups. A Zarr hierarchy would then be defined as a collection of such connected nodes.

Thus by definition there's only one way to add an array to an existing group such that the array is part of the same hierarchy as that group (i.e., by creating the intermediate groups). But applications should not in general be required to always create hierarchies.

[y4n9squared]

That makes sense. We were essentially taking advantage of the fact that we could create disconnected nodes in the Zarr Python 2 implementation.

The auto creation of the nodes from the root to child is nice in some ways. You get discoverability of the entire hierarchy via API and can support things like auto complete.

I think our direction is clear -- assume the spec behavior and write the conflict resolution logic to be smarter so that it knows that two writers writing intermediate nodes doesn't constitute a consistency violation (i.e. if we can show that the hash of the zarr.json file is the same in both writes, then it is safe to accept either.