Support for multiple storage directories

[lfrancke]

Currently, only a single storage directory is supported. We need to support an arbitrary number.
While we're also want to support heterogenous storage tagging.
CRD design

  dataNodes:
    roleGroups:
      default:
        replicas: 1
        config:
          resources:
            storage:
              data: # We need to overwrite the data pvcs coming from the default value
                count: 0
              my-disks:
                capacity: 100Gi
                count: 5
                hdfsStorageType: Disk
              my-ssds:
                capacity: 10Gi
                storageClass: premium
                count: 3
                hdfsStorageType: Ssd

By defaulting to

              data
                capacity: 5Gi
                count: 1
                hdfsStorageType: Disk

this is a non-breaking change for the CRD. Because we name the pvcs data, data-1, data-2 etc. even the pvc names do not change, so the whole feature should be non-breaking

• Multiple pvcs supported
• hdfsStorageType is passed to hdfs
• CRD change is non-breaking
• pvc name change is non-breaking
• Tests added
• Documentation added

[sbernauer]

@soenkeliebau @lfrancke I assume we only need this for the datanodes?

[soenkeliebau]

I'd say so, yes.
iirc dfs.datanode.data.dir is the setting for this, so would only target datanodes..

[lfrancke]

There is an equivalent setting for the namenode. We might want to do that at the same time while we're on it.

[sbernauer]

Do the pvcs need to be identical? Currently i was only thinking about a numberOfDataPvcs: u16 counter and that's it.
But reading a bit i stumbled across

Namenode : 2 directories are enough for backing up namenode metadata in case of any crash of namenode. Usually 1st disk should be local disk and prefer 2nd disk as network storage[san/nas](Just incase local machine goes down you can have backup of namenode metadata on network storage). If you do not have network storage then 2 local disk are fine.

Which let me to the question if we want to support different storageclasses (basically a complete PvcConfig) for every single pvc instance? Currently the user can specify how the data pvc should look like and that get's replicated numberOfDataPvcs times
Use-case would be to use two different storage providers for redundany. But i think we can also solve (maybe even better) this use-case with multiple namenode rolegroups that use different storageclasses. Also i think that hdfs will be pretty f*cked when one of the two storage providers crashes.

TLDR: Going with the one pvs definition replicated numberOfDataPvcs times approach

[sbernauer]

Reading about dfs.namenode.name.dir, it seems to me that it is meant for redundancy and not to improve performance

Determines where on the local filesystem the DFS name node should store the name table(fsimage). If this is a comma-delimited list of directories then the name table is replicated in all of the directories, for redundancy.

Having this in mind is it really a good idea to increase the availability of the namenode this way? With the current approach all the pvcs will be provided from a single storage provider anyway - thus not really increasing the availability. I would think that the correct approach would be to increase the number of datanodes - which resource separation, health-checks and all the other good things. But I'm far from a hdfs expert.

[soenkeliebau]

We might need to look into storage types and maybe storage policies to answer this fully I'm afraid.

Haven't looked at this in ages tbh, but iirc we'd need to be able to specify per volume the storage type that it has.

Then again, that could be accomodated by just having two blocks with

storage:
  - storageClass: default
    numberOfPvs: 5
    storageType: disk
  - storageClass: premium
    numberOfPvs: 2
    storageType: ssd

and maybe make the storageType optional and default it to disk so we'd get

storage:
  - storageClass: default
    numberOfPvs: 5
  - storageClass: premium
    numberOfPvs: 2
    storageType: ssd

[sbernauer]

Good to know.
Maybe we should implement multtiple pvcs in a Version 1 and maybe later on add the storage policies if needed. But we should consider the storage policies in the CRD already, so we can add them non-breaking later on

[soenkeliebau]

Agreed

[sbernauer]

And i assume that a single node should support multiple storage types at once? (If not, things should be okay the way they are now)

[sbernauer]

Normally I would suggest the following:

  dataNodes:
    roleGroups:
      default:
        replicas: 1
        config:
          resources:
            storage:
              - capacity: 100Gi
                hdfsStorageType: disk # field ignored for now
                numberOfPvcs: 5
              - capacity: 10Gi
                hdfsStorageType: ssd # # field ignored for now
                storageClass: premium
                numberOfPvcs: 2

Problem here is, that we can't use Vec with the Fragement/Valdiation/Config merging machinery AFAIK.

What works is

           resources:
            storage:
              disk: # pvcs will still be named data to not force users to rename pvcs.
                numberOfPvcs: 5
                pvc:
                  capacity: 100Gi
              ssd: # supported later on
                numberOfPvcs: 3
                pvc:
                  capacity: 10Gi
                  storageClass: premium

What do you think about the latter solution?