feat(Chart)!: 📦 add separated chart for CRDs

[darkweaver87]

What does this PR do?

This PR introduces a separate chart used by the main one to install CRDs.
The dependency between the main chart and the CRDs one will be added once CRDs one will be released.

Motivation

#1141
#1209

More

• Yes, I updated the tests accordingly
• Yes, I updated the schema accordingly
• Yes, I ran make test and all the tests passed

[mloiseleur]

1. The CI needs to be tested, with a tag on your fork.
2. The documented CRDs upgrade process should still work. I see (at least) two options:
   a. Keep and copy crd files
   b. Update CRD upgrade instruction to use them in the new directory

[darkweaver87]

Tested on my own account as seen with you 👍

[darkweaver87]

I'm not strongly against option 1, but I think we will clearly lost users on the way. The approach of having a dependency chart seduced me at first because of managing all this compatibility complexity being a responsibility delegated to us.
With option 1, we will somehow, have to complete a compatibility matrix for users to know which version of CRD chart comes with which chart version or traefik/traefik-hub version.
Even for us, managing a crds will be a nightmare as we will have to maintain a CRD chat and a crds directory. Or, we need to test if symlinks work on charts.

The other option is keeping it that way:

• people who don't want to bother with it and let us manage their CRDs for them will just need to set:

traefik-crds:
  traefik: true

• people who want to manage two or more instances of traefik will just need to install the chart as previously and either manage CRDs by themselves or install the CRD chart (they will be able to find which version of the CRD chart comes with which version of traefik or traefik-hub following the dependencies section of the main chart)
• people who want a specific version of traefik will fall in the previous case

However, even with the option 1, the main points remains how users can migrate from the previous system to the new one ? Adding the annotation works but the command I provided is mainly linux friendly :-)

[genebean]

It might be worth checking out how https://github.com/prometheus-community/helm-charts/tree/main/charts/kube-prometheus-stack is setup. It bundles https://github.com/prometheus-community/helm-charts/tree/main/charts/prometheus-operator-crds in but has an option to not install them for when you want to manage them separately. I’ve been using that lately and it works well and isn’t too difficult.