Basically unusable for multiple tenants with multiple users

[SebestyenBartha]

We have two different tenants in two different datacenters. I have two different users to go with that and I have to switch contexts all the time. When I switched from one datacenter to the other using kubectl the user experience with the azure auth plugin was that I need to log in once for both the first time I switch and then I can seamlessly switch between the two without re-authenticating every time, because the tokens were saved individually for both clusters, so kubectl always knew which token to use for which context. That is impossible to set up with kubelogin. When I convert the config, everything is converted but the two contexts will use the same token, and only the one I was using at the time of convert will work, the other context will provide an authorization error. Kubelogin is unable to figure out that I had multiple users for the different tenants. To fix it, I need to log out, remove the tokens from kubelogin, re-authenticate, convert the config again. And I have to do these steps every single time I switch contexts. It's awful.
The azure auth plugin had sooo much better user experience... :(
Can something be done about this?

[weinong]

did you try to specify --token-cache-dir in convert-kubeconfig? i suppose you can create unique temp dir using mktemp to store the cached token

[SebestyenBartha]

I'm sorry I don't understand how using a token cache dir would help me in this? From the documentation it is not clear to me.
What I have problem with is the following:

1. I have logged in to two different tenants with: az login --tenant A then, az login --tenant B,
2. az account set --tenant A
3. az aks get-credentials --name clusterA --resource-group rgA
4. kubectl get pods -> I get the pods
5. az account set --tenant B
6. az aks get-credentials --name clusterB --resource-group rgB
7. kubectl get pods -> I get the pods
8. kubectl config use-context clusterA, then get pods -> I get the pods
9. kubectl config use-context clusterB, then get pods -> I get the pods.

Now, when I try to use kubelogin I have to do the following:
10. az account set --tenant A
11. kubelogin convert-kubeconfig -l azurecli
12. kubectl get pods -> I get the pods
13. az account set --tenant B
14. kubectl config use-context clusterB, then get pods -> error: You must be logged in to the server (Unauthorized)
15. kubelogin remove-tokens
16. kubectl get pods -> I get the pods

So essentially instead of just switching seamlessly like in step 8 and 9, I have to call az account set and kubelogin remove-tokens every time I switch contexts, which is very annoying and also much slower than previously. I don't see how the token-cache-dir helps with this...

[weinong]

please look at this proposal, this would work, at least it works for me when i manually edited the kubeconfig

1. az login --tenant A
2. az login --tenant B,
3. az account set --tenant A
4. az aks get-credentials --name clusterA --resource-group rgA
5. az account set --tenant B
6. az aks get-credentials --name clusterB --resource-group rgB
7. kubelogin convert-kubeconfig -l devicecode --context clusterA --token-cache-dir $(mktemp -d)
8. kubelogin convert-kubeconfig -l devicecode --context clusterB --token-cache-dir $(mktemp -d)

after this, you can change contexts without interfering each other.

you can confirm this proposal by editing the kubeconfig to add unique token-cache-dir, but you have to use devicecode login

[ScottS-byte]

I have to side with @SebestyenBartha on this - kubelogin is almost useless. I can connect fine to AKS clusters but not to our legacy k8s 1.17, 1.18 local clusters nor the legacy clusters on Azure. For the legacy clusters the process is kubectl use-context... or kubectx clustername and k9s. Once kubelogin is used as minimally directed nothing can connect to the legacy clusters at all, including kubectl cluster-info.

[weinong]

@ScottS-byte would you mind elaborating? does your legacy k8s use AAD or not? if you use kubelogin convert-kubeconfig --context ... --token-cache-dir $(mktemp -d), it will only change the config for selected context. the rest should remain the same, and you can switch by using kubectl config use-context XYZ