"Unauthorized" Error in Dagster cloud hybrid deployment with k8s

[rrazavipour]

Hi All, I am trying to setup a hybrid cloud environment and I see this error everytime I materialize an asset manually.
The agent is running in AWS kubernetes.
Anyone has seen this error:

dagster_k8s.client.DagsterK8sAPIRetryLimitExceeded: Retry limit of 3 exceeded: Unexpected error encountered in Kubernetes API Client.
  File "/dagster-cloud/dagster_cloud/agent/dagster_cloud_agent.py", line 799, in _process_api_request
    api_result = self._handle_api_request(
  File "/dagster-cloud/dagster_cloud/agent/dagster_cloud_agent.py", line 697, in _handle_api_request
    launcher.launch_run(LaunchRunContext(dagster_run=run, workspace=None))
  File "/dagster-k8s/dagster_k8s/launcher.py", line 282, in launch_run
    self._launch_k8s_job_with_args(job_name, args, run)
  File "/dagster-k8s/dagster_k8s/launcher.py", line 263, in _launch_k8s_job_with_args
    self._api_client.create_namespaced_job_with_retries(body=job, namespace=namespace)
  File "/dagster-k8s/dagster_k8s/client.py", line 826, in create_namespaced_job_with_retries
    k8s_api_retry_creation_mutation(
  File "/dagster-k8s/dagster_k8s/client.py", line 176, in k8s_api_retry_creation_mutation
    raise DagsterK8sAPIRetryLimitExceeded(
The above exception was caused by the following exception:
kubernetes.client.exceptions.ApiException: (401)
Reason: Unauthorized
HTTP response headers: HTTPHeaderDict({'Audit-Id': '2b136821-9688-41b1-8f78-10e37134da6e', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'Date': 'Tue, 12 Dec 2023 18:09:53 GMT', 'Content-Length': '129'})
HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}
  File "/dagster-k8s/dagster_k8s/client.py", line 157, in k8s_api_retry_creation_mutation
    fn()
  File "/dagster-k8s/dagster_k8s/client.py", line 827, in <lambda>
    lambda: self.batch_api.create_namespaced_job(body=body, namespace=namespace),
  File "/usr/local/lib/python3.10/site-packages/kubernetes/client/api/batch_v1_api.py", line 210, in create_namespaced_job
    return self.create_namespaced_job_with_http_info(namespace, body, **kwargs)  # noqa: E501
  File "/usr/local/lib/python3.10/site-packages/kubernetes/client/api/batch_v1_api.py", line 309, in create_namespaced_job_with_http_info
    return self.api_client.call_api(
  File "/usr/local/lib/python3.10/site-packages/kubernetes/client/api_client.py", line 348, in call_api
    return self.__call_api(resource_path, method,
  File "/usr/local/lib/python3.10/site-packages/kubernetes/client/api_client.py", line 180, in __call_api
    response_data = self.request(
  File "/usr/local/lib/python3.10/site-packages/kubernetes/client/api_client.py", line 391, in request
    return self.rest_client.POST(url,
  File "/usr/local/lib/python3.10/site-packages/kubernetes/client/rest.py", line 279, in POST
    return self.request("POST", url,
  File "/usr/local/lib/python3.10/site-packages/kubernetes/client/rest.py", line 238, in request
    raise ApiException(http_resp=r)