Added a cronjob to delete orphaned jenkins job namespaces #1296
Conversation
| - name: CLIENT_ID | ||
| description: The Azure Client ID used for federation | ||
| required: true | ||
| - name: MGMT_ORPHANED_NAMESPACE |
There was a problem hiding this comment.
| - name: MGMT_ORPHANED_NAMESPACE | |
| - name: ORPHANED_NAMESPACE_CLEANER |
Not to confuse it with the MGMT cluster.
| - name: MGMT_ORPHANED_NAMESPACE | ||
| description: The namespace to create to have a cronjob which will delete the orphaned namespace which are not deleted due to any issues with the jenkins job. | ||
| required: true | ||
| value: orphaned-namepsace-mgmt |
There was a problem hiding this comment.
| value: orphaned-namepsace-mgmt | |
| value: orphaned-namepsace-cleaner |
There was a problem hiding this comment.
Do we need to define this namespace in the Kubernetes resources?
There was a problem hiding this comment.
No actually, I intend it to be used with the default value. Let me make the required flag to false.
| - apiVersion: v1 | ||
| kind: ServiceAccount | ||
| metadata: | ||
| name: orphaned-namepsace-mgmt-cronjob |
There was a problem hiding this comment.
Could we use the existing service account cluster-service-mgmt for cleanup? if we are binding the same role namespace-admin to both.
What is the advantage of defining a new service account?
There was a problem hiding this comment.
Since they are in different namespaces I had to create a new SA .. if we would be using the same ns where the sa cluster-service-mgmt, then we would not have created another one.
| - name: KUBECTL-IMAGE | ||
| description: An image which have the `kubectl` binary in it. | ||
| required: true | ||
| value: quay.io/rhn_support_ansverma/ubi8-minimal-kubectl:latest |
There was a problem hiding this comment.
Is this image always going to be public and available to the cronjob? If the cronjob runs on the AKS cluster, can it access Quay? IIRC there is some mechanism to pull images on demand to ACR, but maybe that's only needed for private repos.
@geoberle do you know what are the alternatives for a public image with kubectl?
There was a problem hiding this comment.
This image is public, we can have this image in ACR also, however, AKS will be able to pull it. I have tested this with my mgmt AKS cluster. It worked fine.
| spec: | ||
| template: | ||
| spec: | ||
| serviceAccountName: orphaned-namepsace-mgmt-cronjob |
There was a problem hiding this comment.
| serviceAccountName: orphaned-namepsace-mgmt-cronjob | |
| serviceAccountName: orphaned-namespace-mgmt-cronjob |
There was a problem hiding this comment.
Changed it to orphaned-namespace-cleaner-cronjob
| kubectl config set-cluster $CLUSTER_NAME \ | ||
| --server=$KUBE_API_SERVER_HOST \ | ||
| --embed-certs \ | ||
| --certificate-authority=/var/run/secrets/kubernetes.io/serviceaccount/ca.crt \ |
There was a problem hiding this comment.
| --certificate-authority=/var/run/secrets/kubernetes.io/serviceaccount/ca.crt \ | |
| --certificate-authority=/var/run/secrets/kubernetes.io/serviceaccount/ca.crt |
Is the \ at the end needed?
There was a problem hiding this comment.
nope, will remove it.
| --embed-certs \ | ||
| --certificate-authority=/var/run/secrets/kubernetes.io/serviceaccount/ca.crt \ | ||
|
|
||
| echo "setting kubectl scredential" |
There was a problem hiding this comment.
| echo "setting kubectl scredential" | |
| echo "setting kubectl credential" |
| kubectl config use-context current-context | ||
|
|
||
| echo "deleting the orphaned namespaces" | ||
| kubectl get namespaces -o json | jq -r '.items[] | select(.metadata.labels."sandbox-jenkins-type"=="aro-hcp") | select((now - (.metadata.creationTimestamp | fromdate)) / 60 > 60) | .metadata.name' | xargs kubectl delete namespace |
There was a problem hiding this comment.
Do we need to handle a case where no spaces are matched?
There was a problem hiding this comment.
Sorry, what do you mean by no spaces ?
There was a problem hiding this comment.
Sorry, a typo, a scenario where no namespaces match the filtering.
| description: The namespace to create to have a cronjob which will delete the orphaned namespace which are not deleted due to any issues with the jenkins job. | ||
| required: true | ||
| value: orphaned-namepsace-mgmt | ||
| - name: KUBECTL-IMAGE |
There was a problem hiding this comment.
nit: lowercase for consistency with the other env vars
| kubectl config set-cluster $CLUSTER_NAME \ | ||
| --server=$KUBE_API_SERVER_HOST \ | ||
| --embed-certs \ | ||
| --certificate-authority=/var/run/secrets/kubernetes.io/serviceaccount/ca.crt \ | ||
|
|
||
| echo "setting kubectl scredential" | ||
| kubectl config set-credentials $SERVICE_ACCOUNT_NAME \ | ||
| --token=$(cat /var/run/secrets/kubernetes.io/serviceaccount/token) | ||
| kubectl config set-context current-context --cluster=$CLUSTER_NAME --user=$SERVICE_ACCOUNT_NAME | ||
| kubectl config use-context current-context |
There was a problem hiding this comment.
since the pod runs within the cluster, there is no need to set up a kubeconfig. just make sure that the job runs as the service account specified by $SERVICE_ACCOUNT_NAME
There was a problem hiding this comment.
Ohh yeah, you are right. I did the changes, testing them, once done, I will push the changes.
anshulvermapatel
force-pushed
the
orphaned-ns-mgmt
branch
from
15f63ac to
3453745
Compare
What this PR does
This PR added few more resource including a cronjob to the integration environment in the svc cluster. The cronjob is run once a day to clear the orphaned namespaces which are created by the pr_check jenkins job. Sometime, the namespaces does not get deleted due to some jenkins issue like a crashed jenkins job before executing the namespace delete logic. The cronjob targets such namespaces and delete them.
The logic is delete the namespaces which has the label
sandbox-jenkins-type: aro-hcpand is >60 mins old.Jira: ARO-10434