Overhaul patching and webhook configuration (#25)

Updated chart version to 1.0.0 - Backwards incompatible, major revision update Overhaul patching and webhook configuration Overhaul to remove `haystack` specific references Directly read configmaps instead of hardcoded sidecar names. Remove old deployment methods, only support helm. Remove old mutation code Remove support for annotations Update DEVELOP.md and README.md
ExpediaGroup · Mar 25, 2022 · 27217c8 · 27217c8
1 parent 8264661
commit 27217c8
Show file tree

Hide file tree

Showing 38 changed files with 1,079 additions and 1,134 deletions.
diff --git a/.run/go build kubernetes-sidecar-injector.run.xml b/.run/go build kubernetes-sidecar-injector.run.xml
@@ -2,7 +2,7 @@
   <configuration default="false" name="go build kubernetes-sidecar-injector" type="GoApplicationRunConfiguration" factoryName="Go Application">
     <module name="kubernetes-sidecar-injector" />
     <working_directory value="$PROJECT_DIR$" />
-    <parameters value="--port=8080 --local=true --sideCar=./sample/sidecar.yaml" />
+    <parameters value="--port=8080 --local=true" />
     <kind value="PACKAGE" />
     <package value="github.com/expediagroup/kubernetes-sidecar-injector" />
     <directory value="$PROJECT_DIR$" />

diff --git a/DEVELOP.md b/DEVELOP.md
@@ -13,109 +13,49 @@ Code contributions are always welcome.
 
 
 ## Dependencies
-```bash
-go get -u golang.org/x/lint/golint
-```
 
 * Ensure [GOROOT, GOPATH and GOBIN](https://www.programming-books.io/essential/go/d6da4b8481f94757bae43be1fdfa9e73-gopath-goroot-gobin) environment variables are set correctly.
 
-## Build and run locally
-
-* Build
-
-```bash
-make build
-```
-
-* Run
-
-```bash
-./kubernetes-sidecar-injector -port=8443 -certFile=sample/certs/cert.pem  -keyFile=sample/certs/key.pem -sideCar=sample/sidecar.yaml -logtostderr
-```
-
-* Send a sample request
-
-```bash
-curl -kvX POST --header "Content-Type: application/json" -d @sample/admission-request.json https://localhost:8443/mutate
-```
-
-## Build and run with docker
-
-* Build
-
-```bash
-make docker
-```
-
-* Run
-
-```bash
-docker run -d --name injector -p 8443:443 --mount type=bind,src=${PWD}/sample,dst=/etc/mutator expediagroup/kubernetes-sidecar-injector:latest -logtostderr
-```
-
-* Send a sample request
-
-```bash
-curl -kvX POST --header "Content-Type: application/json" -d @sample/admission-request.json https://localhost:8443/mutate
-```
+## Build and run using an IDE (JetBrains)
+Run the included [`go build kubernetes-sidecar-injector`](.run/go build kubernetes-sidecar-injector.run.xml) `Go Build` job.
 
 ## Build and deploy in Kubernetes
 
 ### Deploy using Kubectl
 
-To deploy and test this in [minikube](https://kubernetes.io/docs/tasks/tools/install-minikube/)
-* build docker container locally with the fixes
-
-```
-eval $(minikube docker-env)
-make release
-```
+To deploy and test this in [kind](https://kind.sigs.k8s.io/docs/user/quick-start/#installation/)
 
 * run the command
 
 ```bash
-./deployment/kubectl/deploy.sh
+make kind-install
 ```
 (To understand the command above - check the [README](README.md) file)
 
 After deployment, one can check the service running by
 
 ```bash
-kubectl get pod
-
-NAME                                                        READY     STATUS    RESTARTS   AGE
-kubernetes-sidecar-injector-deployment-5b5874466-k4gnk   1/1       Running   0          1m
-
-```
-
-### Label the namespace
+kubectl get pods -n kubernetes-sidecar-injector
 
-Before deploying a pod to see the side car being injected, one needs to do one additional step.  
-
-[Registration spec of this mutating webhook](deployment/mutatingwebhook-template.yaml#L22) specifies that this webhook be called only for pods deployed in namespaces with a label `kubernetes-sidecar-injector: enabled`
-
-Following spec applies this label to `default` namespace
-
-```bash
-kubectl apply -f sample/namespace-label.yaml
+NAME                                           READY   STATUS    RESTARTS   AGE
+kubernetes-sidecar-injector-78648d458b-7cv7l   1/1     Running   0          32m
 ```
 
 ### Test the webhook
 
-One can run the following command to deploy a sample `echo-server`. Note, this [deployment spec carries an annotation](sample/echo-server.yaml#L12) `sidecar-injector.expedia.com/inject: "haystack-agent"` that triggers injection of `haystack-agent` sidecar defined in [sidecar-configmap.yaml](deployment/kubectl/sidecar-configmap.yaml) file.
+Run the following command to deploy a sample `echo-server`. Note, this [deployment spec carries an annotation](sample/chart/echo-server/templates/echo-server.yaml#L16) `sidecar-injector.expedia.com/inject: "haystack-agent"` that triggers injection of `haystack-agent` sidecar defined in [sidecar-configmap.yaml](sample/chart/echo-server/templates/sidecar-configmap.yaml) file.
 
 ```bash
-kubectl apply -f sample/echo-server.yaml
+make install-sample
 ```
 
 One can then run the following command to confirm the sidecar has been injected
 
 ```bash
-kubectl get pod
+kubectl get pod -n sample
 
 NAME                                                        READY     STATUS             RESTARTS   AGE
 echo-server-deployment-849b87649d-9x95k                     2/2       Running            0          4m
-kubernetes-sidecar-injector-deployment-cc4648b7f-bdk2v   1/1       Running            0          6m
 ```
 
 Note the **2 containers** in the echo-server pod instead of one. 
@@ -125,12 +65,8 @@ Note the **2 containers** in the echo-server pod instead of one.
 Run the following commands to delete and cleanup the deployed webhook
 
 ```
-kubectl delete mutatingwebhookconfiguration kubernetes-sidecar-injector-webhook
-kubectl delete service kubernetes-sidecar-injector-svc
-kubectl delete deployment kubernetes-sidecar-injector-deployment
-kubectl delete configmap haystack-agent-conf-configmap
-kubectl delete configmap kubernetes-sidecars-configmap
-kubectl delete secret kubernetes-sidecar-injector-certs
+helm delete -n kubernetes-sidecar-injector kubernetes-sidecar-injector
+helm delete -n sample sample-echo-server-sidecar-injector
 ```
 
 

diff --git a/Dockerfile b/Dockerfile
@@ -1,5 +1,5 @@
-FROM golang:1.16 as build
-RUN go get -u golang.org/x/lint/golint
+FROM golang:1.18 as build
+RUN go install golang.org/x/lint/golint@latest
 WORKDIR /build
 COPY . ./
 RUN make release

diff --git a/Makefile b/Makefile
@@ -26,6 +26,13 @@ release: clean vet lint
 docker:
 	docker build --no-cache -t ${CONTAINER_NAME}:${IMAGE_TAG} .
 
-kind-install: docker
+kind-load: docker
 	kind load docker-image ${CONTAINER_NAME}:${IMAGE_TAG} --name ${KIND_CLUSTER}
-	helm upgrade -i kubernetes-sidecar-injector ./charts/kubernetes-sidecar-injector/. --namespace=kubernetes-sidecar-injector --create-namespace --set image.tag=${IMAGE_TAG}
+
+helm-install:
+	helm upgrade -i kubernetes-sidecar-injector ./charts/kubernetes-sidecar-injector/. --namespace=kubernetes-sidecar-injector --create-namespace --set image.tag=${IMAGE_TAG}
+
+kind-install: kind-load helm-install
+
+install-sample:
+	helm upgrade -i sample-echo-server-sidecar-injector ./sample/chart/echo-server/. --namespace=sample --create-namespace
diff --git a/README.md b/README.md
@@ -4,9 +4,7 @@
 Kubernetes Mutating Webhook
 ===========
 
-This [mutating webhook](https://kubernetes.io/docs/reference/access-authn-authz/admission-controllers/#mutatingadmissionwebhook) was developed to inject [Haystack](http://expediadotcom.github.io/haystack/)'s agent as a sidecar to a Kubernetes pod so applications can ship trace data to Haystack server. 
-
-Though this was primarily written to inject [haystack-agent](https://github.com/ExpediaDotCom/haystack-agent) as a sidecar, __one can use this to inject any container as a sidecar in a pod__.
+This [mutating webhook](https://kubernetes.io/docs/reference/access-authn-authz/admission-controllers/#mutatingadmissionwebhook) was developed to inject sidecars to a Kubernetes pod. 
 
 ## Developing
 
@@ -18,67 +16,13 @@ We have provided two ways to deploy this webhook. Using [Helm](https://helm.sh/)
 
 ### How to enable sidecar injection using this webhook
 
-1. One can simply deploy this mutating webhook by cloning this repository and running the following command (needs kubectl installed and configured to point to the kubernets cluster or minikube)
-
-    ```bash
-    ./deployment/kubectl/deploy.sh
-    ```
-
-    or using helm
-
-    ```bash
-    helm init
-    helm install --name kubernetes-sidecar-injector-webhook ./deployment/helm
-    ```
-
-2. The command above installs the webhook and a map of named sidecars to be injected. One can find the map in [this config map file in kubectl folder](deployment/kubectl/sidecar-configmap.yaml) or [this configmap in helm folder](charts/kubernetes-sidecar-injector/templates/sidecar-configmap.yaml). In these files only one sidecar named `haystack-agent`has been configured.
-
-3. Apply the label `kubernetes-sidecar-injector: enabled` in the namespaces where the sidecar injection should be considered. [This sample](sample/namespace-label.yaml) file applies the label mentioned to _default_ namespace
-
-4. Add an annotation `sidecar-injector.expedia.com/inject`  with name of the sidecar to inject in pod spec where sidecar needs to be injected. [This sample spec](sample/echo-server.yaml#L12) shows such an annotation added to a pod spec to inject `haystack-agent`. 
-
-### Kubectl deployment files
-
-Lets go over the files in the __deployment/kubectl__ folder.
-
-1. __sidecar-configmap.yaml__:  This file contains two _configmap_ entries.  First one, _kubernetes-sidecars-configmap_ contains a map of named sidecar containers to be injected. In this case, we have only one named sidecar called `hatrack-agent`. Second one _haystack-agent-conf-configmap_ contains a configuration file that is used by haystack-agent sidecar. 
-
-    _Though this file carries only haystack-agent, one can_  __replace this or add more sidecars with to be injected__. 
-
-2. __sidecar-injector-deployment.yaml__: This file deploys _kubernetes-sidecar-injector_ pod and _kubernetes-sidecar-injector-svc_ service. This is the mutating webhook admission controller service. This is invoked by kebernetes while creating a new pod with the pod spec that is being created. That allows this webhook to inspect and make a decision on whether to inject the sidecar or not. This webhook checks for two conditions to determine whether to inject a sidecar or not
-    1. __Namespace check__:  Sidecar injection will be attempted _only_ if the the pod is being created in a namespace with the label `kubernetes-sidecar-injector: enabled` __and__  the namespace is NOT `kube-system` or `kube-public`
-
-    2. __Annotation check__: Sidecar inkection will be attempted _only_ if the pod being created carries an annotation `sidecar-injector.expedia.com/inject`.  Value of this annotation will be used to locate the sidecar to be injected from the configmap in _sidecar-configmap.yaml_.
-
-       __Note__: One can have a __comma separated list of sidecar names__ if more than one sidecar needs to be injected
-
-3. __create-server-cert.sh__: Mutating webhook admission controllers need to listen on `https (TLS)`. This script generates a key, a certificate request and gets that request signed by Kubernetes CA. i.e., produces a signed certificate and deploys it as a kubernets secret to be used by the service defined in #2
-
-4. __mutatingwebhook-template.yaml__: This file registers the mutating webhook admission controller. This spec carries the CA file that will validate the server certificate used by the service. This file is a template and the `caBundle` field in it is populated by the script `replace-ca-token.sh` file
-
-5. __deploy.sh__: This is a simple bash script that deploys the webhook by executing the scripts / deployment specifications mentioned above.  
-
-### Helm deployment files
-
-Files in __deployment/helm/templates__ are the same as the files in kubectl folder and provide the same functionality.
-
-### Addendum
-
-#### Injecting env variables in the sidecar
-
-At times one may have to pass additional information to the sidecar from the pod spec. For example, a pod specific `api-key` to be used by a sidecar. To allow that, this webhook looks for special annotations with prefix `sidecar-injector.expedia.com` in the pod spec and adds the annotation key-value as environment variables to the sidecar. 
-
-For example, this [sample pod specification](sample/echo-server.yaml#L13) has the following annotation 
-
-  ```yaml
-  sidecar-injector.expedia.com/some-api-key: "6feab492-fc9b-4c38-b50d-3791718c8203"
-  ```
+1. Deploy this mutating webhook by cloning this repository and running the following command (needs kubectl installed and configured to point to the kubernetes cluster or minikube)
 
-and this will cause this webhook to inject
+```bash
+make helm-install
+```
 
-  ```yaml
-  some-api-key: "6feab492-fc9b-4c38-b50d-3791718c8203"
-  ```
+2. By default, all namespaces are watched except `kube-system` and `kube-public`. This can be configured in your [helm values](charts/kubernetes-sidecar-injector/values.yaml#L13-L19).
 
-as an environment variable in all the sidecars injected.
+3. Add the annotation ([`sidecar-injector.expedia.com/inject`](charts/kubernetes-sidecar-injector/values.yaml#L9-L10) by default) with ConfigMap sidecar name to inject in pod spec where sidecar needs to be injected. [This sample spec](sample/chart/echo-server/templates/echo-server.yaml#L16) shows such an annotation added to a pod spec to inject `haystack-agent`.
 
diff --git a/charts/kubernetes-sidecar-injector/Chart.yaml b/charts/kubernetes-sidecar-injector/Chart.yaml
@@ -1,14 +1,11 @@
 name: kubernetes-sidecar-injector
 home: https://github.com/expediagroup/kubernetes-sidecar-injector
-version: 0.0.1
-appVersion: 0.1
+version: 1.0.0
+appVersion: "0.1"
 keywords:
   - kubernetes
   - sidecar-injection
 description:
 apiVersion:
 sources:
 - https://github.com/expediagroup/kubernetes-sidecar-injector
-maintainers:
-- name: Ayan Sen
-  email: [email protected]
diff --git a/charts/kubernetes-sidecar-injector/templates/_vars.tpl b/charts/kubernetes-sidecar-injector/templates/_vars.tpl
@@ -10,4 +10,16 @@ app.kubernetes.io/instance: {{ .Release.Name }}
 
 {{- define "service.name" }}
 {{- .Release.Name }}
+{{- end }}
+
+{{- define "serviceaccount.name" }}
+{{- .Release.Name }}
+{{- end }}
+
+{{- define "clusterrole.name" }}
+{{- .Release.Name }}
+{{- end }}
+
+{{- define "clusterrolebinding.name" }}
+{{- .Release.Name }}
 {{- end }}
diff --git a/charts/kubernetes-sidecar-injector/templates/mutatingwebhook-template.yaml b/charts/kubernetes-sidecar-injector/templates/mutatingwebhook-template.yaml
@@ -42,9 +42,34 @@ webhooks:
           - "*"
         operations:
           - CREATE
+        scope: Namespaced
     namespaceSelector:
+      matchExpressions:
+        {{- with .Values.selectors.namespaceSelector.matchExpressions }}
+        {{- toYaml . | nindent 8 }}
+        {{- end }}
+        - key: {{ .Values.selectors.injectPrefix }}/{{ .Values.selectors.disableInjectLabel }}
+          operator: NotIn
+          values:
+            - "true"
+        - key: name
+          operator: NotIn
+          values:
+            - {{ .Release.Namespace }}
+    objectSelector:
+      {{- with .Values.selectors.objectSelector.matchLabels }}
       matchLabels:
-        kubernetes-sidecar-injector: enabled
+      {{- toYaml . | nindent 8 }}
+      {{- end }}
+      matchExpressions:
+        - key: {{ .Values.selectors.injectPrefix }}/{{ .Values.selectors.injectName }}
+          operator: NotIn
+          values:
+            - skip
+        - key: {{ .Values.selectors.injectPrefix }}/{{ .Values.selectors.disableInjectLabel }}
+          operator: NotIn
+          values:
+            - "true"
 ---
 apiVersion: apps/v1
 kind: Deployment
@@ -65,6 +90,7 @@ spec:
       annotations:
         generated-cert: {{ sha256sum $cert.Cert }}
     spec:
+      serviceAccountName: {{ include "serviceaccount.name" . }}
       containers:
         - name: kubernetes-sidecar-injector
           image: "{{ .Values.image.repository }}:{{ .Values.image.tag }}"
@@ -73,17 +99,14 @@ spec:
             - --port=8443
             - --certFile=/opt/kubernetes-sidecar-injector/certs/cert.pem
             - --keyFile=/opt/kubernetes-sidecar-injector/certs/key.pem
-            - --sideCar=/opt/kubernetes-sidecar-injector/sidecarconfig.yaml
+            - --injectPrefix={{ trimSuffix "/" .Values.selectors.injectPrefix }}
+            - --injectName={{ .Values.selectors.injectName }}
+            - --sidecarDataKey={{ .Values.sidecars.dataKey }}
           volumeMounts:
             - name: {{ .Release.Name }}-certs
               mountPath: /opt/kubernetes-sidecar-injector/certs
               readOnly: true
-            - name: {{ .Release.Name }}-configmap
-              mountPath: /opt/kubernetes-sidecar-injector
       volumes:
         - name: {{ .Release.Name }}-certs
           secret:
             secretName: {{ include "certs.secret.name" . }}
-        - name: {{ .Release.Name }}-configmap
-          configMap:
-            name: kubernetes-sidecars-configmap
diff --git a/charts/kubernetes-sidecar-injector/templates/rbac.yaml b/charts/kubernetes-sidecar-injector/templates/rbac.yaml
@@ -0,0 +1,38 @@
+---
+apiVersion: v1
+kind: ServiceAccount
+metadata:
+  name: {{ include "serviceaccount.name" . }}
+  namespace: {{ .Release.Namespace }}
+  labels:
+    {{- include "common.labels" . | indent 4 }}
+---
+apiVersion: rbac.authorization.k8s.io/v1
+kind: ClusterRole
+metadata:
+  name: {{ include "clusterrole.name" . }}
+  labels:
+    {{- include "common.labels" . | indent 4 }}
+rules:
+  - apiGroups:
+      - ""
+    resources:
+      - configmaps
+    verbs:
+      - get
+      - list
+---
+apiVersion: rbac.authorization.k8s.io/v1
+kind: ClusterRoleBinding
+metadata:
+  name: {{ include "clusterrolebinding.name" . }}
+  labels:
+    {{- include "common.labels" . | indent 4 }}
+roleRef:
+  apiGroup: rbac.authorization.k8s.io
+  kind: ClusterRole
+  name: {{ include "clusterrole.name" . }}
+subjects:
+  - kind: ServiceAccount
+    name: {{ include "serviceaccount.name" . }}
+    namespace: {{ .Release.Namespace }}