diff --git a/charts/airflow/README.md b/charts/airflow/README.md index 6c311af9..fa029dc5 100644 --- a/charts/airflow/README.md +++ b/charts/airflow/README.md @@ -213,6 +213,7 @@ Parameter | Description | Default `airflow.pools` | a list airflow pools to create | `` `airflow.poolsUpdate` | if we create a Deployment to perpetually sync `airflow.pools` | `true` `airflow.defaultNodeSelector` | default nodeSelector for airflow Pods (is overridden by pod-specific values) | `{}` +`airflow.defaultTopologySpreadConstraints` | default topologySpreadConstraints for airflow Pods (is overridden by pod-specific values) | `[]` `airflow.defaultAffinity` | default affinity configs for airflow Pods (is overridden by pod-specific values) | `{}` `airflow.defaultTolerations` | default toleration configs for airflow Pods (is overridden by pod-specific values) | `[]` `airflow.defaultSecurityContext` | default securityContext configs for Pods (is overridden by pod-specific values) | `{fsGroup: 0}` @@ -240,6 +241,7 @@ Parameter | Description | Default `scheduler.replicas` | the number of scheduler Pods to run | `1` `scheduler.resources` | resource requests/limits for the scheduler Pods | `{}` `scheduler.nodeSelector` | the nodeSelector configs for the scheduler Pods | `{}` +`scheduler.topologySpreadConstraints` | the topologySpreadConstraints configs for the scheduler Pods | `[]` `scheduler.affinity` | the affinity configs for the scheduler Pods | `{}` `scheduler.tolerations` | the toleration configs for the scheduler Pods | `[]` `scheduler.securityContext` | the security context for the scheduler Pods | `{}` @@ -268,6 +270,7 @@ Parameter | Description | Default `web.replicas` | the number of web Pods to run | `1` `web.resources` | resource requests/limits for the airflow web pods | `{}` `web.nodeSelector` | the number of web Pods to run | `{}` +`web.topologySpreadConstraints` | the topologySpreadConstraints configs for the web Pods | `[]` `web.affinity` | the affinity configs for the web Pods | `{}` `web.tolerations` | the toleration configs for the web Pods | `[]` `web.securityContext` | the security context for the web Pods | `{}` @@ -295,6 +298,7 @@ Parameter | Description | Default `workers.replicas` | the number of workers Pods to run | `1` `workers.resources` | resource requests/limits for the airflow worker Pods | `{}` `workers.nodeSelector` | the nodeSelector configs for the worker Pods | `{}` +`workers.topologySpreadConstraints` | the topologySpreadConstraints configs for the worker Pods | `[]` `workers.affinity` | the affinity configs for the worker Pods | `{}` `workers.tolerations` | the toleration configs for the worker Pods | `[]` `workers.securityContext` | the security context for the worker Pods | `{}` @@ -323,6 +327,7 @@ Parameter | Description | Default `triggerer.replicas` | the number of triggerer Pods to run | `1` `triggerer.resources` | resource requests/limits for the airflow triggerer Pods | `{}` `triggerer.nodeSelector` | the nodeSelector configs for the triggerer Pods | `{}` +`triggerer.topologySpreadConstraints` | the topologySpreadConstraints configs for the triggerer Pods | `[]` `triggerer.affinity` | the affinity configs for the triggerer Pods | `{}` `triggerer.tolerations` | the toleration configs for the triggerer Pods | `[]` `triggerer.securityContext` | the security context for the triggerer Pods | `{}` @@ -348,6 +353,7 @@ Parameter | Description | Default `flower.enabled` | if the Flower UI should be deployed | `true` `flower.resources` | resource requests/limits for the flower Pods | `{}` `flower.nodeSelector` | the nodeSelector configs for the flower Pods | `{}` +`flower.topologySpreadConstraints` | the topologySpreadConstraints configs for the flower Pods | `[]` `flower.affinity` | the affinity configs for the flower Pods | `{}` `flower.tolerations` | the toleration configs for the flower Pods | `[]` `flower.securityContext` | the security context for the flower Pods | `{}` @@ -438,6 +444,7 @@ Parameter | Description | Default `pgbouncer.image.*` | configs for the pgbouncer container image | `` `pgbouncer.resources` | resource requests/limits for the pgbouncer Pods | `{}` `pgbouncer.nodeSelector` | the nodeSelector configs for the pgbouncer Pods | `{}` +`pgbouncer.topologySpreadConstraints` | the topologySpreadConstraints configs for the pgbouncer Pods | `[]` `pgbouncer.affinity` | the affinity configs for the pgbouncer Pods | `{}` `pgbouncer.tolerations` | the toleration configs for the pgbouncer Pods | `[]` `pgbouncer.securityContext` | the security context for the pgbouncer Pods | `{}` diff --git a/charts/airflow/docs/faq/kubernetes/affinity-node-selectors-tolerations.md b/charts/airflow/docs/faq/kubernetes/affinity-node-selectors-tolerations.md index 7e4c79de..11d604c4 100644 --- a/charts/airflow/docs/faq/kubernetes/affinity-node-selectors-tolerations.md +++ b/charts/airflow/docs/faq/kubernetes/affinity-node-selectors-tolerations.md @@ -5,13 +5,13 @@ # Configure Pod Affinity/Selectors/Tolerations If your environment needs to use Pod [affinity](https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/#affinity-and-anti-affinity), -[nodeSelector](https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/#nodeselector), +[nodeSelector](https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/#nodeselector), [topologySpreadConstraints](https://kubernetes.io/docs/concepts/scheduling-eviction/topology-spread-constraints/#topologyspreadconstraints-field), or [tolerations](https://kubernetes.io/docs/concepts/scheduling-eviction/taint-and-toleration/), we provide many values that allow fine-grained control over the Pod definitions. ## Global Configs -To set affinity, nodeSelector, and tolerations for all airflow Pods, you may use the `airflow.{defaultNodeSelector,defaultAffinity,defaultTolerations}` values: +To set affinity, nodeSelector, topologySpreadConstraints, and tolerations for all airflow Pods, you may use the `airflow.{defaultNodeSelector,defaultTopologySpreadConstraints,defaultAffinity,defaultTolerations}` values: ```yaml airflow: @@ -20,6 +20,16 @@ airflow: # my_node_label_1: value1 # my_node_label_2: value2 + ## https://kubernetes.io/docs/concepts/scheduling-eviction/topology-spread-constraints/#topologyspreadconstraints-field + defaultTopologySpreadConstraints: [] + # - maxSkew: 1 + # topologyKey: topology.kubernetes.io/zone + # whenUnsatisfiable: DoNotSchedule + # labelSelector: + # matchLabels: + # my_label_1: value1 + # my_label_2: value2 + ## https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.20/#affinity-v1-core defaultAffinity: {} # podAffinity: @@ -69,53 +79,60 @@ redis: ## Per-Resource Configs -To set affinity, nodeSelector, and tolerations for specific pods, you may use the following values: +To set affinity, nodeSelector, topologySpreadConstraints, and tolerations for specific pods, you may use the following values: ```yaml airflow: ## airflow KubernetesExecutor pod_template kubernetesPodTemplate: nodeSelector: {} + topologySpreadConstraints: [] affinity: {} tolerations: [] ## sync deployments sync: nodeSelector: {} + topologySpreadConstraints: [] affinity: {} tolerations: [] ## airflow schedulers scheduler: nodeSelector: {} + topologySpreadConstraints: [] affinity: {} tolerations: [] ## airflow webserver web: nodeSelector: {} + topologySpreadConstraints: [] affinity: {} tolerations: [] ## airflow workers workers: nodeSelector: {} + topologySpreadConstraints: [] affinity: {} tolerations: [] ## airflow triggerer triggerer: nodeSelector: {} + topologySpreadConstraints: [] affinity: {} tolerations: [] ## airflow workers flower: nodeSelector: {} + topologySpreadConstraints: [] affinity: {} tolerations: [] ``` > 🟦 __Tip__ 🟦 > -> The `airflow.{defaultNodeSelector,defaultAffinity,defaultTolerations}` values are overridden by the per-resource values like `scheduler.{nodeSelector,affinity,tolerations}`. +> The `airflow.{defaultNodeSelector,defaultTopologySpreadConstraints,defaultAffinity,defaultTolerations}` values are overridden by the per-resource values like `scheduler.{nodeSelector,topologySpreadConstraints,affinity,tolerations}`. diff --git a/charts/airflow/files/pod_template.kubernetes-helm-yaml b/charts/airflow/files/pod_template.kubernetes-helm-yaml index b4c23254..41a36973 100644 --- a/charts/airflow/files/pod_template.kubernetes-helm-yaml +++ b/charts/airflow/files/pod_template.kubernetes-helm-yaml @@ -1,4 +1,5 @@ {{- $podNodeSelector := include "airflow.podNodeSelector" (dict "Release" .Release "Values" .Values "nodeSelector" .Values.airflow.kubernetesPodTemplate.nodeSelector) }} +{{- $podTopologySpreadConstraints := include "airflow.podTopologySpreadConstraints" (dict "Release" .Release "Values" .Values "topologySpreadConstraints" .Values.airflow.kubernetesPodTemplate.topologySpreadConstraints) }} {{- $podAffinity := include "airflow.podAffinity" (dict "Release" .Release "Values" .Values "affinity" .Values.airflow.kubernetesPodTemplate.affinity) }} {{- $podTolerations := include "airflow.podTolerations" (dict "Release" .Release "Values" .Values "tolerations" .Values.airflow.kubernetesPodTemplate.tolerations) }} {{- $podSecurityContext := include "airflow.podSecurityContext" (dict "Release" .Release "Values" .Values "securityContext" .Values.airflow.kubernetesPodTemplate.securityContext) }} @@ -31,6 +32,10 @@ spec: nodeSelector: {{- $podNodeSelector | nindent 4 }} {{- end }} + {{- if $podTopologySpreadConstraints }} + topologySpreadConstraints: + {{- $podTopologySpreadConstraints | nindent 4 }} + {{- end }} {{- if $podAffinity }} affinity: {{- $podAffinity | nindent 4 }} diff --git a/charts/airflow/templates/_helpers/pods.tpl b/charts/airflow/templates/_helpers/pods.tpl index 3cacda28..0af90d0b 100644 --- a/charts/airflow/templates/_helpers/pods.tpl +++ b/charts/airflow/templates/_helpers/pods.tpl @@ -29,6 +29,14 @@ EXAMPLE USAGE: {{ include "airflow.nodeSelector" (dict "Release" .Release "Value {{- .nodeSelector | default .Values.airflow.defaultNodeSelector | toYaml }} {{- end }} +{{/* +Define the topologySpreadConstraints for airflow pods +EXAMPLE USAGE: {{ include "airflow.podTopologySpreadConstraints" (dict "Release" .Release "Values" .Values "topologySpreadConstraints" $topologySpreadConstraints) }} +*/}} +{{- define "airflow.podTopologySpreadConstraints" }} +{{- .topologySpreadConstraints | default .Values.airflow.defaultTopologySpreadConstraints | toYaml }} +{{- end }} + {{/* Define the Affinity for airflow pods EXAMPLE USAGE: {{ include "airflow.podAffinity" (dict "Release" .Release "Values" .Values "affinity" $affinity) }} diff --git a/charts/airflow/templates/db-migrations/db-migrations-deployment.yaml b/charts/airflow/templates/db-migrations/db-migrations-deployment.yaml index 1ad60348..153cf94d 100644 --- a/charts/airflow/templates/db-migrations/db-migrations-deployment.yaml +++ b/charts/airflow/templates/db-migrations/db-migrations-deployment.yaml @@ -1,5 +1,6 @@ {{- if and (.Values.airflow.dbMigrations.enabled) (not .Values.airflow.dbMigrations.runAsJob) }} {{- $podNodeSelector := include "airflow.podNodeSelector" (dict "Release" .Release "Values" .Values "nodeSelector" .Values.airflow.dbMigrations.nodeSelector) }} +{{- $podTopologySpreadConstraints := include "airflow.podTopologySpreadConstraints" (dict "Release" .Release "Values" .Values "topologySpreadConstraints" .Values.airflow.dbMigrations.topologySpreadConstraints) }} {{- $podAffinity := include "airflow.podAffinity" (dict "Release" .Release "Values" .Values "affinity" .Values.airflow.dbMigrations.affinity) }} {{- $podTolerations := include "airflow.podTolerations" (dict "Release" .Release "Values" .Values "tolerations" .Values.airflow.dbMigrations.tolerations) }} {{- $podSecurityContext := include "airflow.podSecurityContext" (dict "Release" .Release "Values" .Values "securityContext" .Values.airflow.dbMigrations.securityContext) }} @@ -65,6 +66,10 @@ spec: nodeSelector: {{- $podNodeSelector | nindent 8 }} {{- end }} + {{- if $podTopologySpreadConstraints }} + topologySpreadConstraints: + {{- $podTopologySpreadConstraints | nindent 8 }} + {{- end }} {{- if $podAffinity }} affinity: {{- $podAffinity | nindent 8 }} diff --git a/charts/airflow/templates/db-migrations/db-migrations-job.yaml b/charts/airflow/templates/db-migrations/db-migrations-job.yaml index 2e729af6..ee65fb7b 100644 --- a/charts/airflow/templates/db-migrations/db-migrations-job.yaml +++ b/charts/airflow/templates/db-migrations/db-migrations-job.yaml @@ -1,5 +1,6 @@ {{- if and (.Values.airflow.dbMigrations.enabled) (.Values.airflow.dbMigrations.runAsJob) }} {{- $podNodeSelector := include "airflow.podNodeSelector" (dict "Release" .Release "Values" .Values "nodeSelector" .Values.airflow.dbMigrations.nodeSelector) }} +{{- $podTopologySpreadConstraints := include "airflow.podTopologySpreadConstraints" (dict "Release" .Release "Values" .Values "topologySpreadConstraints" .Values.airflow.dbMigrations.topologySpreadConstraints) }} {{- $podAffinity := include "airflow.podAffinity" (dict "Release" .Release "Values" .Values "affinity" .Values.airflow.dbMigrations.affinity) }} {{- $podTolerations := include "airflow.podTolerations" (dict "Release" .Release "Values" .Values "tolerations" .Values.airflow.dbMigrations.tolerations) }} {{- $podSecurityContext := include "airflow.podSecurityContext" (dict "Release" .Release "Values" .Values "securityContext" .Values.airflow.dbMigrations.securityContext) }} @@ -59,6 +60,10 @@ spec: nodeSelector: {{- $podNodeSelector | nindent 8 }} {{- end }} + {{- if $podTopologySpreadConstraints }} + topologySpreadConstraints: + {{- $podTopologySpreadConstraints | nindent 8 }} + {{- end }} {{- if $podAffinity }} affinity: {{- $podAffinity | nindent 8 }} diff --git a/charts/airflow/templates/flower/flower-deployment.yaml b/charts/airflow/templates/flower/flower-deployment.yaml index 436f7de4..dca41531 100644 --- a/charts/airflow/templates/flower/flower-deployment.yaml +++ b/charts/airflow/templates/flower/flower-deployment.yaml @@ -1,5 +1,6 @@ {{- if .Values.flower.enabled }} {{- $podNodeSelector := include "airflow.podNodeSelector" (dict "Release" .Release "Values" .Values "nodeSelector" .Values.flower.nodeSelector) }} +{{- $podTopologySpreadConstraints := include "airflow.podTopologySpreadConstraints" (dict "Release" .Release "Values" .Values "topologySpreadConstraints" .Values.flower.topologySpreadConstraints) }} {{- $podAffinity := include "airflow.podAffinity" (dict "Release" .Release "Values" .Values "affinity" .Values.flower.affinity) }} {{- $podTolerations := include "airflow.podTolerations" (dict "Release" .Release "Values" .Values "tolerations" .Values.flower.tolerations) }} {{- $podSecurityContext := include "airflow.podSecurityContext" (dict "Release" .Release "Values" .Values "securityContext" .Values.flower.securityContext) }} @@ -69,6 +70,10 @@ spec: nodeSelector: {{- $podNodeSelector | nindent 8 }} {{- end }} + {{- if $podTopologySpreadConstraints }} + topologySpreadConstraints: + {{- $podTopologySpreadConstraints | nindent 8 }} + {{- end }} {{- if $podAffinity }} affinity: {{- $podAffinity | nindent 8 }} diff --git a/charts/airflow/templates/pgbouncer/pgbouncer-deployment.yaml b/charts/airflow/templates/pgbouncer/pgbouncer-deployment.yaml index 84cef513..dadc5d25 100644 --- a/charts/airflow/templates/pgbouncer/pgbouncer-deployment.yaml +++ b/charts/airflow/templates/pgbouncer/pgbouncer-deployment.yaml @@ -1,5 +1,6 @@ {{- if include "airflow.pgbouncer.should_use" . }} {{- $podNodeSelector := include "airflow.podNodeSelector" (dict "Release" .Release "Values" .Values "nodeSelector" .Values.pgbouncer.nodeSelector) }} +{{- $podTopologySpreadConstraints := include "airflow.podTopologySpreadConstraints" (dict "Release" .Release "Values" .Values "topologySpreadConstraints" .Values.pgbouncer.topologySpreadConstraints) }} {{- $podAffinity := include "airflow.podAffinity" (dict "Release" .Release "Values" .Values "affinity" .Values.pgbouncer.affinity) }} {{- $podTolerations := include "airflow.podTolerations" (dict "Release" .Release "Values" .Values "tolerations" .Values.pgbouncer.tolerations) }} {{- $podSecurityContext := include "airflow.podSecurityContext" (dict "Release" .Release "Values" .Values "securityContext" .Values.pgbouncer.securityContext) }} @@ -63,6 +64,10 @@ spec: nodeSelector: {{- $podNodeSelector | nindent 8 }} {{- end }} + {{- if $podTopologySpreadConstraints }} + topologySpreadConstraints: + {{- $podTopologySpreadConstraints | nindent 8 }} + {{- end }} {{- if $podAffinity }} affinity: {{- $podAffinity | nindent 8 }} diff --git a/charts/airflow/templates/scheduler/scheduler-deployment.yaml b/charts/airflow/templates/scheduler/scheduler-deployment.yaml index 9660f83b..cf8e1438 100644 --- a/charts/airflow/templates/scheduler/scheduler-deployment.yaml +++ b/charts/airflow/templates/scheduler/scheduler-deployment.yaml @@ -1,4 +1,5 @@ {{- $podNodeSelector := include "airflow.podNodeSelector" (dict "Release" .Release "Values" .Values "nodeSelector" .Values.scheduler.nodeSelector) }} +{{- $podTopologySpreadConstraints := include "airflow.podTopologySpreadConstraints" (dict "Release" .Release "Values" .Values "topologySpreadConstraints" .Values.scheduler.topologySpreadConstraints) }} {{- $podAffinity := include "airflow.podAffinity" (dict "Release" .Release "Values" .Values "affinity" .Values.scheduler.affinity) }} {{- $podTolerations := include "airflow.podTolerations" (dict "Release" .Release "Values" .Values "tolerations" .Values.scheduler.tolerations) }} {{- $podSecurityContext := include "airflow.podSecurityContext" (dict "Release" .Release "Values" .Values "securityContext" .Values.scheduler.securityContext) }} @@ -77,6 +78,10 @@ spec: nodeSelector: {{- $podNodeSelector | nindent 8 }} {{- end }} + {{- if $podTopologySpreadConstraints }} + topologySpreadConstraints: + {{- $podTopologySpreadConstraints | nindent 8 }} + {{- end }} {{- if $podAffinity }} affinity: {{- $podAffinity | nindent 8 }} diff --git a/charts/airflow/templates/sync/sync-connections-deployment.yaml b/charts/airflow/templates/sync/sync-connections-deployment.yaml index 82734305..bc085d3c 100644 --- a/charts/airflow/templates/sync/sync-connections-deployment.yaml +++ b/charts/airflow/templates/sync/sync-connections-deployment.yaml @@ -1,5 +1,6 @@ {{- if and (.Values.airflow.connections) (.Values.airflow.connectionsUpdate) }} {{- $podNodeSelector := include "airflow.podNodeSelector" (dict "Release" .Release "Values" .Values "nodeSelector" .Values.airflow.sync.nodeSelector) }} +{{- $podTopologySpreadConstraints := include "airflow.podTopologySpreadConstraints" (dict "Release" .Release "Values" .Values "topologySpreadConstraints" .Values.airflow.sync.topologySpreadConstraints) }} {{- $podAffinity := include "airflow.podAffinity" (dict "Release" .Release "Values" .Values "affinity" .Values.airflow.sync.affinity) }} {{- $podTolerations := include "airflow.podTolerations" (dict "Release" .Release "Values" .Values "tolerations" .Values.airflow.sync.tolerations) }} {{- $podSecurityContext := include "airflow.podSecurityContext" (dict "Release" .Release "Values" .Values "securityContext" .Values.airflow.sync.securityContext) }} @@ -65,6 +66,10 @@ spec: nodeSelector: {{- $podNodeSelector | nindent 8 }} {{- end }} + {{- if $podTopologySpreadConstraints }} + topologySpreadConstraints: + {{- $podTopologySpreadConstraints | nindent 8 }} + {{- end }} {{- if $podAffinity }} affinity: {{- $podAffinity | nindent 8 }} diff --git a/charts/airflow/templates/sync/sync-connections-job.yaml b/charts/airflow/templates/sync/sync-connections-job.yaml index c77002ad..ba1d8a84 100644 --- a/charts/airflow/templates/sync/sync-connections-job.yaml +++ b/charts/airflow/templates/sync/sync-connections-job.yaml @@ -1,5 +1,6 @@ {{- if and (.Values.airflow.connections) (not .Values.airflow.connectionsUpdate) }} {{- $podNodeSelector := include "airflow.podNodeSelector" (dict "Release" .Release "Values" .Values "nodeSelector" .Values.scheduler.nodeSelector) }} +{{- $podTopologySpreadConstraints := include "airflow.podTopologySpreadConstraints" (dict "Release" .Release "Values" .Values "topologySpreadConstraints" .Values.scheduler.topologySpreadConstraints) }} {{- $podAffinity := include "airflow.podAffinity" (dict "Release" .Release "Values" .Values "affinity" .Values.scheduler.affinity) }} {{- $podTolerations := include "airflow.podTolerations" (dict "Release" .Release "Values" .Values "tolerations" .Values.scheduler.tolerations) }} {{- $podSecurityContext := include "airflow.podSecurityContext" (dict "Release" .Release "Values" .Values "securityContext" .Values.airflow.sync.securityContext) }} @@ -59,6 +60,10 @@ spec: nodeSelector: {{- $podNodeSelector | nindent 8 }} {{- end }} + {{- if $podTopologySpreadConstraints }} + topologySpreadConstraints: + {{- $podTopologySpreadConstraints | nindent 8 }} + {{- end }} {{- if $podAffinity }} affinity: {{- $podAffinity | nindent 8 }} diff --git a/charts/airflow/templates/sync/sync-pools-deployment.yaml b/charts/airflow/templates/sync/sync-pools-deployment.yaml index 1635a987..c896ccf6 100644 --- a/charts/airflow/templates/sync/sync-pools-deployment.yaml +++ b/charts/airflow/templates/sync/sync-pools-deployment.yaml @@ -1,5 +1,6 @@ {{- if and (.Values.airflow.pools) (.Values.airflow.poolsUpdate) }} {{- $podNodeSelector := include "airflow.podNodeSelector" (dict "Release" .Release "Values" .Values "nodeSelector" .Values.airflow.sync.nodeSelector) }} +{{- $podTopologySpreadConstraints := include "airflow.podTopologySpreadConstraints" (dict "Release" .Release "Values" .Values "topologySpreadConstraints" .Values.airflow.sync.topologySpreadConstraints) }} {{- $podAffinity := include "airflow.podAffinity" (dict "Release" .Release "Values" .Values "affinity" .Values.airflow.sync.affinity) }} {{- $podTolerations := include "airflow.podTolerations" (dict "Release" .Release "Values" .Values "tolerations" .Values.airflow.sync.tolerations) }} {{- $podSecurityContext := include "airflow.podSecurityContext" (dict "Release" .Release "Values" .Values "securityContext" .Values.airflow.sync.securityContext) }} @@ -65,6 +66,10 @@ spec: nodeSelector: {{- $podNodeSelector | nindent 8 }} {{- end }} + {{- if $podTopologySpreadConstraints }} + topologySpreadConstraints: + {{- $podTopologySpreadConstraints | nindent 8 }} + {{- end }} {{- if $podAffinity }} affinity: {{- $podAffinity | nindent 8 }} diff --git a/charts/airflow/templates/sync/sync-pools-job.yaml b/charts/airflow/templates/sync/sync-pools-job.yaml index cda0f7cd..48f9030c 100644 --- a/charts/airflow/templates/sync/sync-pools-job.yaml +++ b/charts/airflow/templates/sync/sync-pools-job.yaml @@ -1,5 +1,6 @@ {{- if and (.Values.airflow.pools) (not .Values.airflow.poolsUpdate) }} {{- $podNodeSelector := include "airflow.podNodeSelector" (dict "Release" .Release "Values" .Values "nodeSelector" .Values.airflow.sync.nodeSelector) }} +{{- $podTopologySpreadConstraints := include "airflow.podTopologySpreadConstraints" (dict "Release" .Release "Values" .Values "topologySpreadConstraints" .Values.airflow.sync.topologySpreadConstraints) }} {{- $podAffinity := include "airflow.podAffinity" (dict "Release" .Release "Values" .Values "affinity" .Values.airflow.sync.affinity) }} {{- $podTolerations := include "airflow.podTolerations" (dict "Release" .Release "Values" .Values "tolerations" .Values.airflow.sync.tolerations) }} {{- $podSecurityContext := include "airflow.podSecurityContext" (dict "Release" .Release "Values" .Values "securityContext" .Values.airflow.sync.securityContext) }} @@ -59,6 +60,10 @@ spec: nodeSelector: {{- $podNodeSelector | nindent 8 }} {{- end }} + {{- if $podTopologySpreadConstraints }} + topologySpreadConstraints: + {{- $podTopologySpreadConstraints | nindent 8 }} + {{- end }} {{- if $podAffinity }} affinity: {{- $podAffinity | nindent 8 }} diff --git a/charts/airflow/templates/sync/sync-users-deployment.yaml b/charts/airflow/templates/sync/sync-users-deployment.yaml index d61d8943..7403d2e7 100644 --- a/charts/airflow/templates/sync/sync-users-deployment.yaml +++ b/charts/airflow/templates/sync/sync-users-deployment.yaml @@ -1,5 +1,6 @@ {{- if and (.Values.airflow.users) (.Values.airflow.usersUpdate) }} {{- $podNodeSelector := include "airflow.podNodeSelector" (dict "Release" .Release "Values" .Values "nodeSelector" .Values.airflow.sync.nodeSelector) }} +{{- $podTopologySpreadConstraints := include "airflow.podTopologySpreadConstraints" (dict "Release" .Release "Values" .Values "topologySpreadConstraints" .Values.airflow.sync.topologySpreadConstraints) }} {{- $podAffinity := include "airflow.podAffinity" (dict "Release" .Release "Values" .Values "affinity" .Values.airflow.sync.affinity) }} {{- $podTolerations := include "airflow.podTolerations" (dict "Release" .Release "Values" .Values "tolerations" .Values.airflow.sync.tolerations) }} {{- $podSecurityContext := include "airflow.podSecurityContext" (dict "Release" .Release "Values" .Values "securityContext" .Values.airflow.sync.securityContext) }} @@ -65,6 +66,10 @@ spec: nodeSelector: {{- $podNodeSelector | nindent 8 }} {{- end }} + {{- if $podTopologySpreadConstraints }} + topologySpreadConstraints: + {{- $podTopologySpreadConstraints | nindent 8 }} + {{- end }} {{- if $podAffinity }} affinity: {{- $podAffinity | nindent 8 }} diff --git a/charts/airflow/templates/sync/sync-users-job.yaml b/charts/airflow/templates/sync/sync-users-job.yaml index d96c8da7..452bef3c 100644 --- a/charts/airflow/templates/sync/sync-users-job.yaml +++ b/charts/airflow/templates/sync/sync-users-job.yaml @@ -1,5 +1,6 @@ {{- if and (.Values.airflow.users) (not .Values.airflow.usersUpdate) }} {{- $podNodeSelector := include "airflow.podNodeSelector" (dict "Release" .Release "Values" .Values "nodeSelector" .Values.airflow.sync.nodeSelector) }} +{{- $podTopologySpreadConstraints := include "airflow.podTopologySpreadConstraints" (dict "Release" .Release "Values" .Values "topologySpreadConstraints" .Values.airflow.sync.topologySpreadConstraints) }} {{- $podAffinity := include "airflow.podAffinity" (dict "Release" .Release "Values" .Values "affinity" .Values.airflow.sync.affinity) }} {{- $podTolerations := include "airflow.podTolerations" (dict "Release" .Release "Values" .Values "tolerations" .Values.airflow.sync.tolerations) }} {{- $podSecurityContext := include "airflow.podSecurityContext" (dict "Release" .Release "Values" .Values "securityContext" .Values.airflow.sync.securityContext) }} @@ -59,6 +60,10 @@ spec: nodeSelector: {{- $podNodeSelector | nindent 8 }} {{- end }} + {{- if $podTopologySpreadConstraints }} + topologySpreadConstraints: + {{- $podTopologySpreadConstraints | nindent 8 }} + {{- end }} {{- if $podAffinity }} affinity: {{- $podAffinity | nindent 8 }} diff --git a/charts/airflow/templates/sync/sync-variables-deployment.yaml b/charts/airflow/templates/sync/sync-variables-deployment.yaml index 21926197..3564bddb 100644 --- a/charts/airflow/templates/sync/sync-variables-deployment.yaml +++ b/charts/airflow/templates/sync/sync-variables-deployment.yaml @@ -1,5 +1,6 @@ {{- if and (.Values.airflow.variables) (.Values.airflow.variablesUpdate) }} {{- $podNodeSelector := include "airflow.podNodeSelector" (dict "Release" .Release "Values" .Values "nodeSelector" .Values.airflow.sync.nodeSelector) }} +{{- $podTopologySpreadConstraints := include "airflow.podTopologySpreadConstraints" (dict "Release" .Release "Values" .Values "topologySpreadConstraints" .Values.airflow.sync.topologySpreadConstraints) }} {{- $podAffinity := include "airflow.podAffinity" (dict "Release" .Release "Values" .Values "affinity" .Values.airflow.sync.affinity) }} {{- $podTolerations := include "airflow.podTolerations" (dict "Release" .Release "Values" .Values "tolerations" .Values.airflow.sync.tolerations) }} {{- $podSecurityContext := include "airflow.podSecurityContext" (dict "Release" .Release "Values" .Values "securityContext" .Values.airflow.sync.securityContext) }} @@ -65,6 +66,10 @@ spec: nodeSelector: {{- $podNodeSelector | nindent 8 }} {{- end }} + {{- if $podTopologySpreadConstraints }} + topologySpreadConstraints: + {{- $podTopologySpreadConstraints | nindent 8 }} + {{- end }} {{- if $podAffinity }} affinity: {{- $podAffinity | nindent 8 }} diff --git a/charts/airflow/templates/sync/sync-variables-job.yaml b/charts/airflow/templates/sync/sync-variables-job.yaml index bd50349e..22995f84 100644 --- a/charts/airflow/templates/sync/sync-variables-job.yaml +++ b/charts/airflow/templates/sync/sync-variables-job.yaml @@ -1,5 +1,6 @@ {{- if and (.Values.airflow.variables) (not .Values.airflow.variablesUpdate) }} {{- $podNodeSelector := include "airflow.podNodeSelector" (dict "Release" .Release "Values" .Values "nodeSelector" .Values.airflow.sync.nodeSelector) }} +{{- $podTopologySpreadConstraints := include "airflow.podTopologySpreadConstraints" (dict "Release" .Release "Values" .Values "topologySpreadConstraints" .Values.airflow.sync.topologySpreadConstraints) }} {{- $podAffinity := include "airflow.podAffinity" (dict "Release" .Release "Values" .Values "affinity" .Values.airflow.sync.affinity) }} {{- $podTolerations := include "airflow.podTolerations" (dict "Release" .Release "Values" .Values "tolerations" .Values.airflow.sync.tolerations) }} {{- $podSecurityContext := include "airflow.podSecurityContext" (dict "Release" .Release "Values" .Values "securityContext" .Values.airflow.sync.securityContext) }} @@ -59,6 +60,10 @@ spec: nodeSelector: {{- $podNodeSelector | nindent 8 }} {{- end }} + {{- if $podTopologySpreadConstraints }} + topologySpreadConstraints: + {{- $podTopologySpreadConstraints | nindent 8 }} + {{- end }} {{- if $podAffinity }} affinity: {{- $podAffinity | nindent 8 }} diff --git a/charts/airflow/templates/triggerer/triggerer-deployment.yaml b/charts/airflow/templates/triggerer/triggerer-deployment.yaml index 2497f68f..480a281a 100644 --- a/charts/airflow/templates/triggerer/triggerer-deployment.yaml +++ b/charts/airflow/templates/triggerer/triggerer-deployment.yaml @@ -1,5 +1,6 @@ {{- if include "airflow.triggerer.should_use" . }} {{- $podNodeSelector := include "airflow.podNodeSelector" (dict "Release" .Release "Values" .Values "nodeSelector" .Values.triggerer.nodeSelector) }} +{{- $podTopologySpreadConstraints := include "airflow.podTopologySpreadConstraints" (dict "Release" .Release "Values" .Values "topologySpreadConstraints" .Values.triggerer.topologySpreadConstraints) }} {{- $podAffinity := include "airflow.podAffinity" (dict "Release" .Release "Values" .Values "affinity" .Values.triggerer.affinity) }} {{- $podTolerations := include "airflow.podTolerations" (dict "Release" .Release "Values" .Values "tolerations" .Values.triggerer.tolerations) }} {{- $podSecurityContext := include "airflow.podSecurityContext" (dict "Release" .Release "Values" .Values "securityContext" .Values.triggerer.securityContext) }} @@ -69,6 +70,10 @@ spec: nodeSelector: {{- $podNodeSelector | nindent 8 }} {{- end }} + {{- if $podTopologySpreadConstraints }} + topologySpreadConstraints: + {{- $podTopologySpreadConstraints | nindent 8 }} + {{- end }} {{- if $podAffinity }} affinity: {{- $podAffinity | nindent 8 }} diff --git a/charts/airflow/templates/webserver/webserver-deployment.yaml b/charts/airflow/templates/webserver/webserver-deployment.yaml index 605b012c..4aa33bdf 100644 --- a/charts/airflow/templates/webserver/webserver-deployment.yaml +++ b/charts/airflow/templates/webserver/webserver-deployment.yaml @@ -1,4 +1,5 @@ {{- $podNodeSelector := include "airflow.podNodeSelector" (dict "Release" .Release "Values" .Values "nodeSelector" .Values.web.nodeSelector) }} +{{- $podTopologySpreadConstraints := include "airflow.podTopologySpreadConstraints" (dict "Release" .Release "Values" .Values "topologySpreadConstraints" .Values.web.topologySpreadConstraints) }} {{- $podAffinity := include "airflow.podAffinity" (dict "Release" .Release "Values" .Values "affinity" .Values.web.affinity) }} {{- $podTolerations := include "airflow.podTolerations" (dict "Release" .Release "Values" .Values "tolerations" .Values.web.tolerations) }} {{- $podSecurityContext := include "airflow.podSecurityContext" (dict "Release" .Release "Values" .Values "securityContext" .Values.web.securityContext) }} @@ -71,6 +72,10 @@ spec: nodeSelector: {{- $podNodeSelector | nindent 8 }} {{- end }} + {{- if $podTopologySpreadConstraints }} + topologySpreadConstraints: + {{- $podTopologySpreadConstraints | nindent 8 }} + {{- end }} {{- if $podAffinity }} affinity: {{- $podAffinity | nindent 8 }} diff --git a/charts/airflow/templates/worker/worker-statefulset.yaml b/charts/airflow/templates/worker/worker-statefulset.yaml index d2e50bc2..81efb7d2 100644 --- a/charts/airflow/templates/worker/worker-statefulset.yaml +++ b/charts/airflow/templates/worker/worker-statefulset.yaml @@ -1,5 +1,6 @@ {{- if .Values.workers.enabled }} {{- $podNodeSelector := include "airflow.podNodeSelector" (dict "Release" .Release "Values" .Values "nodeSelector" .Values.workers.nodeSelector) }} +{{- $podTopologySpreadConstraints := include "airflow.podTopologySpreadConstraints" (dict "Release" .Release "Values" .Values "topologySpreadConstraints" .Values.workers.topologySpreadConstraints) }} {{- $podAffinity := include "airflow.podAffinity" (dict "Release" .Release "Values" .Values "affinity" .Values.workers.affinity) }} {{- $podTolerations := include "airflow.podTolerations" (dict "Release" .Release "Values" .Values "tolerations" .Values.workers.tolerations) }} {{- $podSecurityContext := include "airflow.podSecurityContext" (dict "Release" .Release "Values" .Values "securityContext" .Values.workers.securityContext) }} @@ -75,6 +76,10 @@ spec: nodeSelector: {{- $podNodeSelector | nindent 8 }} {{- end }} + {{- if $podTopologySpreadConstraints }} + topologySpreadConstraints: + {{- $podTopologySpreadConstraints | nindent 8 }} + {{- end }} {{- if $podAffinity }} affinity: {{- $podAffinity | nindent 8 }} diff --git a/charts/airflow/values.yaml b/charts/airflow/values.yaml index 9bbe6aa2..50e6a7fe 100644 --- a/charts/airflow/values.yaml +++ b/charts/airflow/values.yaml @@ -184,6 +184,11 @@ airflow: ## defaultNodeSelector: {} + ## default topologySpreadConstraints for airflow Pods (is overridden by pod-specific values) + ## - docs for topologySpreadConstraints: + ## https://kubernetes.io/docs/concepts/scheduling-eviction/topology-spread-constraints/#topologyspreadconstraints-field + defaultTopologySpreadConstraints: [] + ## default affinity configs for airflow Pods (is overridden by pod-specific values) ## - spec for Affinity: ## https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.20/#affinity-v1-core @@ -310,6 +315,12 @@ airflow: ## nodeSelector: {} + ## the topologySpreadConstraints configs for the Pod template + ## - docs for topologySpreadConstraints: + ## https://kubernetes.io/docs/concepts/scheduling-eviction/topology-spread-constraints/#topologyspreadconstraints-field + ## + topologySpreadConstraints: [] + ## the affinity configs for the Pod template ## - spec for Affinity: ## https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.20/#affinity-v1-core @@ -402,6 +413,12 @@ airflow: ## nodeSelector: {} + ## the topologySpreadConstraints configs for the db-migrations Pods + ## - docs for topologySpreadConstraints: + ## https://kubernetes.io/docs/concepts/scheduling-eviction/topology-spread-constraints/#topologyspreadconstraints-field + ## + topologySpreadConstraints: [] + ## the affinity configs for the db-migrations Pods ## - spec for Affinity: ## https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.20/#affinity-v1-core @@ -463,6 +480,12 @@ airflow: ## nodeSelector: {} + ## the topologySpreadConstraints configs for the sync Pods + ## - docs for topologySpreadConstraints: + ## https://kubernetes.io/docs/concepts/scheduling-eviction/topology-spread-constraints/#topologyspreadconstraints-field + ## + topologySpreadConstraints: [] + ## the affinity configs for the sync Pods ## - spec for Affinity: ## https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.20/#affinity-v1-core @@ -522,6 +545,12 @@ scheduler: ## nodeSelector: {} + ## the topologySpreadConstraints configs for the scheduler Pods + ## - docs for topologySpreadConstraints: + ## https://kubernetes.io/docs/concepts/scheduling-eviction/topology-spread-constraints/#topologyspreadconstraints-field + ## + topologySpreadConstraints: [] + ## the affinity configs for the scheduler Pods ## - spec of Affinity: ## https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.20/#affinity-v1-core @@ -721,6 +750,12 @@ web: ## nodeSelector: {} + ## the topologySpreadConstraints configs for the web Pods + ## - docs for topologySpreadConstraints: + ## https://kubernetes.io/docs/concepts/scheduling-eviction/topology-spread-constraints/#topologyspreadconstraints-field + ## + topologySpreadConstraints: [] + ## the affinity configs for the web Pods ## - spec for Affinity: ## https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.20/#affinity-v1-core @@ -857,6 +892,12 @@ workers: ## nodeSelector: {} + ## the topologySpreadConstraints configs for the worker Pods + ## - docs for topologySpreadConstraints: + ## https://kubernetes.io/docs/concepts/scheduling-eviction/topology-spread-constraints/#topologyspreadconstraints-field + ## + topologySpreadConstraints: [] + ## the affinity configs for the worker Pods ## - spec for Affinity: ## https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.20/#affinity-v1-core @@ -1039,6 +1080,12 @@ triggerer: ## nodeSelector: {} + ## the topologySpreadConstraints configs for the triggerer Pods + ## - docs for topologySpreadConstraints: + ## https://kubernetes.io/docs/concepts/scheduling-eviction/topology-spread-constraints/#topologyspreadconstraints-field + ## + topologySpreadConstraints: [] + ## the affinity configs for the triggerer Pods ## - spec for Affinity: ## https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.20/#affinity-v1-core @@ -1156,6 +1203,12 @@ flower: ## nodeSelector: {} + ## the topologySpreadConstraints configs for the flower Pods + ## - docs for topologySpreadConstraints: + ## https://kubernetes.io/docs/concepts/scheduling-eviction/topology-spread-constraints/#topologyspreadconstraints-field + ## + topologySpreadConstraints: [] + ## the affinity configs for the flower Pods ## - spec for Affinity: ## https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.20/#affinity-v1-core @@ -1714,6 +1767,12 @@ pgbouncer: ## nodeSelector: {} + ## the topologySpreadConstraints configs for the pgbouncer Pods + ## - docs for topologySpreadConstraints: + ## https://kubernetes.io/docs/concepts/scheduling-eviction/topology-spread-constraints/#topologyspreadconstraints-field + ## + topologySpreadConstraints: [] + ## the affinity configs for the pgbouncer Pods ## - spec for Affinity: ## https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.20/#affinity-v1-core