gke-nodepool

GKE nodepool module

This module allows simplified creation and management of individual GKE nodepools, setting sensible defaults (eg a service account is created for nodes if none is set) and allowing for less verbose usage in most use cases.

Example usage

Module defaults

If no specific node configuration is set via variables, the module uses the provider's defaults only setting OAuth scopes to a minimal working set and the node machine type to n1-standard-1. The service account set by the provider in this case is the GCE default service account.

module "cluster-1-nodepool-1" {
  source       = "./fabric/modules/gke-nodepool"
  project_id   = "myproject"
  cluster_name = "cluster-1"
  location     = "europe-west1-b"
  name         = "nodepool-1"
}
# tftest modules=1 resources=1 inventory=basic.yaml

Internally managed service account

There are three different approaches to defining the nodes service account, all depending on the service_account variable where the create attribute controls creation of a new service account by this module, and the email attribute controls the actual service account to use.

If you create a new service account, its resource and email (in both plain and IAM formats) are then available in outputs to reference it in other modules or resources.

GCE default service account

To use the GCE default service account, you can ignore the variable which is equivalent to { create = null, email = null }. This is what the first example of this document does.

Externally defined service account

To use an existing service account, pass in just the email attribute. If you do this, will most likely want to use the cloud-platform scope.

module "cluster-1-nodepool-1" {
  source       = "./fabric/modules/gke-nodepool"
  project_id   = "myproject"
  cluster_name = "cluster-1"
  location     = "europe-west1-b"
  name         = "nodepool-1"
  service_account = {
    email        = "[email protected]"
    oauth_scopes = ["https://www.googleapis.com/auth/cloud-platform"]
  }
}
# tftest modules=1 resources=1 inventory=external-sa.yaml

Auto-created service account

To have the module create a service account, set the create attribute to true and optionally pass the desired account id in email.

module "cluster-1-nodepool-1" {
  source       = "./fabric/modules/gke-nodepool"
  project_id   = "myproject"
  cluster_name = "cluster-1"
  location     = "europe-west1-b"
  name         = "nodepool-1"
  service_account = {
    create       = true
    email        = "spam-eggs" # optional
    oauth_scopes = ["https://www.googleapis.com/auth/cloud-platform"]
  }
}
# tftest modules=1 resources=2 inventory=create-sa.yaml

Node & node pool configuration

module "cluster-1-nodepool-1" {
  source       = "./fabric/modules/gke-nodepool"
  project_id   = "myproject"
  cluster_name = "cluster-1"
  location     = "europe-west1-b"
  name         = "nodepool-1"
  k8s_labels   = { environment = "dev" }
  service_account = {
    create       = true
    email        = "nodepool-1" # optional
    oauth_scopes = ["https://www.googleapis.com/auth/cloud-platform"]
  }
  node_config = {
    machine_type        = "n2-standard-2"
    disk_size_gb        = 50
    disk_type           = "pd-ssd"
    ephemeral_ssd_count = 1
    gvnic               = true
    spot                = true
  }
  nodepool_config = {
    autoscaling = {
      max_node_count = 10
      min_node_count = 1
    }
    management = {
      auto_repair  = true
      auto_upgrade = false
    }
  }
}
# tftest modules=1 resources=2 inventory=config.yaml

GPU Node & node pool configuration

module "cluster-1-nodepool-gpu-1" {
  source       = "./fabric/modules/gke-nodepool"
  project_id   = "myproject"
  cluster_name = "cluster-1"
  location     = "europe-west4-a"
  name         = "nodepool-gpu-1"
  k8s_labels   = { environment = "dev" }
  service_account = {
    create       = true
    email        = "nodepool-gpu-1" # optional
    oauth_scopes = ["https://www.googleapis.com/auth/cloud-platform"]
  }
  node_config = {
    machine_type        = "a2-highgpu-1g"
    disk_size_gb        = 50
    disk_type           = "pd-ssd"
    ephemeral_ssd_count = 1
    gvnic               = true
    spot                = true
    guest_accelerator = {
      type  = "nvidia-tesla-a100"
      count = 1
      gpu_driver = {
        version = "LATEST"
      }
    }
  }
}
# tftest modules=1 resources=2 inventory=guest-accelerator.yaml

Dynamic Workload Scheduler (DWS) & node pool configuration

This example uses Dynamic Workload Scheduler (DWS) to configure a GPU nodepool.

module "cluster-1-nodepool-dws" {
  source       = "./fabric/modules/gke-nodepool"
  project_id   = "myproject"
  cluster_name = "cluster-1"
  location     = "europe-west4-a"
  name         = "nodepool-dws"
  k8s_labels   = { environment = "dev" }
  service_account = {
    create       = true
    email        = "nodepool-gpu-1" # optional
    oauth_scopes = ["https://www.googleapis.com/auth/cloud-platform"]
  }
  node_config = {
    machine_type        = "g2-standard-4"
    disk_size_gb        = 50
    disk_type           = "pd-ssd"
    ephemeral_ssd_count = 1
    gvnic               = true
    spot                = true
    guest_accelerator = {
      type  = "nvidia-l4"
      count = 1
      gpu_driver = {
        version = "LATEST"
      }
    }
  }
  nodepool_config = {
    autoscaling = {
      max_node_count = 10
      min_node_count = 0
    }
    queued_provisioning = true
  }
  node_count = {
    initial = 0
  }
  reservation_affinity = {
    consume_reservation_type = "NO_RESERVATION"
  }
}
# tftest modules=1 resources=2 inventory=dws.yaml

Variables

name	description	type	required	default
cluster_name	Cluster name.	string	✓
location	Cluster location.	string	✓
project_id	Cluster project id.	string	✓
cluster_id	Cluster id. Optional, but providing cluster_id is recommended to prevent cluster misconfiguration in some of the edge cases.	string		null
gke_version	Kubernetes nodes version. Ignored if auto_upgrade is set in management_config.	string		null
k8s_labels	Kubernetes labels applied to each node.	map(string)		{}
labels	The resource labels to be applied each node (vm).	map(string)		{}
max_pods_per_node	Maximum number of pods per node.	number		null
name	Optional nodepool name.	string		null
node_config	Node-level configuration.	object({…})		{}
node_count	Number of nodes per instance group. Initial value can only be changed by recreation, current is ignored when autoscaling is used.	object({…})		{…}
node_locations	Node locations.	list(string)		null
nodepool_config	Nodepool-level configuration.	object({…})		null
pod_range	Pod secondary range configuration.	object({…})		null
reservation_affinity	Configuration of the desired reservation which instances could take capacity from.	object({…})		null
service_account	Nodepool service account. If this variable is set to null, the default GCE service account will be used. If set and email is null, a service account will be created. If scopes are null a default will be used.	object({…})		{}
sole_tenant_nodegroup	Sole tenant node group.	string		null
tags	Network tags applied to nodes.	list(string)		null
taints	Kubernetes taints applied to all nodes.	map(object({…}))		{}

Outputs

name	description	sensitive
id	Fully qualified nodepool id.
name	Nodepool name.
service_account_email	Service account email.
service_account_iam_email	Service account email.