Skip to content

Latest commit

 

History

History
723 lines (494 loc) · 80.9 KB

CHANGELOG-1.5.md

File metadata and controls

723 lines (494 loc) · 80.9 KB

v1.5.8 - 2023-04-14

Changes by Kind

Bug or Regression

Updates

machine-controller and operating-system-manager

Go

v1.5.7 - 2023-03-23

Changelog since v1.5.6

Changes by Kind

Bug or Regression

  • Download GPG key for the apt repository from dl.k8s.io instead of packages.cloud.google.com (#2725, @kubermatic-bot)

Updates

operating-system-manager

  • Update Operating System Manager to v1.1.3. Fix an issue where cloud-init scripts re-ran on machine reboot (#2717, @ahmedwaleedmalik)

v1.5.6 - 2023-01-27

Changelog since v1.5.5

Changes by Kind

Feature

  • Update kubernetes-cni to v1.2.0 and cri-tools to v1.26.0. This fixes an issue with installing/updating packages when trying to install Kubernetes v1.26.1, v1.25.6, v1.24.10, and v1.23.16 (#2608, @xmudrii)

v1.5.5 - 2023-01-17

Changelog since v1.5.4

Changes by Kind

Feature

  • Update Terraform provider for VMware Cloud Director to v3.8.1 (#2584, @ahmedwaleedmalik)
  • Add support for insecure HTTPS connection to the VMware Cloud Director API in example Terraform configs (#2584, @ahmedwaleedmalik)

Bug or Regression

  • Fix AMI filter for CentOS 7 in Terraform configs for AWS (#2559, @xmudrii)
  • Fix an issue where custom CA bundle was not being propagated to machine-controller-webhook (#2587, @kubermatic-bot)
  • Remove the leftover /tmp/k1-etc-environment file. This fixes an issue with kubeone apply failing if the username is changed (#2563, @kubermatic-bot)
  • Run kubeadm with increased verbosity unconditionally. This only changes the behavior if KubeOne is run without the verbose flag but kubeadm fails, in which case kubeadm is going to print more information about the issue (#2565, @kubermatic-bot)

v1.5.4 - 2022-12-12

Important Registry Change Information

In November, we announced that we are changing all image references from k8s.gcr.io to registry.k8s.io to keep up with the latest upstream changes. This patch release includes this change. Please ensure that any mirrors you use are able to host registry.k8s.io and/or that firewall rules are going to allow access to registry.k8s.io to pull images before upgrading to this KubeOne patch release.

The December Kubernetes patch releases (1.25.5, 1.24.9, 1.23.15, and 1.22.17) are enforcing registry.k8s.io by default. Please keep this in mind if you're using an older KubeOne patch release with the latest Kubernetes patch releases. We strongly advise that you use KubeOne v1.5.4 or newer with the latest Kubernetes patch releases.

Changelog since v1.5.3

Changes by Kind

API Change

  • Image references are changed from k8s.gcr.io to registry.k8s.io. This is done to keep up with the latest upstream changes. Please ensure that any mirrors you use are able to host registry.k8s.io and/or that firewall rules are going to allow access to registry.k8s.io to pull images before applying the next KubeOne patch releases. (#2505, @xmudrii)

Feature

Bug or Regression

v1.5.3 - 2022-11-11

Important Registry Change Information

For the next series of KubeOne and KKP patch releases, image references will move from k8s.gcr.io to registry.k8s.io. This will be done to keep up with the latest upstream changes. Please ensure that any mirrors you use are able to host registry.k8s.io and/or that firewall rules are going to allow access to registry.k8s.io to pull images before applying the next KubeOne patch releases. This is not included in this patch release but just a notification of future changes.

Important Security Information

Kubernetes releases prior to 1.25.4, 1.24.8, 1.23.14, and 1.22.16 are affected by two Medium CVEs in kube-apiserver: CVE-2022-3162 (Unauthorized read of Custom Resources) and CVE-2022-3294 (Node address isn't always verified when proxying). We strongly recommend upgrading to 1.25.4, 1.24.8, 1.23.14, or 1.22.16 as soon as possible.

Changelog since v1.5.2

Changes by Kind

API Change

  • .cloudProvider.csiConfig is now a mandatory field for vSphere clusters using the external cloud provider (.cloudProvider.external: true). .cloudProvider.csiConfig can be specified even if the in-tree provider is used, but the provided CSIConfig is ignored in such cases (a warning about this is printed) (#2447, @kubermatic-bot)

Feature

  • Add allow_insecure variable (default false) to Terraform configs for vSphere. The value of this variable is propagated to the MachineDeployment template in output.tf (#2449, @xmudrii)
  • Add a new addon parameter called HubbleIPv6 (true/false, default: true) for Cilium CNI used to enable/disable Hubble UI listening on an IPv6 interface (#2451, @kubermatic-bot)
  • Update OpenStack CCM and CSI to v1.24.5 and v1.22.2 (#2445, @xmudrii)
  • Update etcd to 3.5.5 or use the version provided by kubeadm if it's newer (#2443, @kubermatic-bot)

Other (Cleanup or Flake)

  • Expose machine-controller metrics port (8080/TCP), so Prometheus ServiceMonitor can be used for scraping (#2439, @kubermatic-bot)
  • Make volume size for worker nodes configurable in Terraform configs for AWS (50 GB by default) (#2450, @xmudrii)

Chore

  • Rename generate-internal-groups Make target to update-codegen (#2450, @xmudrii)
  • KubeOne is now built using Go 1.19.3 (#2462, @xmudrii)
  • The kubeone-e2e image is moved from Docker Hub to Quay (quay.io/kubermatic/kubeone-e2e) (#2464, @xmudrii)

v1.5.2 - 2022-10-20

Changelog since v1.5.1

Changes by Kind

Feature

Updates

v1.5.1 - 2022-09-26

Changelog since v1.5.0

Changes by Kind

Feature

  • Add a new NodeLocalDNS field to the KubeOneCluster API used to control should the NodeLocalDNSCache component be deployed or not. Run kubeone config print --full for details on how to use this field (#2377, @kron4eg)
  • Upgrade Cilium from v1.12.0 to v1.12.2 (#2376, @ahmedwaleedmalik)

Bug or Regression

  • Automatically delete the CoreDNS PodDistruptionBudget if the feature is disabled (#2365, @xmudrii)
  • Fix NPE when machine-controller deployment is disabled (#2357, @kron4eg)
  • Fix NPE with Operating System Manager (OSM) when the KubeOneCluster v1beta1 API is used (#2357, @kron4eg)
  • Explicitly disable Operating System Manager (OSM) when the KubeOneCluster v1beta1 is used (#2357, @kron4eg)
  • Recreate SSH connection in the case of errors with session (#2357, @kron4eg)
  • Update the kubernetes-cni package from 0.8.7 to 1.1.1 to support the latest Kubernetes patch releases (#2357, @kron4eg)
  • Use vmware-system-csi namespace when generating certs for the vSphere CSI webhooks (#2374, @xmudrii)

v1.5.0 - 2022-08-30

We're happy to announce a new KubeOne minor release — KubeOne 1.5! Please consult the changelog below, as well as, the following two documents before upgrading:

Changelog since v1.4.0

Urgent Upgrade Notes

(No, really, you MUST read this before you upgrade)

  • If you have RHEL-based MachineDeployments on Azure, we strongly recommend upgrading to KubeOne 1.4.8 and rotating those MachineDeployments BEFORE upgrading to KubeOne 1.5. If not done, the Canal CNI update might break the cluster networking when upgrading to KubeOne 1.5. (#2333, @xmudrii)
  • The minimum Kubernetes version has been increased to v1.22.0. If you're still using Kubernetes v1.21 or earlier, you have to upgrade the cluster to v1.22 or newer before upgrading to KubeOne 1.5. (#2236, @xmudrii)
  • Operating System Manager is enabled by default and is responsible for generating and managing user-data used for provisioning worker nodes
  • Automatically apply the node-role.kubernetes.io/control-plane taint to nodes running Kubernetes 1.24. The taint is also applied when upgrading nodes from Kubernetes 1.23 to 1.24. You might need to adjust your workloads to tolerate the node-role.kubernetes.io/control-plane taint (in addition to the node-role.kubernetes.io/master taint). Workloads deployed by KubeOne will be adjusted automatically. (#2019, @xmudrii)
  • Kubeadm is now applying the node-role.kubernetes.io/control-plane label for Kubernetes 1.24 nodes. The old label (node-role.kubernetes.io/master) will be removed when upgrading the cluster to Kubernetes 1.24. All addons are updated to use the node-role.kubernetes.io/control-plane label selector instead. All addons now have toleration for node-role.kubernetes.io/control-plane taint in addition to toleration for node-role.kubernetes.io/master taint. If you are overriding addons, make sure to apply those changes before upgrading to Kubernetes 1.24. (#2017, @xmudrii)
  • workers_replicas variable has been renamed to initial_machinedeployment_replicas in example Terraform configs for Hetzner (#2115, @adeniyistephen)
  • Change default instance size in example Terraform configs for Equinix Metal to c3.small.x86 because t1.small.x86 is not available any longer. If you're using the latest Terraform configs for Equinix Metal with an existing cluster, make sure to explicitly set the instance size (device_type and lb_device_type) in terraform.tfvars or otherwise your instances might get recreated (#2054, @xmudrii)
  • Remove defaulting for Flatcar provisioning utility in example Terraform configs for AWS (defaulted to Ignition by machine-controller). If you have Flatcar-based MachineDeployments that use the cloud-init provisioning utility, you must change the provisioning utility to ignition (or leave it empty) for Operating System Manager (OSM) to work properly (#2285, @xmudrii)
  • Remove the hcloud-volumes StorageClass deployed automatically by Hetzner CSI driver in favor of hcloud-volumes StorageClass deployed by the default-storage-class addon. If you're using hcloud-volumes StorageClass, make sure that you have the default-storage-class addon enabled before upgrading to KubeOne 1.5 (#2269, @xmudrii)
  • Update secret name for backup-restic addon to kubeone-backups-credentials. Manual migration steps are needed for users running KKP on top of a KubeOne installation and using both backup-restic addon from KubeOne and s3-exporter from KKP. Ensure that the s3-credentials Secret with keys ACCESS_KEY_ID and SECRET_ACCESS_KEY exists in kube-system namespace and doesn't have the label kubeone.io/addon:. Remove the label if it exists. Otherwise, s3-exporter won't be functional. (#1880, @ahmedwaleedmalik)

Known Issues

  • Calico VXLAN addon has an issue with broken network connectivity for pods running on the same node. If you're using Calico VXLAN, we recommend staying on KubeOne 1.4 until the issue is not fixed. Follow #2192 for updates.
  • KubeOne is failing to provision a cluster on Flatcar VMs that are upgraded from a version prior to 2969.0.0 to a newer version. This only affects VMs that were never used with KubeOne; existing KubeOne clusters are not affected by this issue. If you're affected by this issue, we recommend creating VMs with newer Flatcar version or following cgroups v2 migration instructions. For more technical details, check the issue #2318.
  • If CoreDNS PodDisruptionBudget is enabled in the KubeOneCluster API, and then disabled, kubeone apply will not remove the PDB object from the cluster; user has to do it manually. This issue will be fixed in the next KubeOne 1.5 patch release (#2322)
  • kubeone apply might fail if the SSH connection is interrupted (e.g. VM is restarted while kubeone apply is running). In this case, it's enough to run kubeone apply again and KubeOne should be able to continue as usual (#2319).

Changes by Kind

API Change

  • Extend KubeOneCluster API with the CoreDNS feature allowing users to configure the number of CoreDNS replicas and whether should KubeOne create a PodDistruptionBudget for CoreDNS. Default values are 2 replicas and create PDB. Run kubeone config print --full for more details
    • Add Pod Anti Affinity to the CoreDNS deployment to avoid having multiple CoreDNS pods on the same node (#2165, @xmudrii)
  • Add MaxPods field to the KubeletConfig used to control the maximum number of pods per node (#2075, @xmudrii)
  • Add machineObjectAnnotations field to DynamicWorkerNodes used to apply annotations to resulting Machine objects Add nodeAnnotations field to DynamicWorkerNodes Config as a replacement for deprecated machineAnnotations field (#2074, @xmudrii)
  • Add new HostConfig.Labels map to manage custom labels on the static worker nodes (#2130, @kron4eg)
  • Allow having no OIDC GroupsPrefix (#1942, @kron4eg)

Deprecation

  • We announced with the KubeOne 1.4.0 release that kubeone install and kubeone upgrade commands are deprecated in favor of kubeone apply. This time we're marking those commands as hidden, so they'll not show in the help output. In the next release, we'll completely remove those commands, so we strongly recommend migrating to kubeone apply as soon as possible. (#2258, @kron4eg)

Feature

General

  • Add support for Rocky Linux operating system (#2121, @ahmedwaleedmalik)
  • Introduce additional safeguards in the KubeOne reconciliation process to disallow upgrading to Kubernetes 1.24 if there are pods that use removed master node-role (node-role.kubernetes.io/master), and if there are Flatcar-based MachineDeployments that use the cloud-init provisioningUtility in a cluster with Operating System Manager (OSM) enabled. (#2290, @xmudrii)
  • Enable the etcd integrity checks (on startup and every 4 hours) for Kubernetes 1.22+ clusters. See the official etcd announcement for more details (https://groups.google.com/a/kubernetes.io/g/dev/c/B7gJs88XtQc/m/rSgNOzV2BwAJ). (#1907, @xmudrii)
  • Add kubeone local subcommand used to provision single-node Kubernetes cluster on current machine (#2125, @kron4eg)
  • Implement the kubeone config dump command used to merge the KubeOneCluster manifest with the Terraform output. The resulting (merged) manifest is printed to stdout. (#1874, @xmudrii)
  • Rollout pods that are using kubeone-*-credentials Secrets if credentials are changed (#2214, @xmudrii)
  • Error reporting in CLI now exists with different codes for different error reasons (#1882, @kron4eg)
  • More error handling with new error types (#1890, @kron4eg)
  • Add dedicated error type (and error code) for exec adapter (#2139, @kron4eg)
  • Strict Terraform output reading (#1833, @kron4eg)
  • --log-format flag is introduced to choose between text and JSON formatted logging (#2060, @ahmedwaleedmalik)
  • [EXPERIMENTAL] Add the KubeOne container image. This image should NOT be used in the production. (#1875, @xmudrii)

Cloud Providers

Addons

  • Add CSI snapshot controller and webhook to the Cinder CSI driver (#2067, @xmudrii)
  • Add missing Snapshot CRDs for Openstack CSI (#1871, @WeirdMachine)
  • Add default VolumeSnapshotClass for OpenStack Cinder CSI (#2217, @xmudrii)
  • Add CSI snapshot controller and webhook to the vSphere CSI driver. Add the default VolumeSnapshotClass for vSphere (#2050, @xmudrii)
  • Add GCP Compute Persistent Disk CSI driver. The CSI driver is deployed by default for all GCE clusters running Kubernetes 1.23 or newer. (#2137, @xmudrii)
  • Add the VMware Cloud Director CSI driver addon. Add default StorageClass for the VMware Cloud Director CSI driver. (#2092, @ahmedwaleedmalik)
  • Add Secrets Store CSI driver and Hashicorp Vault provider as optional addons. See addons' README files for more information on how to activate and use those addons. (#2022, @kron4eg)
  • Add .Params.RequestsCPU parameter to cni-canal addon (#1925, @kron4eg)
  • Create PodDistruptionBudget objects for all Deployments created by KubeOne addons (#1906, @kron4eg)

Updates

Go

etcd

containerd

  • Update containerd to 1.5. Amazon Linux 2 is still using containerd 1.4 because 1.5 is not available. (#2020, @xmudrii)

machine-controller

Operating System Manager (OSM)

CNI

  • Update Canal and Calico VXLAN to v3.23.3. This allows users to use kube-proxy in IPVS mode on ARM64 clusters running Kubernetes 1.23 and newer (#2188, @xmudrii)
  • Update Canal and Calico VXLAN to v3.22.2. This allows users to use kube-proxy in IPVS mode on AMD64 clusters running Kubernetes 1.23 and newer (#2041, @xmudrii)
  • Update Flannel to v0.15.1 to fix an issue with Flannel causing iptables segfaults (#1986, @mfranczy)
  • Switching to quay.io from docker.io for Calico CNI images (#2043, @ahmedwaleedmalik)
  • Update Cilium to v1.12.0 (#2220, @xmudrii)
  • Update Cilium to v1.11.5 (#2049, @xmudrii)

AWS

  • Update AWS CCM to the latest releases for all supported Kubernetes versions. Update AWS EBS CSI driver to v1.9.0 (#2171, @xmudrii)
  • Update AWS CCM to v1.24.0, v1.23.1, v1.22.2, v1.21.1, v1.20.1. Update AWS EBS CSI driver to v1.6.2 (#2055, @xmudrii)

Azure

  • Update Azure CCM to the latest releases for all supported Kubernetes versions. Update AzureDisk CSI driver to v1.21.0. Update AzureFile CSI driver to v1.20.0 (#2172, @xmudrii)
  • Update Azure CCM to v1.24.0, v1.23.11, v1.1.14 (for Kubernetes 1.22), v1.0.18 (for Kubernetes 1.21), v0.7.21 (for Kubernetes 1.20). Update AzureDisk CSI driver to v1.18.0. Update AzureFile CSI driver to v1.18.0 (#2058, @xmudrii)

DigitalOcean

Equinix Metal

Nutanix

GCP

OpenStack

  • Update OpenStack CCM and Cinder CSI to v1.24.2 for Kubernetes 1.24 clusters and v1.23.4 for Kubernetes 1.23 clusters (#2195, @xmudrii)
  • Update OpenStack CCM and Cinder CSI to v1.24.0 for Kubernetes 1.24 clusters (#2061, @xmudrii)

vSphere

  • Update vSphere CSI driver to v2.6.0 (#2169, @xmudrii)
  • Update vSphere CCM to v1.24.0 for Kubernetes 1.24+ clusters. Update vSphere CCM to v1.23.1 for Kubernetes 1.23 clusters (#2169, @xmudrii)
  • Update the vSphere CCM to v1.23.0, v1.22.6, v1.21.3, v1.20.1. Update the vSphere CSI driver to v2.5.1
    • The maximum Kubernetes version for vSphere clusters has been increased from 1.22 to 1.25
    • Apply credentials and cloud-config Secrets before applying addons. This ensures that addons depending on those Secrets are applied properly (#2050, @xmudrii)

Other Addons

  • Update metrics-server to v0.6.1. The listen port for metrics-server has been changed from 443 to 4443. This change shouldn't affect you if you see the metrics-server Service (#2079, @xmudrii)
  • Update NodeLocalDNS Cache to v1.21.1 (#2079, @xmudrii)
  • Update cluster-autoscaler to the latest available releases (#2175, @xmudrii)
  • Update cluster-autoscaler to v1.24.0, v1.23.0, v1.22.2, v1.21.2, v1.20.2 (#2052, @xmudrii)

Terraform Integration

General

  • Automate generating terraform configs README files (#2117, @kron4eg)
  • initial_machinedeployment_operating_system_profile was added to specify operating system profile for initial MachineDeployments. (#2097, @ahmedwaleedmalik)

AWS

  • Rollback to CentOS 7 in Terraform configs for AWS because CentOS 8 reached EOL (#2264, @xmudrii)
  • Introduce initial_machinedeployment_spotinstances_max_price in example Terraform configs for AWS. When set, spot instances will be used for initial MachineDeployments (#1924, @ahmedwaleedmalik)
  • Example Terraform configs for AWS are now using Ignition instead of cloud-init for Flatcar worker nodes (#2157, @ahmedwaleedmalik)
  • Let OSM default the OperatingSystemProfiles (OSPs) in the example Terraform configs for AWS (#2198, @kron4eg)

Azure

  • Introduce a new os variable (defaults to ubuntu) in Terraform configs for Azure to allow choosing an operating system other than Ubuntu (#2266, @xmudrii)
  • Extend example Terraform configs for Azure to automatically subscribe RHEL instances to RHSM (see the PR for more details and instructions on how to opt-out). Important: VMs created by Terraform are NOT automatically unregistered on deletion. You have to manually unregister those VMs by running sudo subscription-manager unregister. The worker nodes created by machine-controller are automatically unregistered as long as the RHSM Offline Token (rhsm_offline_token) is provided. (#2306, @xmudrii)
  • Update Terraform integration for Azure with new fields (#2081, @xmudrii)
  • Update Flatcar to 3227.2.1 in the example Terraform configs for Azure (#2331, @xmudrii)
  • Use the same image reference and plan for the initial Azure MachineDeployment as for the control plane (#2331, @xmudrii)

Other providers

  • Increases default MachineDeployment replicas to 2 for all non-AWS Terraform configs (#2159, @xmudrii)
  • Terraform configs for GCP are now using the default network instead of creating a new one. For production usage, it's recommended to modify configs to create a dedicated network for your cluster. (#2143, @kron4eg)
  • Example Terraform configs for OpenStack are no longer attaching a Floating IP address to the initial MachineDeployment. This matches the behavior of not attaching Floating IP addresses to the control plane nodes. (#2299, @xmudrii)
  • Add vSphere anti-affinity rule for the control plane to avoid a single point of failure. (#2124, @mihiragrawal)

Bug or Regression

General

  • Merge the CCM/CSI migration steps for updating the control plane static pod manifests and Kubelet configuration into a single step. This fixes an issue with the CCM/CSI migration failing on clusters running Kubernetes 1.24+ when the API endpoint is one of the control plane nodes. (#2326, @xmudrii)
  • Enable nf_conntrack (nf_conntrack_ipv4) module by default on all operating systems. This fixes an issue with pods unable to reach services running on a host on operating systems that are using the NFT backend. (#2282, @xmudrii)
  • Explicitly create /opt/bin on Flatcar before trying to untar anything to that directory (#2302, @xmudrii)
  • Set rp_filter=0 on all interfaces when Cilium is used. This fixes an issue with Cilium clusters losing pod connectivity after upgrading the cluster (#2089, @xmudrii)
  • Approve pending CSRs when upgrading control plane and static worker nodes (#1887, @xmudrii)
  • Force regenerating CSRs for Kubelet serving certificates after CCM is deployed. This fixes an issue with Kubelet generating CSRs that are stuck in Pending. (#2199, @xmudrii)
  • Fix CSR approving issue for existing nodes with already approved and GCed CSRs (#1894, @kron4eg)
  • Fix wrong maxPods value on follower control plane nodes and static worker nodes (#2112, @xmudrii)
  • Fix KubeletConfiguration and KubeProxyConfiguration for Kubernetes prior v1.23.x (#2138, @kron4eg)
  • Fix missing reading of the static workers defined in Terraform (#2015, @kron4eg)
  • Fix containerd upgrade on Debian-based distros (#1930, @kron4eg)
  • Fix NPE on SSH connection close (#2154, @kron4eg)
  • Fix the GoBetween script failing to install the zip package on Flatcar Linux (#1904, @xmudrii)
  • Fix issue with installer.sh on mac (BSD sed) (#2161, @dermorz)
  • Fix "latest version" in install.sh. (#1949, @dermorz)
  • Fix an issue with kubeone config migrate failing to migrate configs with the containerRuntime block (#1860, @xmudrii)
  • Fix overwriteRegistry not overwriting the Kubernetes control plane images (#1884, @xmudrii)
  • Fix pre-pull images (#2029, @kron4eg)
  • Use kubeadm config when pre-pulling images (#2026, @kron4eg)
  • Add missing volumeattachments permissions to machine-controller (#2031, @kron4eg)
  • Avoid creating and validating MC credentials when MC is disabled (#1939, @kron4eg)
  • Ensure old machine-controller MutatingWebhookConfiguration is deleted (#1900, @kron4eg)
  • Escape docker/containerd versions to avoid wildcard matching (#1941, @kron4eg)
  • Expand path to SSH private key file (#1849, @ahmedwaleedmalik)
  • Add missing systemctl daemon-reload when removing binaries (#2064, @kron4eg)
  • Regenerate container runtime configurations based on KubeOneCluster manifest during control plane upgrades on Flatcar Linux nodes, not only on the initial installation. (#1910, @dermorz)
  • Remove the --network-plugin Kubelet flag when migrating from Docker to containerd and when upgrading from Kubernetes 1.23.x to 1.24.x (#2024, @xmudrii)
  • Restart kubelet after upgrading containerd (#1944, @kron4eg)
  • Update kubeadm-flags.env file when upgrading static worker nodes (#2123, @xmudrii)
  • Don't ignore clientset error when resetting cluster (#1950, @xmudrii)
  • Show "Ensure MachineDeployments" as an action to be taken only when provisioning a cluster for the first time (#1927, @xmudrii)
  • Lower exponential backoff times (#2231, @kron4eg)

Addons

  • Set iptables backend (FELIX_IPTABLESBACKEND) to NFT for Canal and Calico VXLAN on clusters running Flatcar Linux and RHEL. For non Flatcar/RHEL clusters, iptables backend is set to Auto, which is the default value and results in Calico determining the iptables backend automatically. The value can be overridden by setting the iptablesBackend addon parameter (see the PR description for an example). (#2331, #2301, @xmudrii)
  • Move the vSphere CSI driver to vmware-system-csi namespace to fix a bug where the CSI driver requires to run in its dedicated namespace (#2292, @WeirdMachine)
  • Properly propagate external cloud provider and CSI migration options to OSM (#2202, @xmudrii)
  • Replace operator: Exists toleration with the control plane tolerations for metrics-server. This fixes an issue with metrics-server pods breaking eviction (#2205, @xmudrii)
  • Fix the logic for determining if the CSI driver is deployed in the default-storage-class addon. This fixes an issue with deploying the default-storage-class addon on vSphere clusters using the in-tree cloud provider (#2167, @xmudrii)
  • Azure: Migrate AzureDisk CSIDriver to set fsGroupPolicy to File (#2082, @xmudrii)
  • Azure: Disable --configure-cloud-routes on Azure CCM to fix errors when starting the CCM (#2184, @xmudrii)
  • Azure: Disable node IPAM in Azure CCM (#2106, @rastislavs)
  • GCE: Migrate GCE standard default StorageClass to set volumeBindingMode to WaitForFirstConsumer. The StorageClass will be automatically recreated the next time you run kubeone apply (#2142, @xmudrii)
  • Hetzner: Disable Node IPAM in Hetzner CCM. This fixes network connectivity issues on the worker nodes. (#2200, @xmudrii)
  • OpenStack: Tenant ID or Tenant Name is not required when using application credentials (#2196, @ahmedwaleedmalik)
  • OpenStack: Mount /usr/share/ca-certificates to the OpenStack CCM pod to fix the OpenStack CCM pod CrashLooping on Flatcar Linux (#1904, @xmudrii)
  • Mount /etc/pki to the Azure CCM container to fix CrashLoopBackoff on clusters running CentOS 7 and Rocky Linux (#2308, @xmudrii)
  • Mount /usr/share/ca-certificates to the Azure CCM container to fix CrashLoopBackoff on clusters running Flatcar (#2331, @xmudrii)
  • Mount /etc/pki to the OpenStack CCM container to fix CrashLoopBackoff on clusters running CentOS 7 (#2299, @xmudrii)
  • Fix Rocky Linux OS detection (#2267, @kron4eg)
  • Disable preserveUnknownFields in all Canal CRDs. This fixes an issue preventing upgrading Canal to v3.22 for KubeOne clusters created with KubeOne 1.2 and older (#2103, @xmudrii)

Other

v1.5.0-rc.0 - 2022-08-25

Changelog since v1.5.0-beta.0

Urgent Upgrade Notes

(No, really, you MUST read this before you upgrade)

  • The minimum Kubernetes version has been increased to v1.22.0. If you're still using Kubernetes v1.21 or v1.20, you have to upgrade the cluster to v1.22 or newer before upgrading to KubeOne 1.5. (#2236, @xmudrii)
  • Remove defaulting for Flatcar provisioning utility in example Terraform configs for AWS (defaulted to Ignition by machine-controller). If you have Flatcar-based MachineDeployments that use the cloud-init provisioning utility, you must change the provisioning utility to ignition (or leave it empty) for Operating System Manager (OSM) to work properly (#2285, @xmudrii)
  • Remove the hcloud-volumes StorageClass deployed automatically by Hetzner CSI driver in favor of hcloud-volumes StorageClass deployed by the default-storage-class addon. If you're using hcloud-volumes StorageClass, make sure that you have the default-storage-class addon enabled before upgrading to KubeOne 1.5 (#2269, @xmudrii)

Known Issues

  • Calico VXLAN addon has an issue with broken network connectivity for pods running on the same node. If you're using Calico VXLAN, we recommend staying on KubeOne 1.4 until the issue is not fixed. Follow #2192 for updates.

Changes by Kind

Deprecation

  • We announced with the KubeOne 1.4.0 release that kubeone install and kubeone upgrade commands are deprecated in favor of kubeone apply. This time we're marking those commands as hidden, so they'll not show in the help output. In the next release, we'll completely remove those commands, so we strongly recommend migrating to kubeone apply as soon as possible. (#2258, @kron4eg)

Feature

General

  • Introduce additional safeguards in the KubeOne reconciliation process to disallow upgrading to Kubernetes 1.24 if there are pods that use removed master node-role (node-role.kubernetes.io/master), and if there are Flatcar-based MachineDeployments that use the cloud-init provisioningUtility in a cluster with Operating System Manager (OSM) enabled. (#2290, @xmudrii)

Updates

machine-controller

Operating System Manager (OSM)

Terraform Integration

AWS

  • Rollback to CentOS 7 in Terraform configs for AWS because CentOS 8 reached EOL (#2264, @xmudrii)

Azure

  • Introduce a new os variable (defaults to ubuntu) in Terraform configs for Azure to allow choosing an operating system other than Ubuntu (#2266, @xmudrii)
  • Extend example Terraform configs for Azure to automatically subscribe RHEL instances to RHSM (see the PR for more details and instructions on how to opt-out). Important: VMs created by Terraform are NOT automatically unregistered on deletion. You have to manually unregister those VMs by running sudo subscription-manager unregister. The worker nodes created by machine-controller are automatically unregistered as long as the RHSM Offline Token (rhsm_offline_token) is provided. (#2306, @xmudrii)

OpenStack

  • Example Terraform configs for OpenStack are no longer attaching a Floating IP address to the initial MachineDeployment. This matches the behavior of not attaching Floating IP addresses to the control plane nodes. (#2299, @xmudrii)

Bug or Regression

  • Enable nf_conntrack (nf_conntrack_ipv4) module by default on all operating systems. This fixes an issue with pods unable to reach services running on a host on operating systems that are using the NFT backend. (#2282, @xmudrii)
  • Set iptables backend (FELIX_IPTABLESBACKEND) to NFT for Canal and Calico VXLAN on clusters running Flatcar Linux. For non Flatcar clusters, iptables backend is set to Auto, which is the default value and results in Calico determining the iptables backend automatically. The value can be overridden by setting the iptablesBackend addon parameter (see the PR description for an example). (#2301, @xmudrii)
  • Explicitly create /opt/bin on Flatcar before trying to untar anything to that directory (#2302, @xmudrii)
  • Move the vSphere CSI driver to vmware-system-csi namespace to fix a bug where the CSI driver requires to run in its dedicated namespace (#2292, @WeirdMachine)
  • Mount /etc/pki to the Azure CCM container to fix CrashLoopBackoff on clusters running CentOS 7 and Rocky Linux (#2308, @xmudrii)
  • Mount /etc/pki to the OpenStack CCM container to fix CrashLoopBackoff on clusters running CentOS 7 (#2299, @xmudrii)
  • Fix Rocky Linux OS detection (#2267, @kron4eg)

v1.5.0-beta.0 - 2022-08-04

Changelog since v1.4.0

Urgent Upgrade Notes

(No, really, you MUST read this before you upgrade)

  • Automatically apply the node-role.kubernetes.io/control-plane taint to nodes running Kubernetes 1.24. The taint is also applied when upgrading nodes from Kubernetes 1.23 to 1.24. You might need to adjust your workloads to tolerate the node-role.kubernetes.io/control-plane taint (in addition to the node-role.kubernetes.io/master taint). Workloads deployed by KubeOne will be adjusted automatically. (#2019, @xmudrii)
  • Kubeadm is now applying the node-role.kubernetes.io/control-plane label for Kubernetes 1.24 nodes. The old label (node-role.kubernetes.io/master) will be removed when upgrading the cluster to Kubernetes 1.24. All addons are updated to use the node-role.kubernetes.io/control-plane label selector instead. All addons now have toleration for node-role.kubernetes.io/control-plane taint in addition to toleration for node-role.kubernetes.io/master taint. If you are overriding addons, make sure to apply those changes before upgrading to Kubernetes 1.24. (#2017, @xmudrii)
  • Operating System Manager is enabled by default and is responsible for generating and managing user-data used for provisioning worker nodes
    • Existing worker machines will not be migrated to use OSM automatically. The user needs to manually rollout all MachineDeployments to start using OSM. This can be done by following the steps described in Rolling Restart MachineDeploments document
    • The user can opt-out from OSM by setting .operatingSystemManager.deploy to false in their KubeOneCluster manifest. (#2157, @ahmedwaleedmalik)
  • workers_replicas variable has been renamed to initial_machinedeployment_replicas in example Terraform configs for Hetzner (#2115, @adeniyistephen)
  • Change default instance size in example Terraform configs for Equinix Metal to c3.small.x86 because t1.small.x86 is not available any longer. If you're using the latest Terraform configs for Equinix Metal with an existing cluster, make sure to explicitly set the instance size (device_type and lb_device_type) in terraform.tfvars or otherwise your instances might get recreated (#2054, @xmudrii)
  • Update secret name for backup-restic addon to kubeone-backups-credentials. Manual migration steps are needed for users running KKP on top of a KubeOne installation and using both backup-restic addon from KubeOne and s3-exporter from KKP. Ensure that the s3-credentials Secret with keys ACCESS_KEY_ID and SECRET_ACCESS_KEY exists in kube-system namespace and doesn't have the label kubeone.io/addon:. Remove the label if it exists. Otherwise, s3-exporter won't be functional. (#1880, @ahmedwaleedmalik)

Known Issues

  • Calico VXLAN addon has an issue with broken network connectivity for pods running on the same node. If you're using Calico VXLAN, we recommend staying on KubeOne 1.4 until the issue is not fixed. Follow #2192 for updates.

Changes by Kind

API Change

  • Extend KubeOneCluster API with the CoreDNS feature allowing users to configure the number of CoreDNS replicas and whether should KubeOne create a PodDistruptionBudget for CoreDNS. Default values are 2 replicas and create PDB. Run kubeone config print --full for more details
    • Add Pod Anti Affinity to the CoreDNS deployment to avoid having multiple CoreDNS pods on the same node (#2165, @xmudrii)
  • Add MaxPods field to the KubeletConfig used to control the maximum number of pods per node (#2075, @xmudrii)
  • Add machineObjectAnnotations field to DynamicWorkerNodes used to apply annotations to resulting Machine objects Add nodeAnnotations field to DynamicWorkerNodes Config as a replacement for deprecated machineAnnotations field (#2074, @xmudrii)
  • Add new HostConfig.Labels map to manage custom labels on the static worker nodes (#2130, @kron4eg)
  • Allow having no OIDC GroupsPrefix (#1942, @kron4eg)

Feature

General

  • Add support for Rocky Linux operating system (#2121, @ahmedwaleedmalik)
  • Enable the etcd integrity checks (on startup and every 4 hours) for Kubernetes 1.22+ clusters. See the official etcd announcement for more details (https://groups.google.com/a/kubernetes.io/g/dev/c/B7gJs88XtQc/m/rSgNOzV2BwAJ). (#1907, @xmudrii)
  • Add kubeone local subcommand used to provision single-node Kubernetes cluster on current machine (#2125, @kron4eg)
  • Implement the kubeone config dump command used to merge the KubeOneCluster manifest with the Terraform output. The resulting (merged) manifest is printed to stdout. (#1874, @xmudrii)
  • Rollout pods that are using kubeone-*-credentials Secrets if credentials are changed (#2214, @xmudrii)
  • Error reporting in CLI now exists with different codes for different error reasons (#1882, @kron4eg)
  • More error handling with new error types (#1890, @kron4eg)
  • Add dedicated error type (and error code) for exec adapter (#2139, @kron4eg)
  • Strict Terraform output reading (#1833, @kron4eg)
  • --log-format flag is introduced to choose between text and JSON formatted logging (#2060, @ahmedwaleedmalik)
  • [EXPERIMENTAL] Add the KubeOne container image. This image should NOT be used in the production. (#1875, @xmudrii)

Cloud Providers

Addons

  • Add CSI snapshot controller and webhook to the Cinder CSI driver (#2067, @xmudrii)
  • Add missing Snapshot CRDs for Openstack CSI (#1871, @WeirdMachine)
  • Add default VolumeSnapshotClass for OpenStack Cinder CSI (#2217, @xmudrii)
  • Add CSI snapshot controller and webhook to the vSphere CSI driver. Add the default VolumeSnapshotClass for vSphere (#2050, @xmudrii)
  • Add GCP Compute Persistent Disk CSI driver. The CSI driver is deployed by default for all GCE clusters running Kubernetes 1.23 or newer. (#2137, @xmudrii)
  • Add the VMware Cloud Director CSI driver addon. Add default StorageClass for the VMware Cloud Director CSI driver. (#2092, @ahmedwaleedmalik)
  • Add Secrets Store CSI driver and Hashicorp Vault provider as optional addons. See addons' README files for more information on how to activate and use those addons. (#2022, @kron4eg)
  • Add .Params.RequestsCPU parameter to cni-canal addon (#1925, @kron4eg)
  • Create PodDistruptionBudget objects for all Deployments created by KubeOne addons (#1906, @kron4eg)

Updates

Go

etcd

containerd

  • Update containerd to 1.5. Amazon Linux 2 is still using containerd 1.4 because 1.5 is not available. (#2020, @xmudrii)

machine-controller

Operating System Manager (OSM)

CNI

  • Update Canal and Calico VXLAN to v3.23.3. This allows users to use kube-proxy in IPVS mode on ARM64 clusters running Kubernetes 1.23 and newer (#2188, @xmudrii)
  • Update Canal and Calico VXLAN to v3.22.2. This allows users to use kube-proxy in IPVS mode on AMD64 clusters running Kubernetes 1.23 and newer (#2041, @xmudrii)
  • Update Flannel to v0.15.1 to fix an issue with Flannel causing iptables segfaults (#1986, @mfranczy)
  • Switching to quay.io from docker.io for Calico CNI images (#2043, @ahmedwaleedmalik)
  • Update Cilium to v1.12.0 (#2220, @xmudrii)
  • Update Cilium to v1.11.5 (#2049, @xmudrii)

AWS

  • Update AWS CCM to the latest releases for all supported Kubernetes versions. Update AWS EBS CSI driver to v1.9.0 (#2171, @xmudrii)
  • Update AWS CCM to v1.24.0, v1.23.1, v1.22.2, v1.21.1, v1.20.1. Update AWS EBS CSI driver to v1.6.2 (#2055, @xmudrii)

Azure

  • Update Azure CCM to the latest releases for all supported Kubernetes versions. Update AzureDisk CSI driver to v1.21.0. Update AzureFile CSI driver to v1.20.0 (#2172, @xmudrii)
  • Update Azure CCM to v1.24.0, v1.23.11, v1.1.14 (for Kubernetes 1.22), v1.0.18 (for Kubernetes 1.21), v0.7.21 (for Kubernetes 1.20). Update AzureDisk CSI driver to v1.18.0. Update AzureFile CSI driver to v1.18.0 (#2058, @xmudrii)

DigitalOcean

Equinix Metal

Nutanix

GCP

OpenStack

  • Update OpenStack CCM and Cinder CSI to v1.24.2 for Kubernetes 1.24 clusters and v1.23.4 for Kubernetes 1.23 clusters (#2195, @xmudrii)
  • Update OpenStack CCM and Cinder CSI to v1.24.0 for Kubernetes 1.24 clusters (#2061, @xmudrii)

vSphere

  • Update vSphere CSI driver to v2.6.0 (#2169, @xmudrii)
  • Update vSphere CCM to v1.24.0 for Kubernetes 1.24+ clusters. Update vSphere CCM to v1.23.1 for Kubernetes 1.23 clusters (#2169, @xmudrii)
  • Update the vSphere CCM to v1.23.0, v1.22.6, v1.21.3, v1.20.1. Update the vSphere CSI driver to v2.5.1
    • The maximum Kubernetes version for vSphere clusters has been increased from 1.22 to 1.25
    • Apply credentials and cloud-config Secrets before applying addons. This ensures that addons depending on those Secrets are applied properly (#2050, @xmudrii)

Other Addons

  • Update metrics-server to v0.6.1. The listen port for metrics-server has been changed from 443 to 4443. This change shouldn't affect you if you see the metrics-server Service (#2079, @xmudrii)
  • Update NodeLocalDNS Cache to v1.21.1 (#2079, @xmudrii)
  • Update cluster-autoscaler to the latest available releases (#2175, @xmudrii)
  • Update cluster-autoscaler to v1.24.0, v1.23.0, v1.22.2, v1.21.2, v1.20.2 (#2052, @xmudrii)

Terraform Integration

General

  • Automate generating terraform configs README files (#2117, @kron4eg)
  • initial_machinedeployment_operating_system_profile was added to specify operating system profile for initial MachineDeployments. (#2097, @ahmedwaleedmalik)

AWS

  • Introduce initial_machinedeployment_spotinstances_max_price in example Terraform configs for AWS. When set, spot instances will be used for initial MachineDeployments (#1924, @ahmedwaleedmalik)
  • Example Terraform configs for AWS are now using Ignition instead of cloud-init for Flatcar worker nodes (#2157, @ahmedwaleedmalik)
  • Let OSM default the OperatingSystemProfiles (OSPs) in the example Terraform configs for AWS (#2198, @kron4eg)

Other providers

  • Increases default MachineDeployment replicas to 2 for all non-AWS Terraform configs (#2159, @xmudrii)
  • Update Terraform integration for Azure with new fields (#2081, @xmudrii)
  • Terraform configs for GCP are now using the default network instead of creating a new one. For production usage, it's recommended to modify configs to create a dedicated network for your cluster. (#2143, @kron4eg)
  • Add vSphere anti-affinity rule for the control plane to avoid a single point of failure. (#2124, @mihiragrawal)

Bug or Regression

General

  • Set rp_filter=0 on all interfaces when Cilium is used. This fixes an issue with Cilium clusters losing pod connectivity after upgrading the cluster (#2089, @xmudrii)
  • Approve pending CSRs when upgrading control plane and static worker nodes (#1887, @xmudrii)
  • Force regenerating CSRs for Kubelet serving certificates after CCM is deployed. This fixes an issue with Kubelet generating CSRs that are stuck in Pending. (#2199, @xmudrii)
  • Fix CSR approving issue for existing nodes with already approved and GCed CSRs (#1894, @kron4eg)
  • Fix wrong maxPods value on follower control plane nodes and static worker nodes (#2112, @xmudrii)
  • Fix KubeletConfiguration and KubeProxyConfiguration for Kubernetes prior v1.23.x (#2138, @kron4eg)
  • Fix missing reading of the static workers defined in Terraform (#2015, @kron4eg)
  • Fix containerd upgrade on Debian-based distros (#1930, @kron4eg)
  • Fix NPE on SSH connection close (#2154, @kron4eg)
  • Fix the GoBetween script failing to install the zip package on Flatcar Linux (#1904, @xmudrii)
  • Fix issue with installer.sh on mac (BSD sed) (#2161, @dermorz)
  • Fix "latest version" in install.sh. (#1949, @dermorz)
  • Fix an issue with kubeone config migrate failing to migrate configs with the containerRuntime block (#1860, @xmudrii)
  • Fix overwriteRegistry not overwriting the Kubernetes control plane images (#1884, @xmudrii)
  • Fix pre-pull images (#2029, @kron4eg)
  • Use kubeadm config when pre-pulling images (#2026, @kron4eg)
  • Add missing volumeattachments permissions to machine-controller (#2031, @kron4eg)
  • Avoid creating and validating MC credentials when MC is disabled (#1939, @kron4eg)
  • Ensure old machine-controller MutatingWebhookConfiguration is deleted (#1900, @kron4eg)
  • Escape docker/containerd versions to avoid wildcard matching (#1941, @kron4eg)
  • Expand path to SSH private key file (#1849, @ahmedwaleedmalik)
  • Add missing systemctl daemon-reload when removing binaries (#2064, @kron4eg)
  • Regenerate container runtime configurations based on KubeOneCluster manifest during control plane upgrades on Flatcar Linux nodes, not only on the initial installation. (#1910, @dermorz)
  • Remove the --network-plugin Kubelet flag when migrating from Docker to containerd and when upgrading from Kubernetes 1.23.x to 1.24.x (#2024, @xmudrii)
  • Restart kubelet after upgrading containerd (#1944, @kron4eg)
  • Update kubeadm-flags.env file when upgrading static worker nodes (#2123, @xmudrii)
  • Don't ignore clientset error when resetting cluster (#1950, @xmudrii)
  • Show "Ensure MachineDeployments" as an action to be taken only when provisioning a cluster for the first time (#1927, @xmudrii)
  • Lower exponential backoff times (#2231, @kron4eg)

Addons

  • Properly propagate external cloud provider and CSI migration options to OSM (#2202, @xmudrii)
  • Replace operator: Exists toleration with the control plane tolerations for metrics-server. This fixes an issue with metrics-server pods breaking eviction (#2205, @xmudrii)
  • Fix the logic for determining if the CSI driver is deployed in the default-storage-class addon. This fixes an issue with deploying the default-storage-class addon on vSphere clusters using the in-tree cloud provider (#2167, @xmudrii)
  • Azure: Migrate AzureDisk CSIDriver to set fsGroupPolicy to File (#2082, @xmudrii)
  • Azure: Disable --configure-cloud-routes on Azure CCM to fix errors when starting the CCM (#2184, @xmudrii)
  • Azure: Disable node IPAM in Azure CCM (#2106, @rastislavs)
  • GCE: Migrate GCE standard default StorageClass to set volumeBindingMode to WaitForFirstConsumer. The StorageClass will be automatically recreated the next time you run kubeone apply (#2142, @xmudrii)
  • Hetzner: Disable Node IPAM in Hetzner CCM. This fixes network connectivity issues on the worker nodes. (#2200, @xmudrii)
  • OpenStack: Tenant ID or Tenant Name is not required when using application credentials (#2196, @ahmedwaleedmalik)
  • OpenStack: Mount /usr/share/ca-certificates to the OpenStack CCM pod to fix the OpenStack CCM pod CrashLooping on Flatcar Linux (#1904, @xmudrii)
  • Disable preserveUnknownFields in all Canal CRDs. This fixes an issue preventing upgrading Canal to v3.22 for KubeOne clusters created with KubeOne 1.2 and older (#2103, @xmudrii)

Other