✨ Add bootCommands to cloud-init file generation #11271

davidumea · 2024-10-08T14:17:23Z

What this PR does / why we need it:
Adds the ability to provide bootcmd commands to cloud-init via the KubeadmConfig or KubeadmConfigTemplates custom resources. (Same for KubeadmControlPlane)

One thing that I know that does not work at the time of creating this PR is validating/mutating webhooks for KubeadmConfig and KubeadmConfigTemplates. Some pointers to where to change that would be appreciated.

Which issue(s) this PR fixes
Fixes #

k8s-ci-robot · 2024-10-08T14:17:32Z

Welcome @davidumea!

It looks like this is your first PR to kubernetes-sigs/cluster-api 🎉. Please refer to our pull request process documentation to help your PR have a smooth ride to approval.

You will be prompted by a bot to use commands during the review process. Do not be afraid to follow the prompts! It is okay to experiment. Here is the bot commands documentation.

You can also check if kubernetes-sigs/cluster-api has its own contribution guidelines.

You may want to refer to our testing guide if you run into trouble with your tests not passing.

If you are having difficulty getting your pull request seen, please follow the recommended escalation practices. Also, for tips and tricks in the contribution process you may want to read the Kubernetes contributor cheat sheet. We want to make sure your contribution gets all the attention it needs!

Thank you, and welcome to Kubernetes. 😃

k8s-ci-robot · 2024-10-08T14:17:33Z

Hi @davidumea. Thanks for your PR.

I'm waiting for a kubernetes-sigs member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

chrischdi · 2024-10-08T14:21:48Z

Are there version constraints around bootcmd? Does it already exist in cloud-init for a long time?

What happens if we render it but the cloud-init version in the machine does not support it? Does it make cloud-init fail as a whole or is it just ignored?

Edit: looks like it is already there for a long time (from the start?) at 0.7.7 https://cloudinit.readthedocs.io/en/0.7.7/topics/examples.html

davidumea · 2024-10-09T12:28:47Z

Are there version constraints around bootcmd? Does it already exist in cloud-init for a long time?

Edit: looks like it is already there for a long time (from the start?) at 0.7.7 https://cloudinit.readthedocs.io/en/0.7.7/topics/examples.html

Exactly I don't think that should be an issue as bootcmd has been part of cloud-init for a long time.

What happens if we render it but the cloud-init version in the machine does not support it? Does it make cloud-init fail as a whole or is it just ignored?

During my testing, if the bootcmd module fails e.g. due to invalid commands, cloud-init still continues with the following modules. I haven't been able to test running this with a cloud-init version that doesn't support bootcmd.

davidumea · 2024-10-17T12:30:16Z

@chrischdi Is there anything I can do to get this reviewed?

fabriziopandini · 2024-10-18T08:37:00Z

/ok-to-test

cc @enxebre @vincepri @randomvariable

randomvariable

Bootcmd module has been in cloud-init for as long as I can remember (which is back to 2010), so don't see an issue there.

I would like the documentation to be more explicit though, particularly about how this will have no effect on Ignition. Have suggested some changes.

bootstrap/kubeadm/api/v1beta1/kubeadmconfig_types.go

neolit123 · 2024-10-18T15:05:09Z

internal/apis/bootstrap/kubeadm/v1alpha3/kubeadmconfig_types.go

+	// BootCommands specifies extra commands to run very early in the boot process via the cloud-init bootcmd
+	// module. This is typically run in the cloud-init.service systemd unit. This has no effect in Ignition.
+	// +optional
+	BootCommands []BootCommand `json:"bootCommands,omitempty"`


shouldn't it be added only to v1beta1 which is the actively maintained API version?

If I remove this and run make verify, it removes bootCommands from the CRDs.

Might be an issue in the verify logic, what do you think?

diff --git a/bootstrap/kubeadm/config/crd/bases/bootstrap.cluster.x-k8s.io_kubeadmconfigs.yaml b/bootstrap/kubeadm/config/crd/bases/bootstrap.cluster.x-k8s.io_kubeadmconfigs.yaml index 79c3b19e0..4e1852f77 100644 --- a/bootstrap/kubeadm/config/crd/bases/bootstrap.cluster.x-k8s.io_kubeadmconfigs.yaml +++ b/bootstrap/kubeadm/config/crd/bases/bootstrap.cluster.x-k8s.io_kubeadmconfigs.yaml @@ -47,17 +47,6 @@ spec: KubeadmConfigSpec defines the desired state of KubeadmConfig. Either ClusterConfiguration and InitConfiguration should be defined or the JoinConfiguration should be defined. properties: - bootCommands: - description: |- - BootCommands specifies extra commands to run very early in the boot process via the cloud-init bootcmd - module. This is typically run in the cloud-init.service systemd unit. This has no effect in Ignition. - items: - description: BootCommand defines input for each bootcmd command - in cloud-init. - items: - type: string - type: array - type: array clusterConfiguration: description: ClusterConfiguration along with InitConfiguration are the configurations necessary for the init command

etc

New fields should not be added to older API versions.
Conversions should be implemented instead.
See eg

cluster-api/internal/apis/bootstrap/kubeadm/v1alpha3/conversion.go

Lines 47 to 56 in 3db18fb

dst.Files = restored.Files

dst.Users = restored.Users

if restored.Users != nil {

for i := range restored.Users {

if restored.Users[i].PasswdFrom != nil {

dst.Users[i].PasswdFrom = restored.Users[i].PasswdFrom

}

}

}

I reverted my changes to the v1alpha3 and v1alpha4 versions and added conversions instead a9c426a

If I remove this and run make verify, it removes bootCommands from the CRDs.

Disregard this, I didn't realize that all versions of the crds were in the same manifest.

fabriziopandini · 2024-10-21T08:36:03Z

internal/apis/bootstrap/kubeadm/v1alpha3/kubeadmconfig_types.go

+	// BootCommands specifies extra commands to run very early in the boot process via the cloud-init bootcmd
+	// module. This is typically run in the cloud-init.service systemd unit. This has no effect in Ignition.
+	// +optional
+	BootCommands []BootCommand `json:"bootCommands,omitempty"`


New fields should not be added to older API versions.
Conversions should be implemented instead.
See eg

cluster-api/internal/apis/bootstrap/kubeadm/v1alpha3/conversion.go

Lines 47 to 56 in 3db18fb

dst.Files = restored.Files

dst.Users = restored.Users

if restored.Users != nil {

for i := range restored.Users {

if restored.Users[i].PasswdFrom != nil {

dst.Users[i].PasswdFrom = restored.Users[i].PasswdFrom

}

}

}

fabriziopandini · 2024-10-21T08:37:14Z

bootstrap/kubeadm/internal/cloudinit/cloudinit_test.go

@@ -34,6 +34,7 @@ func TestNewInitControlPlaneAdditionalFileEncodings(t *testing.T) {
 	cpinput := &ControlPlaneInput{
 		BaseUserData: BaseUserData{
 			Header:              "test",
+			BootCommands:        nil,


Can we have a test for BootCommands not nil for init, join CP, join workers?

I added non-nil values for BootCommands in TestNewInitControlPlaneCommands and TestNewJoinControlPlaneAdditionalFileEncodings, did you want me to add more tests or is this enough? I couldn't find one for "join workers" daf4f97

WRT to changes in TestNewInitControlPlaneCommands, TestNewJoinControlPlaneAdditionalFileEncodings I have dedicated comments.

We still need test coverage for join and join CP (if they are missing, kindly add them so we pay down some tech debt before adding more on top)

fabriziopandini · 2024-10-21T08:39:02Z

controlplane/kubeadm/internal/filters_test.go

@@ -548,7 +548,7 @@ func TestMatchInitOrJoinConfiguration(t *testing.T) {
    },
    JoinConfiguration: nil,
    Files:             nil,
-    ... // 10 identical fields
+    ... // 11 identical fields


I need to take some time to dig into impact of this change on existing clusters, because we need to make sure this change does not trigger rollouts
However, I'm not sure when I will get to it 😓

Do you think it's feasible to have this included in the next minor release?

Sorry if I missed this notification before.
Unfortunately code freeze is tomorrow, but let's first focus on getting the PR right first, then we can think about what options do we have

I took some time to look into this, and it should not trigger a rollout even if message in the logs could change after we deploy this change.

@chrischdi could you kindly double check this assumption

Should be fine. Especially clusterctl upgrade tests should catch this if it happens and we are wrong.

Edit: sorry for the long round-trip, forgot to send the review / comment... (was pending)

… PostKubeadmCommands

…is used

anmazzotti

All looks good to me, beside the minor issue of not using the constant "cloud-config"

bootstrap/kubeadm/internal/webhooks/kubeadmconfig.go

bootstrap/kubeadm/internal/webhooks/kubeadmconfig_test.go

Danil-Grigorev · 2025-01-24T14:39:43Z

bootstrap/kubeadm/internal/webhooks/kubeadmconfig.go

@@ -65,6 +65,9 @@ func (webhook *KubeadmConfig) ValidateCreate(_ context.Context, obj runtime.Obje
 	if !ok {
 		return nil, apierrors.NewBadRequest(fmt.Sprintf("expected a KubeadmConfig but got a %T", obj))
 	}
+	if c.Spec.BootCommands != nil && c.Spec.Format != bootstrapv1.CloudConfig {


It seems this commands need to be a part of webhook.validate, otherwise the check looks fine.

I had to update the validateIgnition function so that allErrs wouldn't be empty, which was probably for the best, thanks

3db48fa

I guess the webhook check isn't needed anymore then? 🤔

@Danil-Grigorev Please take a look if you have time 🙂

Danil-Grigorev · 2025-02-05T14:03:05Z

/lgtm

Seems good to me now, thank you

k8s-ci-robot · 2025-02-05T14:03:12Z

LGTM label has been added.

Git tree hash: cfb7529d6229e954dee72fe86ec8bc891b33ffb2

davidumea · 2025-02-17T09:39:32Z

/assign justinsb

Pinging @justinsb for approval

chrischdi · 2025-02-18T16:29:29Z

bootstrap/kubeadm/internal/ignition/clc/clc_test.go

+				BootCommands: []bootstrapv1.BootCommand{
+					{
+						"boot-command", "another-boot-command",
+					},
+				},


This does not make sense to get set right? Because ignition does not have bootCommands and we block this getting set?

(also below?)

You're right, it doesn't make sense to have that there.

chrischdi · 2025-02-18T16:31:07Z

bootstrap/kubeadm/internal/cloudinit/cloudinit_test.go

@@ -94,9 +95,12 @@ func TestNewInitControlPlaneCommands(t *testing.T) {

 	cpinput := &ControlPlaneInput{
 		BaseUserData: BaseUserData{
+			BootCommands: []bootstrapv1.BootCommand{
+				{"echo", "$(date) hello BootCommands!"},


Question: can't this be the same as below?

Suggested change

{"echo", "$(date) hello BootCommands!"},

{`"echo $(date) ': hello BootCommands!'"`},

I wanted to test the output when having multiple commands, but it didn't make sense to split it there.

This should be clearer 1874ba8

chrischdi

Two small things, otherwise lgtm

k8s-ci-robot · 2025-02-19T08:10:51Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

chrischdi

/lgtm

k8s-ci-robot · 2025-02-19T08:34:53Z

LGTM label has been added.

Git tree hash: d22ebeacfe3f5a77c7855fcad5303341e19d2e92

k8s-ci-robot added cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. do-not-merge/needs-area PR is missing an area label labels Oct 8, 2024

k8s-ci-robot requested a review from elmiko October 8, 2024 14:17

k8s-ci-robot requested a review from enxebre October 8, 2024 14:17

k8s-ci-robot added size/M Denotes a PR that changes 30-99 lines, ignoring generated files. needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. labels Oct 8, 2024

randomvariable suggested changes Oct 18, 2024

View reviewed changes

bootstrap/kubeadm/api/v1beta1/kubeadmconfig_types.go Outdated Show resolved Hide resolved

bootstrap/kubeadm/api/v1beta1/kubeadmconfig_types.go Outdated Show resolved Hide resolved

davidumea requested a review from randomvariable October 18, 2024 13:45

fabriziopandini added the area/provider/bootstrap-kubeadm Issues or PRs related to CAPBK label Oct 18, 2024

k8s-ci-robot removed the do-not-merge/needs-area PR is missing an area label label Oct 18, 2024

neolit123 reviewed Oct 18, 2024

View reviewed changes

fabriziopandini reviewed Oct 21, 2024

View reviewed changes

k8s-ci-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Nov 1, 2024

davidumea added 7 commits November 5, 2024 09:16

Add bootCommands cloud-init file generation

31406da

fixup: go lint

97311d6

fixup: lint: added missing newline

4367074

fixup: add changes from verify-gen

0c23af8

fixup: filter tests

504dfd5

fixup: clearer documentation for BootCommands, PreKubeadmCommands and…

19d52df

… PostKubeadmCommands

fixup: verify: reflect new documentation in the crds

51068a2

davidumea added 2 commits January 10, 2025 13:11

fixup: update copyright year

03cbf01

fixup: make sure bootCommands is not configured when ignition format …

b7c575a

…is used

davidumea requested a review from Danil-Grigorev January 13, 2025 14:07

anmazzotti suggested changes Jan 24, 2025

View reviewed changes

bootstrap/kubeadm/internal/webhooks/kubeadmconfig.go Outdated Show resolved Hide resolved

bootstrap/kubeadm/internal/webhooks/kubeadmconfig.go Outdated Show resolved Hide resolved

bootstrap/kubeadm/internal/webhooks/kubeadmconfig_test.go Outdated Show resolved Hide resolved

fixup: use constant when checking kubeadmconfig format

0484cdc

davidumea requested a review from anmazzotti January 24, 2025 14:34

anmazzotti approved these changes Jan 24, 2025

View reviewed changes

Danil-Grigorev reviewed Jan 24, 2025

View reviewed changes

fixup: add bootcommands check to validateIgnition function

3db48fa

davidumea requested a review from Danil-Grigorev January 24, 2025 15:46

fixup: remove webhook check

23a608c

k8s-ci-robot assigned Danil-Grigorev Feb 5, 2025

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Feb 5, 2025

k8s-ci-robot assigned justinsb Feb 17, 2025

chrischdi reviewed Feb 18, 2025

View reviewed changes

davidumea added 2 commits February 19, 2025 08:54

fixup: remove bootcommand from ignition clc test

ccd5c8b

fixup: clarified bootcommands test in cloudinit test

1874ba8

k8s-ci-robot removed the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Feb 19, 2025

k8s-ci-robot requested a review from justinsb February 19, 2025 08:10

davidumea requested a review from chrischdi February 19, 2025 08:12

chrischdi reviewed Feb 19, 2025

View reviewed changes

k8s-ci-robot assigned chrischdi Feb 19, 2025

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Feb 19, 2025

	dst.Files = restored.Files

	dst.Users = restored.Users
	if restored.Users != nil {
	for i := range restored.Users {
	if restored.Users[i].PasswdFrom != nil {
	dst.Users[i].PasswdFrom = restored.Users[i].PasswdFrom
	}
	}
	}

	{"echo", "$(date) hello BootCommands!"},
	{`"echo $(date) ': hello BootCommands!'"`},

✨ Add bootCommands to cloud-init file generation #11271

Are you sure you want to change the base?

✨ Add bootCommands to cloud-init file generation #11271

Conversation

davidumea commented Oct 8, 2024 • edited Loading

k8s-ci-robot commented Oct 8, 2024

k8s-ci-robot commented Oct 8, 2024

chrischdi commented Oct 8, 2024 • edited Loading

davidumea commented Oct 9, 2024 • edited Loading

davidumea commented Oct 17, 2024

fabriziopandini commented Oct 18, 2024

randomvariable left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

davidumea Oct 22, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

davidumea Oct 22, 2024 • edited Loading

Choose a reason for hiding this comment

fabriziopandini Nov 25, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

davidumea Nov 5, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

anmazzotti left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Danil-Grigorev commented Feb 5, 2025

k8s-ci-robot commented Feb 5, 2025

davidumea commented Feb 17, 2025

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

chrischdi left a comment

Choose a reason for hiding this comment

k8s-ci-robot commented Feb 19, 2025

chrischdi left a comment

Choose a reason for hiding this comment

k8s-ci-robot commented Feb 19, 2025

davidumea commented Oct 8, 2024 •

edited

Loading

chrischdi commented Oct 8, 2024 •

edited

Loading

davidumea commented Oct 9, 2024 •

edited

Loading

davidumea Oct 22, 2024 •

edited

Loading

davidumea Oct 22, 2024 •

edited

Loading

fabriziopandini Nov 25, 2024 •

edited

Loading

davidumea Nov 5, 2024 •

edited

Loading