OCPBUGS-43777: certrotationcontroller: run tests which runs deployment and creates projects #1759

vrutkovs · 2024-10-24T07:59:20Z

Tests we run after cert rotation should ensure that pod gets created from deployment, scheduled on the node and openshift-apiserver can create projects to validate that all component certificates have been regenerated. The test names are included in
certificates.openshift.io/auto-regenerate-after-offline-expiry annotation

openshift-ci-robot · 2024-10-24T07:59:26Z

@vrutkovs: This pull request references Jira Issue OCPBUGS-43777, which is valid.

3 validation(s) were run on this bug

bug is open, matching expected state (open)
bug target version (4.18.0) matches configured target version for branch (4.18.0)
bug is in the state POST, which is one of the valid states (NEW, ASSIGNED, POST)

Requesting review from QA contact:
/cc @wangke19

The bug has been updated to refer to the pull request using the external bug tracker.

In response to this:

Tests we run after cert rotation should ensure that pod gets created from deployment, scheduled on the node and openshift-apiserver can use imagestreams to validate that all component certificates have been regenerated. The test names are included in
certificates.openshift.io/auto-regenerate-after-offline-expiry annotation

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

openshift-ci · 2024-10-24T08:01:19Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: vrutkovs

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [vrutkovs]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

vrutkovs · 2024-10-25T12:16:58Z

/retest

deads2k · 2024-10-25T13:34:24Z

pkg/operator/certrotationcontroller/certrotationcontroller.go

@@ -324,7 +324,7 @@ func newCertRotationController(
 			Name:      "service-network-serving-certkey",
 			AdditionalAnnotations: certrotation.AdditionalAnnotations{
 				JiraComponent:                    "kube-apiserver",
-				AutoRegenerateAfterOfflineExpiry: "https://github.com/openshift/cluster-kube-apiserver-operator/pull/1631,'operator conditions kube-apiserver'",
+				AutoRegenerateAfterOfflineExpiry: "https://github.com/openshift/cluster-kube-apiserver-operator/pull/1631,'[sig-apps] Deployment RollingUpdateDeployment should delete old pods and create new ones [Conformance] [Suite:openshift/conformance/parallel/minimal] [Suite:k8s]'",


I don't understand how this one tests the service network.

test to external LB to create deployment

kcm uses internal LB to create replicaset and pod

scheduler uses internal LB to scheduler pod

kubelet uses internal LB to get pods and report available status

test uses external LB to confirm available replicas -- and FYI, this is the bit that requires the scheduler. interestingly, the deployment logic itself does not.

which hop uses the service network?

@vrutkovs, if you could provide an explanation, that would be very helpful.

I also think that adding a comment with an explanation for each certificate/test pair would be greatly beneficial for future us and for easier reviewing.

Ah, sorry, previously this was using

[sig-apps] Deployment RollingUpdateDeployment should delete old pods and create new ones [Conformance] [Suite:openshift/conformance/parallel/minimal] [Suite:k8s]

but deployment rollout doesn't use service network, instead its

[sig-network] Services should serve a basic endpoint from pods [Conformance] [Suite:openshift/conformance/parallel/minimal] [Suite:k8s]

now.

I'm not sure its worth commenting tests (we probably should have an easy to understand alias), but there will be an update for certificate descriptions: #1763 - would it be sufficient? Do you want me to add comments on what these tests do in the code (and not touch the annotations)?

deads2k · 2024-10-25T13:36:01Z

pkg/operator/certrotationcontroller/certrotationcontroller.go

@@ -646,7 +646,7 @@ func newCertRotationController(
 			Name:      "control-plane-node-admin-client-cert-key",
 			AdditionalAnnotations: certrotation.AdditionalAnnotations{
 				JiraComponent:                    "kube-apiserver",
-				AutoRegenerateAfterOfflineExpiry: "https://github.com/openshift/cluster-kube-apiserver-operator/pull/1631,'operator conditions kube-apiserver'",
+				AutoRegenerateAfterOfflineExpiry: "https://github.com/openshift/cluster-kube-apiserver-operator/pull/1631,'[sig-apps] Deployment RollingUpdateDeployment should delete old pods and create new ones [Conformance] [Suite:openshift/conformance/parallel/minimal] [Suite:k8s]'",


I don't think this test uses the kubeconfig that this certificate is embedded into. this is for being able to use oc if you ssh to the node.

deads2k · 2024-10-25T13:36:45Z

pkg/operator/certrotationcontroller/certrotationcontroller.go

@@ -698,7 +698,7 @@ func newCertRotationController(
 			Name:      "check-endpoints-client-cert-key",
 			AdditionalAnnotations: certrotation.AdditionalAnnotations{
 				JiraComponent:                    "kube-apiserver",
-				AutoRegenerateAfterOfflineExpiry: "https://github.com/openshift/cluster-kube-apiserver-operator/pull/1631,'operator conditions kube-apiserver'",
+				AutoRegenerateAfterOfflineExpiry: "https://github.com/openshift/cluster-kube-apiserver-operator/pull/1631,'[sig-apps] Deployment RollingUpdateDeployment should delete old pods and create new ones [Conformance] [Suite:openshift/conformance/parallel/minimal] [Suite:k8s]'",


I don't think that this test confirms that the check endpoints controller (which may or may not still exist) can function.

deads2k · 2024-10-25T13:36:58Z

pkg/operator/certrotationcontroller/certrotationcontroller.go

@@ -723,7 +723,7 @@ func newCertRotationController(
 			Name:      "node-system-admin-signer",
 			AdditionalAnnotations: certrotation.AdditionalAnnotations{
 				JiraComponent:                    "kube-apiserver",
-				AutoRegenerateAfterOfflineExpiry: "https://github.com/openshift/cluster-kube-apiserver-operator/pull/1631,'operator conditions kube-apiserver'",
+				AutoRegenerateAfterOfflineExpiry: "https://github.com/openshift/cluster-kube-apiserver-operator/pull/1631,'[sig-apps] Deployment RollingUpdateDeployment should delete old pods and create new ones [Conformance] [Suite:openshift/conformance/parallel/minimal] [Suite:k8s]'",


same as client-cert comment above.

p0lyn0mial · 2024-11-29T09:50:28Z

I can tag this PR to get it merged, as it simply updates some descriptions, but I'm not sure if that is the intended goal.

I think the goal is to ensure we actually have a list of tests that can validate whether certificates have been regenerated properly. To do that, I would have to carefully check each test listed here.

If you want me to confirm the tests, I would suggest changing just one item. That would make it much easier for me to check and make some progress.

vrutkovs · 2024-11-29T09:57:48Z

The plan here is:

have this PR merged to add annotations
merge OCPBUGS-43777: cert-rotation: run tests which runs deployment and create projects release#58130 - it already has been rehearsed to ensure the tests are passing
Update WIP Update TLS registry origin#29213 with new nightly data to include annotations in TLS registry

p0lyn0mial

okay, let's start with the aggregation layer.

Could you remind me of the purpose of the certificates and explain how the[sig-cli] oc adm new-project [apigroup:project.openshift.io][apigroup:authorization.openshift.io] [Suite:openshift/conformance/parallel] test is going to validate each certificate?

p0lyn0mial · 2024-12-02T09:24:42Z

pkg/operator/certrotationcontroller/certrotationcontroller.go

@@ -135,7 +135,7 @@ func newCertRotationController(
 			Name:      "aggregator-client-signer",


is this the signer used to sign --proxy-client-cert ?

p0lyn0mial · 2024-12-02T09:25:12Z

pkg/operator/certrotationcontroller/certrotationcontroller.go

@@ -150,7 +150,7 @@ func newCertRotationController(
 			Name:      "kube-apiserver-aggregator-client-ca",


is this the CA that is added to --requestheader-client-ca-file ?

Yes (that's the entire CA btw)

p0lyn0mial · 2024-12-02T09:25:35Z

pkg/operator/certrotationcontroller/certrotationcontroller.go

@@ -162,7 +162,7 @@ func newCertRotationController(
 			Name:      "aggregator-client",


is this --proxy-client-cert ?

vrutkovs · 2025-01-10T13:43:05Z

/retest

Tests we run after cert rotation should ensure that pod gets created from deployment, scheduled on the node and openshift-apiserver can create projects to validate that all component certificates have been regenerated. The test names are included in certificates.openshift.io/auto-regenerate-after-offline-expiry annotation

vrutkovs · 2025-01-13T07:56:33Z

/retest

openshift-ci · 2025-01-13T10:37:08Z

@vrutkovs: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
ci/prow/e2e-aws-ovn-single-node	`8e7d25b`	link	false	`/test e2e-aws-ovn-single-node`
ci/prow/e2e-aws-ovn-upgrade	`8e7d25b`	link	true	`/test e2e-aws-ovn-upgrade`
ci/prow/e2e-gcp-operator	`8e7d25b`	link	true	`/test e2e-gcp-operator`
ci/prow/okd-scos-e2e-aws-ovn	`8e7d25b`	link	false	`/test okd-scos-e2e-aws-ovn`
ci/prow/k8s-e2e-gcp-serial	`8e7d25b`	link	false	`/test k8s-e2e-gcp-serial`
ci/prow/e2e-gcp-operator-single-node	`8e7d25b`	link	false	`/test e2e-gcp-operator-single-node`
ci/prow/e2e-aws-ovn	`8e7d25b`	link	true	`/test e2e-aws-ovn`
ci/prow/k8s-e2e-gcp	`8e7d25b`	link	true	`/test k8s-e2e-gcp`
ci/prow/e2e-aws-operator-disruptive-single-node	`8e7d25b`	link	false	`/test e2e-aws-operator-disruptive-single-node`
ci/prow/e2e-aws-ovn-serial	`8e7d25b`	link	true	`/test e2e-aws-ovn-serial`

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

hasbro17 · 2025-01-14T20:46:41Z

pkg/operator/certrotationcontroller/certrotationcontroller.go

-				AutoRegenerateAfterOfflineExpiry: "https://github.com/openshift/cluster-kube-apiserver-operator/pull/1631,'operator conditions openshift-apiserver'",
+				JiraComponent: "kube-apiserver",
+				// This test would ensure kube-apiserver request would be passed to openshift-apiserver
+				AutoRegenerateAfterOfflineExpiry: "https://github.com/openshift/cluster-kube-apiserver-operator/pull/1631,'[sig-cli] oc adm new-project [apigroup:project.openshift.io][apigroup:authorization.openshift.io] [Suite:openshift/conformance/parallel]'",


// AutoRegenerateAfterOfflineExpiry contains a link to PR and an e2e test name which verifies
// that TLS artifact is correctly regenerated after it has expired

Would the PR not be openshift/release#58130 where we add and rehearse these tests to the cert rotation jobs?

According to https://github.com/openshift/origin/blob/master/pkg/cmd/update-tls-artifacts/generate-owners/tlsmetadata/autoregenerate_after_expiry/requirement.go#L23 it needs to be the PR which added the annotation, not necessarily the one which runs these tests

openshift-ci-robot added jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. jira/valid-bug Indicates that a referenced Jira bug is valid for the branch this PR is targeting. labels Oct 24, 2024

openshift-ci bot requested review from wangke19 and sanchezl October 24, 2024 07:59

openshift-ci bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Oct 24, 2024

vrutkovs force-pushed the cert-rotation-additional-tests branch from 896b6ba to 1eb6081 Compare October 24, 2024 11:45

vrutkovs changed the title ~~OCPBUGS-43777: certrotationcontroller: run tests which runs deployment and use imagestreams~~ OCPBUGS-43777: certrotationcontroller: run tests which runs deployment and creates projects Oct 24, 2024

deads2k reviewed Oct 25, 2024

View reviewed changes

vrutkovs force-pushed the cert-rotation-additional-tests branch from 1eb6081 to 12ec413 Compare October 25, 2024 15:16

vrutkovs force-pushed the cert-rotation-additional-tests branch from 12ec413 to d00bec9 Compare November 28, 2024 10:35

p0lyn0mial reviewed Dec 2, 2024

View reviewed changes

vrutkovs force-pushed the cert-rotation-additional-tests branch 4 times, most recently from 4051bd1 to fdcdee7 Compare December 2, 2024 10:43

openshift-ci-robot mentioned this pull request Jan 10, 2025

OCPBUGS-43777: apiserver: add tests to verify kube-apiserver is accessible via different endpoints openshift/origin#29339

Merged

vrutkovs force-pushed the cert-rotation-additional-tests branch from fdcdee7 to fd2867a Compare January 10, 2025 11:14

vrutkovs force-pushed the cert-rotation-additional-tests branch from fd2867a to 8e7d25b Compare January 10, 2025 14:40

hasbro17 reviewed Jan 14, 2025

View reviewed changes

openshift-ci-robot mentioned this pull request Jan 26, 2025

OCPBUGS-43777: tests: extend kubeconfig tests openshift/origin#29335

Merged

2 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

OCPBUGS-43777: certrotationcontroller: run tests which runs deployment and creates projects #1759

OCPBUGS-43777: certrotationcontroller: run tests which runs deployment and creates projects #1759

vrutkovs commented Oct 24, 2024 •

edited

Loading

openshift-ci-robot commented Oct 24, 2024

openshift-ci bot commented Oct 24, 2024

vrutkovs commented Oct 25, 2024

deads2k Oct 25, 2024 •

edited

Loading

p0lyn0mial Nov 29, 2024

vrutkovs Nov 29, 2024

deads2k Oct 25, 2024

deads2k Oct 25, 2024

deads2k Oct 25, 2024

p0lyn0mial commented Nov 29, 2024

vrutkovs commented Nov 29, 2024

p0lyn0mial left a comment

p0lyn0mial Dec 2, 2024

vrutkovs Dec 2, 2024

p0lyn0mial Dec 2, 2024

vrutkovs Dec 2, 2024

p0lyn0mial Dec 2, 2024

vrutkovs Dec 2, 2024

vrutkovs commented Jan 10, 2025

vrutkovs commented Jan 13, 2025

openshift-ci bot commented Jan 13, 2025

hasbro17 Jan 14, 2025

vrutkovs Jan 15, 2025

		@@ -135,7 +135,7 @@ func newCertRotationController(
		Name: "aggregator-client-signer",

		@@ -150,7 +150,7 @@ func newCertRotationController(
		Name: "kube-apiserver-aggregator-client-ca",

		@@ -162,7 +162,7 @@ func newCertRotationController(
		Name: "aggregator-client",

OCPBUGS-43777: certrotationcontroller: run tests which runs deployment and creates projects #1759

Are you sure you want to change the base?

OCPBUGS-43777: certrotationcontroller: run tests which runs deployment and creates projects #1759

Conversation

vrutkovs commented Oct 24, 2024 • edited Loading

openshift-ci-robot commented Oct 24, 2024

openshift-ci bot commented Oct 24, 2024

vrutkovs commented Oct 25, 2024

deads2k Oct 25, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

p0lyn0mial commented Nov 29, 2024

vrutkovs commented Nov 29, 2024

p0lyn0mial left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

vrutkovs commented Jan 10, 2025

vrutkovs commented Jan 13, 2025

openshift-ci bot commented Jan 13, 2025

Choose a reason for hiding this comment

Choose a reason for hiding this comment

vrutkovs commented Oct 24, 2024 •

edited

Loading

deads2k Oct 25, 2024 •

edited

Loading