feat!: Add working conversion webhook with cert rotation #1066

sbernauer · 2025-06-30T14:07:23Z

Description

An working example usage can be found in stackabletech/zookeeper-operator#958 (mainly look at rust/operator-binary/src/main.rs)

Definition of Done Checklist

Not all of these items are applicable to all PRs, the author should update this template to only leave the boxes in that are relevant
Please make sure all these things are done and tick the boxes

Author

Changes are OpenShift compatible
CRD changes approved
CRD documentation for all fields, following the style guide.
Integration tests passed (for non trivial changes)
Changes need to be "offline" compatible

Reviewer

Code contains useful comments
Code contains useful logging statements
(Integration-)Test cases added
Documentation added or updated. Follows the style guide.
Changelog updated
Cargo.toml only contains references to git tags (not specific commits or branches)

Acceptance

Feature Tracker has been updated
Proper release label has been added

sbernauer · 2025-07-02T06:11:49Z

crates/stackable-webhook/src/tls/mod.rs

+pub const WEBHOOK_CA_LIFETIME: Duration = Duration::from_minutes_unchecked(3);
+pub const WEBHOOK_CERTIFICATE_LIFETIME: Duration = Duration::from_minutes_unchecked(2);
+pub const WEBHOOK_CERTIFICATE_ROTATION_INTERVAL: Duration = Duration::from_minutes_unchecked(1);


reminder to bump these before merging. Currently they are so low for easy testing

Techassi

Partial review, I didn't look at the CertificateResolver yet.

crates/stackable-operator/src/cli.rs

crates/stackable-webhook/src/lib.rs

crates/stackable-webhook/src/servers/conversion.rs

crates/stackable-webhook/src/tls/mod.rs

Co-authored-by: Techassi <[email protected]>

Techassi

I left a bunch of comments.

I encountered the ConversionWebhookServer which is missing the changes we talked about a few weeks back. As such, it doesn't make a whole lot of sense to continue the review before these changes are implemented. I've sent you an appropriate patch as a private message which should be a good starting point for the changes we discussed.

Techassi · 2025-08-06T14:31:10Z

crates/stackable-operator/src/cli.rs

+            "--operator-namespace",
+            "stackable-operators",
+            "--operator-service-name",
+            "foo-operator",


note: I still feel like all these CLI unit tests are pretty much useless and should be removed to speed up the testing runs.

I don't think they have a noticeable effect on the test runtime. Compare to the really slow doc tests, those take 1000x longer (not joking ^^)
I think the maintenance effort is a valid point, removed the CLI parsing tests in 5e41bf1

crates/stackable-operator/CHANGELOG.md

Techassi · 2025-08-06T14:46:07Z

crates/stackable-webhook/src/servers/conversion.rs

+use kube::{
+    Api, Client, ResourceExt,
+    api::{Patch, PatchParams},
+};
+use snafu::{OptionExt, ResultExt, Snafu};
+use stackable_operator::cli::OperatorEnvironmentOptions;
+use tokio::{sync::mpsc, try_join};
 use tracing::instrument;
+use x509_cert::{
+    Certificate,
+    der::{EncodePem, pem::LineEnding},
+};


note: Please combine these imports with the others above.

hmm, what imports exactly can be combined? rustfmt groups them like this

Yeah this is sub-optimal formatting sadly. You need to manually move them to the top imports above line 11.

I'm too stupid, can you please provide a patch that "survives" cargo fmt? 🤦‍♂️

crates/stackable-webhook/src/servers/conversion.rs

Techassi

We are mostly there, just some small things left.

crates/stackable-operator/CHANGELOG.md

crates/stackable-webhook/src/servers/conversion.rs

Techassi · 2025-08-13T12:46:21Z

crates/stackable-webhook/src/servers/conversion.rs

+    /// The environment the operator is running in, notably the namespace and service name it is
+    /// reachable at.
+    pub operator_environment: OperatorEnvironmentOptions,


note: Let's avoid pulling in a type from stackable_operator here. Instead, split it into two separate fields.

Suggested change

/// The environment the operator is running in, notably the namespace and service name it is

/// reachable at.

pub operator_environment: OperatorEnvironmentOptions,

pub namespace: String,

pub service_name: String,

Hmm, I intentionally sticked with the struct that describes the k8s env. I understand your concerns, but I also really don't like too many function arguments when there is a perfectly fine struct for what we want as "information about the environment of the operator".
Maybe tomorrow we need the k8s version or if we are on OpenShift or whatnot

The comment of the field perfectly sums this up:

The environment the operator is running in, notably the namespace and service name it is reachable at.

Currently, we only need the namespace and the service name. That's it. Why are we optimizing for something which might or might not come in the future? Regarding how the crates are currently structured, I would like to avoid more entanglement between those.

As discussed multiple times already, this needs better structure in the long run, which I can hopefully pick up soon.

I also really don't like too many function arguments

There is only a single parameter: ConversionWebhookOptions. Everything is contained in there. And having two fields instead of one, doesn't really justify mixing even more types between crates (which are not designed for that currently).

Techassi · 2025-08-13T12:54:08Z

crates/stackable-webhook/src/servers/mod.rs

@@ -2,4 +2,5 @@
 //! purposes.
 mod conversion;

-pub use conversion::*;
+pub use conversion::{ConversionWebhookError, ConversionWebhookOptions, ConversionWebhookServer};
+pub use kube::core::conversion::ConversionReview;


note: This is already re-exported via stackable_webhook::servers::conversion::ConversionReview. As such, please remove it from here again.

Correct, removed in 8dc1235

crates/stackable-webhook/src/tls/cert_resolver.rs

Techassi · 2025-08-13T13:37:11Z

crates/stackable-webhook/src/tls/cert_resolver.rs

+        let (cert, certified_key) = Self::generate_new_cert(subject_alterative_dns_names.clone())
+            .await
+            .context(GenerateNewCertificateSnafu)?;
+
+        cert_tx
+            .send(cert)
+            .await
+            .map_err(|_err| CertificateResolverError::SendCertificateToChannel)?;


note: I feel like this can be moved into its own function, because we repeat the exact same code below.

Any reason why you didn't include the certificate generation in that function as well? That's basically also the same code across two functions.

crates/stackable-webhook/src/options.rs

crates/stackable-webhook/CHANGELOG.md

Techassi · 2025-08-13T13:45:03Z

crates/stackable-webhook/CHANGELOG.md

+  Also, `TlsServer::new` now returns an additional `mpsc::Receiver<Certificate>`, so that the caller
+  can get notified about certificate rotations happening ([#1066]).


note: (Addition to the comment above) This would be the perfect trigger for the operator (the caller of this function) to reconcile the CRDs.

Co-authored-by: Techassi <[email protected]>

sbernauer added 9 commits June 27, 2025 17:11

WIP

0ef29ab

Actually rotate certs

ed58563

clippy

3be083e

doctests

fd4d94b

HashMap -> Vec

b3c17b3

Set correct CA lifetime

bdbcee0

Result type

b4c22b0

fix rustdocs

31b128b

Add some docs

b84cf05

sbernauer mentioned this pull request Jun 30, 2025

tracking: CRD versioning stackabletech/issues#642

Open

88 tasks

Update rustdoc

0390680

sbernauer changed the title ~~feat(webhook): Add functioning conversion webhook with cert rotation~~ feat(webhook): Add working conversion webhook with cert rotation Jul 2, 2025

sbernauer added 3 commits July 2, 2025 08:09

Handle cert rottion loop falures

2e1d085

changelog

3f8e744

link to decision

6f0040c

sbernauer commented Jul 2, 2025

View reviewed changes

sbernauer mentioned this pull request Jul 2, 2025

feat!: Add conversion webhook stackabletech/zookeeper-operator#958

Draft

20 tasks

sbernauer self-assigned this Jul 2, 2025

fix rustdocs

941ee6b

sbernauer added this to Stackable Engineering Jul 2, 2025

sbernauer moved this to Development: In Progress in Stackable Engineering Jul 2, 2025

sbernauer moved this from Development: In Progress to Development: In Review in Stackable Engineering Jul 2, 2025

sbernauer moved this from Development: In Review to Development: Waiting for Review in Stackable Engineering Jul 2, 2025

sbernauer mentioned this pull request Jul 2, 2025

feat: Add Service on top of operators stackabletech/operator-templating#534

Merged

Techassi self-requested a review July 3, 2025 06:43

Techassi moved this from Development: Waiting for Review to Development: In Review in Stackable Engineering Jul 3, 2025

sbernauer mentioned this pull request Jul 3, 2025

Use the local kubelet's cluster domain, rather than a random one #1071

Merged

8 tasks

Techassi requested changes Jul 3, 2025

View reviewed changes

Techassi changed the title ~~feat(webhook): Add working conversion webhook with cert rotation~~ feat!: Add working conversion webhook with cert rotation Jul 3, 2025

Merge remote-tracking branch 'origin/main' into feat/conversion-webhook

0c52f5f

Techassi reviewed Jul 8, 2025

View reviewed changes

crates/stackable-webhook/src/tls/mod.rs Outdated Show resolved Hide resolved

sbernauer and others added 3 commits July 8, 2025 15:56

mount -> project

9da5588

Update crates/stackable-webhook/src/tls/mod.rs

758202f

Co-authored-by: Techassi <[email protected]>

changelog

29b0c6c

This was referenced Jul 9, 2025

feat!(stackable-certs): Use builder pattern and support SAN entries #1044

Closed

feat!: Re-export stackable-certs and stackable-webhook crates #1074

Open

Techassi requested changes Aug 6, 2025

View reviewed changes

sbernauer added 5 commits August 11, 2025 17:10

Add ConversionWebhookOptions struct

8b83131

Merge remote-tracking branch 'origin/main' into feat/conversion-webhook

e9e182e

PascalCase error variants

fe1749d

changelog

3894d3b

changelog

0936315

sbernauer requested a review from Techassi August 11, 2025 15:36

fix rustdocs

99f1dfe

Techassi requested changes Aug 13, 2025

View reviewed changes

sbernauer and others added 10 commits August 15, 2025 12:09

Update crates/stackable-operator/CHANGELOG.md

daac003

Co-authored-by: Techassi <[email protected]>

ReceiverCertificateFromChannel -> ReceiveCertificateFromChannel

0e1a160

Remove double re-export

8dc1235

fixup

f87c468

Update crates/stackable-webhook/src/tls/cert_resolver.rs

d8765d8

Co-authored-by: Techassi <[email protected]>

Use CryptoProvider::get_default

0c490c2

Update crates/stackable-webhook/src/tls/cert_resolver.rs

4e4cc70

Co-authored-by: Techassi <[email protected]>

changelog

0dd99ed

Remove CLI parsing tests

5e41bf1

clippy

320e821

sbernauer requested a review from Techassi August 15, 2025 10:53

sbernauer added 4 commits August 15, 2025 12:53

Merge remote-tracking branch 'origin/main' into feat/conversion-webhook

221bc36

Move stuff into send_certificate_to_channel

37818fd

OptionsBuilder -> WebhookOptionsBuilder

c925bd0

fix typo with read and write

8c13fd4

		Also, `TlsServer::new` now returns an additional `mpsc::Receiver<Certificate>`, so that the caller
		can get notified about certificate rotations happening ([#1066]).

Uh oh!

feat!: Add working conversion webhook with cert rotation #1066

Are you sure you want to change the base?

feat!: Add working conversion webhook with cert rotation #1066

Conversation

sbernauer commented Jun 30, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Definition of Done Checklist

Author

Reviewer

Acceptance

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Techassi left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Techassi left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Techassi left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Techassi Aug 15, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Techassi Aug 13, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

sbernauer commented Jun 30, 2025 •

edited

Loading

Techassi left a comment •

edited

Loading

Techassi Aug 15, 2025 •

edited

Loading

Techassi Aug 13, 2025 •

edited

Loading