Add Secret Management Container to ClickHouse Keeper Installation

[murtazayusuf]

Background

We have a ClickHouse Keeper installation in our Kubernetes cluster that requires named collection configuration. Currently, we manage this through a separate job that runs to populate the named collections in the ClickHouse Keeper. We want to integrate this functionality more tightly with the ClickHouse Keeper pods to ensure better lifecycle management and reliability.

Current Setup

We currently have:

1. A ClickHouseKeeperInstallation custom resource that manages our ClickHouse Keeper cluster
2. A ConfigMap containing the initialization script for named collections
3. A ServiceAccount with necessary permissions to access GCP secrets
4. A separate job that runs to configure the named collections

The initialization script works by:

• Fetching necessary credentials from GCP Secret Manager
• Waiting for the ClickHouse Keeper to be available
• Creating the named collection with the fetched credentials

Objective

We want to integrate the secret management functionality directly into the ClickHouse Keeper deployment rather than running it as a separate job. This would ensure that the named collections are always properly configured and maintained alongside the ClickHouse Keeper instances.

Attempted Solutions

Attempt 1: Adding Sidecar Container

First, we tried adding a sidecar container to the existing ClickHouse Keeper pods using a kustomize patch:

- op: add
  path: "/spec/templates/podTemplates/0/spec/serviceAccountName"
  value: clickhouse-keeper-secret-manager
- op: add
  path: "/spec/templates/podTemplates/0/spec/volumes"
  value:
    - name: init-scripts
      configMap:
        name: keeper-init-configmap
        defaultMode: 0755
- op: add
  path: "/spec/templates/podTemplates/0/spec/containers/-"
  value:
    name: secret-setup
    image: us-docker.pkg.dev/brightedge-gcp-staging1/gcr.io/gcloud-zk:latest
    command:
      - /scripts/init-named-collections.sh
    volumeMounts:
      - name: init-scripts
        mountPath: /scripts
    env:
      - name: KEEPER_HOST
        value: "127.0.0.1"
      - name: KEEPER_PORT
        value: "2181"
    resources:
      requests:
        memory: "256Mi"
      limits:
        memory: "512Mi"

Issue: This resulted in incorrect container specification nesting, where the configurations were being mixed between the main ClickHouse Keeper container and our secret management container. This never brought up the chk-3-nodes-0 pod up and kept chk-3-nodes clickhousekeeperinstallation in in-progress state.

Attempt 2: Replacing Container Array

We then tried replacing the entire containers array to ensure proper structure:

- op: add
  path: "/spec/templates/podTemplates/0/spec/containers"
  value:
    - image: clickhouse/clickhouse-keeper:head-alpine
      imagePullPolicy: IfNotPresent
      name: clickhouse-keeper
      resources:
        limits:
          memory: 1Gi
        requests:
          memory: 500Mi
    - name: secret-setup
      image: us-docker.pkg.dev/brightedge-gcp-staging1/gcr.io/gcloud-zk:latest
      command:
        - /scripts/init-named-collections.sh
      env:
        - name: KEEPER_HOST
          value: "127.0.0.1"
        - name: KEEPER_PORT
          value: "2181"
      resources:
        limits:
          memory: "512Mi"
        requests:
          memory: "256Mi"
      volumeMounts:
        - name: init-scripts
          mountPath: /scripts
- op: add
  path: "/spec/templates/podTemplates/0/spec/volumes"
  value:
    - name: init-scripts
      configMap:
        name: keeper-init-configmap
        defaultMode: 493
- op: add
  path: "/spec/templates/podTemplates/0/spec/serviceAccountName"
  value: clickhouse-keeper-secret-manager

Issue: This approach still resulted in configuration problems and risked disrupting the main ClickHouse Keeper container's operation.

Attempt 3: Using Init Container

Before trying the new pod template approach, we attempted to use an init container configuration instead of a sidecar. The reasoning was that init containers are specifically designed to run setup operations before the main container starts, which aligns well with our need to configure named collections:

- op: add
  path: "/spec/templates/podTemplates/0/spec/initContainers"
  value:
    - name: secret-setup
      image: us-docker.pkg.dev/brightedge-gcp-staging1/gcr.io/gcloud-zk:latest
      command:
        - /scripts/init-named-collections.sh
      env:
        - name: KEEPER_HOST
          value: "127.0.0.1"
        - name: KEEPER_PORT
          value: "2181"
      volumeMounts:
        - name: init-scripts
          mountPath: /scripts

However, this attempt faced two significant problems:

Similar to the sidecar container attempt, we encountered issues with the ClickHouseKeeperInstallation operator's handling of pod template modifications. The operator either:

Didn't properly apply the init container configuration
Or merged the configurations in unexpected ways

Attempt 4: Adding New Pod Template

Our last attempt was to add a completely new pod template for the secret management functionality:

- op: add
  path: "/spec/templates/podTemplates/-"
  value:
    name: secret-manager
    spec:
      serviceAccountName: clickhouse-keeper-secret-manager
      containers:
        - name: secret-setup
          image: us-docker.pkg.dev/brightedge-gcp-staging1/gcr.io/gcloud-zk:latest
          command:
            - /scripts/init-named-collections.sh
          env:
            - name: KEEPER_HOST
              value: "chk-3-nodes.clickhouse"
            - name: KEEPER_PORT
              value: "2181"
          resources:
            requests:
              memory: "256Mi"
            limits:
              memory: "512Mi"
          volumeMounts:
            - name: init-scripts
              mountPath: /scripts
      volumes:
        - name: init-scripts
          configMap:
            name: keeper-init-configmap
            defaultMode: 493
      nodeSelector:
        type: static
      tolerations:
        - effect: NoSchedule
          key: static-nodes
          operator: Exists

Issue: When applied, this only created pods for the ClickHouse Keeper itself and did not create the additional secret management pod. This suggests that the ClickHouseKeeperInstallation operator might not support multiple pod templates in the way we expected.

Current Blocker

The ClickHouseKeeperInstallation operator appears to have specific expectations about pod templates that we don't fully understand. Despite adding a new pod template to the specification, the operator only creates pods using the default template. This suggests that either:

1. The operator doesn't support multiple pod templates
2. We need to reference the additional pod template in a different way
3. The operator has specific requirements for pod template configuration that we haven't met

Does the ClickHouseKeeperInstallation operator support running multiple pod types within a single installation? If not, what's the recommended way to deploy tightly coupled auxiliary services alongside ClickHouse Keeper?

The current separate job approach works but isn't ideal because:

• It requires manual intervention if the configuration needs to be updated
• There's no automatic recovery if the ClickHouse Keeper cluster is recreated
• It's harder to maintain version consistency between the configurations and the cluster

Looking for guidance on the best way to implement this integration while maintaining reliability and manageability of the system.

[murtazayusuf]

@johnny @vitaliyf @Slach @yuzhichang Can you please help with this?

[murtazayusuf]

@sunsingerus can you take a look, help us out what we can do?