From 8680ed165c1bcc4304a8d74d54a6983eac90bbc8 Mon Sep 17 00:00:00 2001 From: Frank Budinsky Date: Tue, 3 Dec 2024 15:21:37 -0500 Subject: [PATCH 1/2] ibmcloud: PeerPod demo instructions for OpenShift clusters Added doc with streamlined instructions to set up the peer pod demo environment using an OpenShift cluster. Also added references and made small corrections to the README. Signed-off-by: Frank Budinsky --- src/cloud-api-adaptor/ibmcloud/README.md | 15 +- src/cloud-api-adaptor/ibmcloud/ROKS_SETUP.md | 288 +++++++++++++++++++ 2 files changed, 298 insertions(+), 5 deletions(-) create mode 100644 src/cloud-api-adaptor/ibmcloud/ROKS_SETUP.md diff --git a/src/cloud-api-adaptor/ibmcloud/README.md b/src/cloud-api-adaptor/ibmcloud/README.md index 5ebc192cf..315c2a89d 100644 --- a/src/cloud-api-adaptor/ibmcloud/README.md +++ b/src/cloud-api-adaptor/ibmcloud/README.md @@ -10,6 +10,9 @@ The high level flow involved is: - Deploy and validate that the nginx demo works - Clean-up and deprovision +For streamlined instructions to set up a simple demo for a Red Hat OpenShift Kubernetes cluster, refer to +[PeerPod setup using Red Hat OpenShift on IBM Cloud](./ROKS_SETUP.md). + ## Pre-reqs When building the peer pod VM image, it is simplest to use the container based approach, which only requires either @@ -76,8 +79,10 @@ needed in later steps. ## Create a 'self-managed' Kubernetes cluster on IBM Cloud provided infrastructure If you don't have a Kubernetes cluster for testing, you can follow the open-source -[instructions](./cluster) - to set up a basic cluster where the Kubernetes nodes run on IBM Cloud provided infrastructure. +[instructions](./cluster) to set up a basic cluster where the Kubernetes nodes run on IBM Cloud provided infrastructure. + +If you want to create, or use an existing, Red Hat OpenShift Kubernetes cluster, follow the instructions in +[Set up an OpenShift Kubernetes cluster for PeerPod VMs](./ROKS_SETUP.md#set-up-an-openshift-kubernetes-cluster-for-peerpod-vms). ## Deploy PeerPod Webhook @@ -94,7 +99,7 @@ If you don't have a Kubernetes cluster for testing, you can follow the open-sour #### Deploy the peer-pods webhook - From within the root directory of the `cloud-api-adaptor` repository, deploy the [webhook](../../webhook/) with: ``` - kubectl apply -f ./webhook/hack/webhook-deploy.yaml + kubectl apply -f ./src/webhook/hack/webhook-deploy.yaml ``` - Wait for the pods to all be in running state with: ``` @@ -103,7 +108,7 @@ If you don't have a Kubernetes cluster for testing, you can follow the open-sour - Advertise the extended resource `kata.peerpods.io/vm.` by running the following commands: ``` - pushd webhook/hack/extended-resources + pushd src/webhook/hack/extended-resources ./setup.sh popd ``` @@ -112,7 +117,7 @@ If you don't have a Kubernetes cluster for testing, you can follow the open-sour The `caa-provisioner-cli` simplifies deploying the operator and the cloud-api-adaptor resources on to any cluster. See the [test/tools/README.md](../test/tools/README.md) for full instructions. To create an ibmcloud ready version follow these steps ```bash -# Starting from the cloud-api-adaptor root directory +# Starting from directory src/cloud-api-adaptor of the cloud-api-adapter repository pushd test/tools make BUILTIN_CLOUD_PROVIDERS="ibmcloud" all popd diff --git a/src/cloud-api-adaptor/ibmcloud/ROKS_SETUP.md b/src/cloud-api-adaptor/ibmcloud/ROKS_SETUP.md new file mode 100644 index 000000000..4b0ca5ec5 --- /dev/null +++ b/src/cloud-api-adaptor/ibmcloud/ROKS_SETUP.md @@ -0,0 +1,288 @@ +# PeerPod setup using Red Hat OpenShift on IBM Cloud + +This guide describes how to set up a simple peer pod demo environment with a Red Hat OpenShift Kubernetes cluster on IBM Cloud. This involves the following steps: + +1. [Set up an OpenShift Kubernetes cluster for PeerPod VMs](#set-up-an-openshift-kubernetes-cluster-for-peerpod-vms) +1. [Upload a PeerPod VM Custom Image](#upload-a-peerpod-vm-custom-image) +1. [Deploy the PeerPod Webhook](#deploy-the-peerpod-webhook) +1. [Deploy the Confidential-containers operator](#deploy-the-confidential-containers-operator) +1. [Run a Helloworld sample](#run-a-helloworld-sample) + +## Pre-reqs + +Before proceeding you will need to install: + +1. the [Pre-reqs in README.md](./README.md#pre-reqs) but not including Terraform and Ansible, which are not used in this guide. +1. ibmcloud plugins: + - container-service[kubernetes-service/ks] + - vpc-infrastructure[infrastructure-service/is] +1. the OpenShift [oc CLI](https://cloud.ibm.com/docs/openshift?topic=openshift-cli-install#install-kubectl-cli) + +## Set up an OpenShift Kubernetes cluster for PeerPod VMs + +To set up a cluster for peer pod VMs, you need an IBM Cloud VPC, a subnet with a public gateway, and a COS instance. +Set the following environment variables to the values for your setup (note: make sure that your subnet has an attached public gateway): + +```bash +export IBMCLOUD_API_KEY= +export VPC_ID= +export SUBNET_ID= +export COS_CRN= +``` + +Log in to ibmcloud in the region corresponding to your subnet and then set zone and region environnment variables using the following commands: + +```bash +export ZONE="$(ibmcloud is subnet $SUBNET_ID -json | jq -r .zone.name)" +export REGION="$(ibmcloud is zone $ZONE -json | jq -r .region.name)" +``` + +If you have an existing OpenShift cluster, set the following variable to the name of your cluster. Otherwise set it to whatever name you'd like. + +```bash +export CLUSTER_NAME=kata-test-roks +``` + +### Create an Openshift cluster + +If you are using an existing cluster, you can skip this section and proceed to [configuring the cluster](#configure-an-openshift-cluster). + +1. Create a ROKS cluster + + ```bash + ibmcloud ks cluster create vpc-gen2 --flavor bx2.4x16 --name "$CLUSTER_NAME" --subnet-id "$SUBNET_ID" --vpc-id "$VPC_ID" --zone "$ZONE" --operating-system RHCOS --workers 2 --version 4.16.23_openshift --disable-outbound-traffic-protection --cos-instance "$COS_CRN" + + ``` + + Note: you will need to update the value of the `--version` argument to the current default version returned by `ibmcloud ks versions`, if it is different. + +1. Wait until the cluster is completely up and running before proceeding ... + +1. Get the `kubeconfig` for your cluster + + ```bash + ibmcloud ks cluster config --cluster "$CLUSTER_NAME" --admin + ``` + +### Configure an OpenShift cluster + +By default, your Red Hat OpenShift cluster will not work with the peer pod components. Using the environment variables set in the previous section, proceed with the following steps. + +1. Add a cluster inbound security group rule for the `kata-agent` client + + ```bash + export CLUSTER_ID=$(ibmcloud ks cluster get --cluster "$CLUSTER_NAME" --output json | jq -r .id) + export CLUSTER_SG="kube-$CLUSTER_ID" + export KATA_SG=$(ibmcloud is vpc "$VPC_ID" -json | jq -r .default_security_group.id) + ibmcloud is sg-rulec "$CLUSTER_SG" inbound udp --port-min 4789 --port-max 4789 --remote "$KATA_SG" + ``` + +1. Allow `cloud-api-adapter` to update pod finalizers + + ```bash + kubectl apply -n default -f - < with id is available`. Make note of the `image-id`, which will be +needed below. + +Note: If the import.sh script fails and the CLI has not been configured with the COS instance before, you will need to include the `--instance` argument. Refer to [IMPORT_PODVM_TO_VPC.md](https://github.com/confidential-containers/cloud-api-adaptor/blob/main/src/cloud-api-adaptor/ibmcloud/IMPORT_PODVM_TO_VPC.md#running) for details. + +## Deploy the PeerPod Webhook + +Follow the [webhook instructions in README.md](./README.md#deploy-peerpod-webhook) to deploy cert-manager and the peer-pods webhook. + +## Deploy the Confidential-containers operator + +The `caa-provisioner-cli` command can be use to simplify deployment of the operator and the cloud-api-adaptor resources on to any cluster. See the [test/tools/README.md](../test/tools/README.md) for full instructions. To create an ibmcloud-ready version of the provisioner CLI, run the following make command: + +```bash +# Starting from directory src/cloud-api-adaptor of the cloud-api-adapter repository +pushd test/tools +make BUILTIN_CLOUD_PROVIDERS="ibmcloud" all +popd +``` + +This will create `caa-provisioner-cli` in the `src/cloud-api-adaptor/test/tools` directory. To use the command you will need to set up a `.properties` file containing the relevant ibmcloud information to enable your cluster to create and use peer-pods. + +Set the SSH_KEY_ID and PODVM_IMAGE_ID environment variables and use the following command to generate the `.properties` file (Note that the IBMCLOUD_API_KEY, VPC_ID, and SUBNET_ID environment variables have already been set in [Set up an OpenShift Kubernetes cluster for PeerPod VMs +](#set-up-an-openshift-kubernetes-cluster-for-peerpod-vms)): + +```bash +export SSH_KEY_ID= # your ssh key id +export PODVM_IMAGE_ID= # the image id of the peerpod vm uploaded to ibmcloud +#export IBMCLOUD_API_KEY= # your ibmcloud apikey +#export VPC_ID= # vpc that the cluster is in +#export SUBNET_ID= # subnet to use (must have a public gateway attached) + +cat < ~/peerpods-cluster.properties +APIKEY="$IBMCLOUD_API_KEY" +SSH_KEY_ID="$SSH_KEY_ID" +PODVM_IMAGE_ID="$PODVM_IMAGE_ID" +VPC_ID="$VPC_ID" +VPC_SUBNET_ID="$SUBNET_ID" +VPC_SECURITY_GROUP_ID="$(ibmcloud is vpc "$VPC_ID" -json | jq -r .default_security_group.id)" +RESOURCE_GROUP_ID="$(ibmcloud is vpc "$VPC_ID" -json | jq -r .resource_group.id)" +ZONE="$(ibmcloud is subnet $SUBNET_ID -json | jq -r .zone.name)" +REGION="$(ibmcloud is zone $ZONE -json | jq -r .region.name)" +IBMCLOUD_PROVIDER="ibmcloud" +INSTANCE_PROFILE_NAME="bx2-2x8" +CAA_IMAGE_TAG="latest-amd64" +EOF +``` + +This will create a `peerpods-cluster.properties` files in your home directory. + +Finally, run the `caa-provisioner-cli` command to install the operator and cloud-api-adaptor: + +```bash +export CLOUD_PROVIDER=ibmcloud +export TEST_PROVISION_FILE="$HOME/peerpods-cluster.properties" +export TEST_TEARDOWN="no" +pushd test/tools +./caa-provisioner-cli -action=install +popd +``` + +Run the following command to confirm that the operator and cloud-api-adapter have been deployed: + +```bash +kubectl get pods -n confidential-containers-system +``` + +Once everything is up and ruuning, you should see output similar to the following: + +``` +cc-operator-controller-manager-7f8db55b55-r9thx 2/2 Running 0 7m55s +cc-operator-daemon-install-2dsnz 1/1 Running 0 6m59s +cc-operator-daemon-install-d522m 1/1 Running 0 6m58s +cc-operator-pre-install-daemon-gzl8w 1/1 Running 0 7m29s +cc-operator-pre-install-daemon-w5whl 1/1 Running 0 7m29s +cloud-api-adaptor-daemonset-m5x5s 1/1 Running 0 7m29s +cloud-api-adaptor-daemonset-v6jdr 1/1 Running 0 7m29s +peerpod-ctrl-controller-manager-65f76cb59-vhbt4 2/2 Running 0 5m20s +``` + +## Run a Helloworld sample + +You can run the following commands to validate that your cluster has been set up properly and is working as expected. + +```bash +kubectl apply -n default -f https://raw.githubusercontent.com/istio/istio/release-1.24/samples/curl/curl.yaml +kubectl apply -n default -f - < Date: Thu, 5 Dec 2024 11:19:12 -0500 Subject: [PATCH 2/2] ibmcloud: Remove unneeded step from README Remove unneeded advertise the extended resource step from README. Signed-off-by: Frank Budinsky --- src/cloud-api-adaptor/ibmcloud/README.md | 7 ------- 1 file changed, 7 deletions(-) diff --git a/src/cloud-api-adaptor/ibmcloud/README.md b/src/cloud-api-adaptor/ibmcloud/README.md index 315c2a89d..08a1ca2f8 100644 --- a/src/cloud-api-adaptor/ibmcloud/README.md +++ b/src/cloud-api-adaptor/ibmcloud/README.md @@ -106,13 +106,6 @@ If you want to create, or use an existing, Red Hat OpenShift Kubernetes cluster, kubectl get pods -n peer-pods-webhook-system --watch ``` -- Advertise the extended resource `kata.peerpods.io/vm.` by running the following commands: - ``` - pushd src/webhook/hack/extended-resources - ./setup.sh - popd - ``` - ## Deploy the Confidential-containers operator The `caa-provisioner-cli` simplifies deploying the operator and the cloud-api-adaptor resources on to any cluster. See the [test/tools/README.md](../test/tools/README.md) for full instructions. To create an ibmcloud ready version follow these steps