Extend tests and automation scripts for airgapped installations (#682)

* scripts: Include utility scripts for airgap

Include helper scripts for:
1. Generating a list of all images
2. Pulling images to docker cache
3. Retagging images, and using shas
4. Saving images in a tar.gz
5. Saving charms in a tar.gz

Signed-off-by: Kimonas Sotirchos <[email protected]>

* airgap: LXC and LXD profiles

Profiles used during an airgap installation for spinning up
lxc containers.

Signed-off-by: Kimonas Sotirchos <[email protected]>

* tests: Test driver for airgap installation

Script for
1. Pulling all images and creating tarbal
2. Pulling all charms and creating tarbal
3. Creating a LXC container
4. Install MicroK8s in it
5. Setup a Docker registry, as running container
6. Copies tarbals to LXC container
7. Loads all the images into docker registry
8. Cut off network connection

Signed-off-by: Kimonas Sotirchos <[email protected]>

* gitignore: Ignore files generated from airgap

Signed-off-by: Kimonas Sotirchos <[email protected]>

* review

---------

Signed-off-by: Kimonas Sotirchos <[email protected]>

.gitignore

@@ -6,3 +6,17 @@ build/
 sel-screenshots/*
 geckodriver.log
 .vscode/*
+*.tar.gz
+
+# vim
+[._]*.s[a-v][a-z]
+!*.svg  # comment out if you don't need vector files
+[._]*.sw[a-p]
+[._]s[a-rt-v][a-z]
+[._]ss[a-gi-z]
+[._]sw[a-p]
+
+# airgapped
+images.txt
+charms.txt
+retagged-images.txt

gcloud-publish-file.sh

@@ -0,0 +1,41 @@
+set -eu
+
+# This script is responsible for the following:
+# 1. Upload a file $ARTIFACT to a $GS_URL
+# 2. Creates a service account key from $GCLOUD_SA
+# 3. Creates a signed URL for the updoaded file with the SA key created
+#    from the previous step
+#
+# The script expects that the user has
+# 1. logged in to the gcloud CLI
+# 2. selected in gcloud the project they want to use
+# 3. created a service account key, for pushing to bucket
+#
+# Some helper commands for the above are the following:
+#
+# gcloud auth login --no-launch-browser
+# gcloud projects list
+# gcloud config set project PROJECT_ID
+# gcloud iam service-accounts keys create \
+#     --iam-account=ckf-artifacts-storage-sa@thermal-creek-391110.iam.gserviceaccount.com
+#     signing-sa-key.json \
+#
+# For more information you can take a look on the following links
+# https://cloud.google.com/iam/docs/keys-create-delete#iam-service-account-keys-create-gcloud
+# https://cloud.google.com/storage/docs/access-control/signing-urls-with-helpers
+
+echo $FILE
+echo $GS_URL
+echo $GCLOUD_SA_KEY
+
+FILE_URL=$GS_URL/$(basename $FILE)
+
+echo "Copying \"$FILE\" to \"$GS_URL\""
+gcloud storage cp -r $FILE $FILE_URL
+echo "Successfully uploaded!"
+
+echo "Creating signed url"
+gcloud storage sign-url \
+    --private-key-file=$GCLOUD_SA_KEY \
+    --duration=7d \
+    $FILE_URL

scripts/airgapped/README.md

@@ -0,0 +1,85 @@
+# Airgap Utility Scripts
+
+This directory contains bash and python scripts that are useful for performing
+an airgapped installation. These scripts could either be used independently
+to create airgap artifacts or via our testing scripts.
+
+We'll document some use-case scenarios here for the different scripts.
+
+## Prerequisites
+
+To use the scripts in this directory you'll need to install a couple of Python
+and Ubuntu packages on the host machine, driving the test (not the LXC machine
+that will contain the airgapped environment).
+```
+pip3 install -r requirements.txt
+sudo apt install pigz
+sudo snap install docker
+sudo snap install yq
+sudo snap install jq
+```
+
+## Get list of all images from a bundle definition
+
+Use the following script to get the list of all OCI images used by a bundle.
+This script makes the following assumptions:
+1. Every charm in the bundle has a `_github_repo_name` metadata field,
+   containing the repository name of the charm (the org is assumed to be
+   canonical)
+2. Every charm in the bundle has a `_github_repo_branch` metadata field,
+   containing the branch of the source code
+3. There is a script called `tools/get_images.sh` in each repo that gathers
+   the images for that repo
+
+```bash
+./scripts/airgapped/get-all-images.sh releases/1.7/stable/kubeflow/bundle.yaml > images.txt
+```
+
+## Pull images to docker cache
+
+We have a couple of scripts that are using `docker` commands to pull images,
+retag them and compress them in a final `tar.gz` file. Those scripts require
+that the images are already in docker's cache. This script pull a list of images
+provided by a txt file.
+
+```bash
+python3 scripts/airgapped/save-images-to-cache.py images.txt
+```
+
+## Retag images to cache
+
+In airgap environments users push their images in their own registries. So we'll
+need to rename prefixes like `docker.io` to the server that users would use.
+
+Note that this script will produce by default a `retagged-images.txt` file,
+containing the names of all re-tagged images.
+
+```bash
+python3 scripts/airgapped/retag-images-to-cache.py images.txt
+```
+
+Or if you'd like to use a different prefix, i.e. `registry.example.com`
+```bash
+python3 scripts/airgapped/retag-images-to-cache.py --new-registry=registry.example.com images.txt
+```
+
+## Save images to tar
+
+Users will need to inject the OCI images in their registry in an airgap
+environment. For this we'll be preparing a `tar.gz` file with all OCI images.
+
+```bash
+python3 scripts/airgapped/save-images-to-tar.py retagged-images.txt
+```
+
+## Save charms to tar
+
+Users in an airgap env will need to deploy charms from local files. To assist this
+we'll use this script to create a `tar.gz` containing all the charms referenced
+in a bundle.
+
+```bash
+BUNDLE_PATH=releases/1.7/stable/kubeflow/bundle.yaml
+
+python3 scripts/airgapped/save-charms-to-tar.py $BUNDLE_PATH
+```

scripts/airgapped/get-all-images.sh

@@ -0,0 +1,50 @@
+#!/usr/bin/bash
+#
+# This script parses given bundle file for github repositories and branches. Then checks out each
+# charm's repository one by one using specified branch and collects images referred by that charm
+# using that repository's image collection script
+#
+BUNDLE_FILE=$1
+IMAGES=()
+# retrieve all repositories and branches for CKF
+REPOS_BRANCHES=($(yq -r '.applications[] | select(._github_repo_name) | [(._github_repo_name, ._github_repo_branch)] | join(":")' $BUNDLE_FILE | sort --unique))
+
+# TODO: We need to not hardcode this and be able to deduce all images from the bundle
+# https://github.com/canonical/bundle-kubeflow/issues/789
+RESOURCE_DISPATCHER_BRANCH=track/1.0
+RESOURCE_DISPATCHER_REPO=https://github.com/canonical/resource-dispatcher
+
+for REPO_BRANCH in "${REPOS_BRANCHES[@]}"; do
+  IFS=: read -r REPO BRANCH <<< "$REPO_BRANCH"
+  git clone --branch $BRANCH https://github.com/canonical/$REPO
+  cd $REPO
+  IMAGES+=($(bash ./tools/get-images.sh))
+  cd - > /dev/null
+  rm -rf $REPO
+done
+
+# retrieve all repositories and branches for dependencies
+DEP_REPOS_BRANCHES=($(yq -r '.applications[] | select(._github_dependency_repo_name) | [(._github_dependency_repo_name, ._github_dependency_repo_branch)] | join(":")' $BUNDLE_FILE | sort --unique))
+
+for REPO_BRANCH in "${DEP_REPOS_BRANCHES[@]}"; do
+  IFS=: read -r REPO BRANCH <<< "$REPO_BRANCH"
+  git clone --branch $BRANCH https://github.com/canonical/$REPO
+  cd $REPO
+  # for dependencies only retrieve workload containers from metadata.yaml
+  IMAGES+=($(find -type f -name metadata.yaml -exec yq '.resources | to_entries | map(select(.value.upstream-source != null)) | .[] | .value | ."upstream-source"' {} \;))
+  cd - > /dev/null
+  rm -rf $REPO
+done
+
+# manually retrieve resource-dispatcher
+git clone --branch $RESOURCE_DISPATCHER_BRANCH $RESOURCE_DISPATCHER_REPO
+cd resource-dispatcher
+IMAGES+=($(bash ./tools/get-images.sh))
+cd ..
+rm -rf resource-dispatcher
+
+# ensure we only show unique images
+IMAGES=($(echo "${IMAGES[@]}" | tr ' ' '\n' | sort -u | tr '\n' ' '))
+
+# print full list of images
+printf "%s\n" "${IMAGES[@]}"

scripts/airgapped/prerequisites.sh

@@ -0,0 +1,22 @@
+#!/usr/bin/env bash
+set -xe
+
+SCRIPT_DIR=$( cd -- "$( dirname -- "${BASH_SOURCE[0]}" )" &> /dev/null && pwd )
+
+echo "Installing dependencies..."
+pip3 install -r $SCRIPT_DIR/requirements.txt
+sudo apt update
+
+echo "Installing Docker"
+sudo snap install docker
+sudo groupadd docker
+sudo usermod -aG docker $USER
+sudo snap disable docker
+sudo snap enable docker
+
+echo "Installing parsers"
+sudo snap install yq
+sudo snap install jq
+
+echo "Installing pigz for compression"
+sudo apt install pigz

scripts/airgapped/push-images-to-registry.py

@@ -0,0 +1,27 @@
+import argparse
+import logging
+
+import docker
+
+from utils import get_images_list_from_file
+
+docker_client = docker.client.from_env()
+
+log = logging.getLogger(__name__)
+
+
+if __name__ == "__main__":
+    parser = argparse.ArgumentParser(description="Push images from list.")
+    parser.add_argument("images")
+    args = parser.parse_args()
+
+    images_ls = get_images_list_from_file(args.images)
+    images_len = len(images_ls)
+    new_images_ls = []
+    for idx, image_nm in enumerate(images_ls):
+        log.info("%s/%s", idx + 1, images_len)
+
+        logging.info("Pushing image: %s", image_nm)
+        docker_client.images.push(image_nm)
+
+    log.info("Successfully pushed all images!")

scripts/airgapped/requirements.txt

@@ -0,0 +1,2 @@
+docker
+PyYAML

scripts/airgapped/retag-images-to-cache.py

@@ -0,0 +1,83 @@
+import argparse
+import logging
+
+import docker
+
+from utils import (delete_file_if_exists, get_images_list_from_file,
+                   get_or_pull_image)
+
+cli = docker.client.from_env()
+
+log = logging.getLogger(__name__)
+
+SHA_TOKEN = "@sha256"
+
+
+def retag_image_with_sha(image):
+    """Retag the image by using the sha value."""
+    log.info("Retagging image digest: %s", image)
+    repo_digest = image.attrs["RepoDigests"][0]
+    [repository_name, sha_value] = repo_digest.split("@sha256:")
+
+    tagged_image = "%s:%s" % (repository_name, sha_value)
+    log.info("Retagging to: %s", tagged_image)
+    image.tag(tagged_image)
+
+    log.info("Tagged image successfully: %s", tagged_image)
+    return cli.images.get(tagged_image)
+
+
+def get_retagged_image_name(image_nm: str, new_registry: str) -> str:
+    """Given an image name replace the repo and use sha as tag."""
+    if SHA_TOKEN in image_nm:
+        log.info("Provided image has sha. Using it's value as tag.")
+        image_nm = image_nm.replace(SHA_TOKEN, "")
+
+    if len(image_nm.split("/")) == 1:
+        # docker.io/library image, i.e. ubuntu:22.04
+        return "%s/%s" % (new_registry, image_nm)
+
+    if len(image_nm.split("/")) == 2:
+        # classic docker.io image, i.e. argoproj/workflow-controller
+        return "%s/%s" % (new_registry, image_nm)
+
+    # There are more than 2 / in the image name. Replace first part
+    # Example image: quay.io/metallb/speaker:v0.13.3
+    _, image_nm = image_nm.split("/", 1)
+    return "%s/%s" % (new_registry, image_nm)
+
+
+if __name__ == "__main__":
+    parser = argparse.ArgumentParser(description="Retag list of images")
+    parser.add_argument("images")
+    parser.add_argument("--new-registry", default="172.17.0.2:5000")
+    parser.add_argument("--retagged-images", default="retagged-images.txt")
+    # The reason we are using this IP as new registry is because this will end
+    # up being the IP of the Registry we'll run as a Container. We'll need to
+    # do docker push <...> so we'll have to use the IP directly, or mess with
+    # the environment's /etc/hosts file
+
+    args = parser.parse_args()
+
+    images_ls = get_images_list_from_file(args.images)
+    images_len = len(images_ls)
+    new_images_ls = []
+    for idx, image_nm in enumerate(images_ls):
+        log.info("%s/%s", idx + 1, images_len)
+
+        retagged_image_nm = get_retagged_image_name(
+            image_nm, args.new_registry
+        )
+
+        img = get_or_pull_image(image_nm)
+        log.info("%s: Retagging to %s", image_nm, retagged_image_nm)
+        img.tag(retagged_image_nm)
+
+        new_images_ls.append(retagged_image_nm)
+
+    log.info("Saving the produced list of images.")
+    delete_file_if_exists(args.retagged_images)
+    with open(args.retagged_images, "w+") as f:
+        f.write("\n".join(new_images_ls))
+
+    log.info("Successfully saved list of images in '%s'", args.retagged_images)

scripts/airgapped/save-charms-to-tar.py

@@ -0,0 +1,57 @@
+import argparse
+import logging
+import subprocess
+
+import os
+import yaml
+
+import utils as airgap_utils
+
+log = logging.getLogger(__name__)
+
+
+def download_bundle_charms(bundle: dict, no_zip: bool,
+                           skip_resource_dispatcher: bool,
+                           output_tar: str) -> None:
+    """Given a bundle dict download all the charms using juju download."""
+
+    log.info("Downloading all charms...")
+    applications = bundle.get("applications")
+    for app in applications.values():
+        subprocess.run(["juju", "download", "--channel", app["channel"],
+                        app["charm"]])
+
+    # FIXME: https://github.com/canonical/bundle-kubeflow/issues/789
+    if not skip_resource_dispatcher:
+        log.info("Fetching charm of resource-dispatcher.")
+        subprocess.run(["juju", "download", "--channel", "1.0/stable",
+                        "resource-dispatcher"])
+
+    if not no_zip:
+        # python3 download_bundle_charms.py $BUNDLE_PATH --zip_all
+        log.info("Creating the tar with all the charms...")
+        cmd = "tar -cv --use-compress-program=pigz -f %s *.charm" % output_tar
+        subprocess.run(cmd, shell=True)
+        log.info("Created %s file will all charms.", output_tar)
+
+        log.info("Removing downloaded charms...")
+        airgap_utils.delete_files_with_extension(os.getcwd(), ".charm")
+
+
+if __name__ == "__main__":
+    parser = argparse.ArgumentParser(description="Bundle Charms Downloader")
+    parser.add_argument("--no-zip", action="store_true")
+    parser.add_argument("--skip-resource-dispatcher", action="store_true")
+    parser.add_argument("--output-tar", default="charms.tar.gz")
+    parser.add_argument("bundle")
+    args = parser.parse_args()
+    log.info(args.no_zip)
+
+    bundle_dict = {}
+    with open(args.bundle, 'r') as file:
+        bundle_dict = yaml.safe_load(file)
+
+    airgap_utils.delete_file_if_exists(args.output_tar)
+    download_bundle_charms(bundle_dict, args.no_zip,
+                           args.skip_resource_dispatcher,
+                           args.output_tar)