Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

plugincontainer: Fix rootless tests #123

Draft
wants to merge 1 commit into
base: main
Choose a base branch
from

Conversation

tomhjp
Copy link
Contributor

@tomhjp tomhjp commented Apr 4, 2024

It seems like gVisor fixed a bug where a file's capabilities weren't being properly applied when creating the process: google/gvisor@586c38d

I haven't fully finished investigating yet, but I think it's the above commit that caused the tests to start failing once it got released and used in CI.

It seems like gVisor fixed a bug where a file's capabilities weren't being properly
applied when creating the process: google/gvisor@586c38d

I haven't fully finished investigating yet, but I think it's the above commit that
caused the tests to start failing once it got released and used in CI.
@tomhjp
Copy link
Contributor Author

tomhjp commented May 16, 2024

In case it's helpful to anyone picking this up, I wrote a lima config file to help quickly spin up a VM that is ready to test this:

default.yaml

# Based on https://github.com/lima-vm/lima/blob/9d31f2a7ee7c6a699298c66d80f26ad2d39fae76/examples/docker.yaml

# A template to use Docker instead of containerd & nerdctl
# $ limactl start ./docker.yaml
# $ limactl shell docker docker run -it -v $HOME:$HOME --rm alpine

# To run `docker` on the host (assumes docker-cli is installed):
# $ export DOCKER_HOST=$(limactl list docker --format 'unix://{{.Dir}}/sock/docker.sock')
# $ docker ...

# This template requires Lima v0.8.0 or later
images:
# Try to use release-yyyyMMdd image if available. Note that release-yyyyMMdd will be removed after several months.
- location: "https://cloud-images.ubuntu.com/releases/22.04/release-20231026/ubuntu-22.04-server-cloudimg-amd64.img"
  arch: "x86_64"
  digest: "sha256:054db2d88c454bb0ad8dfd8883955e3946b57d2b0bf0d023f3ade3c93cdd14e5"
- location: "https://cloud-images.ubuntu.com/releases/22.04/release-20231026/ubuntu-22.04-server-cloudimg-arm64.img"
  arch: "aarch64"
  digest: "sha256:eafa7742ce5ff109222ea313d31ea366d587b4e89b900b11d8285ae775dfe8c3"

cpus: 8
memory: 16GiB
mounts:
- location: "~"
  writable: true
- location: "/tmp/lima"
  writable: true
# containerd is managed by Docker, not by Lima, so the values are set to false here.
containerd:
  system: false
  user: false
provision:
- mode: system
  script: |
    #!/bin/bash
    apt-get install -y jq make
- mode: system
  # This script defines the host.docker.internal hostname when hostResolver is disabled.
  # It is also needed for lima 0.8.2 and earlier, which does not support hostResolver.hosts.
  # Names defined in /etc/hosts inside the VM are not resolved inside containers when
  # using the hostResolver; use hostResolver.hosts instead (requires lima 0.8.3 or later).
  script: |
    #!/bin/sh
    sed -i 's/host.lima.internal.*/host.lima.internal host.docker.internal/' /etc/hosts
# Install docker packages
- mode: system
  script: |
    #!/bin/bash
    set -eux -o pipefail
    command -v docker >/dev/null 2>&1 && exit 0
    export DEBIAN_FRONTEND=noninteractive
    curl -fsSL https://get.docker.com | sh
    # NOTE: you may remove the lines below, if you prefer to use rootful docker, not rootless
    systemctl disable --now docker
    apt-get install -y uidmap dbus-user-session
# Setup rootless docker
- mode: user
  script: |
    #!/bin/bash
    set -eux -o pipefail
    systemctl --user start dbus
    dockerd-rootless-setuptool.sh install
    docker context use rootless
# Install gVisor (runsc)
- mode: user
  script: |
    mkdir -p "$HOME/bin"
    (
      set -e
      ARCH="$(uname -m)"
      URL="https://storage.googleapis.com/gvisor/releases/release/latest/${ARCH}"
      wget --quiet "${URL}/runsc" "${URL}/runsc.sha512" \
        "${URL}/containerd-shim-runsc-v1" "${URL}/containerd-shim-runsc-v1.sha512"
      sha512sum -c runsc.sha512 \
        -c containerd-shim-runsc-v1.sha512
      rm -f -- *.sha512
      chmod a+rx runsc containerd-shim-runsc-v1
      mv runsc containerd-shim-runsc-v1 "$HOME/bin"
    )
    mkdir -p ~/.config/docker/
    tee ~/.config/docker/daemon.json <<EOF
    {
      "runtimes": {
        "runsc": {
          "path": "$HOME/bin/runsc",
          "runtimeArgs": [
            "--host-uds=create",
            "--ignore-cgroups"
          ]
        }
      }
    }
    EOF
    systemctl --user restart docker
# Install go
- mode: user
  script: |
    set -euo pipefail
    VERSION="$(curl --silent https://go.dev/dl/?mode=json | jq -r '.[0].version')"
    wget "https://dl.google.com/go/${VERSION}.linux-arm64.tar.gz"
    mkdir -p "$HOME/bin"
    rm -rf "$HOME/bin/go" && tar -C "$HOME/bin" -xzf "${VERSION}.linux-arm64.tar.gz"
    echo '[ "${PATH#*$HOME/bin/go/bin}" == "$PATH" ] && export PATH="$PATH:$HOME/bin/go/bin"' >> "${HOME}/.bashrc"
probes:
- script: |
    #!/bin/bash
    set -eux -o pipefail
    if ! timeout 30s bash -c "until command -v docker >/dev/null 2>&1; do sleep 3; done"; then
      echo >&2 "docker is not installed yet"
      exit 1
    fi
    if ! timeout 30s bash -c "until pgrep rootlesskit; do sleep 3; done"; then
      echo >&2 "rootlesskit (used by rootless docker) is not running"
      exit 1
    fi
  hint: See "/var/log/cloud-init-output.log". in the guest
hostResolver:
  # hostResolver.hosts requires lima 0.8.3 or later. Names defined here will also
  # resolve inside containers, and not just inside the VM itself.
  hosts:
    host.docker.internal: host.lima.internal
portForwards:
- guestSocket: "/run/user/{{.UID}}/docker.sock"
  hostSocket: "{{.Dir}}/sock/docker.sock"
message: |
  cd to the Vault repo on your host, run `lima` to enter the VM, and then:
  make dev # Optional: Pull all the dependencies to populate the VM's Go module cache etc.

  go test -v -count=1 -run="^TestExternalPluginInContainer_MountAndUnmount/rootless_runsc$" github.com/hashicorp/vault/vault

tvoran added a commit that referenced this pull request Jul 25, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant