-
Notifications
You must be signed in to change notification settings - Fork 24
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
plugincontainer: Fix rootless tests #123
Draft
tomhjp
wants to merge
1
commit into
main
Choose a base branch
from
plugincontainer-fix-rootless-tests
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Draft
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
It seems like gVisor fixed a bug where a file's capabilities weren't being properly applied when creating the process: google/gvisor@586c38d I haven't fully finished investigating yet, but I think it's the above commit that caused the tests to start failing once it got released and used in CI.
In case it's helpful to anyone picking this up, I wrote a lima config file to help quickly spin up a VM that is ready to test this: default.yaml
# Based on https://github.com/lima-vm/lima/blob/9d31f2a7ee7c6a699298c66d80f26ad2d39fae76/examples/docker.yaml
# A template to use Docker instead of containerd & nerdctl
# $ limactl start ./docker.yaml
# $ limactl shell docker docker run -it -v $HOME:$HOME --rm alpine
# To run `docker` on the host (assumes docker-cli is installed):
# $ export DOCKER_HOST=$(limactl list docker --format 'unix://{{.Dir}}/sock/docker.sock')
# $ docker ...
# This template requires Lima v0.8.0 or later
images:
# Try to use release-yyyyMMdd image if available. Note that release-yyyyMMdd will be removed after several months.
- location: "https://cloud-images.ubuntu.com/releases/22.04/release-20231026/ubuntu-22.04-server-cloudimg-amd64.img"
arch: "x86_64"
digest: "sha256:054db2d88c454bb0ad8dfd8883955e3946b57d2b0bf0d023f3ade3c93cdd14e5"
- location: "https://cloud-images.ubuntu.com/releases/22.04/release-20231026/ubuntu-22.04-server-cloudimg-arm64.img"
arch: "aarch64"
digest: "sha256:eafa7742ce5ff109222ea313d31ea366d587b4e89b900b11d8285ae775dfe8c3"
cpus: 8
memory: 16GiB
mounts:
- location: "~"
writable: true
- location: "/tmp/lima"
writable: true
# containerd is managed by Docker, not by Lima, so the values are set to false here.
containerd:
system: false
user: false
provision:
- mode: system
script: |
#!/bin/bash
apt-get install -y jq make
- mode: system
# This script defines the host.docker.internal hostname when hostResolver is disabled.
# It is also needed for lima 0.8.2 and earlier, which does not support hostResolver.hosts.
# Names defined in /etc/hosts inside the VM are not resolved inside containers when
# using the hostResolver; use hostResolver.hosts instead (requires lima 0.8.3 or later).
script: |
#!/bin/sh
sed -i 's/host.lima.internal.*/host.lima.internal host.docker.internal/' /etc/hosts
# Install docker packages
- mode: system
script: |
#!/bin/bash
set -eux -o pipefail
command -v docker >/dev/null 2>&1 && exit 0
export DEBIAN_FRONTEND=noninteractive
curl -fsSL https://get.docker.com | sh
# NOTE: you may remove the lines below, if you prefer to use rootful docker, not rootless
systemctl disable --now docker
apt-get install -y uidmap dbus-user-session
# Setup rootless docker
- mode: user
script: |
#!/bin/bash
set -eux -o pipefail
systemctl --user start dbus
dockerd-rootless-setuptool.sh install
docker context use rootless
# Install gVisor (runsc)
- mode: user
script: |
mkdir -p "$HOME/bin"
(
set -e
ARCH="$(uname -m)"
URL="https://storage.googleapis.com/gvisor/releases/release/latest/${ARCH}"
wget --quiet "${URL}/runsc" "${URL}/runsc.sha512" \
"${URL}/containerd-shim-runsc-v1" "${URL}/containerd-shim-runsc-v1.sha512"
sha512sum -c runsc.sha512 \
-c containerd-shim-runsc-v1.sha512
rm -f -- *.sha512
chmod a+rx runsc containerd-shim-runsc-v1
mv runsc containerd-shim-runsc-v1 "$HOME/bin"
)
mkdir -p ~/.config/docker/
tee ~/.config/docker/daemon.json <<EOF
{
"runtimes": {
"runsc": {
"path": "$HOME/bin/runsc",
"runtimeArgs": [
"--host-uds=create",
"--ignore-cgroups"
]
}
}
}
EOF
systemctl --user restart docker
# Install go
- mode: user
script: |
set -euo pipefail
VERSION="$(curl --silent https://go.dev/dl/?mode=json | jq -r '.[0].version')"
wget "https://dl.google.com/go/${VERSION}.linux-arm64.tar.gz"
mkdir -p "$HOME/bin"
rm -rf "$HOME/bin/go" && tar -C "$HOME/bin" -xzf "${VERSION}.linux-arm64.tar.gz"
echo '[ "${PATH#*$HOME/bin/go/bin}" == "$PATH" ] && export PATH="$PATH:$HOME/bin/go/bin"' >> "${HOME}/.bashrc"
probes:
- script: |
#!/bin/bash
set -eux -o pipefail
if ! timeout 30s bash -c "until command -v docker >/dev/null 2>&1; do sleep 3; done"; then
echo >&2 "docker is not installed yet"
exit 1
fi
if ! timeout 30s bash -c "until pgrep rootlesskit; do sleep 3; done"; then
echo >&2 "rootlesskit (used by rootless docker) is not running"
exit 1
fi
hint: See "/var/log/cloud-init-output.log". in the guest
hostResolver:
# hostResolver.hosts requires lima 0.8.3 or later. Names defined here will also
# resolve inside containers, and not just inside the VM itself.
hosts:
host.docker.internal: host.lima.internal
portForwards:
- guestSocket: "/run/user/{{.UID}}/docker.sock"
hostSocket: "{{.Dir}}/sock/docker.sock"
message: |
cd to the Vault repo on your host, run `lima` to enter the VM, and then:
make dev # Optional: Pull all the dependencies to populate the VM's Go module cache etc.
go test -v -count=1 -run="^TestExternalPluginInContainer_MountAndUnmount/rootless_runsc$" github.com/hashicorp/vault/vault |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
It seems like gVisor fixed a bug where a file's capabilities weren't being properly applied when creating the process: google/gvisor@586c38d
I haven't fully finished investigating yet, but I think it's the above commit that caused the tests to start failing once it got released and used in CI.