Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add --log-file argument and log test result details #54

Open
wants to merge 10 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 9 additions & 3 deletions cmd/validate.go
Original file line number Diff line number Diff line change
Expand Up @@ -34,11 +34,11 @@ var validateCmd = &cobra.Command{
SilenceUsage: true,
SilenceErrors: false,
RunE: func(cmd *cobra.Command, args []string) error {
file, err := cmd.Flags().GetString("file")
configPath, err := cmd.Flags().GetString("file")
if err != nil {
return err
}
if file == "" {
if configPath == "" {
return errors.New("you must specify a manifest with '--file path/url'")
}

Expand All @@ -48,7 +48,12 @@ var validateCmd = &cobra.Command{
return err
}

v, err := validator.Validate(image, file, cmd, debug)
logPath, err := cmd.Flags().GetString("log-file")
if err != nil {
return err
}

v, err := validator.Validate(image, configPath, logPath, cmd, debug)
if err != nil {
cmd.Printf("Error: %s\n", err.Error())
return err
Expand Down Expand Up @@ -85,6 +90,7 @@ func imageArg(cmd *cobra.Command, args []string) error {
func init() {
rootCmd.AddCommand(validateCmd)
validateCmd.PersistentFlags().String("file", "", "Path or URL of a manifest to validate against.")
validateCmd.PersistentFlags().String("log-file", "", "File to print log output to.")
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why not use stdout? That gives more flexibility for redirection.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I believe stdout is already consumed by the user-readable output. Writing to a separate log file allows us to get separate machine-readable output, and CI can always cat the file if we want the developer to see it (that's what I plan on doing).

validateCmd.PersistentFlags().Bool("debug", false, "Keep container running on failure for debugging.")

}
4 changes: 2 additions & 2 deletions examples/kubeflow.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -40,7 +40,7 @@ checks:
description: 🌏 Exposes an HTTP interface on port 8888
probe:
httpGet:
path: /
path: /hub/jovyan
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm curious about the motivation here. This feels orthogonal to the rest of this PR.

It's my understanding that Kubeflow only expects a service on port 8888, not any particular path on that web server. You could equally run VSCode Server or some other application and expect it to work.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This PR has some bug fixes and tests too. One of the bugs was that it wasn't checking the status code when doing httpGet. I fixed that, but as a result this test started failing (I don't remember what the return code was). GETting a different path got a successful status code.

port: 8888
failureThreshold: 30
- name: NB_PREFIX
Expand All @@ -54,7 +54,7 @@ checks:
description: "🔓 Sets 'Access-Control-Allow-Origin: *' header"
probe:
httpGet:
path: /
path: /hub/jovyan
port: 8888
httpHeaders:
- name: User-Agent
Expand Down
2 changes: 2 additions & 0 deletions go.mod
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,7 @@ require (
github.com/charmbracelet/bubbletea v0.25.0
github.com/charmbracelet/lipgloss v0.5.0
github.com/google/uuid v1.3.0
github.com/rs/zerolog v1.32.0
github.com/spf13/cobra v1.3.0
github.com/stretchr/testify v1.7.0
golang.org/x/text v0.14.0
Expand All @@ -26,6 +27,7 @@ require (
github.com/inconshreveable/mousetrap v1.0.0 // indirect
github.com/json-iterator/go v1.1.12 // indirect
github.com/lucasb-eyer/go-colorful v1.2.0 // indirect
github.com/mattn/go-colorable v0.1.13 // indirect
github.com/mattn/go-isatty v0.0.20 // indirect
github.com/mattn/go-localereader v0.0.1 // indirect
github.com/mattn/go-runewidth v0.0.15 // indirect
Expand Down
10 changes: 10 additions & 0 deletions go.sum
Original file line number Diff line number Diff line change
Expand Up @@ -107,6 +107,7 @@ github.com/containerd/console v1.0.4 h1:F2g4+oChYvBTsASRTz8NP6iIAi97J3TtSAsLbIFn
github.com/containerd/console v1.0.4/go.mod h1:YynlIjWYF8myEu6sdkwKIvGQq+cOckRm6So2avqoYAk=
github.com/coreos/go-semver v0.3.0/go.mod h1:nnelYz7RCh+5ahJtPPxZlU+153eP4D4r3EedlOD2RNk=
github.com/coreos/go-systemd/v22 v22.3.2/go.mod h1:Y58oyj3AT4RCenI/lSvhwexgC+NSVTIJ3seZv2GcEnc=
github.com/coreos/go-systemd/v22 v22.5.0/go.mod h1:Y58oyj3AT4RCenI/lSvhwexgC+NSVTIJ3seZv2GcEnc=
github.com/cpuguy83/go-md2man/v2 v2.0.1/go.mod h1:tgQtvFlXSQOSOSIRvRPT7W67SCa46tRHOmNcaadrF8o=
github.com/creack/pty v1.1.9/go.mod h1:oKZEueFk5CKHvIhNR5MUki03XCEU+Q6VDXinZuGJ33E=
github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
Expand Down Expand Up @@ -303,13 +304,17 @@ github.com/mattn/go-colorable v0.1.4/go.mod h1:U0ppj6V5qS13XJ6of8GYAs25YV2eR4EVc
github.com/mattn/go-colorable v0.1.6/go.mod h1:u6P/XSegPjTcexA+o6vUJrdnUu04hMope9wVRipJSqc=
github.com/mattn/go-colorable v0.1.9/go.mod h1:u6P/XSegPjTcexA+o6vUJrdnUu04hMope9wVRipJSqc=
github.com/mattn/go-colorable v0.1.12/go.mod h1:u5H1YNBxpqRaxsYJYSkiCWKzEfiAb1Gb520KVy5xxl4=
github.com/mattn/go-colorable v0.1.13 h1:fFA4WZxdEF4tXPZVKMLwD8oUnCTTo08duU7wxecdEvA=
github.com/mattn/go-colorable v0.1.13/go.mod h1:7S9/ev0klgBDR4GtXTXX8a3vIGJpMovkB8vQcUbaXHg=
github.com/mattn/go-isatty v0.0.3/go.mod h1:M+lRXTBqGeGNdLjl/ufCoiOlB5xdOkqRJdNxMWT7Zi4=
github.com/mattn/go-isatty v0.0.8/go.mod h1:Iq45c/XA43vh69/j3iqttzPXn0bhXyGjM0Hdxcsrc5s=
github.com/mattn/go-isatty v0.0.10/go.mod h1:qgIWMr58cqv1PHHyhnkY9lrL7etaEgOFcMEpPG5Rm84=
github.com/mattn/go-isatty v0.0.11/go.mod h1:PhnuNfih5lzO57/f3n+odYbM4JtupLOxQOAqxQCu2WE=
github.com/mattn/go-isatty v0.0.12/go.mod h1:cbi8OIDigv2wuxKPP5vlRcQ1OAZbq2CE4Kysco4FUpU=
github.com/mattn/go-isatty v0.0.13/go.mod h1:cbi8OIDigv2wuxKPP5vlRcQ1OAZbq2CE4Kysco4FUpU=
github.com/mattn/go-isatty v0.0.14/go.mod h1:7GGIvUiUoEMVVmxf/4nioHXj79iQHKdU27kJ6hsGG94=
github.com/mattn/go-isatty v0.0.16/go.mod h1:kYGgaQfpe5nmfYZH+SKPsOc2e4SrIfOl2e/yFXSvRLM=
github.com/mattn/go-isatty v0.0.19/go.mod h1:W+V8PltTTMOvKvAeJH7IuucS94S2C6jfK/D7dTCTo3Y=
github.com/mattn/go-isatty v0.0.20 h1:xfD0iDuEKnDkl03q4limB+vH+GxLEtL/jb4xVJSWWEY=
github.com/mattn/go-isatty v0.0.20/go.mod h1:W+V8PltTTMOvKvAeJH7IuucS94S2C6jfK/D7dTCTo3Y=
github.com/mattn/go-localereader v0.0.1 h1:ygSAOl7ZXTx4RdPYinUpg6W99U8jWvWi9Ye2JC/oIi4=
Expand Down Expand Up @@ -391,6 +396,9 @@ github.com/rivo/uniseg v0.4.7 h1:WUdvkW8uEhrYfLC4ZzdpI2ztxP1I582+49Oc5Mq64VQ=
github.com/rivo/uniseg v0.4.7/go.mod h1:FN3SvrM+Zdj16jyLfmOkMNblXMcoc8DfTHruCPUcx88=
github.com/rogpeppe/fastuuid v1.2.0/go.mod h1:jVj6XXZzXRy/MSR5jhDC/2q6DgLz+nrA6LYCDYWNEvQ=
github.com/rogpeppe/go-internal v1.3.0/go.mod h1:M8bDsm7K2OlrFYOpmOWEs/qY81heoFRclV5y23lUDJ4=
github.com/rs/xid v1.5.0/go.mod h1:trrq9SKmegXys3aeAKXMUTdJsYXVwGY3RLcfgqegfbg=
github.com/rs/zerolog v1.32.0 h1:keLypqrlIjaFsbmJOBdB/qvyF8KEtCWHwobLp5l/mQ0=
github.com/rs/zerolog v1.32.0/go.mod h1:/7mN4D5sKwJLZQ2b/znpjC3/GQWY/xaDXUM0kKWRHss=
github.com/russross/blackfriday/v2 v2.1.0/go.mod h1:+Rmxgy9KzJVeS9/2gXHxylqXiyQDYRxCVz55jmeOWTM=
github.com/ryanuber/columnize v0.0.0-20160712163229-9b3edd62028f/go.mod h1:sm1tb6uqfes/u+d4ooFouqFdy9/2g9QGwK3SQygK0Ts=
github.com/sagikazarmark/crypt v0.3.0/go.mod h1:uD/D+6UF4SrIR1uGEv7bBNkNqLGqUr43MRiaGWX1Nig=
Expand Down Expand Up @@ -635,8 +643,10 @@ golang.org/x/sys v0.0.0-20210927094055-39ccf1dd6fa6/go.mod h1:oPkhp1MJrh7nUepCBc
golang.org/x/sys v0.0.0-20211007075335-d3039528d8ac/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.0.0-20211124211545-fe61309f8881/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.0.0-20211205182925-97ca703d548d/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.0.0-20220811171246-fbc7d0a398ab/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.1.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.6.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.12.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.18.0 h1:DBdB3niSjOA/O0blCZBqDefyWNYveAYMNF1Wum0DYQ4=
golang.org/x/sys v0.18.0/go.mod h1:/VUhepiaJMQUp4+oa/7Zr1D23ma6VTLIYjOOTFZPUcA=
golang.org/x/term v0.0.0-20201126162022-7de9c90e9dd1/go.mod h1:bj7SfCRtBDWHUb9snDiAeCFNEtKQo2Wmx5Cou7ajbmo=
Expand Down
2 changes: 1 addition & 1 deletion internal/config/load_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -46,7 +46,7 @@ func TestValidator(t *testing.T) {
assert.Equal("allow-origin-all", check.Name)
assert.Equal("🔓 Sets 'Access-Control-Allow-Origin: *' header", check.Description)

assert.Equal("/", check.Probe.HTTPGet.Path)
assert.Equal("/hub/jovyan", check.Probe.HTTPGet.Path)
assert.Equal(8888, check.Probe.HTTPGet.Port)

header := check.Probe.HTTPGet.HTTPHeaders[0]
Expand Down
2 changes: 1 addition & 1 deletion internal/container/container.go
Original file line number Diff line number Diff line change
Expand Up @@ -40,7 +40,7 @@ type ContainerInterface interface {
Start() error
Remove() error
Status() (*ContainerInfo, error)
Exec(command ...string) (string, error)
Exec(command ...string) (exitCode int, stdout string, stderr string, err error)
Logs() (string, error)
}

Expand Down
38 changes: 35 additions & 3 deletions internal/container/docker.go
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,7 @@ import (
"encoding/json"
"errors"
"fmt"
"io"
"os/exec"
"strings"
"time"
Expand Down Expand Up @@ -136,11 +137,42 @@ func (c DockerContainer) Status() (*ContainerInfo, error) {
}

// Exec a command inside a container
func (c DockerContainer) Exec(command ...string) (string, error) {
func (c DockerContainer) Exec(command ...string) (exitCode int, stdout string, stderr string, err error) {

args := append([]string{"exec", c.Name}, command...)
out, err := exec.Command("docker", args...).Output()
return string(out), err
cmd := exec.Command("docker", args...)
stdoutPipe, err := cmd.StdoutPipe()
if err != nil {
return
}
stderrPipe, err := cmd.StderrPipe()
if err != nil {
return
}

if err = cmd.Start(); err != nil {
return
}

stdoutBytes, err := io.ReadAll(stdoutPipe)
if err != nil {
return
}
stderrBytes, err := io.ReadAll(stderrPipe)
if err != nil {
return
}

if err = cmd.Wait(); err != nil {
switch t := err.(type) {
case *exec.ExitError:
exitCode = t.ExitCode()
default:
return
}
}

return exitCode, string(stdoutBytes), string(stderrBytes), nil
}

// Get container logs
Expand Down
9 changes: 8 additions & 1 deletion internal/container/docker_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -58,15 +58,22 @@ func TestDockerContainer(t *testing.T) {
}
assert.Contains(status.RunCommand, "docker run", "Run command not stored correctly")

uname, err := c.Exec("uname", "-a")
exitCode, uname, _, err := c.Exec("uname", "-a")
if err != nil {
t.Errorf("Failed to exec command in container: %s", err.Error())
return
}
assert.Equal(0, exitCode)
if !strings.Contains(uname, "Linux") {
t.Error("Output for command 'uname' did not contain expected string 'Linux'")
return
}

exitCode, stdout, stderr, err := c.Exec("sh", "-c", `echo "This is stdout" && echo "This is stderr" >&2 && exit 42`)
assert.NoError(err)
assert.Equal("This is stdout\n", stdout)
assert.Equal("This is stderr\n", stderr)
assert.Equal(42, exitCode)
}
func TestDockerContainerRemoves(t *testing.T) {
c := New("nginx", nil, nil, nil, nil)
Expand Down
2 changes: 1 addition & 1 deletion internal/testdata/containers/kubeflow.Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -13,4 +13,4 @@ ENV PATH=/home/jovyan/.local/bin:$PATH

RUN pip install jupyterlab

CMD jupyter lab --ip=0.0.0.0
CMD jupyter lab --ip=0.0.0.0 --ServerApp.allow_origin="*" --ServerApp.base_url="$NB_PREFIX"
2 changes: 1 addition & 1 deletion internal/testdata/containers/kubeflow_broken.Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -16,4 +16,4 @@ ENV PATH=/home/bob/.local/bin:$PATH

RUN pip install jupyterlab

CMD jupyter lab --ip=0.0.0.0
CMD jupyter lab --ip=0.0.0.0 --ServerApp.allow_origin="*" --ServerApp.base_url="$NB_PREFIX"
11 changes: 7 additions & 4 deletions internal/validator/exec.go
Original file line number Diff line number Diff line change
Expand Up @@ -18,15 +18,18 @@
package validator

import (
"github.com/rs/zerolog"

canaryv1 "github.com/nvidia/container-canary/internal/apis/v1"
"github.com/nvidia/container-canary/internal/container"
)

func ExecCheck(c container.ContainerInterface, probe *canaryv1.Probe) (bool, error) {
func ExecCheck(c container.ContainerInterface, probe *canaryv1.Probe, e *zerolog.Event) (bool, error) {
action := probe.Exec
_, err := c.Exec(action.Command...)
exitCode, stdout, stderr, err := c.Exec(action.Command...)
if err != nil {
return false, nil
return false, err
}
return true, nil
e.Int("exitCode", exitCode).Str("stdout", stdout).Str("stderr", stderr)
return exitCode == 0, nil
}
84 changes: 84 additions & 0 deletions internal/validator/exec_test.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,84 @@
/*
* SPDX-FileCopyrightText: Copyright (c) <2024> NVIDIA CORPORATION & AFFILIATES. All rights reserved.
* SPDX-License-Identifier: Apache-2.0
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/

package validator

import (
"testing"

canaryv1 "github.com/nvidia/container-canary/internal/apis/v1"
"github.com/stretchr/testify/assert"
v1 "k8s.io/api/core/v1"
)

func TestExecSuccess(t *testing.T) {
assert := assert.New(t)

c := &dummyContainer{}
probe := &canaryv1.Probe{
Exec: &v1.ExecAction{
Command: []string{"success"},
},
}

logger, buf := logger()
e := logger.Info()
result, err := ExecCheck(c, probe, e)
e.Send()

assert.True(result)
assert.NoError(err)
assert.Equal("{\"level\":\"info\",\"exitCode\":0,\"stdout\":\"Success stdout\",\"stderr\":\"Success stderr\"}\n", buf.String())
}

func TestExecFailure(t *testing.T) {
assert := assert.New(t)

c := &dummyContainer{}
probe := &canaryv1.Probe{
Exec: &v1.ExecAction{
Command: []string{"failure"},
},
}

logger, buf := logger()
e := logger.Info()
result, err := ExecCheck(c, probe, e)
e.Send()
assert.False(result)
assert.NoError(err)
assert.Equal("{\"level\":\"info\",\"exitCode\":1,\"stdout\":\"Failure stdout\",\"stderr\":\"Failure stderr\"}\n", buf.String())
}

func TestExecError(t *testing.T) {
assert := assert.New(t)

c := &dummyContainer{}
probe := &canaryv1.Probe{
Exec: &v1.ExecAction{
Command: []string{"error"},
},
}

logger, buf := logger()
e := logger.Info()
result, err := ExecCheck(c, probe, e)
e.Send()
assert.False(result)
assert.Error(err, "This command is an error")
assert.Equal("{\"level\":\"info\"}\n", buf.String())
}
17 changes: 15 additions & 2 deletions internal/validator/httpget.go
Original file line number Diff line number Diff line change
Expand Up @@ -24,9 +24,10 @@ import (

canaryv1 "github.com/nvidia/container-canary/internal/apis/v1"
"github.com/nvidia/container-canary/internal/container"
"github.com/rs/zerolog"
)

func HTTPGetCheck(c container.ContainerInterface, probe *canaryv1.Probe) (bool, error) {
func HTTPGetCheck(c container.ContainerInterface, probe *canaryv1.Probe, e *zerolog.Event) (bool, error) {
action := probe.HTTPGet
client := &http.Client{}
req, err := http.NewRequest("GET", fmt.Sprintf("http://localhost:%d%s", action.Port, action.Path), nil)
Expand All @@ -39,14 +40,26 @@ func HTTPGetCheck(c container.ContainerInterface, probe *canaryv1.Probe) (bool,
req.Header.Set(header.Name, header.Value)
}
resp, err := client.Do(req)
if resp != nil {
headers := zerolog.Dict()
for name, value := range resp.Header {
headers.Str(name, strings.Join(value[:], ""))
}
e.Dict("headers", headers).Int("status", resp.StatusCode)
}
if err != nil {
return false, nil
}
if resp.StatusCode < 200 || resp.StatusCode > 299 {
return false, nil
}
for _, header := range action.ResponseHTTPHeaders {
if val, ok := resp.Header[header.Name]; ok {
if val := resp.Header.Values(header.Name); len(val) != 0 {
if header.Value != strings.Join(val[:], "") {
return false, nil
}
} else {
return false, nil
KyleFromNVIDIA marked this conversation as resolved.
Show resolved Hide resolved
}
}
defer resp.Body.Close()
Expand Down
Loading
Loading