Add a command structure to Go driver #73
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,37 @@ | ||
| package drive | ||
|
|
||
| import ( | ||
| "log" | ||
|
|
||
| "github.com/spf13/cobra" | ||
| ) | ||
|
|
||
| var cleanupCmd = &cobra.Command{ | ||
| Use: "cleanup", | ||
| Short: "Cleans up resources, containers, etc. at job completion", | ||
| Long: `The Cleanup stage is executed by "cleanup_exec". | ||
|
|
||
| This final stage is executed even if one of the previous stages failed. | ||
| The main goal for this stage is to clean up any of the environments that | ||
| might have been set up. For example, turning off VMs or deleting | ||
| containers. | ||
|
|
||
| The result of cleanup_exec does not affect job statuses. For example, a | ||
| job will be marked as successful even if the following occurs: | ||
|
|
||
| Both prepare_exec and run_exec are successful. | ||
| cleanup_exec fails. | ||
|
|
||
| The user can set cleanup_exec_timeout if they want to set some kind of | ||
| deadline of how long GitLab Runner should wait to clean up the | ||
| environment before terminating the process. | ||
|
|
||
| The STDOUT of this executable will be printed to GitLab Runner logs at a | ||
| DEBUG level. The STDERR will be printed to the logs at a WARN level. | ||
|
|
||
| Read more in GitLab's documentation: | ||
| https://docs.gitlab.com/runner/executors/custom.html#cleanup`, | ||
| Run: func(cmd *cobra.Command, args []string) { | ||
| log.Println("cleaning up...") | ||
| }, | ||
| } | ||
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,22 @@ | ||
| package drive | ||
|
|
||
| import ( | ||
| "log" | ||
|
|
||
| "github.com/spf13/cobra" | ||
| ) | ||
|
|
||
| var configCmd = &cobra.Command{ | ||
| Use: "config", | ||
| Short: "Configure various jobs settings before they run", | ||
| Long: `The Config stage is executed by "config_exec". | ||
|
|
||
| Sometimes you might want to set some settings during execution time. For | ||
| example, setting a build directory depending on the project ID. | ||
|
|
||
| For a detailed list of settings that can be configured, read more at: | ||
| https://docs.gitlab.com/runner/executors/custom.html#config.`, | ||
| Run: func(cmd *cobra.Command, args []string) { | ||
| log.Println("configuring...") | ||
| }, | ||
| } |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,50 @@ | ||
| package drive | ||
|
|
||
| import ( | ||
| "github.com/spf13/cobra" | ||
| ) | ||
|
|
||
| func init() { | ||
| cobra.EnableCommandSorting = false | ||
| DriveCmd.AddCommand(configCmd, prepareCmd, runCmd, cleanupCmd) | ||
| } | ||
|
|
||
| var DriveCmd = &cobra.Command{ | ||
| Use: "drive", | ||
|
There was a problem hiding this comment. [question] is there a planned use for calling There was a problem hiding this comment. My thinking was to segregate the I'm not super attached to the idea though. There was a problem hiding this comment. That makes sense, and yeah, since these are basically never typed I'm not opposed to the extra characters. |
||
| Short: "Drive stages requested by gitlab-runner's executor", | ||
| Long: `Drive holds subcommands to run each of executor's stages. | ||
|
|
||
| The Custom executor provides the stages for you to configure some | ||
| details of the job, prepare and clean up the environment and run the job | ||
| script within it. Each stage is responsible for specific things and has | ||
| different things to keep in mind. | ||
|
|
||
| Each stage executed by the Custom executor is executed at the time a | ||
| builtin GitLab Runner executor would execute them. | ||
|
|
||
| For each step that will be executed, specific environment variables are | ||
| exposed to the executable, which can be used to get information about the | ||
| specific Job that is running. All stages will have the following | ||
| environment variables available to them: | ||
|
|
||
| - Standard CI/CD environment variables, including predefined variables. | ||
| - All environment variables provided by the Custom executor Runner host | ||
| system. | ||
| - All services and their available settings. Exposed in JSON format as | ||
| CUSTOM_ENV_CI_JOB_SERVICES. | ||
|
|
||
| Both CI/CD environment variables and predefined variables are prefixed | ||
| with CUSTOM_ENV_ to prevent conflicts with system environment variables. | ||
| For example, CI_BUILDS_DIR will be available as CUSTOM_ENV_CI_BUILDS_DIR. | ||
|
|
||
| The stages run in the following sequence: | ||
|
|
||
| 1. config_exec | ||
| 2. prepare_exec | ||
| 3. run_exec | ||
| 4. cleanup_exec | ||
|
|
||
| Read more in GitLab's documentation: | ||
| https://docs.gitlab.com/runner/executors/custom.html#stages | ||
| `, | ||
| } | ||
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,34 @@ | ||
| package drive | ||
|
|
||
| import ( | ||
| "log" | ||
|
|
||
| "github.com/spf13/cobra" | ||
| ) | ||
|
|
||
| var prepareCmd = &cobra.Command{ | ||
| Use: "prepare", | ||
| Short: "Prepare for jobs by starting containers, services, etc.", | ||
| Long: `The Prepare stage is executed by "prepare_exec". | ||
|
|
||
| At this point, GitLab Runner knows everything about the job (where and | ||
| how it’s going to run). The only thing left is for the environment to be | ||
| set up so the job can run. Prepare will execute the steps necessary to | ||
| create that environment. | ||
|
|
||
| This is responsible for setting up the environment (for example, | ||
| creating the virtual machine or container, services or anything else). | ||
| After this is done, we expect that the environment is ready to run the | ||
| job. | ||
|
|
||
| This stage is executed only once in a job execution. | ||
|
|
||
| The STDOUT and STDERR returned from this executable will print to the | ||
| job log. | ||
|
|
||
| Read more in GitLab's documentation: | ||
| https://docs.gitlab.com/runner/executors/custom.html#prepare`, | ||
| Run: func(cmd *cobra.Command, args []string) { | ||
| log.Println("preparing...") | ||
| }, | ||
| } |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,61 @@ | ||
| package drive | ||
|
|
||
| import ( | ||
| "log" | ||
|
|
||
| "github.com/spf13/cobra" | ||
| ) | ||
|
|
||
| var runCmd = &cobra.Command{ | ||
| Use: "run", | ||
| Short: "Executes substages (e.g., cache download) and job steps", | ||
| Long: `The Run stage is executed by "run_exec". | ||
|
|
||
| The STDOUT and STDERR returned from this executable will print to the job | ||
| log. | ||
|
|
||
| Unlike the other stages, the run_exec stage is executed multiple times, | ||
| since it’s split into sub stages listed below in sequential order: | ||
|
|
||
| prepare_script | ||
| get_sources | ||
| restore_cache | ||
| download_artifacts | ||
| step_* | ||
| build_script | ||
| step_* | ||
| after_script | ||
| archive_cache OR archive_cache_on_failure | ||
| upload_artifacts_on_success OR upload_artifacts_on_failure | ||
| cleanup_file_variables | ||
|
|
||
| In GitLab Runner 14.0 and later, build_script will be replaced with | ||
| step_script. For more information, see this issue. | ||
|
|
||
| For each stage mentioned above, the run_exec executable will be executed | ||
| with (1) the path to the script that GitLab Runner creates for the Custom | ||
| executor to run, and (2) the name of the stage. | ||
|
|
||
| This executable should be responsible for executing the scripts that are | ||
| specified in the first argument. They contain all the scripts any GitLab | ||
| Runner executor would run normally to clone, download artifacts, run | ||
| user scripts and all the other steps described below. The scripts can be | ||
| of the following shells: | ||
|
|
||
| Bash | ||
| PowerShell Desktop | ||
| PowerShell Core | ||
| Batch (deprecated) | ||
|
|
||
| We generate the script using the shell configured by shell inside of | ||
| [[runners]]. If none is provided the defaults for the OS platform are used. | ||
|
|
||
| The table below is a detailed explanation of what each script does and | ||
| what the main goal of that script is. | ||
|
|
||
| Read more in GitLab's documentation: | ||
| https://docs.gitlab.com/runner/executors/custom.html#run`, | ||
| Run: func(cmd *cobra.Command, args []string) { | ||
| log.Println("running...") | ||
| }, | ||
| } |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,30 @@ | ||
| package cmd | ||
|
|
||
| import ( | ||
| "fmt" | ||
| "os" | ||
|
|
||
| "github.com/GSA-TTS/gitlab-runner-cloudgov/runner/cf-driver/cmd/drive" | ||
|
|
||
| "github.com/spf13/cobra" | ||
| ) | ||
|
|
||
| func init() { | ||
| rootCmd.AddCommand(drive.DriveCmd) | ||
| } | ||
|
|
||
| var rootCmd = &cobra.Command{ | ||
| Use: "cfd", | ||
|
There was a problem hiding this comment. [question] - the docs mention There was a problem hiding this comment. Good catch, I haven't been using the actual executable much & didn't think of it. |
||
| Short: "CloudFoundry Driver", | ||
| Long: `This is CloudFoundry Driver for the GitLab Custom executor. | ||
|
|
||
| The gitlab-runner service should run cfd with it's "drive" subcommands, | ||
| e.g., "cfd drive prepare".`, | ||
| } | ||
|
|
||
| func Execute() { | ||
| if err := rootCmd.Execute(); err != nil { | ||
| fmt.Println(err) | ||
| os.Exit(1) | ||
| } | ||
| } | ||
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
ooooh - we should replace the current
CUSTOM_ENV_PRESERVE_*behavior with a long (1 hour?) timeout instead of just skippingThere was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Mm yeah that would be good. An hour feels long to me, but I'm supposing it must not to you?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Oh I misunderstood you when I first read this and thought you were just talking about adding a timeout to make sure broken jobs get cleaned up.
How are you thinking this would work? We could put something like a long sleep in the run step? It does seem like an easy way to take care of at least half of #34.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Ideally it'd be something like what circleCI does, which is 10 minutes + whatever time you're ssh'd in to the runner to do your debugging. 1 hour was assuming that we wouldn't be able to have a dynamic timeout like that.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
🤷🏻 this might also be a misinterpretation of the
cleanup_exec_timeoutdocs that triggered this whole thread. I was thinking that we could set that config and gitlab would take care of not calling cleanup right away, but that's probably wrong.In any case, it was a thought for 34 or similar, not this PR
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think I misunderstood it… I was thinking it was a timeout on how long it would wait to kill
run, but it's a timeout on how long it will wait to killcleanup. Every stage butrunhas one of the timeouts, andrun's time is managed in GitLab's settings globally and per runner.But I do think we could use it to do the debugging
PRESERVE's better: