diff --git a/platform_versioned_docs/version-23.3/cli/_images/tw-info.jpg b/platform_versioned_docs/version-23.3/cli/_images/tw-info.jpg new file mode 100644 index 000000000..c5c9ae9e7 Binary files /dev/null and b/platform_versioned_docs/version-23.3/cli/_images/tw-info.jpg differ diff --git a/platform_versioned_docs/version-23.3/cli/_images/tw-screenshot.png b/platform_versioned_docs/version-23.3/cli/_images/tw-screenshot.png new file mode 100644 index 000000000..0cce6f21e Binary files /dev/null and b/platform_versioned_docs/version-23.3/cli/_images/tw-screenshot.png differ diff --git a/platform_versioned_docs/version-23.3/cli/cli.mdx b/platform_versioned_docs/version-23.3/cli/cli.mdx deleted file mode 100644 index 140dc53ac..000000000 --- a/platform_versioned_docs/version-23.3/cli/cli.mdx +++ /dev/null @@ -1,30 +0,0 @@ ---- -title: "Seqera Platform CLI" -description: "Instructions to use Seqera CLI." -date: "24 Apr 2023" -tags: [cli, tower] ---- - -# Seqera Platform CLI - -The command line is called `tw`. This allows you to manage Seqera Platform elements like pipelines, actions, and compute environments in the terminal. - -Seqera Platform is a full-stack application for the management of data pipelines and compute resources. It enables collaborative data analysis at scale, on-premises or in any cloud. - -The CLI interacts with Seqera, providing an interface to launch pipelines, manage cloud resources, and administer your analysis. - -![tw](../_images/tw_cli.png) - -### Key features - -- **A Nextflow-like experience**: The CLI provides a developer-friendly environment. Pipelines can be launched with the CLI similar to Nextflow, but with the benefits of the Seqera Platform which provides monitoring, logging, resource provisioning, dataset management, and collaborative sharing. - -- **Infrastructure as code**: All Seqera resources, including pipelines and compute environments, can be described in a declarative manner. This allows a complete definition of an analysis environment that can be versioned and treated as code. It greatly simplifies sharing and reusing configurations and routine administration. - -- **Built on OpenAPI**: The CLI interacts with Seqera via the [Seqera Platform API](../api/overview.mdx) which is created using the latest OpenAPI 3.0 specification. The CLI provides full control of the application, facilitating maximum insights into your pipeline submissions and execution environments. - -### Availability - -The CLI can be installed on macOS, Windows, and Linux. - -Visit the [CLI](https://github.com/seqeralabs/tower-cli/) GitHub repository for installation and configuration details. diff --git a/platform_versioned_docs/version-23.3/cli/commands.mdx b/platform_versioned_docs/version-23.3/cli/commands.mdx new file mode 100644 index 000000000..5af743503 --- /dev/null +++ b/platform_versioned_docs/version-23.3/cli/commands.mdx @@ -0,0 +1,1064 @@ +--- +title: "Commands" +description: "Seqera Platform CLI commands." +date: "16 Jul 2024" +tags: [cli, commands] +--- + +import Tabs from '@theme/Tabs'; +import TabItem from '@theme/TabItem'; + +:::note +The CLI performs operations in the user workspace context by default. Use the `TOWER_WORKSPACE_ID` environment variable or the `--workspace` parameter to specify an organization workspace ID. +::: + +Use the `-h` or `--help` parameter to list the available commands and their associated options. + +[![`tw --help`](./_images/tw-info.jpg)](https://github.com/seqeralabs/tower-cli) + +For help with a specific subcommand, run the command with `-h` or `--help` appended. For example, `tw credentials add google -h`. + +:::tip +Use `tw --output=json ` to dump and store Seqera Platform entities in JSON format. + +Use `tw --output=json | jq -r '.[].'` to pipe the command to use jq to retrieve specific values in the JSON output. For example, `tw --output=json workspaces list | jq -r '.workspaces[].orgId'` returns the organization ID for each workspace listed. +::: + +
+ **Credentials** + + To launch pipelines in a Platform workspace, you need [credentials][credentials] for: + + 1. Compute environments + 2. Pipeline repository Git providers + 3. (Optional) [Tower agent][tower-agent] — used with HPC clusters + 4. (Optional) Container registries, such as docker.io + + ### Add credentials + + Run `tw credentials add -h` to view a list of providers. + + Run `tw credentials add -h` to view the required fields for your provider. + + :::note + You can add multiple credentials from the same provider in the same workspace. + ::: + + #### Compute environment credentials + + Platform requires credentials to access your cloud compute environments. See the [compute environment page][compute-envs] for your cloud provider for more information. + + ```console + tw credentials add aws --name=my_aws_creds --access-key= --secret-key= + + New AWS credentials 'my_aws_creds (1sxCxvxfx8xnxdxGxQxqxH)' added at user workspace + ``` + + #### Git credentials + + Platform requires access credentials to interact with pipeline Git repositories. See [Git integration][git-integration] for more information. + + ```console + tw credentials add github -n=my_GH_creds -u= -p= + + New GITHUB credentials 'my_GH_creds (xxxxx3prfGlpxxxvR2xxxxo7ow)' added at user workspace + ``` + + #### Container registry credentials + + Configure credentials for the Nextflow Wave container service to authenticate to private and public container registries. See the **Container registry credentials** section under [Credentials][credentials] for registry-specific instructions. + + :::note + Container registry credentials are only used by the Wave container service. See [Wave containers][wave-docs] for more information. + ::: + + ### List credentials + + ```console + tw credentials list + + Credentials at user workspace: + + ID | Provider | Name | Last activity + ------------------------+-----------+------------------------------------+------------------------------- + 1x1HxFxzxNxptxlx4xO7Gx | aws | my_aws_creds_1 | Wed, 6 Apr 2022 08:40:49 GMT + 1sxCxvxfx8xnxdxGxQxqxH | aws | my_aws_creds_2 | Wed, 9 Apr 2022 08:40:49 GMT + 2x7xNsf2xkxxUIxXKxsTCx | ssh | my_ssh_key | Thu, 8 Jul 2021 07:09:46 GMT + 4xxxIeUx7xex1xqx1xxesk | github | my_github_cred | Wed, 22 Jun 2022 09:18:05 GMT + ``` + + ### Delete credentials + + ```console + tw credentials delete --name=my_aws_creds + + Credentials '1sxCxvxfx8xnxdxGxQxqxH' deleted at user workspace + ``` + +
+
+ **Compute environments** + + Compute environments define the execution platform where a pipeline runs. A compute environment is composed of the credentials, configuration, and storage options related to a particular computing platform. See [Compute environments][compute-envs] for more information on supported providers. + +Run `tw compute-envs -h` to view the list of supported compute environment operations. + + ### Add a compute environment + + Run `tw compute-envs add -h` to view the list of supported providers. + + Run `tw compute-envs add -h` to view the required and optional fields for your provider. + + You must add the credentials for your provider before creating your compute environment. + + ```console + tw compute-envs add aws-batch forge --name=my_aws_ce \ + --credentials= --region=eu-west-1 --max-cpus=256 \ + --work-dir=s3:// --wait=AVAILABLE + + New AWS-BATCH compute environment 'my_aws_ce' added at user workspace + ``` + + This command will: + + - Use **Batch Forge** to automatically manage the AWS Batch resource lifecycle (`forge`) + - Use the credentials previously added to the workspace (`--credentials`) + - Create the required AWS Batch resources in the AWS Ireland (`eu-west-1`) region + - Provision a maximum of 256 CPUs in the compute environment (`--max-cpus`) + - Use an existing S3 bucket to store the Nextflow work directory (`--work-dir`) + - Wait until the compute environment has been successfully created and is ready to use (`--wait`) + + See the [compute environment][compute-envs] page for your provider for detailed information on Batch Forge and manual compute environment creation. + + ### Delete a compute environment + + ```console + tw compute-envs delete --name=my_aws_ce + + Compute environment '1sxCxvxfx8xnxdxGxQxqxH' deleted at user workspace + ``` + + ### Default compute environment + + Select a **primary** compute environment to be used by default in a workspace. You can override the workspace primary compute environment by explicitly specifying an alternative compute environment when you create or launch a pipeline. + + ```console + tw compute-envs primary set --name=my_aws_ce + + Primary compute environment for workspace 'user' was set to 'my_aws_ce (1sxCxvxfx8xnxdxGxQxqxH)' + ``` + + ### Import and export a compute environment + + Export the configuration details of a compute environment in JSON format for scripting and reproducibility purposes. + + ```console + tw compute-envs export --name=my_aws_ce my_aws_ce_v1.json + + Compute environment exported into 'my_aws_ce_v1.json' + ``` + + Similarly, a compute environment can be imported to a workspace from a previously exported JSON file. + + ```console + tw compute-envs import --name=my_aws_ce_v1 ./my_aws_ce_v1.json + + New AWS-BATCH compute environment 'my_aws_ce_v1' added at user workspace + ``` +
+
+ **Datasets** + + Run `tw datasets -h` to view the list of supported operations. + + [Datasets](../data/datasets.mdx) are CSV (comma-separated values) and TSV (tab-separated values) files stored in a workspace, used as inputs during pipeline execution. The most commonly used datasets for Nextflow pipelines are samplesheets, where each row consists of a sample, the location of files for that sample (such as FASTQ files), and other sample details. + + ### Add a dataset + + Run `tw datasets add -h` to view the required and optional fields for adding a dataset. + + Add a preconfigured dataset file to a workspace (include the `--header` flag if the first row of your samplesheet file is a header): + + ```console + tw datasets add --name=samplesheet1 --header samplesheet_test.csv + + Dataset 'samplesheet1' added at user workspace with id '60gGrD4I2Gk0TUpEGOj5Td' + ``` + + :::note + The maximum supported dataset file size is 10 MB. + ::: + + ### Delete a dataset + + To delete a workspace dataset, specify either the dataset name (`-n` flag) or ID (`-i` flag): + + ```console + tw datasets delete -i 6tYMjGqCUJy6dEXNK9y8kh + + Dataset '6tYMjGqCUJy6dEXNK9y8kh' deleted at 97652229034604 workspace + ``` + + ### Download a dataset + + View a stored dataset's contents: + + ```console + tw datasets download -n samplesheet1 + + sample,fastq_1,fastq_2,strandedness + WT_REP1,https://raw.githubusercontent.com/nf-core/test-datasets/rnaseq/testdata/GSE110004/SRR6357070_1.fastq.gz,https://raw.githubusercontent.com/nf-core/test-datasets/rnaseq/testdata/GSE110004/SRR6357070_2.fastq.gz,auto + WT_REP1,https://raw.githubusercontent.com/nf-core/test-datasets/rnaseq/testdata/GSE110004/SRR6357071_1.fastq.gz,https://raw.githubusercontent.com/nf-core/test-datasets/rnaseq/testdata/GSE110004/SRR6357071_2.fastq.gz,auto + WT_REP2,https://raw.githubusercontent.com/nf-core/test-datasets/rnaseq/testdata/GSE110004/SRR6357072_1.fastq.gz,https://raw.githubusercontent.com/nf-core/test-datasets/rnaseq/testdata/GSE110004/SRR6357072_2.fastq.gz,reverse + RAP1_UNINDUCED_REP1,https://raw.githubusercontent.com/nf-core/test-datasets/rnaseq/testdata/GSE110004/SRR6357073_1.fastq.gz,,reverse + RAP1_UNINDUCED_REP2,https://raw.githubusercontent.com/nf-core/test-datasets/rnaseq/testdata/GSE110004/SRR6357074_1.fastq.gz,,reverse + RAP1_UNINDUCED_REP2,https://raw.githubusercontent.com/nf-core/test-datasets/rnaseq/testdata/GSE110004/SRR6357075_1.fastq.gz,,reverse + RAP1_IAA_30M_REP1,https://raw.githubusercontent.com/nf-core/test-datasets/rnaseq/testdata/GSE110004/SRR6357076_1.fastq.gz,https://raw.githubusercontent.com/nf-core/test-datasets/rnaseq/testdata/GSE110004/SRR6357076_2.fastq.gz,reverse + ``` + + ### List datasets + + Run `tw datasets list -h` to view the optional fields for listing and filtering datasets. + + ```console + tw datasets list -f data + + Datasets at 97652229034604 workspace: + + ID | Name | Created + ------------------------+----------+------------------------------- + 6vBGj6aWWpBuLpGKjJDpZy | dataset2 | Tue, 27 Aug 2024 14:49:32 GMT + ``` + + ### View a dataset + + Run `tw datasets view -h` to view the required and optional fields for viewing a stored dataset's details. + + ```console + tw datasets view -n samplesheet1 + + Dataset at 97652229034604 workspace: + + -------------+------------------------------- + ID | 60gGrD4I2Gk0TUpEGOj5Td + Name | samplesheet1 + Description | + Media Type | text/csv + Created | Mon, 19 Aug 2024 07:59:16 GMT + Updated | Mon, 19 Aug 2024 07:59:17 GMT + ``` + + ### Update a dataset + + Run `tw datasets update -h` to view the required and optional fields for updating a dataset. + + ```console + tw datasets update -n dataset1 --new-name=dataset2 -f samplesheet_test.csv + + Dataset 'dataset1' updated at 97652229034604 workspace with id '6vBGj6aWWpBuLpGKjJDpZy' + ``` + + ### Obtain a dataset URL + + Run `tw datasets url -h` to view the required and optional fields for obtaining dataset URLs. + + ```console + tw datasets url -n dataset2 + + Dataset URL + + ----------- + + https://api.cloud.seqera.io/workspaces/97652229034xxx/datasets/6vBGj6aWWpBuLpGKjJDxxx/v/2/n/samplesheet_test.csv + ``` + +
+
+ **Pipelines** + + Run `tw pipelines -h` to view the list of supported operations. + + Pipelines define pre-configured workflows in a workspace. A pipeline consists of a workflow repository, launch parameters, and a compute environment. + + ### Add a pipeline + + Run `tw pipelines add -h` to view the required and optional fields for adding your pipeline. + + Add a pre-configured pipeline to the Launchpad: + + ```console + tw pipelines add --name=my_rnaseq_nf_pipeline \ + --params-file=my_rnaseq_nf_pipeline_params.yaml \ + --config= \ + https://github.com/nextflow-io/rnaseq-nf + + New pipeline 'my_rnaseq_nf_pipeline' added at user workspace + ``` + + The optional `--params-file` flag is used to pass a set of default parameters that will be associated with the pipeline in the Launchpad. + + The optional `--config` flag is used to pass a custom Nextflow configuration file — configuration values passed here override the same values in the default pipeline repository `nextflow.conf` file. When this flag is set, all configuration values specified in Platform pipeline or compute environment **Nextflow config** fields are ignored. + + :::tip + The `params-file` or `--config` file must be a YAML or JSON file using [Nextflow configuration][nextflow-config] syntax. + ::: + + ### Import and export a pipeline + + Export the configuration details of a pipeline in JSON format for scripting and reproducibility purposes. + + ```console + tw pipelines export --name=my_rnaseq_nf_pipeline my_rnaseq_nf_pipeline_v1.json + + Pipeline exported into 'my_rnaseq_nf_pipeline_v1.json' + ``` + + Similarly, a pipeline can be imported to a workspace from a previously exported JSON file. + + ```console + tw pipelines import --name=my_rnaseq_nf_pipeline_v1 ./my_rnaseq_nf_pipeline_v1.json + + New pipeline 'my_rnaseq_nf_pipeline_v1' added at user workspace + ``` + + ### Update a pipeline + + The default launch parameters can be changed with the `update` command: + + ```console + tw pipelines update --name=my_rnaseq_nf_pipeline \ + --params-file=my_rnaseq_nf_pipeline_params_2.yaml + ``` +
+
+ **Launch pipelines** + + Run `tw launch -h` to view supported launch options. + + ### Launch a preconfigured pipeline + + If no custom parameters are passed via the CLI during launch, the defaults set for the pipeline in the Launchpad will be used. + + :::note + tw CLI users are bound to the same user permissions that apply in the Platform UI. Launch users can launch pre-configured pipelines in the workspaces they have access to, but they cannot add or run new pipelines. + ::: + + ```console + tw launch my_rnaseq_nf_pipeline \ + --config= \ + + + Workflow 1XCXxX0vCX8xhx submitted at user workspace. + + https://cloud.seqera.io/user/user1/watch/1XCXxX0vCX8xhx + ``` + + The optional `--config` flag is used to pass a custom Nextflow configuration file — configuration values passed here override the same values in the default pipeline repository `nextflow.conf` file. When this flag is set, all configuration values specified in Platform pipeline or compute environment **Nextflow config** fields are ignored. + + When using `--wait`, `tw` can exit with one of two exit codes: + + - `0`: When the run reaches the desired state. + - `1`: When the run reaches a state that makes it impossible to reach the desired state. + + :::tip + Use `--wait=SUCCEEDED` if you want the command to wait until the pipeline execution is complete. + ::: + + ### Launch a pipeline with custom parameters + + To specify custom parameters during pipeline launch, specify a custom `--params-file`: + + ```console + tw launch my_rnaseq_nf_pipeline --params-file=my_rnaseq_nf_pipeline_params_2.yaml + + Workflow 2XDXxX0vCX8xhx submitted at user workspace. + + https://cloud.seqera.io/user/user1/watch/2XDXxX0vCX8xhx + ``` + + See [Nextflow configuration][nextflow-config] for more information. + + ### Launch an unsaved pipeline + + The CLI can directly launch pipelines that have not been added to the Launchpad in a Platform workspace by using the full pipeline repository URL: + + ```console + tw launch https://github.com/nf-core/rnaseq \ + --params-file=./custom_rnaseq_params.yaml \ + --config= \ + --compute-env=my_aws_ce --revision 3.8.1 \ + --profile=test,docker + + Workflow 2XDXxX0vCX8xhx submitted at user workspace. + + https://cloud.seqera.io/user/user1/watch/2XDXxX0vCX8xhx + ``` + + - Pipeline parameters are defined within the `custom_rnaseq_params.yaml` file. + - The optional `--config` flag is used to pass a custom Nextflow configuration file — configuration values passed here override the same values in the default pipeline repository `nextflow.conf` file. When this flag is set, all configuration values specified in Platform pipeline or compute environment **Nextflow config** fields are ignored. + - Other parameters such as `--profile` and `--revision` can also be specified. + - A non-primary compute environment can be used to launch the pipeline. Omit `--compute-env` to launch with the workspace default compute environment. + + :::note + CLI users are bound to the same user permissions that apply in the Platform UI. Launch users can launch pre-configured pipelines in the workspaces they have access to, but they cannot add or run new pipelines. + ::: + +
+
+ **Runs** + + Run `tw runs -h` to view supported runs operations. + + Runs display all the current and previous pipeline runs in the specified workspace. Each new or resumed run is given a random name such as _grave_williams_ by default, which can be overridden with a custom value at launch. See [Run details](../monitoring/run-details.mdx) for more information. As a run executes, it can transition through the following states: + + - `submitted`: Pending execution + - `running`: Running + - `succeeded`: Completed successfully + - `failed`: Successfully executed, where at least one task failed with a terminate [error strategy](https://www.nextflow.io/docs/latest/process.html#errorstrategy) + - `cancelled`: Stopped manually during execution + - `unknown`: Indeterminate status + + ### View pipeline's runs + + Run `tw runs view -h` to view all the required and optional fields for viewing a pipeline's runs. + + ```console + tw runs view -i 2vFUbBx63cfsBY -w seqeralabs/showcase + + Run at [seqeralabs / showcase] workspace: + + + General + ---------------------+------------------------------------------------- + ID | 2vFUbBx63cfsBY + Operation ID | b5d55384-734e-4af0-8e47-0d3abec71264 + Run name | adoring_brown + Status | SUCCEEDED + Starting date | Fri, 31 May 2024 10:38:30 GMT + Commit ID | b89fac32650aacc86fcda9ee77e00612a1d77066 + Session ID | 9365c6f4-6d79-4ca9-b6e1-2425f4d957fe + Username | user1 + Workdir | s3://seqeralabs-showcase/scratch/2vFUbBx63cfsBY + Container | No container was reported + Executors | awsbatch + Compute Environment | seqera_aws_ireland_fusionv2_nvme + Nextflow Version | 23.10.1 + Labels | star_salmon,yeast + ``` + + ### List runs + + Run `tw runs list -h` to view all the required and optional fields for listing runs in a workspace. + + ```console + tw runs list + + Pipeline runs at [seqeralabs / testing] workspace: + + ID | Status | Project Name | Run Name | Username | Submit Date + ----------------+-----------+----------------------------+---------------------------------+-----------------------+------------------------------- + 49Gb5XVMud2e7H | FAILED | seqeralabs/nf-aggregate | distraught_archimedes | user1 | Fri, 31 May 2024 16:22:10 GMT + 4anNFvTUwRFDp | SUCCEEDED | nextflow-io/rnaseq-nf | nasty_kilby | user1 | Fri, 31 May 2024 15:23:12 GMT + 3wo3Kfni6Kl3hO | SUCCEEDED | nf-core/proteinfold | reverent_linnaeus | user2 | Fri, 31 May 2024 15:22:38 GMT + + + + 4fIRrFgZV3eDb1 | FAILED | nextflow-io/hello | gigantic_lichterman | user1 | Mon, 29 Apr 2024 08:44:47 GMT + cHEdKBXmdoQQM | FAILED | mathysgrapotte/stimulus | mighty_poitras | user3 | Mon, 29 Apr 2024 08:08:52 GMT + ``` + + Use the optional `--filter` flag to filter the list of runs returned by one or more `keyword:value` entries: + + - `status` + - `label` + - `workflowId` + - `runName` + - `username` + - `projectName` + - `after` + - `before` + - `sessionId` + - `is:starred` + + If no `keyword` is defined, the filtering is applied to the `runName`, `projectName` (the pipeline name), and `username`. + + :::note + The `after` and `before` flags require an [ISO 8601](https://en.wikipedia.org/wiki/ISO_8601) timestamp with UTC timezone (`YYYY-MM-DDThh:mm:ss.sssZ`). + ::: + + ```console + tw runs list --filter hello_slurm_20240530 + + Pipeline runs at [seqeralabs / showcase] workspace: + + ID | Status | Project Name | Run Name | Username | Submit Date + ---------------+-----------+-------------------+--------------------------------------+------------+------------------------------- + pZeJBOLtIvP7R | SUCCEEDED | nextflow-io/hello | hello_slurm_20240530_e75584566f774e7 | user1 | Thu, 30 May 2024 09:12:51 GMT + ``` + + Multiple filter criteria can be defined: + + ```console + tw runs list --filter="after:2024-05-29T00:00:00.000Z before:2024-05-30T00:00:00.000Z username:user1" + + Pipeline runs at [seqeralabs / testing] workspace: + + ID | Status | Project Name | Run Name | Username | Submit Date + ----------------+-----------+-----------------------+--------------------+-------------+------------------------------- + xJvK95W6YUmEz | SUCCEEDED | nextflow-io/rnaseq-nf | ondemand2 | user1 | Wed, 29 May 2024 20:35:28 GMT + 1c1ckn9a3j0xF0 | SUCCEEDED | nextflow-io/rnaseq-nf | fargate | user1 | Wed, 29 May 2024 20:28:02 GMT + 3sYX1acJ01T7rL | SUCCEEDED | nextflow-io/rnaseq-nf | min1vpcu-spot | user1 | Wed, 29 May 2024 20:27:47 GMT + 4ZYJGWJCttXqXq | SUCCEEDED | nextflow-io/rnaseq-nf | min1cpu-ondemand | user1 | Wed, 29 May 2024 20:25:21 GMT + 4LCxsffTqf3ysT | SUCCEEDED | nextflow-io/rnaseq-nf | lonely_northcutt | user1 | Wed, 29 May 2024 20:09:51 GMT + 4Y8EcyopNiYBlJ | SUCCEEDED | nextflow-io/rnaseq-nf | fargate | user1 | Wed, 29 May 2024 18:53:47 GMT + dyKevNwxK50XX | SUCCEEDED | mark814/nr-test | cheeky_cuvier | user1 | Wed, 29 May 2024 12:21:10 GMT + eS6sVB5A387aR | SUCCEEDED | mark814/nr-test | evil_murdock | user1 | Wed, 29 May 2024 12:11:08 GMT + ``` + + A leading and trailing `*` wildcard character is supported: + + ```console + tw runs list --filter="*man/rnaseq-*" + + Pipeline runs at [seqeralabs / testing] workspace: + + ID | Status | Project Name | Run Name | Username | Submit Date + ----------------+-----------+---------------------+---------------------+----------------+------------------------------- + 5z4AMshti4g0GK | SUCCEEDED | robnewman/rnaseq-nf | admiring_darwin | user1 | Tue, 16 Jan 2024 19:56:29 GMT + 62LqiS4O4FatSy | SUCCEEDED | robnewman/rnaseq-nf | cheeky_yonath | user1 | Wed, 3 Jan 2024 12:36:09 GMT + 3k2nu8ZmcBFSGv | SUCCEEDED | robnewman/rnaseq-nf | compassionate_jones | user3 | Tue, 2 Jan 2024 16:22:26 GMT + 3zG2ggf5JsniNW | SUCCEEDED | robnewman/rnaseq-nf | fervent_payne | user1 | Wed, 20 Dec 2023 23:55:17 GMT + 1SNIcSXRuJMSNZ | SUCCEEDED | robnewman/rnaseq-nf | curious_babbage | user3 | Thu, 28 Sep 2023 17:48:04 GMT + 5lI2fZUZfiokBI | SUCCEEDED | robnewman/rnaseq-nf | boring_heisenberg | user2 | Thu, 28 Sep 2023 12:29:27 GMT + 5I4lsRXIHVEjNB | SUCCEEDED | robnewman/rnaseq-nf | ecstatic_ptolemy | user2 | Wed, 27 Sep 2023 22:06:19 GMT + ``` + + ### Relaunch run + + Run `tw runs relaunch -h` to view all the required and optional fields for relaunching a run in a workspace. + + ### Cancel a run + + Run `tw runs cancel -h` to view all the required and optional fields for canceling a run in a workspace. + + ### Manage labels for runs + + Run `tw runs labels -h` to view all the required and optional fields for managing labels for runs in a workspace. + + In the example below, we add the labels `test` and `rnaseq-demo` to the run with ID `5z4AMshti4g0GK`: + + ```console + tw runs labels -i 5z4AMshti4g0GK test,rnaseq-demo + + 'set' labels on 'run' with id '5z4AMshti4g0GK' at 34830707738561 workspace + ``` + + ### Delete a run + + Run `tw runs delete -h` to view all the required and optional fields for deleting a run in a workspace. + + ### Dump all logs and details of a run + + Run `tw runs dump -h` to view all the required and optional fields for dumping all logs and details of a run in a workspace. The supported formats are `.tar.xz` and `.tar.gz`. In the example below, we dump all the logs and details for the run with ID `5z4AMshti4g0GK` to the output file `file.tar.gz`. + + ```console + tw runs dump -i 5z4AMshti4g0GK -o file.tar.gz + - Tower info + - Workflow details + - Task details + + Pipeline run '5z4AMshti4g0GK' at [seqeralabs / testing] workspace details dump at 'file.tar.gz' + ``` + +
+
+ **Labels** + + Run `tw labels -h` to view supported label operations. + + Manage labels and resource labels. + + ### Add a label + + Run `tw labels add -h` to view the required and optional fields for adding a label. + + :::note + [Resource labels](../resource-labels/overview.mdx) consist of a `name=value` pair. + [Labels](../labels/overview.mdx) require only a name. + ::: + + ```console + tw labels add -n Label1 -w DocTestOrg2/Testing -v Value1 + + Label 'Label1=Value1' added at 'DocTestOrg2/Testing' workspace with id '268741348267491' + ``` + + ### List labels + + Run `tw labels list -h` to view the optional fields for filtering labels. + + ```console + tw labels list + + Labels at 97652229034604 workspace: + + ID | Name | Value | Type + -----------------+------------------------+-----------+---------- + 116734717739444 | manual-fusion-amd64 | | Normal + 120599302764779 | test-with-prefix | | Normal + 128477232893714 | manual-fusion-arm64 | | Normal + 214201679620273 | test-config-link | | Normal + 244634136444435 | manual-nonfusion-amd64 | | Normal + 9184612610501 | Resource1 | Value1 | Resource + ``` + + ### Update a label + + Run `tw labels update -h` to view the required and optional fields for updating labels. + + ### Delete a label + + Run `tw labels delete -h` to view the required and optional fields for deleting labels. + + ```console + tw labels delete -i 203879852150462 + + Label '203879852150462' deleted at '97652229034604' workspace + ``` + +
+
+ **Data links** + + Run `tw data-links -h` to view supported data link operations. + + Data links allow you to work with public and private cloud storage buckets in [Data Explorer](../data/data-explorer.mdx) in the specified workspace. AWS S3, Azure Blob Storage, and Google Cloud Storage are supported. The full list of operations are: + + - `list`: List data links in a workspace + - `add`: Add a custom data link to a workspace + - `update`: Update a custom data link in a workspace + - `delete`: Delete a custom data link from a workspace + - `browse`: Browse the contents of a data link in a workspace + + ### List data links + + Run `tw data-links list -h` to view all the optional fields for listing data links in a workspace. + + If a workspace is not defined, the `TOWER_WORKSPACE_ID` workspace is used by default. Data links can be one of two types: + + - `v1-cloud-`: Cloud data links auto-discovered using credentials attached to the workspace. + - `v1-user-`: Custom data links created by users. + + ```console + tw data-links list -w seqeralabs/showcase + + Data links at [seqeralabs / showcase] workspace: + + ID | Provider | Name | Resource ref | Region + -------------------------------------------+----------+--------------------------------+-----------------------------------------------------------------+----------- + v1-cloud-833bb845bd9ec1970c4a7b0bb7b8c4ad | aws | e2e-data-explorer-tests-aws | s3://e2e-data-explorer-tests-aws | eu-west-2 + v1-cloud-60700a33ec3fae68d424cf948fa8d10c | aws | nf-tower-bucket | s3://nf-tower-bucket | eu-west-1 + v1-user-09705781697816b62f9454bc4b9434b4 | aws | vscode-analysis-demo | s3://seqera-development-permanent-bucket/studios-demo/vscode/ | eu-west-2 + v1-user-0dede00fabbc4b9e2610261822a2d6ae | aws | seqeralabs-showcase | s3://seqeralabs-showcase | eu-west-1 + v1-user-171aa8801cabe4af71500335f193d649 | aws | projectA-rnaseq-analysis | s3://seqeralabs-showcase/demo/nf-core-rnaseq/ | eu-west-1 + + + + v1-user-bb4fa9625a44721510c47ac1cb97905b | aws | genome-in-a-bottle | s3://giab | us-east-1 + v1-user-e7bf26921ba74032bd6ae1870df381fc | aws | NCBI_Sequence_Read_Archive_SRA | s3://sra-pub-src-1/ | us-east-1 + + Showing from 0 to 99 from a total of 16 entries. + ``` + + ### Add a custom data link + + Run `tw data-links add -h` to view all the required and optional fields for adding a custom data link to a workspace. + + Users with the workspace `MAINTAIN` role and above can add custom data links. The data link `name`, `uri`, and `provider` (`aws`, `azure`, or `google`) fields are required. If adding a custom data link for a private bucket, the credentials identifier field is also required. Adding a custom data link for a public bucket doesn't require credentials. + + ```console + tw data-links add -w seqeralabs/showcase -n FOO -u az://seqeralabs.azure-benchmarking \ + -p azure -c seqera_azure_credentials + + Data link created: + + ID | Provider | Name | Resource ref | Region + ------------------------------------------+----------+------+------------------------------------+-------- + v1-user-152116183ee325463901430bb9efb8c9 | azure | FOO | az://seqeralabs.azure-benchmarking | + ``` + + ### Update a custom data link + + Run `tw data-links update -h` to view all the required and optional fields for updating a custom data link in a workspace. Users with the `MAINTAIN` role and above for a workspace can update custom data links. + + ```console + tw data-links update -w seqeralabs/showcase -i v1-user-152116183ee325463901430bb9efb8c9 -n BAR + + Data link updated: + + ID | Provider | Name | Resource ref | Region + ------------------------------------------+----------+------+------------------------------------+-------- + v1-user-152116183ee325463901430bb9efb8c9 | azure | BAR | az://seqeralabs.azure-benchmarking | + ``` + + ### Delete a custom data link + + Run `tw data-links delete -h` to view all the required and optional fields for deleting a custom data link from a workspace. + + Users with the `MAINTAIN` role and above for a workspace can delete custom data links. + + ```console + tw data-links delete -w seqeralabs/showcase -i v1-user-152116183ee325463901430bb9efb8c9 + + Data link 'v1-user-152116183ee325463901430bb9efb8c9' deleted at '138659136604200' workspace. + ``` + + ### Browse data link contents + + Run `tw data-links browse -h` to view all the required and optional fields for browsing a data link in a workspace. + + Define the data link ID using the required `-i` or `--id` argument, which can be found by first using the list operation for a workspace. In the example below, a name is defined to only retrieve data links with names that start with the given word: + + ```console + tw data-links list -w seqeralabs/showcase -n 1000genomes + + Data links at [seqeralabs / showcase] workspace: + + ID | Provider | Name | Resource ref | Region + ------------------------------------------+----------+-------------+------------------+----------- + v1-user-6d8f44c239e2a098b3e02e918612452a | aws | 1000genomes | s3://1000genomes | us-east-1 + + Showing from 0 to 99 from a total of 1 entries. + + tw data-links browse -w seqeralabs/showcase -i v1-user-6d8f44c239e2a098b3e02e918612452a + + Content of 's3://1000genomes' and path 'null': + + Type | Name | Size + --------+--------------------------------------------+---------- + FILE | 20131219.populations.tsv | 1663 + FILE | 20131219.superpopulations.tsv | 97 + FILE | CHANGELOG | 257098 + FILE | README.alignment_data | 15977 + FILE | README.analysis_history | 5289 + FILE | README.complete_genomics_data | 5967 + FILE | README.crams | 563 + FILE | README.ebi_aspera_info | 935 + FILE | README.ftp_structure | 8408 + FILE | README.pilot_data | 2082 + FILE | README.populations | 1938 + FILE | README.sequence_data | 7857 + FILE | README_missing_files_20150612 | 672 + FILE | README_phase3_alignments_sequence_20150526 | 136 + FILE | README_phase3_data_move_20150612 | 273 + FILE | alignment.index | 3579471 + FILE | analysis.sequence.index | 54743580 + FILE | exome.alignment.index | 3549051 + FILE | sequence.index | 67069489 + FOLDER | 1000G_2504_high_coverage/ | 0 + FOLDER | alignment_indices/ | 0 + FOLDER | changelog_details/ | 0 + FOLDER | complete_genomics_indices/ | 0 + FOLDER | data/ | 0 + FOLDER | hgsv_sv_discovery/ | 0 + FOLDER | phase1/ | 0 + FOLDER | phase3/ | 0 + FOLDER | pilot_data/ | 0 + FOLDER | release/ | 0 + FOLDER | sequence_indices/ | 0 + FOLDER | technical/ | 0 + ``` +
+
+ **Organizations** + + Run `tw organizations -h` to view supported workspace operations. + + Organizations are the top-level structure and contain workspaces, members, and teams. You can also add external collaborators to an organization. See [Organization management](../orgs-and-teams/organizations.mdx) for more information. + + ### Add an organization + + Run `tw organizations add -h` to view the required and optional fields for adding your workspace. + + ```console + tw organizations add -n TestOrg2 -f 2nd\ Test\ Organization\ LLC -l RSA + + Organization 'TestOrg2' with ID '204336622618177' was added + ``` + +
+
+ **Members** + + Run `tw members -h` to view supported member operations. + + Manage organization members. Organization membership management requires organization `OWNER` permissions. + + ### List members + + Run `tw members list -h` view all the optional fields for listing organization members. + + ```console + tw members list -o TestOrg2 + + Members for TestOrg2 organization: + + ID | Username | Email | Role + -----------------+----------------------+---------------------------------+-------- + 200954501314303 | user1 | user1@domain.com | MEMBER + 277776534946151 | user2 | user2@domain.com | MEMBER + 243277166855716 | user3 | user3@domain.com | OWNER + + ``` + + ### Add a member + + Run `tw members add -h` view all the required and optional fields for adding organization members. + + ```console + tw members add -u user1@domain.com -o DocTestOrg2 + + Member 'user1' with ID '134534064600266' was added in organization 'TestOrg2' + ``` + + ### Delete a member + + Run `tw members delete -h` view all the required and optional fields for deleting organization members. + + ```console + tw members delete -u user1 -o TestOrg2 + + Member 'user1' deleted from organization 'TestOrg2' + ``` + + ### Update member role + + Run `tw members update -h` view all the required and optional fields for updating organization members. + + ```console + tw members update -u user1 -r OWNER -o TestOrg2 + + Member 'user1' updated to role 'owner' in organization 'TestOrg2' + ``` + + ### Leave an organization + + Run `tw members leave -o ` to be removed from the given organization's members. + +
+
+ **Workspaces** + + Run `tw workspaces -h` to view supported workspace operations. + + Workspaces provide the context in which a user launches workflow executions, defines the available resources, and manages who can access those resources. Workspaces contain pipelines, runs, actions, datasets, compute environments, credentials, and secrets. Access permissions are controlled with participants, collaborators, and teams. + + See [User workspaces][user-workspaces] for more information. + + ### Add a workspace + + :::note + Workspace management operations require organization `OWNER` permissions. + ::: + + Run `tw workspaces add -h` to view the required and optional fields for adding your workspace. + + In the example below, we create a shared workspace to be used for sharing pipelines with other private workspaces. See [Shared workspaces][shared-workspaces] for more information. + + ```console + tw workspaces add --name=shared-workspace --full-name=shared-workspace-for-all --org=my-tower-org --visibility=SHARED + + A 'SHARED' workspace 'shared-workspace' added for 'my-tower-org' organization + ``` + + :::note + By default, a workspace is set to private when created. + ::: + + ### List workspaces + + List all the workspaces in which you are a participant: + + ```console + tw workspaces list + + Workspaces for default user: + + Workspace ID | Workspace Name | Organization Name | Organization ID + -----------------+------------------+-------------------+----------------- + 26002603030407 | shared-workspace | my-tower-org | 04303000612070 + ``` + +
+
+ **Participants** + + Run `tw participants -h` to view supported participant operations. + + Manage workspace participants. + + :::note + The operations listed below require workspace `OWNER` or `ADMIN` permissions. + ::: + + ### List participants + + ```console + tw participants list + + Participants for 'my-tower-org/shared-workspace' workspace: + + ID | Participant Type | Name | Workspace Role + ----------------+------------------+-----------------------------+---------------- + 45678460861822 | MEMBER | user (user@mydomain.com) | owner + ``` + + ### Add participants + + Run `tw participants add -h` to view the required and optional fields for adding a participant. + + To add a new _collaborator_ to the workspace, use the `add` subcommand. The default role assigned to a _collaborator_ is `Launch`. + + See [Participant roles][participant-roles] for more information. + + ```console + tw participants add --name=collaborator@mydomain.com --type=MEMBER + + User 'collaborator' was added as participant to 'shared-workspace' workspace with role 'launch' + ``` + + ### Update participant roles + + To update the role of a _Collaborator_ to `ADMIN` or `MAINTAIN`, use the `update` subcommand: + + ```console + tw participants update --name=collaborator@mydomain.com --type=COLLABORATOR --role=MAINTAIN + + Participant 'collaborator@mydomain.com' has now role 'maintain' for workspace 'shared-workspace' + ``` + +
+
+ **Teams** + + Run `tw teams -h` to view supported team operations. + + Manage organization teams. + + :::note + Team management operations require organization `OWNER` permissions. + ::: + + ### List teams + + Run `tw teams list -h` to view the required and optional fields for listing teams. + + ```console + tw teams list -o TestOrg2 + + Teams for TestOrg2 organization: + + Team ID | Team Name | Members Count Name + ----------------+-----------+-------------------- + 84866234211969 | Testing | 1 + ``` + + ### Add a team + + Run `tw teams add -h` to view the required and optional fields for creating a team. + + ```console + tw teams add -n team1 -o TestOrg2 -d testing + + A 'team1' team added for 'TestOrg2' organization + ``` + + ### Delete a team + + ```console + tw teams delete -i 169283393825479 -o TestOrg2 + + Team '169283393825479' deleted for TestOrg2 organization + ``` + + ### Manage team members + + Run `tw teams members -h` to view the fields and additional commands for managing team members. + + To add a new team member, include an existing username or new user email: + + ```console + tw teams members -t Testing -o TestOrg2 add -m user1@domain.com + + Member 'user1' added to team 'Testing' with id '243206491381406' + ``` + + To delete a team member, include the member's username: + + ```console + tw teams members -t Testing -o TestOrg2 delete -m user1 + + Team member 'user1' deleted at 'Testing' team + ``` + +
+
+ **Collaborators** + +Run `tw collaborators -h` view all the required and optional fields for managing organization collaborators. + + Manage organization collaborators. + + ### List collaborators + + ```console + tw collaborators list -o seqeralabs + + Collaborators for 88848180287xxx organization: + + ID | Username | Email + -----------------+----------------------+-------------------- + 13136942731xxx | external_user1 | user1@domain.com + 127726720173xxx | external_user2 | user2@domain.com + 59151157784xxx | external_user3 | user3@domain.com + 132868466675xxx | external_user4 | user4@domain.com + 178756942629xxx | external_user5 | user5@domain.com + ``` + +
+
+ **Actions** + + Run `tw actions -h` to view supported pipeline action operations. + + [Actions](../pipeline-actions/overview.mdx) enable event-based pipeline execution, such as triggering a pipeline launch with a GitHub webhook whenever the pipeline repository is updated. + + ### Add a pipeline action + + Run `tw actions add -h` to view the required and optional fields for adding an action. + + +
+
+ **Secrets** + + Run `tw secrets -h` to view supported workspace secret operations. + + [Secrets](../secrets/overview.mdx) are used to store the keys and tokens used by workflow tasks to interact with external systems, such as a password to connect to an external database or an API token. + + ### Add a workspace secret + + Run `tw secrets add -h` to view the required and optional fields for adding a secret. + + +
+ +[compute-envs]: ../compute-envs/overview.mdx +[credentials]: ../credentials/overview.mdx +[git-integration]: ../git/overview.mdx +[nextflow-config]: https://www.nextflow.io/docs/latest/config.html#config-syntax +[participant-roles]: ../orgs-and-teams/workspace-management.mdx#participant-roles +[shared-workspaces]: ../orgs-and-teams/workspace-management.mdx +[tower-agent]: ../supported_software/agent/agent.mdx +[user-workspaces]: ../orgs-and-teams/workspace-management.mdx +[wave-docs]: https://docs.seqera.io/wave diff --git a/platform_versioned_docs/version-23.3/cli/installation.mdx b/platform_versioned_docs/version-23.3/cli/installation.mdx new file mode 100644 index 000000000..762b67abf --- /dev/null +++ b/platform_versioned_docs/version-23.3/cli/installation.mdx @@ -0,0 +1,169 @@ +--- +title: "Installation" +description: "Seqera Platform CLI installation and configuration instructions." +date: "16 Jul 2024" +tags: [cli, platform, installation, configuration] +--- + +import Tabs from '@theme/Tabs'; +import TabItem from '@theme/TabItem'; + +1. Download the latest [version][releases] for your OS from the CLI GitHub repository. +1. Rename the file and and make it executable: + + ```bash + mv tw-* tw + chmod +x ./tw + ``` + +1. Move the file to a directory accessible to your `$PATH` variable: + + ```bash + sudo mv tw /usr/local/bin/ + ``` + +### Configuration + +The CLI requires an access token to interact with your Seqera Platform instance. Select **User tokens** from the user menu in the [Platform UI](https://cloud.seqera.io), then select **Add token** to create a new token. + +Copy the access token value and use it with the CLI in one of two ways: + +- **Environment variable**: + + 1. Export the token as a shell variable directly into your terminal: + + ```bash + export TOWER_ACCESS_TOKEN= + ``` + + 2. Add the `export` command to your `.bashrc`, `.zshrc`, or `.bash_profile` file for it to be permanently added to your environment. + +- **tw command flag**: + + Provide the access token directly in your `tw` command with `--access-token`: + + ```bash + tw --access-token= + ``` + +If required, configure the following optional environment variables using the same methods above: + +- `TOWER_WORKSPACE_ID`: Workspace ID. Default: Your user workspace. +- `TOWER_API_ENDPOINT`: Seqera API URL. Default: `api.cloud.seqera.io`. + +:::tip +Find your `TOWER_WORKSPACE_ID` from the **Workspaces** tab on your organization page. Alternatively, list all the workspaces your token can access with `tw workspaces list` and copy the workspace ID from the command output. +::: + +### Health check + +Confirm the installation, configuration, and connection: + +```console +tw info + + Details + -------------------------+---------------------- + Tower API endpoint | + Tower API version | 1.25.0 + Tower version | 24.2.0_cycle22 + CLI version | 0.9.4 (f3e846e) + CLI minimum API version | 1.15 + Authenticated user | + + System health status + ---------------------------------------+------------------ + Remote API server connection check | OK + Tower API version check | OK + Authentication API credential's token | OK +``` + +### Commands + +See [Commands](./commands.mdx) for detailed instructions to use the CLI. + +### Autocompletion + +Activate autocompletion in your current session with this command: + +```bash +source <(tw generate-completion) +``` + +### Custom SSL certificate authority store + +If you are using a Private CA SSL certificate not recognized by the default Java certificate authorities, use a [custom](https://www.baeldung.com/jvm-certificate-store-errors) `cacerts` store: + +```bash +tw -Djavax.net.ssl.trustStore=/absolute/path/to/cacerts -Djavax.net.ssl.trustStorePassword= info +``` + +Replace `` with your keystore password. If you did not set a password when creating the keystore, include the default keystore password `changeit` in the command above. + +You can also rename the binary to `tw-binary` and create a `tw` script to automatically include the custom `cacerts` store in every session: + +```bash +#!/usr/bin/env bash +tw-binary -Djavax.net.ssl.trustStore=/absolute/path/to/cacerts -Djavax.net.ssl.trustStorePassword= $@ +``` + +### Build binary development versions + +tw CLI is a platform binary executable created by a native compilation from Java GraalVM. To compile and build a development version of the binary: + +1. If necessary, install [SDKMan!](https://sdkman.io/) +1. From the root of the tower-cli project, install GraalVM: + + ```bash + sdk env install + ``` + + This ensures that SDKMan uses the tower-cli project-specific `.sdkmanrc` configuration. + +1. Install `native-image`: + + ```bash + gu install native-image + ``` + +1. Export your Github credentials. Github requires authentication for public packages (the token only requires the `read:packages` scope): + + ```bash + export GITHUB_USERNAME=... + export GITHUB_TOKEN=... + ``` + +1. Create the native client: + + ```bash + ./gradlew nativeCompile + ``` + + This will install a locally compiled version of `tw` in the nativeCompile directory: + + ```console + Produced artifacts: + /build/native/nativeCompile/tw (executable) + ======================================================================================================================== + Finished generating 'tw' in 1m 6s. + [native-image-plugin] Native Image written to: /build/native/nativeCompile + + BUILD SUCCESSFUL in 1m 8s + 6 actionable tasks: 2 executed, 4 up-to-date + ``` + +1. Run `tw`: + + ```bash + ./build/native/nativeCompile/tw + ``` + +### Non-binary development versions + +Run a non-binary development version by executing the [`./tw`](https://github.com/seqeralabs/tower-cli/blob/master/tw) script in the root of the CLI repository. + +### License + +[Mozilla Public License v2.0](https://github.com/seqeralabs/tower-cli/blob/master/LICENSE.txt) + +[releases]: https://github.com/seqeralabs/tower-cli/releases \ No newline at end of file diff --git a/platform_versioned_docs/version-23.3/cli/overview.mdx b/platform_versioned_docs/version-23.3/cli/overview.mdx new file mode 100644 index 000000000..1362e4413 --- /dev/null +++ b/platform_versioned_docs/version-23.3/cli/overview.mdx @@ -0,0 +1,29 @@ +--- +title: "Overview" +description: "Seqera Platform CLI overview." +date: "24 Apr 2023" +tags: [cli, tower] +--- + +import Tabs from '@theme/Tabs'; +import TabItem from '@theme/TabItem'; + +Seqera Platform CLI brings concepts like pipelines and compute environments to the terminal. + +The CLI interacts with Platform to provide an interface to launch pipelines, manage cloud resources, and administer your analysis. + +![tw](./_images/tw-screenshot.png) + +## Key features + +- **A Nextflow-like experience**: tw CLI provides a developer-friendly environment. Pipelines can be launched with the CLI similarly to Nextflow but with the Platform benefits of monitoring, logging, resource provisioning, dataset management, and collaborative sharing. + +- **Infrastructure as Code**: All Platform resources, including pipelines and compute environments, can be described in a declarative manner. This enables a complete definition of an analysis environment that can be versioned and treated as code. It greatly simplifies configuration sharing and routine administration. + +- **Built on OpenAPI**: tw CLI interacts with Platform via the [Seqera Platform API](https://cloud.seqera.io/openapi/index.html) which uses the OpenAPI 3.0 specification. The CLI provides full control of the Platform application, allowing users to get maximum insights into pipeline submissions and execution environments. + +## Availability + +The CLI can be installed on macOS, Windows, and Linux. It is compatible with [Seqera Platform Cloud](https://cloud.seqera.io/) and Enterprise versions 21.08 and later. + +See [Installation](./installation.mdx) to get started. diff --git a/platform_versioned_docs/version-23.3/getting-started/deployment-options.mdx b/platform_versioned_docs/version-23.3/getting-started/deployment-options.mdx index 2507fcc80..dbadc5913 100644 --- a/platform_versioned_docs/version-23.3/getting-started/deployment-options.mdx +++ b/platform_versioned_docs/version-23.3/getting-started/deployment-options.mdx @@ -5,7 +5,7 @@ date: "21 Apr 2023" tags: [deployment] --- -Seqera Platform is available in two deployment editions and can be accessed via the web user interface (UI), [API](../api/overview.mdx), [CLI](../cli/cli.mdx), or in Nextflow directly using a flag. +Seqera Platform is available in two deployment editions and can be accessed via the web user interface (UI), [API](../api/overview.mdx), [CLI](../cli/overview.mdx), or in Nextflow directly using a flag. ## Seqera Platform editions @@ -30,7 +30,7 @@ To install Seqera Platform in your organization's infrastructure, [contact us](h ## How to use Seqera Platform -You can access your Seqera instance through the UI, the [API](../api/overview.mdx), the [CLI](../cli/cli.mdx), or in Nextflow directly using the `-with-tower` option. +You can access your Seqera instance through the UI, the [API](../api/overview.mdx), the [CLI](../cli/overview.mdx), or in Nextflow directly using the `-with-tower` option. ### Seqera Platform UI @@ -49,7 +49,7 @@ See [API](../api/overview.mdx). ### Seqera CLI -See [CLI](../cli/cli.mdx). +See [CLI](../cli/overview.mdx). ### Nextflow `-with-tower` diff --git a/platform_versioned_docs/version-23.3/getting-started/overview.mdx b/platform_versioned_docs/version-23.3/getting-started/overview.mdx index 637811099..d3485a6b0 100644 --- a/platform_versioned_docs/version-23.3/getting-started/overview.mdx +++ b/platform_versioned_docs/version-23.3/getting-started/overview.mdx @@ -5,7 +5,7 @@ date: "15 September 2023" tags: [overview] --- -Seqera Platform is available in two [deployment editions](../getting-started/deployment-options.mdx) and can be accessed via web user interface (UI), [API](../api/overview.mdx), [CLI](../cli/cli.mdx), or directly in your Nextflow command using a flag. +Seqera Platform is available in two [deployment editions](../getting-started/deployment-options.mdx) and can be accessed via web user interface (UI), [API](../api/overview.mdx), [CLI](../cli/overview.mdx), or directly in your Nextflow command using a flag. ## Community Showcase diff --git a/platform_versioned_docs/version-23.3/seqerakit/commands.mdx b/platform_versioned_docs/version-23.3/seqerakit/commands.mdx new file mode 100644 index 000000000..533d64777 --- /dev/null +++ b/platform_versioned_docs/version-23.3/seqerakit/commands.mdx @@ -0,0 +1,126 @@ +--- +title: "Commands" +description: "Seqerakit command options" +date: "21 Oct 2024" +tags: [seqerakit, cli, automation, commands] +--- + +Use the `--help` or `-h` option to list available commands and options: + +```shell-session +seqerakit --help +``` + +### Input + +Seqerakit supports input through paths to YAML configuration files or directly from standard input (`stdin`). + +- Using file path: + + ```shell-session + seqerakit file.yaml + ``` + +- Using `stdin`: + + ```shell-session + cat file.yaml | seqerakit - + ``` + +See [Define your YAML file using CLI options](#define-your-yaml-file-using-cli-options) for guidance on formatting your input YAML files. + +### Dryrun + +Confirm that your configuration and command are correct before creating resources in your Seqera account, particularly when automating the end-to-end creation of multiple entities at once. To print the commands that would be executed with Platform CLI when using a YAML file, run your `seqerakit` command with the `--dryrun` option: + +```shell-session +seqerakit file.yaml --dryrun +``` + +### Specify targets + +When using a YAML file as an input that defines multiple resources, use the `--targets` option to specify which resources to create. This option accepts a comma-separated list of resource names. + +Supported resource names include: + +- `actions` +- `compute-envs` +- `credentials` +- `datasets` +- `labels` +- `launch` +- `members` +- `organizations` +- `participants` +- `pipelines` +- `secrets` +- `teams` +- `workspaces` + +For example, given a `test.yaml` file that defines the following resources: + +```yaml +workspaces: + - name: 'workspace-1' + organization: 'seqerakit' +... +compute-envs: + - name: 'compute-env' + type: 'aws-batch forge' + workspace: 'seqerakit/workspace-1' +... +pipelines: + - name: 'hello-world' + url: 'https://github.com/nextflow-io/hello' + workspace: 'seqerakit/workspace-1' + compute-env: 'compute-env' +... +``` + +You can target the creation of `pipelines` only by running: + +```shell-session +seqerakit test.yaml --targets pipelines +``` + +This command will create only the pipelines defined in the YAML file and ignore `workspaces` and `compute-envs`. + +To create both workspaces and pipelines, run: + +```shell-session +seqerakit test.yaml --targets workspaces,pipelines +``` + +### Delete resources + +Instead of adding or creating resources, specify the `--delete` option to recursively delete resources in your YAML file: + +```shell-session +seqerakit file.yaml --delete +``` + +For example, if you have a `file.yaml` that defines an organization, workspace, team, credentials, and compute environment that have already been created, run `seqerakit file.yaml --delete` to recursively delete the same resources. + +### Use `tw`-specific CLI options + +Specify `tw`-specific CLI options with the `--cli=` option: + +```shell-session +seqerakit file.yaml --cli="--arg1 --arg2" +``` + +See [CLI commands](../cli/commands.mdx) or run `tw -h` for the full list of options. + +:::note +The `--verbose` option for `tw` CLI is currently not supported in `seqerakit` commands. +::: + +#### Example: HTTP-only connections + +The Platform CLI expects to connect to a Seqera instance that is secured by a TLS certificate. If your Seqera Enterprise instance does not present a certificate, you must run your `tw` commands with the `--insecure` option. + +To use `tw`-specific CLI options such as `--insecure`, use the `--cli=` option, followed by the options to use enclosed in double quotes: + +```shell-session +seqerakit file.yaml --cli="--insecure" +``` diff --git a/platform_versioned_docs/version-23.3/seqerakit/installation.mdx b/platform_versioned_docs/version-23.3/seqerakit/installation.mdx new file mode 100644 index 000000000..58eac0d41 --- /dev/null +++ b/platform_versioned_docs/version-23.3/seqerakit/installation.mdx @@ -0,0 +1,117 @@ +--- +title: "Installation" +description: "Seqerakit installation options" +date: "21 Oct 2024" +tags: [seqerakit, cli, automation, installation] +--- + + +Seqerakit is a Python wrapper that sets [Platform CLI](../cli/overview.mdx) command options using YAML configuration files. Individual commands and configuration parameters can be chained together to automate the end-to-end creation of all Seqera Platform entities. + +As an extension of the Platform CLI, Seqerakit enables: + +- **Infrastructure as code**: Users manage and provision their infrastructure from the command line. +- **Simple configuration**: All Platform CLI command-line options can be defined in simple YAML format. +- **Automation**: End-to-end creation of Seqera entities, from adding an organization to launching pipelines. + +### Installation + +Seqerakit has three dependencies: + +1. [Seqera Platform CLI (`>=0.10.1`)](https://github.com/seqeralabs/tower-cli/releases) +2. [Python (`>=3.8`)](https://www.python.org/downloads/) +3. [PyYAML](https://pypi.org/project/PyYAML/) + +#### Pip + +If you already have [Platform CLI](../cli/installation.mdx) and Python installed on your system, install Seqerakit directly from [PyPI](https://pypi.org/project/seqerakit/): + +```console +pip install seqerakit +``` + +Overwrite an existing installation to use the latest version: + +```console +pip install --upgrade --force-reinstall seqerakit +``` + +#### Conda + +To install `seqerakit` and its dependencies via Conda, first configure the correct channels: + +```console +conda config --add channels bioconda +conda config --add channels conda-forge +conda config --set channel_priority strict +``` + +Then create a conda environment with `seqerakit` installed: + +```console +conda env create -n seqerakit seqerakit +conda activate seqerakit +``` + +#### Local development installation + +Install the development branch of `seqerakit` on your local machine to test the latest features and updates: + +1. You must have [Python](https://www.python.org/downloads/) and [Git](https://git-scm.com/downloads) installed on your system. +1. To install directly from pip: + ```shell-session + pip install git+https://github.com/seqeralabs/seqera-kit.git@dev + ``` +1. Alternatively, clone the repository locally and install manually: + ```shell-session + git clone https://github.com/seqeralabs/seqera-kit.git + cd seqera-kit + git checkout dev + pip install . + ``` +1. Verify your installation: + ```shell-session + pip show seqerakit + ``` + +### Configuration + +Create a [Seqera](https://cloud.seqera.io/tokens) access token via **Your Tokens** in the user menu. + +Seqerakit reads your access token from the `TOWER_ACCESS_TOKEN` environment variable: + +```shell-session +export TOWER_ACCESS_TOKEN= +``` + +For Enterprise installations, specify the custom API endpoint used to connect to Seqera. Export the API endpoint environment variable: + +```shell-session +export TOWER_API_ENDPOINT= +``` + +By default, this is set to `https://api.cloud.seqera.io` to connect to Seqera Cloud. + +### Usage + +To confirm the installation of `seqerakit`, configuration of the Platform CLI, and connection to Seqera is working as expected, run this command: + +```shell-session +seqerakit --info +``` + +This runs the `tw info` command under the hood. + +Use `--version` or `-v` to retrieve the current version of your `seqerakit` installation: + +```shell-session +seqerakit --version +``` + +Use the `--help` or `-h` option to list the available commands and their associated options: + +```shell-session +seqerakit --help +``` + +See [Commands](./commands.mdx) for detailed instructions to use Seqerakit. \ No newline at end of file diff --git a/platform_versioned_docs/version-23.3/seqerakit/templates.mdx b/platform_versioned_docs/version-23.3/seqerakit/templates.mdx new file mode 100644 index 000000000..f33062192 --- /dev/null +++ b/platform_versioned_docs/version-23.3/seqerakit/templates.mdx @@ -0,0 +1,200 @@ +--- +title: "Templates" +description: "Seqerakit YAML configuration file templates and instructions" +date: "8 Nov 2024" +tags: [seqerakit, cli, automation, yaml, configuration] +toc_max_heading_level: 4 +--- + +import CodeBlock from '@theme/CodeBlock'; +import Organizations from '!!raw-loader!./templates/organizations.yaml'; +import Members from '!!raw-loader!./templates/members.yaml'; +import Workspaces from '!!raw-loader!./templates/workspaces.yaml'; +import Teams from '!!raw-loader!./templates/teams.yaml'; +import Participants from '!!raw-loader!./templates/participants.yaml'; +import ComputeEnvironments from '!!raw-loader!./templates/compute-envs.yaml'; +import Pipelines from '!!raw-loader!./templates/pipelines.yaml'; +import Credentials from '!!raw-loader!./templates/credentials.yaml'; +import Datasets from '!!raw-loader!./templates/datasets.yaml'; +import Labels from '!!raw-loader!./templates/labels.yaml'; +import Launch from '!!raw-loader!./templates/launch.yaml'; +import Secrets from '!!raw-loader!./templates/secrets.yaml'; +import Actions from '!!raw-loader!./templates/actions.yaml'; +import EndToEnd from '!!raw-loader!./templates/seqerakit-e2e.yaml'; + +Customize YAML configuration templates to use in `seqerakit` commands to create, update, or delete Seqera resources. Create or delete multiple resources with a single command by combining them into a single configuration file. + +To use the templates on this page: +1. Copy the template text or download the YAML files you need. +1. Edit the values to specify your resource details, and save as a `.yaml` file. +1. Specify the YAML template file in your `seqerakit` commands: + - To create the resources specified in the file: + ```shell-session + seqerakit file.yaml + ``` + - To delete the existing resources specified in the file: + ```shell-session + seqerakit file.yaml --delete + ``` + +:::info +See [Specify targets](./commands.mdx#specify-targets) to create or delete only selected resources from configuration templates that contain multiple resource entries. +::: + +See [End-to-end example](#end-to-end-example) for a template that contains examples of all Seqera resources that can be created with Seqerakit. + +### Administration + +Manage organizations, organization members, workspaces, teams, and participants. + +#### Organizations + +Add or delete organizations. + + +{Organizations} + + +[Download organizations.yaml](./templates/organizations.yaml) + +#### Members + +Add or delete organization members. + + +{Members} + + +[Download members.yaml](./templates/members.yaml) + +#### Workspaces + +Add or delete workspaces. + + +{Workspaces} + + +[Download workspaces.yaml](./templates/workspaces.yaml) + +#### Teams + +Add or delete teams. + + +{Teams} + + +[Download teams.yaml](./templates/teams.yaml) + +#### Participants + +Add or delete participants in workspaces and teams. + + +{Participants} + + +[Download participants.yaml](./templates/participants.yaml) + +### Credentials + +Add or delete compute environment, Git, and container registry credentials in workspaces. + + +{Credentials} + + +[Download credentials.yaml](./templates/credentials.yaml) + +### Compute environments + +Add or delete compute environments. + + +{ComputeEnvironments} + + +[Download compute-envs.yaml](./templates/compute-envs.yaml) + +### Pipelines + +Add or delete pipelines in workspace Launchpads. + + +{Pipelines} + + +[Download pipelines.yaml](./templates/pipelines.yaml) + +### Launch + +Launch a Nextflow pipeline. + + +{Launch} + + +[Download launch.yaml](./templates/launch.yaml) + +### Datasets + +Add or delete workspace datasets for pipeline input data. + + +{Datasets} + + +[Download datasets.yaml](./templates/datasets.yaml) + +### Labels + +Add or delete labels and resource labels to apply to workspace compute environments, pipelines, and runs. + + +{Labels} + + +[Download labels.yaml](./templates/labels.yaml) + +### Secrets + +Add or delete user and workspace secrets. + + +{Secrets} + + +[Download secrets.yaml](./templates/secrets.yaml) + +### Actions + +Add or delete pipeline actions. + + +{Actions} + + +[Download actions.yaml](./templates/actions.yaml) + +### End-to-end example + +A template to create the follwing resources: + +- An organization +- A workspace +- A team +- Participants +- Credentials +- Secrets +- Compute environments +- Datasets +- Pipelines + +The templates also contains `launch` entries to launch saved pipelines. + + +{EndToEnd} + + +[Download seqerakit-e2e.yaml](./templates/seqerakit-e2e.yaml) \ No newline at end of file diff --git a/platform_versioned_docs/version-23.3/seqerakit/templates/actions.yaml b/platform_versioned_docs/version-23.3/seqerakit/templates/actions.yaml new file mode 100644 index 000000000..5b65933ea --- /dev/null +++ b/platform_versioned_docs/version-23.3/seqerakit/templates/actions.yaml @@ -0,0 +1,24 @@ +## To see the full list of options available, run: "tw actions add -h" +actions: + - type: 'github' # required + name: 'your-github-action' # required + pipeline: 'https://github.com/my_username/my_repo' # required + workspace: 'your-organization/workspace-1' # required + compute-env: 'your-aws-compute-environment' # required + work-dir: 's3://your-bucket' # required + revision: 'main' # required + profile: 'test' # optional + params: # optional + outdir: 's3://your-bucket/results' + overwrite: True # optional + - type: 'tower' # required + name: 'your-tower-action' # required + pipeline: 'https://github.com/my_username/my_repo' # required + workspace: 'your-organization/workspace-1' # required + compute-env: 'your-aws-compute-environment' # required + work-dir: 's3://your-bucket' # required + revision: 'main' # required + profile: 'test' # optional + params: # optional + outdir: 's3://your-bucket/results' + overwrite: True # optional diff --git a/platform_versioned_docs/version-23.3/seqerakit/templates/compute-envs.yaml b/platform_versioned_docs/version-23.3/seqerakit/templates/compute-envs.yaml new file mode 100644 index 000000000..237f66421 --- /dev/null +++ b/platform_versioned_docs/version-23.3/seqerakit/templates/compute-envs.yaml @@ -0,0 +1,36 @@ +## To see the full list of options available, run: "tw compute-envs add -h" +## The options required to create compute environments can be specified: +## 1. Explicitly in this file +## 2. Via a JSON file exported from Seqera with the "tw compute-envs export" command +compute-envs: +# To create a compute environment from a JSON configuration file (AWS Example) + - name: 'your-aws-compute-environment' # required + workspace: 'your-organization/workspace-1' # required + credentials: 'your-aws-credentials' # required + wait: 'AVAILABLE' # optional + file-path: './compute-envs/aws_compute_environment.json' # required + overwrite: True # optional + +# To create a compute environment with options specified through YAML (AWS Example) + - type: aws-batch # required + config-mode: forge # required for AWS and Azure (forge or manual) + name: 'your-aws-compute-environment' # required + workspace: 'your-organization/workspace-1' # required + credentials: 'your-aws-credentials' # required + region: 'eu-west-1' # required + work-dir: 's3://your-bucket' # required + provisioning-model: 'SPOT' # optional + fusion-v2: False # optional + wave: False # optional + fargate: False # optional + fast-storage: False # optional + instance-types: 'c6i,r6i,m6i' # optional, comma-separated list + no-ebs-auto-scale: True # optional + max-cpus: 500 # required + labels: 'label1,label2' # optional, comma-separated list + vpc-id: 'vpc-1234567890' # optional + subnets: 'subnet-1234567890,subnet-1234567891' # optional, comma-separated list + security-groups: 'sg-1234567890,sg-1234567891' # optional, comma-separated list + allow-buckets: 's3://your-bucket,s3://your-other-bucket' # optional, comma-separated list + wait: 'AVAILABLE' # optional + overwrite: False # optional \ No newline at end of file diff --git a/platform_versioned_docs/version-23.3/seqerakit/templates/credentials.yaml b/platform_versioned_docs/version-23.3/seqerakit/templates/credentials.yaml new file mode 100644 index 000000000..e63db59eb --- /dev/null +++ b/platform_versioned_docs/version-23.3/seqerakit/templates/credentials.yaml @@ -0,0 +1,49 @@ +## To see the full list of options available, run: "tw credentials add -h" +## To avoid exposing sensitive information about your credentials, +## use environment variables to supply passwords and secret keys. +credentials: + - type: 'github' # required + name: 'your-github-credentials' # required + workspace: 'your-organization/workspace-1' # required + username: 'your-username' # required + password: '$SEQPLATFORM_GITHUB_PASSWORD' # required + overwrite: True # optional + + - type: 'container-reg' # required + name: 'your-dockerhub-credentials' # required + workspace: 'your-organization/workspace-1' # required + username: 'your-username' # required + password: '$DOCKERHUB_PASSWORD' # required + registry: 'docker.io' # required + overwrite: True # optional + + - type: 'google' # required + name: 'your-google-credentials' # required + workspace: 'your-organization/workspace-1' # required + key: '$GOOGLE_KEY' # required + overwrite: True # optional + + - type: 'aws' # required + name: 'your-aws-credentials' # required + workspace: 'your-organization/workspace-1' # required + access-key: '$AWS_ACCESS_KEY_ID' # required + secret-key: '$AWS_SECRET_ACCESS_KEY' # required + assume-role-arn: '$AWS_ASSUME_ROLE_ARN' # required + overwrite: True # optional + + - type: 'azure' # required + name: 'your-azure-credentials' # required + workspace: 'your-organization/workspace-1' # required + batch-key: '$AZURE_BATCH_KEY' # required + batch-name: 'your-batch-name' # required + storage-key: '$AZURE_STORAGE_KEY' # required + storage-name: 'your-storage-name' # required + overwrite: True # optional + + - type: 'codecommit' # required + name: 'your-codecommit-credentials' # required + workspace: 'your-organization/workspace-1' # required + access-key: '$CODECOMMIT_USER' # required + secret-key: '$CODECOMMIT_PASSWORD' # required + base-url: '$CODECOMMIT_BASEURL' # optional + overwrite: False # optional diff --git a/platform_versioned_docs/version-23.3/seqerakit/templates/datasets.yaml b/platform_versioned_docs/version-23.3/seqerakit/templates/datasets.yaml new file mode 100644 index 000000000..b93364ddc --- /dev/null +++ b/platform_versioned_docs/version-23.3/seqerakit/templates/datasets.yaml @@ -0,0 +1,8 @@ +## To see the full list of options available, run: "tw datasets add -h" +datasets: + - name: 'your-dataset' # required + description: 'Your dataset description' # optional + header: true # optional + workspace: 'your-organization/workspace-1' # required + file-path: './datasets/dataset.csv' # required + overwrite: True # optional \ No newline at end of file diff --git a/platform_versioned_docs/version-23.3/seqerakit/templates/labels.yaml b/platform_versioned_docs/version-23.3/seqerakit/templates/labels.yaml new file mode 100644 index 000000000..f8496a7f4 --- /dev/null +++ b/platform_versioned_docs/version-23.3/seqerakit/templates/labels.yaml @@ -0,0 +1,6 @@ +## To see the full list of options available, run: "tw labels add -h" +labels: + - name: 'label_name' # required + value: 'label_value' # required + workspace: 'your-organization/workspace-1' # required + overwrite: True # optional diff --git a/platform_versioned_docs/version-23.3/seqerakit/templates/launch.yaml b/platform_versioned_docs/version-23.3/seqerakit/templates/launch.yaml new file mode 100644 index 000000000..745c63c04 --- /dev/null +++ b/platform_versioned_docs/version-23.3/seqerakit/templates/launch.yaml @@ -0,0 +1,23 @@ +## To see the full list of options available, run: "tw launch -h" +## The options will vary if you are launching a pipeline: +## 1. Pre-configured in the Seqera Launchpad with most options already specified +## 2. From source via a remote Git repository URL where you must explicitly specify all launch options +## Note: overwrite is not supported for "tw launch" + +launch: + - name: 'launchpad-launch' # required + workspace: 'your-organization/workspace-1' # required + pipeline: 'nf-core-rnaseq' # required + params: # optional + outdir: 's3://your-bucket/results' + + - name: 'remote-launch' # required + workspace: 'your-organization/workspace-1' # required + compute-env: 'your-aws-compute-environment' # required + pipeline: 'https://github.com/my_username/my_repo' # required + work-dir: 's3://your-bucket' # optional + profile: 'test' # optional + revision: 'main' # optional + params-file: './pipelines/params.yml' # optional + config: './pipelines/nextflow.config' # optional + pre-run: './pipelines/pre-run.txt' # optional diff --git a/platform_versioned_docs/version-23.3/seqerakit/templates/members.yaml b/platform_versioned_docs/version-23.3/seqerakit/templates/members.yaml new file mode 100644 index 000000000..f06b9f4af --- /dev/null +++ b/platform_versioned_docs/version-23.3/seqerakit/templates/members.yaml @@ -0,0 +1,5 @@ +## To see the full list of options available, run: "tw members add -h" +members: + - user: 'user@domain.com' # required + organization: 'your-organization' # required + overwrite: True # optional \ No newline at end of file diff --git a/platform_versioned_docs/version-23.3/seqerakit/templates/organizations.yaml b/platform_versioned_docs/version-23.3/seqerakit/templates/organizations.yaml new file mode 100644 index 000000000..6cf0518d2 --- /dev/null +++ b/platform_versioned_docs/version-23.3/seqerakit/templates/organizations.yaml @@ -0,0 +1,8 @@ +## To see the full list of options available, run: "tw organizations add -h" +organizations: + - name: 'your-organization' # required + full-name: 'Your Organization LLC' # required + description: 'Your organization description' # optional + location: 'Global' # optional + website: 'https://domain.com/' # optional + overwrite: True # optional \ No newline at end of file diff --git a/platform_versioned_docs/version-23.3/seqerakit/templates/participants.yaml b/platform_versioned_docs/version-23.3/seqerakit/templates/participants.yaml new file mode 100644 index 000000000..3c5e77da6 --- /dev/null +++ b/platform_versioned_docs/version-23.3/seqerakit/templates/participants.yaml @@ -0,0 +1,10 @@ +## To see the full list of options available, run: "tw participants add -h" +participants: + - name: 'your-team' # required + type: 'TEAM' # required + workspace: 'your-organization/workspace-1' # required + role: 'ADMIN' # required + - name: 'team-member@domain.com' # required + type: 'MEMBER' # required + workspace: 'your-organization/workspace-1' # required + role: 'LAUNCH' # required \ No newline at end of file diff --git a/platform_versioned_docs/version-23.3/seqerakit/templates/pipelines.yaml b/platform_versioned_docs/version-23.3/seqerakit/templates/pipelines.yaml new file mode 100644 index 000000000..017d72527 --- /dev/null +++ b/platform_versioned_docs/version-23.3/seqerakit/templates/pipelines.yaml @@ -0,0 +1,24 @@ +## To see the full list of options available, run: "tw pipelines add -h" +## The options required to create pipelines can be specified: +## 1. Explicitly in this file +## 2. Via a JSON file exported from Seqera with the "tw pipelines export" command +pipelines: + - name: 'your-first-pipeline' # required + workspace: 'your-organization/workspace-1' # required + description: 'Options specified in this file' # optional + compute-env: 'your-aws-compute-environment' # required + work-dir: 's3://your-bucket' # optional + profile: 'test' # optional + revision: 'main' # required + params: # optional + outdir: 's3://your-bucket/results' + config: './pipelines/nextflow.config' # optional + pre-run: './pipelines/pre-run.txt' # optional + url: 'https://github.com/my_username/my_repo' # required + overwrite: True # optional + - name: 'your-second-pipeline' # required + workspace: 'your-organization/workspace-1' # required + description: 'Options specified via JSON file' # optional + compute-env: 'your-aws-compute-environment' # required + file-path: './pipelines/pipeline.json' # required + overwrite: True # optional diff --git a/platform_versioned_docs/version-23.3/seqerakit/templates/secrets.yaml b/platform_versioned_docs/version-23.3/seqerakit/templates/secrets.yaml new file mode 100644 index 000000000..743faba3b --- /dev/null +++ b/platform_versioned_docs/version-23.3/seqerakit/templates/secrets.yaml @@ -0,0 +1,6 @@ +## To see the full list of options available, run: "tw secrets add -h" +secrets: + - name: 'your-secret' # required + workspace: 'your-organization/workspace-1' # required + value: 'your-secret-value' # required + overwrite: True # optional \ No newline at end of file diff --git a/platform_versioned_docs/version-23.3/seqerakit/templates/seqerakit-e2e.yaml b/platform_versioned_docs/version-23.3/seqerakit/templates/seqerakit-e2e.yaml new file mode 100644 index 000000000..fcdea506e --- /dev/null +++ b/platform_versioned_docs/version-23.3/seqerakit/templates/seqerakit-e2e.yaml @@ -0,0 +1,148 @@ +organizations: + - name: 'your-organization' # required + full-name: 'Your organization LLC' # required + description: 'Organization created E2E with seqerakit CLI scripting' # optional + location: 'Global' # optional + website: 'https://yourdomain.com/' # optional + overwrite: True # optional + +teams: + - name: 'pipelines-team' # required + organization: 'your-organization' # required + description: 'Pipelines team @ Your organization' # optional + members: # optional + - 'user1@domain.com' + - 'user2@domain.com' + - 'user3@domain.com' + overwrite: True # optional + +workspaces: + - name: 'workspace-1' # required + full-name: 'Workspace one' # required + organization: 'your-organization' # required + description: 'Workspace created E2E with seqerakit CLI scripting' # optional + visibility: 'PRIVATE' # optional + overwrite: True # optional + +participants: + - name: 'pipelines-team' # required + type: 'TEAM' # required + workspace: 'your-organization/workspace-1' # required + role: 'ADMIN' # required + overwrite: True # optional + - name: 'user@domain.com' # required + type: 'MEMBER' # required + workspace: 'your-organization/workspace-1' # required + role: 'LAUNCH' # required + overwrite: True # optional + +credentials: + - type: 'github' # required + name: 'github-credentials' # required + workspace: 'your-organization/workspace-1' # required + username: 'user1' # required + password: '$TOWER_GITHUB_PASSWORD' # required + overwrite: True # optional + - type: 'container-reg' # required + name: 'dockerhub-credentials' # required + workspace: 'your-organization/workspace-1' # required + username: 'user1' # required + password: '$DOCKERHUB_PASSWORD' # required + registry: 'docker.io' # required + overwrite: True # optional + - type: 'aws' # required + name: 'aws-credentials' # required + workspace: 'your-organization/workspace-1' # required + access-key: '$AWS_ACCESS_KEY_ID' # required + secret-key: '$AWS_SECRET_ACCESS_KEY' # required + assume-role-arn: '$AWS_ASSUME_ROLE_ARN' # required + overwrite: True # optional + +secrets: + - name: 'SENTIEON_LICENSE_BASE64' # required + workspace: 'your-organization/workspace-1' # required + value: '$SENTIEON_LICENSE_BASE64' # required + overwrite: True # optional + +compute-envs: +# To create a compute environment with options specified through YAML (AWS Example) + - type: aws-batch # required + config-mode: forge # required for AWS and Azure + name: 'your-aws-compute-environment' # required + workspace: 'your-organization/workspace-1' # required + credentials: 'your-aws-credentials' # required + region: 'eu-west-1' # required + work-dir: 's3://your-bucket' # required + provisioning-model: 'SPOT' # optional + fusion-v2: False # optional + wave: False # optional + fargate: False # optional + fast-storage: False # optional + instance-types: 'c6i,r6i,m6i' # optional, comma-separated list + no-ebs-auto-scale: True # optional + max-cpus: 500 # required + labels: 'label1,label2' # optional, comma-separated list + vpc-id: 'vpc-1234567890' # optional + subnets: 'subnet-1234567890,subnet-1234567891' # optional, comma-separated list + security-groups: 'sg-1234567890,sg-1234567891' # optional, comma-separated list + allow-buckets: 's3://your-bucket,s3://your-other-bucket' # optional, comma-separated list + wait: 'AVAILABLE' # optional + overwrite: False # optional + +# To create a compute environment from a JSON configuration file (AWS Example) + - name: 'your-aws-compute-environment' # required + workspace: 'your-organization/workspace-1' # required + credentials: 'your-aws-credentials' # required + wait: 'AVAILABLE' # optional + file-path: './compute-envs/your_aws_compute_environment.json' # required + overwrite: True # optional + +datasets: + - name: 'rnaseq_samples' # required + description: 'Samplesheet to run the nf-core/rnaseq pipeline end-to-end' # optional + header: true # optional + workspace: 'your-organization/workspace-1' # required + file-path: 'path/to/rnaseq_samples.csv' # required + overwrite: True # optional + +pipelines: +# To create a pipeline with options specified through YAML + - name: 'nf-core-rnaseq' # required + url: 'https://github.com/nf-core/rnaseq' # required + workspace: 'your-organization/workspace-1' # required + description: 'RNA sequencing analysis pipeline with gene/isoform counts and extensive quality control.' # optional + compute-env: 'your-aws-compute-environment' # required + work-dir: 's3://your-bucket/pipeline/working/directory' # optional + profile: 'test' # optional + revision: '3.12.0' # required + params: # optional + outdir: 's3://your-bucket/nf-core-rnaseq/results' + config: 'path/to/nextflow.config' # optional + pre-run: 'path/to/pipeline/pre_run.txt' # optional + overwrite: True # optional + +# To create a pipeline from a JSON configuration file + - name: 'nf-core-sarek' # required + workspace: 'your-organization/workspace-1' # required + compute-env: 'your-aws-compute-environment' # required + file-path: 'path/to/nf-core-sarek_pipeline.json' # required + overwrite: True # optional + +launch: +# To launch a preconfigured pipeline from your workspace Launchpad + - name: 'nf-core-rnaseq-launchpad' # required + workspace: 'your-organization/workspace-1' # required + pipeline: 'nf-core-rnaseq' # required + compute-env: 'your-aws-compute-environment' # optional (defaults to workspace primary compute environment) + +# To launch an unsaved pipeline, include the pipeline repository URL and launch details + - name: 'nf-core-rnaseq-remote' # required + workspace: 'your-organization/workspace-1' # required + pipeline: 'https://github.com/nf-core/rnaseq' # required + compute-env: 'your-aws-compute-environment' # optional (defaults to workspace primary compute environment) + work-dir: 's3://your-bucket/pipeline/working/directory' # optional + profile: 'test' # optional + revision: '3.12.0' # optional + params-file: 'path/to/pipelines/nf_core_rnaseq_params.yml' # optional + config: 'path/to/pipelines/nextflow.config' # optional + pre-run: 'path/to/pipelines/pre_run.txt' # optional diff --git a/platform_versioned_docs/version-23.3/seqerakit/templates/teams.yaml b/platform_versioned_docs/version-23.3/seqerakit/templates/teams.yaml new file mode 100644 index 000000000..b1258de24 --- /dev/null +++ b/platform_versioned_docs/version-23.3/seqerakit/templates/teams.yaml @@ -0,0 +1,8 @@ +## To see the full list of options available, run: "tw teams add -h" +teams: + - name: 'your-team' # required + organization: 'your-organization' # required + description: 'Your team description.' # optional + members: # optional + - 'team-member@domain.com' + overwrite: True # optional \ No newline at end of file diff --git a/platform_versioned_docs/version-23.3/seqerakit/templates/workspaces.yaml b/platform_versioned_docs/version-23.3/seqerakit/templates/workspaces.yaml new file mode 100644 index 000000000..ca9392340 --- /dev/null +++ b/platform_versioned_docs/version-23.3/seqerakit/templates/workspaces.yaml @@ -0,0 +1,8 @@ +## To see the full list of options available, run: "tw workspaces add -h" +workspaces: + - name: 'workspace-1' # required + full-name: 'Workspace one' # required + organization: 'your-organization' # required + description: 'Your workspace description' # optional + visibility: 'PRIVATE' # optional + overwrite: True # optional \ No newline at end of file diff --git a/platform_versioned_docs/version-23.3/seqerakit/yaml-configuration.mdx b/platform_versioned_docs/version-23.3/seqerakit/yaml-configuration.mdx new file mode 100644 index 000000000..965161262 --- /dev/null +++ b/platform_versioned_docs/version-23.3/seqerakit/yaml-configuration.mdx @@ -0,0 +1,154 @@ +--- +title: "YAML configuration" +description: "seqerakit YAML configuration file options" +date: "21 Oct 2024" +tags: [seqerakit, cli, automation, yaml, configuration] +--- + +Seqerakit supports the creation and deletion of the following Seqera Platform resources, listed here with their respective Platform CLI resource names: + +- Pipeline actions: `actions` +- Compute environments: `compute-envs` +- Credentials: `credentials` +- Datasets: `datasets` +- Labels (including resource labels): `labels` +- Pipeline launch: `launch` +- Organization members: `members` +- Organizations: `organizations` +- Workspace and team participants: `participants` +- Pipelines: `pipelines` +- Pipeline secrets: `secrets` +- Teams: `teams` +- Workspaces: `workspaces` + +To determine the options to provide as definitions in your YAML file, run the Platform CLI help command for the resource you want to create. + +1. Retrieve CLI options: + + Obtain a list of available CLI options for defining your YAML file with the Platform CLI `help` command. For example, to add a pipeline to your workspace, view the options for adding a pipeline: + + ```shell-session + tw pipelines add -h + ``` + + ```shell-session + Usage: tw pipelines add [OPTIONS] PIPELINE_URL + + Add a workspace pipeline. + + Parameters: + * PIPELINE_URL Nextflow pipeline URL. + + Options: + * -n, --name= Pipeline name. + -w, --workspace= Workspace numeric identifier (TOWER_WORKSPACE_ID as default) or workspace reference as OrganizationName/WorkspaceName + -d, --description= Pipeline description. + --labels=[,...] List of labels seperated by coma. + -c, --compute-env= Compute environment name. + --work-dir= Path where the pipeline scratch data is stored. + -p, --profile=[,...] Comma-separated list of one or more configuration profile names you want to use for this pipeline execution. + --params-file= Pipeline parameters in either JSON or YML format. + --revision= A valid repository commit Id, tag or branch name. + ... + ``` + +1. Define key-value pairs in YAML: + + Translate each CLI option into a key-value pair in the YAML file. The structure of your YAML file should reflect the hierarchy and format of the CLI options. For example: + + ```yaml + pipelines: + - name: 'my_first_pipeline' + url: 'https://github.com/username/my_pipeline' + workspace: 'my_organization/my_workspace' + description: 'My test pipeline' + labels: 'yeast,test_data' + compute-env: 'my_compute_environment' + work-dir: 's3://my_bucket' + profile: 'test' + params-file: '/path/to/params.yaml' + revision: '1.0' + ``` + + In this example: + + - The keys (`name`, `url`, `workspace`, and so forth) are the keys derived from the CLI options. + - The corresponding values are user-defined. + +#### Best practices + +- The indentation and structure of the YAML file must be correct — YAML is sensitive to formatting. +- Use quotes around strings that contain special characters or spaces. +- To list multiple values (such as multiple `labels`, `instance-types`, or `allow-buckets`), separate values with commas. This is shown with `labels` in the preceding example. +- For complex configurations, see [Templates](./templates.mdx). + +### Templates + +See [Templates](./templates.mdx) for YAML file templates for each of the entities that can be created in Seqera. + +### YAML Configuration Options + +Some options handled specially by `seqerakit` or not exposed as `tw` CLI options can be provided in your YAML configuration file. + +#### Pipeline parameters using `params` and `params-file` + +To specify pipeline parameters, use `params:` to specify a list of parameters or `params-file:` to point to a parameters file. + +For example, to specify pipeline parameters within your YAML: + +```yaml +params: + outdir: 's3://path/to/outdir' + fasta: 's3://path/to/reference.fasta' +``` + +To specify a file containing pipeline parameters: + +```yaml +params-file: '/path/to/my/parameters.yaml' +``` + +Or provide both: + +```yaml +params-file: '/path/to/my/parameters.yaml' +params: + outdir: 's3://path/to/outdir' + fasta: 's3://path/to/reference.fasta' +``` + +:::note +If duplicate parameters are provided, the parameters provided as key-value pairs inside the `params` nested dictionary of the YAML file will take precedence **over** values in the `params-file`. +::: + +#### Overwrite + +For every entity defined in your YAML file, specify `overwrite: True` to overwrite any existing Seqera entities of the same name. + +Seqerakit will first check to see if the name of the entity exists. If so, it will invoke a `tw delete` command before attempting to create it based on the options defined in the YAML file. + +```shell-session +DEBUG:root: Overwrite is set to 'True' for organizations + +DEBUG:root: Running command: tw -o json organizations list +DEBUG:root: The attempted organizations resource already exists. Overwriting. + +DEBUG:root: Running command: tw organizations delete --name $SEQERA_ORGANIZATION_NAME +DEBUG:root: Running command: tw organizations add --name $SEQERA_ORGANIZATION_NAME --full-name $SEQERA_ORGANIZATION_NAME --description 'Example of an organization' +``` + +#### Specify JSON configuration files with `file-path` + +The Platform CLI allows the export and import of entities through JSON configuration files for pipelines and compute environments. To use these files to add a pipeline or compute environment to a workspace, use the `file-path` key to specify a path to a JSON configuration file. + +An example of the `file-path` option is provided in the [compute-envs.yaml](./templates/compute-envs.yaml) template: + +```yaml +compute-envs: + - name: 'my_aws_compute_environment' # required + workspace: 'my_organization/my_workspace' # required + credentials: 'my_aws_credentials' # required + wait: 'AVAILABLE' # optional + file-path: './compute-envs/my_aws_compute_environment.json' # required + overwrite: True +``` diff --git a/platform_versioned_docs/version-23.4/seqerakit/commands.mdx b/platform_versioned_docs/version-23.4/seqerakit/commands.mdx new file mode 100644 index 000000000..533d64777 --- /dev/null +++ b/platform_versioned_docs/version-23.4/seqerakit/commands.mdx @@ -0,0 +1,126 @@ +--- +title: "Commands" +description: "Seqerakit command options" +date: "21 Oct 2024" +tags: [seqerakit, cli, automation, commands] +--- + +Use the `--help` or `-h` option to list available commands and options: + +```shell-session +seqerakit --help +``` + +### Input + +Seqerakit supports input through paths to YAML configuration files or directly from standard input (`stdin`). + +- Using file path: + + ```shell-session + seqerakit file.yaml + ``` + +- Using `stdin`: + + ```shell-session + cat file.yaml | seqerakit - + ``` + +See [Define your YAML file using CLI options](#define-your-yaml-file-using-cli-options) for guidance on formatting your input YAML files. + +### Dryrun + +Confirm that your configuration and command are correct before creating resources in your Seqera account, particularly when automating the end-to-end creation of multiple entities at once. To print the commands that would be executed with Platform CLI when using a YAML file, run your `seqerakit` command with the `--dryrun` option: + +```shell-session +seqerakit file.yaml --dryrun +``` + +### Specify targets + +When using a YAML file as an input that defines multiple resources, use the `--targets` option to specify which resources to create. This option accepts a comma-separated list of resource names. + +Supported resource names include: + +- `actions` +- `compute-envs` +- `credentials` +- `datasets` +- `labels` +- `launch` +- `members` +- `organizations` +- `participants` +- `pipelines` +- `secrets` +- `teams` +- `workspaces` + +For example, given a `test.yaml` file that defines the following resources: + +```yaml +workspaces: + - name: 'workspace-1' + organization: 'seqerakit' +... +compute-envs: + - name: 'compute-env' + type: 'aws-batch forge' + workspace: 'seqerakit/workspace-1' +... +pipelines: + - name: 'hello-world' + url: 'https://github.com/nextflow-io/hello' + workspace: 'seqerakit/workspace-1' + compute-env: 'compute-env' +... +``` + +You can target the creation of `pipelines` only by running: + +```shell-session +seqerakit test.yaml --targets pipelines +``` + +This command will create only the pipelines defined in the YAML file and ignore `workspaces` and `compute-envs`. + +To create both workspaces and pipelines, run: + +```shell-session +seqerakit test.yaml --targets workspaces,pipelines +``` + +### Delete resources + +Instead of adding or creating resources, specify the `--delete` option to recursively delete resources in your YAML file: + +```shell-session +seqerakit file.yaml --delete +``` + +For example, if you have a `file.yaml` that defines an organization, workspace, team, credentials, and compute environment that have already been created, run `seqerakit file.yaml --delete` to recursively delete the same resources. + +### Use `tw`-specific CLI options + +Specify `tw`-specific CLI options with the `--cli=` option: + +```shell-session +seqerakit file.yaml --cli="--arg1 --arg2" +``` + +See [CLI commands](../cli/commands.mdx) or run `tw -h` for the full list of options. + +:::note +The `--verbose` option for `tw` CLI is currently not supported in `seqerakit` commands. +::: + +#### Example: HTTP-only connections + +The Platform CLI expects to connect to a Seqera instance that is secured by a TLS certificate. If your Seqera Enterprise instance does not present a certificate, you must run your `tw` commands with the `--insecure` option. + +To use `tw`-specific CLI options such as `--insecure`, use the `--cli=` option, followed by the options to use enclosed in double quotes: + +```shell-session +seqerakit file.yaml --cli="--insecure" +``` diff --git a/platform_versioned_docs/version-23.4/seqerakit/installation.mdx b/platform_versioned_docs/version-23.4/seqerakit/installation.mdx new file mode 100644 index 000000000..58eac0d41 --- /dev/null +++ b/platform_versioned_docs/version-23.4/seqerakit/installation.mdx @@ -0,0 +1,117 @@ +--- +title: "Installation" +description: "Seqerakit installation options" +date: "21 Oct 2024" +tags: [seqerakit, cli, automation, installation] +--- + + +Seqerakit is a Python wrapper that sets [Platform CLI](../cli/overview.mdx) command options using YAML configuration files. Individual commands and configuration parameters can be chained together to automate the end-to-end creation of all Seqera Platform entities. + +As an extension of the Platform CLI, Seqerakit enables: + +- **Infrastructure as code**: Users manage and provision their infrastructure from the command line. +- **Simple configuration**: All Platform CLI command-line options can be defined in simple YAML format. +- **Automation**: End-to-end creation of Seqera entities, from adding an organization to launching pipelines. + +### Installation + +Seqerakit has three dependencies: + +1. [Seqera Platform CLI (`>=0.10.1`)](https://github.com/seqeralabs/tower-cli/releases) +2. [Python (`>=3.8`)](https://www.python.org/downloads/) +3. [PyYAML](https://pypi.org/project/PyYAML/) + +#### Pip + +If you already have [Platform CLI](../cli/installation.mdx) and Python installed on your system, install Seqerakit directly from [PyPI](https://pypi.org/project/seqerakit/): + +```console +pip install seqerakit +``` + +Overwrite an existing installation to use the latest version: + +```console +pip install --upgrade --force-reinstall seqerakit +``` + +#### Conda + +To install `seqerakit` and its dependencies via Conda, first configure the correct channels: + +```console +conda config --add channels bioconda +conda config --add channels conda-forge +conda config --set channel_priority strict +``` + +Then create a conda environment with `seqerakit` installed: + +```console +conda env create -n seqerakit seqerakit +conda activate seqerakit +``` + +#### Local development installation + +Install the development branch of `seqerakit` on your local machine to test the latest features and updates: + +1. You must have [Python](https://www.python.org/downloads/) and [Git](https://git-scm.com/downloads) installed on your system. +1. To install directly from pip: + ```shell-session + pip install git+https://github.com/seqeralabs/seqera-kit.git@dev + ``` +1. Alternatively, clone the repository locally and install manually: + ```shell-session + git clone https://github.com/seqeralabs/seqera-kit.git + cd seqera-kit + git checkout dev + pip install . + ``` +1. Verify your installation: + ```shell-session + pip show seqerakit + ``` + +### Configuration + +Create a [Seqera](https://cloud.seqera.io/tokens) access token via **Your Tokens** in the user menu. + +Seqerakit reads your access token from the `TOWER_ACCESS_TOKEN` environment variable: + +```shell-session +export TOWER_ACCESS_TOKEN= +``` + +For Enterprise installations, specify the custom API endpoint used to connect to Seqera. Export the API endpoint environment variable: + +```shell-session +export TOWER_API_ENDPOINT= +``` + +By default, this is set to `https://api.cloud.seqera.io` to connect to Seqera Cloud. + +### Usage + +To confirm the installation of `seqerakit`, configuration of the Platform CLI, and connection to Seqera is working as expected, run this command: + +```shell-session +seqerakit --info +``` + +This runs the `tw info` command under the hood. + +Use `--version` or `-v` to retrieve the current version of your `seqerakit` installation: + +```shell-session +seqerakit --version +``` + +Use the `--help` or `-h` option to list the available commands and their associated options: + +```shell-session +seqerakit --help +``` + +See [Commands](./commands.mdx) for detailed instructions to use Seqerakit. \ No newline at end of file diff --git a/platform_versioned_docs/version-23.4/seqerakit/templates.mdx b/platform_versioned_docs/version-23.4/seqerakit/templates.mdx new file mode 100644 index 000000000..f33062192 --- /dev/null +++ b/platform_versioned_docs/version-23.4/seqerakit/templates.mdx @@ -0,0 +1,200 @@ +--- +title: "Templates" +description: "Seqerakit YAML configuration file templates and instructions" +date: "8 Nov 2024" +tags: [seqerakit, cli, automation, yaml, configuration] +toc_max_heading_level: 4 +--- + +import CodeBlock from '@theme/CodeBlock'; +import Organizations from '!!raw-loader!./templates/organizations.yaml'; +import Members from '!!raw-loader!./templates/members.yaml'; +import Workspaces from '!!raw-loader!./templates/workspaces.yaml'; +import Teams from '!!raw-loader!./templates/teams.yaml'; +import Participants from '!!raw-loader!./templates/participants.yaml'; +import ComputeEnvironments from '!!raw-loader!./templates/compute-envs.yaml'; +import Pipelines from '!!raw-loader!./templates/pipelines.yaml'; +import Credentials from '!!raw-loader!./templates/credentials.yaml'; +import Datasets from '!!raw-loader!./templates/datasets.yaml'; +import Labels from '!!raw-loader!./templates/labels.yaml'; +import Launch from '!!raw-loader!./templates/launch.yaml'; +import Secrets from '!!raw-loader!./templates/secrets.yaml'; +import Actions from '!!raw-loader!./templates/actions.yaml'; +import EndToEnd from '!!raw-loader!./templates/seqerakit-e2e.yaml'; + +Customize YAML configuration templates to use in `seqerakit` commands to create, update, or delete Seqera resources. Create or delete multiple resources with a single command by combining them into a single configuration file. + +To use the templates on this page: +1. Copy the template text or download the YAML files you need. +1. Edit the values to specify your resource details, and save as a `.yaml` file. +1. Specify the YAML template file in your `seqerakit` commands: + - To create the resources specified in the file: + ```shell-session + seqerakit file.yaml + ``` + - To delete the existing resources specified in the file: + ```shell-session + seqerakit file.yaml --delete + ``` + +:::info +See [Specify targets](./commands.mdx#specify-targets) to create or delete only selected resources from configuration templates that contain multiple resource entries. +::: + +See [End-to-end example](#end-to-end-example) for a template that contains examples of all Seqera resources that can be created with Seqerakit. + +### Administration + +Manage organizations, organization members, workspaces, teams, and participants. + +#### Organizations + +Add or delete organizations. + + +{Organizations} + + +[Download organizations.yaml](./templates/organizations.yaml) + +#### Members + +Add or delete organization members. + + +{Members} + + +[Download members.yaml](./templates/members.yaml) + +#### Workspaces + +Add or delete workspaces. + + +{Workspaces} + + +[Download workspaces.yaml](./templates/workspaces.yaml) + +#### Teams + +Add or delete teams. + + +{Teams} + + +[Download teams.yaml](./templates/teams.yaml) + +#### Participants + +Add or delete participants in workspaces and teams. + + +{Participants} + + +[Download participants.yaml](./templates/participants.yaml) + +### Credentials + +Add or delete compute environment, Git, and container registry credentials in workspaces. + + +{Credentials} + + +[Download credentials.yaml](./templates/credentials.yaml) + +### Compute environments + +Add or delete compute environments. + + +{ComputeEnvironments} + + +[Download compute-envs.yaml](./templates/compute-envs.yaml) + +### Pipelines + +Add or delete pipelines in workspace Launchpads. + + +{Pipelines} + + +[Download pipelines.yaml](./templates/pipelines.yaml) + +### Launch + +Launch a Nextflow pipeline. + + +{Launch} + + +[Download launch.yaml](./templates/launch.yaml) + +### Datasets + +Add or delete workspace datasets for pipeline input data. + + +{Datasets} + + +[Download datasets.yaml](./templates/datasets.yaml) + +### Labels + +Add or delete labels and resource labels to apply to workspace compute environments, pipelines, and runs. + + +{Labels} + + +[Download labels.yaml](./templates/labels.yaml) + +### Secrets + +Add or delete user and workspace secrets. + + +{Secrets} + + +[Download secrets.yaml](./templates/secrets.yaml) + +### Actions + +Add or delete pipeline actions. + + +{Actions} + + +[Download actions.yaml](./templates/actions.yaml) + +### End-to-end example + +A template to create the follwing resources: + +- An organization +- A workspace +- A team +- Participants +- Credentials +- Secrets +- Compute environments +- Datasets +- Pipelines + +The templates also contains `launch` entries to launch saved pipelines. + + +{EndToEnd} + + +[Download seqerakit-e2e.yaml](./templates/seqerakit-e2e.yaml) \ No newline at end of file diff --git a/platform_versioned_docs/version-23.4/seqerakit/templates/actions.yaml b/platform_versioned_docs/version-23.4/seqerakit/templates/actions.yaml new file mode 100644 index 000000000..5b65933ea --- /dev/null +++ b/platform_versioned_docs/version-23.4/seqerakit/templates/actions.yaml @@ -0,0 +1,24 @@ +## To see the full list of options available, run: "tw actions add -h" +actions: + - type: 'github' # required + name: 'your-github-action' # required + pipeline: 'https://github.com/my_username/my_repo' # required + workspace: 'your-organization/workspace-1' # required + compute-env: 'your-aws-compute-environment' # required + work-dir: 's3://your-bucket' # required + revision: 'main' # required + profile: 'test' # optional + params: # optional + outdir: 's3://your-bucket/results' + overwrite: True # optional + - type: 'tower' # required + name: 'your-tower-action' # required + pipeline: 'https://github.com/my_username/my_repo' # required + workspace: 'your-organization/workspace-1' # required + compute-env: 'your-aws-compute-environment' # required + work-dir: 's3://your-bucket' # required + revision: 'main' # required + profile: 'test' # optional + params: # optional + outdir: 's3://your-bucket/results' + overwrite: True # optional diff --git a/platform_versioned_docs/version-23.4/seqerakit/templates/compute-envs.yaml b/platform_versioned_docs/version-23.4/seqerakit/templates/compute-envs.yaml new file mode 100644 index 000000000..237f66421 --- /dev/null +++ b/platform_versioned_docs/version-23.4/seqerakit/templates/compute-envs.yaml @@ -0,0 +1,36 @@ +## To see the full list of options available, run: "tw compute-envs add -h" +## The options required to create compute environments can be specified: +## 1. Explicitly in this file +## 2. Via a JSON file exported from Seqera with the "tw compute-envs export" command +compute-envs: +# To create a compute environment from a JSON configuration file (AWS Example) + - name: 'your-aws-compute-environment' # required + workspace: 'your-organization/workspace-1' # required + credentials: 'your-aws-credentials' # required + wait: 'AVAILABLE' # optional + file-path: './compute-envs/aws_compute_environment.json' # required + overwrite: True # optional + +# To create a compute environment with options specified through YAML (AWS Example) + - type: aws-batch # required + config-mode: forge # required for AWS and Azure (forge or manual) + name: 'your-aws-compute-environment' # required + workspace: 'your-organization/workspace-1' # required + credentials: 'your-aws-credentials' # required + region: 'eu-west-1' # required + work-dir: 's3://your-bucket' # required + provisioning-model: 'SPOT' # optional + fusion-v2: False # optional + wave: False # optional + fargate: False # optional + fast-storage: False # optional + instance-types: 'c6i,r6i,m6i' # optional, comma-separated list + no-ebs-auto-scale: True # optional + max-cpus: 500 # required + labels: 'label1,label2' # optional, comma-separated list + vpc-id: 'vpc-1234567890' # optional + subnets: 'subnet-1234567890,subnet-1234567891' # optional, comma-separated list + security-groups: 'sg-1234567890,sg-1234567891' # optional, comma-separated list + allow-buckets: 's3://your-bucket,s3://your-other-bucket' # optional, comma-separated list + wait: 'AVAILABLE' # optional + overwrite: False # optional \ No newline at end of file diff --git a/platform_versioned_docs/version-23.4/seqerakit/templates/credentials.yaml b/platform_versioned_docs/version-23.4/seqerakit/templates/credentials.yaml new file mode 100644 index 000000000..e63db59eb --- /dev/null +++ b/platform_versioned_docs/version-23.4/seqerakit/templates/credentials.yaml @@ -0,0 +1,49 @@ +## To see the full list of options available, run: "tw credentials add -h" +## To avoid exposing sensitive information about your credentials, +## use environment variables to supply passwords and secret keys. +credentials: + - type: 'github' # required + name: 'your-github-credentials' # required + workspace: 'your-organization/workspace-1' # required + username: 'your-username' # required + password: '$SEQPLATFORM_GITHUB_PASSWORD' # required + overwrite: True # optional + + - type: 'container-reg' # required + name: 'your-dockerhub-credentials' # required + workspace: 'your-organization/workspace-1' # required + username: 'your-username' # required + password: '$DOCKERHUB_PASSWORD' # required + registry: 'docker.io' # required + overwrite: True # optional + + - type: 'google' # required + name: 'your-google-credentials' # required + workspace: 'your-organization/workspace-1' # required + key: '$GOOGLE_KEY' # required + overwrite: True # optional + + - type: 'aws' # required + name: 'your-aws-credentials' # required + workspace: 'your-organization/workspace-1' # required + access-key: '$AWS_ACCESS_KEY_ID' # required + secret-key: '$AWS_SECRET_ACCESS_KEY' # required + assume-role-arn: '$AWS_ASSUME_ROLE_ARN' # required + overwrite: True # optional + + - type: 'azure' # required + name: 'your-azure-credentials' # required + workspace: 'your-organization/workspace-1' # required + batch-key: '$AZURE_BATCH_KEY' # required + batch-name: 'your-batch-name' # required + storage-key: '$AZURE_STORAGE_KEY' # required + storage-name: 'your-storage-name' # required + overwrite: True # optional + + - type: 'codecommit' # required + name: 'your-codecommit-credentials' # required + workspace: 'your-organization/workspace-1' # required + access-key: '$CODECOMMIT_USER' # required + secret-key: '$CODECOMMIT_PASSWORD' # required + base-url: '$CODECOMMIT_BASEURL' # optional + overwrite: False # optional diff --git a/platform_versioned_docs/version-23.4/seqerakit/templates/datasets.yaml b/platform_versioned_docs/version-23.4/seqerakit/templates/datasets.yaml new file mode 100644 index 000000000..b93364ddc --- /dev/null +++ b/platform_versioned_docs/version-23.4/seqerakit/templates/datasets.yaml @@ -0,0 +1,8 @@ +## To see the full list of options available, run: "tw datasets add -h" +datasets: + - name: 'your-dataset' # required + description: 'Your dataset description' # optional + header: true # optional + workspace: 'your-organization/workspace-1' # required + file-path: './datasets/dataset.csv' # required + overwrite: True # optional \ No newline at end of file diff --git a/platform_versioned_docs/version-23.4/seqerakit/templates/labels.yaml b/platform_versioned_docs/version-23.4/seqerakit/templates/labels.yaml new file mode 100644 index 000000000..f8496a7f4 --- /dev/null +++ b/platform_versioned_docs/version-23.4/seqerakit/templates/labels.yaml @@ -0,0 +1,6 @@ +## To see the full list of options available, run: "tw labels add -h" +labels: + - name: 'label_name' # required + value: 'label_value' # required + workspace: 'your-organization/workspace-1' # required + overwrite: True # optional diff --git a/platform_versioned_docs/version-23.4/seqerakit/templates/launch.yaml b/platform_versioned_docs/version-23.4/seqerakit/templates/launch.yaml new file mode 100644 index 000000000..745c63c04 --- /dev/null +++ b/platform_versioned_docs/version-23.4/seqerakit/templates/launch.yaml @@ -0,0 +1,23 @@ +## To see the full list of options available, run: "tw launch -h" +## The options will vary if you are launching a pipeline: +## 1. Pre-configured in the Seqera Launchpad with most options already specified +## 2. From source via a remote Git repository URL where you must explicitly specify all launch options +## Note: overwrite is not supported for "tw launch" + +launch: + - name: 'launchpad-launch' # required + workspace: 'your-organization/workspace-1' # required + pipeline: 'nf-core-rnaseq' # required + params: # optional + outdir: 's3://your-bucket/results' + + - name: 'remote-launch' # required + workspace: 'your-organization/workspace-1' # required + compute-env: 'your-aws-compute-environment' # required + pipeline: 'https://github.com/my_username/my_repo' # required + work-dir: 's3://your-bucket' # optional + profile: 'test' # optional + revision: 'main' # optional + params-file: './pipelines/params.yml' # optional + config: './pipelines/nextflow.config' # optional + pre-run: './pipelines/pre-run.txt' # optional diff --git a/platform_versioned_docs/version-23.4/seqerakit/templates/members.yaml b/platform_versioned_docs/version-23.4/seqerakit/templates/members.yaml new file mode 100644 index 000000000..f06b9f4af --- /dev/null +++ b/platform_versioned_docs/version-23.4/seqerakit/templates/members.yaml @@ -0,0 +1,5 @@ +## To see the full list of options available, run: "tw members add -h" +members: + - user: 'user@domain.com' # required + organization: 'your-organization' # required + overwrite: True # optional \ No newline at end of file diff --git a/platform_versioned_docs/version-23.4/seqerakit/templates/organizations.yaml b/platform_versioned_docs/version-23.4/seqerakit/templates/organizations.yaml new file mode 100644 index 000000000..6cf0518d2 --- /dev/null +++ b/platform_versioned_docs/version-23.4/seqerakit/templates/organizations.yaml @@ -0,0 +1,8 @@ +## To see the full list of options available, run: "tw organizations add -h" +organizations: + - name: 'your-organization' # required + full-name: 'Your Organization LLC' # required + description: 'Your organization description' # optional + location: 'Global' # optional + website: 'https://domain.com/' # optional + overwrite: True # optional \ No newline at end of file diff --git a/platform_versioned_docs/version-23.4/seqerakit/templates/participants.yaml b/platform_versioned_docs/version-23.4/seqerakit/templates/participants.yaml new file mode 100644 index 000000000..3c5e77da6 --- /dev/null +++ b/platform_versioned_docs/version-23.4/seqerakit/templates/participants.yaml @@ -0,0 +1,10 @@ +## To see the full list of options available, run: "tw participants add -h" +participants: + - name: 'your-team' # required + type: 'TEAM' # required + workspace: 'your-organization/workspace-1' # required + role: 'ADMIN' # required + - name: 'team-member@domain.com' # required + type: 'MEMBER' # required + workspace: 'your-organization/workspace-1' # required + role: 'LAUNCH' # required \ No newline at end of file diff --git a/platform_versioned_docs/version-23.4/seqerakit/templates/pipelines.yaml b/platform_versioned_docs/version-23.4/seqerakit/templates/pipelines.yaml new file mode 100644 index 000000000..017d72527 --- /dev/null +++ b/platform_versioned_docs/version-23.4/seqerakit/templates/pipelines.yaml @@ -0,0 +1,24 @@ +## To see the full list of options available, run: "tw pipelines add -h" +## The options required to create pipelines can be specified: +## 1. Explicitly in this file +## 2. Via a JSON file exported from Seqera with the "tw pipelines export" command +pipelines: + - name: 'your-first-pipeline' # required + workspace: 'your-organization/workspace-1' # required + description: 'Options specified in this file' # optional + compute-env: 'your-aws-compute-environment' # required + work-dir: 's3://your-bucket' # optional + profile: 'test' # optional + revision: 'main' # required + params: # optional + outdir: 's3://your-bucket/results' + config: './pipelines/nextflow.config' # optional + pre-run: './pipelines/pre-run.txt' # optional + url: 'https://github.com/my_username/my_repo' # required + overwrite: True # optional + - name: 'your-second-pipeline' # required + workspace: 'your-organization/workspace-1' # required + description: 'Options specified via JSON file' # optional + compute-env: 'your-aws-compute-environment' # required + file-path: './pipelines/pipeline.json' # required + overwrite: True # optional diff --git a/platform_versioned_docs/version-23.4/seqerakit/templates/secrets.yaml b/platform_versioned_docs/version-23.4/seqerakit/templates/secrets.yaml new file mode 100644 index 000000000..743faba3b --- /dev/null +++ b/platform_versioned_docs/version-23.4/seqerakit/templates/secrets.yaml @@ -0,0 +1,6 @@ +## To see the full list of options available, run: "tw secrets add -h" +secrets: + - name: 'your-secret' # required + workspace: 'your-organization/workspace-1' # required + value: 'your-secret-value' # required + overwrite: True # optional \ No newline at end of file diff --git a/platform_versioned_docs/version-23.4/seqerakit/templates/seqerakit-e2e.yaml b/platform_versioned_docs/version-23.4/seqerakit/templates/seqerakit-e2e.yaml new file mode 100644 index 000000000..fcdea506e --- /dev/null +++ b/platform_versioned_docs/version-23.4/seqerakit/templates/seqerakit-e2e.yaml @@ -0,0 +1,148 @@ +organizations: + - name: 'your-organization' # required + full-name: 'Your organization LLC' # required + description: 'Organization created E2E with seqerakit CLI scripting' # optional + location: 'Global' # optional + website: 'https://yourdomain.com/' # optional + overwrite: True # optional + +teams: + - name: 'pipelines-team' # required + organization: 'your-organization' # required + description: 'Pipelines team @ Your organization' # optional + members: # optional + - 'user1@domain.com' + - 'user2@domain.com' + - 'user3@domain.com' + overwrite: True # optional + +workspaces: + - name: 'workspace-1' # required + full-name: 'Workspace one' # required + organization: 'your-organization' # required + description: 'Workspace created E2E with seqerakit CLI scripting' # optional + visibility: 'PRIVATE' # optional + overwrite: True # optional + +participants: + - name: 'pipelines-team' # required + type: 'TEAM' # required + workspace: 'your-organization/workspace-1' # required + role: 'ADMIN' # required + overwrite: True # optional + - name: 'user@domain.com' # required + type: 'MEMBER' # required + workspace: 'your-organization/workspace-1' # required + role: 'LAUNCH' # required + overwrite: True # optional + +credentials: + - type: 'github' # required + name: 'github-credentials' # required + workspace: 'your-organization/workspace-1' # required + username: 'user1' # required + password: '$TOWER_GITHUB_PASSWORD' # required + overwrite: True # optional + - type: 'container-reg' # required + name: 'dockerhub-credentials' # required + workspace: 'your-organization/workspace-1' # required + username: 'user1' # required + password: '$DOCKERHUB_PASSWORD' # required + registry: 'docker.io' # required + overwrite: True # optional + - type: 'aws' # required + name: 'aws-credentials' # required + workspace: 'your-organization/workspace-1' # required + access-key: '$AWS_ACCESS_KEY_ID' # required + secret-key: '$AWS_SECRET_ACCESS_KEY' # required + assume-role-arn: '$AWS_ASSUME_ROLE_ARN' # required + overwrite: True # optional + +secrets: + - name: 'SENTIEON_LICENSE_BASE64' # required + workspace: 'your-organization/workspace-1' # required + value: '$SENTIEON_LICENSE_BASE64' # required + overwrite: True # optional + +compute-envs: +# To create a compute environment with options specified through YAML (AWS Example) + - type: aws-batch # required + config-mode: forge # required for AWS and Azure + name: 'your-aws-compute-environment' # required + workspace: 'your-organization/workspace-1' # required + credentials: 'your-aws-credentials' # required + region: 'eu-west-1' # required + work-dir: 's3://your-bucket' # required + provisioning-model: 'SPOT' # optional + fusion-v2: False # optional + wave: False # optional + fargate: False # optional + fast-storage: False # optional + instance-types: 'c6i,r6i,m6i' # optional, comma-separated list + no-ebs-auto-scale: True # optional + max-cpus: 500 # required + labels: 'label1,label2' # optional, comma-separated list + vpc-id: 'vpc-1234567890' # optional + subnets: 'subnet-1234567890,subnet-1234567891' # optional, comma-separated list + security-groups: 'sg-1234567890,sg-1234567891' # optional, comma-separated list + allow-buckets: 's3://your-bucket,s3://your-other-bucket' # optional, comma-separated list + wait: 'AVAILABLE' # optional + overwrite: False # optional + +# To create a compute environment from a JSON configuration file (AWS Example) + - name: 'your-aws-compute-environment' # required + workspace: 'your-organization/workspace-1' # required + credentials: 'your-aws-credentials' # required + wait: 'AVAILABLE' # optional + file-path: './compute-envs/your_aws_compute_environment.json' # required + overwrite: True # optional + +datasets: + - name: 'rnaseq_samples' # required + description: 'Samplesheet to run the nf-core/rnaseq pipeline end-to-end' # optional + header: true # optional + workspace: 'your-organization/workspace-1' # required + file-path: 'path/to/rnaseq_samples.csv' # required + overwrite: True # optional + +pipelines: +# To create a pipeline with options specified through YAML + - name: 'nf-core-rnaseq' # required + url: 'https://github.com/nf-core/rnaseq' # required + workspace: 'your-organization/workspace-1' # required + description: 'RNA sequencing analysis pipeline with gene/isoform counts and extensive quality control.' # optional + compute-env: 'your-aws-compute-environment' # required + work-dir: 's3://your-bucket/pipeline/working/directory' # optional + profile: 'test' # optional + revision: '3.12.0' # required + params: # optional + outdir: 's3://your-bucket/nf-core-rnaseq/results' + config: 'path/to/nextflow.config' # optional + pre-run: 'path/to/pipeline/pre_run.txt' # optional + overwrite: True # optional + +# To create a pipeline from a JSON configuration file + - name: 'nf-core-sarek' # required + workspace: 'your-organization/workspace-1' # required + compute-env: 'your-aws-compute-environment' # required + file-path: 'path/to/nf-core-sarek_pipeline.json' # required + overwrite: True # optional + +launch: +# To launch a preconfigured pipeline from your workspace Launchpad + - name: 'nf-core-rnaseq-launchpad' # required + workspace: 'your-organization/workspace-1' # required + pipeline: 'nf-core-rnaseq' # required + compute-env: 'your-aws-compute-environment' # optional (defaults to workspace primary compute environment) + +# To launch an unsaved pipeline, include the pipeline repository URL and launch details + - name: 'nf-core-rnaseq-remote' # required + workspace: 'your-organization/workspace-1' # required + pipeline: 'https://github.com/nf-core/rnaseq' # required + compute-env: 'your-aws-compute-environment' # optional (defaults to workspace primary compute environment) + work-dir: 's3://your-bucket/pipeline/working/directory' # optional + profile: 'test' # optional + revision: '3.12.0' # optional + params-file: 'path/to/pipelines/nf_core_rnaseq_params.yml' # optional + config: 'path/to/pipelines/nextflow.config' # optional + pre-run: 'path/to/pipelines/pre_run.txt' # optional diff --git a/platform_versioned_docs/version-23.4/seqerakit/templates/teams.yaml b/platform_versioned_docs/version-23.4/seqerakit/templates/teams.yaml new file mode 100644 index 000000000..b1258de24 --- /dev/null +++ b/platform_versioned_docs/version-23.4/seqerakit/templates/teams.yaml @@ -0,0 +1,8 @@ +## To see the full list of options available, run: "tw teams add -h" +teams: + - name: 'your-team' # required + organization: 'your-organization' # required + description: 'Your team description.' # optional + members: # optional + - 'team-member@domain.com' + overwrite: True # optional \ No newline at end of file diff --git a/platform_versioned_docs/version-23.4/seqerakit/templates/workspaces.yaml b/platform_versioned_docs/version-23.4/seqerakit/templates/workspaces.yaml new file mode 100644 index 000000000..ca9392340 --- /dev/null +++ b/platform_versioned_docs/version-23.4/seqerakit/templates/workspaces.yaml @@ -0,0 +1,8 @@ +## To see the full list of options available, run: "tw workspaces add -h" +workspaces: + - name: 'workspace-1' # required + full-name: 'Workspace one' # required + organization: 'your-organization' # required + description: 'Your workspace description' # optional + visibility: 'PRIVATE' # optional + overwrite: True # optional \ No newline at end of file diff --git a/platform_versioned_docs/version-23.4/seqerakit/yaml-configuration.mdx b/platform_versioned_docs/version-23.4/seqerakit/yaml-configuration.mdx new file mode 100644 index 000000000..965161262 --- /dev/null +++ b/platform_versioned_docs/version-23.4/seqerakit/yaml-configuration.mdx @@ -0,0 +1,154 @@ +--- +title: "YAML configuration" +description: "seqerakit YAML configuration file options" +date: "21 Oct 2024" +tags: [seqerakit, cli, automation, yaml, configuration] +--- + +Seqerakit supports the creation and deletion of the following Seqera Platform resources, listed here with their respective Platform CLI resource names: + +- Pipeline actions: `actions` +- Compute environments: `compute-envs` +- Credentials: `credentials` +- Datasets: `datasets` +- Labels (including resource labels): `labels` +- Pipeline launch: `launch` +- Organization members: `members` +- Organizations: `organizations` +- Workspace and team participants: `participants` +- Pipelines: `pipelines` +- Pipeline secrets: `secrets` +- Teams: `teams` +- Workspaces: `workspaces` + +To determine the options to provide as definitions in your YAML file, run the Platform CLI help command for the resource you want to create. + +1. Retrieve CLI options: + + Obtain a list of available CLI options for defining your YAML file with the Platform CLI `help` command. For example, to add a pipeline to your workspace, view the options for adding a pipeline: + + ```shell-session + tw pipelines add -h + ``` + + ```shell-session + Usage: tw pipelines add [OPTIONS] PIPELINE_URL + + Add a workspace pipeline. + + Parameters: + * PIPELINE_URL Nextflow pipeline URL. + + Options: + * -n, --name= Pipeline name. + -w, --workspace= Workspace numeric identifier (TOWER_WORKSPACE_ID as default) or workspace reference as OrganizationName/WorkspaceName + -d, --description= Pipeline description. + --labels=[,...] List of labels seperated by coma. + -c, --compute-env= Compute environment name. + --work-dir= Path where the pipeline scratch data is stored. + -p, --profile=[,...] Comma-separated list of one or more configuration profile names you want to use for this pipeline execution. + --params-file= Pipeline parameters in either JSON or YML format. + --revision= A valid repository commit Id, tag or branch name. + ... + ``` + +1. Define key-value pairs in YAML: + + Translate each CLI option into a key-value pair in the YAML file. The structure of your YAML file should reflect the hierarchy and format of the CLI options. For example: + + ```yaml + pipelines: + - name: 'my_first_pipeline' + url: 'https://github.com/username/my_pipeline' + workspace: 'my_organization/my_workspace' + description: 'My test pipeline' + labels: 'yeast,test_data' + compute-env: 'my_compute_environment' + work-dir: 's3://my_bucket' + profile: 'test' + params-file: '/path/to/params.yaml' + revision: '1.0' + ``` + + In this example: + + - The keys (`name`, `url`, `workspace`, and so forth) are the keys derived from the CLI options. + - The corresponding values are user-defined. + +#### Best practices + +- The indentation and structure of the YAML file must be correct — YAML is sensitive to formatting. +- Use quotes around strings that contain special characters or spaces. +- To list multiple values (such as multiple `labels`, `instance-types`, or `allow-buckets`), separate values with commas. This is shown with `labels` in the preceding example. +- For complex configurations, see [Templates](./templates.mdx). + +### Templates + +See [Templates](./templates.mdx) for YAML file templates for each of the entities that can be created in Seqera. + +### YAML Configuration Options + +Some options handled specially by `seqerakit` or not exposed as `tw` CLI options can be provided in your YAML configuration file. + +#### Pipeline parameters using `params` and `params-file` + +To specify pipeline parameters, use `params:` to specify a list of parameters or `params-file:` to point to a parameters file. + +For example, to specify pipeline parameters within your YAML: + +```yaml +params: + outdir: 's3://path/to/outdir' + fasta: 's3://path/to/reference.fasta' +``` + +To specify a file containing pipeline parameters: + +```yaml +params-file: '/path/to/my/parameters.yaml' +``` + +Or provide both: + +```yaml +params-file: '/path/to/my/parameters.yaml' +params: + outdir: 's3://path/to/outdir' + fasta: 's3://path/to/reference.fasta' +``` + +:::note +If duplicate parameters are provided, the parameters provided as key-value pairs inside the `params` nested dictionary of the YAML file will take precedence **over** values in the `params-file`. +::: + +#### Overwrite + +For every entity defined in your YAML file, specify `overwrite: True` to overwrite any existing Seqera entities of the same name. + +Seqerakit will first check to see if the name of the entity exists. If so, it will invoke a `tw delete` command before attempting to create it based on the options defined in the YAML file. + +```shell-session +DEBUG:root: Overwrite is set to 'True' for organizations + +DEBUG:root: Running command: tw -o json organizations list +DEBUG:root: The attempted organizations resource already exists. Overwriting. + +DEBUG:root: Running command: tw organizations delete --name $SEQERA_ORGANIZATION_NAME +DEBUG:root: Running command: tw organizations add --name $SEQERA_ORGANIZATION_NAME --full-name $SEQERA_ORGANIZATION_NAME --description 'Example of an organization' +``` + +#### Specify JSON configuration files with `file-path` + +The Platform CLI allows the export and import of entities through JSON configuration files for pipelines and compute environments. To use these files to add a pipeline or compute environment to a workspace, use the `file-path` key to specify a path to a JSON configuration file. + +An example of the `file-path` option is provided in the [compute-envs.yaml](./templates/compute-envs.yaml) template: + +```yaml +compute-envs: + - name: 'my_aws_compute_environment' # required + workspace: 'my_organization/my_workspace' # required + credentials: 'my_aws_credentials' # required + wait: 'AVAILABLE' # optional + file-path: './compute-envs/my_aws_compute_environment.json' # required + overwrite: True +``` diff --git a/platform_versioned_docs/version-24.1/cli/installation.mdx b/platform_versioned_docs/version-24.1/cli/installation.mdx index 762b67abf..4acbbd0e8 100644 --- a/platform_versioned_docs/version-24.1/cli/installation.mdx +++ b/platform_versioned_docs/version-24.1/cli/installation.mdx @@ -65,9 +65,9 @@ tw info Details -------------------------+---------------------- Tower API endpoint | - Tower API version | 1.25.0 - Tower version | 24.2.0_cycle22 - CLI version | 0.9.4 (f3e846e) + Tower API version | 1.25.0 + Tower version | 24.2.0_cycle27 + CLI version | 0.10.3 (a86eeed) CLI minimum API version | 1.15 Authenticated user | diff --git a/platform_versioned_docs/version-24.1/seqerakit/commands.mdx b/platform_versioned_docs/version-24.1/seqerakit/commands.mdx new file mode 100644 index 000000000..533d64777 --- /dev/null +++ b/platform_versioned_docs/version-24.1/seqerakit/commands.mdx @@ -0,0 +1,126 @@ +--- +title: "Commands" +description: "Seqerakit command options" +date: "21 Oct 2024" +tags: [seqerakit, cli, automation, commands] +--- + +Use the `--help` or `-h` option to list available commands and options: + +```shell-session +seqerakit --help +``` + +### Input + +Seqerakit supports input through paths to YAML configuration files or directly from standard input (`stdin`). + +- Using file path: + + ```shell-session + seqerakit file.yaml + ``` + +- Using `stdin`: + + ```shell-session + cat file.yaml | seqerakit - + ``` + +See [Define your YAML file using CLI options](#define-your-yaml-file-using-cli-options) for guidance on formatting your input YAML files. + +### Dryrun + +Confirm that your configuration and command are correct before creating resources in your Seqera account, particularly when automating the end-to-end creation of multiple entities at once. To print the commands that would be executed with Platform CLI when using a YAML file, run your `seqerakit` command with the `--dryrun` option: + +```shell-session +seqerakit file.yaml --dryrun +``` + +### Specify targets + +When using a YAML file as an input that defines multiple resources, use the `--targets` option to specify which resources to create. This option accepts a comma-separated list of resource names. + +Supported resource names include: + +- `actions` +- `compute-envs` +- `credentials` +- `datasets` +- `labels` +- `launch` +- `members` +- `organizations` +- `participants` +- `pipelines` +- `secrets` +- `teams` +- `workspaces` + +For example, given a `test.yaml` file that defines the following resources: + +```yaml +workspaces: + - name: 'workspace-1' + organization: 'seqerakit' +... +compute-envs: + - name: 'compute-env' + type: 'aws-batch forge' + workspace: 'seqerakit/workspace-1' +... +pipelines: + - name: 'hello-world' + url: 'https://github.com/nextflow-io/hello' + workspace: 'seqerakit/workspace-1' + compute-env: 'compute-env' +... +``` + +You can target the creation of `pipelines` only by running: + +```shell-session +seqerakit test.yaml --targets pipelines +``` + +This command will create only the pipelines defined in the YAML file and ignore `workspaces` and `compute-envs`. + +To create both workspaces and pipelines, run: + +```shell-session +seqerakit test.yaml --targets workspaces,pipelines +``` + +### Delete resources + +Instead of adding or creating resources, specify the `--delete` option to recursively delete resources in your YAML file: + +```shell-session +seqerakit file.yaml --delete +``` + +For example, if you have a `file.yaml` that defines an organization, workspace, team, credentials, and compute environment that have already been created, run `seqerakit file.yaml --delete` to recursively delete the same resources. + +### Use `tw`-specific CLI options + +Specify `tw`-specific CLI options with the `--cli=` option: + +```shell-session +seqerakit file.yaml --cli="--arg1 --arg2" +``` + +See [CLI commands](../cli/commands.mdx) or run `tw -h` for the full list of options. + +:::note +The `--verbose` option for `tw` CLI is currently not supported in `seqerakit` commands. +::: + +#### Example: HTTP-only connections + +The Platform CLI expects to connect to a Seqera instance that is secured by a TLS certificate. If your Seqera Enterprise instance does not present a certificate, you must run your `tw` commands with the `--insecure` option. + +To use `tw`-specific CLI options such as `--insecure`, use the `--cli=` option, followed by the options to use enclosed in double quotes: + +```shell-session +seqerakit file.yaml --cli="--insecure" +``` diff --git a/platform_versioned_docs/version-24.1/seqerakit/installation.mdx b/platform_versioned_docs/version-24.1/seqerakit/installation.mdx new file mode 100644 index 000000000..58eac0d41 --- /dev/null +++ b/platform_versioned_docs/version-24.1/seqerakit/installation.mdx @@ -0,0 +1,117 @@ +--- +title: "Installation" +description: "Seqerakit installation options" +date: "21 Oct 2024" +tags: [seqerakit, cli, automation, installation] +--- + + +Seqerakit is a Python wrapper that sets [Platform CLI](../cli/overview.mdx) command options using YAML configuration files. Individual commands and configuration parameters can be chained together to automate the end-to-end creation of all Seqera Platform entities. + +As an extension of the Platform CLI, Seqerakit enables: + +- **Infrastructure as code**: Users manage and provision their infrastructure from the command line. +- **Simple configuration**: All Platform CLI command-line options can be defined in simple YAML format. +- **Automation**: End-to-end creation of Seqera entities, from adding an organization to launching pipelines. + +### Installation + +Seqerakit has three dependencies: + +1. [Seqera Platform CLI (`>=0.10.1`)](https://github.com/seqeralabs/tower-cli/releases) +2. [Python (`>=3.8`)](https://www.python.org/downloads/) +3. [PyYAML](https://pypi.org/project/PyYAML/) + +#### Pip + +If you already have [Platform CLI](../cli/installation.mdx) and Python installed on your system, install Seqerakit directly from [PyPI](https://pypi.org/project/seqerakit/): + +```console +pip install seqerakit +``` + +Overwrite an existing installation to use the latest version: + +```console +pip install --upgrade --force-reinstall seqerakit +``` + +#### Conda + +To install `seqerakit` and its dependencies via Conda, first configure the correct channels: + +```console +conda config --add channels bioconda +conda config --add channels conda-forge +conda config --set channel_priority strict +``` + +Then create a conda environment with `seqerakit` installed: + +```console +conda env create -n seqerakit seqerakit +conda activate seqerakit +``` + +#### Local development installation + +Install the development branch of `seqerakit` on your local machine to test the latest features and updates: + +1. You must have [Python](https://www.python.org/downloads/) and [Git](https://git-scm.com/downloads) installed on your system. +1. To install directly from pip: + ```shell-session + pip install git+https://github.com/seqeralabs/seqera-kit.git@dev + ``` +1. Alternatively, clone the repository locally and install manually: + ```shell-session + git clone https://github.com/seqeralabs/seqera-kit.git + cd seqera-kit + git checkout dev + pip install . + ``` +1. Verify your installation: + ```shell-session + pip show seqerakit + ``` + +### Configuration + +Create a [Seqera](https://cloud.seqera.io/tokens) access token via **Your Tokens** in the user menu. + +Seqerakit reads your access token from the `TOWER_ACCESS_TOKEN` environment variable: + +```shell-session +export TOWER_ACCESS_TOKEN= +``` + +For Enterprise installations, specify the custom API endpoint used to connect to Seqera. Export the API endpoint environment variable: + +```shell-session +export TOWER_API_ENDPOINT= +``` + +By default, this is set to `https://api.cloud.seqera.io` to connect to Seqera Cloud. + +### Usage + +To confirm the installation of `seqerakit`, configuration of the Platform CLI, and connection to Seqera is working as expected, run this command: + +```shell-session +seqerakit --info +``` + +This runs the `tw info` command under the hood. + +Use `--version` or `-v` to retrieve the current version of your `seqerakit` installation: + +```shell-session +seqerakit --version +``` + +Use the `--help` or `-h` option to list the available commands and their associated options: + +```shell-session +seqerakit --help +``` + +See [Commands](./commands.mdx) for detailed instructions to use Seqerakit. \ No newline at end of file diff --git a/platform_versioned_docs/version-24.1/seqerakit/templates.mdx b/platform_versioned_docs/version-24.1/seqerakit/templates.mdx new file mode 100644 index 000000000..f33062192 --- /dev/null +++ b/platform_versioned_docs/version-24.1/seqerakit/templates.mdx @@ -0,0 +1,200 @@ +--- +title: "Templates" +description: "Seqerakit YAML configuration file templates and instructions" +date: "8 Nov 2024" +tags: [seqerakit, cli, automation, yaml, configuration] +toc_max_heading_level: 4 +--- + +import CodeBlock from '@theme/CodeBlock'; +import Organizations from '!!raw-loader!./templates/organizations.yaml'; +import Members from '!!raw-loader!./templates/members.yaml'; +import Workspaces from '!!raw-loader!./templates/workspaces.yaml'; +import Teams from '!!raw-loader!./templates/teams.yaml'; +import Participants from '!!raw-loader!./templates/participants.yaml'; +import ComputeEnvironments from '!!raw-loader!./templates/compute-envs.yaml'; +import Pipelines from '!!raw-loader!./templates/pipelines.yaml'; +import Credentials from '!!raw-loader!./templates/credentials.yaml'; +import Datasets from '!!raw-loader!./templates/datasets.yaml'; +import Labels from '!!raw-loader!./templates/labels.yaml'; +import Launch from '!!raw-loader!./templates/launch.yaml'; +import Secrets from '!!raw-loader!./templates/secrets.yaml'; +import Actions from '!!raw-loader!./templates/actions.yaml'; +import EndToEnd from '!!raw-loader!./templates/seqerakit-e2e.yaml'; + +Customize YAML configuration templates to use in `seqerakit` commands to create, update, or delete Seqera resources. Create or delete multiple resources with a single command by combining them into a single configuration file. + +To use the templates on this page: +1. Copy the template text or download the YAML files you need. +1. Edit the values to specify your resource details, and save as a `.yaml` file. +1. Specify the YAML template file in your `seqerakit` commands: + - To create the resources specified in the file: + ```shell-session + seqerakit file.yaml + ``` + - To delete the existing resources specified in the file: + ```shell-session + seqerakit file.yaml --delete + ``` + +:::info +See [Specify targets](./commands.mdx#specify-targets) to create or delete only selected resources from configuration templates that contain multiple resource entries. +::: + +See [End-to-end example](#end-to-end-example) for a template that contains examples of all Seqera resources that can be created with Seqerakit. + +### Administration + +Manage organizations, organization members, workspaces, teams, and participants. + +#### Organizations + +Add or delete organizations. + + +{Organizations} + + +[Download organizations.yaml](./templates/organizations.yaml) + +#### Members + +Add or delete organization members. + + +{Members} + + +[Download members.yaml](./templates/members.yaml) + +#### Workspaces + +Add or delete workspaces. + + +{Workspaces} + + +[Download workspaces.yaml](./templates/workspaces.yaml) + +#### Teams + +Add or delete teams. + + +{Teams} + + +[Download teams.yaml](./templates/teams.yaml) + +#### Participants + +Add or delete participants in workspaces and teams. + + +{Participants} + + +[Download participants.yaml](./templates/participants.yaml) + +### Credentials + +Add or delete compute environment, Git, and container registry credentials in workspaces. + + +{Credentials} + + +[Download credentials.yaml](./templates/credentials.yaml) + +### Compute environments + +Add or delete compute environments. + + +{ComputeEnvironments} + + +[Download compute-envs.yaml](./templates/compute-envs.yaml) + +### Pipelines + +Add or delete pipelines in workspace Launchpads. + + +{Pipelines} + + +[Download pipelines.yaml](./templates/pipelines.yaml) + +### Launch + +Launch a Nextflow pipeline. + + +{Launch} + + +[Download launch.yaml](./templates/launch.yaml) + +### Datasets + +Add or delete workspace datasets for pipeline input data. + + +{Datasets} + + +[Download datasets.yaml](./templates/datasets.yaml) + +### Labels + +Add or delete labels and resource labels to apply to workspace compute environments, pipelines, and runs. + + +{Labels} + + +[Download labels.yaml](./templates/labels.yaml) + +### Secrets + +Add or delete user and workspace secrets. + + +{Secrets} + + +[Download secrets.yaml](./templates/secrets.yaml) + +### Actions + +Add or delete pipeline actions. + + +{Actions} + + +[Download actions.yaml](./templates/actions.yaml) + +### End-to-end example + +A template to create the follwing resources: + +- An organization +- A workspace +- A team +- Participants +- Credentials +- Secrets +- Compute environments +- Datasets +- Pipelines + +The templates also contains `launch` entries to launch saved pipelines. + + +{EndToEnd} + + +[Download seqerakit-e2e.yaml](./templates/seqerakit-e2e.yaml) \ No newline at end of file diff --git a/platform_versioned_docs/version-24.1/seqerakit/templates/actions.yaml b/platform_versioned_docs/version-24.1/seqerakit/templates/actions.yaml new file mode 100644 index 000000000..5b65933ea --- /dev/null +++ b/platform_versioned_docs/version-24.1/seqerakit/templates/actions.yaml @@ -0,0 +1,24 @@ +## To see the full list of options available, run: "tw actions add -h" +actions: + - type: 'github' # required + name: 'your-github-action' # required + pipeline: 'https://github.com/my_username/my_repo' # required + workspace: 'your-organization/workspace-1' # required + compute-env: 'your-aws-compute-environment' # required + work-dir: 's3://your-bucket' # required + revision: 'main' # required + profile: 'test' # optional + params: # optional + outdir: 's3://your-bucket/results' + overwrite: True # optional + - type: 'tower' # required + name: 'your-tower-action' # required + pipeline: 'https://github.com/my_username/my_repo' # required + workspace: 'your-organization/workspace-1' # required + compute-env: 'your-aws-compute-environment' # required + work-dir: 's3://your-bucket' # required + revision: 'main' # required + profile: 'test' # optional + params: # optional + outdir: 's3://your-bucket/results' + overwrite: True # optional diff --git a/platform_versioned_docs/version-24.1/seqerakit/templates/compute-envs.yaml b/platform_versioned_docs/version-24.1/seqerakit/templates/compute-envs.yaml new file mode 100644 index 000000000..237f66421 --- /dev/null +++ b/platform_versioned_docs/version-24.1/seqerakit/templates/compute-envs.yaml @@ -0,0 +1,36 @@ +## To see the full list of options available, run: "tw compute-envs add -h" +## The options required to create compute environments can be specified: +## 1. Explicitly in this file +## 2. Via a JSON file exported from Seqera with the "tw compute-envs export" command +compute-envs: +# To create a compute environment from a JSON configuration file (AWS Example) + - name: 'your-aws-compute-environment' # required + workspace: 'your-organization/workspace-1' # required + credentials: 'your-aws-credentials' # required + wait: 'AVAILABLE' # optional + file-path: './compute-envs/aws_compute_environment.json' # required + overwrite: True # optional + +# To create a compute environment with options specified through YAML (AWS Example) + - type: aws-batch # required + config-mode: forge # required for AWS and Azure (forge or manual) + name: 'your-aws-compute-environment' # required + workspace: 'your-organization/workspace-1' # required + credentials: 'your-aws-credentials' # required + region: 'eu-west-1' # required + work-dir: 's3://your-bucket' # required + provisioning-model: 'SPOT' # optional + fusion-v2: False # optional + wave: False # optional + fargate: False # optional + fast-storage: False # optional + instance-types: 'c6i,r6i,m6i' # optional, comma-separated list + no-ebs-auto-scale: True # optional + max-cpus: 500 # required + labels: 'label1,label2' # optional, comma-separated list + vpc-id: 'vpc-1234567890' # optional + subnets: 'subnet-1234567890,subnet-1234567891' # optional, comma-separated list + security-groups: 'sg-1234567890,sg-1234567891' # optional, comma-separated list + allow-buckets: 's3://your-bucket,s3://your-other-bucket' # optional, comma-separated list + wait: 'AVAILABLE' # optional + overwrite: False # optional \ No newline at end of file diff --git a/platform_versioned_docs/version-24.1/seqerakit/templates/credentials.yaml b/platform_versioned_docs/version-24.1/seqerakit/templates/credentials.yaml new file mode 100644 index 000000000..e63db59eb --- /dev/null +++ b/platform_versioned_docs/version-24.1/seqerakit/templates/credentials.yaml @@ -0,0 +1,49 @@ +## To see the full list of options available, run: "tw credentials add -h" +## To avoid exposing sensitive information about your credentials, +## use environment variables to supply passwords and secret keys. +credentials: + - type: 'github' # required + name: 'your-github-credentials' # required + workspace: 'your-organization/workspace-1' # required + username: 'your-username' # required + password: '$SEQPLATFORM_GITHUB_PASSWORD' # required + overwrite: True # optional + + - type: 'container-reg' # required + name: 'your-dockerhub-credentials' # required + workspace: 'your-organization/workspace-1' # required + username: 'your-username' # required + password: '$DOCKERHUB_PASSWORD' # required + registry: 'docker.io' # required + overwrite: True # optional + + - type: 'google' # required + name: 'your-google-credentials' # required + workspace: 'your-organization/workspace-1' # required + key: '$GOOGLE_KEY' # required + overwrite: True # optional + + - type: 'aws' # required + name: 'your-aws-credentials' # required + workspace: 'your-organization/workspace-1' # required + access-key: '$AWS_ACCESS_KEY_ID' # required + secret-key: '$AWS_SECRET_ACCESS_KEY' # required + assume-role-arn: '$AWS_ASSUME_ROLE_ARN' # required + overwrite: True # optional + + - type: 'azure' # required + name: 'your-azure-credentials' # required + workspace: 'your-organization/workspace-1' # required + batch-key: '$AZURE_BATCH_KEY' # required + batch-name: 'your-batch-name' # required + storage-key: '$AZURE_STORAGE_KEY' # required + storage-name: 'your-storage-name' # required + overwrite: True # optional + + - type: 'codecommit' # required + name: 'your-codecommit-credentials' # required + workspace: 'your-organization/workspace-1' # required + access-key: '$CODECOMMIT_USER' # required + secret-key: '$CODECOMMIT_PASSWORD' # required + base-url: '$CODECOMMIT_BASEURL' # optional + overwrite: False # optional diff --git a/platform_versioned_docs/version-24.1/seqerakit/templates/datasets.yaml b/platform_versioned_docs/version-24.1/seqerakit/templates/datasets.yaml new file mode 100644 index 000000000..b93364ddc --- /dev/null +++ b/platform_versioned_docs/version-24.1/seqerakit/templates/datasets.yaml @@ -0,0 +1,8 @@ +## To see the full list of options available, run: "tw datasets add -h" +datasets: + - name: 'your-dataset' # required + description: 'Your dataset description' # optional + header: true # optional + workspace: 'your-organization/workspace-1' # required + file-path: './datasets/dataset.csv' # required + overwrite: True # optional \ No newline at end of file diff --git a/platform_versioned_docs/version-24.1/seqerakit/templates/labels.yaml b/platform_versioned_docs/version-24.1/seqerakit/templates/labels.yaml new file mode 100644 index 000000000..f8496a7f4 --- /dev/null +++ b/platform_versioned_docs/version-24.1/seqerakit/templates/labels.yaml @@ -0,0 +1,6 @@ +## To see the full list of options available, run: "tw labels add -h" +labels: + - name: 'label_name' # required + value: 'label_value' # required + workspace: 'your-organization/workspace-1' # required + overwrite: True # optional diff --git a/platform_versioned_docs/version-24.1/seqerakit/templates/launch.yaml b/platform_versioned_docs/version-24.1/seqerakit/templates/launch.yaml new file mode 100644 index 000000000..745c63c04 --- /dev/null +++ b/platform_versioned_docs/version-24.1/seqerakit/templates/launch.yaml @@ -0,0 +1,23 @@ +## To see the full list of options available, run: "tw launch -h" +## The options will vary if you are launching a pipeline: +## 1. Pre-configured in the Seqera Launchpad with most options already specified +## 2. From source via a remote Git repository URL where you must explicitly specify all launch options +## Note: overwrite is not supported for "tw launch" + +launch: + - name: 'launchpad-launch' # required + workspace: 'your-organization/workspace-1' # required + pipeline: 'nf-core-rnaseq' # required + params: # optional + outdir: 's3://your-bucket/results' + + - name: 'remote-launch' # required + workspace: 'your-organization/workspace-1' # required + compute-env: 'your-aws-compute-environment' # required + pipeline: 'https://github.com/my_username/my_repo' # required + work-dir: 's3://your-bucket' # optional + profile: 'test' # optional + revision: 'main' # optional + params-file: './pipelines/params.yml' # optional + config: './pipelines/nextflow.config' # optional + pre-run: './pipelines/pre-run.txt' # optional diff --git a/platform_versioned_docs/version-24.1/seqerakit/templates/members.yaml b/platform_versioned_docs/version-24.1/seqerakit/templates/members.yaml new file mode 100644 index 000000000..f06b9f4af --- /dev/null +++ b/platform_versioned_docs/version-24.1/seqerakit/templates/members.yaml @@ -0,0 +1,5 @@ +## To see the full list of options available, run: "tw members add -h" +members: + - user: 'user@domain.com' # required + organization: 'your-organization' # required + overwrite: True # optional \ No newline at end of file diff --git a/platform_versioned_docs/version-24.1/seqerakit/templates/organizations.yaml b/platform_versioned_docs/version-24.1/seqerakit/templates/organizations.yaml new file mode 100644 index 000000000..6cf0518d2 --- /dev/null +++ b/platform_versioned_docs/version-24.1/seqerakit/templates/organizations.yaml @@ -0,0 +1,8 @@ +## To see the full list of options available, run: "tw organizations add -h" +organizations: + - name: 'your-organization' # required + full-name: 'Your Organization LLC' # required + description: 'Your organization description' # optional + location: 'Global' # optional + website: 'https://domain.com/' # optional + overwrite: True # optional \ No newline at end of file diff --git a/platform_versioned_docs/version-24.1/seqerakit/templates/participants.yaml b/platform_versioned_docs/version-24.1/seqerakit/templates/participants.yaml new file mode 100644 index 000000000..3c5e77da6 --- /dev/null +++ b/platform_versioned_docs/version-24.1/seqerakit/templates/participants.yaml @@ -0,0 +1,10 @@ +## To see the full list of options available, run: "tw participants add -h" +participants: + - name: 'your-team' # required + type: 'TEAM' # required + workspace: 'your-organization/workspace-1' # required + role: 'ADMIN' # required + - name: 'team-member@domain.com' # required + type: 'MEMBER' # required + workspace: 'your-organization/workspace-1' # required + role: 'LAUNCH' # required \ No newline at end of file diff --git a/platform_versioned_docs/version-24.1/seqerakit/templates/pipelines.yaml b/platform_versioned_docs/version-24.1/seqerakit/templates/pipelines.yaml new file mode 100644 index 000000000..017d72527 --- /dev/null +++ b/platform_versioned_docs/version-24.1/seqerakit/templates/pipelines.yaml @@ -0,0 +1,24 @@ +## To see the full list of options available, run: "tw pipelines add -h" +## The options required to create pipelines can be specified: +## 1. Explicitly in this file +## 2. Via a JSON file exported from Seqera with the "tw pipelines export" command +pipelines: + - name: 'your-first-pipeline' # required + workspace: 'your-organization/workspace-1' # required + description: 'Options specified in this file' # optional + compute-env: 'your-aws-compute-environment' # required + work-dir: 's3://your-bucket' # optional + profile: 'test' # optional + revision: 'main' # required + params: # optional + outdir: 's3://your-bucket/results' + config: './pipelines/nextflow.config' # optional + pre-run: './pipelines/pre-run.txt' # optional + url: 'https://github.com/my_username/my_repo' # required + overwrite: True # optional + - name: 'your-second-pipeline' # required + workspace: 'your-organization/workspace-1' # required + description: 'Options specified via JSON file' # optional + compute-env: 'your-aws-compute-environment' # required + file-path: './pipelines/pipeline.json' # required + overwrite: True # optional diff --git a/platform_versioned_docs/version-24.1/seqerakit/templates/secrets.yaml b/platform_versioned_docs/version-24.1/seqerakit/templates/secrets.yaml new file mode 100644 index 000000000..743faba3b --- /dev/null +++ b/platform_versioned_docs/version-24.1/seqerakit/templates/secrets.yaml @@ -0,0 +1,6 @@ +## To see the full list of options available, run: "tw secrets add -h" +secrets: + - name: 'your-secret' # required + workspace: 'your-organization/workspace-1' # required + value: 'your-secret-value' # required + overwrite: True # optional \ No newline at end of file diff --git a/platform_versioned_docs/version-24.1/seqerakit/templates/seqerakit-e2e.yaml b/platform_versioned_docs/version-24.1/seqerakit/templates/seqerakit-e2e.yaml new file mode 100644 index 000000000..fcdea506e --- /dev/null +++ b/platform_versioned_docs/version-24.1/seqerakit/templates/seqerakit-e2e.yaml @@ -0,0 +1,148 @@ +organizations: + - name: 'your-organization' # required + full-name: 'Your organization LLC' # required + description: 'Organization created E2E with seqerakit CLI scripting' # optional + location: 'Global' # optional + website: 'https://yourdomain.com/' # optional + overwrite: True # optional + +teams: + - name: 'pipelines-team' # required + organization: 'your-organization' # required + description: 'Pipelines team @ Your organization' # optional + members: # optional + - 'user1@domain.com' + - 'user2@domain.com' + - 'user3@domain.com' + overwrite: True # optional + +workspaces: + - name: 'workspace-1' # required + full-name: 'Workspace one' # required + organization: 'your-organization' # required + description: 'Workspace created E2E with seqerakit CLI scripting' # optional + visibility: 'PRIVATE' # optional + overwrite: True # optional + +participants: + - name: 'pipelines-team' # required + type: 'TEAM' # required + workspace: 'your-organization/workspace-1' # required + role: 'ADMIN' # required + overwrite: True # optional + - name: 'user@domain.com' # required + type: 'MEMBER' # required + workspace: 'your-organization/workspace-1' # required + role: 'LAUNCH' # required + overwrite: True # optional + +credentials: + - type: 'github' # required + name: 'github-credentials' # required + workspace: 'your-organization/workspace-1' # required + username: 'user1' # required + password: '$TOWER_GITHUB_PASSWORD' # required + overwrite: True # optional + - type: 'container-reg' # required + name: 'dockerhub-credentials' # required + workspace: 'your-organization/workspace-1' # required + username: 'user1' # required + password: '$DOCKERHUB_PASSWORD' # required + registry: 'docker.io' # required + overwrite: True # optional + - type: 'aws' # required + name: 'aws-credentials' # required + workspace: 'your-organization/workspace-1' # required + access-key: '$AWS_ACCESS_KEY_ID' # required + secret-key: '$AWS_SECRET_ACCESS_KEY' # required + assume-role-arn: '$AWS_ASSUME_ROLE_ARN' # required + overwrite: True # optional + +secrets: + - name: 'SENTIEON_LICENSE_BASE64' # required + workspace: 'your-organization/workspace-1' # required + value: '$SENTIEON_LICENSE_BASE64' # required + overwrite: True # optional + +compute-envs: +# To create a compute environment with options specified through YAML (AWS Example) + - type: aws-batch # required + config-mode: forge # required for AWS and Azure + name: 'your-aws-compute-environment' # required + workspace: 'your-organization/workspace-1' # required + credentials: 'your-aws-credentials' # required + region: 'eu-west-1' # required + work-dir: 's3://your-bucket' # required + provisioning-model: 'SPOT' # optional + fusion-v2: False # optional + wave: False # optional + fargate: False # optional + fast-storage: False # optional + instance-types: 'c6i,r6i,m6i' # optional, comma-separated list + no-ebs-auto-scale: True # optional + max-cpus: 500 # required + labels: 'label1,label2' # optional, comma-separated list + vpc-id: 'vpc-1234567890' # optional + subnets: 'subnet-1234567890,subnet-1234567891' # optional, comma-separated list + security-groups: 'sg-1234567890,sg-1234567891' # optional, comma-separated list + allow-buckets: 's3://your-bucket,s3://your-other-bucket' # optional, comma-separated list + wait: 'AVAILABLE' # optional + overwrite: False # optional + +# To create a compute environment from a JSON configuration file (AWS Example) + - name: 'your-aws-compute-environment' # required + workspace: 'your-organization/workspace-1' # required + credentials: 'your-aws-credentials' # required + wait: 'AVAILABLE' # optional + file-path: './compute-envs/your_aws_compute_environment.json' # required + overwrite: True # optional + +datasets: + - name: 'rnaseq_samples' # required + description: 'Samplesheet to run the nf-core/rnaseq pipeline end-to-end' # optional + header: true # optional + workspace: 'your-organization/workspace-1' # required + file-path: 'path/to/rnaseq_samples.csv' # required + overwrite: True # optional + +pipelines: +# To create a pipeline with options specified through YAML + - name: 'nf-core-rnaseq' # required + url: 'https://github.com/nf-core/rnaseq' # required + workspace: 'your-organization/workspace-1' # required + description: 'RNA sequencing analysis pipeline with gene/isoform counts and extensive quality control.' # optional + compute-env: 'your-aws-compute-environment' # required + work-dir: 's3://your-bucket/pipeline/working/directory' # optional + profile: 'test' # optional + revision: '3.12.0' # required + params: # optional + outdir: 's3://your-bucket/nf-core-rnaseq/results' + config: 'path/to/nextflow.config' # optional + pre-run: 'path/to/pipeline/pre_run.txt' # optional + overwrite: True # optional + +# To create a pipeline from a JSON configuration file + - name: 'nf-core-sarek' # required + workspace: 'your-organization/workspace-1' # required + compute-env: 'your-aws-compute-environment' # required + file-path: 'path/to/nf-core-sarek_pipeline.json' # required + overwrite: True # optional + +launch: +# To launch a preconfigured pipeline from your workspace Launchpad + - name: 'nf-core-rnaseq-launchpad' # required + workspace: 'your-organization/workspace-1' # required + pipeline: 'nf-core-rnaseq' # required + compute-env: 'your-aws-compute-environment' # optional (defaults to workspace primary compute environment) + +# To launch an unsaved pipeline, include the pipeline repository URL and launch details + - name: 'nf-core-rnaseq-remote' # required + workspace: 'your-organization/workspace-1' # required + pipeline: 'https://github.com/nf-core/rnaseq' # required + compute-env: 'your-aws-compute-environment' # optional (defaults to workspace primary compute environment) + work-dir: 's3://your-bucket/pipeline/working/directory' # optional + profile: 'test' # optional + revision: '3.12.0' # optional + params-file: 'path/to/pipelines/nf_core_rnaseq_params.yml' # optional + config: 'path/to/pipelines/nextflow.config' # optional + pre-run: 'path/to/pipelines/pre_run.txt' # optional diff --git a/platform_versioned_docs/version-24.1/seqerakit/templates/teams.yaml b/platform_versioned_docs/version-24.1/seqerakit/templates/teams.yaml new file mode 100644 index 000000000..b1258de24 --- /dev/null +++ b/platform_versioned_docs/version-24.1/seqerakit/templates/teams.yaml @@ -0,0 +1,8 @@ +## To see the full list of options available, run: "tw teams add -h" +teams: + - name: 'your-team' # required + organization: 'your-organization' # required + description: 'Your team description.' # optional + members: # optional + - 'team-member@domain.com' + overwrite: True # optional \ No newline at end of file diff --git a/platform_versioned_docs/version-24.1/seqerakit/templates/workspaces.yaml b/platform_versioned_docs/version-24.1/seqerakit/templates/workspaces.yaml new file mode 100644 index 000000000..ca9392340 --- /dev/null +++ b/platform_versioned_docs/version-24.1/seqerakit/templates/workspaces.yaml @@ -0,0 +1,8 @@ +## To see the full list of options available, run: "tw workspaces add -h" +workspaces: + - name: 'workspace-1' # required + full-name: 'Workspace one' # required + organization: 'your-organization' # required + description: 'Your workspace description' # optional + visibility: 'PRIVATE' # optional + overwrite: True # optional \ No newline at end of file diff --git a/platform_versioned_docs/version-24.1/seqerakit/yaml-configuration.mdx b/platform_versioned_docs/version-24.1/seqerakit/yaml-configuration.mdx new file mode 100644 index 000000000..965161262 --- /dev/null +++ b/platform_versioned_docs/version-24.1/seqerakit/yaml-configuration.mdx @@ -0,0 +1,154 @@ +--- +title: "YAML configuration" +description: "seqerakit YAML configuration file options" +date: "21 Oct 2024" +tags: [seqerakit, cli, automation, yaml, configuration] +--- + +Seqerakit supports the creation and deletion of the following Seqera Platform resources, listed here with their respective Platform CLI resource names: + +- Pipeline actions: `actions` +- Compute environments: `compute-envs` +- Credentials: `credentials` +- Datasets: `datasets` +- Labels (including resource labels): `labels` +- Pipeline launch: `launch` +- Organization members: `members` +- Organizations: `organizations` +- Workspace and team participants: `participants` +- Pipelines: `pipelines` +- Pipeline secrets: `secrets` +- Teams: `teams` +- Workspaces: `workspaces` + +To determine the options to provide as definitions in your YAML file, run the Platform CLI help command for the resource you want to create. + +1. Retrieve CLI options: + + Obtain a list of available CLI options for defining your YAML file with the Platform CLI `help` command. For example, to add a pipeline to your workspace, view the options for adding a pipeline: + + ```shell-session + tw pipelines add -h + ``` + + ```shell-session + Usage: tw pipelines add [OPTIONS] PIPELINE_URL + + Add a workspace pipeline. + + Parameters: + * PIPELINE_URL Nextflow pipeline URL. + + Options: + * -n, --name= Pipeline name. + -w, --workspace= Workspace numeric identifier (TOWER_WORKSPACE_ID as default) or workspace reference as OrganizationName/WorkspaceName + -d, --description= Pipeline description. + --labels=[,...] List of labels seperated by coma. + -c, --compute-env= Compute environment name. + --work-dir= Path where the pipeline scratch data is stored. + -p, --profile=[,...] Comma-separated list of one or more configuration profile names you want to use for this pipeline execution. + --params-file= Pipeline parameters in either JSON or YML format. + --revision= A valid repository commit Id, tag or branch name. + ... + ``` + +1. Define key-value pairs in YAML: + + Translate each CLI option into a key-value pair in the YAML file. The structure of your YAML file should reflect the hierarchy and format of the CLI options. For example: + + ```yaml + pipelines: + - name: 'my_first_pipeline' + url: 'https://github.com/username/my_pipeline' + workspace: 'my_organization/my_workspace' + description: 'My test pipeline' + labels: 'yeast,test_data' + compute-env: 'my_compute_environment' + work-dir: 's3://my_bucket' + profile: 'test' + params-file: '/path/to/params.yaml' + revision: '1.0' + ``` + + In this example: + + - The keys (`name`, `url`, `workspace`, and so forth) are the keys derived from the CLI options. + - The corresponding values are user-defined. + +#### Best practices + +- The indentation and structure of the YAML file must be correct — YAML is sensitive to formatting. +- Use quotes around strings that contain special characters or spaces. +- To list multiple values (such as multiple `labels`, `instance-types`, or `allow-buckets`), separate values with commas. This is shown with `labels` in the preceding example. +- For complex configurations, see [Templates](./templates.mdx). + +### Templates + +See [Templates](./templates.mdx) for YAML file templates for each of the entities that can be created in Seqera. + +### YAML Configuration Options + +Some options handled specially by `seqerakit` or not exposed as `tw` CLI options can be provided in your YAML configuration file. + +#### Pipeline parameters using `params` and `params-file` + +To specify pipeline parameters, use `params:` to specify a list of parameters or `params-file:` to point to a parameters file. + +For example, to specify pipeline parameters within your YAML: + +```yaml +params: + outdir: 's3://path/to/outdir' + fasta: 's3://path/to/reference.fasta' +``` + +To specify a file containing pipeline parameters: + +```yaml +params-file: '/path/to/my/parameters.yaml' +``` + +Or provide both: + +```yaml +params-file: '/path/to/my/parameters.yaml' +params: + outdir: 's3://path/to/outdir' + fasta: 's3://path/to/reference.fasta' +``` + +:::note +If duplicate parameters are provided, the parameters provided as key-value pairs inside the `params` nested dictionary of the YAML file will take precedence **over** values in the `params-file`. +::: + +#### Overwrite + +For every entity defined in your YAML file, specify `overwrite: True` to overwrite any existing Seqera entities of the same name. + +Seqerakit will first check to see if the name of the entity exists. If so, it will invoke a `tw delete` command before attempting to create it based on the options defined in the YAML file. + +```shell-session +DEBUG:root: Overwrite is set to 'True' for organizations + +DEBUG:root: Running command: tw -o json organizations list +DEBUG:root: The attempted organizations resource already exists. Overwriting. + +DEBUG:root: Running command: tw organizations delete --name $SEQERA_ORGANIZATION_NAME +DEBUG:root: Running command: tw organizations add --name $SEQERA_ORGANIZATION_NAME --full-name $SEQERA_ORGANIZATION_NAME --description 'Example of an organization' +``` + +#### Specify JSON configuration files with `file-path` + +The Platform CLI allows the export and import of entities through JSON configuration files for pipelines and compute environments. To use these files to add a pipeline or compute environment to a workspace, use the `file-path` key to specify a path to a JSON configuration file. + +An example of the `file-path` option is provided in the [compute-envs.yaml](./templates/compute-envs.yaml) template: + +```yaml +compute-envs: + - name: 'my_aws_compute_environment' # required + workspace: 'my_organization/my_workspace' # required + credentials: 'my_aws_credentials' # required + wait: 'AVAILABLE' # optional + file-path: './compute-envs/my_aws_compute_environment.json' # required + overwrite: True +``` diff --git a/platform_versioned_docs/version-24.2/seqerakit/commands.mdx b/platform_versioned_docs/version-24.2/seqerakit/commands.mdx new file mode 100644 index 000000000..533d64777 --- /dev/null +++ b/platform_versioned_docs/version-24.2/seqerakit/commands.mdx @@ -0,0 +1,126 @@ +--- +title: "Commands" +description: "Seqerakit command options" +date: "21 Oct 2024" +tags: [seqerakit, cli, automation, commands] +--- + +Use the `--help` or `-h` option to list available commands and options: + +```shell-session +seqerakit --help +``` + +### Input + +Seqerakit supports input through paths to YAML configuration files or directly from standard input (`stdin`). + +- Using file path: + + ```shell-session + seqerakit file.yaml + ``` + +- Using `stdin`: + + ```shell-session + cat file.yaml | seqerakit - + ``` + +See [Define your YAML file using CLI options](#define-your-yaml-file-using-cli-options) for guidance on formatting your input YAML files. + +### Dryrun + +Confirm that your configuration and command are correct before creating resources in your Seqera account, particularly when automating the end-to-end creation of multiple entities at once. To print the commands that would be executed with Platform CLI when using a YAML file, run your `seqerakit` command with the `--dryrun` option: + +```shell-session +seqerakit file.yaml --dryrun +``` + +### Specify targets + +When using a YAML file as an input that defines multiple resources, use the `--targets` option to specify which resources to create. This option accepts a comma-separated list of resource names. + +Supported resource names include: + +- `actions` +- `compute-envs` +- `credentials` +- `datasets` +- `labels` +- `launch` +- `members` +- `organizations` +- `participants` +- `pipelines` +- `secrets` +- `teams` +- `workspaces` + +For example, given a `test.yaml` file that defines the following resources: + +```yaml +workspaces: + - name: 'workspace-1' + organization: 'seqerakit' +... +compute-envs: + - name: 'compute-env' + type: 'aws-batch forge' + workspace: 'seqerakit/workspace-1' +... +pipelines: + - name: 'hello-world' + url: 'https://github.com/nextflow-io/hello' + workspace: 'seqerakit/workspace-1' + compute-env: 'compute-env' +... +``` + +You can target the creation of `pipelines` only by running: + +```shell-session +seqerakit test.yaml --targets pipelines +``` + +This command will create only the pipelines defined in the YAML file and ignore `workspaces` and `compute-envs`. + +To create both workspaces and pipelines, run: + +```shell-session +seqerakit test.yaml --targets workspaces,pipelines +``` + +### Delete resources + +Instead of adding or creating resources, specify the `--delete` option to recursively delete resources in your YAML file: + +```shell-session +seqerakit file.yaml --delete +``` + +For example, if you have a `file.yaml` that defines an organization, workspace, team, credentials, and compute environment that have already been created, run `seqerakit file.yaml --delete` to recursively delete the same resources. + +### Use `tw`-specific CLI options + +Specify `tw`-specific CLI options with the `--cli=` option: + +```shell-session +seqerakit file.yaml --cli="--arg1 --arg2" +``` + +See [CLI commands](../cli/commands.mdx) or run `tw -h` for the full list of options. + +:::note +The `--verbose` option for `tw` CLI is currently not supported in `seqerakit` commands. +::: + +#### Example: HTTP-only connections + +The Platform CLI expects to connect to a Seqera instance that is secured by a TLS certificate. If your Seqera Enterprise instance does not present a certificate, you must run your `tw` commands with the `--insecure` option. + +To use `tw`-specific CLI options such as `--insecure`, use the `--cli=` option, followed by the options to use enclosed in double quotes: + +```shell-session +seqerakit file.yaml --cli="--insecure" +``` diff --git a/platform_versioned_docs/version-24.2/seqerakit/installation.mdx b/platform_versioned_docs/version-24.2/seqerakit/installation.mdx new file mode 100644 index 000000000..58eac0d41 --- /dev/null +++ b/platform_versioned_docs/version-24.2/seqerakit/installation.mdx @@ -0,0 +1,117 @@ +--- +title: "Installation" +description: "Seqerakit installation options" +date: "21 Oct 2024" +tags: [seqerakit, cli, automation, installation] +--- + + +Seqerakit is a Python wrapper that sets [Platform CLI](../cli/overview.mdx) command options using YAML configuration files. Individual commands and configuration parameters can be chained together to automate the end-to-end creation of all Seqera Platform entities. + +As an extension of the Platform CLI, Seqerakit enables: + +- **Infrastructure as code**: Users manage and provision their infrastructure from the command line. +- **Simple configuration**: All Platform CLI command-line options can be defined in simple YAML format. +- **Automation**: End-to-end creation of Seqera entities, from adding an organization to launching pipelines. + +### Installation + +Seqerakit has three dependencies: + +1. [Seqera Platform CLI (`>=0.10.1`)](https://github.com/seqeralabs/tower-cli/releases) +2. [Python (`>=3.8`)](https://www.python.org/downloads/) +3. [PyYAML](https://pypi.org/project/PyYAML/) + +#### Pip + +If you already have [Platform CLI](../cli/installation.mdx) and Python installed on your system, install Seqerakit directly from [PyPI](https://pypi.org/project/seqerakit/): + +```console +pip install seqerakit +``` + +Overwrite an existing installation to use the latest version: + +```console +pip install --upgrade --force-reinstall seqerakit +``` + +#### Conda + +To install `seqerakit` and its dependencies via Conda, first configure the correct channels: + +```console +conda config --add channels bioconda +conda config --add channels conda-forge +conda config --set channel_priority strict +``` + +Then create a conda environment with `seqerakit` installed: + +```console +conda env create -n seqerakit seqerakit +conda activate seqerakit +``` + +#### Local development installation + +Install the development branch of `seqerakit` on your local machine to test the latest features and updates: + +1. You must have [Python](https://www.python.org/downloads/) and [Git](https://git-scm.com/downloads) installed on your system. +1. To install directly from pip: + ```shell-session + pip install git+https://github.com/seqeralabs/seqera-kit.git@dev + ``` +1. Alternatively, clone the repository locally and install manually: + ```shell-session + git clone https://github.com/seqeralabs/seqera-kit.git + cd seqera-kit + git checkout dev + pip install . + ``` +1. Verify your installation: + ```shell-session + pip show seqerakit + ``` + +### Configuration + +Create a [Seqera](https://cloud.seqera.io/tokens) access token via **Your Tokens** in the user menu. + +Seqerakit reads your access token from the `TOWER_ACCESS_TOKEN` environment variable: + +```shell-session +export TOWER_ACCESS_TOKEN= +``` + +For Enterprise installations, specify the custom API endpoint used to connect to Seqera. Export the API endpoint environment variable: + +```shell-session +export TOWER_API_ENDPOINT= +``` + +By default, this is set to `https://api.cloud.seqera.io` to connect to Seqera Cloud. + +### Usage + +To confirm the installation of `seqerakit`, configuration of the Platform CLI, and connection to Seqera is working as expected, run this command: + +```shell-session +seqerakit --info +``` + +This runs the `tw info` command under the hood. + +Use `--version` or `-v` to retrieve the current version of your `seqerakit` installation: + +```shell-session +seqerakit --version +``` + +Use the `--help` or `-h` option to list the available commands and their associated options: + +```shell-session +seqerakit --help +``` + +See [Commands](./commands.mdx) for detailed instructions to use Seqerakit. \ No newline at end of file diff --git a/platform_versioned_docs/version-24.2/seqerakit/templates.mdx b/platform_versioned_docs/version-24.2/seqerakit/templates.mdx new file mode 100644 index 000000000..f33062192 --- /dev/null +++ b/platform_versioned_docs/version-24.2/seqerakit/templates.mdx @@ -0,0 +1,200 @@ +--- +title: "Templates" +description: "Seqerakit YAML configuration file templates and instructions" +date: "8 Nov 2024" +tags: [seqerakit, cli, automation, yaml, configuration] +toc_max_heading_level: 4 +--- + +import CodeBlock from '@theme/CodeBlock'; +import Organizations from '!!raw-loader!./templates/organizations.yaml'; +import Members from '!!raw-loader!./templates/members.yaml'; +import Workspaces from '!!raw-loader!./templates/workspaces.yaml'; +import Teams from '!!raw-loader!./templates/teams.yaml'; +import Participants from '!!raw-loader!./templates/participants.yaml'; +import ComputeEnvironments from '!!raw-loader!./templates/compute-envs.yaml'; +import Pipelines from '!!raw-loader!./templates/pipelines.yaml'; +import Credentials from '!!raw-loader!./templates/credentials.yaml'; +import Datasets from '!!raw-loader!./templates/datasets.yaml'; +import Labels from '!!raw-loader!./templates/labels.yaml'; +import Launch from '!!raw-loader!./templates/launch.yaml'; +import Secrets from '!!raw-loader!./templates/secrets.yaml'; +import Actions from '!!raw-loader!./templates/actions.yaml'; +import EndToEnd from '!!raw-loader!./templates/seqerakit-e2e.yaml'; + +Customize YAML configuration templates to use in `seqerakit` commands to create, update, or delete Seqera resources. Create or delete multiple resources with a single command by combining them into a single configuration file. + +To use the templates on this page: +1. Copy the template text or download the YAML files you need. +1. Edit the values to specify your resource details, and save as a `.yaml` file. +1. Specify the YAML template file in your `seqerakit` commands: + - To create the resources specified in the file: + ```shell-session + seqerakit file.yaml + ``` + - To delete the existing resources specified in the file: + ```shell-session + seqerakit file.yaml --delete + ``` + +:::info +See [Specify targets](./commands.mdx#specify-targets) to create or delete only selected resources from configuration templates that contain multiple resource entries. +::: + +See [End-to-end example](#end-to-end-example) for a template that contains examples of all Seqera resources that can be created with Seqerakit. + +### Administration + +Manage organizations, organization members, workspaces, teams, and participants. + +#### Organizations + +Add or delete organizations. + + +{Organizations} + + +[Download organizations.yaml](./templates/organizations.yaml) + +#### Members + +Add or delete organization members. + + +{Members} + + +[Download members.yaml](./templates/members.yaml) + +#### Workspaces + +Add or delete workspaces. + + +{Workspaces} + + +[Download workspaces.yaml](./templates/workspaces.yaml) + +#### Teams + +Add or delete teams. + + +{Teams} + + +[Download teams.yaml](./templates/teams.yaml) + +#### Participants + +Add or delete participants in workspaces and teams. + + +{Participants} + + +[Download participants.yaml](./templates/participants.yaml) + +### Credentials + +Add or delete compute environment, Git, and container registry credentials in workspaces. + + +{Credentials} + + +[Download credentials.yaml](./templates/credentials.yaml) + +### Compute environments + +Add or delete compute environments. + + +{ComputeEnvironments} + + +[Download compute-envs.yaml](./templates/compute-envs.yaml) + +### Pipelines + +Add or delete pipelines in workspace Launchpads. + + +{Pipelines} + + +[Download pipelines.yaml](./templates/pipelines.yaml) + +### Launch + +Launch a Nextflow pipeline. + + +{Launch} + + +[Download launch.yaml](./templates/launch.yaml) + +### Datasets + +Add or delete workspace datasets for pipeline input data. + + +{Datasets} + + +[Download datasets.yaml](./templates/datasets.yaml) + +### Labels + +Add or delete labels and resource labels to apply to workspace compute environments, pipelines, and runs. + + +{Labels} + + +[Download labels.yaml](./templates/labels.yaml) + +### Secrets + +Add or delete user and workspace secrets. + + +{Secrets} + + +[Download secrets.yaml](./templates/secrets.yaml) + +### Actions + +Add or delete pipeline actions. + + +{Actions} + + +[Download actions.yaml](./templates/actions.yaml) + +### End-to-end example + +A template to create the follwing resources: + +- An organization +- A workspace +- A team +- Participants +- Credentials +- Secrets +- Compute environments +- Datasets +- Pipelines + +The templates also contains `launch` entries to launch saved pipelines. + + +{EndToEnd} + + +[Download seqerakit-e2e.yaml](./templates/seqerakit-e2e.yaml) \ No newline at end of file diff --git a/platform_versioned_docs/version-24.2/seqerakit/templates/actions.yaml b/platform_versioned_docs/version-24.2/seqerakit/templates/actions.yaml new file mode 100644 index 000000000..5b65933ea --- /dev/null +++ b/platform_versioned_docs/version-24.2/seqerakit/templates/actions.yaml @@ -0,0 +1,24 @@ +## To see the full list of options available, run: "tw actions add -h" +actions: + - type: 'github' # required + name: 'your-github-action' # required + pipeline: 'https://github.com/my_username/my_repo' # required + workspace: 'your-organization/workspace-1' # required + compute-env: 'your-aws-compute-environment' # required + work-dir: 's3://your-bucket' # required + revision: 'main' # required + profile: 'test' # optional + params: # optional + outdir: 's3://your-bucket/results' + overwrite: True # optional + - type: 'tower' # required + name: 'your-tower-action' # required + pipeline: 'https://github.com/my_username/my_repo' # required + workspace: 'your-organization/workspace-1' # required + compute-env: 'your-aws-compute-environment' # required + work-dir: 's3://your-bucket' # required + revision: 'main' # required + profile: 'test' # optional + params: # optional + outdir: 's3://your-bucket/results' + overwrite: True # optional diff --git a/platform_versioned_docs/version-24.2/seqerakit/templates/compute-envs.yaml b/platform_versioned_docs/version-24.2/seqerakit/templates/compute-envs.yaml new file mode 100644 index 000000000..237f66421 --- /dev/null +++ b/platform_versioned_docs/version-24.2/seqerakit/templates/compute-envs.yaml @@ -0,0 +1,36 @@ +## To see the full list of options available, run: "tw compute-envs add -h" +## The options required to create compute environments can be specified: +## 1. Explicitly in this file +## 2. Via a JSON file exported from Seqera with the "tw compute-envs export" command +compute-envs: +# To create a compute environment from a JSON configuration file (AWS Example) + - name: 'your-aws-compute-environment' # required + workspace: 'your-organization/workspace-1' # required + credentials: 'your-aws-credentials' # required + wait: 'AVAILABLE' # optional + file-path: './compute-envs/aws_compute_environment.json' # required + overwrite: True # optional + +# To create a compute environment with options specified through YAML (AWS Example) + - type: aws-batch # required + config-mode: forge # required for AWS and Azure (forge or manual) + name: 'your-aws-compute-environment' # required + workspace: 'your-organization/workspace-1' # required + credentials: 'your-aws-credentials' # required + region: 'eu-west-1' # required + work-dir: 's3://your-bucket' # required + provisioning-model: 'SPOT' # optional + fusion-v2: False # optional + wave: False # optional + fargate: False # optional + fast-storage: False # optional + instance-types: 'c6i,r6i,m6i' # optional, comma-separated list + no-ebs-auto-scale: True # optional + max-cpus: 500 # required + labels: 'label1,label2' # optional, comma-separated list + vpc-id: 'vpc-1234567890' # optional + subnets: 'subnet-1234567890,subnet-1234567891' # optional, comma-separated list + security-groups: 'sg-1234567890,sg-1234567891' # optional, comma-separated list + allow-buckets: 's3://your-bucket,s3://your-other-bucket' # optional, comma-separated list + wait: 'AVAILABLE' # optional + overwrite: False # optional \ No newline at end of file diff --git a/platform_versioned_docs/version-24.2/seqerakit/templates/credentials.yaml b/platform_versioned_docs/version-24.2/seqerakit/templates/credentials.yaml new file mode 100644 index 000000000..e63db59eb --- /dev/null +++ b/platform_versioned_docs/version-24.2/seqerakit/templates/credentials.yaml @@ -0,0 +1,49 @@ +## To see the full list of options available, run: "tw credentials add -h" +## To avoid exposing sensitive information about your credentials, +## use environment variables to supply passwords and secret keys. +credentials: + - type: 'github' # required + name: 'your-github-credentials' # required + workspace: 'your-organization/workspace-1' # required + username: 'your-username' # required + password: '$SEQPLATFORM_GITHUB_PASSWORD' # required + overwrite: True # optional + + - type: 'container-reg' # required + name: 'your-dockerhub-credentials' # required + workspace: 'your-organization/workspace-1' # required + username: 'your-username' # required + password: '$DOCKERHUB_PASSWORD' # required + registry: 'docker.io' # required + overwrite: True # optional + + - type: 'google' # required + name: 'your-google-credentials' # required + workspace: 'your-organization/workspace-1' # required + key: '$GOOGLE_KEY' # required + overwrite: True # optional + + - type: 'aws' # required + name: 'your-aws-credentials' # required + workspace: 'your-organization/workspace-1' # required + access-key: '$AWS_ACCESS_KEY_ID' # required + secret-key: '$AWS_SECRET_ACCESS_KEY' # required + assume-role-arn: '$AWS_ASSUME_ROLE_ARN' # required + overwrite: True # optional + + - type: 'azure' # required + name: 'your-azure-credentials' # required + workspace: 'your-organization/workspace-1' # required + batch-key: '$AZURE_BATCH_KEY' # required + batch-name: 'your-batch-name' # required + storage-key: '$AZURE_STORAGE_KEY' # required + storage-name: 'your-storage-name' # required + overwrite: True # optional + + - type: 'codecommit' # required + name: 'your-codecommit-credentials' # required + workspace: 'your-organization/workspace-1' # required + access-key: '$CODECOMMIT_USER' # required + secret-key: '$CODECOMMIT_PASSWORD' # required + base-url: '$CODECOMMIT_BASEURL' # optional + overwrite: False # optional diff --git a/platform_versioned_docs/version-24.2/seqerakit/templates/datasets.yaml b/platform_versioned_docs/version-24.2/seqerakit/templates/datasets.yaml new file mode 100644 index 000000000..b93364ddc --- /dev/null +++ b/platform_versioned_docs/version-24.2/seqerakit/templates/datasets.yaml @@ -0,0 +1,8 @@ +## To see the full list of options available, run: "tw datasets add -h" +datasets: + - name: 'your-dataset' # required + description: 'Your dataset description' # optional + header: true # optional + workspace: 'your-organization/workspace-1' # required + file-path: './datasets/dataset.csv' # required + overwrite: True # optional \ No newline at end of file diff --git a/platform_versioned_docs/version-24.2/seqerakit/templates/labels.yaml b/platform_versioned_docs/version-24.2/seqerakit/templates/labels.yaml new file mode 100644 index 000000000..f8496a7f4 --- /dev/null +++ b/platform_versioned_docs/version-24.2/seqerakit/templates/labels.yaml @@ -0,0 +1,6 @@ +## To see the full list of options available, run: "tw labels add -h" +labels: + - name: 'label_name' # required + value: 'label_value' # required + workspace: 'your-organization/workspace-1' # required + overwrite: True # optional diff --git a/platform_versioned_docs/version-24.2/seqerakit/templates/launch.yaml b/platform_versioned_docs/version-24.2/seqerakit/templates/launch.yaml new file mode 100644 index 000000000..745c63c04 --- /dev/null +++ b/platform_versioned_docs/version-24.2/seqerakit/templates/launch.yaml @@ -0,0 +1,23 @@ +## To see the full list of options available, run: "tw launch -h" +## The options will vary if you are launching a pipeline: +## 1. Pre-configured in the Seqera Launchpad with most options already specified +## 2. From source via a remote Git repository URL where you must explicitly specify all launch options +## Note: overwrite is not supported for "tw launch" + +launch: + - name: 'launchpad-launch' # required + workspace: 'your-organization/workspace-1' # required + pipeline: 'nf-core-rnaseq' # required + params: # optional + outdir: 's3://your-bucket/results' + + - name: 'remote-launch' # required + workspace: 'your-organization/workspace-1' # required + compute-env: 'your-aws-compute-environment' # required + pipeline: 'https://github.com/my_username/my_repo' # required + work-dir: 's3://your-bucket' # optional + profile: 'test' # optional + revision: 'main' # optional + params-file: './pipelines/params.yml' # optional + config: './pipelines/nextflow.config' # optional + pre-run: './pipelines/pre-run.txt' # optional diff --git a/platform_versioned_docs/version-24.2/seqerakit/templates/members.yaml b/platform_versioned_docs/version-24.2/seqerakit/templates/members.yaml new file mode 100644 index 000000000..f06b9f4af --- /dev/null +++ b/platform_versioned_docs/version-24.2/seqerakit/templates/members.yaml @@ -0,0 +1,5 @@ +## To see the full list of options available, run: "tw members add -h" +members: + - user: 'user@domain.com' # required + organization: 'your-organization' # required + overwrite: True # optional \ No newline at end of file diff --git a/platform_versioned_docs/version-24.2/seqerakit/templates/organizations.yaml b/platform_versioned_docs/version-24.2/seqerakit/templates/organizations.yaml new file mode 100644 index 000000000..6cf0518d2 --- /dev/null +++ b/platform_versioned_docs/version-24.2/seqerakit/templates/organizations.yaml @@ -0,0 +1,8 @@ +## To see the full list of options available, run: "tw organizations add -h" +organizations: + - name: 'your-organization' # required + full-name: 'Your Organization LLC' # required + description: 'Your organization description' # optional + location: 'Global' # optional + website: 'https://domain.com/' # optional + overwrite: True # optional \ No newline at end of file diff --git a/platform_versioned_docs/version-24.2/seqerakit/templates/participants.yaml b/platform_versioned_docs/version-24.2/seqerakit/templates/participants.yaml new file mode 100644 index 000000000..3c5e77da6 --- /dev/null +++ b/platform_versioned_docs/version-24.2/seqerakit/templates/participants.yaml @@ -0,0 +1,10 @@ +## To see the full list of options available, run: "tw participants add -h" +participants: + - name: 'your-team' # required + type: 'TEAM' # required + workspace: 'your-organization/workspace-1' # required + role: 'ADMIN' # required + - name: 'team-member@domain.com' # required + type: 'MEMBER' # required + workspace: 'your-organization/workspace-1' # required + role: 'LAUNCH' # required \ No newline at end of file diff --git a/platform_versioned_docs/version-24.2/seqerakit/templates/pipelines.yaml b/platform_versioned_docs/version-24.2/seqerakit/templates/pipelines.yaml new file mode 100644 index 000000000..017d72527 --- /dev/null +++ b/platform_versioned_docs/version-24.2/seqerakit/templates/pipelines.yaml @@ -0,0 +1,24 @@ +## To see the full list of options available, run: "tw pipelines add -h" +## The options required to create pipelines can be specified: +## 1. Explicitly in this file +## 2. Via a JSON file exported from Seqera with the "tw pipelines export" command +pipelines: + - name: 'your-first-pipeline' # required + workspace: 'your-organization/workspace-1' # required + description: 'Options specified in this file' # optional + compute-env: 'your-aws-compute-environment' # required + work-dir: 's3://your-bucket' # optional + profile: 'test' # optional + revision: 'main' # required + params: # optional + outdir: 's3://your-bucket/results' + config: './pipelines/nextflow.config' # optional + pre-run: './pipelines/pre-run.txt' # optional + url: 'https://github.com/my_username/my_repo' # required + overwrite: True # optional + - name: 'your-second-pipeline' # required + workspace: 'your-organization/workspace-1' # required + description: 'Options specified via JSON file' # optional + compute-env: 'your-aws-compute-environment' # required + file-path: './pipelines/pipeline.json' # required + overwrite: True # optional diff --git a/platform_versioned_docs/version-24.2/seqerakit/templates/secrets.yaml b/platform_versioned_docs/version-24.2/seqerakit/templates/secrets.yaml new file mode 100644 index 000000000..743faba3b --- /dev/null +++ b/platform_versioned_docs/version-24.2/seqerakit/templates/secrets.yaml @@ -0,0 +1,6 @@ +## To see the full list of options available, run: "tw secrets add -h" +secrets: + - name: 'your-secret' # required + workspace: 'your-organization/workspace-1' # required + value: 'your-secret-value' # required + overwrite: True # optional \ No newline at end of file diff --git a/platform_versioned_docs/version-24.2/seqerakit/templates/seqerakit-e2e.yaml b/platform_versioned_docs/version-24.2/seqerakit/templates/seqerakit-e2e.yaml new file mode 100644 index 000000000..fcdea506e --- /dev/null +++ b/platform_versioned_docs/version-24.2/seqerakit/templates/seqerakit-e2e.yaml @@ -0,0 +1,148 @@ +organizations: + - name: 'your-organization' # required + full-name: 'Your organization LLC' # required + description: 'Organization created E2E with seqerakit CLI scripting' # optional + location: 'Global' # optional + website: 'https://yourdomain.com/' # optional + overwrite: True # optional + +teams: + - name: 'pipelines-team' # required + organization: 'your-organization' # required + description: 'Pipelines team @ Your organization' # optional + members: # optional + - 'user1@domain.com' + - 'user2@domain.com' + - 'user3@domain.com' + overwrite: True # optional + +workspaces: + - name: 'workspace-1' # required + full-name: 'Workspace one' # required + organization: 'your-organization' # required + description: 'Workspace created E2E with seqerakit CLI scripting' # optional + visibility: 'PRIVATE' # optional + overwrite: True # optional + +participants: + - name: 'pipelines-team' # required + type: 'TEAM' # required + workspace: 'your-organization/workspace-1' # required + role: 'ADMIN' # required + overwrite: True # optional + - name: 'user@domain.com' # required + type: 'MEMBER' # required + workspace: 'your-organization/workspace-1' # required + role: 'LAUNCH' # required + overwrite: True # optional + +credentials: + - type: 'github' # required + name: 'github-credentials' # required + workspace: 'your-organization/workspace-1' # required + username: 'user1' # required + password: '$TOWER_GITHUB_PASSWORD' # required + overwrite: True # optional + - type: 'container-reg' # required + name: 'dockerhub-credentials' # required + workspace: 'your-organization/workspace-1' # required + username: 'user1' # required + password: '$DOCKERHUB_PASSWORD' # required + registry: 'docker.io' # required + overwrite: True # optional + - type: 'aws' # required + name: 'aws-credentials' # required + workspace: 'your-organization/workspace-1' # required + access-key: '$AWS_ACCESS_KEY_ID' # required + secret-key: '$AWS_SECRET_ACCESS_KEY' # required + assume-role-arn: '$AWS_ASSUME_ROLE_ARN' # required + overwrite: True # optional + +secrets: + - name: 'SENTIEON_LICENSE_BASE64' # required + workspace: 'your-organization/workspace-1' # required + value: '$SENTIEON_LICENSE_BASE64' # required + overwrite: True # optional + +compute-envs: +# To create a compute environment with options specified through YAML (AWS Example) + - type: aws-batch # required + config-mode: forge # required for AWS and Azure + name: 'your-aws-compute-environment' # required + workspace: 'your-organization/workspace-1' # required + credentials: 'your-aws-credentials' # required + region: 'eu-west-1' # required + work-dir: 's3://your-bucket' # required + provisioning-model: 'SPOT' # optional + fusion-v2: False # optional + wave: False # optional + fargate: False # optional + fast-storage: False # optional + instance-types: 'c6i,r6i,m6i' # optional, comma-separated list + no-ebs-auto-scale: True # optional + max-cpus: 500 # required + labels: 'label1,label2' # optional, comma-separated list + vpc-id: 'vpc-1234567890' # optional + subnets: 'subnet-1234567890,subnet-1234567891' # optional, comma-separated list + security-groups: 'sg-1234567890,sg-1234567891' # optional, comma-separated list + allow-buckets: 's3://your-bucket,s3://your-other-bucket' # optional, comma-separated list + wait: 'AVAILABLE' # optional + overwrite: False # optional + +# To create a compute environment from a JSON configuration file (AWS Example) + - name: 'your-aws-compute-environment' # required + workspace: 'your-organization/workspace-1' # required + credentials: 'your-aws-credentials' # required + wait: 'AVAILABLE' # optional + file-path: './compute-envs/your_aws_compute_environment.json' # required + overwrite: True # optional + +datasets: + - name: 'rnaseq_samples' # required + description: 'Samplesheet to run the nf-core/rnaseq pipeline end-to-end' # optional + header: true # optional + workspace: 'your-organization/workspace-1' # required + file-path: 'path/to/rnaseq_samples.csv' # required + overwrite: True # optional + +pipelines: +# To create a pipeline with options specified through YAML + - name: 'nf-core-rnaseq' # required + url: 'https://github.com/nf-core/rnaseq' # required + workspace: 'your-organization/workspace-1' # required + description: 'RNA sequencing analysis pipeline with gene/isoform counts and extensive quality control.' # optional + compute-env: 'your-aws-compute-environment' # required + work-dir: 's3://your-bucket/pipeline/working/directory' # optional + profile: 'test' # optional + revision: '3.12.0' # required + params: # optional + outdir: 's3://your-bucket/nf-core-rnaseq/results' + config: 'path/to/nextflow.config' # optional + pre-run: 'path/to/pipeline/pre_run.txt' # optional + overwrite: True # optional + +# To create a pipeline from a JSON configuration file + - name: 'nf-core-sarek' # required + workspace: 'your-organization/workspace-1' # required + compute-env: 'your-aws-compute-environment' # required + file-path: 'path/to/nf-core-sarek_pipeline.json' # required + overwrite: True # optional + +launch: +# To launch a preconfigured pipeline from your workspace Launchpad + - name: 'nf-core-rnaseq-launchpad' # required + workspace: 'your-organization/workspace-1' # required + pipeline: 'nf-core-rnaseq' # required + compute-env: 'your-aws-compute-environment' # optional (defaults to workspace primary compute environment) + +# To launch an unsaved pipeline, include the pipeline repository URL and launch details + - name: 'nf-core-rnaseq-remote' # required + workspace: 'your-organization/workspace-1' # required + pipeline: 'https://github.com/nf-core/rnaseq' # required + compute-env: 'your-aws-compute-environment' # optional (defaults to workspace primary compute environment) + work-dir: 's3://your-bucket/pipeline/working/directory' # optional + profile: 'test' # optional + revision: '3.12.0' # optional + params-file: 'path/to/pipelines/nf_core_rnaseq_params.yml' # optional + config: 'path/to/pipelines/nextflow.config' # optional + pre-run: 'path/to/pipelines/pre_run.txt' # optional diff --git a/platform_versioned_docs/version-24.2/seqerakit/templates/teams.yaml b/platform_versioned_docs/version-24.2/seqerakit/templates/teams.yaml new file mode 100644 index 000000000..b1258de24 --- /dev/null +++ b/platform_versioned_docs/version-24.2/seqerakit/templates/teams.yaml @@ -0,0 +1,8 @@ +## To see the full list of options available, run: "tw teams add -h" +teams: + - name: 'your-team' # required + organization: 'your-organization' # required + description: 'Your team description.' # optional + members: # optional + - 'team-member@domain.com' + overwrite: True # optional \ No newline at end of file diff --git a/platform_versioned_docs/version-24.2/seqerakit/templates/workspaces.yaml b/platform_versioned_docs/version-24.2/seqerakit/templates/workspaces.yaml new file mode 100644 index 000000000..ca9392340 --- /dev/null +++ b/platform_versioned_docs/version-24.2/seqerakit/templates/workspaces.yaml @@ -0,0 +1,8 @@ +## To see the full list of options available, run: "tw workspaces add -h" +workspaces: + - name: 'workspace-1' # required + full-name: 'Workspace one' # required + organization: 'your-organization' # required + description: 'Your workspace description' # optional + visibility: 'PRIVATE' # optional + overwrite: True # optional \ No newline at end of file diff --git a/platform_versioned_docs/version-24.2/seqerakit/yaml-configuration.mdx b/platform_versioned_docs/version-24.2/seqerakit/yaml-configuration.mdx new file mode 100644 index 000000000..965161262 --- /dev/null +++ b/platform_versioned_docs/version-24.2/seqerakit/yaml-configuration.mdx @@ -0,0 +1,154 @@ +--- +title: "YAML configuration" +description: "seqerakit YAML configuration file options" +date: "21 Oct 2024" +tags: [seqerakit, cli, automation, yaml, configuration] +--- + +Seqerakit supports the creation and deletion of the following Seqera Platform resources, listed here with their respective Platform CLI resource names: + +- Pipeline actions: `actions` +- Compute environments: `compute-envs` +- Credentials: `credentials` +- Datasets: `datasets` +- Labels (including resource labels): `labels` +- Pipeline launch: `launch` +- Organization members: `members` +- Organizations: `organizations` +- Workspace and team participants: `participants` +- Pipelines: `pipelines` +- Pipeline secrets: `secrets` +- Teams: `teams` +- Workspaces: `workspaces` + +To determine the options to provide as definitions in your YAML file, run the Platform CLI help command for the resource you want to create. + +1. Retrieve CLI options: + + Obtain a list of available CLI options for defining your YAML file with the Platform CLI `help` command. For example, to add a pipeline to your workspace, view the options for adding a pipeline: + + ```shell-session + tw pipelines add -h + ``` + + ```shell-session + Usage: tw pipelines add [OPTIONS] PIPELINE_URL + + Add a workspace pipeline. + + Parameters: + * PIPELINE_URL Nextflow pipeline URL. + + Options: + * -n, --name= Pipeline name. + -w, --workspace= Workspace numeric identifier (TOWER_WORKSPACE_ID as default) or workspace reference as OrganizationName/WorkspaceName + -d, --description= Pipeline description. + --labels=[,...] List of labels seperated by coma. + -c, --compute-env= Compute environment name. + --work-dir= Path where the pipeline scratch data is stored. + -p, --profile=[,...] Comma-separated list of one or more configuration profile names you want to use for this pipeline execution. + --params-file= Pipeline parameters in either JSON or YML format. + --revision= A valid repository commit Id, tag or branch name. + ... + ``` + +1. Define key-value pairs in YAML: + + Translate each CLI option into a key-value pair in the YAML file. The structure of your YAML file should reflect the hierarchy and format of the CLI options. For example: + + ```yaml + pipelines: + - name: 'my_first_pipeline' + url: 'https://github.com/username/my_pipeline' + workspace: 'my_organization/my_workspace' + description: 'My test pipeline' + labels: 'yeast,test_data' + compute-env: 'my_compute_environment' + work-dir: 's3://my_bucket' + profile: 'test' + params-file: '/path/to/params.yaml' + revision: '1.0' + ``` + + In this example: + + - The keys (`name`, `url`, `workspace`, and so forth) are the keys derived from the CLI options. + - The corresponding values are user-defined. + +#### Best practices + +- The indentation and structure of the YAML file must be correct — YAML is sensitive to formatting. +- Use quotes around strings that contain special characters or spaces. +- To list multiple values (such as multiple `labels`, `instance-types`, or `allow-buckets`), separate values with commas. This is shown with `labels` in the preceding example. +- For complex configurations, see [Templates](./templates.mdx). + +### Templates + +See [Templates](./templates.mdx) for YAML file templates for each of the entities that can be created in Seqera. + +### YAML Configuration Options + +Some options handled specially by `seqerakit` or not exposed as `tw` CLI options can be provided in your YAML configuration file. + +#### Pipeline parameters using `params` and `params-file` + +To specify pipeline parameters, use `params:` to specify a list of parameters or `params-file:` to point to a parameters file. + +For example, to specify pipeline parameters within your YAML: + +```yaml +params: + outdir: 's3://path/to/outdir' + fasta: 's3://path/to/reference.fasta' +``` + +To specify a file containing pipeline parameters: + +```yaml +params-file: '/path/to/my/parameters.yaml' +``` + +Or provide both: + +```yaml +params-file: '/path/to/my/parameters.yaml' +params: + outdir: 's3://path/to/outdir' + fasta: 's3://path/to/reference.fasta' +``` + +:::note +If duplicate parameters are provided, the parameters provided as key-value pairs inside the `params` nested dictionary of the YAML file will take precedence **over** values in the `params-file`. +::: + +#### Overwrite + +For every entity defined in your YAML file, specify `overwrite: True` to overwrite any existing Seqera entities of the same name. + +Seqerakit will first check to see if the name of the entity exists. If so, it will invoke a `tw delete` command before attempting to create it based on the options defined in the YAML file. + +```shell-session +DEBUG:root: Overwrite is set to 'True' for organizations + +DEBUG:root: Running command: tw -o json organizations list +DEBUG:root: The attempted organizations resource already exists. Overwriting. + +DEBUG:root: Running command: tw organizations delete --name $SEQERA_ORGANIZATION_NAME +DEBUG:root: Running command: tw organizations add --name $SEQERA_ORGANIZATION_NAME --full-name $SEQERA_ORGANIZATION_NAME --description 'Example of an organization' +``` + +#### Specify JSON configuration files with `file-path` + +The Platform CLI allows the export and import of entities through JSON configuration files for pipelines and compute environments. To use these files to add a pipeline or compute environment to a workspace, use the `file-path` key to specify a path to a JSON configuration file. + +An example of the `file-path` option is provided in the [compute-envs.yaml](./templates/compute-envs.yaml) template: + +```yaml +compute-envs: + - name: 'my_aws_compute_environment' # required + workspace: 'my_organization/my_workspace' # required + credentials: 'my_aws_credentials' # required + wait: 'AVAILABLE' # optional + file-path: './compute-envs/my_aws_compute_environment.json' # required + overwrite: True +``` diff --git a/platform_versioned_sidebars/version-23.3-sidebars.json b/platform_versioned_sidebars/version-23.3-sidebars.json index 7401ef888..6e92cde60 100644 --- a/platform_versioned_sidebars/version-23.3-sidebars.json +++ b/platform_versioned_sidebars/version-23.3-sidebars.json @@ -199,8 +199,28 @@ "collapsed": true, "items": [ "api/overview", - "cli/cli" - ] + { + "type": "category", + "label": "CLI", + "collapsed": true, + "items": [ + "cli/overview", + "cli/installation", + "cli/commands" + ] + }, + { + "type": "category", + "label": "Seqerakit", + "collapsed": true, + "items": [ + "seqerakit/installation", + "seqerakit/commands", + "seqerakit/yaml-configuration", + "seqerakit/templates" + ] + } + ] }, { "type": "category", diff --git a/platform_versioned_sidebars/version-23.4-sidebars.json b/platform_versioned_sidebars/version-23.4-sidebars.json index d377bb9c6..66de4899e 100644 --- a/platform_versioned_sidebars/version-23.4-sidebars.json +++ b/platform_versioned_sidebars/version-23.4-sidebars.json @@ -195,7 +195,18 @@ "cli/installation", "cli/commands" ] - } + }, + { + "type": "category", + "label": "Seqerakit", + "collapsed": true, + "items": [ + "seqerakit/installation", + "seqerakit/commands", + "seqerakit/yaml-configuration", + "seqerakit/templates" + ] + } ] }, { diff --git a/platform_versioned_sidebars/version-24.1-sidebars.json b/platform_versioned_sidebars/version-24.1-sidebars.json index 189b21703..b38f75a99 100644 --- a/platform_versioned_sidebars/version-24.1-sidebars.json +++ b/platform_versioned_sidebars/version-24.1-sidebars.json @@ -196,7 +196,7 @@ "type": "category", "label": "Developer tools", "collapsed": true, - "items": [ + "items": [ "api/overview", { "type": "category", @@ -208,7 +208,18 @@ "cli/commands" ] }, - "pipeline-actions/overview" + { + "type": "category", + "label": "Seqerakit", + "collapsed": true, + "items": [ + "seqerakit/installation", + "seqerakit/commands", + "seqerakit/yaml-configuration", + "seqerakit/templates" + ] + }, + "pipeline-actions/overview" ] }, { diff --git a/platform_versioned_sidebars/version-24.2-sidebars.json b/platform_versioned_sidebars/version-24.2-sidebars.json index 134c1232e..85a299f6b 100644 --- a/platform_versioned_sidebars/version-24.2-sidebars.json +++ b/platform_versioned_sidebars/version-24.2-sidebars.json @@ -208,6 +208,17 @@ "cli/commands" ] }, + { + "type": "category", + "label": "Seqerakit", + "collapsed": true, + "items": [ + "seqerakit/installation", + "seqerakit/commands", + "seqerakit/yaml-configuration", + "seqerakit/templates" + ] + }, "pipeline-actions/overview" ] },