From ab1ad3c9a293e70d28e84f4b0fa75090f5baabcc Mon Sep 17 00:00:00 2001 From: Rosie Yohannan Date: Mon, 2 Dec 2024 23:36:03 +0000 Subject: [PATCH 01/20] convert databases --- jekyll/_cci2/{databases.md => databases.adoc} | 104 +++++++++--------- 1 file changed, 50 insertions(+), 54 deletions(-) rename jekyll/_cci2/{databases.md => databases.adoc} (52%) diff --git a/jekyll/_cci2/databases.md b/jekyll/_cci2/databases.adoc similarity index 52% rename from jekyll/_cci2/databases.md rename to jekyll/_cci2/databases.adoc index 36c9b197fd..ae73af9203 100644 --- a/jekyll/_cci2/databases.md +++ b/jekyll/_cci2/databases.adoc @@ -1,45 +1,52 @@ --- -layout: classic-docs -title: "Configure Databases" -description: "This document describes how to use the official CircleCI pre-built Docker container images for a database service in CircleCI." contentTags: platform: - Cloud - Server v4.x - - Server v3.x --- += Configure databases +:page-description: This document describes how to use the official CircleCI pre-built Docker container images for a database service in CircleCI. +:experimental: +:icons: font +:page-layout: classic-docs +:page-liquid: -This document describes how to use the official CircleCI pre-built Docker container images for a database service in CircleCI. +This page describes how to use the official CircleCI pre-built Docker container images for a database service in CircleCI. -* TOC -{:toc} +[#overview] +== Introduction -## Overview -{: #overview } +CircleCI provides pre-built images (Docker and VM) for languages and services, like databases, with commonly-used tools preinstalled. For a full list, see the link:https://circleci.com/developer/images[CircleCI Developer Hub]. +When configuring a job, the first executor you select is the primary container. The primary container is where all execution occurs. You can add additional containers to the job, these are service, or secondary containers. -CircleCI provides pre-built images for languages and services like databases with a lot of conveniences added into the images on [CircleCI Developer Hub](https://circleci.com/developer/images). +The example in the next section shows a xref:configuration-reference#[CircleCI configuration] file with the following: -The following example shows a [`.circleci/config.yml`]({{ site.baseurl }}/configuration-reference/) file with one job called `build`. Docker is selected for the executor and the first image is the primary container where all execution occurs. This example has a second image and this will be used as the service image. The first image is the programming language Python. The Python image has `pip` installed and `-browsers` for browser testing. The service image gives access to additional services like databases. +* One job named `build`. +* Docker is selected for the execution environment. +* The first image is the primary container where all execution occurs. The first image is the CircleCI Python image. The Python image has `pip` installed and `-browsers` for browser testing. +* The second image specified is for a service container. The service image gives access to additional services, like databases. -## PostgreSQL database testing example -{: #postgresql-database-testing-example } +[#postgresql-database-testing-example] +== PostgreSQL database testing example -In the primary image, the config defines an environment variable with the `environment` key, giving it a URL. The URL tells it that it is a PostgreSQL database, so it will default to the PostgreSQL default port. This pre-built circleci image includes a database and a user already. The username is `postgres` and database is `circle_test`. So, you can begin with using that user and database without having to set it up yourself. +In the primary image, the config defines an environment variable with the `environment` key, giving it a URL. The URL tells it that it is a PostgreSQL database, so it will default to the PostgreSQL default port. This pre-built CircleCI image includes a database and a user already. The username is `postgres` and database is `circle_test`. You can begin with using that user and database without having to set it up yourself. Set the `POSTGRES_USER` environment variable in your CircleCI config to `postgres` to add the role to the image as follows: -{% include snippets/docker-auth.md %} +{% include snippets/docker-auth.adoc %} -```yml +[,yml] +---- - image: cimg/postgres:14.0 environment: POSTGRES_USER: postgres -``` +---- {% raw %} -```yml +[,yml] +---- version: 2.1 jobs: @@ -69,46 +76,45 @@ jobs: psql \ -d $TEST_DATABASE_URL \ -c "INSERT INTO test VALUES ('John'), ('Joanna'), ('Jennifer');" -``` +---- {% endraw %} -The `steps` run `checkout` first, then install the Postgres client tools. The `cimg/postgres:14.0` image doesn't install any client-specific database adapters. For example, for Python, you might install [`psycopg2`](https://www.psycopg.org/) so that you can interface with the PostgreSQL database. See [Pre-Built CircleCI Services Images]({{ site.baseurl }}/circleci-images/#next-gen-service-images) for the list of images. +The `steps` run `checkout` first, then install the PostgreSQL client tools. The `cimg/postgres:14.0` image doesn't install any client-specific database adapters. For example, for Python, you might install link:https://www.psycopg.org/[`psycopg2`] so that you can interface with the PostgreSQL database. In this example, the config installs the PostgreSQL client tools, `postgresql-client` via `apt-get`, to get access to `psql`. Installing packages in images requires administrator privileges, therefore `sudo` is used - a password is not required. Two commands follow the `postgresql-client` installation that interact with the database service. These are SQL commands that create a table called test and insert a value into that table. After committing changes and pushing them, the build is automatically triggered on CircleCI and spins up the primary container. -CircleCI injects a number of convenience environment variables into the primary container that you can use in conditionals throughout the rest of your build. For example, CIRCLE_NODE_INDEX and CIRCLE_NODE_TOTAL are related to concurrent execution environments. See the [Project values and variables]({{site.baseurl}}/variables#built-in-environment-variables) document for details. -{: class="alert alert-info" } +NOTE: CircleCI injects a number of convenience environment variables into the primary container that you can use in conditionals throughout the rest of your build. For example, CIRCLE_NODE_INDEX and CIRCLE_NODE_TOTAL are related to concurrent execution environments. See the link:{{site.baseurl}}/variables#built-in-environment-variables[Project values and variables] document for details. When the database service spins up, it automatically creates the database `circle_test` and the `postgres` role that you can use to log in and run your tests. Then the database tests run to create a table and insert a value into it. -## Optional customization -{: #optional-customization } +[#optional-customization] +== Optional customization This section describes additional optional configuration for further customizing your build and avoiding race conditions. -### Using binaries -{: #using-binaries } - +[#using-binaries] +=== Using binaries To use `pg_dump`, `pg_restore` and similar utilities requires some extra configuration to ensure that `pg_dump` invocations will also use the correct version. Add the following to your `config.yml` file to enable `pg_*` or equivalent database utilities: -```yml +[,yml] +---- steps: # Add the Postgres 12.0 binaries to the path. - run: echo 'export PATH=/usr/lib/postgresql/1bin/:"$PATH"' >> "$BASH_ENV" -``` - -### Using Dockerize to wait for dependencies -{: #using-dockerize-to-wait-for-dependencies } +---- +[#using-dockerize-to-wait-for-dependencies] +=== Using Dockerize to wait for dependencies -Using multiple Docker containers for your jobs may cause race conditions if the service in a container does not start before the job tries to use it. For example, your PostgreSQL container might be running, but might not be ready to accept connections. Work around this problem by using `dockerize` to wait for dependencies. -Following is an example of how to do this in your CircleCI `config.yml` file: +Using multiple Docker containers for your jobs may cause race conditions if the service in a container does not start before the job tries to use it. For example, your PostgreSQL container might be running, but might not be ready to accept connections. Work around this problem by using `dockerize` to wait for dependencies. +Following is an example of how to do this in your CircleCI config file: -```yml +[,yml] +---- version: 2.1 jobs: build: @@ -129,29 +135,19 @@ jobs: - run: name: Wait for db command: dockerize -wait tcp://localhost:5432 -timeout 1m -``` +---- It is possible to apply the same principle for the following databases: -- MySQL: - -`dockerize -wait tcp://localhost:3306 -timeout 1m` - -- Redis: - -`dockerize -wait tcp://localhost:6379 -timeout 1m` - -Redis also has a CLI available: - -`sudo apt-get install redis-tools ; while ! redis-cli ping 2>/dev/null ; do sleep 1 ; done` - -- Other services such as web servers: - -`dockerize -wait http://localhost:80 -timeout 1m` +* MySQL: `dockerize -wait tcp://localhost:3306 -timeout 1m` -## See also -{: #see-also } +* Redis: `dockerize -wait tcp://localhost:6379 -timeout 1m` ++ +Redis also has a CLI available: `sudo apt-get install redis-tools ; while ! redis-cli ping 2>/dev/null ; do sleep 1 ; done` +* Other services, such as web servers: `+dockerize -wait http://localhost:80 -timeout 1m+` -Refer to the [Database Configuration Examples]({{ site.baseurl }}/postgres-config/) document for additional configuration file examples. +[#see-also] +== See also +Refer to the xref:postgres-config#[Database Configuration Examples] document for additional configuration file examples. From 125ef183bed8e8d85f0519b6fdffda3dbb36f15c Mon Sep 17 00:00:00 2001 From: Rosie Yohannan Date: Mon, 27 Jan 2025 11:46:21 +0000 Subject: [PATCH 02/20] convert circleci-images and deploy to iOS --- jekyll/_cci2/circleci-images.adoc | 358 +++++++++++++++++ jekyll/_cci2/circleci-images.md | 361 ----------------- jekyll/_cci2/deploy-ios-applications.adoc | 432 +++++++++++++++++++++ styles/config/vocabularies/Docs/accept.txt | 7 + 4 files changed, 797 insertions(+), 361 deletions(-) create mode 100644 jekyll/_cci2/circleci-images.adoc delete mode 100644 jekyll/_cci2/circleci-images.md create mode 100644 jekyll/_cci2/deploy-ios-applications.adoc diff --git a/jekyll/_cci2/circleci-images.adoc b/jekyll/_cci2/circleci-images.adoc new file mode 100644 index 0000000000..5aeffa9413 --- /dev/null +++ b/jekyll/_cci2/circleci-images.adoc @@ -0,0 +1,358 @@ +--- +contentTags: + platform: + - Cloud + - Server v4+ +--- += Convenience images +:page-description: Listing of available Docker images maintained by CircleCI +:experimental: +:icons: font +:page-layout: classic-docs +:page-liquid: + +CAUTION: *Legacy images with the prefix `circleci/` were https://discuss.circleci.com/t/legacy-convenience-image-deprecation/41034[deprecated]* on December 31, 2021. For faster builds, upgrade your projects with https://circleci.com/blog/announcing-our-next-generation-convenience-images-smaller-faster-more-deterministic/[next-generation convenience images]. + +This document provides information about convenience images (pre-built Docker images maintained by CircleCI) and a listing by language, service type, and tags. + +[#overview] +== Overview + +For convenience, CircleCI maintains several Docker images. These images are +typically extensions of official Docker images, and include tools especially +useful for CI/CD. + +This document provides an overview of best practices when using a convenience image. We advise using the *next-generation* convenience images (these start `cimg/`) rather than *legacy images*, as explained below. + +If you would like to directly search for an image, you can browse CircleCI Docker images in the following locations: + +* Visit the https://circleci.com/developer/images/[Developer Hub] for links to all the repositories for each next-gen image. +* Find all CircleCI pre-built images available on https://hub.docker.com/u/cimg[Docker Hub]. + +NOTE: CircleCI occasionally makes scheduled changes to images to fix bugs or otherwise improve functionality, and these changes can sometimes affect how images work in CircleCI jobs. Follow the https://discuss.circleci.com/tags/convenience-images[*convenience-images* tag on Discuss] to be notified in advance of scheduled maintenance._ + +[#examples] +=== Examples + +Refer to the link:{{ site.baseurl }}/examples-and-guides-overview/[Examples and Guides Overview] for examples of using pre-built CircleCI Docker Images in a demo application. + +[#next-generation-convenience-images] +== Next-generation convenience images + +The next-generation convenience images in this section were built from the ground up with CI, efficiency, and determinism in mind. Here are some of the highlights: + +*Faster spin-up time* - In Docker terminology, these next-gen images will generally have fewer and smaller layers. Using these new images will lead to faster image downloads when a build starts, and a higher likelihood that the image is already cached on the host. + +*Improved reliability and stability* - The existing legacy convenience images are +rebuilt practically every day with potential changes from upstream that we cannot +always test fast enough. This leads to frequent breaking changes, which is not +the best environment for stable, deterministic builds. Next-gen images will only +be rebuilt for security and critical-bugs, leading to more stable and +deterministic images. + +[#circleci-base-image] +=== CircleCI base image + +Using the `base` image in your config looks like the example shown below: + +[,yaml] +---- +jobs: + myjob: + docker: + - image: cimg/base:2021.04 +---- + +{% include snippets/docker-auth.adoc %} + +`cimg/base:2021.04` is a Ubuntu-based image designed to install the bare minimum. The +next-generation convenience images are based on this image. + +*When to use it?* + +If you need a generic image to run on CircleCI, to use with orbs, or to use as a +base for your own custom Docker image, this image is for you. + +*Resources* + +You can find more config examples for this image on the https://circleci.com/developer/images/image/cimg/base[Developer Hub], and the source code and documentation on https://github.com/CircleCI-Public/cimg-base[GitHub]. + +The example below demonstrates how to use the next-gen Go image, which is based off the `base` image above. + +[,yaml] +---- +jobs: + myjob: + docker: + - image: cimg/go:1.16 +---- + +This Go image is a direct replacement for the legacy CircleCI Go image (`circleci/golang`). Note, the Docker Hub namespace is `cimg`. You can view other next generation images for other languages <>. + +[#best-practices] +== Best practices + +The next-gen convenience images in the following sections are based on the most recent Ubuntu LTS Docker images and installed with the base libraries for the language or services, so it is best practice to use the most specific image possible. This makes your builds more deterministic by preventing an upstream image from introducing unintended changes to your image. + +That is, to prevent unintended changes that come from upstream, instead of using `cimg/ruby:2.4-node` use a more specific version of these containers to ensure the image does not change with upstream changes until you change the tag. + +For example, pin down those images to a specific point version, like `cimg/ruby:2.4.10-node`. Specifying the version is possible for any of the CircleCI images. + +It is also possible to use the specific SHA of a image. For example, you can use +`cimg/ruby@sha256:e4aa60a0a47363eca3bbbb066620f0a5967370f6469f6831ad52231c87ca9390` +instead of `cimg/ruby:2.4.10-node`. Doing so allows you to test specific images +for as long as you would like before making any changes. + +[#notes-on-pinning-images] +=== Notes on pinning images + +* It is not recommended that you use the SHA for extended periods of time. If there is a major bug or security issue that requires a rebuild of the image, your pipeline's dependency on the image could inhibit you from acquiring the update that fixes that bug or patches a security issue. + +* If you are using a legacy image and you do not specify a tag, Docker +applies the `latest` tag. The `latest` tag refers to the most recent stable +release of an image. However, since this tag may change unexpectedly, it is best +practice to add an explicit image tag. + +* For Node.js variant Docker images (tags that end in `-node`) the LTS +release of Node.js is pre-installed. If you would like to include your own +specific version of Node.js / npm you can set it up in a series of `run` steps +in your `.circleci/config.yml`. Consider the example below, which installs a +specific version of Node.js alongside the Ruby image. + +[,yaml] +---- +version: 2.1 + +jobs: + build: + docker: + - image: cimg/ruby:2.7.1-node + steps: + - checkout + - run: + name: "Update Node.js and npm" + command: | + curl -sSL "https://nodejs.org/dist/v11.10.0/node-v11.10.0-linux-x64.tar.xz" | sudo tar --strip-components=2 -xJ -C /usr/local/bin/ node-v11.10.0-linux-x64/bin/node + curl https://www.npmjs.com/install.sh | sudo bash + - run: + name: Check current version of node + command: node -v +---- + +[#finding-an-image-id] +==== Finding an image id + +Follow these steps to find your Docker image id: + +. In the CircleCI application, navigate to the job in your pipeline for which you would like to know the Docker image. +. Toggle open the *Spin up environment* step. +. In the log output, locate the digest for the image. +. Add the image ID to the image name as shown below. + +[,shell] +---- +cimg/python@sha256:bdabda041f88d40d194c65f6a9e2a2e69ac5632db8ece657b15269700b0182cf +---- + +[#image-types] +== Image types + +CircleCI's convenience images fall into two categories: + +* *language* images +* *service* images + +All images add a `circleci` user as a system user. The sections below walk through the available next-generation and legacy images. + +[#next-gen-language-images] +=== Next-gen language images + +Next-gen language images are convenience images for common programming languages. +These images include relevant language tools and <>. +To use a language image, list it first under the `docker` key in your configuration, +making it the link:{{ site.baseurl }}/glossary/#primary-container[primary container] during execution. + +See the https://circleci.com/developer/images?imageType=docker[Developer Hub] for a full list of next-gen images. + +If your language is not listed, feel free to request an image on our https://ideas.circleci.com/[Ideas +Board]. First, check to see if that "idea" is +already on CircleCI Ideas. If it is, up-vote it. If not, create it and set the +category as "images". Finally, go and market your "idea" to friends, co-workers, +forums, and other communities in order to help it build traction. + +If we see an idea on the board take off, we will consider building it officially. + +[#next-gen-language-image-variants] +==== Next-gen language image variants + +CircleCI maintains several variants for the next-gen language image. For +next-gen images be sure to check each image listing for information on each +variant. The `-browsers` variant for next-gen images is still in progress. See +each image listing on the https://circleci.com/developer/images/[Developer Hub] +for details on which variants it supports. + +[#legacy-language-images] +=== Legacy language images + +The legacy language images are convenience images for common programming languages. +These images include both the relevant language and <>. +A language image should be listed first under the `docker` key in your configuration, +making it the link:{{ site.baseurl }}/glossary/#primary-container[primary container] during execution. + +CircleCI maintains legacy images for the languages below. + +* <> +* <> +* <> +* <> +* <> +* <> +* <> +* <> +* <> +* <> +* <> + +[#language-image-variants] +==== Language image variants + +CircleCI maintains several variants for language images. To use these variants, +add one of the following suffixes to the end of an image tag. + +* `-node` includes Node.js for polyglot applications +* `-browsers` includes Chrome, Firefox, OpenJDK v11, and geckodriver +* `-node-browsers` combines the `-node` and `-browsers` variants + +For example, if you want to add browsers to the `circleci/golang:1.9` image, use +the `circleci/golang:1.9-browsers` image. + +[#next-gen-service-images] +=== Next-gen service images + +Service images are convenience images for services like databases. These images should be listed *after* language images so they become secondary service containers. + +* https://circleci.com/developer/images/image/cimg/postgres[PostgreSQL] +* https://circleci.com/developer/images/image/cimg/mysql[MySQL] +* https://circleci.com/developer/images/image/cimg/mariadb[MariaDB] +* https://circleci.com/developer/images/image/cimg/redis[Redis] + +[#legacy-service-images] +=== Legacy service images + +CircleCI maintains legacy images for the services below. + +* <> +* <> +* <> +* <> +* <> +* <> +* <> + +[#service-image-variant] +==== Service image variant + +CircleCI maintains only one variant for service images. To speed up builds using +RAM volume, add the `-ram` suffix to the end of a service image tag. + +For example, if you want the `circleci/postgres:9.5-postgis` image to use RAM +volume, use the `circleci/postgres:9.5-postgis-ram` image. + +[#pre-installed-tools] +== Pre-installed tools + +All convenience images have been extended with additional tools, installed with `apt-get`: + +* `bzip2` +* `ca-certificates` +* `curl` +* `git` +* `gnupg` +* `gzip` +* `locales` +* `mercurial` (legacy images only) +* `net-tools` +* `netcat` +* `openssh-client` +* `parallel` +* `sudo` +* `tar` +* `unzip` +* `wget` +* `xvfb` (legacy images only) +* `zip` + +The specific version of a particular package that gets installed in a particular +CircleCI image variant depends on the default version included in the package +directory for the Linux distribution/version installed in that variant's base +image. The legacy CircleCI convenience images are https://packages.debian.org/jessie/[Debian Jessie]- +or https://packages.debian.org/stretch/[Stretch]-based images, +however the next-gen images, `cimg`, extend the official https://packages.ubuntu.com[Ubuntu] image. +For details on the next-gen images, see the https://circleci.com/developer/images/[Developer Hub]. Each image is tracked in its own repository. + +The following packages are pre-installed in convenience images using `curl` or other means. + +* https://docs.docker.com/install/[Docker client] +* https://docs.docker.com/compose/overview/[Docker Compose] +* https://github.com/jwilder/dockerize[dockerize] +* https://stedolan.github.io/jq/[jq] + +[#out-of-scope] +== Out of scope + +* If an image is not listed above, it is not available. As the convenience image +program is revamped, proposals for new images are not currently being +accepted. +* Old versions of software will not be rebuilt. Once an upstream image stops +building the tag for a specific release, say Node.js v8.1.0, then we stop +building it too. This means other tools in that image, such as `npm` in this +example, will no longer be updated either. +* We do not support building preview, beta, or release candidate images tags. On +occasion they will be available but these tags tend to cause our build system +for convenience images to fail. If you need a non-stable release of a +language, we suggest installing it via https://circleci.com/orbs/[an orb] +or a custom Docker image instead. + +[#legacy-image-tags-by-language] +== Legacy image tags by language + +Below is a list of the latest *legacy* convenience images, sorted by language. + +It is recommended to use next-generation images when possible. +For a list of the latest next-gen convenience images and details about the content of each image, visit the https://circleci.com/developer/[Developer Hub]. + +NOTE: Excluding <> and <>, *for legacy images* CircleCI +does *not* control which tags are used. These tags are chosen and maintained +by upstream projects. Do not assume that a given tag has the same meaning across +images! + +{% assign images = site.data.docker-image-tags | sort %} +{% for image in images %} + +=== {{ image[1].name }} + +{: # {\{image1name}} } + +*Resources:* + +* https://hub.docker.com/r/circleci/{{ image[0] }}[Docker Hub] - where this image is hosted as well as some useful instructions. + +*Usage:* Add the following under `docker:` in your config.yml: + +`- image: circleci/{{ image[0] }}:[TAG]` + +*Recent Tags:* + +See https://hub.docker.com/r/circleci/{{ image[0] }}/tags?ordering=last_updated[the tag list for circleci/{{ image[0\] }} on Docker Hub]. + +''' + +{% endfor %} + +[#next-steps] +== Next steps + +* See xref:private-images#[Using Docker Authenticated Pulls] for information about how to authorize your build to use an image in a private repository or in Amazon ECR. +* For information about macOS images for iOS, see the xref:testing-ios#[Testing iOS] page. +* See xref:building-docker-images#[Running Docker Commands] for information about how to build Docker images. diff --git a/jekyll/_cci2/circleci-images.md b/jekyll/_cci2/circleci-images.md deleted file mode 100644 index 9cc7d5b618..0000000000 --- a/jekyll/_cci2/circleci-images.md +++ /dev/null @@ -1,361 +0,0 @@ ---- -layout: classic-docs -title: "Convenience images" -description: "Listing of available Docker images maintained by CircleCI" -contentTags: - platform: - - Cloud - - Server v4+ ---- - - -**Legacy images with the prefix "circleci/" were [deprecated](https://discuss.circleci.com/t/legacy-convenience-image-deprecation/41034)** on December 31, 2021. For faster builds, upgrade your projects with [next-generation convenience images](https://circleci.com/blog/announcing-our-next-generation-convenience-images-smaller-faster-more-deterministic/). -{: class="alert alert-warning"} - -This document provides information about convenience images (pre-built Docker images maintained by CircleCI) and a listing by language, service type, and tags. - -## Overview -{: #overview } - -For convenience, CircleCI maintains several Docker images. These images are -typically extensions of official Docker images, and include tools especially -useful for CI/CD. - -This document provides an overview of best practices when using a convenience image. Please note that we advise using the **next-generation** convenience images (these start `cimg/`) rather than **legacy images**, as explained below. - -If you would like to directly search for an image, you can browse CircleCI Docker images in the following locations: - -- Visit the [Developer Hub](https://circleci.com/developer/images/) for links to all the repositories for each next-gen image. -- Find all CircleCI pre-built images available on [Docker Hub](https://hub.docker.com/u/cimg). - -_**Note:** CircleCI occasionally makes scheduled changes to images to fix bugs or -otherwise improve functionality, and these changes can sometimes affect -how images work in CircleCI jobs. Please follow the [**convenience-images** tag on -Discuss](https://discuss.circleci.com/tags/convenience-images) to be notified in advance of scheduled maintenance._ - -### Examples -{: #examples } - -Refer to the [Examples and Guides Overview]({{ site.baseurl }}/examples-and-guides-overview/) for examples of using pre-built CircleCI Docker Images in a demo application. - -## Next-generation convenience images -{: #next-generation-convenience-images } - -The next-generation convenience images in this section were built from the ground up with CI, efficiency, and determinism in mind. Here are some of the highlights: - -**Faster spin-up time** - In Docker terminology, these next-gen images will generally have fewer and smaller layers. Using these new images will lead to faster image downloads when a build starts, and a higher likelihood that the image is already cached on the host. - -**Improved reliability and stability** - The existing legacy convenience images are -rebuilt practically every day with potential changes from upstream that we cannot -always test fast enough. This leads to frequent breaking changes, which is not -the best environment for stable, deterministic builds. Next-gen images will only -be rebuilt for security and critical-bugs, leading to more stable and -deterministic images. - -### CircleCI base image -{: #circleci-base-image } -Using the `base` image in your config looks like the example shown below: - -```yaml -jobs: - myjob: - docker: - - image: cimg/base:2021.04 -``` - -{% include snippets/docker-auth.md %} - -This is an Ubuntu-based image designed to install the bare minimum. The -next-generation convenience images are based on this image. - -**When to use it?** - -If you need a generic image to run on CircleCI, to use with orbs, or to use as a -base for your own custom Docker image, this image is for you. - -**Resources** - -You can find more config examples for this image on the [Developer Hub](https://circleci.com/developer/images/image/cimg/base), and the source code and documentation on [GitHub](https://github.com/CircleCI-Public/cimg-base). - -The example below demonstrates how to use the next-gen Go image, which is based off the `base` image above. - -```yaml -jobs: - myjob: - docker: - - image: cimg/go:1.16 -``` - -This is a direct replacement for the legacy CircleCI Go image (`circleci/golang`). Note, the Docker Hub namespace is `cimg`. You can view other next generation images for other languages [below](#next-gen-language-images). - - -## Best practices -{: #best-practices } - -The next-gen convenience images in the following sections are based on the most recent Ubuntu LTS Docker images and installed with the base libraries for the language or services, so it is best practice to use the most specific image possible. This makes your builds more deterministic by preventing an upstream image from introducing unintended changes to your image. - -That is, to prevent unintended changes that come from upstream, instead of using `cimg/ruby:2.4-node` use a more specific version of these containers to ensure the image does not change with upstream changes until you change the tag. - -For example, pin down those images to a specific point version, like `cimg/ruby:2.4.10-node`. Specifying the version is possible for any of the CircleCI images. - -It is also possible to use the specific SHA of a image. For example, you can use -`cimg/ruby@sha256:e4aa60a0a47363eca3bbbb066620f0a5967370f6469f6831ad52231c87ca9390` -instead of `cimg/ruby:2.4.10-node`. Doing so allows you to test specific images -for as long as you would like before making any changes. - - -### Notes on pinning images -{: #notes-on-pinning-images } - -It is not recommended that you use the SHA for extended periods of time. If there is a major bug or security issue that requires a rebuild of the image, your pipeline's dependency on the image could inhibit you from acquiring the update that fixes that bug or patches a security issue. -{: class="alert alert-warning"} - -**Note:** If you are using a legacy image and you do not specify a tag, Docker -applies the `latest` tag. The `latest` tag refers to the most recent stable -release of an image. However, since this tag may change unexpectedly, it is best -practice to add an explicit image tag. - -**Note:** For Node.js variant Docker images (tags that end in `-node`) the LTS -release of Node.js is pre-installed. If you would like to include your own -specific version of Node.js / NPM you can set it up in a series of `run` steps -in your `.circleci/config.yml`. Consider the example below, which installs a -specific version of Node.js alongside the Ruby image. - -```yaml -version: 2.1 - -jobs: - build: - docker: - - image: cimg/ruby:2.7.1-node - steps: - - checkout - - run: - name: "Update Node.js and npm" - command: | - curl -sSL "https://nodejs.org/dist/v11.10.0/node-v11.10.0-linux-x64.tar.xz" | sudo tar --strip-components=2 -xJ -C /usr/local/bin/ node-v11.10.0-linux-x64/bin/node - curl https://www.npmjs.com/install.sh | sudo bash - - run: - name: Check current version of node - command: node -v -``` - -#### Finding an image id -{: #finding-an-image-id } - -Follow these steps to find your docker image id: - -1. In the CircleCI application, navigate to the job in your pipeline for which you would like to know the docker image. -2. Toggle open the **Spin up environment** step. -3. In the log output, locate the digest for the image. -4. Add the image ID to the image name as shown below. - -```shell -cimg/python@sha256:bdabda041f88d40d194c65f6a9e2a2e69ac5632db8ece657b15269700b0182cf -``` - -## Image types -{: #image-types } - -CircleCI's convenience images fall into two categories: -* **language** images -* **service** images - -All images add a `circleci` user as a system user. The sections below walk through the available next-generation and legacy images. - -### Next-gen language images -{: #next-gen-language-images } - -Next-gen language images are convenience images for common programming languages. -These images include relevant language tools and [commonly-used tools](#pre-installed-tools). -To use a language image, list it first under the `docker` key in your configuration, -making it the [primary container]({{ site.baseurl }}/glossary/#primary-container){:target="_blank"} during execution. - -See the [Developer Hub](https://circleci.com/developer/images?imageType=docker) for a full list of next-gen images. - -If your language is not listed, feel free to request an image on our [Ideas -Board](https://ideas.circleci.com/). First, check to see if that "idea" is -already on CircleCI Ideas. If it is, up-vote it. If not, create it and set the -category as "images". Finally, go and market your "idea" to friends, co-workers, -forums, and other communities in order to help it build traction. - -If we see an idea on the board take off, we will consider building it officially. - -#### Next-gen language image variants -{: #next-gen-language-image-variants } - -CircleCI maintains several variants for the next-gen language image. For -next-gen images be sure to check each image listing for information on each -variant. The `-browsers` variant for next-gen images is still in progress. See -each image listing on the [Developer Hub](https://circleci.com/developer/images/) -for details on which variants it supports. - -### Legacy language images -{: #legacy-language-images } - -The legacy language images are convenience images for common programming languages. -These images include both the relevant language and [commonly-used tools](#pre-installed-tools). -A language image should be listed first under the `docker` key in your configuration, -making it the [primary container]({{ site.baseurl }}/glossary/#primary-container){:target="_blank"} during execution. - -CircleCI maintains legacy images for the languages below. - -- [Android](#android) -- [Clojure](#clojure) -- [Elixir](#elixir) -- [Go (Golang)](#go-golang) -- [JRuby](#jruby) -- [Node.js](#nodejs) -- [OpenJDK (Java)](#openjdk) -- [PHP](#php) -- [Python](#python) -- [Ruby](#ruby) -- [Rust](#rust) - -#### Language image variants -{: #language-image-variants } - -CircleCI maintains several variants for language images. To use these variants, -add one of the following suffixes to the end of an image tag. - -- `-node` includes Node.js for polyglot applications -- `-browsers` includes Chrome, Firefox, OpenJDK v11, and Geckodriver -- `-node-browsers` combines the `-node` and `-browsers` variants - -For example, if you want to add browsers to the `circleci/golang:1.9` image, use -the `circleci/golang:1.9-browsers` image. - -### Next-Gen Service images -{: #next-gen-service-images } - -Service images are convenience images for services like databases. These images should be listed **after** language images so they become secondary service containers. - -- [Postgres](https://circleci.com/developer/images/image/cimg/postgres) -- [MySQL](https://circleci.com/developer/images/image/cimg/mysql) -- [MariaDB](https://circleci.com/developer/images/image/cimg/mariadb) -- [Redis](https://circleci.com/developer/images/image/cimg/redis) - -### Legacy Service images -{: #legacy-service-images } - -CircleCI maintains legacy images for the services below. - -- [buildpack-deps](#buildpack-deps) -- [DynamoDB](#dynamodb) -- [MariaDB](#mariadb) -- [MongoDB](#mongodb) -- [MySQL](#mysql) -- [PostgreSQL](#postgresql) -- [Redis](#redis) - -#### Service image variant -{: #service-image-variant } - -CircleCI maintains only one variant for service images. To speed up builds using -RAM volume, add the `-ram` suffix to the end of a service image tag. - -For example, if you want the `circleci/postgres:9.5-postgis` image to use RAM -volume, use the `circleci/postgres:9.5-postgis-ram` image. - -## Pre-installed tools -{: #pre-installed-tools } - -All convenience images have been extended with additional tools, installed with `apt-get`: - -- `bzip2` -- `ca-certificates` -- `curl` -- `git` -- `gnupg` -- `gzip` -- `locales` -- `mercurial` (legacy images only) -- `net-tools` -- `netcat` -- `openssh-client` -- `parallel` -- `sudo` -- `tar` -- `unzip` -- `wget` -- `xvfb` (legacy images only) -- `zip` - -The specific version of a particular package that gets installed in a particular -CircleCI image variant depends on the default version included in the package -directory for the Linux distribution/version installed in that variant's base -image. The legacy CircleCI convenience images are [Debian Jessie](https://packages.debian.org/jessie/)- -or [Stretch](https://packages.debian.org/stretch/)-based images, -however the next-gen images, `cimg`, extend the official [Ubuntu](https://packages.ubuntu.com) image. -For details on the next-gen images, see the [Developer Hub](https://circleci.com/developer/images/). Each image is tracked in its own repository. - -The following packages are pre-installed in convenience images using `curl` or other means. - -- [Docker client](https://docs.docker.com/install/) -- [Docker Compose](https://docs.docker.com/compose/overview/) -- [dockerize](https://github.com/jwilder/dockerize) -- [jq](https://stedolan.github.io/jq/) - - -## Out of scope -{: #out-of-scope } - -1. If an image is not listed above, it is not available. As the convenience image - program is revamped, proposals for new images are not currently being - accepted. -1. Old versions of software will not be rebuilt. Once an upstream image stops - building the tag for a specific release, say Node.js v8.1.0, then we stop - building it too. This means other tools in that image, such as `npm` in this - example, will no longer be updated either. -1. We do not support building preview, beta, or release candidate images tags. On - occasion they will be available but these tags tend to cause our build system - for convenience images to fail. If you need a non-stable release of a - language, we suggest installing it via [an orb](https://circleci.com/orbs/) - or a custom Docker image instead. - -## Legacy image tags by language -{: #legacy-image-tags-by-language } - -Below is a list of the latest **legacy** convenience images, sorted by language. - - -It is recommended to use next-generation images when possible. -For a list of the latest next-gen convenience images and -details about the content of each image, visit -the [Developer Hub](https://circleci.com/developer/). -{: class="alert alert-warning"} - -**Note:** Excluding [language image variants](#language-image-variants) and [the -service image variant](#service-image-variant), **for legacy images** CircleCI -does **not** control which tags are used. These tags are chosen and maintained -by upstream projects. Do not assume that a given tag has the same meaning across -images! - -{% assign images = site.data.docker-image-tags | sort %} -{% for image in images %} - -### {{ image[1].name }} -{: # {{image1name}} } - -**Resources:** - -- [Docker Hub](https://hub.docker.com/r/circleci/{{ image[0] }}) - where this image is hosted as well as some useful instructions. - -**Usage:** Add the following under `docker:` in your config.yml: - -`- image: circleci/{{ image[0] }}:[TAG]` - -**Recent Tags:** - -See [the tag list for circleci/{{ image[0] }} on Docker Hub](https://hub.docker.com/r/circleci/{{ image[0] }}/tags?ordering=last_updated). - ---- - -{% endfor %} - -## Next steps -{: #next-steps } - -- See [Using Docker Authenticated Pulls]({{ site.baseurl }}/private-images/) for information about how to authorize your build to use an image in a private repository or in Amazon ECR. -- For information about macOS images for iOS, see ({{ site.baseurl }}/testing-ios/). -- See [Running Docker Commands]({{ site.baseurl }}/building-docker-images/) for information about how to build Docker images. diff --git a/jekyll/_cci2/deploy-ios-applications.adoc b/jekyll/_cci2/deploy-ios-applications.adoc new file mode 100644 index 0000000000..8bcde3603e --- /dev/null +++ b/jekyll/_cci2/deploy-ios-applications.adoc @@ -0,0 +1,432 @@ +--- +contentTags: + platform: + - Cloud +--- += Deploy iOS applications +:page-description: Deploy iOS Applications +:experimental: +:icons: font +:page-layout: classic-docs +:page-liquid: + +In this how-to guide, you will learn how to configure link:https://fastlane.tools/[Fastlane] to automatically deploy iOS apps from CircleCI to a distribution service. + +[#introduction] +== Introduction + +Using Fastlane, CircleCI can automatically deploy iOS apps to various services. This helps remove the manual steps required to ship a beta or release version of an iOS app to its intended audience. + +Deployment _lanes_ can be combined with testing _lanes_ so that the app is automatically deployed upon a successful build and test. + +TIP: Using the deployment examples on this page requires that code signing is already configured for your project. To learn how to set up code signing, see the xref:ios-codesigning#[Setting up code signing] page. + +[#best-practices] +== Best practices + +[#use-git-branches] +=== Use Git branches + +It is advisable to only run your release lane on a specific branch of your git repository, for example, a dedicated release/beta branch. This will allow releases on only successful merges into the specified branch, and prevent a release every time a push is committed during your development phase. In turn this will also reduce job completion time, as uploading to an external service may take some time depending on the size of the iOS app binary. For information on how to set up a workflow to achieve this, refer to the xref:workflows#branch-level-job-execution[Branch-Level Job Execution] page. + +[#set-the-build-number] +=== Set the build number + +When uploading to a deployment service, it is important to consider the build number of the iOS app binary. Commonly this is set in the `.xcproject` and has to be updated manually to ensure it is unique. If the build number is not updated before each run of the deployment lane, you may find that the receiving service rejects the binary due to a build number conflict. + +Fastlane provides an link:https://docs.fastlane.tools/actions/increment_build_number/[`increment_build_number` action] which allows the build number to be modified during the lane execution. As an example, if you want to tie the build number to a particular CircleCI job, consider using the `$CIRCLE_BUILD_NUM` environment variable: + +[,ruby] +---- +increment_build_number( + build_number: "$CIRCLE_BUILD_NUM" +) +---- + +[#circleci-config-for-fastlane-integration] +== CircleCI configuration for Fastlane integration + +All the examples on this page use Fastlane to configure deployment. For each example the following example `.circleci/config.yml` configuration can be used to integrate your Fastlane setup with CircleCI. The example config should be edited to fit the needs of your project: + +NOTE: The environment variable `FL_OUTPUT_DIR` is the artifact directory where Fastlane logs, and a signed `.ipa` file should be stored. Use this to set the path in the `store_artifacts` step to automatically save logs and build artifacts from Fastlane. + +[,yaml] +---- +# .circleci/config.yml +version: 2.1 +jobs: + build-and-test: + macos: + xcode: 15.4.0 + environment: + FL_OUTPUT_DIR: output + FASTLANE_LANE: test + steps: + - checkout + - run: bundle install + - run: + name: Fastlane + command: bundle exec fastlane $FASTLANE_LANE + - store_artifacts: + path: output + - store_test_results: + path: output/scan + + adhoc: + macos: + xcode: 15.4.0 + environment: + FL_OUTPUT_DIR: output + FASTLANE_LANE: adhoc + steps: + - checkout + - run: bundle install + - run: + name: Fastlane + command: bundle exec fastlane $FASTLANE_LANE + - store_artifacts: + path: output + +workflows: + build-test-adhoc: + jobs: + - build-and-test + - adhoc: + filters: + branches: + only: development + requires: + - build-and-test +---- + +In this example, upon pushing to a development branch, the `adhoc` job enables producing a development build, or upload to TestFlight. + +[#set-up-app-store] +== Set up with App Store Connect + +To set up Fastlane to automatically upload iOS binaries to App Store Connect and/or TestFlight, a few steps need to be followed to allow Fastlane access to your App Store Connect account. + +The recommended way to set this up is to generate and use an App Store Connect API key. This prevents issues occurring with 2FA, which is now mandatory for Apple IDs, and is the most reliable way of interacting with the service. + +To create an API Key, follow the steps outlined in the link:https://developer.apple.com/documentation/appstoreconnectapi/creating_api_keys_for_app_store_connect_api[Apple Developer Documentation]. Once you have the resulting `.p8` file, make a note of the _Issuer ID_ and _Key ID_ which can be found on the link:https://appstoreconnect.apple.com/access/api[App Store Connect API Keys page]. + +TIP: Ensure you download the `.p8` file and store it somewhere safe. The file cannot be downloaded again once you navigate away from the App Store Connect portal. + +Next, a few environment variables need to be set. In your CircleCI project, navigate to *Build Settings \-> Environment Variables* and set the following: + +* `APP_STORE_CONNECT_API_KEY_ISSUER_ID` to the Issuer ID + ** Example: `6053b7fe-68a8-4acb-89be-165aa6465141` +* `APP_STORE_CONNECT_API_KEY_KEY_ID` to your Key ID + ** Example: `D383SF739` +* `APP_STORE_CONNECT_API_KEY_KEY` to the contents of your `.p8` file + ** Example: `-----BEGIN PRIVATE KEY-----\nMIGTAgEAMBMGByqGSM49AgEGCCqGSM49AwEHBHknlhdlYdLu\n-----END PRIVATE KEY-----` + +TIP: To find the contents of the `.p8` file, open it in a text editor. You will need to replace each new line with `\n` so that it forms one long string. + +Finally, Fastlane requires some information from us in order to know which Apple ID to use and which application we are targeting. The Apple ID and application bundle identifier can be set in the `fastlane/Appfile` as follows: + +[,ruby] +---- +# fastlane/Appfile +apple_id "ci@yourcompany.com" +app_identifier "com.example.HelloWorld" +---- + +If you need to use different credentials for App Store Connect and the Apple Developer Portal, check the link:https://docs.fastlane.tools/advanced/Appfile/[Fastlane Appfile documentation] for more details. + +Once this is configured, you just need to call link:https://docs.fastlane.tools/actions/app_store_connect_api_key/#app_store_connect_api_key[`app_store_connect_api_key`] in your lane before calling any actions that interact with App Store Connect (such as `pilot` and `deliver`). + +[#deploy-to-the-app-store] +=== 1. Deploy to the App Store + +The example below shows a basic lane to build, sign and upload a binary to App Store Connect. The link:https://docs.fastlane.tools/actions/deliver/#deliver/[`deliver` action] provided by Fastlane is a powerful tool that automates the App Store submission process. + +Deliver also allows various options such as automatic uploading of metadata and screenshots (which can be generated with the link:https://docs.fastlane.tools/actions/snapshot/[`snapshot`] and link:https://docs.fastlane.tools/actions/frameit/[frameit] actions). For further configuration, refer to the Fastlane link:https://docs.fastlane.tools/actions/deliver/[documentation for `deliver`]. + +[,ruby] +---- +# fastlane/Fastfile +default_platform :ios + +platform :ios do + before_all do + setup_circle_ci + end + + desc "Upload Release to App Store" + lane :upload_release do + # Get the version number from the project and check against + # the latest build already available on App Store Connect, then + # increase the build number by 1. If no build is available + # for that version, then start at 1 + increment_build_number( + build_number: app_store_build_number( + initial_build_number: 1, + version: get_version_number(xcodeproj: "HelloCircle.xcodeproj"), + live: false + ) + 1, + ) + # Set up Distribution code signing and build the app + match(type: "appstore") + gym(scheme: "HelloCircle") + # Upload the binary to App Store Connect + app_store_connect_api_key + deliver( + submit_for_review: false, + force: true + ) + end +end +---- + +[#deploy-to-testflight] +=== 2. Deploy to TestFlight + +TestFlight is Apple's beta distribution service which is tied into App Store Connect. Fastlane provides the link:https://docs.fastlane.tools/actions/pilot/[`pilot` action] to make managing TestFlight distribution simple. + +The example below shows how Fastlane can be configured to automatically build, sign and upload an iOS binary. Pilot has lots of customization options to help deliver apps to TestFlight, so it is highly recommended to check out the link:https://docs.fastlane.tools/actions/pilot/[`pilot` documentation] for further information. + +[,ruby] +---- +# fastlane/Fastfile +default_platform :ios + +platform :ios do + before_all do + setup_circle_ci + end + + desc "Upload to Testflight" + lane :upload_testflight do + # Get the version number from the project and check against + # the latest build already available on TestFlight, then + # increase the build number by 1. If no build is available + # for that version, then start at 1 + increment_build_number( + build_number: latest_testflight_build_number( + initial_build_number: 1, + version: get_version_number(xcodeproj: "HelloWorld.xcodeproj") + ) + 1, + ) + # Set up Distribution code signing and build the app + match(type: "appstore") + gym(scheme: "HelloWorld") + # Upload the binary to TestFlight and automatically publish + # to the configured beta testing group + app_store_connect_api_key + pilot( + distribute_external: true, + notify_external_testers: true, + groups: ["HelloWorld Beta Testers"], + changelog: "This is another new build from CircleCI!" + ) + end +end +---- + +[#deploying-to-firebase] +== Deploy to Firebase + +Firebase is a distribution service from Google. Deploying to Firebase is simplified by installing the link:https://github.com/fastlane/fastlane-plugin-firebase_app_distribution[Firebase app distribution plugin]. + +[#fastlane-plugin-setup] +=== 1. Fastlane plugin setup + +To set up the plugin for your project, on your local machine, open your project directory in Terminal and run the following command: + +[,bash] +---- +fastlane add_plugin firebase_app_distribution +---- + +This will install the plugin and add the required information to `fastlane/Pluginfile` and your `Gemfile`. + +NOTE: It is important that both of these files are checked into your git repo so that this plugin can be installed by CircleCI during the job execution via a `bundle install` step. + +[#generate-a-cli-token] +=== 2. Generate a CLI Token + +Firebase requires a token to used during authentication. To generate the token, we need to use the Firebase CLI and a browser - as CircleCI is a headless environment, we will need to generate this token locally, rather than at runtime, then add it to CircleCI as an environment variable. + +. Download and install the Firebase CLI locally with the command `+curl -sL https://firebase.tools | bash+`. +. Trigger a login by using the command `firebase login:ci`. +. Complete the sign in via the browser window, then copy the token provided in the Terminal output. +. Go to your project settings in CircleCI and create a new environment variable named `FIREBASE_TOKEN` with the value of the token. + +[#fastlane-configuration] +=== 3. Fastlane configuration + +The Firebase plugin requires minimal configuration to upload an iOS binary to Firebase. The main parameter is `app` which will require the App ID set by Firebase. To find this, go to your project in the link:https://console.firebase.google.com[Firebase Console], then go to *Project Settings \-> General*. Under *Your apps*, you will see the list of apps that are part of the project and their information, including the App ID (generally in the format of `1:123456789012:ios:abcd1234abcd1234567890`). + +For more configuration options, see the link:https://firebase.google.com/docs/app-distribution/ios/distribute-fastlane#step_3_set_up_your_fastfile_and_distribute_your_app[Firebase Fastlane plugin documentation]. + +[,ruby] +---- +# Fastlane/fastfile +default_platform :ios + +platform :ios do + before_all do + setup_circle_ci + end + + desc "Upload to Firebase" + lane :upload_firebase do + increment_build_number( + build_number: "$CIRCLE_BUILD_NUM" + ) + match(type: "adhoc") + gym(scheme: "HelloWorld") + firebase_app_distribution( + app: "1:123456789012:ios:abcd1234abcd1234567890", + release_notes: "This is a test release!" + ) + end +end +---- + +To use the Firebase Fastlane plugin, the Firebase CLI must be installed as part of the job via the `+curl -sL https://firebase.tools | bash+` command: + +[,yaml] +---- +version: 2.1 +jobs: + adhoc: + macos: + xcode: "12.5.1" + environment: + FL_OUTPUT_DIR: output + steps: + - checkout + - run: echo 'chruby ruby-2.6' >> ~/.bash_profile + - run: bundle install + - run: curl -sL https://firebase.tools | bash + - run: bundle exec fastlane upload_firebase + +workflows: + adhoc-build: + jobs: + - adhoc +---- + +NOTE: The Firebase plugin may cause errors when run with the macOS system Ruby. It is therefore advisable to xref:testing-ios#using-ruby[switch to a different Ruby version]. + +[#deploy-to-visual-studio-app-center] +== Deploy to Visual Studio App Center + +link:https://appcenter.ms/[Visual Studio App Center] (formally HockeyApp), is a distribution service from Microsoft. App Center integration with Fastlane is enabled by installing the link:https://github.com/microsoft/fastlane-plugin-appcenter[App Center plugin]. + +[#fastlane-plugin-setup] +=== 1. Fastlane plugin setup + +To set up the plugin for your project, on your local machine open your project directory in Terminal and run the following command: + +[,bash] +---- +fastlane add_plugin appcenter +---- + +This will install the plugin and add the required information to `fastlane/Pluginfile` and your `Gemfile`. + +NOTE: It is important that both of these files are checked into your git repo so that this plugin can be installed by CircleCI during the job execution via a `bundle install` step. + +[#app-center-setup] +=== 2. App center setup + +First, the app needs to be created in VS App Center. + +. Log in, or sign up, to link:https://appcenter.ms/[Visual Studio App Center]. +. At the top-right of the page, click on "Add New", then select "Add New App". +. Fill out the required information in the form as required. + +Once this is complete you will need to generate an API token to allow Fastlane to upload to App Center. + +. Go to the link:https://appcenter.ms/settings/apitokens[API Tokens] section in Settings. +. Select "New API Token". +. Enter a description for the token, then set the access to "Full Access". +. When the token is generated, make sure to copy it somewhere safe. +. Go to your project settings in CircleCI and create a new environment variable named `VS_API_TOKEN` with the value of the API Key. + +[#fastlane-configuration-vs-app] +=== 3. Fastlane configuration + +Below is an example of a lane that distributes beta app builds to Visual Studio App Center. Both the username of your App Center account and an API Token with "Full Access" is required to upload the binary to App Center. + +[,ruby] +---- +# Fastlane/fastfile +default_platform :ios + +platform :ios do + before_all do + setup_circle_ci + end + +desc "Upload to VS App Center" + lane :upload_appcenter do + # Here we are using the CircleCI job number + # for the build number + increment_build_number( + build_number: "$CIRCLE_BUILD_NUM" + ) + # Set up Adhoc code signing and build the app + match(type: "adhoc") + gym(scheme: "HelloWorld") + # Set up the required information to upload the + # app binary to VS App Center + appcenter_upload( + api_token: ENV[VS_API_TOKEN], + owner_name: "YOUR_VS_APPCENTER_USERNAME", + owner_type: "user", + app_name: "HelloWorld" + ) + end +end +---- + +[#upload-to-testfairy] +== Upload to TestFairy + +link:https://www.testfairy.com[TestFairy] is another popular Enterprise App distribution and testing service. Fastlane has built-in support for TestFairy, making it simple to upload new builds to the service. + +image::{{site.baseurl}}/assets/img/docs/testfairy-open-preferences.png[TestFairy preferences image] + +. On the TestFairy dashboard, navigate to the Preferences page. +. On the Preferences page, go to the API Key section and copy your API Key. +. Go to your project settings in CircleCI and create a new environment variable named `TESTFAIRY_API_KEY` with the value of the API Key + +[#fastlane-configuration] +=== Fastlane configuration + +To configure uploading to TestFairy within Fastlane, see the following example: + +[,ruby] +---- +# Fastlane/fastfile +default_platform :ios + +platform :ios do + before_all do + setup_circle_ci + end + +desc "Upload to TestFairy" + lane :upload_testfairy do + # Here we are using the CircleCI job number + # for the build number + increment_build_number( + build_number: "$CIRCLE_BUILD_NUM" + ) + # Set up Adhoc code signing and build the app + match(type: "adhoc") + gym(scheme: "HelloWorld") + # Set up the required information to upload the + # app binary to VS App Center + testfairy( + api_key: ENV[TESTFAIRY_API_KEY], + ipa: 'path/to/ipafile.ipa', + comment: ENV[CIRCLE_BUILD_URL] + ) + end +end +---- diff --git a/styles/config/vocabularies/Docs/accept.txt b/styles/config/vocabularies/Docs/accept.txt index 88974cf1e5..47a45315e4 100644 --- a/styles/config/vocabularies/Docs/accept.txt +++ b/styles/config/vocabularies/Docs/accept.txt @@ -31,7 +31,9 @@ Android AMI APIs? \b[Aa]pp\b +Appfile Appium +\bApp Store Connect\b Arm Artifactory [Aa]rchitected @@ -102,6 +104,7 @@ DNS Docker Docker\sHub Dockerfiles? +dockerize [Dd]otfile [Ee]nablement EKS @@ -119,6 +122,7 @@ Fastlane /bFastlane Match/b filename framebuffer +frameit \bFree Plan\b [Ff]ormatters? Gbps @@ -150,6 +154,7 @@ IAM IdP [Ii]nsights invocated +iOS IP Java JavaScript @@ -157,6 +162,7 @@ JavaScript Jira jq\b [JU|ju]nit +JRuby JVM kaniko [Kk]aocha @@ -271,6 +277,7 @@ Telegraf templated [Tt]emplatizable Terraform +TestFairy Tink TLS [Tt]olerations From 8fc9bdfbf4d507b7b3331fe4fba8a731629f8ece Mon Sep 17 00:00:00 2001 From: Rosie Yohannan Date: Mon, 27 Jan 2025 13:30:46 +0000 Subject: [PATCH 03/20] fix images page fornmatting --- jekyll/_cci2/circleci-images.adoc | 4 +- jekyll/_cci2/deploy-ios-applications.md | 423 ------------------------ 2 files changed, 1 insertion(+), 426 deletions(-) delete mode 100644 jekyll/_cci2/deploy-ios-applications.md diff --git a/jekyll/_cci2/circleci-images.adoc b/jekyll/_cci2/circleci-images.adoc index 5aeffa9413..3aa50db405 100644 --- a/jekyll/_cci2/circleci-images.adoc +++ b/jekyll/_cci2/circleci-images.adoc @@ -332,8 +332,6 @@ images! === {{ image[1].name }} -{: # {\{image1name}} } - *Resources:* * https://hub.docker.com/r/circleci/{{ image[0] }}[Docker Hub] - where this image is hosted as well as some useful instructions. @@ -344,7 +342,7 @@ images! *Recent Tags:* -See https://hub.docker.com/r/circleci/{{ image[0] }}/tags?ordering=last_updated[the tag list for circleci/{{ image[0\] }} on Docker Hub]. +See https://hub.docker.com/r/circleci/{{ image[0] }}/tags?ordering=last_updated[the tag list for circleci/{{ image[0] }} on Docker Hub]. ''' diff --git a/jekyll/_cci2/deploy-ios-applications.md b/jekyll/_cci2/deploy-ios-applications.md deleted file mode 100644 index f019250d31..0000000000 --- a/jekyll/_cci2/deploy-ios-applications.md +++ /dev/null @@ -1,423 +0,0 @@ ---- -layout: classic-docs -title: Deploy iOS applications -description: Deploy iOS Applications -redirect-from: /deploying-ios -contentTags: - platform: - - Cloud ---- - -In this how-to guide, you will learn how to to configure [Fastlane](https://fastlane.tools/) to automatically deploy iOS apps from CircleCI to a distribution service. - -## Introduction -{: #introduction } - -Using Fastlane, CircleCI can automatically deploy iOS apps to various services. This helps remove the manual steps required to ship a beta or release version of an iOS app to its intended audience. - -Deployment _lanes_ can be combined with testing _lanes_ so that the app is automatically deployed upon a successful build and test. - -Using the deployment examples on this page requires that code signing is already configured for your project. To learn how to set up code signing, see the [Setting up code signing]({{site.baseurl}}/ios-codesigning/) page. -{: class="alert alert-note"} - -## Best practices -{: #best-practices } - -### Use Git branches -{: #use-git-branches } - -It is advisable to only run your release lane on a specific branch of your git repository, for example, a dedicated release/beta branch. This will allow releases on only successful merges into the specified branch, and prevent a release every time a push is committed during your development phase. In turn this will also reduce job completion time, as uploading to an external service may take some time depending on the size of the iOS app binary. For information on how to set up a workflow to achieve this, refer to the [Branch-Level Job Execution]({{site.baseurl}}/workflows/#branch-level-job-execution) page. - -### Set the build number -{: #set-the-build-number } - -When uploading to a deployment service, it is important to consider the build number of the iOS app binary. Commonly this is set in the `.xcproject` and has to be updated manually to ensure it is unique. If the build number is not updated before each run of the deployment lane, you may find that the receiving service rejects the binary due to a build number conflict. - -Fastlane provides an [`increment_build_number` action](https://docs.fastlane.tools/actions/increment_build_number/) which allows the build number to be modified during the lane execution. As an example, if you want to tie the build number to a particular CircleCI job, consider using the `$CIRCLE_BUILD_NUM` environment variable: - -```ruby -increment_build_number( - build_number: "$CIRCLE_BUILD_NUM" -) -``` - -## CircleCI configuration for Fastlane integration -{: #circleci-config-for-fastlane-integration } - -All the examples on this page use Fastlane to configure deployment. For each example the following example `.circleci/config.yml` configuration can be used to integrate your Fastlane setup with CircleCI. This is an example config which should be edited to fit the needs of your project: - -The environment variable `FL_OUTPUT_DIR` is the artifact directory where FastLane logs, and a signed `.ipa` file should be stored. Use this to set the path in the `store_artifacts` step to automatically save logs and build artifacts from Fastlane. -{: class="alert alert-note"} - -```yaml -# .circleci/config.yml -version: 2.1 -jobs: - build-and-test: - macos: - xcode: 15.4.0 - environment: - FL_OUTPUT_DIR: output - FASTLANE_LANE: test - steps: - - checkout - - run: bundle install - - run: - name: Fastlane - command: bundle exec fastlane $FASTLANE_LANE - - store_artifacts: - path: output - - store_test_results: - path: output/scan - - adhoc: - macos: - xcode: 15.4.0 - environment: - FL_OUTPUT_DIR: output - FASTLANE_LANE: adhoc - steps: - - checkout - - run: bundle install - - run: - name: Fastlane - command: bundle exec fastlane $FASTLANE_LANE - - store_artifacts: - path: output - -workflows: - build-test-adhoc: - jobs: - - build-and-test - - adhoc: - filters: - branches: - only: development - requires: - - build-and-test -``` - -In this example, upon pushing to a development branch, the `adhoc` job enables producing a development build, or upload to Testflight. - - -## Set up with App Store Connect -{: #set-up-app-store } - -To set up Fastlane to automatically upload iOS binaries to App Store Connect and/or TestFlight, a few steps need to be followed to allow Fastlane access to your App Store Connect account. - -The recommended way to set this up is to generate and use an App Store Connect API key. This prevents issues occurring with 2FA, which is now mandatory for Apple IDs, and is the most reliable way of interacting with the service. - -To create an API Key, follow the steps outlined in the [Apple Developer Documentation](https://developer.apple.com/documentation/appstoreconnectapi/creating_api_keys_for_app_store_connect_api). Once you have the resulting `.p8` file, make a note of the *Issuer ID* and *Key ID* which can be found on the [App Store Connect API Keys page](https://appstoreconnect.apple.com/access/api). - -Ensure you download the `.p8` file and store it somewhere safe. The file cannot be downloaded again once you navigate away from the App Store Connect portal. -{: class="alert alert-info" } - -Next, a few environment variables need to be set. In your CircleCI project, navigate to **Build Settings -> Environment Variables** and set the following: - -* `APP_STORE_CONNECT_API_KEY_ISSUER_ID` to the Issuer ID - * Example: `6053b7fe-68a8-4acb-89be-165aa6465141` -* `APP_STORE_CONNECT_API_KEY_KEY_ID` to your Key ID - * Example: `D383SF739` -* `APP_STORE_CONNECT_API_KEY_KEY` to the contents of your `.p8` file - * Example: `-----BEGIN PRIVATE KEY-----\nMIGTAgEAMBMGByqGSM49AgEGCCqGSM49AwEHBHknlhdlYdLu\n-----END PRIVATE KEY-----` - -To find the contents of the `.p8` file, open it in a text editor. You will need to replace each new line with `\n` so that it forms one long string. -{: class="alert alert-info" } - -Finally, Fastlane requires some information from us in order to know which Apple ID to use and which application we are targeting. The Apple ID and application bundle identifier can be set in the `fastlane/Appfile` as follows: - -```ruby -# fastlane/Appfile -apple_id "ci@yourcompany.com" -app_identifier "com.example.HelloWorld" -``` - -If you need to use different credentials for App Store Connect and the Apple Developer Portal, check the [Fastlane Appfile documentation](https://docs.fastlane.tools/advanced/Appfile/) for more details. - -Once this is configured, you just need to call [`app_store_connect_api_key`](https://docs.fastlane.tools/actions/app_store_connect_api_key/#app_store_connect_api_key) in your lane before calling any actions that interact with App Store Connect (such as `pilot` and `deliver`). - -### 1. Deploy to the App Store -{: #deploy-to-the-app-store } - -The example below shows a basic lane to build, sign and upload a binary to App Store Connect. The [`deliver` action](https://docs.fastlane.tools/actions/deliver/#deliver/) provided by Fastlane is a powerful tool that automates the App Store submission process. - -Deliver also allows various options such as automatic uploading of metadata and screenshots (which can be generated with the [`snapshot`](https://docs.fastlane.tools/actions/snapshot/) and [frameit](https://docs.fastlane.tools/actions/frameit/) actions). For further configuration, refer to the Fastlane [documentation for `deliver`](https://docs.fastlane.tools/actions/deliver/). - -```ruby -# fastlane/Fastfile -default_platform :ios - -platform :ios do - before_all do - setup_circle_ci - end - - desc "Upload Release to App Store" - lane :upload_release do - # Get the version number from the project and check against - # the latest build already available on App Store Connect, then - # increase the build number by 1. If no build is available - # for that version, then start at 1 - increment_build_number( - build_number: app_store_build_number( - initial_build_number: 1, - version: get_version_number(xcodeproj: "HelloCircle.xcodeproj"), - live: false - ) + 1, - ) - # Set up Distribution code signing and build the app - match(type: "appstore") - gym(scheme: "HelloCircle") - # Upload the binary to App Store Connect - app_store_connect_api_key - deliver( - submit_for_review: false, - force: true - ) - end -end -``` - -### 2. Deploy to TestFlight -{: #deploy-to-testflight } - -TestFlight is Apple's beta distribution service which is tied into App Store Connect. Fastlane provides the [`pilot` action](https://docs.fastlane.tools/actions/pilot/) to make managing TestFlight distribution simple. - -The example below shows how Fastlane can be configured to automatically build, sign and upload an iOS binary. Pilot has lots of customization options to help deliver apps to TestFlight, so it is highly recommended to check out the [`pilot` documentation](https://docs.fastlane.tools/actions/pilot/) for further information. - -```ruby -# fastlane/Fastfile -default_platform :ios - -platform :ios do - before_all do - setup_circle_ci - end - - desc "Upload to Testflight" - lane :upload_testflight do - # Get the version number from the project and check against - # the latest build already available on TestFlight, then - # increase the build number by 1. If no build is available - # for that version, then start at 1 - increment_build_number( - build_number: latest_testflight_build_number( - initial_build_number: 1, - version: get_version_number(xcodeproj: "HelloWorld.xcodeproj") - ) + 1, - ) - # Set up Distribution code signing and build the app - match(type: "appstore") - gym(scheme: "HelloWorld") - # Upload the binary to TestFlight and automatically publish - # to the configured beta testing group - app_store_connect_api_key - pilot( - distribute_external: true, - notify_external_testers: true, - groups: ["HelloWorld Beta Testers"], - changelog: "This is another new build from CircleCI!" - ) - end -end -``` - -## Deploy to Firebase -{: #deploying-to-firebase } - -Firebase is a distribution service from Google. Deploying to Firebase is simplified by installing the [Firebase app distribution plugin](https://github.com/fastlane/fastlane-plugin-firebase_app_distribution). - -### 1. Fastlane Plugin Setup -{: #fastlane-plugin-setup } - -To set up the plugin for your project, on your local machine, open your project directory in Terminal and run the following command: - -```bash -fastlane add_plugin firebase_app_distribution -``` - -This will install the plugin and add the required information to `fastlane/Pluginfile` and your `Gemfile`. - -It is important that both of these files are checked into your git repo so that this plugin can be installed by CircleCI during the job execution via a `bundle install` step. -{: class="alert alert-info" } - -### 2. Generate a CLI Token -{: #generate-a-cli-token } - -Firebase requires a token to used during authentication. To generate the token, we need to use the Firebase CLI and a browser - as CircleCI is a headless environment, we will need to generate this token locally, rather than at runtime, then add it to CircleCI as an environment variable. - -1. Download and install the Firebase CLI locally with the command `curl -sL https://firebase.tools | bash`. -2. Trigger a login by using the command `firebase login:ci`. -3. Complete the sign in via the browser window, then copy the token provided in the Terminal output. -4. Go to your project settings in CircleCI and create a new environment variable named `FIREBASE_TOKEN` with the value of the token. - -### 3. Fastlane configuration -{: #fastlane-configuration } - -The Firebase plugin requires minimal configuration to upload an iOS binary to Firebase. The main parameter is `app` which will require the App ID set by Firebase. To find this, go to your project in the [Firebase Console](https://console.firebase.google.com), then go to **Project Settings -> General**. Under **Your apps**, you will see the list of apps that are part of the project and their information, including the App ID (generally in the format of `1:123456789012:ios:abcd1234abcd1234567890`). - -For more configuration options, see the [Firebase Fastlane plugin documentation](https://firebase.google.com/docs/app-distribution/ios/distribute-fastlane#step_3_set_up_your_fastfile_and_distribute_your_app). - -```ruby -# Fastlane/fastfile -default_platform :ios - -platform :ios do - before_all do - setup_circle_ci - end - - desc "Upload to Firebase" - lane :upload_firebase do - increment_build_number( - build_number: "$CIRCLE_BUILD_NUM" - ) - match(type: "adhoc") - gym(scheme: "HelloWorld") - firebase_app_distribution( - app: "1:123456789012:ios:abcd1234abcd1234567890", - release_notes: "This is a test release!" - ) - end -end -``` - -To use the Firebase Fastlane plugin, the Firebase CLI must be installed as part of the job via the `curl -sL https://firebase.tools | bash` command: - -```yaml -version: 2.1 -jobs: - adhoc: - macos: - xcode: "12.5.1" - environment: - FL_OUTPUT_DIR: output - steps: - - checkout - - run: echo 'chruby ruby-2.6' >> ~/.bash_profile - - run: bundle install - - run: curl -sL https://firebase.tools | bash - - run: bundle exec fastlane upload_firebase - -workflows: - adhoc-build: - jobs: - - adhoc -``` - -**Note:** The Firebase plugin may cause errors when run with the macOS system Ruby. It is therefore advisable to [switch to a different Ruby version]({{site.baseurl}}/testing-ios/#using-ruby). - -## Deploy to Visual Studio App Center -{: #deploy-to-visual-studio-app-center } - -[Visual Studio App Center](https://appcenter.ms/) (formally HockeyApp), is a distribution service from Microsoft. App Center integration with Fastlane is enabled by installing the [App Center plugin](https://github.com/microsoft/fastlane-plugin-appcenter). - -### 1. Fastlane Plugin Setup -{: #fastlane-plugin-setup } - -To set up the plugin for your project, on your local machine open your project directory in Terminal and run the following command: -```bash -fastlane add_plugin appcenter -``` - This will install the plugin and add the required information to `fastlane/Pluginfile` and your `Gemfile`. - -**Note:** It is important that both of these files are checked into your git repo so that this plugin can be installed by CircleCI during the job execution via a `bundle install` step. - -### 2. App Center setup -{: #app-center-setup } - -First, the app needs to be created in VS App Center. - -1. Log in, or sign up, to [Visual Studio App Center](https://appcenter.ms/). -2. At the top-right of the page, click on "Add New", then select "Add New App". -3. Fill out the required information in the form as required. - -Once this is complete you will need to generate an API token to allow Fastlane to upload to App Center. - -1. Go to the [API Tokens](https://appcenter.ms/settings/apitokens) section in Settings. -2. Click on "New API Token". -3. Enter a description for the token, then set the access to "Full Access". -4. When the token is generated, make sure to copy it somewhere safe. -5. Go to your project settings in CircleCI and create a new environment variable named `VS_API_TOKEN` with the value of the API Key. - -### 3. Fastlane configuration -{: #fastlane-configuration-vs-app } - -Below is an example of a lane that distributes beta app builds to Visual Studio App Center. Both the username of your App Center account and an API Token with "Full Access" is required to upload the binary to App Center. - -```ruby -# Fastlane/fastfile -default_platform :ios - -platform :ios do - before_all do - setup_circle_ci - end - -desc "Upload to VS App Center" - lane :upload_appcenter do - # Here we are using the CircleCI job number - # for the build number - increment_build_number( - build_number: "$CIRCLE_BUILD_NUM" - ) - # Set up Adhoc code signing and build the app - match(type: "adhoc") - gym(scheme: "HelloWorld") - # Set up the required information to upload the - # app binary to VS App Center - appcenter_upload( - api_token: ENV[VS_API_TOKEN], - owner_name: "YOUR_VS_APPCENTER_USERNAME", - owner_type: "user", - app_name: "HelloWorld" - ) - end -end -``` - -## Upload to TestFairy -{: #upload-to-testfairy } - -[TestFairy](https://www.testfairy.com) is another popular Enterprise App distribution and testing service. Fastlane has built-in support for TestFairy, making it quick and easy to upload new builds to the service. - -![TestFairy preferences image]({{site.baseurl}}/assets/img/docs/testfairy-open-preferences.png) - -1. On the TestFairy dashboard, navigate to the Preferences page. -2. On the Preferences page, go to the API Key section and copy your API Key. -3. Go to your project settings in CircleCI and create a new environment variable named `TESTFAIRY_API_KEY` with the value of the API Key - -### Fastlane configuration -{: #fastlane-configuration } - -To configure uploading to TestFairy within Fastlane, see the following example: - -```ruby -# Fastlane/fastfile -default_platform :ios - -platform :ios do - before_all do - setup_circle_ci - end - -desc "Upload to TestFairy" - lane :upload_testfairy do - # Here we are using the CircleCI job number - # for the build number - increment_build_number( - build_number: "$CIRCLE_BUILD_NUM" - ) - # Set up Adhoc code signing and build the app - match(type: "adhoc") - gym(scheme: "HelloWorld") - # Set up the required information to upload the - # app binary to VS App Center - testfairy( - api_key: ENV[TESTFAIRY_API_KEY], - ipa: 'path/to/ipafile.ipa', - comment: ENV[CIRCLE_BUILD_URL] - ) - end -end -``` From 6cfb9fc54c6bbe5d0d227fc1c43e6a4fece7f14c Mon Sep 17 00:00:00 2001 From: Rosie Yohannan Date: Mon, 27 Jan 2025 13:43:07 +0000 Subject: [PATCH 04/20] convert deploy to artifactory guide --- ...ifactory.md => deploy-to-artifactory.adoc} | 90 ++++++++++--------- styles/config/vocabularies/Docs/accept.txt | 1 + 2 files changed, 50 insertions(+), 41 deletions(-) rename jekyll/_cci2/{deploy-to-artifactory.md => deploy-to-artifactory.adoc} (64%) diff --git a/jekyll/_cci2/deploy-to-artifactory.md b/jekyll/_cci2/deploy-to-artifactory.adoc similarity index 64% rename from jekyll/_cci2/deploy-to-artifactory.md rename to jekyll/_cci2/deploy-to-artifactory.adoc index 6893d0b745..c707f2f1ff 100644 --- a/jekyll/_cci2/deploy-to-artifactory.md +++ b/jekyll/_cci2/deploy-to-artifactory.adoc @@ -1,63 +1,69 @@ --- -layout: classic-docs -title: Deploy to Artifactory -description: How to upload artifacts to Artifactory in CircleCI contentTags: platform: - Cloud - Server v4+ --- += Deploy to Artifactory +:page-description: How to upload artifacts to Artifactory in CircleCI +:experimental: +:icons: font +:page-layout: classic-docs +:page-liquid: In this how-to guide, you will learn how to upload artifacts to Artifactory in CircleCI. -## Introduction -{: #introduction } +[#introduction] +== Introduction -Artifactory has documentation explaining how to use their [REST API](https://www.jfrog.com/confluence/display/RTF/Artifactory+REST+API). +Artifactory has documentation explaining how to use their link:https://www.jfrog.com/confluence/display/RTF/Artifactory+REST+API[REST API]. Below are some sample projects showing how to best use CircleCI and Artifactory together. Ensure that you have created your repository before starting this example, otherwise CircleCI will not have a place to store your dependencies. -## Artifactory plugins -{: #artifactory-plugins } +[#artifactory-plugins] +== Artifactory plugins Popular tools like Maven and Gradle have Artifactory plugins, and can deploy to Artifactory using their respective deploy commands. -- [Deploying with Maven](https://jfrog.com/help/r/jfrog-integrations-documentation/ecosystem-integration-maven-artifactory-plugin) -- [Deploying with Gradle](https://www.jfrog.com/confluence/display/RTF/Gradle+Artifactory+Plugin) +* link:https://jfrog.com/help/r/jfrog-integrations-documentation/ecosystem-integration-maven-artifactory-plugin[Deploying with Maven] +* link:https://www.jfrog.com/confluence/display/RTF/Gradle+Artifactory+Plugin[Deploying with Gradle] -## JFrog CLI -{: #jfrog-cli } +[#jfrog-cli] +== JFrog CLI -If you want to use the [JFrog CLI](https://docs.jfrog-applications.jfrog.io/jfrog-applications/jfrog-cli), you can install it by following the steps below. +If you want to use the link:https://docs.jfrog-applications.jfrog.io/jfrog-applications/jfrog-cli[JFrog CLI], you can install it by following the steps below. -### 1. Add JFrog to your configuration -{: #add-jfrog-to-your-configuration } +[#add-jfrog-to-your-configuration] +=== 1. Add JFrog to your configuration -There are a number of methods available for [installing JFrog](https://docs.jfrog-applications.jfrog.io/jfrog-applications/jfrog-cli/install#installation). Please refer to their documentation for the method best suited for your pipeline. +You can install JFrog CLI in one of a few ways. Refer to the link:https://docs.jfrog-applications.jfrog.io/jfrog-applications/jfrog-cli/install#installation[JFrog documentation]. for the method best suited for your pipeline. -For example if using Node, add the following to your `.circleci/config.yml`: +For example, if using Node, add the following to your `.circleci/config.yml`: -```yml +[,yml] +---- - run: name: Install jFrog CLI command: npm install -g jfrog-cli-v2-jf +---- -``` -### 2. Configure credentials -{: #configure-credentials } +[#configure-credentials] +=== 2. Configure credentials -Now you need to configure JFrog to use CircleCI credentials securely. CircleCI configures the client to use `$ARTIFACTORY_URL`, along with `$ARTIFACTORY_USER` and `$ARTIFACTORY_APIKEY`. These can be entered under `Project Settings->Environment Variables`. Configure the CLI to use these settings: +Now you need to configure JFrog to use CircleCI credentials securely. CircleCI configures the client to use `$ARTIFACTORY_URL`, along with `$ARTIFACTORY_USER` and `$ARTIFACTORY_APIKEY`. These can be entered under `+Project Settings->Environment Variables+`. Configure the CLI to use these settings: -```yml +[,yml] +---- - run: ./jf config add --artifactory-url $ARTIFACTORY_URL --user $ARTIFACTORY_USER --password $ARTIFACTORY_APIKEY --interactive=false -``` +---- -### 3. Upload JAR files (optional) -{: #upload-jar-files } +[#upload-jar-files] +=== 3. Upload JAR files (optional) If you would like to upload JAR files use the following example: -```yml +[,yml] +---- command: | ./jf mvnc \ --server-id-resolve \ @@ -69,14 +75,15 @@ command: | --include-patterns "*.jar, *.pom, *.xml" /.jf mvn clean install -``` +---- -### 4. Upload WAR files (optional) -{: #upload-war-files } +[#upload-war-files] +=== 4. Upload WAR files (optional) If you would like to upload WAR files use the following example: -```yml +[,yml] +---- command: | ./jf mvnc \ --server-id-resolve \ @@ -88,16 +95,17 @@ command: | --include-patterns "*.war" /.jf mvn clean install -``` +---- -## Full configuration example -{: #full-configuration-example } +[#full-configuration-example] +== Full configuration example The full `.circleci/config.yml` file would look something like the following: -{% include snippets/docker-auth.md %} +{% include snippets/docker-auth.adoc %} -```yml +[,yml] +---- version: 2.1 jobs: create-build-package: @@ -136,10 +144,10 @@ workflows: jobs: - create-build-package: context: artifactory -``` +---- -## See also -{: #see-also } +[#see-also] +== See also -- [Storing and Accessing Artifacts](/docs/artifacts/) -- [CircleCI and JFrog](https://circleci.com/circleci-and-jfrog/) +* xref:artifacts#[Storing and Accessing Artifacts] +* link:https://circleci.com/circleci-and-jfrog/[CircleCI and JFrog] diff --git a/styles/config/vocabularies/Docs/accept.txt b/styles/config/vocabularies/Docs/accept.txt index 47a45315e4..4a9cd9cc5d 100644 --- a/styles/config/vocabularies/Docs/accept.txt +++ b/styles/config/vocabularies/Docs/accept.txt @@ -159,6 +159,7 @@ IP Java JavaScript [Jj]est +JFrog Jira jq\b [JU|ju]nit From e5c188c7a674e104027b30537b1b0e56d3ceef2a Mon Sep 17 00:00:00 2001 From: Rosie Yohannan Date: Mon, 27 Jan 2025 14:05:45 +0000 Subject: [PATCH 05/20] convert docker-compose page --- ...{docker-compose.md => docker-compose.adoc} | 74 ++++++++++--------- 1 file changed, 40 insertions(+), 34 deletions(-) rename jekyll/_cci2/{docker-compose.md => docker-compose.adoc} (50%) diff --git a/jekyll/_cci2/docker-compose.md b/jekyll/_cci2/docker-compose.adoc similarity index 50% rename from jekyll/_cci2/docker-compose.md rename to jekyll/_cci2/docker-compose.adoc index 7b385c65e3..5a4ad7c5e4 100644 --- a/jekyll/_cci2/docker-compose.md +++ b/jekyll/_cci2/docker-compose.adoc @@ -1,36 +1,39 @@ --- -layout: classic-docs -title: "Installing and Using Docker Compose" -description: "How to enable Docker Compose in your primary container" contentTags: platform: - Cloud - Server v4+ --- += Installing and Using Docker Compose +:page-description: How to enable Docker Compose in your primary container +:experimental: +:icons: font +:page-layout: classic-docs +:page-liquid: This page describes how to use Docker Compose in your CircleCI pipelines. -If you are new to Docker Compose, you can review the [official Docker Compose overview](https://docs.docker.com/compose/), or checking out the [Getting Started guide](https://docs.docker.com/compose/gettingstarted/). +If you are new to Docker Compose, you can review the link:https://docs.docker.com/compose/[official Docker Compose overview], or checking out the link:https://docs.docker.com/compose/gettingstarted/[Getting Started guide]. -## Using Docker Compose with machine executor -{: #using-docker-compose-with-machine-executor } +[#using-docker-compose-with-machine-executor] +== Using Docker Compose with machine executor -If you want to use Docker Compose to manage a multi-container setup with a Docker Compose file, use the `machine` key in your `.circleci/config.yml` file and use `docker compose` as you would normally (see Linux VM execution environment documentation [here]({{site.baseurl}}/using-linuxvm) for more details). That is, if you have a Docker Compose file that shares local directories with a container, this will work as expected. +If you want to use Docker Compose to manage a multi-container setup with a Docker Compose file, use the `machine` key in your `.circleci/config.yml` file and use `docker compose` as you would normally (see Linux VM execution environment documentation xref:using-linuxvm#[here] for more details). That is, if you have a Docker Compose file that shares local directories with a container, this will work as expected. +[#using-docker-compose-with-docker-executor] +== Using Docker Compose with Docker executor -## Using Docker Compose with docker executor -{: #using-docker-compose-with-docker-executor } +Using the `docker` execution environment combined with `setup_remote_docker` enables you to run Docker commands similarly to how you run Docker commands in a machine execution environment. However, volume mounting and port forwarding do *not* work the same way when using the `docker` execution environment. When using the `docker` execution environment with `setup_remote_docker`, a job's commands are executed in a container that has access to an external Docker daemon. Therefore, to use the Docker CLI or Docker Compose, you must move data around. Mounting can typically be solved by making content available in a Docker volume. It is possible to load data into a Docker volume by using `docker cp` to get the data from the CLI host into a location running on the Docker host. -Using the `docker` execution environment combined with `setup_remote_docker` enables you to run Docker commands similarly to how you run Docker commands in a machine execution environment. However, volume mounting and port forwarding do **not** work the same way when using the `docker` execution environment. When using the `docker` execution environment with `setup_remote_docker`, a job's commands are executed in a container that has access to an external Docker daemon. Therefore, to use the Docker CLI or Docker Compose, you must move data around. Mounting can typically be solved by making content available in a Docker volume. It is possible to load data into a Docker volume by using `docker cp` to get the data from the CLI host into a location running on the Docker host. +== Install Docker Compose -## Install Docker Compose +The Docker Compose utility is xref:circleci-images#pre-installed-tools[pre-installed in the CircleCI convenience +images] and machine executor images. -The Docker Compose utility is [pre-installed in the CircleCI convenience -images]({{ site.baseurl }}/circleci-images/#pre-installed-tools) and machine executor images. +If you are using the Docker executor and *are not* using a convenience image, you can install Docker Compose into your xref:glossary#primary-container[primary container] during the job execution and use the xref:building-docker-images#[Remote Docker Environment], as follows: -If you are using the Docker executor and **are not** using a convenience image, you can install Docker Compose into your [primary container]({{ site.baseurl }}/glossary/#primary-container) during the job execution and use the xref:building-docker-images#[Remote Docker Environment], as follows: - -```yml +[,yml] +---- - run: name: Install Docker Compose command: | @@ -50,30 +53,33 @@ If you are using the Docker executor and **are not** using a convenience image, DEBIAN_FRONTEND=noninteractive apt-get install docker-ce-cli docker-buildx-plugin docker-compose-plugin - setup_remote_docker #activate the remote docker environment -``` +---- If you are constructing your own Docker images, consider reading the -[custom docker images page]({{site.baseurl}}/custom-images/). +xref:custom-images#[custom Docker images page]. Once you have Docker Compose installed, you can use it to build images and run containers: -```yml +[,yml] +---- - run: name: Build images of services declared in docker-compose.yml command: docker compose build -``` +---- Or to run the whole system: -```yml +[,yml] +---- - run: name: Start all services declared in docker-compose.yml command: docker compose up -d -``` +---- Or to also verify if a service is running for example: -```yml +[,yml] +---- - run: name: Start Docker Compose and verify service(s) command: | @@ -88,22 +94,22 @@ Or to also verify if a service is running for example: -wait http://localhost:8080/healthcheck \ -wait-retry-interval 2s \ -timeout 20s -``` +---- -## Example project -{: #example-project } +[#example-project] +== Example project -See the [Example docker-compose Project](https://github.com/circleci/cci-demo-docker/tree/docker-compose) on GitHub for a demonstration and use the [full configuration file](https://github.com/circleci/cci-demo-docker/blob/docker-compose/.circleci/config.yml) as a template for your own projects. +See the link:https://github.com/circleci/cci-demo-docker/tree/docker-compose[Example Docker Compose Project] on GitHub for a demonstration and use the link:https://github.com/circleci/cci-demo-docker/blob/docker-compose/.circleci/config.yml[full configuration file] as a template for your own projects. -**Note**: The primary container runs in a separate environment from Remote Docker and the two cannot communicate directly. To interact with a running service, run a container in the service's network. +The primary container runs in a separate environment from Remote Docker and the two cannot communicate directly. To interact with a running service, run a container in the service's network. -## Limitations -{: #limitations } +[#limitations] +== Limitations Using `docker compose` with the `macos` executor is not supported. -See [our support article for more information](https://support.circleci.com/hc/en-us/articles/360045029591-Can-I-use-Docker-within-the-macOS-executor-). +See link:https://support.circleci.com/hc/en-us/articles/360045029591-Can-I-use-Docker-within-the-macOS-executor-[our support article for more information]. -## See also -{: #see-also } +[#see-also] +== See also -[Running Docker Commands]({{site.baseurl}}/building-docker-images/) +xref:building-docker-images#[Running Docker Commands] From 612cb017266c0913a09499c3567ef65ffa090f8c Mon Sep 17 00:00:00 2001 From: Rosie Yohannan Date: Mon, 27 Jan 2025 14:13:16 +0000 Subject: [PATCH 06/20] convert dlc page --- ...r-caching.md => docker-layer-caching.adoc} | 85 ++++++++++--------- styles/config/vocabularies/Docs/accept.txt | 2 + 2 files changed, 47 insertions(+), 40 deletions(-) rename jekyll/_cci2/{docker-layer-caching.md => docker-layer-caching.adoc} (53%) diff --git a/jekyll/_cci2/docker-layer-caching.md b/jekyll/_cci2/docker-layer-caching.adoc similarity index 53% rename from jekyll/_cci2/docker-layer-caching.md rename to jekyll/_cci2/docker-layer-caching.adoc index 31cb82214a..a9eaa4c44d 100644 --- a/jekyll/_cci2/docker-layer-caching.md +++ b/jekyll/_cci2/docker-layer-caching.adoc @@ -1,35 +1,39 @@ --- -layout: classic-docs -title: "Docker layer caching overview" -description: "How to reuse unchanged cache layers in images you build to reduce overall run time" contentTags: platform: - Cloud - Server v4+ --- += Docker layer caching overview +:page-description: How to reuse unchanged cache layers in images you build to reduce overall run time +:experimental: +:icons: font +:page-layout: classic-docs +:page-liquid: Use Docker layer caching (DLC) to reduce Docker image build times on CircleCI. DLC is available on all CircleCI plans. -## Introduction -{: #introduction } +[#introduction] +== Introduction Docker layer caching (DLC) is beneficial if building Docker images is a regular part of your CI/CD process. DLC saves Docker image layers created within your jobs, and caches them to be reused during future builds. DLC caches the individual layers of any Docker images built during your CircleCI jobs, and then reuses unchanged image layers on subsequent job runs, rather than rebuilding the entire image every time. In short, the less your Dockerfiles change from commit to commit, the less time your image-building jobs will take to run. -DLC can be used with both the `machine` executor and in a [remote Docker environment](/docs/building-docker-images/) (`setup_remote_docker`). +DLC can be used with both the `machine` executor and in a link:/docs/building-docker-images/[remote Docker environment] (`setup_remote_docker`). -DLC is charged at 200 credits per job run. For more information about DLC charges, see the [FAQ](/docs/credits/#charge-for-docker-layer-caching). +DLC is charged at 200 credits per job run. For more information about DLC charges, see the link:/docs/credits/#charge-for-docker-layer-caching[FAQ]. -## Quickstart -{: #quickstart } +[#quickstart] +== Quickstart -### Remote Docker environment -{: #remote-docker-environment } +[#remote-docker-environment] +=== Remote Docker environment -To use DLC with the Docker execution environment, you will need to configure your job to run in a [Remote Docker Environment](/docs/building-docker-images/). To do this, add `docker_layer_caching: true` under the `setup_remote_docker` key in your [`.circleci/config.yml`](/docs/configuration-reference/) file: +To use DLC with the Docker execution environment, you will need to configure your job to run in a link:/docs/building-docker-images/[Remote Docker Environment]. To do this, add `docker_layer_caching: true` under the `setup_remote_docker` key in your link:/docs/configuration-reference/[`.circleci/config.yml`] file: -```yaml +[,yaml] +---- version: 2.1 jobs: @@ -41,14 +45,15 @@ jobs: - checkout - setup_remote_docker: docker_layer_caching: true -``` +---- -### Machine executor -{: #machine-executor } +[#machine-executor] +=== Machine executor -DLC can be used when building Docker images using the [`machine` executor](/docs/configuration-reference/#machine). Use DLC with the `machine` executor by adding `docker_layer_caching: true` below your `machine` key: +DLC can be used when building Docker images using the link:/docs/configuration-reference/#machine[`machine` executor]. Use DLC with the `machine` executor by adding `docker_layer_caching: true` below your `machine` key: -```yml +[,yml] +---- version: 2.1 jobs: @@ -59,17 +64,18 @@ jobs: steps: - checkout -``` +---- -## Limitations -{: #limitations } +[#limitations] +== Limitations -### Building Docker images -{: #building-docker-images } +[#building-docker-images] +=== Building Docker images DLC is only useful when creating your own Docker image with `docker build`, `docker compose`, or similar Docker commands. It does not decrease the wall clock time that all builds take to spin up the initial environment. -```yaml +[,yaml] +---- version: 2.1 orbs: @@ -85,20 +91,19 @@ jobs: - setup_remote_docker: docker_layer_caching: true # DLC will explicitly cache layers here and try to avoid rebuilding. - run: docker build . -``` +---- -DLC has **no** effect on Docker images used as build containers. That is, containers that are used to _run_ your jobs are specified with the `image` key when using the [`docker` executor](/docs/using-docker/) and appear in the **Spin up Environment** step on your jobs pages. -{: class="alert alert-info"} +NOTE: DLC has *no* effect on Docker images used as build containers. That is, containers that are used to _run_ your jobs are specified with the `image` key when using the link:/docs/using-docker/[`docker` executor] and appear in the *Spin up Environment* step on your jobs pages. -### Buildx builder instances -{: #buildx-builder-instances } +[#buildx-builder-instances] +=== Buildx builder instances -When using buildx builder instances with DLC, it is important to [name your builders](https://docs.docker.com/engine/reference/commandline/buildx_create/#name). Doing so ensures CircleCI can detect and preserve the Docker volumes for subsequent job runs. If you do not name your builders, each time the job is run they will have different, randomly generated names, and the resulting volumes will be automatically cleaned up. +When using Buildx builder instances with DLC, it is important to https://docs.docker.com/engine/reference/commandline/buildx_create/#name[name your builders]. Doing so ensures CircleCI can detect and preserve the Docker volumes for subsequent job runs. If you do not name your builders, each time the job is run they will have different, randomly generated names, and the resulting volumes will be automatically cleaned up. -If you store Docker build artifacts in a Docker volume, managed by the buildkit inside buildx builder instances, the DLC feature cannot _maintain_ these artifacts, but they can still be supported. DLC is not able to prune these images/build cache, but buildx builders do have some in-built pruning. For more information, see the [Docker docs](https://docs.docker.com/build/cache/garbage-collection/#default-policies). +If you store Docker build artifacts in a Docker volume, managed by the BuildKit inside Buildx builder instances, the DLC feature cannot _maintain_ these artifacts, but they can still be supported. DLC is not able to prune these images/build cache, but Buildx builders do have some in-built pruning. For more information, see the https://docs.docker.com/build/cache/garbage-collection/#default-policies[Docker docs]. -## How DLC works -{: #how-dlc-works } +[#how-dlc-works] +== How DLC works DLC caches your Docker image layers within the container/virtual machine used to run your job. @@ -106,19 +111,19 @@ If, for example, the first run of your job takes over two minutes to build a Doc When none of the layers in the image change between job runs, DLC pulls the layers from the cache and reuses those instead of rebuilding the entire image. -If part of the Dockerfile changes (which changes part of the image), a subsequent run of the exact same job with the modified Dockerfile may still finish faster than rebuilding the entire image. This is because the cache can still be used for the first few steps that did not change in the Dockerfile. The steps that follow the change must be rerun because the Dockerfile change invalidates the cache for those layers. +If part of the Dockerfile changes (which changes part of the image), a subsequent run of the exact same job with the modified Dockerfile may still finish faster than rebuilding the entire image. This speed optimization happens because the cache can still be used for the first few steps that did not change in the Dockerfile. The steps that follow the change must be rerun because the Dockerfile change invalidates the cache for those layers. -If you change something in your Dockerfile, all of the later steps (from the point that the change was made) are invalidated and the layers have to be rebuilt. When some of the steps remain the same (the steps before the one you removed), those steps can be reused. So, it is still faster than rebuilding the entire image. +If you change something in your Dockerfile, all of the later steps (from the point that the change was made) are invalidated and the layers have to be rebuilt. When some of the steps remain the same (the steps before the one you removed), those steps can be reused, therefore, it is still faster than rebuilding the entire image. -There is a “DLC set-up” step at the beginning of each job that uses DLC. You are not charged for this step. At the end of each job that uses DLC, the cache upload is done asynchronously, and does not prevent the workflow from continuing to progress. This means that jobs within the same workflow are unlikely to access a cache uploaded from an upstream job. You are not charged for this “DLC teardown” step. +At the beginning of each job that uses DLC, there is a `DLC set-up` step. You are not charged for this step. At the end of each job that uses DLC, the cache upload is done asynchronously, and does not prevent the workflow from continuing to progress. This means that jobs within the same workflow are unlikely to access a cache uploaded from an upstream job. You are not charged for this `DLC teardown` step. -### Parallelism and DLC -{: #parallelism-and-dlc } +[#parallelism-and-dlc] +=== Parallelism and DLC DLC operates in the same way for jobs that use parallelism. If a `machine` job using DLC is configured with `parallelism: 2`, two jobs run in parallel. Each virtual machine in this case will have a separate DLC cache, and whichever is saved last will be used for the next build. -## Deprecated keys -{: #deprecated-keys } +[#deprecated-keys] +== Deprecated keys DLC was previously enabled via the `reusable: true` key. The `reusable` key has been deprecated in favor of the `docker_layer_caching` key. diff --git a/styles/config/vocabularies/Docs/accept.txt b/styles/config/vocabularies/Docs/accept.txt index 4a9cd9cc5d..b5a91ff091 100644 --- a/styles/config/vocabularies/Docs/accept.txt +++ b/styles/config/vocabularies/Docs/accept.txt @@ -57,7 +57,9 @@ Bitbucket Braintrust BrowserStack Buildah +\bBuildKit\b Buildkite +Buildx Bundler Capistrano Certbot From e08e1cd42a240397c30134b812a6f20edd6e4eb1 Mon Sep 17 00:00:00 2001 From: Rosie Yohannan Date: Mon, 27 Jan 2025 14:30:22 +0000 Subject: [PATCH 07/20] convert docker-to-machine --- jekyll/_cci2/docker-layer-caching.adoc | 14 ++--- jekyll/_cci2/docker-to-machine.adoc | 45 ++++++++++++++++ jekyll/_cci2/docker-to-machine.md | 72 -------------------------- 3 files changed, 52 insertions(+), 79 deletions(-) create mode 100644 jekyll/_cci2/docker-to-machine.adoc delete mode 100644 jekyll/_cci2/docker-to-machine.md diff --git a/jekyll/_cci2/docker-layer-caching.adoc b/jekyll/_cci2/docker-layer-caching.adoc index a9eaa4c44d..14dda25cf0 100644 --- a/jekyll/_cci2/docker-layer-caching.adoc +++ b/jekyll/_cci2/docker-layer-caching.adoc @@ -20,9 +20,9 @@ Docker layer caching (DLC) is beneficial if building Docker images is a regular DLC caches the individual layers of any Docker images built during your CircleCI jobs, and then reuses unchanged image layers on subsequent job runs, rather than rebuilding the entire image every time. In short, the less your Dockerfiles change from commit to commit, the less time your image-building jobs will take to run. -DLC can be used with both the `machine` executor and in a link:/docs/building-docker-images/[remote Docker environment] (`setup_remote_docker`). +DLC can be used with both the `machine` executor and in a xref:building-docker-images#[remote Docker environment] (`setup_remote_docker`). -DLC is charged at 200 credits per job run. For more information about DLC charges, see the link:/docs/credits/#charge-for-docker-layer-caching[FAQ]. +DLC is charged at 200 credits per job run. For more information about DLC charges, see the xref:credits#charge-for-docker-layer-caching[FAQ]. [#quickstart] == Quickstart @@ -30,7 +30,7 @@ DLC is charged at 200 credits per job run. For more information about DLC charge [#remote-docker-environment] === Remote Docker environment -To use DLC with the Docker execution environment, you will need to configure your job to run in a link:/docs/building-docker-images/[Remote Docker Environment]. To do this, add `docker_layer_caching: true` under the `setup_remote_docker` key in your link:/docs/configuration-reference/[`.circleci/config.yml`] file: +To use DLC with the Docker execution environment, you will need to configure your job to run in a xref:building-docker-images#[Remote Docker Environment]. To do this, add `docker_layer_caching: true` under the `setup_remote_docker` key in your xref:configuration-reference#[`.circleci/config.yml`] file: [,yaml] ---- @@ -50,7 +50,7 @@ jobs: [#machine-executor] === Machine executor -DLC can be used when building Docker images using the link:/docs/configuration-reference/#machine[`machine` executor]. Use DLC with the `machine` executor by adding `docker_layer_caching: true` below your `machine` key: +DLC can be used when building Docker images using the xref:configuration-reference#machine[`machine` executor]. Use DLC with the `machine` executor by adding `docker_layer_caching: true` below your `machine` key: [,yml] ---- @@ -93,14 +93,14 @@ jobs: - run: docker build . ---- -NOTE: DLC has *no* effect on Docker images used as build containers. That is, containers that are used to _run_ your jobs are specified with the `image` key when using the link:/docs/using-docker/[`docker` executor] and appear in the *Spin up Environment* step on your jobs pages. +NOTE: DLC has *no* effect on Docker images used as build containers. That is, containers that are used to _run_ your jobs are specified with the `image` key when using the xref:using-docker#[`docker` executor] and appear in the *Spin up Environment* step on your jobs pages. [#buildx-builder-instances] === Buildx builder instances -When using Buildx builder instances with DLC, it is important to https://docs.docker.com/engine/reference/commandline/buildx_create/#name[name your builders]. Doing so ensures CircleCI can detect and preserve the Docker volumes for subsequent job runs. If you do not name your builders, each time the job is run they will have different, randomly generated names, and the resulting volumes will be automatically cleaned up. +When using Buildx builder instances with DLC, it is important to link:https://docs.docker.com/engine/reference/commandline/buildx_create/#name[name your builders]. Doing so ensures CircleCI can detect and preserve the Docker volumes for subsequent job runs. If you do not name your builders, each time the job is run they will have different, randomly generated names, and the resulting volumes will be automatically cleaned up. -If you store Docker build artifacts in a Docker volume, managed by the BuildKit inside Buildx builder instances, the DLC feature cannot _maintain_ these artifacts, but they can still be supported. DLC is not able to prune these images/build cache, but Buildx builders do have some in-built pruning. For more information, see the https://docs.docker.com/build/cache/garbage-collection/#default-policies[Docker docs]. +If you store Docker build artifacts in a Docker volume, managed by the BuildKit inside Buildx builder instances, the DLC feature cannot _maintain_ these artifacts, but they can still be supported. DLC is not able to prune these images/build cache, but Buildx builders do have some in-built pruning. For more information, see the link:https://docs.docker.com/build/cache/garbage-collection/#default-policies[Docker docs]. [#how-dlc-works] == How DLC works diff --git a/jekyll/_cci2/docker-to-machine.adoc b/jekyll/_cci2/docker-to-machine.adoc new file mode 100644 index 0000000000..e0d204417f --- /dev/null +++ b/jekyll/_cci2/docker-to-machine.adoc @@ -0,0 +1,45 @@ +--- +contentTags: + platform: + - Cloud + - Server v4+ +--- += Migrating from Docker to Machine +:description: Best practices and considerations when migrating your executor from Docker to machine +:experimental: +:icons: font +:page-layout: classic-docs +:page-liquid: + +This page contains some general guidelines and considerations to make when moving from the Docker executor to machine, or vice versa. + +[#overview] +== Overview + +In some situations, the Docker executor is not the right fit for your builds, for example if using a Docker resource class is providing a lack of memory or if there is a requirement for more dedicated CPU power. Moving to a dedicated virtual machine can help alleviate these issues. However, changing out an executor is not as simple as replacing a few lines of configuration. You will also need to consider the tools and libraries required to be installed for your application and tests. + +[#pre-installed-software] +== Pre-installed software + +The most up to date list of pre-installed software can be found on the link:https://raw.githubusercontent.com/circleci/image-builder/picard-vm-image/provision.sh[image builder] page. You can also visit link:https://discuss.circleci.com/[the CircleCI community forum] for more information. + +Additional packages can be installed with `sudo apt-get install `. If the package in question is not found, `sudo apt-get update` may be required before installing it. + +[#running-docker-containers-on-machine] +== Running Docker containers on machine + +Machine executors come installed with Docker, which can be used to run your application within a container rather than installing additional dependencies. We recommend using a custom Docker image rather than a CircleCI convenience image, which are built under the assumption they will be used with the Docker executor and may be tricky to work around. Since each machine executor environment is a dedicated virtual machine, commands to run background containers can be used as normal. + +If you have Docker Layer Caching (DLC) enabled for your account, machine executors can utilize this to cache your image layers for subsequent runs. + +[#why-use-docker-executors-at-all] +== Why use Docker executors at all? + +Machine executors offer double the memory and a more isolated environment for running builds, but there is additional overhead regarding spin up time, and, depending on the approach taken for running the application, more time taken to install the required dependencies or pull your Docker image. The Docker executor will also cache as many layers as possible from your image during spin-up, as opposed to the machine executor, where DLC will need to be enabled. + +All executors have their pros and cons, which have been laid out here to help decide which is right for your pipelines. + +[#further-reading] +== Further reading + +We have more details on each specific executor xref:executor-intro#[here], which includes links to specific memory and vCPU allocation details, as well as how to implement each one in your own configuration. diff --git a/jekyll/_cci2/docker-to-machine.md b/jekyll/_cci2/docker-to-machine.md deleted file mode 100644 index dba4a02e31..0000000000 --- a/jekyll/_cci2/docker-to-machine.md +++ /dev/null @@ -1,72 +0,0 @@ ---- -layout: classic-docs -title: "Migrating from Docker to Machine" -description: "Best practices and considerations when migrating your executor from Docker to machine" -contentTags: - platform: - - Cloud - - Server v4+ ---- - -This document contains some general guidelines and considerations to -make when moving from the Docker executor to machine, or vice versa. - -* TOC -{:toc} - -## Overview -{: #overview } - - -Occasionally, the Docker executor isn't quite the right fit for your -builds. This can include a lack of memory or requiring more dedicated -CPU power. Moving to a dedicated virtual machine can help alleviate some -of these issues, but changing out an executor is not as easy as -replacing a few lines of configuration. There are some other -considerations to make, such as the tools and libraries required to be -installed for your application and tests. - -## Pre-installed software -{: #pre-installed-software } - -The most up to date list of pre-installed software can be found on the [image builder](https://raw.githubusercontent.com/circleci/image-builder/picard-vm-image/provision.sh) page. You can also visit the [Discuss](https://discuss.circleci.com/) page for more information. - -Additional packages can be installed with `sudo apt-get install `. If the package in question is not found, `sudo apt-get update` may be required before installing it. - -## Running docker containers on machine -{: #running-docker-containers-on-machine } - -Machine executors come installed with Docker, which can be used -to run your application within a container rather than installing -additional dependencies. Note, it is recommended this is done with a -customer Docker image rather than a CircleCI convenience image, which -are built under the assumption they will be used with the Docker -executor and may be tricky to work around. Since each machine executor -environment is a dedicated virtual machine, commands to run background -containers can be used is normal. - -**Note:** if you have Docker Layer Caching (DLC) enabled for your -account, machine executors can utilize this to cache your image layers -for subsequent runs. - -## Why use docker executors at all? -{: #why-use-docker-executors-at-all } - -While machine executors do offer twice the memory and a more isolated -environment, there is some additional overhead regarding spin up time, -and, depending on the approach taken for running the application, more -time is taken to install the required dependencies or pull your Docker -image. The Docker executor will also cache as many layers as possible -from your image during spin-up, as opposed to the machine executor, -where DLC will need to be enabled. - -All executors have their pros and cons, which have been laid out here to -help decide which is right for your pipelines. - -## Further reading -{: #further-reading } - -We have more details on each specific executor -[here]({{site.baseurl}}/executor-intro/), which includes links to -specific memory and vCPU allocation details, as well as how to implement -each one in your own configuration. From 3d9d3d940cb142698eba1cb17bd93e9a498faa2f Mon Sep 17 00:00:00 2001 From: Rosie Yohannan Date: Mon, 27 Jan 2025 15:00:40 +0000 Subject: [PATCH 08/20] convert testing orbs --- jekyll/_cci2/circleci-images.adoc | 2 +- .../{testing-orbs.md => testing-orbs.adoc} | 220 +++++++++--------- styles/config/vocabularies/Docs/accept.txt | 4 +- 3 files changed, 118 insertions(+), 108 deletions(-) rename jekyll/_cci2/{testing-orbs.md => testing-orbs.adoc} (58%) diff --git a/jekyll/_cci2/circleci-images.adoc b/jekyll/_cci2/circleci-images.adoc index 3aa50db405..d7c12799d1 100644 --- a/jekyll/_cci2/circleci-images.adoc +++ b/jekyll/_cci2/circleci-images.adoc @@ -342,7 +342,7 @@ images! *Recent Tags:* -See https://hub.docker.com/r/circleci/{{ image[0] }}/tags?ordering=last_updated[the tag list for circleci/{{ image[0] }} on Docker Hub]. +See https://hub.docker.com/r/circleci/{{ image[0] }}/tags?ordering=last_updated[the tag list for the {{ image[0] }} image on Docker Hub]. ''' diff --git a/jekyll/_cci2/testing-orbs.md b/jekyll/_cci2/testing-orbs.adoc similarity index 58% rename from jekyll/_cci2/testing-orbs.md rename to jekyll/_cci2/testing-orbs.adoc index 5a4f2c0f9c..efaf344139 100644 --- a/jekyll/_cci2/testing-orbs.md +++ b/jekyll/_cci2/testing-orbs.adoc @@ -1,141 +1,148 @@ --- -layout: classic-docs -title: "Orb Testing Methodologies" -short-title: "Testing Methodologies" -description: "Starting point for Testing CircleCI Orbs" -categories: [getting-started] -order: 1 contentTags: platform: - Cloud + - Server v4+ --- += Orb testing methodologies +:description: Starting point for Testing CircleCI Orbs +:experimental: +:icons: font +:page-layout: classic-docs +:page-liquid: This guide covers various best practices for testing orbs. -* TOC -{:toc} - -## Introduction -{: #introduction } +[#introduction] +== Introduction Orbs are a critical component of a pipeline on CircleCI, responsible for installing tooling, executing tests, deploying applications, and more. As with any software, it is important to implement tests to protect the orb from breaking with new changes. Because orbs are developed in YAML, the testing process is a little different than for a programming language. With the orb development kit, there is a simple path to implementing a full range of robust tests for your orb. -
- -
+video::kTeRJrwxShI[youtube] -## Orb-tools Pipeline Overview -{: #orb-tools-pipeline-overview } +[#orb-tools-pipeline-overview] +== Orb-tools pipeline overview -If you are following this guide and have created your orb using the orb development kit, your orb project will follow the same structure as the [Orb Template](https://github.com/CircleCI-Public/Orb-Template). If you look inside your `.circleci/` directory, you will find two config files, `config.yml` and `test-deploy.yml`, each of which contains a set of tests to run. +If you are following this guide and have created your orb using the orb development kit, your orb project will follow the same structure as the link:https://github.com/CircleCI-Public/Orb-Template[Orb Template]. If you look inside your `.circleci/` directory, you will find two config files, `config.yml` and `test-deploy.yml`, each of which contains a set of tests to run. -### config.yml -{: #configyml } +[#configyml] +=== config.yml The first config file is responsible for publishing a development version of our orb, so that you may run integration tests against it in the second workflow, `test-deploy`. At this point in the pipeline, because the orb is not yet published, you can not test the orb _directly_, but in this stage you can still lint, validate, review, and potentially even run unit tests against your scripts. After the development version of the orb has been published, the final `orb-tools/continue` job will trigger the second workflow, `test-deploy`. -See the full [config.yml template here](https://github.com/CircleCI-Public/Orb-Template/blob/main/.circleci/config.yml). - +See the full link:https://github.com/CircleCI-Public/Orb-Template/blob/main/.circleci/config.yml[config.yml template here]. -### test-deploy.yml -{: #test-deployyml } +[#test-deployyml] +=== test-deploy.yml This second configuration file has two main tasks, as the development version of the orb has been published in the previous config, you may now _directly_ test your orb with integration testing, and in the event that a tag is created, this config will also publish the orb to the CircleCI orb registry. -See the full [test-deploy.yml template here](https://github.com/CircleCI-Public/Orb-Template/blob/main/.circleci/test-deploy.yml). +See the full link:https://github.com/CircleCI-Public/Orb-Template/blob/main/.circleci/test-deploy.yml[test-deploy.yml template here]. -## Validation -{: #validation } +[#validation] +== Validation The most basic forms of testing for orbs are configuration validation and code linting. When an orb is packed and published it must be both valid YAML, and valid CircleCI syntax. Both of these checks are automatically applied when using the orb development kit, through the CI/CD pipeline set out in the project's config file at `.circleci/config.yml`. Config validation and code linting can also be performed manually, locally. -```yaml +[,yaml] +---- # Snippet from lint-pack workflow in config.yml workflows: lint-pack: jobs: - orb-tools/lint # Lints the YAML and CircleCI syntax of the orb - orb-tools/pack # Packs the orb source and validates it -``` -### YAML Linting -{: #yaml-lint } +---- -The first job listed within the workflow, `orb-tools/lint`, is from the [`orb-tools` orb](https://circleci.com/developer/orbs/orb/circleci/orb-tools), which is a major component of the orb development kit. The `orb-tools/lint` job is responsible for basic YAML linting. You can modify the linting rules or other settings via the [job's parameters, which are listed on the orb registry](https://circleci.com/developer/orbs/orb/circleci/orb-tools#jobs-lint). +[#yaml-lint] +=== YAML Linting -#### Local YAML Linting -{: #local-yaml-lint } +The first job listed within the workflow, `orb-tools/lint`, is from the https://circleci.com/developer/orbs/orb/circleci/orb-tools[`orb-tools` orb], which is a major component of the orb development kit. The `orb-tools/lint` job is responsible for basic YAML linting. You can modify the linting rules or other settings via the https://circleci.com/developer/orbs/orb/circleci/orb-tools#jobs-lint[job's parameters, which are listed on the orb registry]. + +[#local-yaml-lint] +==== Local YAML Linting If you have `yamllint` installed locally: -```shell +[,shell] +---- $ yamllint ./src -``` +---- Using CircleCI's Local Execute: -```shell +[,shell] +---- circleci local execute orb-tools/lint -``` - +---- -### Orb Validation -{: #orb-validation } +[#orb-validation] +=== Orb Validation -In addition to YAML Linting, we must validate the "packed" `orb.yml` file to ensure it properly conforms to orb schema. We first [pack]({{site.baseurl}}/orb-concepts/#orb-packing) the orb to combine the multiple source files into an `orb.yml`. Then we run the `circleci orb validate` command for schema checking. +In addition to YAML Linting, we must validate the "packed" `orb.yml` file to ensure it properly conforms to orb schema. We first link:{{site.baseurl}}/orb-concepts/#orb-packing[pack] the orb to combine the multiple source files into an `orb.yml`. Then we run the `circleci orb validate` command for schema checking. -```yaml +[,yaml] +---- # Snippet from lint-pack workflow in config.yml workflows: lint-pack: jobs: - orb-tools/pack # Packs the orb source and validates it -``` +---- -#### Local Orb Validate -{: #local-orb-validate } +[#local-orb-validate] +==== Local orb validate To pack and validate your orb locally, run the following commands: -```shell +[,shell] +---- circleci orb pack src > orb.yml circleci orb validate orb.yml -``` +---- Or, using CircleCI's Local Execute: -```shell + +[,shell] +---- circleci local execute orb-tools/pack -``` -### Shellcheck -{: #shellcheck } +---- + +[#shellcheck] +=== ShellCheck -One of the major benefits of using the orb development kit is the ability to [import external bash scripts]({{site.baseurl}}/orb-concepts/#file-include-syntax) into your final orb. Because you can keep your bash scripts in the [src/scripts](https://github.com/CircleCI-Public/Orb-Template/tree/main/src/scripts) directory, you can run additional tests against your scripts. +One of the major benefits of using the orb development kit is the ability to xref:orb-concepts#file-include-syntax[import external Bash scripts] into your final orb. Because you can keep your Bash scripts in the link:https://github.com/CircleCI-Public/Orb-Template/tree/main/src/scripts[src/scripts] directory, you can run additional tests against your scripts. -The most basic tests to run against bash scripts are a form of validation: "shellchecking". This is similar to a linter for Bash, you can find out more at [shellcheck.net](https://www.shellcheck.net/). +The most basic tests to run against Bash scripts are a form of validation: "shellchecking". Shellchecking is similar to a linter for Bash. Find out more at https://www.shellcheck.net/[shellcheck.net]. -In the `lint-pack` workflow, you will find the [shellcheck orb](https://circleci.com/developer/orbs/orb/circleci/shellcheck) is included. The shellcheck orb steps are completely optional and can be removed, especially, if your orb does not require scripts to be imported. +In the `lint-pack` workflow, you will find the link:https://circleci.com/developer/orbs/orb/circleci/shellcheck[ShellCheck orb] is included. The ShellCheck orb steps are completely optional and can be removed, especially, if your orb does not require scripts to be imported. -#### Local ShellCheck -{: #local-shellcheck } +[#local-shellcheck] +==== Local ShellCheck -To run shellcheck locally, run the following commands: +To run ShellCheck locally, run the following commands: -```shell +[,shell] +---- shellcheck src/scripts/*.sh -``` +---- Or, using CircleCI's Local Execute: -```shell + +[,shell] +---- circleci local execute shellcheck/check -``` +---- -### Review -{: #review } +[#review] +=== Review -The orb-tools orb includes a job `orb-tools/review` which will run a suite of tests against your orb designed to find opportunities to implement best practices and improve the quality of the orb. The "review" job was modeled closely after _ShellCheck_, and operates based on a list of rules called "RC" Review Checks. Each "RC" code corresponds to a specific rule, which can optionally be ignored using the `exclude` parameter in your config.yaml file. +The orb-tools orb includes a job `orb-tools/review` which will run a suite of tests against your orb designed to find opportunities to implement best practices and improve the quality of the orb. The "review" job was modeled closely after _ShellCheck_, and operates based on a list of rules called "RC" Review Checks. Each "RC" code corresponds to a specific rule, which can optionally be ignored using the `exclude` parameter in your `config.yaml` file. -```yaml +[,yaml] +---- version: 2.1 orbs: @@ -146,38 +153,37 @@ workflows: jobs: - orb-tools/review: exclude: RC006,RC007 -``` +---- Review Checks output to JUnit XML format and are automatically uploaded to CircleCI to be displayed natively in the UI. -![orb-tools review check RC008]({{site.baseurl}}/assets/img/docs/orbtools-rc008.png) +image::{{site.baseurl}}/assets/img/docs/orbtools-rc008.png[orb-tools review check RC008] When you click into the error you will receive more information such as what file and at what line in the code the error was found, along with suggestions for resolution. -**Note:** The `orb-tools/review` job currently can not be run locally due to the fact that the results are output as JUnit XML and uploaded to CircleCI, which is not supported by the local execute command at this time. -{: class="alert alert-warning"} +CAUTION: The `orb-tools/review` job currently can not be run locally due to the fact that the results are output as JUnit XML and uploaded to CircleCI, which is not supported by the local execute command at this time. -## Unit testing -{: #unit-testing } +[#unit-testing] +== Unit testing -If you are taking advantage of the orb development kit's [`<>` file inclusion]({{site.baseurl}}/orb-concepts/#file-include-syntax) feature and `src/scripts` directory to store and source your bash files, you can write true integration tests for your scripts. +If you are taking advantage of the orb development kit's xref:orb-concepts#file-include-syntax[`<>` file inclusion] feature and `src/scripts` directory to store and source your Bash files, you can write true integration tests for your scripts. -![Unit testing BASH with BATS-Core]({{site.baseurl}}/assets/img/docs/bats_tests_example.png) +image::{{site.baseurl}}/assets/img/docs/bats_tests_example.png[Unit testing Bash with BATS-Core] If you have an orb with sufficiently complex internal scripts, you may want to implement unit tests for better code quality and easier local development testing. -For Bash unit testing, we recommend the [BATS-Core](https://github.com/bats-core/bats-core) library, which is an open source testing framework for Bash, analogous to [Jest](https://jestjs.io/) for JavaScript. +For Bash unit testing, we recommend the link:https://github.com/bats-core/bats-core[BATS-Core] library, which is an open source testing framework for Bash, analogous to link:https://jestjs.io/[Jest] for JavaScript. -CircleCI has created a [BATS orb](https://circleci.com/developer/orbs/orb/circleci/bats-core) to easily integrate BATS into your CircleCI pipelines. +CircleCI has created a link:https://circleci.com/developer/orbs/orb/circleci/bats-core[BATS orb] to integrate BATS into your CircleCI pipelines. To add BATS to your orb, follow these steps: - 1. Add a `tests` directory to your orb's `src` directory. - 2. Create your tests in the `tests` directory. - 3. Add the [circleci/bats](https://circleci.com/developer/orbs/orb/circleci/bats#usage-run-bats-tests) orb to your `config.yml` file. - 4. Add the `bats/run` job to the pre-publishing jobs in the `config.yml` file. +. Add a `tests` directory to your orb's `src` directory. +. Create your tests in the `tests` directory. +. Add the https://circleci.com/developer/orbs/orb/circleci/bats#usage-run-bats-tests[bats] orb to your `config.yml` file. +. Add the `bats/run` job to the pre-publishing jobs in the `config.yml` file. -``` +---- workflows: lint-pack: jobs: @@ -196,17 +202,17 @@ workflows: - orb-tools/publish: requires: [orb-tools/lint, orb-tools/review, orb-tools/pack, shellcheck/check, bats/run] -``` +---- -Want to see how how CircleCI writes unit tests for Bash? Check out our [Slack orb](https://github.com/CircleCI-Public/slack-orb/blob/master/src/tests/notify.bats). +Want to see how CircleCI writes unit tests for Bash? Check out our link:https://github.com/CircleCI-Public/slack-orb/blob/master/src/tests/notify.bats[Slack orb]. -## Integration testing -{: #integration-testing } +[#integration-testing] +== Integration testing After validating, linting, shellchecking, and any other testing that you can perform on the source code is complete, you must test your orb's functionality in a real CircleCI config. In the second config file (`test-deploy.yml`), you can access the development version of the orb you published in the first config, and attempt to execute your orbs commands and jobs. -### Testing orb commands -{: #testing-orb-commands } +[#testing-orb-commands] +=== Testing orb commands By default, when you author a new orb, you will have an example orb source which comes with a "greet" command. You can test the greet command (and maybe other commands) in your `test-deploy` workflow as an integration test. You will be able to execute the commands to validate they run without error, and could even verify their functionality by running additional checks. @@ -216,7 +222,8 @@ In this job, you can call your orb command, with any parameters you want to test By default you will see the included "greet" command is being tested. Because the greet command only outputs a message to stdout, you can not do any additional validation checks. -```yaml +[,yaml] +---- jobs: command-tests: docker: @@ -224,11 +231,12 @@ jobs: steps: # Run your orb's commands to validate them. - /greet -``` +---- -Here is a snippet of a real example from our [GitHub CLI orb](https://github.com/CircleCI-Public/github-cli-orb): +Here is a snippet of a real example from our link:https://github.com/CircleCI-Public/github-cli-orb[GitHub CLI orb]: -```yaml +[,yaml] +---- jobs: command-tests: docker: @@ -238,23 +246,23 @@ jobs: - run: name: verify Install command: command -v gh -``` +---- In this example we are testing the `github-cli/install` command. This command may pass or fail on its own, but we can also validate in the next step that the GitHub CLI has been installed and is available on the command line. If the `gh` binary is not found in the path, this job will fail at this step. Remember that you can have multiple jobs for testing commands if desired, or if your orb has no commands, you may have no such job. Just ensure that your `orb-tools/publish` job is requiring any jobs that contain your tests. - -### Testing orb jobs -{: #testing-orb-jobs } +[#testing-orb-jobs] +=== Testing orb jobs Testing jobs within your orbs is very similar to testing commands. However, there are a few additional restrictions to consider. First, in your `test-deploy` workflow, you will see, just as we mentioned with testing commands above, there is ultimately an `orb-tools/publish` job which requires every job before it in the workflow to have completed. To test the jobs of your orb, you need to add them to this workflow and ensure they are required in the `orb-tools/publish` job. -Here is an example from CircleCI's [AWS ECR orb](https://github.com/CircleCI-Public/aws-ecr-orb/blob/0c27bfab932b60f1c60a4c2e74bee114f8d4b795/.circleci/test-deploy.yml#L40) +Here is an example from CircleCI's link:https://github.com/CircleCI-Public/aws-ecr-orb/blob/0c27bfab932b60f1c60a4c2e74bee114f8d4b795/.circleci/test-deploy.yml#L40[AWS ECR orb] -```yaml +[,yaml] +---- # Shortened for this example workflows: test-deploy: @@ -284,20 +292,20 @@ workflows: ignore: /.*/ tags: only: /^v[0-9]+\.[0-9]+\.[0-9]+$/ -``` +---- The AWS ECR orb contains a job named "build-and-push-image" which will build and push an image to the AWS ECR repository. We run this job and others with multiple parameter options to test their functionality with each code change. -Similar to how we could use additional steps to test our commands, we can take advantage of [post-steps](https://circleci.com/docs/configuration-reference/#pre-steps-and-post-steps) to validate in the job environment, or as shown in this example, we can "clean up" anything we may have created in the job. Post-Steps are additional steps that can be injected at the end of an existing job. +Similar to how we could use additional steps to test our commands, we can take advantage of link:https://circleci.com/docs/configuration-reference/#pre-steps-and-post-steps[post-steps] to validate in the job environment, or as shown in this example, we can "clean up" anything we may have created in the job. Post-Steps are additional steps that can be injected at the end of an existing job. -## What's next? -{: #whats-next } +[#whats-next] +== What's next? -Once you have added new orb features, and created tests that pass your CI, it is time to publish your orb to the Orb Registry. View the [Orb Publishing Process]({{site.baseurl}}/creating-orbs/) guide for information on releasing production-ready orbs. +Once you have added new orb features, and created tests that pass your CI, it is time to publish your orb to the Orb Registry. View the xref:creating-orbs#[Orb Publishing Process] guide for information on releasing production-ready orbs. -## See also -{: #see-also } +[#see-also] +== See also -- Refer to [Orbs Concepts]({{site.baseurl}}/orb-concepts/) for high-level information about CircleCI orbs. -- Refer to [Orb Publishing Process]({{site.baseurl}}/creating-orbs/) for information about orbs that you may use in your workflows and jobs. -- Refer to [Orbs Reference]({{site.baseurl}}/reusing-config/) for examples of reusable orbs, commands, parameters, and executors. +* Refer to xref:orb-concepts#[Orbs Concepts] for high-level information about CircleCI orbs. +* Refer to xref:creating-orbs#[Orb Publishing Process] for information about orbs that you may use in your workflows and jobs. +* Refer to xref:reusing-config#[Orbs Reference] for examples of reusable orbs, commands, parameters, and executors. diff --git a/styles/config/vocabularies/Docs/accept.txt b/styles/config/vocabularies/Docs/accept.txt index b5a91ff091..ffc037b975 100644 --- a/styles/config/vocabularies/Docs/accept.txt +++ b/styles/config/vocabularies/Docs/accept.txt @@ -164,7 +164,8 @@ JavaScript JFrog Jira jq\b -[JU|ju]nit +\bJUnit\b +junit JRuby JVM kaniko @@ -259,6 +260,7 @@ SDK Selenium SemVer Server Plan +\b[Ss]hellchecking\b \b[Ss]ingle sign-on\b SLAs? [Ss]ignup From 0ca90d8f8fca1c625af86b1891cb920d22421254 Mon Sep 17 00:00:00 2001 From: Rosie Yohannan Date: Mon, 27 Jan 2025 15:02:42 +0000 Subject: [PATCH 09/20] use yml --- jekyll/_cci2/testing-orbs.adoc | 1 + 1 file changed, 1 insertion(+) diff --git a/jekyll/_cci2/testing-orbs.adoc b/jekyll/_cci2/testing-orbs.adoc index efaf344139..8c4b7dc995 100644 --- a/jekyll/_cci2/testing-orbs.adoc +++ b/jekyll/_cci2/testing-orbs.adoc @@ -183,6 +183,7 @@ To add BATS to your orb, follow these steps: . Add the https://circleci.com/developer/orbs/orb/circleci/bats#usage-run-bats-tests[bats] orb to your `config.yml` file. . Add the `bats/run` job to the pre-publishing jobs in the `config.yml` file. +[,yaml] ---- workflows: lint-pack: From d70d932acd28330a3ffacb29be4a76a3f5c7f2a5 Mon Sep 17 00:00:00 2001 From: Rosie Yohannan Date: Mon, 27 Jan 2025 15:15:41 +0000 Subject: [PATCH 10/20] convert security page --- jekyll/_cci2/security.adoc | 67 ++++++++++++++++++++++ jekyll/_cci2/security.md | 66 --------------------- styles/config/vocabularies/Docs/accept.txt | 2 + 3 files changed, 69 insertions(+), 66 deletions(-) create mode 100644 jekyll/_cci2/security.adoc delete mode 100644 jekyll/_cci2/security.md diff --git a/jekyll/_cci2/security.adoc b/jekyll/_cci2/security.adoc new file mode 100644 index 0000000000..3553481a6b --- /dev/null +++ b/jekyll/_cci2/security.adoc @@ -0,0 +1,67 @@ +--- +contentTags: + platform: + - Server v4+ + - Server Admin +--- += How CircleCI handles security +:page-description: An overview of security measures taken at CircleCI. +:experimental: +:icons: font +:page-layout: classic-docs +:page-liquid: + +This document outlines security initiatives taken by CircleCI. + +[#overview] +== Overview + +Proactive security is a top priority at CircleCI, and security issues are acted upon immediately. Report security issues to mailto:security@circleci.com[security@circleci.com] with an encrypted message using our security team's GPG key: + +* *ID:* 0x4013DDA7 +* *Fingerprint:* 3CD2 A48F 2071 61C0 B9B7 1AE2 6170 15B8 4013 DDA7 + +[#encryption] +== Encryption + +CircleCI uses HTTPS or SSH for all networking in and out of our service. This includes from the browser to our services application, from the services application to your builder fleet, from our builder fleet to your source control system, and all other points of communication. In short, none of your code or data travels to or from CircleCI without being encrypted unless you have code in your builds that does so at your discretion. Operators may also choose to go around our SSL configuration or not use TLS for communicating with underlying systems. + +CircleCI's software has access to your code and all data that code interacts with. All jobs on CircleCI run in a sandbox (specifically, a Docker container, or an ephemeral VM) that stands alone from all other builds and is not accessible from the Internet or from your own network. The build agent pulls code via Git over SSH. Your particular test suite or job configurations may call out to external services or integration points within your network, and the response from such calls will be pulled into your jobs and used by your code at your discretion. After a job is complete, the container that ran the job is destroyed and rebuilt. All environment variables are encrypted using link:https://www.vaultproject.io/[HashiCorp Vault], using AES256-GCM96, and are unavailable to CircleCI employees. + +[#sandboxing] +== Sandboxing + +With CircleCI, you control the resources allocated to run the builds of your code. This will be done through instances of our builder boxes that set up the containers in which your builds will run. Build containers pull down source code and run whatever test and deployment scripts are part of the code base or your configuration. The containers are sandboxed, each created and destroyed for one build only (or one slice of a parallel build), and they are not available from outside themselves. CircleCI provides the ability to SSH directly to a particular build container. With the SSH process, user will have complete access to any files or processes being run inside that build container, so provide access to CircleCI _only_ to those also trusted with your source code. + +[#integrations] +== Integrations + +A few different external services and technology integration points touch CircleCI. + +* *Web sockets* CircleCI uses https://pusher.com/[Pusher] client libraries for WebSocket communication between the server and the browser, though for installs an internal server called Slanger is used, so Pusher servers have no access to your instance of CircleCI, nor your source control system. This is how CircleCI updates the builds list dynamically, or shows the output of a build line-by-line as it occurs. CircleCI sends build statuses and lines of your build output through the web socket server (which unless you have configured your installation to run without SSL, is done using the same certs over SSL), so it is encrypted in transit. +* *Source control systems* To use CircleCI you will set up a direct connection with your instance of your source control system (GitHub, Bitbucket, GitLab). When you set up CircleCI, you authorize the system to check out your private repositories. You may revoke this permission at any time through your VCS's settings by removing Circle's deploy keys and service hooks from your repositories' admin pages. While CircleCI allows you to selectively build your projects, your VCS's permissions model is "all or nothing" -- CircleCI gets permission to access all of a user's repositories or none of them. Your instance of CircleCI will have access to anything hosted in those Git repositories, and will create webhooks for a variety of events (for example, when code is pushed, when a user is added, etc.) that will call back to CircleCI, triggering one or more Git commands that will pull down code to your build fleet. +* *Dependency and source caches* Most CircleCI customers use S3 or equivalent cloud-based storage inside their private cloud infrastructure (Amazon VPC, etc.) to store their dependency and source caches. These storage servers are subject to the normal security parameters of anything stored on such services, meaning in most cases, it is suggested to prevent any outside access. +* *Artifacts* To help prevent other builds from accessing your browser's local storage when viewing artifacts, HTML and XHTML pages are hosted on their own project-specific subdomain of `*.circle-artifacts.com`. Non-HTML artifacts will usually be (`302 FOUND`) redirected to an S3 URL to allow for the highest download speed. Because these artifact types are hosted on a single S3 domain, artifacts may access your browser's local storage on HTML and XHTML pages, and so you should avoid entering sensitive data into your browser for these URLs. +* *iOS builds* If you are paying to run iOS builds on CircleCI hardware, your source code will be downloaded to a build box on a macOS fleet where it will be compiled, and any tests will be run. Similar to CircleCI's primary build containers that you control, the iOS builds that run are sandboxed such that they cannot be accessed. +* *Docker* If you are using Docker images, refer to the public Docker link:https://github.com/docker/engine/blob/e76380b67bcdeb289af66ec5d6412ea85063fc04/profiles/seccomp/default.json[seccomp (security computing mode) profile] for the Docker engine. CircleCI appends the following to the Docker default `seccomp` profile: + +{% raw %} +[,shell] +---- +[ + { + "comment": "Allow create user namespaces", + "names": [ + "clone", + "setns", + "unshare" + ], + "action": "SCMP_ACT_ALLOW", + "args": [], + "includes": {}, + "excludes": {} + } +] +---- + +{% endraw %} diff --git a/jekyll/_cci2/security.md b/jekyll/_cci2/security.md deleted file mode 100644 index d1a70441e6..0000000000 --- a/jekyll/_cci2/security.md +++ /dev/null @@ -1,66 +0,0 @@ ---- -layout: classic-docs -title: "How CircleCI handles security" -description: "An overview of security measures taken at CircleCI." -contentTags: - platform: - - Cloud - - Server v4+ ---- - -This document outlines security initiatives taken by CircleCI. - -## Overview -{: #overview } - -Proactive security is a top priority at CircleCI, and security issues are acted upon immediately. Report security issues to with an encrypted message using our security team's GPG key: -- **ID:** 0x4013DDA7 -- **Fingerprint:** 3CD2 A48F 2071 61C0 B9B7 1AE2 6170 15B8 4013 DDA7 - -## Encryption -{: #encryption } - -CircleCI uses HTTPS or SSH for all networking in and out of our service. This includes from the browser to our services application, from the services application to your builder fleet, from our builder fleet to your source control system, and all other points of communication. In short, none of your code or data travels to or from CircleCI without being encrypted unless you have code in your builds that does so at your discretion. Operators may also choose to go around our SSL configuration or not use TLS for communicating with underlying systems. - -CircleCI's software has access to your code and all data that code interacts with. All jobs on CircleCI run in a sandbox (specifically, a Docker container, or an ephemeral VM) that stands alone from all other builds and is not accessible from the Internet or from your own network. The build agent pulls code via Git over SSH. Your particular test suite or job configurations may call out to external services or integration points within your network, and the response from such calls will be pulled into your jobs and used by your code at your discretion. After a job is complete, the container that ran the job is destroyed and rebuilt. All environment variables are encrypted using [HashiCorp Vault](https://www.vaultproject.io/), using AES256-GCM96, and are unavailable to CircleCI employees. - -## Sandboxing -{: #sandboxing } - -With CircleCI, you control the resources allocated to run the builds of your code. This will be done through instances of our builder boxes that set up the containers in which your builds will run. Build containers pull down source code and run whatever test and deployment scripts are part of the code base or your configuration. The containers are sandboxed, each created and destroyed for one build only (or one slice of a parallel build), and they are not available from outside themselves. CircleCI provides the ability to SSH directly to a particular build container. With the SSH process, user will have complete access to any files or processes being run inside that build container, so provide access to CircleCI _only_ to those also trusted with your source code. - -## Integrations -{: #integrations } - -A few different external services and technology integration points touch CircleCI. - -- **Web sockets** CircleCI uses [Pusher](https://pusher.com/) client libraries for WebSocket communication between the server and the browser, though for installs an internal server called slanger is used, so Pusher servers have no access to your instance of CircleCI, nor your source control system. This is how CircleCI updates the builds list dynamically, or shows the output of a build line-by-line as it occurs. CircleCI sends build statuses and lines of your build output through the web socket server (which unless you have configured your installation to run without SSL, is done using the same certs over SSL), so it is encrypted in transit. - -- **Source control systems** To use CircleCI you will set up a direct connection with your instance of your source control system (GitHub, Bitbucket, GitLab). When you set up CircleCI, you authorize the system to check out your private repositories. You may revoke this permission at any time through your VCS's settings by removing Circle's deploy keys and service hooks from your repositories' admin pages. While CircleCI allows you to selectively build your projects, your VCS's permissions model is "all or nothing" — CircleCI gets permission to access all of a user's repositories or none of them. Your instance of CircleCI will have access to anything hosted in those Git repositories, and will create webhooks for a variety of events (eg: when code is pushed, when a user is added, etc.) that will call back to CircleCI, triggering one or more Git commands that will pull down code to your build fleet. - -- **Dependency and source caches** Most CircleCI customers use S3 or equivalent cloud-based storage inside their private cloud infrastructure (Amazon VPC, etc.) to store their dependency and source caches. These storage servers are subject to the normal security parameters of anything stored on such services, meaning in most cases, it is suggested to prevent any outside access. - -- **Artifacts** To help prevent other builds from accessing your browser's local storage when viewing artifacts, HTML and XHTML pages are hosted on their own project-specific subdomain of `*.circle-artifacts.com`. Non-HTML artifacts will usually be (`302 FOUND`) redirected to an S3 URL to allow for the highest download speed. Because these artifact types are hosted on a single S3 domain, artifacts may access your browser's local storage on HTML and XHTML pages, and so you should avoid entering sensitive data into your browser for these URLs. - -- **iOS builds** If you are paying to run iOS builds on CircleCI hardware, your source code will be downloaded to a build box on a macOS fleet where it will be compiled, and any tests will be run. Similar to CircleCI's primary build containers that you control, the iOS builds that run are sandboxed such that they cannot be accessed. - -- **Docker** If you are using Docker images, refer to the public Docker [seccomp (security computing mode) profile](https://github.com/docker/engine/blob/e76380b67bcdeb289af66ec5d6412ea85063fc04/profiles/seccomp/default.json) for the Docker engine. CircleCI appends the following to the Docker default `seccomp` profile: - -{% raw %} -``` -[ - { - "comment": "Allow create user namespaces", - "names": [ - "clone", - "setns", - "unshare" - ], - "action": "SCMP_ACT_ALLOW", - "args": [], - "includes": {}, - "excludes": {} - } -] -``` -{% endraw %} diff --git a/styles/config/vocabularies/Docs/accept.txt b/styles/config/vocabularies/Docs/accept.txt index ffc037b975..0a1fddd2de 100644 --- a/styles/config/vocabularies/Docs/accept.txt +++ b/styles/config/vocabularies/Docs/accept.txt @@ -256,6 +256,7 @@ SSO \bSauce Labs\b \bScale Plan\b SDK +seccomp [Ss]ecrets? Selenium SemVer @@ -263,6 +264,7 @@ Server Plan \b[Ss]hellchecking\b \b[Ss]ingle sign-on\b SLAs? +Slanger [Ss]ignup SKUs? [Ss]nap From 87a651905206113bc6d12cc6a5f637b2a6e06179 Mon Sep 17 00:00:00 2001 From: Rosie Yohannan Date: Mon, 27 Jan 2025 16:01:13 +0000 Subject: [PATCH 11/20] convert security recommendations --- jekyll/_cci2/security-recommendations.adoc | 115 ++++++++++++++++++ jekyll/_cci2/security-recommendations.md | 107 ---------------- ...ly-chain.md => security-supply-chain.adoc} | 68 ++++++----- styles/config/vocabularies/Docs/accept.txt | 1 + 4 files changed, 152 insertions(+), 139 deletions(-) create mode 100644 jekyll/_cci2/security-recommendations.adoc delete mode 100644 jekyll/_cci2/security-recommendations.md rename jekyll/_cci2/{security-supply-chain.md => security-supply-chain.adoc} (53%) diff --git a/jekyll/_cci2/security-recommendations.adoc b/jekyll/_cci2/security-recommendations.adoc new file mode 100644 index 0000000000..88a11fba94 --- /dev/null +++ b/jekyll/_cci2/security-recommendations.adoc @@ -0,0 +1,115 @@ +--- +contentTags: + platform: + - Cloud + - Server v4+ +--- += Secure secrets handling +:description: Learn how to handle secrets securely with CircleCI. +:experimental: +:icons: font +:page-layout: classic-docs +:page-liquid: + +Many builds must reference secret values entrusted to CircleCI. CircleCI understands that security is critical to every organization's success. In addition to the work CircleCI does to keep your secrets safe, there are a few things you can do to help protect secrets at the boundary between CircleCI's systems and yours. + +[#risks-of-using-secrets-on-the-command-line] +== Risks of using secrets on the command-line + +Unix and Linux shells can expose sensitive data in several ways. It is important to consider all of them when working with CircleCI on the command-line. + +* *Command history*: If you include a secret in a command's parameters, such as `export MY_SECRET="value"` or `curl --header "authorization: Basic TOKEN"`, that value could be written into your shell's history file, such as `.bash_history`. Anyone with access to that file could then retrieve the secret. +* *Process arguments*: While a process is running, any user on the same system can view the command that started it. The easiest way to see this is by running `ps -ef`, but there are other methods as well. Critically, this information is exposed after environment variables have been interpreted, so that when running `mycommand "$MYVAR"`, `ps` will show `mycommand `. On some older variants of Unix, such as AIX, it is also possible for all users to see all environment variables for any process. +* *System logs*: Many systems log all commands executed using `sudo` for auditing. There are many auditing services that record all commands. Such services could potentially export those logs into systems that are not designed to keep secret data safe. +* *Console output*: Depending on your threat model and what kind of console is in use, simply printing a secret to the console could carry risk. For example, use of screen-sharing tools for activities like pair-programming can lead to accidental, persistent exposure of secrets transited through untrusted videoconferencing providers, possibly even in video recordings. It is best to choose tools that print secrets to the console only when necessary and explicitly told to do so by the user. +* *Persistent, unencrypted secrets on disk*: Although it is common practice for command-line tools to store and use secrets stored in files in your home directory, such files' availability to all processes and persistence over time may be a significant risk. + +[#mitigation-techniques] +== Mitigation techniques + +In this section, several techniques to help mitigate the risks discussed above are presented. We will focus on methods for using `curl` commands and xref:local-cli#[the CircleCI CLI] securely with the Bash shell. + +[#general-precautions] +=== General precautions + +Avoid running `env` or `printenv`, which will print the values of all environment variables, including secrets. + +Avoid writing secrets into your shell history with these following techniques. However, note that turning off history will not prevent commands from being exposed through audit logs and `ps`: + +* Running `set +o history` before the sensitive commands will prevent them from being written to the history file. `set -o history` will turn history logging back on. +* If your shell supports the `HISTCONTROL` environment variable, and it is set to `ignoreboth` or `ignorespace`, placing a space before your command will prevent it from being written to the history file. + +Be aware that `export` is built in to Bash and other common shells. This means that, with precautions, you can avoid exposure of secrets to the history file, `ps`, and audit logs when using `export`: + +* Make sure to avoid writing to the shell history by using `set +o history` or `HISTCONTROL`. +* Next, if unsure, verify that `export` is really a shell built-in by using the `type` command: `type export`. +* Remember that information will still be exposed in your console, and make sure you are okay with that risk. +* Follow the rest of the precautions on this page related to the use of environment variables. +* As soon as you are finished using a secret with `export`, consider using `unset` to remove it from the shell. Otherwise, the `export` environment variable will still be available to all processes spawned from that console. + +Another shell built-in, `read`, can be used to set an environment variable without exposing it to the console: + +[,shell] +---- +# Check that your version of read supports the -s option +help read + +IFS='' read -r -s MY_VAR +# (enter your secret; press return when done) + +# Alternatively, read from a file +IFS='' read -r MY_VAR < ~/.my_secret + +# Or a process +secret_producer | IFS='' read -r MY_VAR + +# Export the variable so that it is available to subprocesses +export MY_VAR +---- + +[#use-the-circleci-cli] +=== Use the CircleCI CLI + +Use the xref:local-cli#[CircleCI local CLI] instead of `curl` commands when possible. The CLI takes extra precautions to avoid leaking secrets when performing sensitive operations. For example, when xref:contexts#creating-environment-variables[creating environment variables], the CLI will prompt you to enter the secret rather than accepting it as a command line argument. + +If writing a shell script that uses the CircleCI CLI, remember that in Bash you can avoid exposing secrets stored in environment variables or text by using the `<<<` construct, which does not spawn a new process while piping a value: + +[,bash] +---- +`circleci context store-secret --org-id <<< "$MY_SECRET"` +---- + +Using the `<<<` construct is more reliable than using `echo` or `printf`, which may or may not be shell built-ins and could potentially spawn a process. + +[#protect-the-api-token] +=== Protect the API token + +When calling the CircleCI API with `curl` commands, you need to provide an API token. Several ways you can mitigate risk while supplying tokens are as follows: + +* Use a `.netrc` file: The link:https://everything.curl.dev/usingcurl/netrc[`.netrc` file format], which is supported by several different tools, allows you to provide HTTP basic auth credentials in a file, rather than at the command-line. +** Create a file at a location of your choosing. The default used by some tools is `~/.netrc`. Be sure to `chmod 0600` this file before adding the secret, to prevent other users from viewing its contents. +** Add a line in the following format: `machine circleci.com login password` +** When invoking `curl` commands, tell it to look in your `.netrc` file for credentials: `curl --netrc-file ~/.netrc` +* Write the `Circle-Token` header into a file. This requires cURL 7.55 or later, but is a more reliable solution than `.netrc`, because some tools that use `.netrc` files do not understand an empty password field: +** Create a file at a location of your choosing. Be sure to `chmod 0600` the file to prevent other users from viewing its contents. +** Add a line in the following format: `Circle-Token: ` +** When invoking `curl` commands, tell it to read the header from a file: `curl --header @your_filename` +* Pull the token directly from a tool designed to store secrets, such as 1Password. In this case, you can use link:https://en.wikipedia.org/wiki/Process_substitution[process substitution] to retrieve the header without exposing it: +** `curl --header @<(command_to_retrieve_password)` +** If you are sure that `printf` is a built-in in your shell, it should also be safe to write `curl --header @<(printf '%s\n' "$MYVAR")`, allowing you to use environment variables without exposing them through `ps`. + +[#protect-your-secrets] +=== Protect your secrets + +Some API endpoints, such as link:https://circleci.com/docs/api/v2/#operation/addEnvironmentVariableToContext[addEnvironmentVariableToContext], may require secrets to be sent in the body of `PUT` or `POST` requests. Several options to help conceal secrets sent in a request body are as follows: + +* Use a file to compose and store the request body. Be sure to `chmod 0600` this file before adding the secret value to prevent other users from viewing its contents. +** Point `curl` to this file by using the `@` directive: `curl --data @myfile` +* Use a Heredoc to compose the request body, and pass it to cURL on stdin: ++ +[,shell] +---- +curl --data @- <`. On some older variants of Unix, such as AIX, it is also possible for all users to see all environment variables for any process. - -* **System logs**: Many systems log all commands executed using `sudo` for auditing. There are many auditing services that record all commands. Such services could potentially export those logs into systems that are not designed to keep secret data safe. - -* **Console output**: Depending on your threat model and what kind of console is in use, simply printing a secret to the console could carry risk. For example, use of screen-sharing tools for activities like pair-programming can lead to accidental, persistent exposure of secrets transited through untrusted videoconferencing providers, possibly even in video recordings. It is best to choose tools that print secrets to the console only when necessary and explicitly told to do so by the user. - -* **Persistent, unencrypted secrets on disk**: Although it is common practice for command-line tools to store and use secrets stored in files in your home directory, such files' availability to all processes and persistence over time may be a significant risk. - -## Mitigation techniques -{: #mitigation-techniques } - -There are many techniques to help mitigate the risks discussed above. We will focus on methods for using `curl` commands and [the CircleCI CLI]({{site.baseurl}}/local-cli) securely with the bash shell. - -### General precautions -{: #general-precautions } - -Avoid running `env` or `printenv`, which will print the values of all environment variables, including secrets. - -Avoid writing secrets into your shell history with these following techniques. However, note that turning off history will not prevent commands from being exposed through audit logs and `ps`: - - Running `set +o history` before the sensitive commands will prevent them from being written to the history file. `set -o history` will turn history logging back on. - - If your shell supports the `HISTCONTROL` environment variable, and it is set to `ignoreboth` or `ignorespace`, placing a space before your command will prevent it from being written to the history file. - -Be aware that `export` is built in to bash and other common shells. This means that, with precautions, you can avoid exposure of secrets to the history file, `ps`, and audit logs when using `export`: - - Make sure to avoid writing to the shell history by using `set +o history` or `HISTCONTROL`. - - Next, if unsure, verify that `export` is really a shell built-in by using the `type` command: `type export`. - - Remember that information will still be exposed in your console, and make sure you are okay with that risk. - - Follow the rest of the precautions on this page related to the use of environment variables. - - As soon as you are finished using a secret with `export`, consider using `unset` to remove it from the shell. Otherwise, the `export` environment variable will still be available to all processes spawned from that console. - -Another shell built-in, `read`, can be used to set an environment variable without exposing it to the console: -``` -# Check that your version of read supports the -s option -help read - -IFS='' read -r -s MY_VAR -# (enter your secret; press return when done) - -# Alternatively, read from a file -IFS='' read -r MY_VAR < ~/.my_secret - -# Or a process -secret_producer | IFS='' read -r MY_VAR - -# Export the variable so that it is available to subprocesses -export MY_VAR -``` - -### Use the CircleCI CLI -{: #use-the-circleci-cli } - -Use the [the CircleCI local CLI]({{site.baseurl}}/local-cli) instead of `curl` commands when possible. The CLI takes extra precautions to avoid leaking secrets when performing sensitive operations. For example, when [creating environment variables]({{site.baseurl}}/contexts#creating-environment-variables), the CLI will prompt you to enter the secret rather than accepting it as a command line argument. - -If writing a shell script that uses the CircleCI CLI, remember that in bash you can avoid exposing secrets stored in environment variables or text by using the `<<<` construct, which does not spawn a new process while piping a value: -```bash -`circleci context store-secret --org-id <<< "$MY_SECRET"` -``` -This is more reliable than using `echo` or `printf`, which may or may not be shell built-ins and could potentially spawn a process. - -### Protect the API token -{: #protect-the-api-token } - -When calling the CircleCI API with `curl` commands, you need to provide an API token. There are several ways you can mitigate risk while doing so: - -* Use a `.netrc` file: The [netrc file format](https://everything.curl.dev/usingcurl/netrc), which is supported by several different tools, allows you to provide HTTP basic auth credentials in a file, rather than at the command-line. - - Create a file at a location of your choosing. The default used by some tools is `~/.netrc`. Be sure to `chmod 0600` this file before adding the secret, to prevent other users from viewing its contents. - - Add a line in the following format: `machine circleci.com login password` - - When invoking `curl` commands, tell it to look in your `.netrc` file for credentials: `curl --netrc-file ~/.netrc` -* Write the `Circle-Token` header into a file. This requires cURL 7.55 or later, but is a more reliable solution than `.netrc`, because some tools that use `.netrc` files do not understand an empty password field: - - Create a file at a location of your choosing. Be sure to `chmod 0600` the file to prevent other users from viewing its contents. - - Add a line in the following format: `Circle-Token: ` - - When invoking `curl` commands, tell it to read the header from a file: `curl --header @your_filename` -* Pull the token directly from a tool designed to store secrets, such as 1Password. In this case, you can use [process substitution](https://en.wikipedia.org/wiki/Process_substitution) to retrieve the header without exposing it: - - `curl --header @<(command_to_retrieve_password)` - - If you are sure that `printf` is a built-in in your shell, it should also be safe to write `curl --header @<(printf '%s\n' "$MYVAR")`, allowing you to use environment variables without exposing them through `ps`. - -### Protect your secrets -{: #protect-your-secrets } - -Some API endpoints, such as [addEnvironmentVariableToContext](https://circleci.com/docs/api/v2/#operation/addEnvironmentVariableToContext), may require secrets to be sent in the body of `PUT` or `POST` requests. There are several options to help conceal these secrets: - -* Use a file to compose and store the request body. Be sure to `chmod 0600` this file before adding the secret value to prevent other users from viewing its contents. - - Point `curl` to this file by using the `@` directive: `curl --data @myfile` -* Use a heredoc to compose the request body, and pass it to cURL on stdin: -``` -curl --data @- < requirements.in -$ pip-compile --generate-hashes requirements.in --output-file requirements.txt -$ pip install --no-deps -r requirements.txt -``` +[,shell] +---- +echo ‘flask’ > requirements.in +pip-compile --generate-hashes requirements.in --output-file requirements.txt +pip install --no-deps -r requirements.txt +---- This adds a single top-level dependency called `flask` to an input file, then generates secure hashes for all transitive dependencies and locks their versions. Installation using the `--no-deps` flag ensures that only the dependencies listed in the requirements file are installed and nothing else. Likewise, a similar example will ensure only exactly the known dependencies are installed when using `yarn`. -```shell -$ yarn add express +[,shell] +---- +yarn add express # during your build -$ yarn install --frozen-lockfile -``` +yarn install --frozen-lockfile +---- Many tools for scanning dependency files exist, and many are first-party for a given language or tool chain. On CircleCI, there are orbs available that offer -[dependency scanning](https://circleci.com/developer/orbs?query=&category=Security), and cron jobs for periodic scanning to ensure your applications are scanning more often than your pushes. +link:https://circleci.com/developer/orbs?query=&category=Security[dependency scanning], and cron jobs for periodic scanning to ensure your applications are scanning more often than your pushes. Using dependency pinning with hashes like this prevents malicious binaries or packages from silently replacing known good versions. It protects against a narrow range of attacks where the upstream repository is compromised. This can protect your workstation and CI builds. -## See also -{: #see-also } +[#see-also] +== See also - -- [Security recommendations]({{site.baseurl}}/security-recommendations) \ No newline at end of file +* xref:security-recommendations#[Security recommendations] diff --git a/styles/config/vocabularies/Docs/accept.txt b/styles/config/vocabularies/Docs/accept.txt index 0a1fddd2de..b1f77ce38e 100644 --- a/styles/config/vocabularies/Docs/accept.txt +++ b/styles/config/vocabularies/Docs/accept.txt @@ -146,6 +146,7 @@ Gzip HashiCorp [Hh]eadspace Helm +Heredoc Heroku Homebrew [Hh]ostname From e48b1b9b530f5bee83cf82af72082c32abdae1ae Mon Sep 17 00:00:00 2001 From: Rosie Yohannan Date: Mon, 27 Jan 2025 16:18:32 +0000 Subject: [PATCH 12/20] convert sample config page --- .../{sample-config.md => sample-config.adoc} | 160 ++++++++++-------- 1 file changed, 89 insertions(+), 71 deletions(-) rename jekyll/_cci2/{sample-config.md => sample-config.adoc} (82%) diff --git a/jekyll/_cci2/sample-config.md b/jekyll/_cci2/sample-config.adoc similarity index 82% rename from jekyll/_cci2/sample-config.md rename to jekyll/_cci2/sample-config.adoc index 2f92de0d28..e0050e0c4d 100644 --- a/jekyll/_cci2/sample-config.md +++ b/jekyll/_cci2/sample-config.adoc @@ -1,48 +1,51 @@ --- -layout: classic-docs -title: "Sample config.yml Files" -description: "Sample config.yml File" contentTags: platform: - Cloud - Server v4+ --- += Sample config.yml Files +:description: Sample config.yml File +:experimental: +:icons: font +:page-layout: classic-docs +:page-liquid: -This document provides sample `.circleci/config.yml` files that you can use as a starting point when setting up projects, or to better understand different ways to orchestrate jobs using workflows and filters. For information on all configuration elements available to you, see the [Configuration reference](/docs/configuration-reference/) page. +This document provides sample `.circleci/config.yml` files that you can use as a starting point when setting up projects, or to better understand different ways to orchestrate jobs using workflows and filters. For information on all configuration elements available to you, see the xref:configuration-reference#[Configuration reference] page. If you would like to get set up quickly, see our language-specific quickstart guides: -* [Node](/docs/language-javascript/) -* [Python](/docs/language-python/) -* [Go](/docs/language-go/) +* xref:language-javascript#[Node] +* xref:language-python#[Python] +* xref:language-go#[Go] -## Tools for editing configuration files -{: tools-for-editing-configuration-files } +== Tools for editing configuration files -CircleCI has created an [extension for Visual Studio Code](/docs/vs-code-extension-overview/) that reduces context switching between the web app and VS Code through a set of helpful features. +CircleCI has created an xref:vs-code-extension-overview#[extension for Visual Studio Code] that reduces context switching between the web app and VS Code through a set of helpful features. The VS Code extension reduces the time to create, modify, and troubleshoot configuration files through real-time syntax validation, highlighting, and autocomplete suggestions. Authenticating the extension with your CircleCI account will also allow you to visualize and manage your CircleCI pipelines directly from your code editor, and be notified of workflow status changes. -The CircleCI VS Code extension is available to download on the [VS Code marketplace](https://marketplace.visualstudio.com/items?itemName=circleci.circleci). +The CircleCI VS Code extension is available to download on the link:https://marketplace.visualstudio.com/items?itemName=circleci.circleci[VS Code marketplace]. -## Simple configuration examples -{: #simple-configuration-examples } +[#simple-configuration-examples] +== Simple configuration examples -{% include snippets/docker-auth.md %} +{% include snippets/docker-auth.adoc %} -### Concurrent workflow -{: #concurrent-workflow } +[#concurrent-workflow] +=== Concurrent workflow The configuration example below shows a concurrent workflow in which the `build` and `test` jobs run at the same time. Both jobs are run in Docker containers using the base image provided by CircleCI. -* Refer to the [Workflows](/docs/workflows) document for complete details about orchestrating job runs with concurrent, sequential, and manual approval workflows. -* Refer to the [Developer Hub convenience images](https://circleci.com/developer/images?imageType=docker) page to find out about available Docker images for running your jobs. +* Refer to the xref:workflows#[Workflows] document for complete details about orchestrating job runs with concurrent, sequential, and manual approval workflows. +* Refer to the link:https://circleci.com/developer/images?imageType=docker[Developer Hub convenience images] page to find out about available Docker images for running your jobs. This image shows the workflow view for the following configuration example: -![Concurrent Workflow Graph]({{ site.baseurl }}/assets/img/docs/concurrent-workflow-map.png) +image::{{ site.baseurl }}/assets/img/docs/concurrent-workflow-map.png[Concurrent Workflow Graph] -```yaml +[,yaml] +---- version: 2.1 # Define the jobs we want to run for this project @@ -66,21 +69,22 @@ workflows: jobs: - build - test -``` +---- -### Sequential workflow -{: #sequential-workflow } +[#sequential-workflow] +=== Sequential workflow -The configuration example below shows a sequential workflow where the `build` job runs, and then the `test` job runs once `build` has completed. This is achieved by using the [`requires` key](/docs/configuration-reference/#requires), and specifying the `test` job "requires" the `build` job in order to run. Both jobs are run in Docker containers using the base image provided by CircleCI. +The configuration example below shows a sequential workflow where the `build` job runs, and then the `test` job runs once `build` has completed. Configure sequential workflows using the xref:configuration-reference#requires[`requires` key], and specifying the `test` job "requires" the `build` job in order to run. Both jobs are run in Docker containers using the base image provided by CircleCI. -* Refer to the [Workflows](/docs/workflows) document for complete details about orchestrating job runs with concurrent, sequential, and manual approval workflows. -* Refer to the [developer hub convenience images](https://circleci.com/developer/images?imageType=docker) page to find out about available Docker images for running your jobs. +* Refer to the xref:workflows#[Workflows] document for complete details about orchestrating job runs with concurrent, sequential, and manual approval workflows. +* Refer to the link:https://circleci.com/developer/images?imageType=docker[developer hub convenience images] page to find out about available Docker images for running your jobs. This image shows the workflow view for the following configuration example, in which jobs run sequentially (one after the other): -![Sequential Workflow Graph]({{ site.baseurl }}/assets/img/docs/sequential-workflow-map.png) +image::{{ site.baseurl }}/assets/img/docs/sequential-workflow-map.png[Sequential Workflow Graph] -```yaml +[,yaml] +---- version: 2.1 # Define the jobs we want to run for this project @@ -106,14 +110,15 @@ workflows: - test: requires: - build -``` +---- -### Approval job -{: #approval-job } +[#approval-job] +=== Approval job -The example below shows a sequential workflow with an [approval job](/docs/configuration-reference/#type). Use an approval job to require manual approval before downstream jobs may proceed. +The example below shows a sequential workflow with an xref:configuration-reference#type[approval job]. Use an approval job to require manual approval before downstream jobs may proceed. -```yaml +[,yaml] +---- version: 2.1 executors: # Define an executor @@ -153,14 +158,14 @@ workflows: - deploy: requires: - hold -``` +---- The workflow runs as follows: -1. `build` job runs -1. `test` job runs -1. `hold` job, with `type: approval` ensures the workflow waits for manual approval in the CircleCI web app -1. Once `hold` job is approved the `deploy` job runs +. `build` job runs +. `test` job runs +. `hold` job, with `type: approval` ensures the workflow waits for manual approval in the CircleCI web app +. Once `hold` job is approved the `deploy` job runs An approval job can have any name. In the example above the approval job is named `hold`. The name you choose for an approval job should not be used to define a job in the main configuration. An approval job only exists as a workflow orchestration devise. @@ -170,14 +175,15 @@ The image below shows the workflow view for the following configuration example. * The "Needs Approval" popup that appears when you click on a hold job * The workflow view again once the `hold` job has been approved and the `deploy` job has run -![Approval Workflow Graph]({{ site.baseurl }}/assets/img/docs/approval-workflow-map.png) +image::{{ site.baseurl }}/assets/img/docs/approval-workflow-map.png[Approval Workflow Graph] -* Refer to the [Workflows](/docs/workflows) document for complete details about orchestrating job runs with concurrent, sequential, and manual approval workflows. -* Refer to the [developer hub convenience images](https://circleci.com/developer/images?imageType=docker) page to find out about available Docker images for running your jobs. +* Refer to the xref:workflows#[Workflows] document for complete details about orchestrating job runs with concurrent, sequential, and manual approval workflows. +* Refer to the link:https://circleci.com/developer/images?imageType=docker[developer hub convenience images] page to find out about available Docker images for running your jobs. -### Hello world +=== Hello world -```yml +[,yml] +---- # Use the latest 2.1 version of CircleCI pipeline process engine. # See: https://circleci.com/docs/configuration-reference version: 2.1 @@ -209,10 +215,10 @@ workflows: # Inside the workflow, you define the jobs you want to run. jobs: - say-hello -``` +---- -## Sample configuration with sequential workflow and secondary Docker container -{: #sample-configuration-with-sequential-workflow } +[#sample-configuration-with-sequential-workflow] +== Sample configuration with sequential workflow and secondary Docker container Following is a sample `.circleci/config.yml` file using the following configuration features: @@ -222,7 +228,8 @@ Following is a sample `.circleci/config.yml` file using the following configurat * Workspaces * Storing artifacts -```yaml +[,yaml] +---- version: 2.1 orbs: @@ -287,19 +294,21 @@ workflows: - test: requires: - build -``` +---- -This example shows a sequential workflow with the `test` job configured to run only on the main branch. Refer to the [Workflows]({{ site.baseurl }}/workflows) document for complete details about orchestrating job runs with concurrent, sequential, and manual approval workflows. +This example shows a sequential workflow with the `test` job configured to run only on the main branch. Refer to the xref:workflows#[Workflows] document for complete details about orchestrating job runs with concurrent, sequential, and manual approval workflows. + +[#sample-configuration-with-fan-infan-out-workflow] +== Sample configuration with fan-in/fan-out workflow -## Sample configuration with fan-in/fan-out workflow -{: #sample-configuration-with-fan-infan-out-workflow } Below are two sample configurations for a Fan-in/Fan-out workflow. -![Fan-in-out]({{ site.baseurl }}/assets/img/docs/fan-in-out-example.png) +image::{{ site.baseurl }}/assets/img/docs/fan-in-out-example.png[Fan-in-out] -{:.tab.fan-in-out.Cloud} {% raw %} -```yaml + +[,yaml] +---- version: 2.1 orbs: @@ -431,23 +440,26 @@ workflows: requires: - build-docker-image - test -``` +---- + {% endraw %} -**Note:** a job can only run when its dependencies are satisfied therefore it requires the dependencies of all upstream jobs. This means only the immediate upstream dependencies need to be specified in the `requires:` blocks. +NOTE: A job can only run when its dependencies are satisfied therefore it requires the dependencies of all upstream jobs. This means only the immediate upstream dependencies need to be specified in the `requires:` blocks. -## Sample configuration with multiple executor types -{: #sample-configuration-with-multiple-executor-types } +[#sample-configuration-with-multiple-executor-types] +== Sample configuration with multiple executor types -It is possible to use multiple [executor types]({{site.baseurl}}/executor-intro/) +It is possible to use multiple xref:executor-intro#[executor types] in the same workflow. In `Example-1` each push will build and test the project on Linux, Windows and macOS. -In `Example-2` each push of an iOS project will be built on macOS, and additional iOS tools ([SwiftLint](https://github.com/realm/SwiftLint) and [Danger](https://github.com/danger/danger)) will be run in Docker. +In `Example-2` each push of an iOS project will be built on macOS, and additional iOS tools (link:https://github.com/realm/SwiftLint[SwiftLint] and link:https://github.com/danger/danger[Danger]) will be run in Docker. -{:.tab.multiple-executors.Example-1} -```yaml +[tab.multiple-executors.Example_1] +-- +[,yaml] +---- version: 2.1 orbs: @@ -707,11 +719,15 @@ workflows: filters: branches: only: main -``` +---- -{:.tab.multiple-executors.Example-2} {% raw %} -```yaml +-- + +[tab.multiple-executors.Example_2] +-- +[,yaml] +---- version: 2.1 jobs: @@ -762,12 +778,14 @@ workflows: - swiftlint - danger - build-and-test -``` +---- + {% endraw %} +-- -## See also -{: #see-also } +[#see-also] +== See also -* See the [Concepts document]({{ site.baseurl }}/concepts/#configuration) and [Workflows]({{ site.baseurl }}/workflows/) for more details of the concepts covered in this example. -* See the [Configuration Reference]({{ site.baseurl }}/configuration-reference/) document for full details of each individual configuration key. -* See the [Example Public Repos]({{ site.baseurl }}/example-configs/) document for a list of public projects that use CircleCI. +* See the xref:concepts#configuration[Concepts document] and xref:workflows#[Workflows] for more details of the concepts covered in this example. +* See the xref:configuration-reference#[Configuration Reference] document for full details of each individual configuration key. +* See the xref:example-configs#[Example Public Repos] document for a list of public projects that use CircleCI. From 6273e612a5340391b508d439a406991389ec5652 Mon Sep 17 00:00:00 2001 From: Rosie Yohannan Date: Mon, 27 Jan 2025 17:28:46 +0000 Subject: [PATCH 13/20] convert ecs-ecr --- jekyll/_cci2/ecs-ecr.adoc | 152 ++++++++++++++++++++++++++++++++++++++ jekyll/_cci2/ecs-ecr.md | 131 -------------------------------- 2 files changed, 152 insertions(+), 131 deletions(-) create mode 100644 jekyll/_cci2/ecs-ecr.adoc delete mode 100644 jekyll/_cci2/ecs-ecr.md diff --git a/jekyll/_cci2/ecs-ecr.adoc b/jekyll/_cci2/ecs-ecr.adoc new file mode 100644 index 0000000000..1b2a02358b --- /dev/null +++ b/jekyll/_cci2/ecs-ecr.adoc @@ -0,0 +1,152 @@ +--- +layout: classic-docs +contentTags: + platform: + - Cloud + - Server v4+ +--- += Push image to AWS ECR and deploy to AWS ECS +:description: How to use CircleCI to deploy to AWS ECS from ECR +:experimental: +:icons: font +:page-layout: classic-docs +:page-liquid: + +This document describes how to use CircleCI to deploy to Amazon Elastic Container Service (ECS) from Amazon Elastic Container Registry (ECR). + +This page is outdated. CircleCI is working on a new updated sample project. The information on this page is still relevant, but the sample project will be replaced. + +[#overview] +== Overview + +This guide has two phases: + +* Building and pushing a Docker image to AWS ECR +* Deploying the new Docker image to an existing AWS ECS service + +// You can also find the application [building on CircleCI](https://circleci.com/gh/CircleCI-Public/circleci-demo-aws-ecs-ecr){:rel="nofollow"}. + +This project includes a simple https://github.com/CircleCI-Public/circleci-demo-aws-ecs-ecr/blob/master/Dockerfile[Dockerfile]. Visit the link:{{site.baseurl}}/custom-images/#creating-a-custom-image-manually[Creating a custom image manually] page for more information. + +[#prerequisites] +== Prerequisites + +[#use-terraform-to-create-aws-resources] +=== 1. Use Terraform to create AWS resources + +Several AWS resources are required to build and deploy the application in this guide. CircleCI provides https://github.com/CircleCI-Public/circleci-demo-aws-ecs-ecr/tree/master/terraform_setup[several Terraform scripts] to create these resources. To use these scripts, follow the steps below. + +. https://aws.amazon.com/premiumsupport/knowledge-center/create-and-activate-aws-account/[Create an AWS account]. +. https://www.terraform.io/[Install Terraform]. +. Clone the https://github.com/CircleCI-Public/circleci-demo-aws-ecs-ecr[sample project] and go to its root directory. +. Update `~/terraform_setup/terraform.tfvars` with real values for the AWS variables. For more details, see the <> section below. +. Create the AWS resources by running the following commands. + +[,shell] +---- +cd terraform_setup +terraform init +terraform plan # review the plan +terraform apply # apply the plan and create AWS resources +---- + +TIP: You can destroy most AWS resources by running `terraform destroy`. If any resources remain, check the https://console.aws.amazon.com/[AWS Management Console], particularly the *ECS*, *CloudFormation* and *VPC* pages. If `apply` fails, check that the user has permissions for EC2, Elastic Load Balancing, and IAM services. + +[#configure-circleci-environment-variables] +=== 2. Configure CircleCI environment variables + +In the CircleCI application, set the following link:{{ site.baseurl }}/set-environment-variable/#set-an-environment-variable-in-a-project[project environment variables]. + +[.table.table-striped] +[cols=2*, options="header", stripes=even] +|=== +| Variable | Description + +| AWS_ACCESS_KEY_ID +| Security credentials for AWS. + +| AWS_SECRET_ACCESS_KEY +| Security credentials for AWS. + +| AWS_REGION +| Used by the AWS CLI. + +| AWS_ACCOUNT_ID +| Required for deployment. https://docs.aws.amazon.com/IAM/latest/UserGuide/console_account-alias.html#FindingYourAWSId[Find your AWS Account ID]. + +| AWS_RESOURCE_NAME_PREFIX +| Prefix for some required AWS resources. Should correspond to the value of `aws_resource_prefix` in `terraform_setup/terraform.tfvars`. + +| AWS_ECR_REGISTRY_ID +| The 12 digit AWS id associated with the ECR account. +|=== + +[#configuration-walkthrough] +== Configuration walkthrough + +Every CircleCI project requires a configuration file called xref:configuration-reference#[`.circleci/config.yml`]. Follow the steps below to create a complete `config.yml` file. + +NOTE: The sample project described in this section makes use of the CircleCI AWS-ECR and AWS-ECS orbs, which can be found in the CircleCI developer hub using the following links: + +* https://circleci.com/developer/orbs/orb/circleci/aws-ecr[AWS-ECR] +* https://circleci.com/developer/orbs/orb/circleci/aws-ecs[AWS-ECS] + +Notice the orbs are versioned with tags, for example, `aws-ecr: circleci/aws-ecr@x.y.z`. If you copy paste any examples you will need to edit `x.y.z` to specify a version. You can find the available versions listed on the individual orb pages in the link:https://circleci.com/developer/orbs[CircleCI Orbs Registry]. + +[#build-and-push-the-docker-image-to-aws-ecr] +=== 1. Build and push the Docker image to AWS ECR + +The `build-and-push-image` job builds a Docker image from a Dockerfile in the default location (that is, the root of the checkout directory) and pushes it to the specified ECR repository. + +[,yaml] +---- +version: 2.1 + +orbs: + aws-ecr: circleci/aws-ecr@x.y.z + aws-ecs: circleci/aws-ecs@0x.y.z + +workflows: + build-and-deploy: + jobs: + - aws-ecr/build-and-push-image: + repo: "${AWS_RESOURCE_NAME_PREFIX}" + tag: "${CIRCLE_SHA1}" +---- + +[#deploy-the-new-docker-image-to-an-existing-aws-ecs-service] +=== 2. Deploy the new Docker image to an existing AWS ECS service + +The `deploy-service-update` job of the AWS-ECS orb creates a new task definition that is based on the current task definition, but with the new Docker image specified in the task definition's container definitions, and deploys the new task definition to the specified ECS service. If you would like more information about the CircleCI AWS-ECS orb, go to the link:https://circleci.com/developer/orbs/orb/circleci/aws-ecs[CircleCI AWS-ECS orb] page on the developer hub. + +NOTE: The `deploy-service-update` job depends on `build-and-push-image` because of the `requires` key. + +[,yaml] +---- +version: 2.1 + +orbs: + aws-ecr: circleci/aws-ecr@x.y.z + aws-ecs: circleci/aws-ecs@0x.y.z + +workflows: + build-and-deploy: + jobs: + - aws-ecr/build-and-push-image: + repo: "${AWS_RESOURCE_NAME_PREFIX}" + tag: "${CIRCLE_SHA1}" + - aws-ecs/deploy-service-update: + requires: + - aws-ecr/build-and-push-image # only run this job once aws-ecr/build-and-push-image has completed + family: "${AWS_RESOURCE_NAME_PREFIX}-service" + cluster: "${AWS_RESOURCE_NAME_PREFIX}-cluster" + container-image-name-updates: "container=${AWS_RESOURCE_NAME_PREFIX}-service,tag=${CIRCLE_SHA1}" +---- + +Note the use of Workflows to define job run order/concurrency. See the xref:workflows#[Using Workflows to Orchestrate Jobs] page for more information. + +[#see-also] +== See also + +* If you would like to review an example that builds, tests and pushes the Docker image to ECR and then uses the `aws-ecs` orb to deploy the update, go to the link:https://github.com/CircleCI-Public/circleci-demo-aws-ecs-ecr/tree/orbs[AWS-ECS-ECR Orbs] demo page. +* If you would also like to review an example that does **not** use CircleCI orbs, go to the https://github.com/CircleCI-Public/circleci-demo-aws-ecs-ecr/tree/without_orbs[non-orb AWS ECR-ECS demo] page. diff --git a/jekyll/_cci2/ecs-ecr.md b/jekyll/_cci2/ecs-ecr.md deleted file mode 100644 index 4d931fd6c3..0000000000 --- a/jekyll/_cci2/ecs-ecr.md +++ /dev/null @@ -1,131 +0,0 @@ ---- -layout: classic-docs -title: Push image to ECR and deploy to ECS -description: How to use CircleCI to deploy to AWS ECS from ECR -contentTags: - platform: - - Cloud - - Server v4+ ---- - -This document describes how to use CircleCI to deploy to Amazon Elastic Container Service (ECS) from Amazon Elastic Container Registry (ECR). - -This page is outdated. CircleCI is working on a new updated sample project. The information on this page is still relevant, but the sample project will be replaced. -{: class="alert alert-warning" } - -## Overview -{: #overview } - -This guide has two phases: - -- Building and pushing a Docker image to AWS ECR -- Deploying the new Docker image to an existing AWS ECS service - - - -This project includes a simple [Dockerfile](https://github.com/CircleCI-Public/circleci-demo-aws-ecs-ecr/blob/master/Dockerfile). Visit the [Creating a custom image manually]({{site.baseurl}}/custom-images/#creating-a-custom-image-manually) page for more information. - -## Prerequisites -{: #prerequisites } - -### 1. Use Terraform to create AWS resources -{: #use-terraform-to-create-aws-resources } - -Several AWS resources are required to build and deploy the application in this guide. CircleCI provides [several Terraform scripts](https://github.com/CircleCI-Public/circleci-demo-aws-ecs-ecr/tree/master/terraform_setup) to create these resources. To use these scripts, follow the steps below. - -1. [Create an AWS account](https://aws.amazon.com/premiumsupport/knowledge-center/create-and-activate-aws-account/). -2. [Install Terraform](https://www.terraform.io/). -3. Clone the [sample project](https://github.com/CircleCI-Public/circleci-demo-aws-ecs-ecr) and go to its root directory. -4. Update `~/terraform_setup/terraform.tfvars` with real values for the AWS variables. For more details, see the [Configure CircleCI environment variables](#configure-circleci-environment-variables) section below. -5. Create the AWS resources by running the following commands. - -```shell -cd terraform_setup -terraform init -terraform plan # review the plan -terraform apply # apply the plan and create AWS resources -``` - -You can destroy most AWS resources by running `terraform destroy`. If any resources remain, check the [AWS Management Console](https://console.aws.amazon.com/), particularly the **ECS**, **CloudFormation** and **VPC** pages. If `apply` fails, check that the user has permissions for EC2, Elastic Load Balancing, and IAM services. -{: class="alert alert-info" } - -### 2. Configure CircleCI environment variables -{: #configure-circleci-environment-variables } - -In the CircleCI application, set the following [project environment variables]({{ site.baseurl }}/set-environment-variable/#set-an-environment-variable-in-a-project). - -Variable | Description --------------------------|------------ -AWS_ACCESS_KEY_ID | Security credentials for AWS. -AWS_SECRET_ACCESS_KEY | Security credentials for AWS. -AWS_REGION | Used by the AWS CLI. -AWS_ACCOUNT_ID | Required for deployment. [Find your AWS Account ID](https://docs.aws.amazon.com/IAM/latest/UserGuide/console_account-alias.html#FindingYourAWSId). -AWS_RESOURCE_NAME_PREFIX | Prefix for some required AWS resources. Should correspond to the value of `aws_resource_prefix` in `terraform_setup/terraform.tfvars`. -AWS_ECR_REGISTRY_ID | The 12 digit AWS id associated with the ECR account. -{:class="table table-striped"} - -## Configuration walkthrough -{: #configuration-walkthrough } - -Every CircleCI project requires a configuration file called [`.circleci/config.yml`]({{ site.baseurl }}/configuration-reference/). Follow the steps below to create a complete `config.yml` file. - -**Note**: The sample project described in this section makes use of the CircleCI AWS-ECR and AWS-ECS orbs, which can be found here: - - [AWS-ECR](https://circleci.com/developer/orbs/orb/circleci/aws-ecr) - - [AWS-ECS](https://circleci.com/developer/orbs/orb/circleci/aws-ecs) - -Notice the orbs are versioned with tags, for example, `aws-ecr: circleci/aws-ecr@x.y.z`. If you copy paste any examples you will need to edit `x.y.z` to specify a version. You can find the available versions listed on the individual orb pages in the [CircleCI Orbs Registry](https://circleci.com/developer/orbs). - -### 1. Build and push the Docker image to AWS ECR -{: #build-and-push-the-docker-image-to-aws-ecr } - -The `build-and-push-image` job builds a Docker image from a Dockerfile in the default location (i.e. root of the checkout directory) and pushes it to the specified ECR repository. - -```yaml -version: 2.1 - -orbs: - aws-ecr: circleci/aws-ecr@x.y.z - aws-ecs: circleci/aws-ecs@0x.y.z - -workflows: - build-and-deploy: - jobs: - - aws-ecr/build-and-push-image: - repo: "${AWS_RESOURCE_NAME_PREFIX}" - tag: "${CIRCLE_SHA1}" -``` - -### 2. Deploy the new Docker image to an existing AWS ECS service -{: #deploy-the-new-docker-image-to-an-existing-aws-ecs-service } - -The `deploy-service-update` job of the aws-ecs orb creates a new task definition that is based on the current task definition, but with the new Docker image specified in the task definition's container definitions, and deploys the new task definition to the specified ECS service. If you would like more information about the CircleCI AWS-ECS orb, go to: https://circleci.com/developer/orbs/orb/circleci/aws-ecs - -**Note** The `deploy-service-update` job depends on `build-and-push-image` because of the `requires` key. - -```yaml -version: 2.1 - -orbs: - aws-ecr: circleci/aws-ecr@x.y.z - aws-ecs: circleci/aws-ecs@0x.y.z - -workflows: - build-and-deploy: - jobs: - - aws-ecr/build-and-push-image: - repo: "${AWS_RESOURCE_NAME_PREFIX}" - tag: "${CIRCLE_SHA1}" - - aws-ecs/deploy-service-update: - requires: - - aws-ecr/build-and-push-image # only run this job once aws-ecr/build-and-push-image has completed - family: "${AWS_RESOURCE_NAME_PREFIX}-service" - cluster: "${AWS_RESOURCE_NAME_PREFIX}-cluster" - container-image-name-updates: "container=${AWS_RESOURCE_NAME_PREFIX}-service,tag=${CIRCLE_SHA1}" -``` - -Note the use of Workflows to define job run order/concurrency. See the [Using Workflows to Orchestrate Jobs]({{site.baseurl}}/workflows/) page for more information. - -## See also -{: #see-also } -- If you would like to review an example that builds, tests and pushes the Docker image to ECR and then uses the `aws-ecs` orb to deploy the update, go to the [AWS-ECS-ECR Orbs](https://github.com/CircleCI-Public/circleci-demo-aws-ecs-ecr/tree/orbs) demo page. -- If you would also like to review an example that does not use CircleCI orbs, go to the [Non-Orbs AWS ECR-ECS Demo](https://github.com/CircleCI-Public/circleci-demo-aws-ecs-ecr/tree/without_orbs) demo page. From 28dad40ede018c76923e69666f08cc5972da88dc Mon Sep 17 00:00:00 2001 From: Rosie Yohannan Date: Mon, 27 Jan 2025 17:40:25 +0000 Subject: [PATCH 14/20] convert enable-checks --- jekyll/_cci2/enable-checks.adoc | 110 +++++++++++++++++++++ jekyll/_cci2/enable-checks.md | 108 -------------------- styles/config/vocabularies/Docs/accept.txt | 1 + 3 files changed, 111 insertions(+), 108 deletions(-) create mode 100644 jekyll/_cci2/enable-checks.adoc delete mode 100644 jekyll/_cci2/enable-checks.md diff --git a/jekyll/_cci2/enable-checks.adoc b/jekyll/_cci2/enable-checks.adoc new file mode 100644 index 0000000000..2b265419fa --- /dev/null +++ b/jekyll/_cci2/enable-checks.adoc @@ -0,0 +1,110 @@ +--- +layout: classic-docs +contentTags: + platform: + - Cloud +--- += Enable GitHub Checks +:page-description: How to enable GitHub Checks for CircleCI +:experimental: +:icons: font +:page-layout: classic-docs +:page-liquid: + +[#introduction] +== Introduction + +The GitHub Checks feature is not currently supported for projects that have integrated with the CircleCI GitHub App. To find out if you authorized through the GitHub OAuth app or the CircleCI GitHub App, see the xref:github-apps-integration#[GitHub App integration] page. + +This document describes how to enable the GitHub Checks feature and authorize CircleCI to report workflow status to GitHub user interface. *The GitHub Checks integration feature is not currently available on CircleCI server*. + +GitHub Checks provides you with workflow status messages and gives the option to rerun workflows from the GitHub Checks page. + +After GitHub Checks is enabled, CircleCI workflow status is reported under the checks tab on GitHub. + +image::{{site.baseurl}}/assets/img/docs/checks_tab.png[CircleCI checks on GitHub] + +GitHub does not currently provide a granular way for you to rerun workflows. When you select the Re-run checks button, you will automatically re-run all checks, regardless of whether you selected "re-run failed checks" or "rerun all checks" from the Re-run checks button. + +[#github-check-and-github-status-updates] +== GitHub Check and GitHub status updates + +GitHub Checks should not be confused with GitHub status updates: + +* GitHub Checks are administered from the GitHub UI, and are reported in the GitHub UI per _workflow_. +* GitHub status updates are the default way status updates from your builds are reported in the GitHub UI, and they are reported per _job_. + +If both these features are enabled, in a GitHub PR view the Checks tab will show workflow status and the Checks section in the PR conversation view will show job status. + +If you are using GitHuck Checks or GitHub Status updates with the xref:skip-build#skip-jobs[skip jobs feature], +the status of the skipped builds will not be reported even though the checks will be created in GitHub. + +[#prerequisites] +== Prerequisites + +* You must be using the cloud-hosted version of CircleCI. +* Your project must be using xref:workflows#[Workflows]. +* You must be an admin on your GitHub repository to authorize installation of the CircleCI Checks integration. + +[#enable-github-checks] +== Enable GitHub Checks + +. In the CircleCI app sidebar, select *Organization Settings*. +. Select *VCS*. +. Select *Manage GitHub Checks*. ++ +image:{{site.baseurl}}/assets/img/docs/screen_github_checks_new_ui.png[CircleCI VCS Settings Page] +. Select the repositories that should use GitHub Checks and click the *Install* button. + +After installation completes, the Checks tab on the GitHub PR view will be populated with workflow status information, and from here you can rerun workflows or navigate to the CircleCI app to view further information. + +[#checks-status-reporting] +== Checks status reporting + +CircleCI reports the status of workflows and all corresponding jobs under the Checks tab on GitHub. Additionally, GitHub Checks provides a button to rerun all workflows configured for your project. + +After a rerun is initiated, CircleCI reruns the workflows from the start and reports the status in the Checks tab. To navigate to the CircleCI app from GitHub, click the *View more details on CircleCI Checks* link. + +Your project will stop receiving job level status after GitHub Checks is turned on. You can re-enable this in the GitHub Status updates section of the *Project Settings* > *Advanced Settings* page in the CircleCI app. + +[#disable-github-checks-for-a-project] +== Disable GitHub Checks for a project + +To disable the GitHub Checks integration, navigate to the *Organization Settings* page in the CircleCI app, then remove the repositories using GitHub Checks, as follows: + +. Select *Organization Settings* in the CircleCI sidebar. +. Select VCS. +. Select btn:[Manage GitHub Checks]. The "Update CircleCI Checks repository access" page appears. +. Deselect the repository to uninstall the Checks integration. Click Save. If you are removing GitHub Checks from the only repo you have it installed on, you will need to either suspend or uninstall CircleCI Checks. +. Confirm that GitHub status updates have been enabled on your project so you will see job level status within PRs in GitHub. Go to the CircleCI app and navigate to *Project Settings* > *Advanced Settings* and confirm that the setting `GitHub Status Updates` is set to `on`. + +image::{{site.baseurl}}/assets/img/docs/screen_github_checks_disable_new_ui.png[CircleCI VCS Settings Page] + +[#uninstall-checks-for-the-organization] +== Uninstall Checks for the organization + +. Select *Organization Settings* in the CircleCI sidebar. +. Select VCS. +. Select btn:[Manage GitHub Checks]. +. Scroll down and click the Uninstall button to uninstall the GitHub Checks app. + +[#troubleshoot-github-checks-waiting-for-status-in-github] +== Troubleshoot GitHub Checks waiting for status in GitHub + +[,shell] +---- +ci/circleci:build — Waiting for status to be reported +---- + +If you have enabled GitHub Checks in your GitHub repository, but the status check never completes on GitHub Checks tab, there may be status settings in GitHub that you need to deselect. For example, if you choose to protect your branches, you may need to deselect the `ci/circleci:build` status key as this check refers to the job status from CircleCI, as follows: + +image::{{site.baseurl}}/assets/img/docs/github_job_status.png[Uncheck GitHub Job Status Keys] + +Having the `ci/circleci:build` checkbox enabled will prevent the status from showing as completed in GitHub when using a GitHub Check because CircleCI posts statuses to GitHub at a workflow level rather than a job level. + +Go to *Settings* > *Branches* in GitHub and click the *Edit* button on the protected branch to deselect the settings, for example `+https://github.com/your-org/project/settings/branches+`. + +[#next-steps] +== Next steps + +* xref:add-ssh-key#[Add an SSH key to CircleCI] diff --git a/jekyll/_cci2/enable-checks.md b/jekyll/_cci2/enable-checks.md deleted file mode 100644 index 585078a954..0000000000 --- a/jekyll/_cci2/enable-checks.md +++ /dev/null @@ -1,108 +0,0 @@ ---- -layout: classic-docs -title: "Enabling GitHub Checks" -short-title: "Enabling GitHub Checks" -description: "How to enable GitHub Checks for CircleCI" -categories: [getting-started] -contentTags: - platform: - - Cloud ---- - -## Introduction -{: #introduction } - -The GitHub Checks feature is not currently supported for projects that have integrated with the CircleCI GitHub App. To find out if you authorized through the GitHub OAuth app or the CircleCI GitHub App, see the [GitHub App integration](/docs/github-apps-integration/) page. -{: class="alert alert-info" } - -This document describes how to enable the GitHub Checks feature and authorize CircleCI to report workflow status to GitHub user interface. **The GitHub checks integration feature is not currently available on CircleCI server**. - -GitHub Checks provides you with workflow status messages and gives the option to rerun workflows from the GitHub Checks page. - -After GitHub Checks is enabled, CircleCI workflow status is reported under the checks tab on GitHub. - -![CircleCI checks on GitHub]({{site.baseurl}}/assets/img/docs/checks_tab.png) - -GitHub does not currently provide a granular way for you to rerun workflows. When you select the Re-run checks button, you will automatically re-run all checks, regardless of whether you selected "re-run failed checks" or "rerun all checks" from the Re-run checks button. -{: class="alert alert-info" } - -## GitHub Check and GitHub status updates -{: #github-check-and-github-status-updates } - -GitHub Checks should not be confused with GitHub status updates: - -* GitHub Checks are administered from the GitHub UI, and are reported in the GitHub UI per _workflow_. -* GitHub status updates are the default way status updates from your builds are reported in the GitHub UI, and they are reported per _job_. - -If both these features are enabled, in a GitHub PR view the Checks tab will show workflow status and the Checks section in the PR conversation view will show job status. - -If you are using GitHuck Checks or GitHub Status updates with the [skip jobs feature](/docs/skip-build/#skip-jobs), -the status of the skipped builds will not be reported even though the checks will be created in GitHub. - -## Prerequisites -{: #prerequisites } - -- You must be using the cloud-hosted version of CircleCI. -- Your project must be using [Workflows](/docs/workflows/). -- You must be an admin on your GitHub repository to authorize installation of the CircleCI Checks integration. - -## Enable GitHub Checks -{: #enable-github-checks } - -1. In the CircleCI app sidebar, select **Organization Settings**. -2. Select **VCS**. -3. Click the **Manage GitHub Checks** button. -![CircleCI VCS Settings Page]({{site.baseurl}}/assets/img/docs/screen_github_checks_new_ui.png) -4. Select the repositories that should use checks and click the **Install** button. - -After installation completes, the Checks tab on the GitHub PR view will be populated with workflow status information, and from here you can rerun workflows or navigate to the CircleCI app to view further information. - -## Checks status reporting -{: #checks-status-reporting } - -CircleCI reports the status of workflows and all corresponding jobs under the Checks tab on GitHub. Additionally, Checks provides a button to rerun all workflows configured for your project. - -After a rerun is initiated, CircleCI reruns the workflows from the start and reports the status in the Checks tab. To navigate to the CircleCI app from GitHub, click the **View more details on CircleCI Checks** link. - -Your project will stop receiving job level status after GitHub Checks is turned on. You can re-enable this in the GitHub Status updates section of the **Project Settings** > **Advanced Settings** page in the CircleCI app. -{: class="alert alert-info" } - -## Disable GitHub Checks for a project -{: #disable-github-checks-for-a-project } - -To disable the GitHub Checks integration, navigate to the **Organization Settings** page in the CircleCI app, then remove the repositories using GitHub Checks, as follows: - -1. Click the **Organization Settings** option in the CircleCI sidebar. -2. Select VCS. -3. Click the Manage GitHub Checks button. The Update CircleCI Checks repository access page appears. -4. Deselect the repository to uninstall the Checks integration. Click Save. If you are removing GitHub checks from the only repo you have it installed on, you will need to either suspend or uninstall CircleCI Checks. -5. Confirm that GitHub status updates have been enabled on your project so you will see job level status within PRs in GitHub. Go to the CircleCI app and navigate to **Project Settings** > **Advanced Settings** and confirm that the setting `GitHub Status Updates` is set to `on`. - -![CircleCI VCS Settings Page]({{site.baseurl}}/assets/img/docs/screen_github_checks_disable_new_ui.png) - -## Uninstall Checks for the organization -{: #uninstall-checks-for-the-organization } - -1. Click the **Organization Settings** option in the CircleCI sidebar. -2. Select VCS. -3. Click the Manage GitHub Checks button. -4. Scroll down and click the Uninstall button to uninstall the GitHub Checks app. - - -## Troubleshoot GitHub Checks waiting for status in GitHub -{: #troubleshoot-github-checks-waiting-for-status-in-github } - -`ci/circleci:build — Waiting for status to be reported` - -If you have enabled GitHub Checks in your GitHub repository, but the status check never completes on GitHub Checks tab, there may be status settings in GitHub that you need to deselect. For example, if you choose to protect your branches, you may need to deselect the `ci/circleci:build` status key as this check refers to the job status from CircleCI, as follows: - -![Uncheck GitHub Job Status Keys]({{site.baseurl}}/assets/img/docs/github_job_status.png) - -Having the `ci/circleci:build` checkbox enabled will prevent the status from showing as completed in GitHub when using a GitHub Check because CircleCI posts statuses to GitHub at a workflow level rather than a job level. - -Go to **Settings** > **Branches** in GitHub and click the **Edit** button on the protected branch to deselect the settings, for example `https://github.com/your-org/project/settings/branches`. - -## Next steps -{: #next-steps } - -- [Add an SSH key to CircleCI](/docs/add-ssh-key) diff --git a/styles/config/vocabularies/Docs/accept.txt b/styles/config/vocabularies/Docs/accept.txt index b1f77ce38e..f6e6910d8b 100644 --- a/styles/config/vocabularies/Docs/accept.txt +++ b/styles/config/vocabularies/Docs/accept.txt @@ -132,6 +132,7 @@ GCP geckodriver Gemfile GitHub +\bGitHub Checks\b \bGitLab self-managed\b GKE \bGoogle Cloud\b From 7d8a0397b55c4012f47810a1165421c2f65aefc8 Mon Sep 17 00:00:00 2001 From: Rosie Yohannan Date: Mon, 27 Jan 2025 19:03:06 +0000 Subject: [PATCH 15/20] revert changes to enable-checks because another team with working on the same page --- jekyll/_cci2/enable-checks.adoc | 110 -------------------------------- jekyll/_cci2/enable-checks.md | 108 +++++++++++++++++++++++++++++++ 2 files changed, 108 insertions(+), 110 deletions(-) delete mode 100644 jekyll/_cci2/enable-checks.adoc create mode 100644 jekyll/_cci2/enable-checks.md diff --git a/jekyll/_cci2/enable-checks.adoc b/jekyll/_cci2/enable-checks.adoc deleted file mode 100644 index 2b265419fa..0000000000 --- a/jekyll/_cci2/enable-checks.adoc +++ /dev/null @@ -1,110 +0,0 @@ ---- -layout: classic-docs -contentTags: - platform: - - Cloud ---- -= Enable GitHub Checks -:page-description: How to enable GitHub Checks for CircleCI -:experimental: -:icons: font -:page-layout: classic-docs -:page-liquid: - -[#introduction] -== Introduction - -The GitHub Checks feature is not currently supported for projects that have integrated with the CircleCI GitHub App. To find out if you authorized through the GitHub OAuth app or the CircleCI GitHub App, see the xref:github-apps-integration#[GitHub App integration] page. - -This document describes how to enable the GitHub Checks feature and authorize CircleCI to report workflow status to GitHub user interface. *The GitHub Checks integration feature is not currently available on CircleCI server*. - -GitHub Checks provides you with workflow status messages and gives the option to rerun workflows from the GitHub Checks page. - -After GitHub Checks is enabled, CircleCI workflow status is reported under the checks tab on GitHub. - -image::{{site.baseurl}}/assets/img/docs/checks_tab.png[CircleCI checks on GitHub] - -GitHub does not currently provide a granular way for you to rerun workflows. When you select the Re-run checks button, you will automatically re-run all checks, regardless of whether you selected "re-run failed checks" or "rerun all checks" from the Re-run checks button. - -[#github-check-and-github-status-updates] -== GitHub Check and GitHub status updates - -GitHub Checks should not be confused with GitHub status updates: - -* GitHub Checks are administered from the GitHub UI, and are reported in the GitHub UI per _workflow_. -* GitHub status updates are the default way status updates from your builds are reported in the GitHub UI, and they are reported per _job_. - -If both these features are enabled, in a GitHub PR view the Checks tab will show workflow status and the Checks section in the PR conversation view will show job status. - -If you are using GitHuck Checks or GitHub Status updates with the xref:skip-build#skip-jobs[skip jobs feature], -the status of the skipped builds will not be reported even though the checks will be created in GitHub. - -[#prerequisites] -== Prerequisites - -* You must be using the cloud-hosted version of CircleCI. -* Your project must be using xref:workflows#[Workflows]. -* You must be an admin on your GitHub repository to authorize installation of the CircleCI Checks integration. - -[#enable-github-checks] -== Enable GitHub Checks - -. In the CircleCI app sidebar, select *Organization Settings*. -. Select *VCS*. -. Select *Manage GitHub Checks*. -+ -image:{{site.baseurl}}/assets/img/docs/screen_github_checks_new_ui.png[CircleCI VCS Settings Page] -. Select the repositories that should use GitHub Checks and click the *Install* button. - -After installation completes, the Checks tab on the GitHub PR view will be populated with workflow status information, and from here you can rerun workflows or navigate to the CircleCI app to view further information. - -[#checks-status-reporting] -== Checks status reporting - -CircleCI reports the status of workflows and all corresponding jobs under the Checks tab on GitHub. Additionally, GitHub Checks provides a button to rerun all workflows configured for your project. - -After a rerun is initiated, CircleCI reruns the workflows from the start and reports the status in the Checks tab. To navigate to the CircleCI app from GitHub, click the *View more details on CircleCI Checks* link. - -Your project will stop receiving job level status after GitHub Checks is turned on. You can re-enable this in the GitHub Status updates section of the *Project Settings* > *Advanced Settings* page in the CircleCI app. - -[#disable-github-checks-for-a-project] -== Disable GitHub Checks for a project - -To disable the GitHub Checks integration, navigate to the *Organization Settings* page in the CircleCI app, then remove the repositories using GitHub Checks, as follows: - -. Select *Organization Settings* in the CircleCI sidebar. -. Select VCS. -. Select btn:[Manage GitHub Checks]. The "Update CircleCI Checks repository access" page appears. -. Deselect the repository to uninstall the Checks integration. Click Save. If you are removing GitHub Checks from the only repo you have it installed on, you will need to either suspend or uninstall CircleCI Checks. -. Confirm that GitHub status updates have been enabled on your project so you will see job level status within PRs in GitHub. Go to the CircleCI app and navigate to *Project Settings* > *Advanced Settings* and confirm that the setting `GitHub Status Updates` is set to `on`. - -image::{{site.baseurl}}/assets/img/docs/screen_github_checks_disable_new_ui.png[CircleCI VCS Settings Page] - -[#uninstall-checks-for-the-organization] -== Uninstall Checks for the organization - -. Select *Organization Settings* in the CircleCI sidebar. -. Select VCS. -. Select btn:[Manage GitHub Checks]. -. Scroll down and click the Uninstall button to uninstall the GitHub Checks app. - -[#troubleshoot-github-checks-waiting-for-status-in-github] -== Troubleshoot GitHub Checks waiting for status in GitHub - -[,shell] ----- -ci/circleci:build — Waiting for status to be reported ----- - -If you have enabled GitHub Checks in your GitHub repository, but the status check never completes on GitHub Checks tab, there may be status settings in GitHub that you need to deselect. For example, if you choose to protect your branches, you may need to deselect the `ci/circleci:build` status key as this check refers to the job status from CircleCI, as follows: - -image::{{site.baseurl}}/assets/img/docs/github_job_status.png[Uncheck GitHub Job Status Keys] - -Having the `ci/circleci:build` checkbox enabled will prevent the status from showing as completed in GitHub when using a GitHub Check because CircleCI posts statuses to GitHub at a workflow level rather than a job level. - -Go to *Settings* > *Branches* in GitHub and click the *Edit* button on the protected branch to deselect the settings, for example `+https://github.com/your-org/project/settings/branches+`. - -[#next-steps] -== Next steps - -* xref:add-ssh-key#[Add an SSH key to CircleCI] diff --git a/jekyll/_cci2/enable-checks.md b/jekyll/_cci2/enable-checks.md new file mode 100644 index 0000000000..777de30986 --- /dev/null +++ b/jekyll/_cci2/enable-checks.md @@ -0,0 +1,108 @@ +--- +layout: classic-docs +title: "Enabling GitHub Checks" +short-title: "Enabling GitHub Checks" +description: "How to enable GitHub Checks for CircleCI" +categories: [getting-started] +contentTags: + platform: + - Cloud +--- + +## Introduction +{: #introduction } + +The GitHub Checks feature is not currently supported for projects that have integrated with the CircleCI GitHub App. To find out if you authorized through the GitHub OAuth app or the CircleCI GitHub App, see the [GitHub App integration](/docs/github-apps-integration/) page. +{: class="alert alert-info" } + +This document describes how to enable the GitHub Checks feature and authorize CircleCI to report workflow status to GitHub user interface. **The GitHub checks integration feature is not currently available on CircleCI server**. + +GitHub Checks provides you with workflow status messages and gives the option to rerun workflows from the GitHub Checks page. + +After GitHub Checks is enabled, CircleCI workflow status is reported under the checks tab on GitHub. + +![CircleCI checks on GitHub]({{site.baseurl}}/assets/img/docs/checks_tab.png) + +GitHub does not currently provide a granular way for you to rerun workflows. When you select the Re-run checks button, you will automatically re-run all checks, regardless of whether you selected "re-run failed checks" or "rerun all checks" from the Re-run checks button. +{: class="alert alert-info" } + +## GitHub Check and GitHub status updates +{: #github-check-and-github-status-updates } + +GitHub Checks should not be confused with GitHub status updates: + +* GitHub Checks are administered from the GitHub UI, and are reported in the GitHub UI per _workflow_. +* GitHub status updates are the default way status updates from your builds are reported in the GitHub UI, and they are reported per _job_. + +If both these features are enabled, in a GitHub PR view the Checks tab will show workflow status and the Checks section in the PR conversation view will show job status. + +If you are using GitHuck Checks or GitHub Status updates with the [skip jobs feature](/docs/skip-build/#skip-jobs), +the status of the skipped builds will not be reported even though the checks will be created in GitHub. + +## Prerequisites +{: #prerequisites } + +- You must be using the cloud-hosted version of CircleCI. +- Your project must be using [Workflows](/docs/workflows/). +- You must be an admin on your GitHub repository to authorize installation of the CircleCI Checks integration. + +## Enable GitHub Checks +{: #enable-github-checks } + +1. In the CircleCI app sidebar, select **Organization Settings**. +2. Select **VCS**. +3. Click the **Manage GitHub Checks** button. +![CircleCI VCS Settings Page]({{site.baseurl}}/assets/img/docs/screen_github_checks_new_ui.png) +4. Select the repositories that should use checks and click the **Install** button. + +After installation completes, the Checks tab on the GitHub PR view will be populated with workflow status information, and from here you can rerun workflows or navigate to the CircleCI app to view further information. + +## Checks status reporting +{: #checks-status-reporting } + +CircleCI reports the status of workflows and all corresponding jobs under the Checks tab on GitHub. Additionally, Checks provides a button to rerun all workflows configured for your project. + +After a rerun is initiated, CircleCI reruns the workflows from the start and reports the status in the Checks tab. To navigate to the CircleCI app from GitHub, click the **View more details on CircleCI Checks** link. + +Your project will stop receiving job level status after GitHub Checks is turned on. You can re-enable this in the GitHub Status updates section of the **Project Settings** > **Advanced Settings** page in the CircleCI app. +{: class="alert alert-info" } + +## Disable GitHub Checks for a project +{: #disable-github-checks-for-a-project } + +To disable the GitHub Checks integration, navigate to the **Organization Settings** page in the CircleCI app, then remove the repositories using GitHub Checks, as follows: + +1. Click the **Organization Settings** option in the CircleCI sidebar. +2. Select VCS. +3. Click the Manage GitHub Checks button. The Update CircleCI Checks repository access page appears. +4. Deselect the repository to uninstall the Checks integration. Click Save. If you are removing GitHub checks from the only repo you have it installed on, you will need to either suspend or uninstall CircleCI Checks. +5. Confirm that GitHub status updates have been enabled on your project so you will see job level status within PRs in GitHub. Go to the CircleCI app and navigate to **Project Settings** > **Advanced Settings** and confirm that the setting `GitHub Status Updates` is set to `on`. + +![CircleCI VCS Settings Page]({{site.baseurl}}/assets/img/docs/screen_github_checks_disable_new_ui.png) + +## Uninstall Checks for the organization +{: #uninstall-checks-for-the-organization } + +1. Click the **Organization Settings** option in the CircleCI sidebar. +2. Select VCS. +3. Click the Manage GitHub Checks button. +4. Scroll down and click the Uninstall button to uninstall the GitHub Checks app. + + +## Troubleshoot GitHub Checks waiting for status in GitHub +{: #troubleshoot-github-checks-waiting-for-status-in-github } + +`ci/circleci:build — Waiting for status to be reported` + +If you have enabled GitHub Checks in your GitHub repository, but the status check never completes on GitHub Checks tab, there may be status settings in GitHub that you need to deselect. For example, if you choose to protect your branches, you may need to deselect the `ci/circleci:build` status key as this check refers to the job status from CircleCI, as follows: + +![Uncheck GitHub Job Status Keys]({{site.baseurl}}/assets/img/docs/github_job_status.png) + +Having the `ci/circleci:build` checkbox enabled will prevent the status from showing as completed in GitHub when using a GitHub Check because CircleCI posts statuses to GitHub at a workflow level rather than a job level. + +Go to **Settings** > **Branches** in GitHub and click the **Edit** button on the protected branch to deselect the settings, for example `https://github.com/your-org/project/settings/branches`. + +## Next steps +{: #next-steps } + +- [Add an SSH key to CircleCI](/docs/add-ssh-key) \ No newline at end of file From 7d30c55db554ef9ad11b36fc5e90043f61377874 Mon Sep 17 00:00:00 2001 From: Rosie Yohannan Date: Mon, 27 Jan 2025 22:46:09 +0000 Subject: [PATCH 16/20] convert env var page --- jekyll/_cci2/env-vars.adoc | 296 +++++++++++++++++++++++++++++++++++++ jekyll/_cci2/env-vars.md | 287 ----------------------------------- 2 files changed, 296 insertions(+), 287 deletions(-) create mode 100644 jekyll/_cci2/env-vars.adoc delete mode 100644 jekyll/_cci2/env-vars.md diff --git a/jekyll/_cci2/env-vars.adoc b/jekyll/_cci2/env-vars.adoc new file mode 100644 index 0000000000..e537187ee3 --- /dev/null +++ b/jekyll/_cci2/env-vars.adoc @@ -0,0 +1,296 @@ +--- +contentTags: + platform: + - Cloud + - Server v4+ +--- += Introduction to environment variables +:page-description: Introduction to environment variables in CircleCI +:experimental: +:icons: font +:page-layout: classic-docs +:page-liquid: + +[#introduction] +== Introduction + +Use environment variables to set up various configuration options, and keep your set-up secure with secrets, private keys, and contexts. Environment variables in CircleCI are governed by an <>, allowing control at each level in your configuration. + +See the xref:set-environment-variable#[Set an environment variable] page for guidance on the different ways to set an environment variable. + +If you have existing environment variables (or contexts) and you would like to rename your organization or repository, follow the xref:rename-organizations-and-repositories#[Rename organizations and repositories] guide to make sure you do not lose access to environment variables or contexts in the process. + +[#built-in-environment-variables] +== Built-in environment variables + +All projects have access to CircleCI's built-in environment variables. These environment variables are scoped at the job level, so they can be used with the `context` key in a job, but they do not exist at a pipeline level. + +For a full list of built-in environment variables, see the xref:variables#built-in-environment-variables[Project values and variables] page. + +[#private-keys-and-secrets] +== Private keys and secrets + +To add private keys or secrets as environment variables for use throughout your project, navigate to menu:Project Settings[Environment Variables] in the link:https://app.circleci.com/[CircleCI web app]. You can find step-by-step instructions of this process on the xref:set-environment-variable#set-an-environment-variable-in-a-project[Environment variables] page. The variable values are neither readable nor editable in the app after they are set. To change the value of an environment variable, delete the current variable, and add it again with the new value. + +Private environment variables enable you to store secrets safely, even when your project is public. Refer to the xref:oss#[Building open source projects] page for associated security and settings information. + +{% include snippets/secrets-masking.adoc %} + +[#environment-variable-usage-options] +== Environment variable usage options + +CircleCI uses Bash, which follows the POSIX naming convention for environment variables. Valid characters include letters (uppercase and lowercase), digits, and the underscore. The first character of each environment variable name must be an alpha character or an underscore, *not* a digit. + +[#order-of-precedence] +=== Order of precedence + +Environment variables are used according to a specific precedence order, as follows: + +. Environment variables declared xref:set-environment-variable#set-an-environment-variable-in-a-shell-command[inside a shell command] in a `run` step, for example `FOO=bar make install`. +. Environment variables declared with the `environment` key xref:set-environment-variable#set-an-environment-variable-in-a-step[for a `run` step]. +. Environment variables set with the `environment` key link:set-environment-variable#set-an-environment-variable-in-a-job[for a job]. +. Special CircleCI environment variables defined in the xref:variables#built-in-environment-variables[CircleCI Built-in environment variables] document. +. Context environment variables (assuming the user has access to the context). See the xref:contexts#[Contexts] documentation for more information. +. xref:set-environment-variable#set-an-environment-variable-in-a-project[Project-level environment variables] set on the *Project Settings* page in the web app. + +Environment variables declared inside a shell command `run step`, for example `FOO=bar make install`, will override environment variables declared with the `environment` and `contexts` keys. Environment variables added on the *Contexts* page in the web app will take precedence over variables added on the *Project Settings* page. + +image::{{site.baseurl}}/assets/img/docs/env-var-order.png[Environment variable order of precedence] + +[#example-configuration-of-environment-variables] +=== Example configuration of environment variables + +Consider the example `.circleci/config.yml` below: + +[,yaml] +---- +version: 2.1 + +jobs: # basic units of work in a run + build: + docker: # use the Docker executor + # CircleCI Node images available at: https://circleci.com/developer/images/image/cimg/node + - image: cimg/node:18.11.0 + steps: # steps that comprise the `build` job + - checkout # check out source code to working directory + # Run a step to setup an environment variable + # Redirect MY_ENV_VAR into $BASH_ENV + - run: + name: "Setup custom environment variables" + command: echo 'export MY_ENV_VAR="FOO"' >> "$BASH_ENV" + - run: # print the name of the branch we're on + name: "What branch am I on?" + command: echo ${CIRCLE_BRANCH} + # Run another step, the same as above; note that you can + # invoke environment variable without curly braces. + - run: + name: "What branch am I on now?" + command: echo $CIRCLE_BRANCH + - run: + name: "What was my custom environment variable?" + command: echo ${MY_ENV_VAR} + - run: + name: "Print an env var stored in the Project" + command: echo ${PROJECT_ENV_VAR} + - run: + name: "Print an env var stored in a Context" + command: echo ${CONTEXT_ENV_VAR} + +workflows: # a single workflow with a single job called build + build: + jobs: + - build: + context: Testing-Env-Vars +---- + +The above `.circleci/config.yml` demonstrates the following: + +* Setting custom environment variables. +* Reading a built-in environment variable that CircleCI provides (`CIRCLE_BRANCH`). +* How variables are used (or interpolated) in your `.circleci/config.yml`. +* Secrets masking, applied to environment variable set in the project or within a context. + +When the above configuration runs, the output looks like the image below. Notice the environment variables stored in the project is masked, and displays as `+****+`: + +image::{{site.baseurl}}/assets/img/docs/env-vars-example-ui.png[Environment variable interpolation example] + +Notice there are two similar steps in the above image and configuration - "What branch am I on?" These steps illustrate two different methods to read environment variables. + +In the example configuration above, two syntax variants are used: `+${VAR}+` and `$VAR`. Both syntax variants are supported. You can read more about shell parameter expansion in the link:https://www.gnu.org/software/bash/manual/bashref.html#Shell-Parameter-Expansion[Bash documentation]. + +[#parameters-and-bash-environment] +=== Parameters and the Bash environment + +In general, CircleCI does not support interpolating environment variables in the configuration. Values used are treated as literals. This can cause issues when defining `working_directory`, modifying `PATH`, and sharing variables across multiple `run` steps. + +In the example below, `$ORGNAME` and `$REPONAME` will not be interpolated. + +[,yaml] +---- +working_directory: /go/src/github.com/$ORGNAME/$REPONAME +---- + +NOTE: An exception to this interpolation rule is using project environment variables to pull xref:private-images#[private images]. + +You can reuse pieces of configuration across your `.circleci/config.yml` file. By using the `parameters` declaration, you can pass values into reusable `commands`, `jobs`, and `executors`: + +[,yaml] +---- +version: 2.1 # version 2.1 is required for reusing configuration + +jobs: + build: + parameters: + org_name: + type: string + default: my_org + repo_name: + type: string + default: my_repo + docker: + - image: cimg/go:1.17.3 + steps: + - run: echo "project directory is go/src/github.com/<< parameters.org_name >>/<< parameters.repo_name >>" + +workflows: + my_workflow: + jobs: + - build: + org_name: my_organization + repo_name: project1 + + - build: + org_name: my_organization + repo_name: project2 +---- + +For more information, read the documentation on xref:reusing-config#using-the-parameters-declaration[using the parameters declaration]. + +Another possible method to interpolate values into your configuration is to use a `run` step to export environment variables to `BASH_ENV`, as shown below. + +CAUTION: The `$BASH_ENV` workaround only works with `bash`, and has not been confirmed to work with other shells. + +[,yaml] +---- +steps: + - run: + name: Setup Environment Variables + command: | + echo 'export PATH="$GOPATH"/bin:"$PATH"' >> "$BASH_ENV" + echo 'export GIT_SHA1="$CIRCLE_SHA1"' >> "$BASH_ENV" +---- + +In every step, CircleCI uses `bash` to source `BASH_ENV`. This means that `BASH_ENV` is automatically loaded and run, allowing you to use interpolation and share environment variables across `run` steps. + +[#environment-variable-substitution] +=== Environment variable substitution + +The CircleCI CLI offers a wrapper around the link:https://github.com/a8m/envsubst[`envsubst`] tool, available both locally as well as in all jobs running on CircleCI. `envsubst` is a command-line utility used to replace environment variables in text strings. + +CLI command: + +[,sh] +---- +circleci env subst +---- + +[#env-subst-usage] +==== Usage + +The `circleci env subst` command can accept text input from `stdin` or as an argument. + +Within your repository create a file such as `template.json`, with value replaced by environment variable strings + +[,json] +---- +{ + "foo": "$FOO", + "provider": "${PROVIDER}" +} +---- + +`envsubst` can convert all types of environment variable strings, including those encased in curly braces (`{}`). + +The config example below shows the corresponding environment variables as if they were defined directly within a step in the config. However, we strongly recommend creating the environment variables in the CircleCI app, either in xref:set-environment-variable#set-an-environment-variable-in-a-project[Project Settings] or as a xref:contexts#[context]. + +[,yaml] +---- +version: 2.1 +jobs: + process-template: + docker: + - image: cimg/base:current + steps: + - checkout + - run: + name: Process template file + environment: + # Environment variables would typically be served via a context + FOO: bar + PROVIDER: circleci + command: | + circleci env subst < template.json > deploy.json + cat deploy.json +workflows: + env-subst-workflow: + jobs: + - process-template +---- + +In this example, the `<` symbol is used to redirect the contents of the `template.json` file as _input_ to the `env subst` command, while the `>` symbol is used to redirect the output of the `env subst` command to the `deploy.json`. + +You could alternatively pass input to the `circleci env subst` command as an argument: `circleci env subst "hello \$WORLD"` + +Output: + +[,sh] +---- +{ + "foo": "bar", + "provider": "circleci" +} +---- + +For instructions on installing the CircleCI CLI locally, read the xref:local-cli#[Installing the CircleCI local CLI] guide. + +[#alpine-linux] +=== Alpine Linux + +An image that has been based on link:https://alpinelinux.org/[Alpine Linux] (like link:https://hub.docker.com/_/docker[Docker]), uses the `ash` shell. + +To use environment variables with `bash`, add the `shell` and `environment` keys to your job. + +[,yaml] +---- +version: 2.1 + +jobs: + build: + shell: /bin/sh -leo pipefail + environment: + BASH_ENV: /etc/profile +---- + +[#notes-on-security] +== Notes on security + +Do not add secrets or keys inside the `.circleci/config.yml` file. The full text of `.circleci/config.yml` is visible to developers with access to your project on CircleCI. Store secrets or keys in the CircleCI web app under one of the following: + +* In xref:set-environment-variable#set-an-environment-variable-in-a-project[Project settings]. +* In a xref:set-environment-variable#set-an-environment-variable-in-a-context[context]. + +For more information, see the xref:security#encryption[Encryption] section of the security page. + +Running scripts within configuration may expose secret environment variables. See the xref:using-shell-scripts#shell-script-best-practices[Using shell scripts] page for best practices for secure scripts. + +[#contexts] +== Contexts + +You can further restrict access to environment variables using xref:contexts#[contexts]. Contexts are set from the *Organization Settings* in the CircleCI web app. + +[#see-also] +== See also + +* xref:security-recommendations#[Security recommendations] +* xref:set-environment-variable#[Set an environment variable] +* xref:inject-environment-variables-with-api#[Inject variables using the CircleCI API] diff --git a/jekyll/_cci2/env-vars.md b/jekyll/_cci2/env-vars.md deleted file mode 100644 index 7f35e56475..0000000000 --- a/jekyll/_cci2/env-vars.md +++ /dev/null @@ -1,287 +0,0 @@ ---- -layout: classic-docs -title: Introduction to environment variables -short-title: Environment variables -description: Introduction to environment variables in CircleCI -contentTags: - platform: - - Cloud - - Server v4+ -suggested: - - title: Keep environment variables private - link: https://circleci.com/blog/keep-environment-variables-private-with-secret-masking/ - - title: Troubleshoot environment variables settings - link: https://discuss.circleci.com/t/somehow-i-just-cannot-get-the-enviroment-variable-from-enviroment-variable-setting-an-context-in-organization-menu/40342 - - title: Insert files as environment variables - link: https://support.circleci.com/hc/en-us/articles/360003540393?input_string=how+to+i+inject+an+environment+variable+using+the+api%3F ---- - -## Introduction -{: #introduction } - -Use environment variables to set up various configuration options, and keep your set-up secure with secrets, private keys, and contexts. Environment variables in CircleCI are governed by an [order of precedence](#order-of-precedence), allowing control at each level in your configuration. See the [Set an environment variable](/docs/set-environment-variable/) page for guidance on the different ways to set an environment variable. - -If you have existing environment variables (or contexts) and you would like to rename your organization or repository, please follow the [Rename organizations and repositories]({{site.baseurl}}/rename-organizations-and-repositories) guide to make sure you do not lose access to environment variables or contexts in the process. - -## Built-in environment variables -{: #built-in-environment-variables } - -All projects have access to CircleCI's built-in environment variables. These environment variables are scoped at the job level, so they can be used with the `context` key in a job, but they do not exist at a pipeline level. - -For a full list of built-in environment variables, see the [Project values and variables]({{site.baseurl}}/variables#built-in-environment-variables) page. - -## Private keys and secrets -{: #private-keys-and-secrets } - -To add private keys or secrets as environment variables for use throughout your project, navigate to **Project Settings > Environment Variables** in the [CircleCI web app](https://app.circleci.com/). You can find step-by-step instructions of this process on the [Environment variables]({{site.baseurl}}/set-environment-variable/#set-an-environment-variable-in-a-project) page. The variable values are neither readable nor editable in the app after they are set. To change the value of an environment variable, delete the current variable, and add it again with the new value. - -Private environment variables enable you to store secrets safely, even when your project is public. Refer to the [Building open source projects]({{site.baseurl}}/oss/) page for associated security and settings information. - -{% include snippets/secrets-masking.md %} - -## Environment variable usage options -{: #environment-variable-usage-options } - -CircleCI uses Bash, which follows the POSIX naming convention for environment variables. Valid characters include letters (uppercase and lowercase), digits, and the underscore. The first character of each environment variable name must be an alpha character or an underscore, **not** a digit. - -### Order of precedence -{: #order-of-precedence } - -Environment variables are used according to a specific precedence order, as follows: - -1. Environment variables declared [inside a shell command]({{site.baseurl}}/set-environment-variable/#set-an-environment-variable-in-a-shell-command) in a `run` step, for example `FOO=bar make install`. -2. Environment variables declared with the `environment` key [for a `run` step]({{site.baseurl}}/set-environment-variable/#set-an-environment-variable-in-a-step). -3. Environment variables set with the `environment` key [for a job]({{site.baseurl}}/set-environment-variable/#set-an-environment-variable-in-a-job). -4. Special CircleCI environment variables defined in the [CircleCI Built-in environment variables]({{site.baseurl}}/variables#built-in-environment-variables) document. -5. Context environment variables (assuming the user has access to the context). See the [Contexts]({{site.baseurl}}/contexts/) documentation for more information. -6. [Project-level environment variables]({{site.baseurl}}/set-environment-variable/#set-an-environment-variable-in-a-project) set on the **Project Settings** page in the web app. - -Environment variables declared inside a shell command `run step`, for example `FOO=bar make install`, will override environment variables declared with the `environment` and `contexts` keys. Environment variables added on the **Contexts** page in the web app will take precedence over variables added on the **Project Settings** page. - -![Environment variable order of precedence]({{site.baseurl}}/assets/img/docs/env-var-order.png) - -### Example configuration of environment variables -{: #example-configuration-of-environment-variables } - -Consider the example `.circleci/config.yml` below: - -```yaml -version: 2.1 - -jobs: # basic units of work in a run - build: - docker: # use the Docker executor - # CircleCI Node images available at: https://circleci.com/developer/images/image/cimg/node - - image: cimg/node:18.11.0 - steps: # steps that comprise the `build` job - - checkout # check out source code to working directory - # Run a step to setup an environment variable - # Redirect MY_ENV_VAR into $BASH_ENV - - run: - name: "Setup custom environment variables" - command: echo 'export MY_ENV_VAR="FOO"' >> "$BASH_ENV" - - run: # print the name of the branch we're on - name: "What branch am I on?" - command: echo ${CIRCLE_BRANCH} - # Run another step, the same as above; note that you can - # invoke environment variable without curly braces. - - run: - name: "What branch am I on now?" - command: echo $CIRCLE_BRANCH - - run: - name: "What was my custom environment variable?" - command: echo ${MY_ENV_VAR} - - run: - name: "Print an env var stored in the Project" - command: echo ${PROJECT_ENV_VAR} - - run: - name: "Print an env var stored in a Context" - command: echo ${CONTEXT_ENV_VAR} - -workflows: # a single workflow with a single job called build - build: - jobs: - - build: - context: Testing-Env-Vars -``` - -The above `.circleci/config.yml` demonstrates the following: - -- Setting custom environment variables -- Reading a built-in environment variable that CircleCI provides (`CIRCLE_BRANCH`) -- How variables are used (or interpolated) in your `.circleci/config.yml` -- Secrets masking, applied to environment variable set in the project or within a context. - -When the above configuration runs, the output looks like the below image. Notice the environment variables stored in the project is masked, and displays as `****`: - -![Environment variable interpolation example]({{site.baseurl}}/assets/img/docs/env-vars-example-ui.png) - -Notice there are two similar steps in the above image and configuration - "What branch am I on?" These steps illustrate two different methods to read environment variables. - -In the example configuration above, two syntaxes are used: `${VAR}` and `$VAR`. Both syntaxes are supported. You can read more about shell parameter expansion in the [Bash documentation](https://www.gnu.org/software/bash/manual/bashref.html#Shell-Parameter-Expansion). - -### Parameters and bash environment -{: #parameters-and-bash-environment } - -In general, CircleCI does not support interpolating environment variables in the configuration. Values used are treated as literals. This can cause issues when defining `working_directory`, modifying `PATH`, and sharing variables across multiple `run` steps. - -In the example below, `$ORGNAME` and `$REPONAME` will not be interpolated. - -```yaml -working_directory: /go/src/github.com/$ORGNAME/$REPONAME -``` - -An exception to this rule is using project environment variables to pull [private images]({{site.baseurl}}/private-images/). -{: class="alert alert-info" } - -You can reuse pieces of configuration across your `.circleci/config.yml` file. By using the `parameters` declaration, you can pass values into reusable `commands`, `jobs`, and `executors`: - -```yaml -version: 2.1 # version 2.1 is required for reusing configuration - -jobs: - build: - parameters: - org_name: - type: string - default: my_org - repo_name: - type: string - default: my_repo - docker: - - image: cimg/go:1.17.3 - steps: - - run: echo "project directory is go/src/github.com/<< parameters.org_name >>/<< parameters.repo_name >>" - -workflows: - my_workflow: - jobs: - - build: - org_name: my_organization - repo_name: project1 - - - build: - org_name: my_organization - repo_name: project2 -``` - -For more information, read the documentation on [using the parameters declaration]({{site.baseurl}}/reusing-config/#using-the-parameters-declaration). - -Another possible method to interpolate values into your configuration is to use a `run` step to export environment variables to `BASH_ENV`, as shown below. - -The `$BASH_ENV` workaround only works with `bash`, and has not been confirmed to work with other shells. -{: class="alert alert-info"} - -```yaml -steps: - - run: - name: Setup Environment Variables - command: | - echo 'export PATH="$GOPATH"/bin:"$PATH"' >> "$BASH_ENV" - echo 'export GIT_SHA1="$CIRCLE_SHA1"' >> "$BASH_ENV" -``` - -In every step, CircleCI uses `bash` to source `BASH_ENV`. This means that `BASH_ENV` is automatically loaded and run, allowing you to use interpolation and share environment variables across `run` steps. - -### Environment variable substitution -{: #environment-variable-substitution } - -The CircleCI CLI offers a wrapper around the [`envsubst`](https://github.com/a8m/envsubst) tool, available both locally as well as in all jobs running on CircleCI. `envsubst` is a command-line utility used to replace environment variables in text strings. - -CLI command: - -```sh -circleci env subst -``` - -#### Usage -{: #env-subst-usage } - -The `circleci env subst` command can accept text input from `stdin` or as an argument. - -Within your repository create a file such as `template.json`, with value replaced by environment variable strings - -```json -{ - "foo": "$FOO", - "provider": "${PROVIDER}" -} -``` - -`envsubst` can convert all types of environment variable strings, including those encased in curly braces (`{}`). - -The config example below shows the corresponding environment variables as if they were defined directly within a step in the config. However, we strongly recommend creating the environment variables in the CircleCI app, either in [Project Settings](/docs/set-environment-variable/#set-an-environment-variable-in-a-project) or as a [context](/docs/contexts). - -```yaml -version: 2.1 -jobs: - process-template: - docker: - - image: cimg/base:current - steps: - - checkout - - run: - name: Process template file - environment: - # Environment variables would typically be served via a context - FOO: bar - PROVIDER: circleci - command: | - circleci env subst < template.json > deploy.json - cat deploy.json -workflows: - env-subst-workflow: - jobs: - - process-template -``` - -In this example, the `<` symbol is used to redirect the contents of the `template.json` file as _input_ to the `env subst` command, while the `>` symbol is used to redirect the output of the env subst command to the `deploy.json`. - -You could alternatively pass input to the `circleci env subst` command as an argument: `circleci env subst "hello \$WORLD"` - -Output: - -```sh -{ - "foo": "bar", - "provider": "circleci" -} -``` - -For instructions on installing the CircleCI CLI locally, read the [Installing the CircleCI local CLI](/docs/local-cli/) guide. - -### Alpine Linux -{: #alpine-linux } - -An image that has been based on [Alpine Linux](https://alpinelinux.org/) (like [Docker](https://hub.docker.com/_/docker)), uses the `ash` shell. - -To use environment variables with `bash`, add the `shell` and `environment` keys to your job. - -```yaml -version: 2.1 - -jobs: - build: - shell: /bin/sh -leo pipefail - environment: - BASH_ENV: /etc/profile -``` - -## Notes on security -{: #notes-on-security } - -Do not add secrets or keys inside the `.circleci/config.yml` file. The full text of `.circleci/config.yml` is visible to developers with access to your project on CircleCI. Store secrets or keys in [project]({{site.baseurl}}/set-environment-variable/#set-an-environment-variable-in-a-project) or [context]({{site.baseurl}}/set-environment-variable/#set-an-environment-variable-in-a-context) settings in the CircleCI web app. For more information, see the [Encryption]({{site.baseurl}}/security/#encryption) section of the security page. - -Running scripts within configuration may expose secret environment variables. See the [Using shell scripts]({{site.baseurl}}/using-shell-scripts/#shell-script-best-practices) page for best practices for secure scripts. - -## Contexts -{: #contexts } - -You can further restrict access to environment variables using [contexts]({{site.baseurl}}/contexts). Contexts are set from the **Organization Settings** in the CircleCI web app. - -## See also -{: #see-also } - -- [Security recommendations](/docs/security-recommendations) -- [Set an environment variable](/docs/set-environment-variable/) -- [Inject variables using the CircleCI API](/docs/inject-environment-variables-with-api/) From 4d86d0001926c15b874073ad16ad44e04149c8d1 Mon Sep 17 00:00:00 2001 From: Rosie Yohannan Date: Mon, 27 Jan 2025 23:41:09 +0000 Subject: [PATCH 17/20] convert example configs page --- jekyll/_cci2/example-configs.adoc | 137 +++++++++++++++++++++ jekyll/_cci2/example-configs.md | 71 ----------- styles/config/vocabularies/Docs/accept.txt | 5 + 3 files changed, 142 insertions(+), 71 deletions(-) create mode 100644 jekyll/_cci2/example-configs.adoc delete mode 100644 jekyll/_cci2/example-configs.md diff --git a/jekyll/_cci2/example-configs.adoc b/jekyll/_cci2/example-configs.adoc new file mode 100644 index 0000000000..7e6192884f --- /dev/null +++ b/jekyll/_cci2/example-configs.adoc @@ -0,0 +1,137 @@ +--- +contentTags: + platform: + - Cloud + - Server v4+ +--- += Example Public Repos +:page-description: This page lists example CircleCI config files. They contain all the basic steps needed to get started with deploying code using CircleCI. +:experimental: +:icons: font +:page-layout: classic-docs +:page-liquid: + +Refer to the following documents and linked `.circleci/config.yml` files for complete examples of public and open source projects that run on CircleCI. + +[#example-circleci-files-and-public-repos] +== Example CircleCI files and public repos + +[.table.table-striped] +[cols=3*, options="header", stripes=even] +|=== +| Example Name | Description | Link + +| Sample `config.yml` Files +| Four sample `config.yml` files using concurrent Workflows, sequential Workflows, fan-in/fan-out Workflows, and building Linux and iOS in one configuration file. +| xref:sample-config#[Sample `config.yml` Files] + +| Example Database Configurations +| Three example `config.yml` files using PostgreSQL/Rails, Go/PostgreSQL, and Ruby/MySQL +| xref:postgres-config#[Example Database Configurations] + +| Parallel +| Concurrent job run workflow configuration. +| link:https://github.com/CircleCI-Public/circleci-demo-workflows/blob/parallel-jobs/.circleci/config.yml[Parallel jobs config] + +| Sequential +| Sequential job run workflow configuration +| https://github.com/CircleCI-Public/circleci-demo-workflows/blob/sequential-branch-filter/.circleci/config.yml[Sequential jobs config] + +| Fan-in / Fan-out +| Workflow configuration with sequenced jobs followed by concurrent jobs chained to a final sequenced job. +| https://github.com/CircleCI-Public/circleci-demo-workflows/blob/fan-in-fan-out/.circleci/config.yml[Fan-in-out config] + +| Workspace Forwarding +| Jobs configured to share data with workspaces. +| https://github.com/CircleCI-Public/circleci-demo-workflows/blob/workspace-forwarding/.circleci/config.yml[Workspace forwarding config] + +| CircleCI docs +| A static website generated by Jekyll for CircleCI documentation. +| https://github.com/circleci/circleci-docs/blob/master/.circleci/config.yml[CircleCI docs config] + +| CircleCI image builder +| Uses Docker for building container images. +| https://github.com/circleci/image-builder/blob/master/.circleci/config.yml[Image builder config] + +| CircleCI Docker demo +| Example application showcasing how to build Docker images in CircleCI. +| https://github.com/CircleCI-Public/circleci-demo-docker/blob/master/.circleci/config.yml[CircleCI Docker demo config] +|=== + +[#open-source-projects-by-feature] +== Open source projects by feature + +The following projects provide examples of using features and functionality of CircleCI configuration syntax: + +[.table.table-striped] +[cols=3*, options="header", stripes=even] +|=== +| Example Description | Project | config.yml link + +| Build, test, and upload run for macOS and Linux Arm +| *Conda* is a platform- and language-independent package manager that sports distribution, installation and version management of software. +| https://github.com/bioconda/bioconda-recipes/blob/master/.circleci/config.yml[Bioconda CircleCI config] + +| Build and register a Docker image, deploy to AWS ECS +| *Taco* is a next generation repository system for DLSS. +| https://github.com/sul-dlss-labs/taco/blob/master/.circleci/config.yml[Taco CircleCI config] + +| Docker compose and `docker cp` with `store_artifacts` +| *Mayflower* is the enterprise design system for the Commonwealth of Massachusetts. +| https://github.com/massgov/mayflower/blob/develop/.circleci/config.yml[Mayflower CircleCI config] + +| Dynamic config, Remote Docker, Docker Layer Caching, build and push Docker images +| *Aeternity node* is a new blockchain for æpps. +| https://github.com/aeternity/aeternity/blob/master/.circleci/config.yml[Aeternity CircleCI config] + +| Build and test on multiple platforms with tagged releases +| *Crystal* is a programming language. +| https://github.com/crystal-lang/crystal/blob/master/.circleci/config.yml[Crystal CircleCI config] +|=== + +[#open-source-projects-by-language] +== Open source projects by language + +The following projects provide examples for particular programming languages, testing mechanisms, and deployment targets: + +[.table.table-striped] +[cols=3*, options="header", stripes=even] +|=== +| Example Description | Project | config.yml link + +| Salesforce DX +| This repository shows one way you can successfully set up *Salesforce DX* with CircleCI. +| https://github.com/forcedotcom/sfdx-circleci/blob/master/.circleci/config.yml[SFDX CircleCI config] + +| Golang build with `junit` and Kubernetes tests for prod and master job workflow +| The *Azure Container Service Engine* (acs-engine) generates Azure Resource Manager templates for Docker enabled clusters on Microsoft Azure with your choice of DC/OS, Kubernetes, Swarm Mode, or Swarm orchestrators. +| https://github.com/Azure/acs-engine/blob/master/.circleci/config.yml[ACS engine CircleCI config] + +| Go and Node build, package, and deploy run +| *Ignition* is a landing page for developers to self-service their way onto your Pivotal Cloud Foundry (PCF) deployment(s). +| https://github.com/ktpv/ignition/blob/master/.circleci/config.yml[Ignition CircleCI config] + +| JavaScript Node frontend build with `junit` and end-to-end WebDriver screenshot tests +| Frontend code for *cBioPortal* using React, MobX and TypeScript. +| https://github.com/cBioPortal/cbioportal-frontend/blob/master/.circleci/config.yml[cBioPortal CircleCI config] + +| Node and Yarn test and deploy website with caching +| *NEO•ONE* makes coding, testing and deploying your NEO blockchain solutions easier. +| https://github.com/neo-one-suite/neo-one/blob/master/.circleci/config.yml[Neo One CircleCI config] + +| Build and test with npm and Yarn, deploy to S3 +| The official *CLARK Platform* client +| https://github.com/Cyber4All/clark-client/blob/main/.circleci/config.yml[Clark CircleCI config] + +| Python build with PostgreSQL database, Selenium tests, and CodeClimate +| *CALC* (formerly known as "Hourglass"), which stands for Contracts Awarded Labor Category, is a tool to help contracting personnel estimate their per-hour labor costs for a contract, based on historical pricing information. +| https://github.com/18F/calc/blob/develop/.circleci/config.yml[CALC CircleCI config] + +| Python build with `apt` for concurrent job run with webhooks notification +| *SunPy* is an open-source Python library for solar physics data analysis. +| https://github.com/sunpy/sunpy/blob/main/.circleci/config.yml[SunPy CircleCI config] + +| Scala and `sbt` build, test, and release with three workflows +| *Arweave4s* is a lightweight modular HTTP client for the Arweave blockchain. +| https://github.com/toknapp/arweave4s/blob/master/.circleci/config.yml[Arweave4s CircleCI config] +|=== diff --git a/jekyll/_cci2/example-configs.md b/jekyll/_cci2/example-configs.md deleted file mode 100644 index 82c4e421ee..0000000000 --- a/jekyll/_cci2/example-configs.md +++ /dev/null @@ -1,71 +0,0 @@ ---- -layout: classic-docs -title: "Example Public Repos" -short-title: "Example Public Repos" -description: "This page lists example CircleCI config files. They contain all the basic steps needed to get started with deploying code using CircleCI." -categories: [getting-started] -order: 1 -contentTags: - platform: - - Cloud - - Server v4+ ---- - -Refer to the following documents and linked `.circleci/config.yml` files for complete examples of public and open source projects that run on CircleCI. - -## Example CircleCI files and public repos -{: #example-circleci-files-and-public-repos } - -Example Name | Description | Link -----|----------|-------- -Sample `config.yml` Files | Four sample `config.yml` files using concurrent Workflows, sequential Workflows, fan-in/fan-out Workflows, and building Linux and iOS in one configuration file. | Sample `config.yml` Files -Example Database Configurations | Three example `config.yml` files using PostgreSQL/Rails, Go/PostgreSQL, and Ruby/MySQL | [Example Database Configurations]({{ site.baseurl }}/postgres-config/) -Parallel | Concurrent job run workflow configuration. | [parallel-jobs/.circleci/config.yml](https://github.com/CircleCI-Public/circleci-demo-workflows/blob/parallel-jobs/.circleci/config.yml) -Sequential | Sequential job run workflow configuration | [sequential-branch-filter/.circleci/config.yml](https://github.com/CircleCI-Public/circleci-demo-workflows/blob/sequential-branch-filter/.circleci/config.yml) -Fan-in / Fan-out | Workflow configuration with sequenced jobs followed by concurrent jobs chained to a final sequenced job. | [fan-in-fan-out/.circleci/config.yml](https://github.com/CircleCI-Public/circleci-demo-workflows/blob/fan-in-fan-out/.circleci/config.yml) -Workspace Forwarding | Jobs configured to share data with workspaces. | [workspace-forwarding/.circleci/config.yml](https://github.com/CircleCI-Public/circleci-demo-workflows/blob/workspace-forwarding/.circleci/config.yml) -circleci-docs | A static website generated by Jekyll for CircleCI documentation. | [circleci-docs/.circleci/config.yml](https://github.com/circleci/circleci-docs/blob/master/.circleci/config.yml) -circleci image-builder | Uses Docker for building container images. | [image-builder/.circleci/config.yml](https://github.com/circleci/image-builder/blob/master/.circleci/config.yml) -circleci-demo-docker | This is an example application showcasing how to build Docker images in CircleCI. | [.circleci/config.yml](https://github.com/CircleCI-Public/circleci-demo-docker/blob/master/.circleci/config.yml) -{: class="table table-striped"} - -## Open source projects by feature -{: #open-source-projects-by-feature } - -The following projects provide examples of using features and functionality of CircleCI configuration syntax: - -Example Description | Project | config.yml link -------|-----------|------------ -Caching with templates, using workspaces, concurrent workflow jobs | Build mobile apps with **React**. | [react-native/.circleci/config.yml](https://github.com/facebook/react-native/blob/main/.circleci/config.yml) -Build and test with custom container image for ARM and x64 with `xvfb` and mocha | The **Electron** framework lets you write cross-platform desktop applications using JavaScript, HTML and CSS. It is based on Node.js and Chromium. |[electron/.circleci/config.yml](https://github.com/electron/electron/blob/main/.circleci/config.yml) -Build, test, and upload run for macOS and Linux | **Conda** is a platform- and language-independent package manager that sports easy distribution, installation and version management of software. | [bioconda-recipes/.circleci/config.yml](https://github.com/bioconda/bioconda-recipes/blob/master/.circleci/config.yml) -Build and register a Docker image, deploy to AWS ECS | **Taco** is a next generation repository system for DLSS. | [taco/.circleci/config.yml](https://github.com/sul-dlss-labs/taco/blob/master/.circleci/config.yml) -Docker compose and `docker cp` with `store_artifacts` | **Mayflower** is the enterprise design system for the Commonwealth of Massachusetts. | [mayflower/.circleci/config.yml](https://github.com/massgov/mayflower/blob/develop/.circleci/config.yml) -Remote Docker, Docker Layer Caching, build and push Docker images | **Epoch** is a new blockchain for æpps. | [epoch/.circleci/config.yml](https://github.com/aeternity/epoch/blob/master/.circleci/config.yml) -Build and test on multiple platforms with tagged releases | **Crystal** is a programming language. | [crystal/.circleci/config.yml](https://github.com/crystal-lang/crystal/blob/master/.circleci/config.yml) -{: class="table table-striped"} - -## Open source projects by language -{: #open-source-projects-by-language } - -The following projects provide examples for particular programming languages, testing mechanisms, and deployment targets: - -Example Description | Project | config.yml link -------|-----------|------------ -Bazel build and testing with `xvfb` | **Angular** is a development platform for building mobile and desktop web applications using Typescript/JavaScript and other languages. | [angular/.circleci/config.yml](https://github.com/angular/angular/blob/main/.circleci/config.yml) -Salesforce DX | This repository shows one way you can successfully set up **Salesforce DX** with CircleCI. | [sfdx-circleci/.circleci/config.yml](https://github.com/forcedotcom/sfdx-circleci/blob/master/.circleci/config.yml) -Golang build with `junit` and Kubernetes tests for prod and master job workflow | The **Azure Container Service Engine** (acs-engine) generates ARM (Azure Resource Manager) templates for Docker enabled clusters on Microsoft Azure with your choice of DC/OS, Kubernetes, Swarm Mode, or Swarm orchestrators. | [acs-engine/.circleci/config.yml](https://github.com/Azure/acs-engine/blob/master/.circleci/config.yml) -Go and Node build, package, and deploy run | **Ignition** is a landing page for developers to self-service their way onto your Pivotal Cloud Foundry (PCF) deployment(s). | [ignition/.circleci/config.yml](https://github.com/ktpv/ignition/blob/master/.circleci/config.yml) -Javascript Node frontend build with `junit` and end-to-end Webdriver screeshot tests | This is the frontend code for **cBioPortal** using React, MobX and TypeScript. | [cbioportal-frontend/.circleci/config.yml](https://github.com/cBioPortal/cbioportal-frontend/blob/master/.circleci/config.yml) -Node and Yarn with manual deploy gate | **Docusaurus** is a project for easily building, deploying, and maintaining open source project websites. | [Docusaurus/.circleci/config.yml](https://github.com/circleci/circleci-images/blob/master/.circleci/config.yml) -Node and Yarn test and deploy website with caching | **NEO•ONE** makes coding, testing and deploying your NEO blockchain solutions easier. | [neo-one/.circleci/config.yml](https://github.com/neo-one-suite/neo-one/blob/master/.circleci/config.yml) -Build and test with NPM and Yarn, deploy to S3 | The official **CLARK Platform** client | [clark-client/.circleci/config.yml](https://github.com/Cyber4All/clark-client/blob/main/.circleci/config.yml) -Python build with Postgres database, Selenium tests, and CodeClimate | **CALC** (formerly known as "Hourglass"), which stands for Contracts Awarded Labor Category, is a tool to help contracting personnel estimate their per-hour labor costs for a contract, based on historical pricing information. | [calc/.circleci/config.yml](https://github.com/18F/calc/blob/develop/.circleci/config.yml) -Python build with `apt` for concurrent job run with webhooks notification | **SunPy** is an open-source Python library for solar physics data analysis. | [sunpy/.circleci/config.yml](https://github.com/sunpy/sunpy/blob/main/.circleci/config.yml) -Scala and `sbt` build, test, and release with three workflows | **Arweave4s** is a lightweight modular HTTP client for the Arweave blockchain. | [arweave4s/.circleci/config.yml](https://github.com/toknapp/arweave4s/blob/master/.circleci/config.yml) -{: class="table table-striped"} - -## See also -{: #see-also } - -Refer to [Examples and Guides Overview]({{ site.baseurl }}/examples-and-guides-overview/) for configuration walkthroughs with commented examples and detailed explanations for basic applications written in several different languages. diff --git a/styles/config/vocabularies/Docs/accept.txt b/styles/config/vocabularies/Docs/accept.txt index f6e6910d8b..b7d395751f 100644 --- a/styles/config/vocabularies/Docs/accept.txt +++ b/styles/config/vocabularies/Docs/accept.txt @@ -23,6 +23,7 @@ [Ww]alkthrough [Aa]ctions ACR +Aeternity [Aa]ir-gapped Airtable /bAmazon SageMaker/b @@ -37,6 +38,7 @@ Appium Arm Artifactory [Aa]rchitected +Arweave Asciidoctor Atlassian [Aa]utodetected @@ -51,9 +53,11 @@ backoff [Bb]anlist? Bash Behat +Bioconda Bitbucket \bBitbucket Cloud\b [Bb]lobless +[Bb]lockchain Braintrust BrowserStack Buildah @@ -82,6 +86,7 @@ CocoaPods Codecov CodeDeploy Coinbase +\bConda\b [Cc]onfigs?\b ConfigMap /b[Cc]ontainer [Rr]egistry/b From d5ef20e87f0e8ec2f40dfc9fd7940a4f77489c08 Mon Sep 17 00:00:00 2001 From: Rosie Yohannan Date: Tue, 28 Jan 2025 10:50:42 +0000 Subject: [PATCH 18/20] fix links --- jekyll/_cci2/circleci-images.adoc | 2 +- jekyll/_cci2/env-vars.adoc | 2 +- jekyll/_cci2/hello-world.adoc | 4 ++-- 3 files changed, 4 insertions(+), 4 deletions(-) diff --git a/jekyll/_cci2/circleci-images.adoc b/jekyll/_cci2/circleci-images.adoc index d7c12799d1..6d9f725491 100644 --- a/jekyll/_cci2/circleci-images.adoc +++ b/jekyll/_cci2/circleci-images.adoc @@ -206,7 +206,7 @@ CircleCI maintains legacy images for the languages below. * <> * <> * <> -* <> +* <> * <> * <> * <> diff --git a/jekyll/_cci2/env-vars.adoc b/jekyll/_cci2/env-vars.adoc index e537187ee3..0aae64f325 100644 --- a/jekyll/_cci2/env-vars.adoc +++ b/jekyll/_cci2/env-vars.adoc @@ -48,7 +48,7 @@ Environment variables are used according to a specific precedence order, as foll . Environment variables declared xref:set-environment-variable#set-an-environment-variable-in-a-shell-command[inside a shell command] in a `run` step, for example `FOO=bar make install`. . Environment variables declared with the `environment` key xref:set-environment-variable#set-an-environment-variable-in-a-step[for a `run` step]. -. Environment variables set with the `environment` key link:set-environment-variable#set-an-environment-variable-in-a-job[for a job]. +. Environment variables set with the `environment` key xref:set-environment-variable#set-an-environment-variable-in-a-job[for a job]. . Special CircleCI environment variables defined in the xref:variables#built-in-environment-variables[CircleCI Built-in environment variables] document. . Context environment variables (assuming the user has access to the context). See the xref:contexts#[Contexts] documentation for more information. . xref:set-environment-variable#set-an-environment-variable-in-a-project[Project-level environment variables] set on the *Project Settings* page in the web app. diff --git a/jekyll/_cci2/hello-world.adoc b/jekyll/_cci2/hello-world.adoc index 9012634673..4463f9aa6a 100644 --- a/jekyll/_cci2/hello-world.adoc +++ b/jekyll/_cci2/hello-world.adoc @@ -37,7 +37,7 @@ These examples adds a job called `hello-job` that prints `hello world` to the co [.tab.hellocloud.Docker] -- -The job `hello-job` spins up a container running a link:/docs/circleci-images/#nodejs[pre-built CircleCI Docker image for Node]. Refer to link:/docs/using-docker[Using the Docker execution environment] page for more information. +The job `hello-job` spins up a container running a link:/docs/circleci-images/#node-js[pre-built CircleCI Docker image for Node]. Refer to link:/docs/using-docker[Using the Docker execution environment] page for more information. [source,yaml] ---- @@ -197,7 +197,7 @@ These examples add a job called `hello-job` that prints `hello world` to the con [.tab.helloserver4.Docker] -- -The job `hello-job` spins up a container running a link:/docs/circleci-images/#nodejs[pre-built CircleCI Docker image for Node]. Refer to link:/docs/using-docker[Using the Docker execution environment] page for more information. +The job `hello-job` spins up a container running a link:/docs/circleci-images/#node-js[pre-built CircleCI Docker image for Node]. Refer to link:/docs/using-docker[Using the Docker execution environment] page for more information. [source,yaml] ---- From 5fa59135ed372b26e5f3242e8d8a8c3cd65e963c Mon Sep 17 00:00:00 2001 From: Rosie Yohannan Date: Tue, 28 Jan 2025 10:55:58 +0000 Subject: [PATCH 19/20] fix link --- jekyll/_cci2/circleci-images.adoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/jekyll/_cci2/circleci-images.adoc b/jekyll/_cci2/circleci-images.adoc index 6d9f725491..9a3b2dc2fa 100644 --- a/jekyll/_cci2/circleci-images.adoc +++ b/jekyll/_cci2/circleci-images.adoc @@ -206,7 +206,7 @@ CircleCI maintains legacy images for the languages below. * <> * <> * <> -* <> +* <> * <> * <> * <> From 3ad1c2e9fceb7b442180b77e33c2feb77c255840 Mon Sep 17 00:00:00 2001 From: Rosie Yohannan Date: Tue, 28 Jan 2025 11:21:17 +0000 Subject: [PATCH 20/20] convert packagecloud tutorial --- ... => publish-packages-to-packagecloud.adoc} | 109 +++++++++--------- styles/config/vocabularies/Docs/accept.txt | 2 +- 2 files changed, 58 insertions(+), 53 deletions(-) rename jekyll/_cci2/{publish-packages-to-packagecloud.md => publish-packages-to-packagecloud.adoc} (70%) diff --git a/jekyll/_cci2/publish-packages-to-packagecloud.md b/jekyll/_cci2/publish-packages-to-packagecloud.adoc similarity index 70% rename from jekyll/_cci2/publish-packages-to-packagecloud.md rename to jekyll/_cci2/publish-packages-to-packagecloud.adoc index fce403962f..b202b9980d 100644 --- a/jekyll/_cci2/publish-packages-to-packagecloud.md +++ b/jekyll/_cci2/publish-packages-to-packagecloud.adoc @@ -1,66 +1,67 @@ --- -layout: classic-docs -title: Publish packages to Packagecloud -description: How to publish packages to Packagecloud using CircleCI. contentTags: platform: - Cloud - Server v4+ --- += Publish packages to Packagecloud +:description: How to publish packages to Packagecloud using CircleCI. +:experimental: +:icons: font +:page-layout: classic-docs +:page-liquid: In this how-to guide, you will learn how to configure CircleCI to publish packages to Packagecloud. -## Introduction -{: #introduction } +[#introduction] +== Introduction -[Packagecloud](https://packagecloud.io) is a hosted package repository service. It allows users to host npm, Java/Maven, python, apt, yum and rubygem repositories without any pre-configuration. +link:https://packagecloud.io[Packagecloud] is a hosted package repository service. It allows users to host npm, Java/Maven, Python, apt, yum and RubyGem repositories without any pre-configuration. -## 1. Configure environment variables -{: #configure-environment-variables } +[#configure-environment-variables] +== 1. Configure environment variables Under project settings in CircleCI, create an environment variable with the name `PACKAGECLOUD_TOKEN`, containing the value of a packagecloud API token. This environment variable will be used to authenticate with the packagecloud API directly, or using the packagecloud CLI. The packagecloud CLI will automatically read this environment variable from the system when interacting with repositories. -Alternatively, if you prefer to keep your sensitive environment variables checked into git, but encrypted, you can follow the process outlined at [circleci/encrypted-files](https://github.com/circleci/encrypted-files). +[#set-the-packagecloudurl-for-packagecloud-enterprise] +== 2. Set the $PACKAGECLOUD_URL for packagecloud:enterprise +Only set the `$PACKAGECLOUD_URL` if you are a packagecloud:enterprise customer. -## 2. Set the $PACKAGECLOUD_URL for packagecloud:enterprise -{: #set-the-packagecloudurl-for-packagecloud-enterprise } +NOTE: This setting is only for packagecloud:enterprise customers. Under project settings in CircleCI, set the `$PACKAGECLOUD_URL` environment variable to the URL of the packagecloud:enterprise installation. -**Only set the `$PACKAGECLOUD_URL` if you are a packagecloud:enterprise customer**. - -This setting is only for packagecloud:enterprise customers. Under project settings in CircleCI, set the `$PACKAGECLOUD_URL` environment variable to the URL of the packagecloud:enterprise installation. -{: class="alert alert-info" } - -## 3. Install the packagecloud CLI -{: #install-the-packagecloud-cli } +[#install-the-packagecloud-cli] +== 3. Install the packagecloud CLI To use the packagecloud CLI from CircleCI, install it using RubyGems by adding the following `run` step to your `.circleci/config.yml` under the job that is configured to deploy the package: -```yml +[,yml] +---- - run: name: Install packagecloud CLI command: gem install package_cloud -``` +---- The CLI will automatically use the `$PACKAGECLOUD_TOKEN` environment variable to authenticate against the packagecloud service. -## 4. Use dependency caching -{: #use-dependency-caching } +[#use-dependency-caching] +== 4. Use dependency caching -If you want to cache this dependency between builds, you can add the `package_cloud` gem to a `Gemfile` and follow CircleCI's guide for [Caching dependencies](/docs/caching/). +If you want to cache this dependency between builds, you can add the `package_cloud` gem to a `Gemfile` and follow CircleCI's guide for xref:caching#[Caching dependencies]. -## 5. Push packages with the packagecloud CLI -{: #push-packages-with-the-packagecloud-cli } +[#push-packages-with-the-packagecloud-cli] +== 5. Push packages with the packagecloud CLI The build processes for package types will vary, but pushing them into a packagecloud repository is quite simple. To add packages to a repository from your CircleCI builds, add a step in your `deploy` configuration that uses the packagecloud CLI. The following is a full example `.circleci/config.yml` that will checkout a git repository, run a `make` task (this command can be anything configured to build your package), then deploy the package to a packagecloud repo. -{% include snippets/docker-auth.md %} +{% include snippets/docker-auth.adoc %} -```yaml +[,yaml] +---- version: 2.1 defaults: &defaults working_directory: ~/repo @@ -95,19 +96,20 @@ workflows: - deploy: requires: - build -``` +---- -## 6. Deploy npm packages -{: #deploy-npm-packages } +[#deploy-npm-packages] +== 6. Deploy npm packages CircleCI users can deploy packages directly to npm registries hosted on Packagecloud. -### a. Configure the test job -{: #configure-the-test-job } +[#configure-the-test-job] +=== a. Configure the test job This job will retrieve the project code, install its dependencies and run any tests in the NodeJS project: -```yaml +[,yaml] +---- jobs: test: <<: *defaults @@ -133,14 +135,15 @@ jobs: - persist_to_workspace: root: ~/repo paths: . -``` +---- -### b. Configure the deploy job -{: #configure-the-deploy-job } +[#configure-the-deploy-job] +=== b. Configure the deploy job The next job configured is the deploy job. This job will authenticate and publish to the packagecloud npm registry: -```yaml +[,yaml] +---- jobs: ... deploy: @@ -157,21 +160,23 @@ jobs: - run: name: Publish package command: npm publish -``` +---- -* *Set registry URL* : This command sets the registry to URL that will be used by the `npm` CLI. -* *Authenticate with the registry* : This command will set the `authToken` to be used by the `npm` CLI to the environment variable configured in the project settings. -* *Publish package* : Publish the package to the configured npm registry on packagecloud. +* _Set registry URL_ : This command sets the registry to URL that will be used by the `npm` CLI. +* _Authenticate with the registry_ : This command will set the `authToken` to be used by the `npm` CLI to the environment variable configured in the project settings. +* _Publish package_ : Publish the package to the configured npm registry on packagecloud. The packagecloud npm registry URL is in the following format: -``` +[,shell] +---- https://packagecloud.io/:username/:repo_name/npm/ -``` +---- The full `.circleci/config.yml` should look something like this: -```yaml +[,yaml] +---- version: 2.1 defaults: &defaults working_directory: ~/repo @@ -223,18 +228,18 @@ workflows: - deploy: requires: - test -``` +---- The workflows section will tie together both the `test` and `deploy` jobs into sequential steps in the build process. -You can read more about publishing npm packages to packagecloud on the CircleCI blog post: [Publishing npm Packages Using CircleCI](https://circleci.com/blog/publishing-npm-packages-using-circleci-2-0/) +You can read more about publishing npm packages to packagecloud on the CircleCI blog post: link:https://circleci.com/blog/publishing-npm-packages-using-circleci-2-0/[Publishing npm Packages Using CircleCI] -## Use the Packagecloud API -{: #use-the-packagecloud-api } +[#use-the-packagecloud-api] +== Use the Packagecloud API -Packagecloud also provides a robust API to manage package repositories. You can read more about the [Packagecloud API](https://packagecloud.io/docs/api) and how to upload, delete, and promote packages across repositories. +Packagecloud also provides a robust API to manage package repositories. You can read more about the link:https://packagecloud.io/docs/api[Packagecloud API] and how to upload, delete, and promote packages across repositories. -## See also -{: #see-also } +[#see-also] +== See also -- [Storing and accessing artifacts](/docs/artifacts/) +* xref:artifacts#[Storing and accessing artifacts] diff --git a/styles/config/vocabularies/Docs/accept.txt b/styles/config/vocabularies/Docs/accept.txt index b7d395751f..d9ecb4b19a 100644 --- a/styles/config/vocabularies/Docs/accept.txt +++ b/styles/config/vocabularies/Docs/accept.txt @@ -215,7 +215,7 @@ OOM OPA \bOpenID Connect\b orgs? -Packagecloud +[Pp]ackagecloud [Pp]arameterization \bPerformance Plan\b PHP