Pipelines

Having an automated release process is one of the most important parts of software delivery so Copilot wants to make setting up that automated release process as easy as possible 🚀.

In this section, we'll talk about using Copilot to set up a CodePipeline which automatically builds your service code when you push to GitHub, deploys to your environments, and runs automated testing.

Why?

I won't get too philosophical about releasing software, but what's the point of having a release pipeline? With copilot deploy you can deploy your service directly from your computer to ECS, why add a middleman? That's a great question. For some apps, manually using deploy is enough, but as your release process gets more complicated (as you add more environments or add automated testing for example) you want to offload the boring work of repeatably orchestrating that process to a service. Having two services, each having two environments (test and production, say) and wanting to run integration tests after you deploy to your test environment becomes surprisingly cumbersome to do by hand.

Using an automated release tool like CodePipeline helps make your release manageable. Even if your release isn't particularly complicated, knowing that you can just git push to deploy your change always feels a little magical 🌈.

Pipeline structure

Copilot can set up a CodePipeline for you with a few commands - but before we jump into that, let's talk a little bit about the structure of the pipeline we'll be generating. Our pipeline will have the following basic structure:

1. GitHub Source - when you push to a configured branch (master by default), a new pipeline execution is triggered.
2. Build Stage - after your code is pulled from GitHub, your service's container image is built and published to every environment's ECR repository.
3. Deploy Stages - after your code is built, you can deploy to any and all of your environments, with optional post deployment tests or manual approvals.

Once you've set up a CodePipeline using Copilot, all you'll have to do is push to your GitHub repository, and CodePipeline will orchestrate the deployments.

Want to learn more about CodePipeline? Check out their getting started docs.

Creating a Pipeline in 3 steps

Creating a Pipeline only requires three steps:

1. Preparing the pipeline structure.
2. Committing the generated buildspec.yml.
3. Creating the actual CodePipeline.

Follow the three steps below, from your workspace root:

$ copilot pipeline init
$ git add copilot/buildspec.yml && git commit -m "Adding Pipeline Buildspec" && git push
$ copilot pipeline update

✨ And you'll have a new pipeline configured in your application account. Want to understand a little bit more what's going on? Read on!

Setting up a Pipeline, step by step

Step 1: Configuring your Pipeline

Pipeline configurations are created at a workspace level. If your workspace has a single service then your pipeline will be triggered only for that service. However, if you have multiple services in a workspace then it will build all the services in the workspace. To start setting up a pipeline, cd into your service(s)'s workspace and run:

$ copilot pipeline init

This won't create your pipeline, but it will create some local files that will be used when creating your pipeline.

• Release order: You'll be prompted for environments you want to deploy to - select them based on the order you want them to be deployed in your pipeline (deployments happen one environment at a time). You may, for example, want to deploy to your test environment first, and then your prod environment.

• Tracking repository: After you've selected the environments you want to deploy to, you'll be prompted to select which GitHub repository you want your CodePipeline to track. This is the repository that, when pushed to, will trigger a Pipeline execution (if the repository you're interested in doesn't show up, you can pass it in using the --github-url flag).

• Personal access token: In order to allow CodePipeline to track your GitHub repository, you'll need to provide a GitHub Personal Access Token. You can read how to do that here. Your token needs to have repo and admin:repo_hook permissions (so CodePipeline can create a WebHook on your behalf). Your GitHub Personal Access Token is stored securely in AWS Secrets Manager.

Step 2: Updating the Pipeline manifest (optional)

Just like your service has a simple manifest file, so does your pipeline. After you run pipeline init, two files are created, the pipeline.yml and buildspec.yml, both created in your copilot/ directory. If you poke in, you'll see a file that looks something like this (for a service called "api-frontend" with two environments, "test" and "prod"):

# This YAML file defines the relationship and deployment ordering of your environments.

# The name of the pipeline
name: pipeline-ecs-kudos-kohidave-demo-api-frontend

# The version of the schema used in this template
version: 1

# This section defines the source artifacts.
source:
  # The name of the provider that is used to store the source artifacts.
  provider: GitHub
  # Additional properties that further specifies the exact location
  # the artifacts should be sourced from. For example, the GitHub provider
  # has the following properties: repository, branch.
  properties:
    access_token_secret: github-token-ecs-kudos-demo-api-frontend
    branch: master
    repository: https://github.com/kohidave/demo-api-frontend

# The deployment section defines the order the pipeline will deploy
# to your environments.
stages:
    - # The name of the environment to deploy to.
      name: test
      test_commands:
        - make test
        - echo "woo! Tests passed"
    - # The name of the environment to deploy to.
      name: prod

There are 3 main parts of this file, the name field, which is the name of your CodePipeline, the source section, which details the GitHub repository and branch to track, and the stages section, which lists the environments you want this pipeline to deploy to. You can update this any time, but you must run copilot pipeline update afterwards.

Typically, you'll update this file if you add new environments you want to deploy to, or want to track a different branch.

Step 3: Updating the Buildspec (optional)

Along with the pipeline.yml, the pipeline init command also generated a buildspec.yml file in the copilot/ directory. This contains the instructions for building and publishing your service. If you want to run any additional commands, besides docker build, such as unit tests or style checkers, feel free to add it to the buildspec's build phase.

When this buildspec runs, it pulls down the version of Copilot which was used when you ran pipeline init, to ensure backwards compatibility.

Step 4: Creating your Pipeline

Now that your pipeline.yml and buildspec.yml are created, check them in and push them to your GitHub repository. The buildspec.yml is needed for your Pipeline's build stage to run successfully. Once you've done that, to actually create your pipeline run:

copilot pipeline update

This parses your pipeline.yml, creates a CodePipeline in the same account and region as your project (though it can deploy cross account and cross region) and kicks off a pipeline execution. Log into the AWS Console to watch your Pipeline go.