Pipelines

Having an automated release process is one of the most important parts of software delivery so the ECS CLI wants to make setting up that automated release process as easy as possible 🚀.

In this section, we'll talk about using the ECS CLI to set up a CodePipeline which automatically builds your application code when you push to GitHub, deploys to your environments, and runs automated testing.

Why?

I won't get too philosophical about releasing software, but what's the point of having a release pipeline? With ecs app deploy you can deploy your app directly from your computer to ECS, why add a middleman? That's a great question. For some projects, manually using app deploy is enough, but as your release process gets more complicated (as you add more environments or add automated testing for example) you want to offload the boring work of repeatably orchestrating that process to a service. Having two services, each having two environments (test and production, say) and wanting to run integration tests after you deploy to your test environment becomes surprisingly cumbersome to do by hand.

Using an automated release tool like CodePipeline helps make your release manageable. Even if your release isn't particularly complicated, knowing that you can just git push to deploy your change always feels a little magical 🌈.

Pipeline structure

The ECS CLI can set up a CodePipeline for you with a few commands - but before we jump into that, let's talk a little bit about the structure of the pipeline we'll be generating. Our pipeline will have the following basic structure:

1. GitHub Source - when you push to a configured branch (master by default), a new pipeline execution is triggered.
2. Build Stage - after your code is pulled from GitHub, your app's container image is built and published to every environment's ECR repository.
3. Deploy Stages - after your code is built, you can deploy to any and all of your environments, with optional post deployment tests or manual approvals.

Once you've set up a CodePipeline using the ECS CLI, all you'll have to do is push to your GitHub repository, and CodePipeline will orchestrate the deployments.

Want to learn more about CodePipeline? Check out their getting started docs.

Creating a Pipeline in 3 steps

Creating a Pipeline only requires three steps:

1. Preparing the pipeline structure.
2. Committing the generated buildspec.yml.
3. Creating the actual CodePipeline.

Follow the three steps below, from your workspace root:

$ ecs pipeline init
$ git add ecs-project/buildspec.yml && git commit -m "Adding Pipeline Buildspec" && git push
$ ecs pipeline update

✨ And you'll have a new pipeline configured in your project account. Want to understand a little bit more what's going on? Read on!

Setting up a Pipeline, step by step

Step 1: Configuring your Pipeline

Pipeline configurations are created at a workspace level. If your workspace has a single application, then your pipeline will be triggered only for that application. However, if you have multiple applications in a workspace then it will build all the applications in the workspace. To start setting up a pipeline, cd into your app(s)'s workspace and run:

$ ecs pipeline init

This won't create your pipeline, but it will create some local files that will be used when creating your Pipeline.

• Release order: You'll be prompted for environments you want to deploy to - select them based on the order you want them to be deployed in your Pipeline (deployments happen one environment at a time). You may, for example, want to deploy to your test environment first, and then your prod environment.

• Tracking repository: After you've selected the environments you want to deploy to, you'll be prompted to select which GitHub repository you want your CodePipeline to track. This is the repository that, when pushed to, will trigger a Pipeline execution (if the repository you're interested in doesn't show up, you can pass it in using the --github-url flag).

• Personal access token: In order to allow CodePipeline to track your GitHub repository, you'll need to provide a GitHub Personal Access Token. You can read how to do that here. Your token needs to have repo and admin:repo_hook permissions (so CodePipeline can create a WebHook on your behalf). Your GitHub Personal Access Token is stored securely in AWS Secrets Manager.

Step 2: Updating the Pipeline manifest (optional)

Just like your app has a simple manifest file, so does your Pipeline. After you run pipeline init, two files are created, the pipeline.yml and buildspec.yml, both created in your ecs-project directory. If you poke in, you'll see a file that looks something like this (for an app called api-frontend with two environments, test and prod):

# This YAML file defines the relationship and deployment ordering of your environments.

# The name of the pipeline
name: pipeline-ecs-kudos-kohidave-demo-api-frontend

# The version of the schema used in this template
version: 1

# This section defines the source artifacts.
source:
  # The name of the provider that is used to store the source artifacts.
  provider: GitHub
  # Additional properties that further specifies the exact location
  # the artifacts should be sourced from. For example, the GitHub provider
  # has the following properties: repository, branch.
  properties:
    access_token_secret: github-token-ecs-kudos-demo-api-frontend
    branch: master
    repository: https://github.com/kohidave/demo-api-frontend

# The deployment section defines the order the pipeline will deploy
# to your environments.
stages:
    - # The name of the environment to deploy to.
      name: test
    - # The name of the environment to deploy to.
      name: prod

There are 3 main parts of this file, the name field, which is the name of your Pipeline, the source section, which details the GitHub repository and branch to track, and the stages section, which lists the environments you want this Pipeline to deploy to. You can update this any time, but you must run ecs pipeline update afterwards.

Typically, you'll update this file if you add new environments you want to deploy to, or want to track a different branch.

Step 3: Updating the Buildspec (optional)

Along with the pipeline.yml, the init command also generated a buildpsec.yml file in the ecs-project/ directory. This contains the instructions for building and publishing your app. If you want to run any additional commands, besides docker build, such as unit tests or style checkers, feel free to add it to the buildspec's build phase.

When this buildspec runs, it pulls down the version of the ECS CLI which was used when you ran pipeline init, to ensure backwards compatibility.

Step 4: Creating your Pipeline

Now that your pipeline.yml and buildspec.yml are created, check them in and push them to your GitHub repository. The buildspec.yml is needed for your Pipeline's build stage to run successfully. Once you've done that, to actually create your Pipeline run:

ecs pipeline update

This parses your pipeline.yml, creates a CodePipeline in the same account and region as your project (though it can deploy cross account and cross region) and kicks off a Pipeline execution. Log into the AWS Console to watch your Pipeline go.

FAQ

1. How do I add integration tests to my environments?
2. How do I add manual blockers?
3. Can I have one Pipeline per app?
4. Can I have one Pipeline with multiple apps?
5. Can I use a source provider other than GitHub?
6. Will the Pipeline work with environments in different accounts and regions?
7. Can I still use app deploy if I've set up a Pipeline?