AWSome

This is a tool to generate AWS cli commands and scripts based on YAML configuration files.

Although AWS services often do not require much maintainance, many services need to be configured and tuned before they are available.

Serveces like CloudFormation and OpsWorks allow to describe such configurations so they can be automated and versioned (along your code or on their own) but they do not cover all the services or are otherwise impractical (CloudFormation).

The aim here is to change the workflow to be repeatable and versioned. Here is how:

1. A yaml file is created to store one or more configurations. These map directly to an AWS CLI command. The structure of these files is explained below.
2. This tool is used to process the file.
3. One or more AWS CLI command are either outputted to stdout or executed.

And that is it, nothing more to do but check that AWS is set up as expected.

Table of contents

• Requirements
• Installing the AWS client
• Running from source
• Installing and running from an egg
• Building the egg
• YAML configuration format
• Profiles
• AWS CLI Profiles

Requirements

Python 3 and pip are required for this tool to work. Once pip is available other dependencies can be installed with

pip install -r requirements.txt

The following dependencies are also required:

• AWS CLI (see below)

Installing the AWS client

For this tool to execute the commands and not just generate them you need the AWS CLI installed and configured. The AWS CLI documentation is available at http://aws.amazon.com/cli/ and will be the place to look for the options you can place in the YAML files.

To install run the follwoing commands:

curl "https://s3.amazonaws.com/aws-cli/awscli-bundle.zip" -o "awscli-bundle.zip"
unzip awscli-bundle.zip
sudo ./awscli-bundle/install -i /usr/local/aws -b /usr/local/bin/aws
aws help

Instructions from http://docs.aws.amazon.com/cli/latest/userguide/installing.html#install-bundle-other-os

The configuration process is described in details in the documentation but for a quick setup just run aws configure and provide the details of your account.

Running from source

The source directory includes a script called awsome. This is a bash script that will check the location of the tool and call python in the appropriate way.

This will work on Linux only!

Assuming that all dependencies are installed running ./awsome <ARGS> should be enough.

Symlinks to this scripts are supported as well so you can "install" the tool. From the directory that contains the script:

ln -s "${PWD}/awsome" /usr/bin/awsome
awsome --help  # Test the command is now working.

Installing and running from an egg

This will be done when requested or when everything else is finished.

Building the egg

This will be done when requested or when everything else is finished.

YAML configuration format

All YAML files will need to be associative arrays (or dictionaries in python terms) that describe options and services to configure.

All top-level keys except for commands are mapped to global options for aws. The commands key stores a list of commands that will be generated.

For example this yaml file

output: json
commands:
  - ec2 describe-vpcs: ~

will result in the following command being generated:

aws --output="json" ec2 describe-vpcs

The following, more complete example shows how to create a security group:

# Config file 1 creates the security group(s)
commands:
  - ec2 create-security-group:
      description: AWSome security group test
      group-name: AWSome-Group
      vpc-id: your-vpc-id

# Config file 2 adds rules to the group.
# This needs to be separate as we need the group ID generated by AWS.
commands:
  # Allow SSH to the group.
  - ec2 authorize-security-group-ingress:
      group-id: aws-group-id
      protocol: tcp
      port: 22
      cidr: 172.30.0.0/16
  # Allow HTTP to the group.
  - ec2 authorize-security-group-ingress:
      group-id: aws-group-id
      protocol: tcp
      port: 80
      cidr: 172.30.0.0/16

More details on the format are available in the description of the YamlLoader class in AWSome/loader/yaml.py.

Profiles

Commands often need to refer to other items or services (such as vpcs or security groups). These have IDs that are generated by AWS and are not predictable.

To deal with this problem profiles are introduced. Profiles allow the use of "variables" in configuration files. Profiles are YAML files that map names to values.

When profiles are used and a config file is parsed occurrences of {{ var }} are replaced by the value of var indicated by the profile. If the variable is not defined in the profile an error is raised.

Note that if a yaml file with a {{ var }} token is processed without a profile the YAML parser will complain as {{ is not allowed.

AWS CLI Profiles

The AWS command line tool supports profiles too. These profiles are used to act as different users without having to log in multiple times and have nothing to do with AWSome profiles.

There are multiple ways to specify which profile to use but since the command line is called by AWSome and not us the environment variable solution is the simplest:

aws --profile thunder-prod configure
export AWS_DEFAULT_PROFILE="thunder-prod"
awsome --execute ...