CloudFormation provides the ability to describe the AWS architecture to build through JSON or YAML files. Once the files are created, they can be uploaded to CloudFormation, which will then execute them and automatically create or update the associated AWS resources. CloudFormation can be reached at https://console.aws.amazon.com/cloudformation/home.
Using the AWS CLI tools required going through a number of steps to configure the EC2 instance and its security groups. Because that was done in a manual fashion, those steps are not reusable or auditable.
Existing templates can be found here: https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/cfn-sample-templates-us-west-2.html. The CloudFormation service is organized around the concept of stacks. Each stack describes a set of AWS resources and thier configuration in order to start an application.
Templates have the following format:
{
"AWSTemplateFormatVersion" : "2010-09-09",
"Description" : "Description string",
"Resources" : { },
"Parameters" : { },
"Mappings" : { },
"Conditions" : { },
"Metadata" : { },
"Outputs" : { }
}
AWSTemplateFormatVersion: The latest template format version is 2010-09-09 and is currently the only valid value.Description: Provide a summary of what the template does.Resources: Describe which AWS services will be instantiated and what thier configurations are.Parameters: Provide extra information such as which SSH keypair to use.Mappings: Useful when trying to create a more generic template such as defining which AMI to use for a given region so that the same template can be used to start an application in any AWS region.Conditions: Define conditional logic such as if statements, logical operators, and so on.Metadata: Add arbitrary information to resources.Outputs: Allows the ability to extract and print out useful information based on execution of the template such as IP address of the EC2 instance created.
From a DevOps perspective, one of the most powerful aspects of CloudFormation is the ability to write code to dynamically generate those templates. To illustrate that, we are going to use a Python library called Troposphere to generate the Hello World CloudFormation template.
Install the troposphere python library:
sean@vubuntu:~$ sudo pip install troposphere
Create the file helloworld-cf-template-v1.py and start by importing a number of definitions from the troposhere module:
"""Gnerating AWS CloudFormation template."""
from troposphere import (
Base64,
ec2,
GetAtt,
Join,
Output,
Parameter,
Ref,
Template,
)
Create a variable that will make editing the code easier later on.
ApplicationPort = "3000"This var instantiates an object of the type Template that will contain the entire description of the infrastructure.
t = Template()Provide a description of the stack.
t.add_description("Effective DevOps in AWS: HelloWorld web application")Create a Parameter object and intialize it by providing an identifier, description, parameter type, and a constraint description to help make the right decision when launching the stack. This will allow the CloudFormation user the ability to select which key pair to use when launching the EC2 instance.
t.add_parameter(Parameter(
"KeyPair",
Description="Name of an existing EC2 KeyPair to SSH",
Type="AWS::EC2::KeyPair::KeyName",
ConstraintDescription="must be the name of an existing EC2 KeyPair.",
))Create a Security Group object to open up SSH and TCP/3000. Port 3000 was defined in the var ApplicaitonPort declared earlier.
t.add_resource(ec2.SecurityGroup(
"SecurityGroup",
GroupDescription="Allow SSH and TCP/{} access".format(ApplicationPort),
SecurityGroupIngress=[
ec2.SecurityGroupRule(
IpProtocol="tcp",
FromPort="22",
ToPort="22",
CidrIp="0.0.0.0/0",
),
ec2.SecurityGroupRule(
IpProtocol="tcp",
FromPort=ApplicationPort,
ToPort=ApplicationPort,
CidrIp="0.0.0.0/0",
),
],
))Utilize the user-data optional parameter to provide a set of commands to install the helloworld.js file and its init script once the VM is up. This part of the script is the same steps that were done in the first chapter: Deploying your first web application.For more info see user-data.
The script will be base64-encoded
ud = Base64(Join('\n', [
"#!/bin/bash",
"sudo yum install --enablerepo=epel -y nodejs",
"wget http://bit.ly/2vESNuc -O /home/ec2-user/helloworld.js",
"wget http://bit.ly/2vVvT18 -O /etc/init/helloworld.conf",
"start helloworld"
]))Cloud-init is the defacto multi-distribution package that handles early initialization of a cloud instance. It complements the UserData field by moving most standard operations, such as installing packages, creating files, and running commands, into different sections of the template.
Create an EC2 instance object by providing the name, an image ID, instance type, security group, keypair for SSH, and user data. In CloudFormation, you can reference pre-existing subsections of your template by using the keyword Ref. In troposphere, this is done by calling the Ref() function. Add the resulting output to the template with the add_resource function.
t.add_resource(ec2.Instance(
"instance",
ImageId="ami-a0cfeed8",
InstanceType="t2.micro",
SecurityGroups=[Ref("SecurityGroup")],
KeyName=Ref("KeyPair"),
UserData=ud,
))The Outputs section of the tempalte that gets populated when CloudFormation creates a stack. Useful information we need is the URL to access the web application and the public IP address of the instance in order to SSH. Use the CloudFormation function Fn::GetAtt, which in Troposphere is translated to GetAttr() function.
t.add_output(Output(
"InstancePublicIp",
Description="Public IP of our instance.",
Value=GetAtt("instance", "PublicIp"),
))
t.add_output(Output(
"WebUrl",
Description="Application endpoint",
Value=Join("", [
"http://", GetAtt("instance", "PublicDnsName"),
":", ApplicationPort
]),
))Output the final result of the template generated.
print t.to_json()Run the script to generate the CloudFormation template by saving the output of the script to a file.
sean@vubuntu:~$ python helloworld-cf-template.py > helloworld-cf.template
{
"Description": "Effective DevOps in AWS: HelloWorld web application",
"Outputs": {
"InstancePublicIp": {
"Description": "Public IP of our instance.",
"Value": {
"Fn::GetAtt": [
"instance",
"PublicIp"
]
}
},
"WebUrl": {
"Description": "Application endpoint",
"Value": {
"Fn::Join": [
"",
[
"http://",
{
"Fn::GetAtt": [
"instance",
"PublicDnsName"
]
},
":",
"3000"
]
]
}
}
},
"Parameters": {
"KeyPair": {
"ConstraintDescription": "must be the name of an existing EC2 KeyPair.",
"Description": "Name of an existing EC2 KeyPair to SSH",
"Type": "AWS::EC2::KeyPair::KeyName"
}
},
"Resources": {
"SecurityGroup": {
"Properties": {
"GroupDescription": "Allow SSH and TCP/3000 access",
"SecurityGroupIngress": [
{
"CidrIp": "0.0.0.0/0",
"FromPort": "22",
"IpProtocol": "tcp",
"ToPort": "22"
},
{
"CidrIp": "0.0.0.0/0",
"FromPort": "3000",
"IpProtocol": "tcp",
"ToPort": "3000"
}
]
},
"Type": "AWS::EC2::SecurityGroup"
},
"instance": {
"Properties": {
"ImageId": "ami-a0cfeed8",
"InstanceType": "t2.micro",
"KeyName": {
"Ref": "KeyPair"
},
"SecurityGroups": [
{
"Ref": "SecurityGroup"
}
],
"UserData": {
"Fn::Base64": {
"Fn::Join": [
"\n",
[
"#!/bin/bash",
"sudo yum install --enablerepo=epel -y nodejs",
"wget http://bit.ly/2vESNuc -O /home/ec2-user/helloworld.js",
"wget http://bit.ly/2vVvT18 -O /etc/init/helloworld.conf",
"start helloworld"
]
]
}
}
},
"Type": "AWS::EC2::Instance"
}
}
}
-
Open the CloudFormation web console and click on Create Stack.
-
UnderChoose a template select Upload a template to Amazon S3 and Choose File to select the
helloworld-cf.templatefile that was generated from the python script. Click Next. -
Enter the Stack name and select the KeyPair and click Next.
-
On the next screen there is the ability to add optional tags to the resources. In the advanced section there is the option to integrate CloudFormation and SNS, make decisions on what to do when a failure occurs, and even add a stack policy that lets you control who can edit the stack. For this example leave everything blank and click Next.
- On the review screen click on Create.
- In the CloudFormation console the events Events tab shows the status and when the template is completed, click on the Outputs tab, which will revewal the generated outputs from the template.
- Click on the link in the WebUrl key to open up the HelloWorld page.
AWS has a service called AWS [CodeCommit] (https://aws.amazon.com/codecommit/) that allows easy management of Git repositories. However, it less popular that GitHub so GitHub will be used to create a new repository for the CloudFormation template:
- Create a new repository in GitHub.
- Name the new repository
CloudFormation. - Check the Initialize this repository with a README checkbox.
- Click the Create repository button.
- Install Git (on Ubuntu).
sean@vubuntu:~$ sudo apt update
sean@vubuntu:~$ sudo apt install git -y
sean@vubuntu:~$ git --version git version 2.17.1
sean@vubuntu:~$ git config --global user.email "[email protected]"
sean@vubuntu:~$ git config --global user.name "Sean Watson"
- Clone the repository to your system.
sean@vubuntu:~$ git clone https://github.com/seanlwatson/CloudFormation Cloning into 'CloudFormation'... Username for 'https://github.com': seanlwatson Password for 'https://[email protected]': remote: Enumerating objects: 3, done. remote: Counting objects: 100% (3/3), done. remote: Total 3 (delta 0), reused 0 (delta 0), pack-reused 0 Unpacking objects: 100% (3/3), done.
- Now that the repository is cloned, go into the repository and copy the template previously created in the new GitHub repository.
sean@vubuntu:~$ cd CloudFormation/
sean@vubuntu:~/CloudFormation$ cp ../helloworld-cf-template.py .
- Finally, add and commit that new file to the project and push it to GitHub.
sean@vubuntu:~/CloudFormation$ git add helloworld-cf-template.py
sean@vubuntu:~/CloudFormation$ git push Username for 'https://github.com': seanlwatson Password for 'https://[email protected]': Counting objects: 3, done. Delta compression using up to 2 threads. Compressing objects: 100% (3/3), done. Writing objects: 100% (3/3), 1.05 KiB | 1.05 MiB/s, done. Total 3 (delta 0), reused 0 (delta 0) To https://github.com/seanlwatson/CloudFormation 8af6f40..a71e8d8 master -> master
You may need to create a GitHub (personal access) token if you are using two-factor authentication. This can be used in place of a password when performing Git operations over HTTPS with Git on the command line or the API.
When managing code there are two common approaches. You can create a single repository for each project you have or decide to put the entire organizations code under a single repository. There are several open source projects such as Bazel from Google for using a mono repo so it becomes a very compelling option as it avoids juggling between multiple repositories when making big changes to infrastructure and services at the same time.
sean@vubuntu:~/CloudFormation$ python helloworld-cf-template.py > helloworld-cf.template
sean@vubuntu:~/CloudFormation$ git add helloworld-cf.template
sean@vubuntu:~/CloudFormation$ git commit -m "Adding helloworld CloudFormation template" [master 4c5d2f2] Adding helloworld CloudFormation template 1 file changed, 91 insertions(+) create mode 100644 helloworld-cf.template
sean@vubuntu:~/CloudFormation$ git push Username for 'https://github.com': seanlwatson Password for 'https://[email protected]': Counting objects: 3, done. Delta compression using up to 2 threads. Compressing objects: 100% (3/3), done. Writing objects: 100% (3/3), 1.04 KiB | 1.04 MiB/s, done. Total 3 (delta 0), reused 0 (delta 0) To https://github.com/seanlwatson/CloudFormation a71e8d8..4c5d2f2 master -> master
One of the biggest benefits of CloudFormation templates to manage resources is that the resources created from the template are tighly coupled to the stack. In order to make changes to the stack, update the template and apply the change to the existing stack. Install the ipify python package, which is used for IP address lookup.
sean@vubuntu:~$ sudo pip install ipify
The script requires CIDR notation so install the ipaddress package to conver the IP address into CIDR notation.
sean@vubuntu:~$ sudo pip install ipaddress
Modify the helloworld-cf-template.py script to change the CidrIp declaration for the SSH group rules to reference the PublicCidrIp variable:
from ipaddress import ip_network
from ipify import get_ip
PublicCidrIp = str(ip_network(get_ip()))t.add_resource(ec2.SecurityGroup(
"SecurityGroup",
GroupDescription="Allow SSH and TCP/{} access".format(ApplicationPort),
SecurityGroupIngress=[
ec2.SecurityGroupRule(
IpProtocol="tcp",
FromPort="22",
ToPort="22",
CidrIp=PublicCidrIp,
),
ec2.SecurityGroupRule(
IpProtocol="tcp",
FromPort=ApplicationPort,
ToPort=ApplicationPort,
CidrIp=PublicCidrIp,
),
],
))Generate a new template.
sean@vubuntu:~/CloudFormation$ python helloworld-cf-template.py > helloworld-cf.template
After updating the JSON CloudFormation template, update the stack as follows:
- In the CloudFormation web console select the
HelloWorldstack that was previously created. - Click on Action and then Update Stack.
- Check Upload a template to Amazon S3 and click on the Choose file button and select the
helloworld-cf.templateby clicking on the Choose button and then clicking on Next. - On the next screen click Next.
- In the next screen where the options exist to add optional tags to the resources and so on just click Next.
- This brings us to the Review page and after a few seconds a preview of the change is given and more detail can be seen by clicking on the View change set details and then click Execute.
- We can verify the change by extracting the Security Group physical resource ID from the review page and see that the
CidrIpfield inIpPermissions(for ingress) now has the value75.166.145.22/32instead of the allow all0.0.0.0/0.
sean@vubuntu:~$ aws ec2 describe-security-groups --group-names HelloWorld-SecurityGroup-1I2MCWU9SJYHO
{
"SecurityGroups": [
{
"Description": "Allow SSH and TCP/3000 access",
"GroupName": "HelloWorld-SecurityGroup-1I2MCWU9SJYHO",
"IpPermissions": [
{
"FromPort": 22,
"IpProtocol": "tcp",
"IpRanges": [
{
"CidrIp": "75.166.145.22/32"
}
],
"Ipv6Ranges": [],
"PrefixListIds": [],
"ToPort": 22,
"UserIdGroupPairs": []
},
{
"FromPort": 3000,
"IpProtocol": "tcp",
"IpRanges": [
{
"CidrIp": "75.166.145.22/32"
}
],
"Ipv6Ranges": [],
"PrefixListIds": [],
"ToPort": 3000,
"UserIdGroupPairs": []
}
],
"OwnerId": "404297683117",
"GroupId": "sg-09b45294e082298f9",
"IpPermissionsEgress": [
{
"IpProtocol": "-1",
"IpRanges": [
{
"CidrIp": "0.0.0.0/0"
}
],
"Ipv6Ranges": [],
"PrefixListIds": [],
"UserIdGroupPairs": []
}
],
"Tags": [
{
"Key": "aws:cloudformation:logical-id",
"Value": "SecurityGroup"
},
{
"Key": "aws:cloudformation:stack-id",
"Value": "arn:aws:cloudformation:us-west-2:404297683117:stack/HelloWorld/01e63360-dbb4-11e8-a96f-0ad5109330ec"
},
{
"Key": "aws:cloudformation:stack-name",
"Value": "HelloWorld"
}
],
"VpcId": "vpc-b3b5feca"
}
]
}
AWS offers an alternate and safer way to update templates with a feature called change sets. When in the CloudFormation web console select the
HelloWorldstack that was previously created. When clicking on Action then select Create Change Set instead of Update Stack. This allows the ability to audit recent changes using the Change Sets tabl of the stack.
Best practice is to always use CloudFormation to make changes to resources previously intialized with CloudFormation, including deleting them.
- Open the CloudFormation web console and select the
HelloWorldstack that was previously created. - Click on Action and then select Delete Stack and you will have to confirm the delete by clicking Yes, Delete.
Finally commit changes to git for the Troposphere python script and the CloudFormaiton template.
sean@vubuntu:~/CloudFormation$ git commit -am "Only allow SSH from our local IP" [master 7ed693b] Only allow SSH from our local IP 2 files changed, 7 insertions(+), 4 deletions(-)
sean@vubuntu:~/CloudFormation$ git push Username for 'https://github.com': seanlwatson Password for 'https://[email protected]': Counting objects: 4, done. Delta compression using up to 2 threads. Compressing objects: 100% (4/4), done. Writing objects: 100% (4/4), 534 bytes | 534.00 KiB/s, done. Total 4 (delta 2), reused 0 (delta 0) remote: Resolving deltas: 100% (2/2), completed with 2 local objects. To https://github.com/seanlwatson/CloudFormation 4c5d2f2..7ed693b master -> master
Click on the commits link to view all commits performed and then click on 7ed693b to view a diff of changes.
Because CloudFormaiton doesn't keep track of state of resources once they are launched, the only reliable way to update an EC2 instance is to recreate a new instance and swap it with the existing instance once the new instance is ready. This creates a somewhat immutable (unchanging over time) design assuing you don't run any extra commands once the instance is created. Having the ability to have long-running instances where you can quickly and reliably make changes through a controlled pipeline similiar to CloudFormation is what configuration management systems such as Puppet, Chef, SaltStack, and Ansible excel at.
Unlike other configuration mgt. systems, Ansible is built to work without a server, a daemon, or a database. You can simply keep your code in source control and download it on the host whenever you need to run it or use a push mechanism via SSH. The automation code you write is in YAML static files and will use GitHub as our version control system.
AWS OpsWorks has Chef integration which aims at being a "complete life cycle, including resource provisioning, configuration management, application deployment, software updates, monitoring, and access control".
sean@vubuntu:~$ sudo pip install ansible
sean@vubuntu:~$ ansible --version ansible 2.7.1 config file = None configured module search path = [u'/home/sean/.ansible/plugins/modules', u'/usr/share/ansible/plugins/modules'] ansible python module location = /usr/local/lib/python2.7/dist-packages/ansible executable location = /usr/local/bin/ansible python version = 2.7.15rc1 (default, Apr 15 2018, 21:51:34) [GCC 7.3.0]
Re-launch the helloworld application by using the CLI to interface with CloudFormation.
sean@vubuntu:~/CloudFormation$ aws cloudformation create-stack \
> --capabilities CAPABILITY_IAM \
> --stack-name Ansible \
> --template-body file://helloworld-cf.template \
> --parameters ParameterKey=KeyPair,ParameterValue=EffectiveDevOpsAWS
{
"StackId": "arn:aws:cloudformation:us-west-2:404297683117:stack/Ansible/4b8e9f00-dca6-11e8-9ab5-503a90a9c435"
}
- Create a new repository in GitHub.
- Name the new repository
Ansible. - Check the Initialize this repository with a README checkbox.
- Click the Create repository button.
sean@vubuntu:~$ git clone https://github.com/seanlwatson/Ansible Cloning into 'Ansible'... remote: Enumerating objects: 3, done. remote: Counting objects: 100% (3/3), done. remote: Total 3 (delta 0), reused 0 (delta 0), pack-reused 0 Unpacking objects: 100% (3/3), done.
sean@vubuntu:~$ cd Ansible/
Download the EC2 external inventory script and give execution permissions.
sean@vubuntu:~/Ansible$ curl -Lo ec2.py http://bit.ly/2v4SwE5
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 184 100 184 0 0 641 0 --:--:-- --:--:-- --:--:-- 641
100 68938 100 68938 0 0 107k 0 --:--:-- --:--:-- --:--:-- 216k
sean@vubuntu:~/Ansible$ chmod +x ec2.py
Create an ec2.ini file and put the following configuration:
[ec2]
regions = all
regions_exclude = us-gov-west-1,cn-north-1
destination_variable = public_dns_name
vpc_destination_variable = ip_address
route53 = False
cache_path = ~/.ansible/tmp
cache_max_age = 300
rds = False
Install the boto Python package.
sean@vubuntu:~/Ansible$ pip install boto
Validate that inventory is working by executing the ec2.py script.
sean@vubuntu:~/Ansible$ ./ec2.py
{
"404297683117": [
"54.189.226.117"
],
"_meta": {
"hostvars": {
"54.189.226.117": {
"ansible_host": "54.189.226.117",
"ec2__in_monitoring_element": false,
"ec2_account_id": "404297683117",
"ec2_ami_launch_index": "0",
"ec2_architecture": "x86_64",
"ec2_block_devices": {
"xvda": "vol-0c2707aa04d4047c4"
},
"ec2_client_token": "Ansib-insta-23FSM1KDHQUI",
"ec2_dns_name": "ec2-54-189-226-117.us-west-2.compute.amazonaws.com",
"ec2_ebs_optimized": false,
"ec2_eventsSet": "",
"ec2_group_name": "",
"ec2_hypervisor": "xen",
"ec2_id": "i-093a0a830b46ce5a9",
"ec2_image_id": "ami-a0cfeed8",
"ec2_instance_profile": "",
"ec2_instance_type": "t2.micro",
"ec2_ip_address": "54.189.226.117",
"ec2_item": "",
"ec2_kernel": "",
"ec2_key_name": "EffectiveDevOpsAWS",
"ec2_launch_time": "2018-10-31T00:45:54.000Z",
"ec2_monitored": false,
"ec2_monitoring": "",
"ec2_monitoring_state": "disabled",
"ec2_persistent": false,
"ec2_placement": "us-west-2a",
"ec2_platform": "",
"ec2_previous_state": "",
"ec2_previous_state_code": 0,
"ec2_private_dns_name": "ip-172-31-25-188.us-west-2.compute.internal",
"ec2_private_ip_address": "172.31.25.188",
"ec2_public_dns_name": "ec2-54-189-226-117.us-west-2.compute.amazonaws.com",
"ec2_ramdisk": "",
"ec2_reason": "",
"ec2_region": "us-west-2",
"ec2_requester_id": "",
"ec2_root_device_name": "/dev/xvda",
"ec2_root_device_type": "ebs",
"ec2_security_group_ids": "sg-014c8c6e132da8d1b",
"ec2_security_group_names": "Ansible-SecurityGroup-KZAEHA5Z3ZRW",
"ec2_sourceDestCheck": "true",
"ec2_spot_instance_request_id": "",
"ec2_state": "running",
"ec2_state_code": 16,
"ec2_state_reason": "",
"ec2_subnet_id": "subnet-718a3a08",
"ec2_tag_aws_cloudformation_logical_id": "instance",
"ec2_tag_aws_cloudformation_stack_id": "arn:aws:cloudformation:us-west-2:404297683117:stack/Ansible/4b8e9f00-dca6-11e8-9ab5-503a90a9c435",
"ec2_tag_aws_cloudformation_stack_name": "Ansible",
"ec2_virtualization_type": "hvm",
"ec2_vpc_id": "vpc-b3b5feca"
}
}
},
"ami_a0cfeed8": [
"54.189.226.117"
],
"ec2": [
"54.189.226.117"
],
"i-093a0a830b46ce5a9": [
"54.189.226.117"
],
"instance_state_running": [
"54.189.226.117"
],
"key_EffectiveDevOpsAWS": [
"54.189.226.117"
],
"security_group_Ansible_SecurityGroup_KZAEHA5Z3ZRW": [
"54.189.226.117"
],
"tag_aws_cloudformation_logical_id_instance": [
"54.189.226.117"
],
"tag_aws_cloudformation_stack_id_arn_aws_cloudformation_us_west_2_404297683117_stack_Ansible_4b8e9f00_dca6_11e8_9ab5_503a90a9c435": [
"54.189.226.117"
],
"tag_aws_cloudformation_stack_name_Ansible": [
"54.189.226.117"
],
"type_t2_micro": [
"54.189.226.117"
],
"us-west-2": [
"54.189.226.117"
],
"us-west-2a": [
"54.189.226.117"
],
"vpc_id_vpc_b3b5feca": [
"54.189.226.117"
]
}
Create a file called ansible.cfg with the below contents.
[defaults]
inventory = ./ec2.py
remote_user = ec2-user
become = True
become_method = sudo
become_user = root
nocows = 1
Use Ansible and the ping command to discover the IP address of the new EC2 instance that was created using CloudFormation.
sean@vubuntu:~/Ansible$ ansible --private-key ~/.ssh/EffectiveDevOpsAWS.pem ec2 -m ping
The authenticity of host '54.189.226.117 (54.189.226.117)' can't be established.
ECDSA key fingerprint is SHA256:PQ80MvhVozF0/X/HZGJs7UHjJsNayXyNGZe5hI4msoU.
Are you sure you want to continue connecting (yes/no)? yes
54.189.226.117 | SUCCESS => {
"changed": false,
"ping": "pong"
}
Since Ansible relies heavily on SSH configure SSH via the $HOME/.ssh/config file with the below configuration. Once configured, we will no longer have to provide the --private-key option to Ansible.
IdentityFile ~/.ssh/EffectiveDevOpsAWS.pem
User ec2-user
StrictHostKeyChecking no
PasswordAuthentication no
ForwardAgent yessean@vubuntu:~$ touch ~/.ssh/config
Use the below ansible command to run the df disk utility command to show disk space available and human readable -h.
sean@vubuntu:~/Ansible$ ansible '54.189.226.*' -a 'df -h' 54.189.226.117 | CHANGED | rc=0 >> Filesystem Size Used Avail Use% Mounted on devtmpfs 483M 60K 483M 1% /dev tmpfs 493M 0 493M 0% /dev/shm /dev/xvda1 7.8G 1.1G 6.6G 15% /
Playbooks contain Ansible's configuration, deployment, and orchestration language. By defining a playbook, you sequentially define the state of your system from the OS configuration down to the application deployment and monitoring. Playbooks are YAML based.
Creating roles is a key component in making Ansible modular enough for code reuse across services and playbooks.
The UserData block of the CloudFormation template contained the following commands. This prepares teh system to run the helloworld application by installing node.js. Then different resources are copied to run the application such as the JavaScript code and the upstart configuration. Lastly, we start the service.
sudo yum install --enablerepo=epel -y nodejs
wget http://bit.ly/2vESNuc -O /home/ec2-user/helloworld.js
wget http://bit.ly/2vVvT18 -O /etc/init/helloworld.conf
start helloworld
Installing node.js is not unique to our application so to make this reusable piece of code, we will create two roles. One to install node.js and one to deploy and start the helloworld application. Ansible expects to see roles inside a roles directory at the root of the Ansible repository.
sean@vubuntu:~/Ansible$ mkdir roles
sean@vubuntu:~/Ansible$ cd roles
The ansible-galaxy command can be used to initialize the creation of a role.
sean@vubuntu:~/Ansible/roles$ ansible-galaxy init nodejs - nodejs was created successfully
sean@vubuntu:~/Ansible/roles$ cd nodejs/
The most important directory inside the nodejs directory is one called tasks. When Ansible executes a playbook, it run the code present in the file tasks/main.yml
sean@vubuntu:~/Ansible/roles/nodejs$ ls defaults files handlers meta README.md tasks templates tests vars
When creating task we will use a wrapper around the yum command. Documentation about using the yum module is available. This will look at the packages installed on the system and if it doesn't find the nodejs or rpm packages then it will install them from the Extra Packages for Enterprise Linux (EPEL) repository.
--- # tasks file for nodejs - name: Installing Node.js and NPM (Node Package Mgr.) yum: name: "{{ item }}" enablerepo: epel state: installed with_items: - nodejs - npm
Navigate up a directory and initialize the helloworld role. Navigate to the helloworld directory and download the two resource files needed for the application.
sean@vubuntu:~/Ansible/roles/nodejs$ cd ..
sean@vubuntu:~/Ansible/roles$ ansible-galaxy init helloworld - helloworld was created successfully
sean@vubuntu:~/Ansible/roles$ cd helloworld/
sean@vubuntu:~/Ansible/roles/helloworld$ wget http://bit.ly/2vESNuc -O files/helloworld.js --2018-10-30 20:45:44-- http://bit.ly/2vESNuc Resolving bit.ly (bit.ly)... 67.199.248.10, 67.199.248.11 Connecting to bit.ly (bit.ly)|67.199.248.10|:80... connected. HTTP request sent, awaiting response... 301 Moved Permanently Location: https://raw.githubusercontent.com/EffectiveDevOpsWithAWS/code-by-chapter/master/Chapter02/helloworld.js [following] --2018-10-30 20:45:45-- https://raw.githubusercontent.com/EffectiveDevOpsWithAWS/code-by-chapter/master/Chapter02/helloworld.js Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 151.101.192.133, 151.101.128.133, 151.101.64.133, ... Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|151.101.192.133|:443... connected. HTTP request sent, awaiting response... 200 OK Length: 384 [text/plain] Saving to: ‘files/helloworld.js’ files/helloworld.js 100%[==============================================================>] 384 --.-KB/s in 0s 2018-10-30 20:45:45 (74.6 MB/s) - ‘files/helloworld.js’ saved [384/384]
sean@vubuntu:~/Ansible/roles/helloworld$ wget http://bit.ly/2vVvT18 -O files/helloworld.conf --2018-10-30 20:46:24-- http://bit.ly/2vVvT18 Resolving bit.ly (bit.ly)... 67.199.248.11, 67.199.248.10 Connecting to bit.ly (bit.ly)|67.199.248.11|:80... connected. HTTP request sent, awaiting response... 301 Moved Permanently Location: https://raw.githubusercontent.com/EffectiveDevOpsWithAWS/code-by-chapter/master/Chapter02/helloworld.conf [following] --2018-10-30 20:46:24-- https://raw.githubusercontent.com/EffectiveDevOpsWithAWS/code-by-chapter/master/Chapter02/helloworld.conf Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 151.101.192.133, 151.101.128.133, 151.101.64.133, ... Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|151.101.192.133|:443... connected. HTTP request sent, awaiting response... 200 OK Length: 301 [text/plain] Saving to: ‘files/helloworld.conf’ files/helloworld.conf 100%[==============================================================>] 301 --.-KB/s in 0s 2018-10-30 20:46:25 (61.7 MB/s) - ‘files/helloworld.conf’ saved [301/301]
Now create the task file to perform the copy on the remote system by opening the tasks/main.yml file to add the following YAML code. We are using the copy module to copy the application file into the home directory of the ec2-user. Lastly there is a notify action that act as triggers that can be added at the end of each block of task in a playbook. Here the notify is telling Ansible to call the restart helloworld directive if the file helloworld.js changed - the actual restart will be covered in another file. The notify option makes it easy to trigger events when a system changes state.
--- # tasks file for helloworld - name: Copying the application files copy: src: helloworld.js dest: /home/ec2-user/ owner: ec2-user group: ec2-user mode: 0644 notify: restart helloworld
Now add another task to copy the second file, the upstart script. Then the last taks to perform is to start the service by utilizing the service module. The tasks/main.yml file should resemble the below code.
--- # tasks file for helloworld - name: Copying the application file copy: src: helloworld.js dest: /home/ec2-user/ owner: ec2-user group: ec2-user mode: 0644 notify: restart helloworld - name: Copying the upstart file copy: src: helloworld.conf dest: /etc/init/helloworld.conf owner: root group: root mode: 0644 - name: Starting the HelloWorld node service service: name: helloworld state: started
The next file will provide Ansible with the knowlege of how to restart helloworld as called out in the notify paramater of our first task. These types of interactions are defined in the handler section of the role in the file handlers/main.yml/
--- # handlers file for helloworld - name: restart helloworld service: name: helloworld state: restarted
Lastly, we want to create a dependency that the helloworld role depends on the the nodejs role so that when the helloworld role is executed, it will first call the nodejs role and install the necessary requirements to run the application. Edit the meta/main.yml file, which has two sections. The first under galaxy info let you fill in information on the role you are building. This allows you to publish your role on GitHub and link it back into ansible-galaxy to share with the community. The second section at the bottom of hte file is called dependencies and is what should be edited to make sure that the Node.js is present on the system prior to starting the applicaiton. Remove the square brackets ([]) and add an entry to call the nodejs role.
dependencies: # List your role dependencies here, one per line. Be sure to remove the '[]' above, # if you add dependencies to this list. - nodejs
At the top level of the Ansible repository (two directories up from the helloworld role), create a new file called helloworld-pb.yml and add the following code. This tells Ansible to execute the role helloworld on the host listed in the variable target or localhost if the target isn't defined. The become option tells Ansible to execute the role with elevated priviliges (sudo).
sean@vubuntu:~/Ansible/roles/helloworld$ cd ~/Ansible/
--- # playbook file for helloworld application - hosts: "{{ target | default('localhost') }}" become: yes roles: - helloworld
Execution of a playbook uses the ansible-playbook command and relies on the same Ansible configuration file previously defined and therefore should be run at the root of the Ansible repository. The -e option allows the ability to pass in extra options for execution (or --extra-vars). One of the extra options is defining the variable target, which was declared in the hosts section of the playbook and set it to be equal to ec2 to target all EC2 instances. The option --list-hosts makes Ansible return a list of host that match the hosts criteria but will not run anything against those host. This allows you to verify which host would run the actual playbook.
sean@vubuntu:~/Ansible$ ansible-playbook helloworld-pb.yml \
> -e target=ec2 \
> --list-hosts
playbook: helloworld-pb.yml
play #1 (ec2): ec2 TAGS: []
pattern: [u'ec2']
hosts (1):
54.189.226.117
Check what will happen by performing a dry run with the playbook by using the --check option.
sean@vubuntu:~/Ansible$ ansible-playbook helloworld-pb.yml \
> -e target=54.189.226.117 \
> --check
PLAY [54.189.226.117] ********************************************************************************************************************
TASK [Gathering Facts] *******************************************************************************************************************
ok: [54.189.226.117]
TASK [nodejs : Installing Node.js and NPM (Node Package Mgr.)] ***************************************************************************
[DEPRECATION WARNING]: Invoking "yum" only once while using a loop via squash_actions is deprecated. Instead of using a loop to supply
multiple items and specifying `name: {{ item }}`, please use `name: [u'nodejs', u'npm']` and remove the loop. This feature will be
removed in version 2.11. Deprecation warnings can be disabled by setting deprecation_warnings=False in ansible.cfg.
changed: [54.189.226.117] => (item=[u'nodejs', u'npm'])
TASK [helloworld : Copying the application files] ****************************************************************************************
changed: [54.189.226.117]
TASK [helloworld : Copying the upstart file] *********************************************************************************************
ok: [54.189.226.117]
TASK [helloworld : Starting the HelloWorld node service] *********************************************************************************
ok: [54.189.226.117]
RUNNING HANDLER [helloworld : restart helloworld] ****************************************************************************************
changed: [54.189.226.117]
PLAY RECAP *******************************************************************************************************************************
54.189.226.117 : ok=6 changed=3 unreachable=0 failed=0
Having verified the host and code, run the playbook and execute changes.
sean@vubuntu:~/Ansible$ ansible-playbook helloworld-pb.yml -e target=54.189.226.117
PLAY [54.189.226.117] ********************************************************************************************************************
TASK [Gathering Facts] *******************************************************************************************************************
ok: [54.189.226.117]
TASK [nodejs : Installing Node.js and NPM (Node Package Mgr.)] ***************************************************************************
[DEPRECATION WARNING]: Invoking "yum" only once while using a loop via squash_actions is deprecated. Instead of using a loop to supply
multiple items and specifying `name: {{ item }}`, please use `name: [u'nodejs', u'npm']` and remove the loop. This feature will be
removed in version 2.11. Deprecation warnings can be disabled by setting deprecation_warnings=False in ansible.cfg.
changed: [54.189.226.117] => (item=[u'nodejs', u'npm'])
TASK [helloworld : Copying the application files] ****************************************************************************************
changed: [54.189.226.117]
TASK [helloworld : Copying the upstart file] *********************************************************************************************
ok: [54.189.226.117]
TASK [helloworld : Starting the HelloWorld node service] *********************************************************************************
ok: [54.189.226.117]
RUNNING HANDLER [helloworld : restart helloworld] ****************************************************************************************
changed: [54.189.226.117]
PLAY RECAP *******************************************************************************************************************************
54.189.226.117 : ok=6 changed=3 unreachable=0 failed=0
Test the application.
sean@vubuntu:~/Ansible$ curl 54.189.226.117:3000 Hello World
Commit changes to GitHub using 2 commits to break down the initialization of the repository and the creation of the role.
sean@vubuntu:~/Ansible$ git add ansible.cfg ec2.ini ec2.py
sean@vubuntu:~/Ansible$ git commit -m "Configuring Ansible to work with EC2" [master 0d0a9a0] Configuring Ansible to work with EC2 3 files changed, 1626 insertions(+) create mode 100644 ansible.cfg create mode 100644 ec2.ini create mode 100755 ec2.py
sean@vubuntu:~/Ansible$ git add roles helloworld-pb.yml
sean@vubuntu:~/Ansible$ git commit -m "Adding role for nodejs and helloworld" [master 16801ef] Adding role for nodejs and helloworld 19 files changed, 297 insertions(+) create mode 100644 helloworld-pb.yml create mode 100644 roles/helloworld/README.md create mode 100644 roles/helloworld/defaults/main.yml create mode 100644 roles/helloworld/files/helloworld.conf create mode 100644 roles/helloworld/files/helloworld.js create mode 100644 roles/helloworld/handlers/main.yml create mode 100644 roles/helloworld/meta/main.yml create mode 100644 roles/helloworld/tasks/main.yml create mode 100644 roles/helloworld/tests/inventory create mode 100644 roles/helloworld/tests/test.yml create mode 100644 roles/helloworld/vars/main.yml create mode 100644 roles/nodejs/README.md create mode 100644 roles/nodejs/defaults/main.yml create mode 100644 roles/nodejs/handlers/main.yml create mode 100644 roles/nodejs/meta/main.yml create mode 100644 roles/nodejs/tasks/main.yml create mode 100644 roles/nodejs/tests/inventory create mode 100644 roles/nodejs/tests/test.yml create mode 100644 roles/nodejs/vars/main.yml
sean@vubuntu:~/Ansible$ git push Username for 'https://github.com': seanlwatson Password for 'https://[email protected]': Counting objects: 40, done. Delta compression using up to 2 threads. Compressing objects: 100% (24/24), done. Writing objects: 100% (40/40), 18.52 KiB | 2.65 MiB/s, done. Total 40 (delta 3), reused 0 (delta 0) remote: Resolving deltas: 100% (3/3), done. To https://github.com/seanlwatson/Ansible 41231c6..16801ef master -> master
Open up the roles/helloworld/files/helloworld.js and simply change the response in line 11 from Hello World\n to 'Hello New World\m'.
1 var http = require("http")
2
3 http.createServer(function (request, response) {
4
5 // Send the HTTP header
6 // HTTP Status: 200 : OK
7 // Content Type: text/plain
8 response.writeHead(200, {'Content-Type': 'text/plain'})
9
10 // Send the response body as "Hello World"
11 response.end('Hello New World\n')
12 }).listen(3000)
13
14 // Console will print the message
15 console.log('Server running')
Save the file and run the playbook again with the --check option. Ansible will detect a change in the application file, which will trigger the the notify of the restart helloworld handler.
sean@vubuntu:~/Ansible$ ansible-playbook helloworld-pb.yml \
> -e target=54.189.226.117 \
> --check
PLAY [54.189.226.117] ********************************************************************************************************************
TASK [Gathering Facts] *******************************************************************************************************************
ok: [54.189.226.117]
TASK [nodejs : Installing Node.js and NPM (Node Package Mgr.)] ***************************************************************************
[DEPRECATION WARNING]: Invoking "yum" only once while using a loop via squash_actions is deprecated. Instead of using a loop to supply
multiple items and specifying `name: {{ item }}`, please use `name: [u'nodejs', u'npm']` and remove the loop. This feature will be
removed in version 2.11. Deprecation warnings can be disabled by setting deprecation_warnings=False in ansible.cfg.
ok: [54.189.226.117] => (item=[u'nodejs', u'npm'])
TASK [helloworld : Copying the application files] ****************************************************************************************
changed: [54.189.226.117]
TASK [helloworld : Copying the upstart file] *********************************************************************************************
ok: [54.189.226.117]
TASK [helloworld : Starting the HelloWorld node service] *********************************************************************************
ok: [54.189.226.117]
RUNNING HANDLER [helloworld : restart helloworld] ****************************************************************************************
changed: [54.189.226.117]
PLAY RECAP *******************************************************************************************************************************
54.189.226.117 : ok=6 changed=2 unreachable=0 failed=0
Execute the changes.
sean@vubuntu:~/Ansible$ ansible-playbook helloworld-pb.yml -e target=54.189.226.117
PLAY [54.189.226.117] ********************************************************************************************************************
TASK [Gathering Facts] *******************************************************************************************************************
ok: [54.189.226.117]
TASK [nodejs : Installing Node.js and NPM (Node Package Mgr.)] ***************************************************************************
[DEPRECATION WARNING]: Invoking "yum" only once while using a loop via squash_actions is deprecated. Instead of using a loop to supply
multiple items and specifying `name: {{ item }}`, please use `name: [u'nodejs', u'npm']` and remove the loop. This feature will be
removed in version 2.11. Deprecation warnings can be disabled by setting deprecation_warnings=False in ansible.cfg.
ok: [54.189.226.117] => (item=[u'nodejs', u'npm'])
TASK [helloworld : Copying the application files] ****************************************************************************************
changed: [54.189.226.117]
TASK [helloworld : Copying the upstart file] *********************************************************************************************
ok: [54.189.226.117]
TASK [helloworld : Starting the HelloWorld node service] *********************************************************************************
ok: [54.189.226.117]
RUNNING HANDLER [helloworld : restart helloworld] ****************************************************************************************
changed: [54.189.226.117]
PLAY RECAP *******************************************************************************************************************************
54.189.226.117 : ok=6 changed=2 unreachable=0 failed=0
Verify change is in effect.
sean@vubuntu:~/Ansible$ curl 54.189.226.117:3000 Hello New World
If this simple change was done through CloudFormation template, then CloudFormation would have had to create a new EC2 instance to make it happen when all we wanted to do was to update the code of the application and push it through Ansible on the target host.
Now revert the change locally in Git.
sean@vubuntu:~/Ansible$ git checkout roles/helloworld/files/helloworld.js
Install Git from the EPEL yum repository and Ansible using pip. Use the become command in order to run the commands as root.
sean@vubuntu:~/Ansible$ ansible '54.189.226.117' \
> --become \
> -m yum -a 'name=git enablerepo=epel state=installed'
54.189.226.117 | CHANGED => {
"ansible_facts": {
"pkg_mgr": "yum"
},
"changed": true,
"msg": "",
"rc": 0,
"results": [
"Loaded plugins: priorities, update-motd, upgrade-helper\n1054 packages excluded due to repository priority protections\nResolving Dependencies\n--> Running transaction check\n---> Package git.x86_64 0:2.14.5-1.59.amzn1 will be installed\n--> Processing Dependency: perl-Git = 2.14.5-1.59.amzn1 for package: git-2.14.5-1.59.amzn1.x86_64\n--> Processing Dependency: perl(Term::ReadKey) for package: git-2.14.5-1.59.amzn1.x86_64\n--> Processing Dependency: perl(Git::I18N) for package: git-2.14.5-1.59.amzn1.x86_64\n--> Processing Dependency: perl(Git) for package: git-2.14.5-1.59.amzn1.x86_64\n--> Processing Dependency: perl(Error) for package: git-2.14.5-1.59.amzn1.x86_64\n--> Running transaction check\n---> Package perl-Error.noarch 1:0.17020-2.9.amzn1 will be installed\n---> Package perl-Git.noarch 0:2.14.5-1.59.amzn1 will be installed\n---> Package perl-TermReadKey.x86_64 0:2.30-20.9.amzn1 will be installed\n--> Finished Dependency Resolution\n\nDependencies Resolved\n\n================================================================================\n Package Arch Version Repository Size\n================================================================================\nInstalling:\n git x86_64 2.14.5-1.59.amzn1 amzn-updates 12 M\nInstalling for dependencies:\n perl-Error noarch 1:0.17020-2.9.amzn1 amzn-main 33 k\n perl-Git noarch 2.14.5-1.59.amzn1 amzn-updates 69 k\n perl-TermReadKey x86_64 2.30-20.9.amzn1 amzn-main 33 k\n\nTransaction Summary\n================================================================================\nInstall 1 Package (+3 Dependent packages)\n\nTotal download size: 12 M\nInstalled size: 29 M\nDownloading packages:\n--------------------------------------------------------------------------------\nTotal 4.7 MB/s | 12 MB 00:02 \nRunning transaction check\nRunning transaction test\nTransaction test succeeded\nRunning transaction\n Installing : 1:perl-Error-0.17020-2.9.amzn1.noarch 1/4 \n Installing : perl-TermReadKey-2.30-20.9.amzn1.x86_64 2/4 \n Installing : perl-Git-2.14.5-1.59.amzn1.noarch 3/4 \n Installing : git-2.14.5-1.59.amzn1.x86_64 4/4 \n Verifying : 1:perl-Error-0.17020-2.9.amzn1.noarch 1/4 \n Verifying : perl-Git-2.14.5-1.59.amzn1.noarch 2/4 \n Verifying : git-2.14.5-1.59.amzn1.x86_64 3/4 \n Verifying : perl-TermReadKey-2.30-20.9.amzn1.x86_64 4/4 \n\nInstalled:\n git.x86_64 0:2.14.5-1.59.amzn1 \n\nDependency Installed:\n perl-Error.noarch 1:0.17020-2.9.amzn1 perl-Git.noarch 0:2.14.5-1.59.amzn1\n perl-TermReadKey.x86_64 0:2.30-20.9.amzn1\n\nComplete!\n"
]
}
sean@vubuntu:~/Ansible$ ansible '54.189.226.117' \
> --become \
> -m pip -a 'name=ansible state=present'
54.189.226.117 | CHANGED => {
"changed": true,
"cmd": [
"/usr/bin/pip",
"install",
"ansible"
],
"name": [
"ansible"
],
"requirements": null,
"state": "present",
"stderr": "You are using pip version 9.0.3, however version 18.1 is available.\nYou should consider upgrading via the 'pip install --upgrade pip' command.\n",
"stderr_lines": [
"You are using pip version 9.0.3, however version 18.1 is available.",
"You should consider upgrading via the 'pip install --upgrade pip' command."
],
"stdout": "Collecting ansible\n Downloading https://files.pythonhosted.org/packages/ec/ee/1494474b59c6e9cccdfde32da1364b94cdb280ff96b1493deaf4f3ae55f8/ansible-2.7.1.tar.gz (11.7MB)\nRequirement already satisfied: jinja2 in /usr/lib/python2.7/dist-packages (from ansible)\nRequirement already satisfied: PyYAML in /usr/lib64/python2.7/dist-packages (from ansible)\nRequirement already satisfied: paramiko in /usr/lib/python2.7/dist-packages (from ansible)\nCollecting cryptography (from ansible)\n Downloading https://files.pythonhosted.org/packages/87/e6/915a482dbfef98bbdce6be1e31825f591fc67038d4ee09864c1d2c3db371/cryptography-2.3.1-cp27-cp27mu-manylinux1_x86_64.whl (2.1MB)\nRequirement already satisfied: setuptools in /usr/lib/python2.7/dist-packages (from ansible)\nRequirement already satisfied: markupsafe in /usr/lib64/python2.7/dist-packages (from jinja2->ansible)\nRequirement already satisfied: pycrypto!=2.4,>=2.1 in /usr/lib64/python2.7/dist-packages (from paramiko->ansible)\nRequirement already satisfied: ecdsa>=0.11 in /usr/lib/python2.7/dist-packages (from paramiko->ansible)\nCollecting asn1crypto>=0.21.0 (from cryptography->ansible)\n Downloading https://files.pythonhosted.org/packages/ea/cd/35485615f45f30a510576f1a56d1e0a7ad7bd8ab5ed7cdc600ef7cd06222/asn1crypto-0.24.0-py2.py3-none-any.whl (101kB)\nCollecting enum34; python_version < \"3\" (from cryptography->ansible)\n Downloading https://files.pythonhosted.org/packages/c5/db/e56e6b4bbac7c4a06de1c50de6fe1ef3810018ae11732a50f15f62c7d050/enum34-1.1.6-py2-none-any.whl\nRequirement already satisfied: six>=1.4.1 in /usr/lib/python2.7/dist-packages (from cryptography->ansible)\nCollecting cffi!=1.11.3,>=1.7 (from cryptography->ansible)\n Downloading https://files.pythonhosted.org/packages/14/dd/3e7a1e1280e7d767bd3fa15791759c91ec19058ebe31217fe66f3e9a8c49/cffi-1.11.5-cp27-cp27mu-manylinux1_x86_64.whl (407kB)\nCollecting idna>=2.1 (from cryptography->ansible)\n Downloading https://files.pythonhosted.org/packages/4b/2a/0276479a4b3caeb8a8c1af2f8e4355746a97fab05a372e4a2c6a6b876165/idna-2.7-py2.py3-none-any.whl (58kB)\nCollecting ipaddress; python_version < \"3\" (from cryptography->ansible)\n Downloading https://files.pythonhosted.org/packages/fc/d0/7fc3a811e011d4b388be48a0e381db8d990042df54aa4ef4599a31d39853/ipaddress-1.0.22-py2.py3-none-any.whl\nCollecting pycparser (from cffi!=1.11.3,>=1.7->cryptography->ansible)\n Downloading https://files.pythonhosted.org/packages/68/9e/49196946aee219aead1290e00d1e7fdeab8567783e83e1b9ab5585e6206a/pycparser-2.19.tar.gz (158kB)\nInstalling collected packages: asn1crypto, enum34, pycparser, cffi, idna, ipaddress, cryptography, ansible\n Running setup.py install for pycparser: started\n Running setup.py install for pycparser: finished with status 'done'\n Running setup.py install for ansible: started\n Running setup.py install for ansible: finished with status 'done'\nSuccessfully installed ansible-2.7.1 asn1crypto-0.24.0 cffi-1.11.5 cryptography-2.3.1 enum34-1.1.6 idna-2.7 ipaddress-1.0.22 pycparser-2.19\n",
"stdout_lines": [
"Collecting ansible",
" Downloading https://files.pythonhosted.org/packages/ec/ee/1494474b59c6e9cccdfde32da1364b94cdb280ff96b1493deaf4f3ae55f8/ansible-2.7.1.tar.gz (11.7MB)",
"Requirement already satisfied: jinja2 in /usr/lib/python2.7/dist-packages (from ansible)",
"Requirement already satisfied: PyYAML in /usr/lib64/python2.7/dist-packages (from ansible)",
"Requirement already satisfied: paramiko in /usr/lib/python2.7/dist-packages (from ansible)",
"Collecting cryptography (from ansible)",
" Downloading https://files.pythonhosted.org/packages/87/e6/915a482dbfef98bbdce6be1e31825f591fc67038d4ee09864c1d2c3db371/cryptography-2.3.1-cp27-cp27mu-manylinux1_x86_64.whl (2.1MB)",
"Requirement already satisfied: setuptools in /usr/lib/python2.7/dist-packages (from ansible)",
"Requirement already satisfied: markupsafe in /usr/lib64/python2.7/dist-packages (from jinja2->ansible)",
"Requirement already satisfied: pycrypto!=2.4,>=2.1 in /usr/lib64/python2.7/dist-packages (from paramiko->ansible)",
"Requirement already satisfied: ecdsa>=0.11 in /usr/lib/python2.7/dist-packages (from paramiko->ansible)",
"Collecting asn1crypto>=0.21.0 (from cryptography->ansible)",
" Downloading https://files.pythonhosted.org/packages/ea/cd/35485615f45f30a510576f1a56d1e0a7ad7bd8ab5ed7cdc600ef7cd06222/asn1crypto-0.24.0-py2.py3-none-any.whl (101kB)",
"Collecting enum34; python_version < \"3\" (from cryptography->ansible)",
" Downloading https://files.pythonhosted.org/packages/c5/db/e56e6b4bbac7c4a06de1c50de6fe1ef3810018ae11732a50f15f62c7d050/enum34-1.1.6-py2-none-any.whl",
"Requirement already satisfied: six>=1.4.1 in /usr/lib/python2.7/dist-packages (from cryptography->ansible)",
"Collecting cffi!=1.11.3,>=1.7 (from cryptography->ansible)",
" Downloading https://files.pythonhosted.org/packages/14/dd/3e7a1e1280e7d767bd3fa15791759c91ec19058ebe31217fe66f3e9a8c49/cffi-1.11.5-cp27-cp27mu-manylinux1_x86_64.whl (407kB)",
"Collecting idna>=2.1 (from cryptography->ansible)",
" Downloading https://files.pythonhosted.org/packages/4b/2a/0276479a4b3caeb8a8c1af2f8e4355746a97fab05a372e4a2c6a6b876165/idna-2.7-py2.py3-none-any.whl (58kB)",
"Collecting ipaddress; python_version < \"3\" (from cryptography->ansible)",
" Downloading https://files.pythonhosted.org/packages/fc/d0/7fc3a811e011d4b388be48a0e381db8d990042df54aa4ef4599a31d39853/ipaddress-1.0.22-py2.py3-none-any.whl",
"Collecting pycparser (from cffi!=1.11.3,>=1.7->cryptography->ansible)",
" Downloading https://files.pythonhosted.org/packages/68/9e/49196946aee219aead1290e00d1e7fdeab8567783e83e1b9ab5585e6206a/pycparser-2.19.tar.gz (158kB)",
"Installing collected packages: asn1crypto, enum34, pycparser, cffi, idna, ipaddress, cryptography, ansible",
" Running setup.py install for pycparser: started",
" Running setup.py install for pycparser: finished with status 'done'",
" Running setup.py install for ansible: started",
" Running setup.py install for ansible: finished with status 'done'",
"Successfully installed ansible-2.7.1 asn1crypto-0.24.0 cffi-1.11.5 cryptography-2.3.1 enum34-1.1.6 idna-2.7 ipaddress-1.0.22 pycparser-2.19"
],
"version": null,
"virtualenv": null
}
Because ansible-pull uses Git to clone locally the repository and execute it there is no need for SSH. At the root repository for Ansible create a file
called localhost with the following information. This creates a static inventory and run commands locally as apposed to using SSH when the target is defined as localhost.
[localhost]
localhost ansible_connection=local
sean@vubuntu:~/Ansible$ git add localhost
sean@vubuntu:~/Ansible$ git commit -m "Adding localhost inventory" [master 91a63ca] Adding localhost inventory 1 file changed, 2 insertions(+) create mode 100644 localhost
sean@vubuntu:~/Ansible$ git push Username for 'https://github.com': seanlwatson Password for 'https://[email protected]': Counting objects: 3, done. Delta compression using up to 2 threads. Compressing objects: 100% (2/2), done. Writing objects: 100% (3/3), 325 bytes | 325.00 KiB/s, done. Total 3 (delta 1), reused 0 (delta 0) remote: Resolving deltas: 100% (1/1), completed with 1 local object. To https://github.com/seanlwatson/Ansible 16801ef..91a63ca master -> master
Add a crontable entry to periodically call ansible-pull. The below command tells Ansible to use the cron module targteting the EC2 instance. We provide a name that Ansible will use to track the cronjob over time, telling cron to run the job every 10 minutes and finally the command to execute and its parameters that include the GitHub URL of our branch, the inventory file we just added and a sleep command that makes the command start at a random time between 1-60 sec. after the call starts. This helps spread out the load on the networks and prevent all node services from restarting at the same time.
sean@vubuntu:~/Ansible$ ansible '54.189.226.117' -m cron -a 'name=ansible-pull minute="*/10" job="/usr/local/bin/ansible-pull -U https://github.com/seanlwatson/Ansible helloworld-pb.yml -i localhost --sleep 60"'
54.189.226.117 | CHANGED => {
"changed": true,
"envs": [],
"jobs": [
"ansible-pull"
]
}
You can log into the EC2 instance and verify the cron job and once you see it execute then check that the change is effective on the server (Hello New World should change back to Hello World)
sean@vubuntu:~/Ansible$ ssh [email protected] -i ~/.ssh/EffectiveDevOpsAWS.pem Last login: Sat Nov 3 00:05:34 2018 from 75-166-145-22.hlrn.qwest.net __| __|_ ) _| ( / Amazon Linux AMI ___|\___|___| https://aws.amazon.com/amazon-linux-ami/2018.03-release-notes/ 14 package(s) needed for security, out of 25 available Run "sudo yum update" to apply all updates. [ec2-user@ip-172-31-25-188 ~]$
[ec2-user@ip-172-31-25-188 ~]$ crontab -l #Ansible: ansible-pull */10 * * * * /usr/local/bin/ansible-pull -U https://github.com/seanlwatson/Ansible helloworld-pb.yml -i localhost --sleep 60
[ec2-user@ip-172-31-25-188 ~]$ sudo grep -E "CROND.*CMD" /var/log/cron ... Nov 3 00:30:01 ip-172-31-25-188 CROND[3851]: (ec2-user) CMD (/usr/local/bin/ansible-pull -U https://github.com/seanlwatson/Ansible helloworld-pb.yml -i localhost --sleep 60)
[ec2-user@ip-172-31-25-188 ~]$ exit logout Connection to 54.189.226.117 closed.
sean@vubuntu:~/Ansible$ curl 54.189.226.117:3000 Hello World
Go to the CloudFormation repository and duplicate the previous Python Troposphere script.
sean@vubuntu:~/Ansible$ cd ~/CloudFormation/
sean@vubuntu:~/CloudFormation$ cp helloworld-cf-template.py ansiblebase-cf-template.py
Then modify the ansiblebase-cf-template.py script with the following changes. Before the declaration of the appliation port, add the application name.
ApplicationName = 'helloworld'Then add some GitHub information such as your username for your GitHub account (or organization name) and the URL to the Ansible repository on GitHub .
GitHubAccount = "seanlwatson"
GitHubAnsibleUrl = "https://github.com/{}/Ansible".format(GitHubAccount)Create one more var that will contain the command line to execute ansible-pull command that will configure the host. Use the vars GitHubAnsibleUrl and ApplicationName that was just previously defined.
AnsiblePullCmd = \
"/usr/local/bin/ansible-pull -U {} {}.yml -i localhost".format(
GitHubAnsibleUrl,
ApplicationName
)Now delete the previous ud var definition and replace it with the below code. This will update the AWS user-data optional parameter to install Git and Ansible, execute the command contained within AnsiblePullCmd, and lastly craete a cronjob to re-execute the command every 10 minutes.
ud = Base64(Join('\n', [
"#!/bin/bash",
"yum install --enablerepo=epel -y git",
"pip install ansible",
AnsiblePullCmd,
"echo '*/10 * * * * {}' > /etc/cron.d/ansible-pull".format(AnsiblePullCmd)
]))Create the CloudFormation JSON template and test it.
sean@vubuntu:~/CloudFormation$ python ansiblebase-cf-template.py > ansiblebase-cf.template
sean@vubuntu:~/CloudFormation$ aws cloudformation update-stack --stack-name Ansible --template-body file://ansiblebase-cf.template --parameters ParameterKey=KeyPair,ParameterValue=EffectiveDevOpsAWS
{
"StackId": "arn:aws:cloudformation:us-west-2:404297683117:stack/Ansible/4b8e9f00-dca6-11e8-9ab5-503a90a9c435"
}
This command will wait until stack status is UPDATE_COMPLETE. It will poll every 30 seconds until a successful state has been reached.
sean@vubuntu:~/CloudFormation$ aws cloudformation wait stack-update-complete \ > --stack-name Ansible
Query CloudFormation to get the stack outputs, specifically the Public IP. Then test the server application.
sean@vubuntu:~/CloudFormation$ aws cloudformation describe-stacks \
> --stack-name Ansible \
> --query 'Stacks[0].Outputs[0]'
{
"OutputKey": "InstancePublicIp",
"OutputValue": "34.221.214.133",
"Description": "Public IP of our instance."
}
sean@vubuntu:~/CloudFormation$ curl 34.221.214.133:3000 Hello World
Commit the newly created Python Troposphere script to the CloudFormation repository.
sean@vubuntu:~/CloudFormation$ git add ansiblebase-cf-template.py
sean@vubuntu:~/CloudFormation$ git commit -m "Adding helloworld Python Troposphere script to create a stack that relies on Ansible to manage the application" [master 3bdc58b] Adding helloworld Python Troposphere script to create a stack that relies on Ansible to manage the application 1 file changed, 92 insertions(+) create mode 100644 ansiblebase-cf-template.py
sean@vubuntu:~/CloudFormation$ git push Username for 'https://github.com': seanlwatson Password for 'https://[email protected]': Counting objects: 3, done. Delta compression using up to 2 threads. Compressing objects: 100% (3/3), done. Writing objects: 100% (3/3), 1.29 KiB | 1.29 MiB/s, done. Total 3 (delta 0), reused 0 (delta 0) To https://github.com/seanlwatson/CloudFormation 7ed693b..3bdc58b master -> master
We now have a complete solution to efficiently manage our infrastructure using code. While this is a simple example, everything done here is applicable to bigger infrastructure with a greater number of services.
Finally delete the stack.
sean@vubuntu:~/CloudFormation$ aws cloudformation delete-stack --stack-name Ansible