Recently, Everyday Hero has been doing a heap of work around the automation and provisioning of resources with AWS cloud services. This entails working frequently with the AWS API.
A useful filter we have been using in Ansible is taking output from a shell action and turning it into something we can consume via variables.
An example playbook is below:
- shell: |
export AWS_ACCESS_KEY_ID=$AWS_ACCESS_KEY
export AWS_SECRET_ACCESS_KEY=$AWS_SECRET_KEY
lib/ec2.py --list --refresh-cache
register: output
- set_fact:
ec2_output: "{{ output.stdout|from_json }}"
- shell: "rm addresses.txt"
- shell: "echo '{{ item.value['ec2_private_ip_address'] }}' >> addresses.txt"
when: item.value["ec2_tag_Name"] is defined and item.value["ec2_tag_Name"] == "deis-{{ environment_unique_id }}"
with_dict: ec2_output._meta.hostvars
- shell: "cat addresses.txt"
register: output
- add_host:
hostname: "{{ output.stdout_lines.0 }}"
groupname: "deis_node"
- debug: var={{ groups['deis_node'].0 }}
This example uses an Ansible plugin script ec2.py to retrieve a list of instances
running in an AWS account. It then filters out the ones that we want via the
ec2_tag_Name key and adds the private IP address to a file.
After running the following shell action, output holds a heap of information
about what the commands did and the output from stdout and stderr (if there was
any.)
- shell: |
export AWS_ACCESS_KEY_ID=$AWS_ACCESS_KEY
export AWS_SECRET_ACCESS_KEY=$AWS_SECRET_KEY
lib/ec2.py --list --refresh-cache
register: output
The Ansible output for the registered variable output looks something like this:
TASK: [debug var=output] ******************************************************
ok: [localhost] => {
"output": {
"changed": true,
"cmd": "export AWS_ACCESS_KEY_ID=$AWS_ACCESS_KEY \nexport AWS_SECRET_ACCESS_KEY=$AWS_SECRET_KEY \nlib/ec2.py --list --refresh-cache",
"delta": "0:00:05.268682",
"end": "2014-10-05 11:43:05.235594",
"invocation": {
"module_args": "export AWS_ACCESS_KEY_ID=$AWS_ACCESS_KEY \nexport AWS_SECRET_ACCESS_KEY=$AWS_SECRET_KEY \nlib/ec2.py --list --refresh-cache",
"module_name": "shell"
},
"rc": 0,
"start": "2014-10-05 11:42:59.966912",
"stderr": "",
"stdout": "<suppressed to keep it short>",
"stdout_lines": [
"{",
" \"_meta\": {",
" \"hostvars\": {",
" \"54.164.119.177\": {",
" \"ec2__in_monitoring_element\": false, ",
" \"ec2_ami_launch_index\": \"0\", ",
" \"ec2_architecture\": \"x86_64\", ",
" \"ec2_client_token\": \"a31d6387-be14-4182-9eca-c55977fc0029_us-east-1a_1\", ",
" \"ec2_dns_name\": \"ec2-54-164-119-177.compute-1.amazonaws.com\", ",
" \"ec2_ebs_optimized\": false, ",
" \"ec2_eventsSet\": \"\", ",
" \"ec2_group_name\": \"\", ",
" \"ec2_hypervisor\": \"xen\", ",
" \"ec2_id\": \"i-3ff468d1\", ",
" \"ec2_image_id\": \"ami-eb6b0182\", ",
" \"ec2_instance_profile\": \"\", ",
" \"ec2_instance_type\": \"m1.large\", ",
" \"ec2_ip_address\": \"54.164.119.177\", ",
" \"ec2_item\": \"\", ",
" \"ec2_kernel\": \"aki-88aa75e1\", ",
" \"ec2_key_name\": \"ambari\", ",
" \"ec2_launch_time\": \"2014-10-01T04:53:00.000Z\", ",
" \"ec2_monitored\": true, ",
" \"ec2_monitoring\": \"\", ",
" \"ec2_monitoring_state\": \"enabled\", ",
" \"ec2_persistent\": false, ",
" \"ec2_placement\": \"us-east-1a\", ",
" \"ec2_platform\": \"\", ",
" \"ec2_previous_state\": \"\", ",
" \"ec2_previous_state_code\": 0, ",
" \"ec2_private_dns_name\": \"ip-10-21-1-250.ec2.internal\", ",
" \"ec2_private_ip_address\": \"10.21.1.250\", ",
" \"ec2_public_dns_name\": \"ec2-54-164-119-177.compute-1.amazonaws.com\", ",
" \"ec2_ramdisk\": \"\", ",
" \"ec2_reason\": \"\", ",
" \"ec2_region\": \"us-east-1\", ",
" \"ec2_requester_id\": \"\", ",
" \"ec2_root_device_name\": \"/dev/sda1\", ",
" \"ec2_root_device_type\": \"ebs\", ",
" \"ec2_security_group_ids\": \"sg-016f5e64\", ",
" \"ec2_security_group_names\": \"ambari\", ",
" \"ec2_sourceDestCheck\": \"true\", ",
" \"ec2_spot_instance_request_id\": \"\", ",
" \"ec2_state\": \"running\", ",
" \"ec2_state_code\": 16, ",
" \"ec2_state_reason\": \"\", ",
" \"ec2_subnet_id\": \"subnet-98988bb0\", ",
" \"ec2_tag_Name\": \"hdpmaster1\", ",
" \"ec2_tag_aws_autoscaling_groupName\": \"ambari-LargeClusterGroup-148W3OVR9LSSE\", ",
" \"ec2_tag_aws_cloudformation_logical-id\": \"LargeClusterGroup\", ",
" \"ec2_tag_aws_cloudformation_stack-id\": \"arn:aws:cloudformation:us-east-1:167609138788:stack/ambari/c86f4dc0-4926-11e4-af0b-50fa526be49c\", ",
" \"ec2_tag_aws_cloudformation_stack-name\": \"ambari\", ",
" \"ec2_tag_long_hostname\": \"hdpmaster1.everydayhero-staging.io\", ",
" \"ec2_virtualization_type\": \"paravirtual\", ",
" \"ec2_vpc_id\": \"vpc-311faf54\"",
" }, ",
"}"
]
}
}
As you can see stdout and stdout_lines aren't in a great format to do much with, here enters from_json:
- set_fact:
ec2_output: "{{ output.stdout|from_json }}"
This then allows us to do the following and ec2_output is now in a standard
JSON format that Ansible can then use to iterate over like so:
- shell: "echo '{{ item.value['ec2_private_ip_address'] }}' >> addresses.txt"
when: item.value["ec2_tag_Name"] is defined and item.value["ec2_tag_Name"] == "deis-{{ environment_unique_id }}"
with_dict: ec2_output._meta.hostvars
This Ansible task creates a file that contains all of the private IP addresses
of ec2 instances that match the ec2_tag_Name criteria.
We could then take the first of these IP addresses and use it later in our playbook:
- shell: "cat addresses.txt"
register: output
- debug: var={{ output.stdout_lines.0 }}
This becomes a very powerful pattern when you mix in RESTful APIs, calls to them and processing output from them.