Building AWS infrastructure via the AWS console is pretty easy using your mad pointy clicky skills. However, perhaps you wondered if there was a different way? A repeatable, “programmatic” way with less pointy clicky action.

Well, there is, and it’s called Cloud Formation. Amazon describes Cloud Formation as:

AWS CloudFormation is a service that helps you model and set up your Amazon Web Services resources so that you can spend less time managing those resources and more time focusing on your applications that run in AWS.

You create a template that describes all the AWS resources that you want (like Amazon EC2 instances or Amazon RDS DB instances), and AWS CloudFormation takes care of provisioning and configuring those resources for you.

You don’t need to individually create and configure AWS resources and figure out what’s dependent on what; AWS CloudFormation handles all of that.

Check out the Cloud Formation documentation for more information.

I recently build my first Cloud Formation template by hand, and, this article discusses some of my observations whilst doing so.

First Up

Draw (design) what you need first

If you don’t have a good idea of what your infrastructure should look like, you’ve lost already. You’ll probably end up with a Frankenstein of a build with bits and pieces missing, or, additional components which will just cost you additional money.

Draw it out first using your favourite design tool such as Visio, Dia, Gliffy, OmniGraffle, PowerPoint, Paint, draw.io, cloudcraft.co, Pen and Paper or whatever you want to draw your infrastructure diagram with.

The first CF JSON file I was looking to build was based on the following diagram.

Diagram 1. High level diagram of the solution we are aiming to implement with CF.

Note: Check out the AWS architecture icon packs. Above was created with the Visio stencils.

Creation Options

Build via Console, dump configuration with Cloud Former

A manual option is to build your infrastructure as normal via the AWS console. Once complete, search for Cloud Formation and click on “Launch CloudFormer”. Cloud Former fires up some AWS instances and walks you through exporting your running AWS configuration into a JSON file, which, you can later import (Create a stack) back into Cloud Formation.

Using this option gives you an insight as to how a CF JSON file should look, the names it gives each element are not the names you gave the elements but the under the hood names AWS uses. Eg; vpc-4b7e932e

The names are stored as Tags for each resource { ‘Name’: ‘MyVPC’ }

Whilst I preferred manually building the JSON file with the VS Code extensions mentioned below, the Cloud Former created JSON file helped identify some of the missing resources from the diagram. For example, AWS::EC2::SubnetRouteTableAssociation is not represented in the diagram.

Note: CloudFormer is a cloud formation stack of it’s own, and, you will be charged for the use of the resources it consumes. I suggest shutting it down as soon as possible after dumping your configuration.

Build with Cloud Formation Designer (drag and drop)

I really liked the idea of CF Designer. It’s a graphical tool to drag and drop elements onto a canvas and link them together. Unfortunately, I couldn’t get to grips with it and I felt clumsy working with it.

CF Designer allows you start from scratch, or, load a JSON file and it will display what it looks like as a diagram. For example, the export I took from CloudFormer (above) looks like this when loaded into CF Designer:

Diagram 2. CloudFormation Designer diagram

Build manually, “Create a stack”

Building out your JSON file by hand can be rewarding, but, time consuming. You’ll be wanting to refer back to the AWS Resource Types on a regular basis.

Whilst CF will be able to work out dependencies (if you’ve done your references correctly), the order of your resources in the CF stack JSON file won’t matter… however, it seemed easier to refer back to your design diagram and start from the outside working inwards.

eg. Outside to inside

VPC
Subnets
Availability zone
Security groups
Elastic IP/Internet Gateway
NAT gateways
Load balancer nodes
Compute nodes

As mentioned above, the diagram doesn’t capture everything you need in the CF JSON template to make the solution actually work. Below is a list of all the AWS Resource Types used in the final JSON template file.

AWS::EC2::VPC
AWS::EC2::Subnet
AWS::EC2::InternetGateway
AWS::EC2::VPCGatewayAttachment
AWS::EC2::RouteTable
AWS::EC2::SubnetRouteTableAssociation
AWS::EC2::Route
AWS::EC2::SecurityGroup
AWS::EC2::EIP
AWS::EC2::NatGateway
AWS::ElasticLoadBalancing::LoadBalancer
AWS::EC2::Instance

Check out some of the Visual Code Studio tools (below) to assist building the JSON file out.

Rinse and repeat (for free – mostly)

You will probably find yourself “Creating stack” and “Delete stack” on a regular basis until you get your JSON file just right.

Save yourself some money, and ensure you try to stick to the order above. There are a number of resources that are not chargeable, so, you can tinker with them in your JSON until your heart’s content.

Once you start spinning up Load Balancers, NAT Gateways, Elastic IP’s and Instances (and others) you will start getting charged.

Working with JSON

Handraulic

I started off without a decent IDE, and, without any extensions/plugins to make life easier. Whilst this was OK to start with, as more resources found their way into the CF JSON template it started becoming a mess.

IDE/Text Editor

Hopefully you’ve got a favourite text editor that will save you some time. Some considerations for a good IDE/text editor are intellisense and extensions/plugins.

A couple of decent editors include VIM, Atom, Notepad++, Visual Code studios. Obviously there are heaps of good editors other there.

Visual Code Studio

When working with Microsoft Visual Code Studio I found the following extensions super helpful for building out the JSON file.

Access the extensions area of visual studio code via (1) on the image.

Other useful extensions to load/install were 2 & 3.

2. CloudFormation by aws-scripting-guy

This is a great extension to help build out your CF JSON file. Install the extension, open a new file, set syntax to JSON, type start + tab and watch it populate a basic AWS CF template for you.

{

“AWSTemplateFormatVersion”: “2010-09-09”,

“Description”: “”,

“Metadata”: {

},

“Parameters”: {

},

“Mappings”: {

},

“Conditions”: {

},

“Resources”: {

},

“Outputs”: {

}

}

Start typing the type of resource you wish to add (VPC for example), and, watch it populate a default VPC resource into your JSON. Then, you just work through filling in the blanks appropriately.

“name“: {

“Type”: “AWS::EC2::VPC”,

“Properties”: {

“CidrBlock”: “—-/–“,

“Tags”: [–]

}

}

The extension was missing a few resource types, so, you’ll be referring back to the AWS Resource Types to fill in the blanks.

3. JSON Tools by Erik Lynd

If your JSON is getting a bit unwieldy or ugly, JSON Tools helps fix up any alignment issues.

Ctrl(Cmd)+Alt+M for JSON pretty.

Alt+M for JSON minify.

Troubleshooting

Bastion Host

Sometimes, you’re not quite sure why a stack is not working as desired… ie. Why hasn’t my load balancers become active.

Well, you could look to get console access, or, I found it quicker and easier to setup another subnet, security gateway and micro EC2 instance to act as a bastion host. Make sure you allow it to be reachable from the internet with a Elastic IP and required security group.

Best Practices

Make sure you check out the AWS CF Best Practices.

Resulting Files

Will upload if any demand.

Overview

The requirement for dynamic DNS has been around for decades and companies like DynDNS have been an enabler for just as long. Script kiddies, IRC dudes, gamers & professionals often want to host services out of their homes for various reasons, but, may not have a static IP address for their internet connection. Dynamic DNS services allows the user to update a hostname with a provider to point back to the dynamic IP address allocated to the users modem. This allows people to reference the domain name record to return the IP address of the modem.

Note: I’m not talking about RFC2136 which includes a dynamic DNS mechanism in the DNS protocol.

I host a few services at home which I like to reach remotely from time to time, and, I’m too tight to pay for a static IP address. A few years ago this was the task I decided to force myself to solve using python in an attempt to learn. Whilst ugly, it served its purpose for quite some time until last night when I set myself a task to do this with Ansible in the evening.

The Players

Ansible is a simple automation tool with use cases across a number of use cases such as Provisioning, Configuration Management, Application Deployment, Orchestration and others. Ansible has plugins and modules which extend it’s functionality. In this case we are using the ipinfoio_facts and cloudflare_dns modules to query/communicate with…

Cloudflare I see as the Content Delivery Network (CDN) for the people. Free basic plans, API interfaces, proxying and DNS management.

ipinfo.io, a neat little site/service to give you geolocation information about where you are browsing from. This site also returns the data in JSON format if requested, which, makes it nice an easy to query programatically.

A linux Ansible command host to run the Ansible playbooks from…. and setup a crontab to continually run the playbooks.

Some domain names that I have registered with various domain registrars.

The Process (TL;DR)

Ensure you have a domain name to use.
Ensure you have a Cloudflare account, with, the domain name associated.
1. Take note of your cloudflare API token which is found under My Profile > API Key
Ensure you have a linux box with Ansible installed on it (tested with 2.3.x)
Clone https://github.com/Im0/CloudFlare_DyDNS_Playbook.git
Update the following fields in the cf_dydns_update.yml file
1. cf_api_token: ‘YOUR API KEY’
2. cf_email: ‘YOUR CLOUDFLARE EMAIL’
3. with_items: – The domain names you want to update
Run ansible with:
ansible-playbook cf_dydns_update.yml

Obviously, you’ll probably want different DNS records updated. Change the ‘record: mail’ an A record of your choice.

More detail

1 --- 2 - hosts: localhost 3 gather_facts: no 4 vars: 5 cf_api_token: 'CF API token under My Profile.. API key' 6 cf_email: 'Cloud Flare email address' 7 8 tasks: 9 - name: get current IP geolocation data 10 ipinfoio_facts: 11 timeout: 5 12 register: ipdata 13 14 # - debug: 15 # var: ipdata.ansible_facts.ip 16 17 - name: Update mail A record 18 cloudflare_dns: 19 zone: '{{ item }}' 20 record: mail 21 type: A 22 value: '{{ ipdata.ansible_facts.ip }}' 23 account_email: '{{ cf_email }}' 24 account_api_token: '{{ cf_api_token }}' 25 register: record 26 with_items: 27 - domain1 28 - domain2 29 30 # - debug: 31 # var: record

Breaking down the YML file..

Required at the top of oru YAML files
As we are not configuring any nodes, we set localhost as the only node we want to call against.
As we aren’t using any facts, we don’t need to collect them.
Variables we’re going to need to talk to cloudflare
The API token found under our profile
Our sign up email address for cloudflare
.
The tasks section for all tasks we are going to execute in this playbook
.
Using the ipinfoio_facts module we query ipinfo.io for our externally visible IP address. Note: If we are being a proxy of some sort this will likely break what we are trying to achieve.
.
This could probably be done a bit better and dropped. We are registering the output of the module to the ipdata variable. This could probably be removed as the returned data ends up in the gathered facts which we could use.
.
If we want to see what useful little nuggets of information that have come back, dump the variable contents.
.
.
.
Use the cloudflare_dns module to start talking to cloudflare
Which domain (zone) are we talking about? In this case we iterate over the domains listed starting line 26 ‘with_items’:
record: is the record we wish to update.
type, is the type of record we are working with. A few other examples are on the cloudflare_dns module page.
Use the data we received from ipinfoio. We’ve stashed this away in the data structure: ipdata.ansible_facts.ip
Our cloudflare email
Our cloudflare API key
Capture the output from the cloudflare_dns queries, if we want to dump it in debug later.
With items is a list of items we iterate over… instead of hosts.

Category: Automation

Dynamic DNS in less than 25 lines of Ansible YAML and CloudFlare CDN

Overview

The Players

The Process (TL;DR)

More detail