Sharing snapshots & dodging yaml landmines

So recently I had a real world problem to solve from a developer;

“We need a DEV SQL server in a secondary account… and we also need to restore production snapshots from the production account INTO the DEV SQL server“

This is probably a very real requirement people come across every day. Developers needing to restore production data into test and dev servers to ensure the products they are building are fit for purpose.

This is a simple process, documented below, involved a CloudWatch event and a Lambda function.

This process would be pretty simple to setup in a single environment, but I wanted to respect that this could be setup across many and any accounts, and package it into a sam-cli template. Spoiler alert, yaml hurts if it’s wrong!

TL;DR – https://github.com/alanblockley/ec2-shapshot-share

Firstly, I’m going to put praise on the sam-cli framework for making the creation of a small app/function/package like this so easy from the get go. A simple sam init and not only do we have a “Hello World” but also a reasonable folder structure and a README.md, pre-written and ready to be customised for your app.

This app was written in Python2.7, utilising boto3.

sam init --runtime python2.7

Simple hey?

Now, a quick butchery of the template.yaml and we’re on our way, renaming the HelloWorld references with our function name. Also, rename the directory structure too to reference your app and then give it a quick local test;

sam local invoke --event event.json EC2ShapshotShare

Et, voila! We have our baseline framework in place. Now to add our own function in place of the hello world example. Remember, we’re using boto3 and we’ll be using an ec2 client to gain access to the EBS Snapshot info.

https://raw.githubusercontent.com/alanblockley/ec2-shapshot-share/master/ec2-snapshot-share/app.py

As you can see this is a very generic script, requires event input and environment variables to work. This could be laborious having to setup on each account with its partner account. However, as we’ve started this project using sam-cli, we can leverage it’s template.yaml to build a cloud-formation stack to help build the rest.

PSA: I’m not here to teach Python… so please, double check my syntax’s against the AWS SDK Documentation to understand WHAT is happening. Needless to say, this is my second Python project and it probably shows.

If at first you don’t succeed… YAML! When building the template.yaml there’s 1 rule to remember throughout the whole process. “YAML IS GOD”. This is not a preference, but a mandate. I hate yaml as much as the next person but unfortunately was forced to obey it’s heavy demands of alignment. By example, below, I was creating the Cloudwatch Event and spent over an hour not understanding why i could not get it to trigger on “createSnapshot” AND “succeeded”. I checked my typo’s and even tried adding a debug SNS topic (email to myself), to which it would not trigger. In the end I had the wrong amount of spaces in my indents for the Event… and a silent fail.

https://raw.githubusercontent.com/alanblockley/ec2-shapshot-share/master/template.yaml

The last thing to be concerned about is the deployment of the script.

Make any changes you require and test them using the local invoke (as above).

Edit the template.yaml file and change the DEST_ACCOUNT to reflect your destination account you are sharing to.

Then simply use the deploy command.

(You may need to make an S3 bucket and amend the script to reflect your installation bucket)

./deploy.sh

This is basically a premade deployment script to run the following;

sam package --template-file template.yaml --output-template-file ec2-snapshot-share.yaml --s3-bucket mybucket

aws cloudformation deploy --template-file ec2-snapshot-share.yaml --stack-name ec2-snapshot-share --capabilities CAPABILITY_IAM --region ap-southeast-2

This will deploy the package into your bucket and then initialise a Cloudformation Stack OR create a change set and apply it to the stack.