— Development, AWS, Lambda, Python, Automation — 1 min read — Mike Lapidakis
Update [February 10, 2021]: The Github community has actively contributed to the below code snippets over the past five years. I'd recommend reading the comments on the Gist page before proceeding with this post. Thank you, Github community!
One of the more challenging aspects to managing a large AWS infrastructure can be tag management for cost allocation and tracking. When you create an EC2 instance, several other assets are created with it, some of which generate charges that should be tracked over time. While keeping an instance's tags updated is fairly straightforward, ensuring that its EBS volumes, elastic IPs, elastic network interfaces and snapshots stay tagged appropriately can be a real headache.
In my quest to streamline the operations of my clients' AWS infrastructure using Lambda, I've created a Lambda Function that will write and update specific tags from an EC2 instance to that instance's attached volumes and network interfaces. I have this function's event set to trigger every hour to ensure the tags stay up to date.
Active Gist
1from __future__ import print_function23import json4import boto35import logging67#setup simple logging for INFO8logger = logging.getLogger()9logger.setLevel(logging.ERROR)1011#define the connection region12ec2 = boto3.resource('ec2', region_name="us-west-2")1314#Set this to True if you don't want the function to perform any actions15debugMode = False1617def lambda_handler(event, context):18 #List all EC2 instances19 base = ec2.instances.all()2021 #loop through by running instances22 for instance in base:23 24 #Tag the Volumes25 for vol in instance.volumes.all():26 #print(vol.attachments[0]['Device'])27 if debugMode == True:28 print("[DEBUG] " + str(vol))29 tag_cleanup(instance, vol.attachments[0]['Device'])30 else:31 tag = vol.create_tags(Tags=tag_cleanup(instance, vol.attachments[0]['Device']))32 print("[INFO]: " + str(tag))33 34 #Tag the Network Interfaces35 for eni in instance.network_interfaces:36 #print(eni.attachment['DeviceIndex'])37 if debugMode == True:38 print("[DEBUG] " + str(eni))39 tag_cleanup(instance, "eth"+str(eni.attachment['DeviceIndex']))40 else:41 tag = eni.create_tags(Tags=tag_cleanup(instance, "eth"+str(eni.attachment['DeviceIndex'])))42 print("[INFO]: " + str(tag))43 44#------------- Functions ------------------45#returns the type of configuration that was performed 46 47def tag_cleanup(instance, detail):48 tempTags=[]49 v={}50 51 for t in instance.tags:52 #pull the name tag53 if t['Key'] == 'Name':54 v['Value'] = t['Value'] + " - " + str(detail)55 v['Key'] = 'Name'56 tempTags.append(v)57 #Set the important tags that should be written here 58 elif t['Key'] == 'Application Owner':59 print("[INFO]: Application Owner Tag " + str(t))60 tempTags.append(t) 61 elif t['Key'] == 'Cost Center':62 print("[INFO]: Cost Center Tag " + str(t))63 tempTags.append(t) 64 elif t['Key'] == 'Date Created':65 print("[INFO]: Date Created Tag " + str(t))66 tempTags.append(t) 67 elif t['Key'] == 'Requestor':68 print("[INFO]: Requestor Tag " + str(t))69 tempTags.append(t) 70 elif t['Key'] == 'System Owner':71 print("[INFO]: System Owner Tag " + str(t))72 tempTags.append(t)73 else:74 print("[INFO]: Skip Tag - " + str(t))75 76 print("[INFO] " + str(tempTags))77 return(tempTags)
Using the tagging Lambda function in conjunction with a snapshotting function that copies a volume's tags to all newly created snapshots will ensure your billing reports and charge backs capture all charges associated with running that instance in AWS, nearly automatically.
The following snapshot script also cleans up old snapshot (you can set the offset on line 15). I normally set this function's event to trigger once a day, during a low transactional point. I also recommend setting the timeout on this function to five minutes, as the cleanup process can to take a very long time depending on the number of snapshots you keep and the number of volumes you're snapshotting.
Active Gist
1import boto32import logging3import datetime4import re5import time67#setup simple logging for INFO8logger = logging.getLogger()9logger.setLevel(logging.ERROR)1011#define the connection12ec2 = boto3.resource('ec2', region_name="us-west-2")1314#set the snapshot removal offset15cleanDate = datetime.datetime.now()-datetime.timedelta(days=5)1617#Set this to True if you don't want the function to perform any actions18debugMode = False1920def lambda_handler(event, context):21 22 if debugMode == True:23 print("-------DEBUG MODE----------")24 25 #snapshop the instances26 for vol in ec2.volumes.all():27 tempTags=[]28 29 #Prepare Volume tags to be importated into the snapshot30 if vol.tags != None:31 for t in vol.tags:32 33 #pull the name tag34 if t['Key'] == 'Name':35 instanceName = t['Value']36 tempTags.append(t)37 else:38 tempTags.append(t)39 else:40 print("Issue retriving tag")41 instanceName = "NoName"42 t['Key'] = 'Name'43 t['Value'] = 'Missing'44 tempTags.append(t)45 46 description = str(datetime.datetime.now()) + "-" + instanceName + "-" + vol.id + "-automated"47 48 if debugMode != True:49 #snapshot that server50 snapshot = ec2.create_snapshot(VolumeId=vol.id, Description=description)51 52 #write the tags to the snapshot53 tags = snapshot.create_tags(54 Tags=tempTags55 )56 print("[LOG] " + str(snapshot))57 58 else:59 print("[DEBUG] " + str(tempTags))60 61 print "[LOG] Cleaning out old entries starting on " + str(cleanDate)62 63 #clean up old snapshots64 for snap in ec2.snapshots.all():6566 #veryify results have a value67 if snap.description.endswith("-automated"): 68 69 #Pull the snapshot date70 snapDate = snap.start_time.replace(tzinfo=None)71 if debugMode == True:72 print("[DEBUG] " + str(snapDate) +" vs " + str(cleanDate)) 73 74 #Compare the clean dates75 if cleanDate > snapDate:76 print("[INFO] Deleteing: " + snap.id + " - From: " + str(snapDate))77 if debugMode != True:78 try:79 snapshot = snap.delete()80 except:81 82 #if we timeout because of a rate limit being exceeded, give it a rest of a few seconds83 print("[INFO]: Waiting 5 Seconds for the API to Chill")84 time.sleep(5)85 snapshot = snap.delete()86 print("[INFO] " + str(snapshot))
Using these two function in conjunction with one another will not only streamline your charge back and tagging models, but also ensure you have consistent snapshots of all of your instances over time. While I don't recommend snapshots as your sole backup method, I do recommend keeping at least one per day to speed up the recovery process if a disaster does occur.
If you're looking for a solid snapshotting and tagging solution, give these Lambda Functions a try. If you have any questions or suggestions, please leave a comment below or contact me on Twitter. I'm always looking for better ways to write and run these functions.