Automating AWS EC2 Management with Python and Boto3

Photo by Henri Pham on Unsplash

Introduction

Hello once again dear reader! It’s time for another of my articles with a cool, random picture at the beginning.

We’re looking at Python again, but this time showing how it can save us time managing AWS resources.

Pre-requisites

1. A non-root AWS IAM account with enough rights to perform the required actions.
2. A Python development environment. As before, I am using AWS Cloud9 which is lightweight and integrates seamlessly with AWS services.
3. The Boto3 library installed on your Python environment. The Boto3 library is the official Amazon Web Services (AWS) SDK (Software Development Kit) for Python and it enables our script to communicate with AWS and manipulate AWS resources programmatically using Python code.
4. As ever, some general AWS service and architecture knowledge is useful, but hopefully I will explain everything clearly!
5. Finally knowledge on Git and GitHub would be handy. You can check out some of my previous articles to help with that.

Please also feel free to check out my Python GitHub repository, where you can find the source code from this article as well as some useful code snippets if you’re learning Python too. Link: Johnny Mac’s GitHub repo

The Challenge!

Scenario

Our DevOps engineering team often uses a development lab to test releases of our application. The managers are complaining about the rising cost of our development lab and need to save money by stopping our (for this example) 3 ec2 instances after all engineers are clocked out.

Create a Python script that you can run that will stop all instances.

Advanced

We want to ensure that only our Development instances are stopped to make sure nothing in Production is accidentally stopped. Add logic to your script that only stops running instances that have the Environment: Dev tag.

Environment Setup and Test Strategy

In order to meet the challenge, I have to do a little bit of setup.

The first thing is to install Boto3 on my Cloud9 environment.

Running pip install boto3 in the terminal takes care of that.

Next, we can see I have 4 EC2 instances in my AWS environment.

The machine with the name starting aws is my Cloud9 environment and I don’t want that being powered off since that’s where I’m running my script! So I will make a slight change to the challenge to ensure that is factored in.

The next slight change is to add the Environment: Dev tag to the two instances whose names start with LUITProject.

So when I run the script, only the LUITProject instances (i.e. the Dev machines) will be powered off.

My Cloud9 machine should be unaffected as I’ve coded that into my script and the remaining johnnymac machine should be unaffected as it does not have the Environment: Dev tag.

This should cover all bases to prove my script works as expected.

The Script and Test Results

So I actually just combined the Foundational and Advanced part of the challenge into one script shown below.

I will break down the functions in the following section.

For now, let’s get ready to run the script.

So before running the script, make sure all instances are running, then click Run on our Cloud9 machine.

Terminal output looks good…

…and checking the console confirms success! The two machines tagged Dev are stopped and the others are left alone.

Breaking Down the Code

I tried to make the code as readable as possible by breaking the steps into functions. However, I’ll provide some additional detail here.

import boto3

By importing boto3, the script can use its functionality to:

• Establish a connection to the AWS EC2 service, which is responsible for managing instances.
• Send requests to the EC2 service to get information about instances (for example, their state and tags).
• Issue commands to the EC2 service to perform actions on instances, such as stopping them.

def get_running_instances(ec2_client):
    response = ec2_client.describe_instances(Filters=[{'Name': 'instance-state-name', 'Values': ['running']}])
    return [instance for reservation in response['Reservations'] for instance in reservation['Instances']]

get_running_instances(ec2_client) is a function that retrieves a list of currently running virtual computers (called instances) on the AWS platform. I’ll go into a little more depth on this one as it actually does quite a bit for only a couple of lines!

ec2_client: This is an input to the function. It represents a connection to the AWS EC2 service, which manages the instances. By providing this connection to the function, it knows where to send requests and get information about the instances.

response = ec2_client.describe_instances(…): This line sends a request to the AWS EC2 service, asking it to provide details about the instances. It applies a filter so that only instances with a running state are included in the response. The information received from the AWS service is stored in a variable called response.

return [instance for reservation in response[‘Reservations’] for instance in reservation[‘Instances’]]: The information in the response variable is organized in a hierarchy, with a list of Reservations, and each reservation containing a list of Instances. This line goes through each reservation, and then each instance within the reservations, to create a simplified list containing all the running instances. This list is returned by the function.

def stop_instances_with_dev_tag(instances, ec2_client):
    for instance in instances:
        if should_stop_instance(instance):
            print(f'Stopping instance: {instance["InstanceId"]}')
            ec2_client.stop_instances(InstanceIds=[instance['InstanceId']])

stop_instances_with_dev_tag(instances, ec2_client): This function goes through the list of instances provided and checks if they meet the conditions we specified (having a specific Environment tag and not starting with aws in their name). If an instance meets the conditions, it sends a command to AWS to stop that instance.

def should_stop_instance(instance):
    environment_tag_value = get_tag_value(instance, 'Environment')
    instance_name = get_tag_value(instance, 'Name')
    return environment_tag_value == 'Dev' and not instance_name.startswith('aws')

should_stop_instance(instance): This function checks an individual instance to see if it meets the conditions for stopping (having a specific Environment tag and not starting with aws in their name). It returns True if the conditions are met and False otherwise.

def get_tag_value(instance, tag_key):
    return next((tag['Value'] for tag in instance['Tags'] if tag['Key'] == tag_key), '')

get_tag_value(instance, tag_key): This function retrieves the value of a specific tag associated with an instance. Tags are like labels that help identify and organize our instances.

def main():
    ec2_client = boto3.client('ec2')
    running_instances = get_running_instances(ec2_client)
    stop_instances_with_dev_tag(running_instances, ec2_client)

main(): This is the central function that orchestrates the entire process. It first connects to the AWS EC2 service, gets the list of running instances, and then calls the function to stop instances meeting the specified conditions.

Conclusion

Well, that’s it for another episode! I hope this have given some insight into how powerful Python can be for managing AWS resources.

Stay tuned for my next article where we will take our automation to the next level and look to use this code in a Lambda function and schedule it to run at a specified time.

As ever, please reach out to me with any comments or questions — until next time, take care!