Introduction
Good day dear reader! Once again, we’re diving into Terraform. I’m building on my previous articles which I’ve listed in the Pre-requisites section, so please check them out for background.
I’m going to keep this one short and focused, so not too much pre-amble, other than to say we’re building on what we’ve learned so far and going a step further.
Specifically we are creating our own custom Terraform modules to enhance code reusability and we shall also delve into Terraform Cloud.
Terraform Cloud is an online service provided by HashiCorp that helps teams work together to build and manage digital infrastructure. Its main advantages are that it streamlines teamwork, automates tasks, and improves security compared to working on individual computers.
Pre-requisites
- Please checkout my Introduction to Terraform and Terraform: Creating a Highly Available, Scalable AWS Infrastructure articles which will provide a good foundation for this one.
- A non-root AWS IAM account with enough rights to perform the required actions.
- A development environment for creating and manipulating the Terraform files. I am using a Windows 11 workstation with WSL/PowerShell and VSCode installed.
- The AWS CLI should be installed on your system and your AWS account details configured.
- Terraform installed on your machine. The installers can be found on the official HashiCorp website.
- In this particular article, a good degree of AWS service and architecture knowledge is useful. If you need more detail, please check out any of my previous AWS articles where I explain everything simply and clearly!
- Finally knowledge on Git and GitHub would be handy. You can check out some of my previous articles to help with that.
Please also feel free to check out my GitHub repository, where you can find the source code from this article. Link: Johnny Mac’s Terraform GitHub repo
The Challenge!
Foundational Stage:
- Create a highly available two-tier AWS architecture containing the following:
a) Custom VPC with:
- 2 Public Subnets for the Web Server Tier.
- 2 Private Subnets for the RDS Tier.
- Appropriate route tables.
b) Launch an EC2 Instance with your choice of webserver in each public web tier subnet (apache, NGINX, etc).
c) One RDS MySQL Instance (micro) in the private RDS subnets.
d) Security Groups properly configured for needed resources ( web servers, RDS).
2. Deploy this using Terraform Cloud as a CI/CD tool to check your build.
3. Push your code to GitHub and include the link in your write up.
4. (Optional) Use custom modules to build out your code for repeatability.
NOTE: DO NOT FORGET TO SET YOUR ACCESS KEY, SECRET ACCESS KEY AND REGION ENVIRONMENT VARIABLES IN TERRAFORM CLOUD
Advanced Stage:
- Internet-facing Application Load Balancer targeting web servers.
- ALB Security Group with needed permissions and modifications needed to Web Servers SG to reflect the new architecture
There is a Complex Stage, but I will perhaps visit that in a later article!
So I will be walking through this article in a slightly different order to the challenge steps.
I will be creating the architecture from the Foundational and Advanced stages in one go and testing it locally first.
Only once everything is working will I migrate to Terraform Cloud. This is because I’ve not used and configured Terraform Cloud myself, so I don’t want to be potentially bogged down in troubleshooting should something go awry!
Task Preparation
The first thing we want to do is set up a logical folder structure so that we can organize our modules.
The code is organized into separate directories, each representing a module. These modules include vpc, alb, ec2 and rds, each corresponding to a specific component of our cloud architecture: Virtual Private Cloud (VPC), Application Load Balancer (ALB), Elastic Compute Cloud (EC2), and Amazon Relational Database Service (RDS), respectively.
Each module directory contains three files — main.tf, variables.tf and outputs.tf. The main.tf file holds the code that creates the resource. The variables.tf file contains any input parameters needed for the module, and the outputs.tf file captures the data produced by the module.
At the top level, there’s another main.tf file. This file calls the individual modules, passing along the necessary parameters and orchestrating the deployment.
The benefit of using modules in Terraform is the ability to encapsulate complex functionalities into reusable, manageable units. This approach not only increases the clarity of the code but also offers repeatability, making it easy to create identical environments, which is a common requirement in cloud computing for testing, staging, and production environments.
Here I’ve used the Linux tree command to hopefully give a useful visual representation of what I am discussing.
Please note each module has its own security group setup within its respective main.tf.
Given this, we don’t actually need a separate security_groups module for the setup we’ve described. So the security_groups directory and the corresponding main.tf, variables.tf, and outputs.tf files are not needed in this instance.
If however, there were common security groups used across multiple services, it might make sense to have a separate security_groups module to avoid redundancy and make the code easier to maintain. In that case, we would populate the security_groups directory with the relevant Terraform configuration for those shared security groups. So I’ve left it in my code and you can keep it or remove it to suit your needs.
@echo off
REM Create module directories
md modules\vpc modules\alb modules\ec2 modules\rds modules\security_groups
REM Create required files in each module
for /D %%a in (modules\*) do (
echo. > %%a\main.tf
echo. > %%a\variables.tf
echo. > %%a\outputs.tf
)
REM Create main.tf in the root directory
echo. > main.tf
@echo onAs a bonus for all my Windows readers, here’s a little batch file to create the folder structure for you. Aren’t I generous!?
As for the code in this task, I’m not going to list it all — there’s simply too much! Please check my GitHub repo linked in the Pre-requisites section for all code used in this challenge.
What I will do is list the top level main.tf and the files for the VPC to show how the top level main.tf references a module.
Top Level main.tf
module "vpc" {
source = "./modules/vpc"
vpc_cidr = "10.0.0.0/16"
public_subnet1_cidr = "10.0.1.0/24"
public_subnet2_cidr = "10.0.2.0/24"
private_subnet1_cidr = "10.0.3.0/24"
private_subnet2_cidr = "10.0.4.0/24"
availability_zone1 = "eu-west-3a"
availability_zone2 = "eu-west-3b"
}
module "alb" {
source = "./modules/alb"
public_subnets = module.vpc.public_subnets
vpc_id = module.vpc.vpc_id
}
# Get the latest AWS Linux 2 image
data "aws_ami" "amazon_linux" {
most_recent = true
filter {
name = "name"
values = ["amzn2-ami-hvm-2.0.*-x86_64-gp2"]
}
owners = ["137112412989"] # Amazon
}
module "ec2" {
source = "./modules/ec2"
public_subnets = module.vpc.public_subnets
ami_id = data.aws_ami.amazon_linux.id
instance_type = "t2.micro"
key_name = "jmac-le-pair"
webserver_sg_id = module.alb.alb_sg_id
target_group_arn = module.alb.target_group_arn
}
module "rds" {
source = "./modules/rds"
private_subnets = module.vpc.private_subnets
vpc_id = module.vpc.vpc_id
db_name = "luitweek22db"
db_username = "dbusername"
db_password = "dbpassword"
sg_id = module.alb.alb_sg_id
}So as we mentioned previously, this file has the specific resource parameters defined and it calls the relevant modules through the file paths defined in each source.
Top Level outputs.tf
output "alb_dns_name" {
description = "The DNS name of the ALB"
value = module.alb.alb_dns_name
}Just as an aside, I made a couple of changes as I was writing this article.
I added a variables.tf, a providers.tf and as shown here, an outputs.tf to the top level.
The former two files were to set the region I wanted to use to avoid issues I was having with infrastructure attempting to be brought up in the wrong location.
The outputs.tf I thought was a good additional example on how to reference another module in a different location. In Terraform, modules are isolated from each other unless explicitly referenced.
Here we are explicitly calling a value from the alb module.
VPC main.tf
resource "aws_vpc" "main" {
cidr_block = var.vpc_cidr
enable_dns_support = true
enable_dns_hostnames = true
tags = {
Name = "main_vpc"
}
}
resource "aws_subnet" "public_subnet1" {
vpc_id = aws_vpc.main.id
cidr_block = var.public_subnet1_cidr
map_public_ip_on_launch = true
availability_zone = var.availability_zone1
tags = {
Name = "public_subnet1"
}
}
resource "aws_subnet" "public_subnet2" {
vpc_id = aws_vpc.main.id
cidr_block = var.public_subnet2_cidr
map_public_ip_on_launch = true
availability_zone = var.availability_zone2
tags = {
Name = "public_subnet2"
}
}
resource "aws_subnet" "private_subnet1" {
vpc_id = aws_vpc.main.id
cidr_block = var.private_subnet1_cidr
availability_zone = var.availability_zone1
tags = {
Name = "private_subnet1"
}
}
resource "aws_subnet" "private_subnet2" {
vpc_id = aws_vpc.main.id
cidr_block = var.private_subnet2_cidr
availability_zone = var.availability_zone2
tags = {
Name = "private_subnet2"
}
}
resource "aws_internet_gateway" "igw" {
vpc_id = aws_vpc.main.id
}
resource "aws_route_table" "public" {
vpc_id = aws_vpc.main.id
route {
cidr_block = "0.0.0.0/0"
gateway_id = aws_internet_gateway.igw.id
}
}
resource "aws_main_route_table_association" "a" {
vpc_id = aws_vpc.main.id
route_table_id = aws_route_table.public.id
}In the VPC module, the variables are referenced to customize the resource properties. This allows the module to be reused with different input parameters. For instance, when creating the VPC or subnets, the CIDR blocks are not hard-coded but instead specified as variables like var.vpc_cidr and var.public_subnet1_cidr.
The actual values for these variables are not defined in the main.tf file itself but are passed in when the module is called from the top-level main.tf file.
VPC variables.tf
variable "vpc_cidr" {}
variable "public_subnet1_cidr" {}
variable "public_subnet2_cidr" {}
variable "private_subnet1_cidr" {}
variable "private_subnet2_cidr" {}
variable "availability_zone1" {}
variable "availability_zone2" {}Here we define our VPC variable names of course. Each variable does not have a set value as those are determined by the contents of the top level main.tf. So this is how our modules can be reused between different projects — they do not have hard coded values.
VPC outputs.tf
output "vpc_id" {
value = aws_vpc.main.id
}
output "public_subnets" {
value = [aws_subnet.public_subnet1.id, aws_subnet.public_subnet2.id]
}
output "private_subnets" {
value = [aws_subnet.private_subnet1.id, aws_subnet.private_subnet2.id]
}The main purpose of an outputs.tf in each subfolder or child module is twofold:
- They allow you to access specific data from your child modules within your root module. Once a value is outputted from a child module, it can be referenced in the root module, enabling inter-module data sharing.
- They can be useful for debugging and testing individual modules. When you’re developing or testing a module, you can run Terraform commands directly in the module directory and see the output values.
Terraform Commands on our Local Workstation
Now that our files are set up the way we want, we can run our Terraform commands to check everything meets the required standards and then bring up our infrastructure.
We’re in the correct working directory, so let’s kick things off with a terraform init.
Everything looks good.
terraform fmt is next and only our main.tf needed a bit of formatting.
terraform validate confirms our code is free from typos and mistakes.
terraform plan previews all the resources to be created and no errors are returned at this stage.
Finally, terraform apply -auto-approve brings up our infrastructure and our configuration outputs the DNS name of our ALB.
As the eagle eyed may notice, I made a few revisions and corrections between taking the terraform plan screenshot and the terraform apply screenshot!
Anyway, inputting our ALB’s DNS name into a browser confirms our web server is working as it should…
…and a couple of refreshes confirms the load balancing is working too as we are directed to our other web server.
Our last check is to confirm our database instance is working. To do that, we will need to connect to it via one of our EC2 instances. Remember our database instance is in a private subnet, so we can’t connect to it externally via the internet.
So we check an instance in the AWS Console and grab the public IP address…
…which allows us to SSH in.
Then we need to install MySQL, which we do via the sudo yum install mysql command.
Next, we go to the Amazon RDS Dashboard and confirm our database instance is available. Looks good, so click on the DB Identifier…
On the Connectivity & security tab, take a copy of the Endpoint name.
mysql -h {hostname} -P {port} -u {username} -p{password}The Endpoint name is our hostname in the command we used to connect to our database. Also, watch out as there is no space between the -p switch and the password.
If I planned ahead better, I should have actually output the RDS Endpoint name in my terminal in the same manner as I did with the ALB DNS name. This would have avoided the need for going into the console. I’ll know next time!
Back to the terminal, we enter the mysql command and as shown, we get a successful connection.
show databases; lists the databases on our RDS instance and is the last step in confirming our infrastructure works as expected.
Let’s tidy up by running terraform destroy.
Now, we can migrate to the Cloud!
Terraform Cloud
Okay dear reader, time is pressing now, so let’s crack on!
Step 1 is to go to https://app.terraform.io/signup/account and create a Terraform Cloud account if you do not have one.
I’m not going to run through all the steps in detail like I usually do, but you are welcome to checkout the official HashiCorp cloud login instructions which go over everything in depth.
Step 2, once you’ve authenticated your login, is to Create a new organization and associate it with your email address.
Step 3 is to Create a new Workspace.
I will select CLI-driven workflow as I am using the command line on my workstation.
Next configure your workspace settings and click Create workspace when ready.
Over to the menu on the left, I will make one change to the default settings under General and set the Terraform Working Directory to match the folder for the code in my GitHub repo.
Don’t forget the press the Save settings button at the bottom of the screen!
Next, select Version Control and click the Connect to version control button.
We’re going to select Version control workflow to link to our GitHub repo.
Choose your provider and in my case, I click on a pop up to allow Terraform Cloud connection to my selected GitHub repository.
Note: you may need to disable your pop up blocker for this step.
Once you have selected and confirmed which repositories you will use, click the Update VCS settings button at the bottom of the page.
Step 4: Next, we want to configure our AWS access.
Navigate back to Overview on the left hand menu and click Configure variables.
I’ve added 3 variables. My AWS ID and Access Key are confidential, so they can be marked as sensitive and will not be displayed openly.
I have also set my default AWS region. Note that the AWS_DEFAULT_REGION environment variable will overwrite any region code you have set in your AWS provider block.
Step 5: Go to your terminal and type terraform login. You will get a browser window opened on which you press a button to generate a token.
Copy the token and save it somewhere safe!
Enter the generated token where prompted on your terminal and if everything works, you’ll get the Terraform Cloud login screen.
Step 6: We now need to alter a couple of our top level configuration files to ensure we use Terraform Cloud.
terraform {
backend "remote" {
organization = "TopKoda-luit"
workspaces {
name = "luit-week22"
}
}
}So this code block will be added to the top level main.tf to tell Terraform to use the remote Cloud backend.
provider "aws" {
# AWS details are provided as environment variables in Terraform Cloud now
}Our top level providers.tf will be changed as above. In fact, we could probably delete it altogether, but I will leave the file in place for now.
Step 7: terraform.init
We can see that the backend configuration is now using (remote) Terraform Cloud.
Final Testing of Terraform Cloud
Next, I ran a terraform plan just to check the CI/CD pipeline was working properly.
That was confirmed by the bottom entry on the Run List.
Next, I changed the user_data section in the main.tf of my ec2 module. I added a line to print the current date and time on my web server page.
Upon committing the change to my GitHub repository, Terraform Cloud automatically detected the change and started a new run as shown in the other runs listed above.
Lastly, let’s bring up the infrastructure one final time.
I chose to be prompted before applying, so I have to manually confirm that I want to proceed.
We get confirmation in the Terraform page that our infrastructure is up and our new ALB DNS name is output.
Let’s put that into a browser…
…and hurrah! Our updated web page with the current date and time is displayed.
After a couple of refreshes, I remembered I had two user_data sections in my ec2 module — one for each web server. However, I only updated one, so one web server displays the date and time and the other doesn’t. Oops! Although, I guess we confirmed the load balancer works anyway.
Tidy Up
Time to bring everything down. Press Queue destroy plan on the Destruction and Deletion menu.
Confirm the workspace as directed and click Queue destroy plan.
Confirm & Apply if appropriate…
…and a few minutes later, everything is gone. Without even going into the AWS Console.
Conclusion
Well, there we are dear reader. The end of another epic!
I’m not going to lie, I spent weeks on this. I really struggled to get this over the line due to lots of other real life stuff. So I really hope you appreciate and enjoy the results of my extended labour.
Our exploration into the realms of AWS and Terraform was a testament to the potency of infrastructure as code.
Our aim was clear: create a robust and dynamic web service, ready to cater to the unpredictable ebb and flow of the digital world. The recipe we used combined several key ingredients, including a VPC, EC2 instances, an ALB and an RDS, all within the AWS ecosystem.
The magic, however, was in how these elements were combined and managed. That’s where Terraform came in. It allowed us to orchestrate our infrastructure, evolving it as easily as we would our own application code.
By leveraging Terraform and AWS, we crafted an infrastructure ready to embrace change. We’ve established a platform that isn’t just prepared for the future; it’s designed to make the most of it.
