Organizing the terraform configuration files

Terraform configuration is easy to write and the syntax of HCL made it easy for the infrastructure developers to write down their Infra quickly and fast. Each developer has their own way of organizing the terraform configuration files.

In this article, we will talk about some of the ways to organize the terraform configuration files.

• Do factor out the resources that are repeated in nature and time-consuming with dependencies. For other users to reuse these resources, deliver these repeatable resources as modules.
  • A VPC Solution that has NAT, IGW and creates few public and private subnets
  • An EKS solution that creates an EKS cluster with all required components like ECR, S3, etc
  • A Hybrid solution that creates multiple VPCs with VPC Peering, Private endpoints, and routing tables that enable the communication between VPCs.
• Instead of storing the modules locally or in file shares, store each module as a separate repository, so that you can work on the module independently and version it every time you add a change. This allows the users to refer to the modules in their solution with the version that they are looking for. The module object in terraforming allows the user to give the source module a remote location as well. You can deliver one module for basic VPC setup, and create another version of the module that is VPC with Public and Private subnets. Refer to the below pseudo-code:

module "vpc_basic" {
   source: "git@github.com/abcd/vpc?ref=v1.0"
}

module "vpc_subnets" {
   source: "git@github.com/abcd/vpc?ref=v2.0"
}

1. Generalize the terraform configuration file instead of hardcoding the values. This enables us to use the variable file as input for the terraform configuration files. This makes it easy for other people to use it without changing the logic and they need simply to change the values in the variable files.
2. Reduce the number of input values to take. For example, you no need to take names as input for each resource, instead, you can build a logic that frames the names of the resources on the fly.

For example: You take ENV, PROJECT as input and you can build names using concatenation.

resource "aws_vpc" "vpc" {
    name = "vpc-${var.ENV}-${var.PROJECT}"
}

This will generate the VPC Name as “vpc-dev-poc”. Like this, you can avoid some of the inputs asking the user to provide.

• Use the variables file to declare the variables and use the .tfvar file to define the input values.

• DO NOT inject the credentials into the terraform provider.tf, instead use the environment variables in your system. If you are using Azure DevOps for deploying terraform files, use the service connections for authentication.

• Terraform creates the resources in parallel so make ensure that you add the dependencies properly. For example, you want to create VPC and IGW attach to it. If you do not mention the dependency for VPC in IGW, then IGW creation will fail. Because the VPC creation may not be completed by the time IGW is being provisioned. So, you need make ensure that IGW gets created only when VPC provisioning is completed.

For Example:

resources "aws_vpc" "vpc" {
  name = "demovpc"
}
resource "aws_internet_gateway" "igw" {
  name = "demovpc"
  vpcId = aws_vpc.vpc.id # this creates dependency since VPCID will only be generated when it VPC provisioning is completed
}

• Make sure that the module defines the outputs.tf, since the caller resource objects depend on the output of the module. You have to explicitly define the values that are needed to be exposed to the caller blocks.

• Have all configuration files in one place or a workspaces, so that you can manage then easily

• Terraform creates a state file to store the state of the resources created. if you lose the state file, terraform will fail to identify the resources that were created. So, do not store the state file locally because if your system gets crashed, you will lose the state file. Instead store the state file remotely such as on S3, Azure Storage, etc. Cloud service provides the object versioning, snapshots for the objects. These also provide better availability and durability.

These are some of the points that I usually follow during my infra development. Hope you find the article useful.