Copier Template for TF Tasks

This Copier template provides files for a Terraform or OpenTofu in a monorepo.

The tooling uses Task as the task runner for the template and the generated projects. It provides an opinionated configuration for Terraform and OpenTofu. This configuration enables projects to use built-in features of these tools to support:

Multiple infrastructure components in the same code repository. Each unit is a complete root module.
Multiple instances of the same component with different configurations
Disposable instances of a component for development and testing. Use this to create instances for the branches of your code as you need them.
Integration testing for every component.
Migrating from Terraform to OpenTofu. You use the same tasks for both.

This uses the identifier TF or tf for Terraform and OpenTofu. Both tools accept the same commands and have the same behavior. The tooling itself is just called tft in the documentation and code.

Quick Examples

To start a new project:

uvx copier copy git+https://github.com/stuartellis/tf-tasks my-project
cd my-project
TFT_CONTEXT=dev task tft:context:new
TFT_UNIT=my-app task tft:new

The tft:new task creates a unit, a complete Terraform root module. This root module includes code for AWS, so that it can work immediately. Enable remote state storage by adding the settings to the context, or use local state. Set to the AWS IAM role for TF with the tfvar tf_exec_role_arn.

You can then start working with your TF module:

# Set a default context and unit
export TFT_CONTEXT=dev TFT_UNIT=my-app

# Run tasks on the unit with the configuration from the context
task tft:init
task tft:plan
task tft:apply

You can also specifically set the unit and context for one task. This example runs the integration tests for the module:

TFT_CONTEXT=dev TFT_UNIT=my-app task tft:test

To create a disposable copy of the resources for a module, just set an identifier for the extra copy:

export TFT_CONTEXT=dev TFT_UNIT=my-app

# Create a disposable copy of my-app
TFT_EDITION=copy2 task tft:plan
TFT_EDITION=copy2 task tft:apply

# Destroy the extra copy of my-app
TFT_EDITION=copy2 task tft:destroy

# Clean-up: Delete the remote TF state for the extra copy of my-app
TFT_EDITION=copy2 task tft:forget

To see a list of all of the available tasks in a project, enter task in a terminal window:

task

If you have autocompletion for Task, this will show you commands as you type.

How It Works

First, you run Copier to either generate a new project, or to add this tooling to an existing project.

The tooling uses specific files and directories:

|- tasks/
|   |
|   |- tft/
|       |- Taskfile.yaml
|
|- tf/
|    |- .gitignore
|    |
|    |- contexts/
|    |   |
|    |   |- all/
|    |   |
|    |   |- template/
|    |   |
|    |   |- <generated contexts>
|    |
|    |- units/
|    |    |
|    |    |- template/
|    |    |
|    |    |- <generated unit definitions>
|    |
|    |- modules/
|
|- tmp/
|    |
|    |- tf/
|
|- .gitignore
|- .terraform-version
|- Taskfile.yaml

The Copier template:

Adds a .gitignore file and a Taskfile.yaml file to the root directory of the project, if these do not already exist.
Provides a .terraform-version file.
Provides the file tasks/tft/Taskfile.yaml to the project. This file contains the task definitions.
Provides a tf/ directory structure for TF files and configuration.

The tasks:

Generate a tmp/tf/ directory for artifacts.
Only change the contents of the tf/ and tmp/tf/ directories.
Copy the contents of the template/ directories to new units and contexts. These provide consistent structures for each component.

Units

You define a set of infrastructure code as a component. Each of the infrastructure components in a project is a separate TF root module, so that it can be created, tested, updated or destroyed independently of the others.

This tooling refers to these TF root modules as units. Each TF unit is a subdirectory in the directory tf/units/.

To create a new unit, use the tft:new task:

TFT_UNIT=my-app task tft:new

The tooling creates each new unit as a copy of the files in tf/units/template/. The template directory contains a complete and working TF module for AWS resources. This means that each new unit is immediately ready to use. You are free to change units as you need. For example, you can completely remove the AWS resources from a unit, or have different versions of the same providers in separate units.

The tooling only requires that each unit is a valid TF module with these tfvars:

environment_name (string)
product_name (string)
unit_name (string)
edition (string)

To avoid compatibility issues, I recommend that you use names that only include lowercase letters, numbers and hyphen characters, with the first character being a lowercase letter. Avoid defining environment and edition names that are longer than 7 characters, and unit names that are longer than 12 characters.

If you amend a module to not use AWS, ensure that you change the tests.

Contexts

This tooling has contexts to provide profiles for TF. Contexts enable you to deploy multiple instances of the same unit with different configurations, instead of needing to maintain separate sets of code for different instances.

To create a new context, use the tft:context:new task:

TFT_CONTEXT=dev task tft:context:new

Each context is a subdirectory in the directory tf/contexts/ that contains a context.json file and one .tfvars file per unit.

The context.json file is the configuration file for the context. It specifies metadata and settings for TF remote state. Each context.json file specifies two items of metadata:

description
environment

The description is deliberately not used by the tooling, so that you can leave it empty, or do whatever you wish with it. The environment is a string that is automatically provided to TF as the tfvar environment_name. There are no limitations on how your code uses this tfvar.

Here is an example of a context.json file:

{
  "metadata": {
    "description": "Cloud development environment",
    "environment": "dev"
  },
  "backend_s3": {
    "tfstate_bucket": "789000123456-tf-state-dev-eu-west-2",
    "tfstate_ddb_table": "789000123456-tf-lock-dev-eu-west-2",
    "tfstate_dir": "dev",
    "region": "eu-west-2",
    "role_arn": "arn:aws:iam::789000123456:role/my-tf-state-role"
  }
}

To enable you to share common tfvars across all of the contexts for a unit, the directory tf/contexts/all/ contains one .tfvars file for each unit. The .tfvars file for a unit in the all directory is always used, along with .tfvars for the current context.

The tooling creates each new context as a copy of files in tf/contexts/template/. It copies the standard.tfvars file to create the tfvars files that are created for new units.

To avoid issues, I recommend that you use context names that only include lowercase letters, numbers and hyphen characters, with the first character being a lowercase letter. Avoid defining context names that are longer than 12 characters.

Contexts exist to configure TF. To avoid coupling live resources directly to individual contexts, the tooling does not pass the name of the active context to the TF code, only the environment name that it specifies.

Extra Instances

TF has two different ways to create extra copies of the same infrastructure from a root module: the test feature and workspaces.

The test feature destroys the resources at the end of each test run. The state information about the resources is only held in the memory of the system. This means that it is not stored and no existing state data is updated.

Workspaces means that TF creates a new, separate state for the extra copy, so that you can maintain and update an extra copy for as long as you need it. We often use workspaces to deploy separate copies of infrastructure for development and testing, with different copies from features branches of the same code.

However, if you try to create multiple instances of the same infrastructure from the same root module with the same configuration then it will probably fail. TF will try to create new resources that use exactly the same attributes as the resources for the first copy. The provider will then receive requests from TF to create resources that have the same names as existing resources, and it is likely to handle the problem by refusing to create these new resources.

This tooling implements both TF test and workspaces features with a consistent means of ensuring that every copy of infrastructure can have unique names. It simply defines that every module has a tfvar called edition.

This ensures that every instance has an identifier that you can use in your TF code. You include the edition identifier in resource names to avoid conflicts between copies. For convenience, the template TF code provides locals that you can use to create unique resource names. The next section has more details about resource names.

For workspaces, the tooling automatically sets the value of the tfvar edition to match the variable TFT_EDITION. Set this variable in any way that you want. For example, you can configure your CI system to set the variable to match branch names. If you do not specify a edition identifier, TF uses the default workspace for state, and the value of the tfvar edition is default.

For tests, we should use a unique identifier per test run. The test in the unit template includes code to set the value of edition to a random string with the prefix tt, so that every test copy of infrastructure will have a unique value for the edition tfvar.

If you follow this example in your own test code and have used the tfvar edition in resource names then you can run tests and create multiple extra instances of infrastructure in parallel without issues.

Managing Resource Names

Use the environment, unit_name and edition tfvars in your TF code to define resource names that are both meaningful to humans and unique for each instance of the resource. This avoids conflicts between copies of infrastructure.

For convenience, the code in the unit template includes locals and outputs to help with this:

tft_handle - Normalizes the unit_name to the first 12 characters, in lowercase
tft_standard_prefix - Combines environment, edition and tft_handle, separated by hyphens

To avoid compatibility issues, I recommend that you use names that only include lowercase letters, numbers and hyphen characters, with the first character being a lowercase letter. Avoid defining environment and edition names that are longer than 7 characters, and unit names that are longer than 12 characters.

To ensure that the template code is compatible with older versions of Terraform, it currently does not use validations on the tfvars.

The test in the unit template includes code to set the value of the tfvar edition to a random string with the prefix tt. If you use the edition in resource names, this ensures that test copies of resources do not conflict with existing resources that were deployed with the same TF module.

Shared Modules

The project structure also includes a tf/shared/ directory to hold TF modules that are shared between the root modules in the same project. By design, the tooling does not manage any of these shared modules, and does not impose any requirements on them.

To share modules between projects, publish them to a registry.

Dependencies Between Units

By design, this tooling does not specify or enforce any dependencies between infrastructure components. If you need to execute changes in a particular order, specify that order in whichever system you use to carry out deployments.

This tooling does not explicitly support or conflict with the stacks feature of Terraform. I do not currently test with the stacks feature. It is unclear when this feature will be finalised, or if an equivalent will be implemented by OpenTofu.

Working with TF Versions

By design, this tooling uses the copy of Terraform or OpenTofu that you provide. It does not install or manage copies of Terraform and OpenTofu, and it is not dependent on specific versions of these tools.

You will need to use different versions of Terraform and OpenTofu for different projects. To handle this, use a tool version manager. The version manager will install the versions that you need and automatically switch between them as needed. Consider using tenv, which is a version manager that is specifically designed for TF tools. For projects that use multiple technologies, consider using mise, which can manage versions of many tools and programming languages.

The generated projects include a .terraform-version file so that your tool version manager installs and use the Terraform version that you specify. To use OpenTofu, add an .opentofu-version file to enable your tool version manager to install and use the OpenTofu version that you specify.

This tooling can switch between Terraform and OpenTofu. This is specifically to help you migrate projects from one of these tools to the other.

Install

This project uses several command-line tools. We can install all of these tools on Linux or macOS with Homebrew:

Git - brew install git
Task - brew install go-task
pipx OR uv - brew install pipx OR brew install uv

Set up shell completions for Task after you install it. Task supports bash, zsh, fish and PowerShell.

The helpers pipx and uv enable us to run Copier without installing it:

pipx run copier copy git+https://github.com/stuartellis/tf-tasks my-project

uvx copier copy git+https://github.com/stuartellis/tf-tasks my-project

You can install Terraform or OpenTofu any way that you wish. I recommend that you use tenv. The tenv tool automatically installs and uses the required version of Terraform or OpenTofu for the project. If cosign is present, tenv uses it to carry out signature verification on OpenTofu binaries.

# Install tenv with cosign
brew install tenv cosign

The tasks do not use Python or Copier. They only need a UNIX shell, Git, Task and Terraform or OpenTofu. This means that they can be run in a restricted environment, such as a continuous integration job. We can add tenv to any environment and then use it to install Terraform or OpenTofu.

Usage

To use the tasks in a generated project you always need:

A UNIX shell, such as Bash or Fish
Git
Task
Terraform or OpenTofu

The TF tasks in the template do not use Python or Copier. This means that they can be run in a restricted environment, such as a continuous integration job.

To see a list of the available tasks in a project, enter task in a terminal window:

task

The tasks use the namespace tft. This means that they do not conflict with any other tasks in the project.

Before you manage resources with TF, first create at least one context:

TFT_CONTEXT=dev task tft:context:new

This creates a new context. Edit the context.json file in the directory tf/contexts/<CONTEXT>/ to set the environment name and specify the settings for the remote state storage that you want to use.

This tooling currently only supports Amazon S3 for remote state storage.

Next, create a unit:

TFT_UNIT=my-app task tft:new

Use TFT_CONTEXT and TFT_UNIT to create a deployment of the unit with the configuration from the specified context:

export TFT_CONTEXT=dev TFT_UNIT=my-app
task tft:init
task tft:plan
task tft:apply

You will see a warning when you run init with a current version of Terraform. This is because Hashicorp are deprecating the use of DynamoDB with S3 remote state. To support older versions of Terraform, this tooling will continue to use DynamoDB for a period of time.

The `tft` Tasks

Name	Description
tft:apply	terraform apply for a unit*
tft:check-fmt	Checks whether terraform fmt would change the code for a unit
tft:clean	Remove the generated files for a unit
tft:console	terraform console for a unit*
tft:destroy	terraform apply -destroy for a unit*
tft:fmt	terraform fmt for a unit
tft:forget	terraform workspace delete*
tft:init	terraform init for a unit. An alias for `tft:init:s3`.
tft:new	Add the source code for a new unit. Copies content from the tf/units/template/ directory
tft:plan	terraform plan for a unit*
tft:rm	Delete the source code for a unit
tft:test	terraform test for a unit*
tft:units	List the units.
tft:validate	terraform validate for a unit*

*: These tasks require that you first initialise the unit.

The `tft:context` Tasks

Name	Description
tft:context	An alias for `tft:context:list`.
tft:context:list	List the contexts
tft:context:new	Add a new context. Copies content from the tf/contexts/template/ directory
tft:context:rm	Delete the directory for a context

The `tft:init` Tasks

Name	Description
tft:init	terraform init for a unit. An alias for `tft:init:s3`.
tft:init:local	terraform init for a unit, with local state.
tft:init:s3	terraform init for a unit, with Amazon S3 remote state.

Settings for Features

Set these variables to override the defaults:

TFT_PRODUCT_NAME - The name of the project
TFT_CLI_EXE - The Terraform or OpenTofu executable to use
TFT_REMOTE_BACKEND - Set to false to force the use of local TF state
TFT_EDITION - See the section on extra instances

Updating TF Tasks

To update projects with the latest version of this template, use the update feature of Copier:

cd my-project
uvx copier update -A -a .copier-answers-tf-task.yaml .

This synchronizes the files in your project that the template manages with the latest release of the template.

Copier only changes the files and directories that are managed by the template.

Using Extra Instances

Specify TFT_EDITION to create an extra instance of a unit:

export TFT_CONTEXT=dev TFT_UNIT=my-app TFT_EDITION=feature1
task tft:plan
task tft:apply

Each instance of a unit has an identical configuration as other instances that use the specified context, apart from the tfvar edition. The tooling automatically sets the value of the tfvar edition to match TFT_EDITION. This ensures that every edition has a unique identifier that can be used in TF code.

Only set TFT_EDITION when you want to create an extra copy of a unit. If you do not specify a edition identifier, TF uses the default workspace for state, and the value of the tfvar edition is default.

Once you no longer need the extra instance, run tft:destroy to delete the resources, and then run tft:forget to delete the TF remote state for the extra instance:

export TFT_CONTEXT=dev TFT_UNIT=my-app TFT_EDITION=copy2
task tft:destroy
task tft:forget

Testing

This tooling supports the validate and test features of TF. Each unit includes a minimum test configuration, so that you can run immediately run tests on the module as soon as it is created.

A test creates and then immediately destroys resources without storing the state. To ensure that temporary test copies of units do not conflict with other copies of the resources, the test in the unit template includes code to set the value of edition to a random string with the prefix tt.

To validate a unit before any resources are deployed, use the tft:validate task:

TFT_UNIT=my-app task tft:validate

To run tests on a unit, use the tft:test task:

TFT_CONTEXT=dev TFT_UNIT=my-app task tft:test

The tests create and destroy temporary copies of resources on the cloud services that being managed. Check the expected behaviour of the resources, because cloud services may not immediately remove some types of resources.

Using Local TF State

By default, this tooling uses Amazon S3 for remote state storage. To initialize a unit with local state storage, use the task tft:init:local rather than tft:init:

task tft:init:local

To use local state, you will also need to comment out the backend "s3" {} block in the main.tf file.

I highly recommend that you only use TF local state for prototyping. Local state means that the resources can only be managed from a computer that has access to the state files.

Using OpenTofu

By default, this tooling uses the copy of Terraform that is found on your PATH. Set TFT_CLI_EXE as an environment variable to specify the path to the tool that you wish to use. For example, to use OpenTofu, set TFT_CLI_EXE with the value tofu:

TFT_CLI_EXE=tofu

To specify which version of OpenTofu to use, create a .opentofu-version file. This file should contain the version of OpenTofu and nothing else, like this:

1.9.1

Remember that if you switch between Terraform and OpenTofu, you will need to initialise your unit again, and when you run apply it will migrate the TF state. The OpenTofu Website provides migration guides, which includes information about code changes that you may need to make.

Contributing

This tooling was built for my personal use. I will consider suggestions and Pull Requests, but I may decline anything that makes it less useful for my needs. You are welcome to fork the project.

Some of the configuration files for this project template are provided by my project baseline Copier template. To synchronize a copy of this project template with the baseline template, run these commands:

cd tf-tasks
copier update -A -a .copier-answers-baseline.yaml .

Name		Name	Last commit message	Last commit date
Latest commit History 223 Commits
.github		.github
LICENSES		LICENSES
template		template
.copier-answers-baseline.yaml		.copier-answers-baseline.yaml
.editorconfig		.editorconfig
.gitignore		.gitignore
.markdownlint-cli2.yaml		.markdownlint-cli2.yaml
.pre-commit-config.yaml		.pre-commit-config.yaml
.prettierignore		.prettierignore
.prettierrc.yaml		.prettierrc.yaml
.yamllint.yaml		.yamllint.yaml
LICENSE.md		LICENSE.md
README.md		README.md
REUSE.toml		REUSE.toml
Taskfile.yaml		Taskfile.yaml
copier.yaml		copier.yaml
pyproject.toml		pyproject.toml
trivy.yaml		trivy.yaml

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Repository files navigation

Copier Template for TF Tasks

Table of Contents

Quick Examples

How It Works

Units

Contexts

Extra Instances

Managing Resource Names

Shared Modules

Dependencies Between Units

Working with TF Versions

Install

Usage

The `tft` Tasks

The `tft:context` Tasks

The `tft:init` Tasks

Settings for Features

Updating TF Tasks

Using Extra Instances

Testing

Using Local TF State

Using OpenTofu

Contributing

License

About

Uh oh!

Releases 46

Uh oh!

Contributors 2

Uh oh!

Languages

License

stuartellis/tf-tasks

Folders and files

Latest commit

History

Repository files navigation

Copier Template for TF Tasks

Table of Contents

Quick Examples

How It Works

Units

Contexts

Extra Instances

Managing Resource Names

Shared Modules

Dependencies Between Units

Working with TF Versions

Install

Usage

The tft Tasks

The tft:context Tasks

The tft:init Tasks

Settings for Features

Updating TF Tasks

Using Extra Instances

Testing

Using Local TF State

Using OpenTofu

Contributing

License

About

Topics

Resources

License

Uh oh!

Stars

Watchers

Forks

Releases 46

Uh oh!

Contributors 2

Uh oh!

Languages

The `tft` Tasks

The `tft:context` Tasks

The `tft:init` Tasks