Add Terraform module support to manage the infrastructure lifecycle

[yufeiminds]

Hi, @daniel-hutao @IronCore864

Devstream is an awesome tool for managing the DevOps infrastructure lifecycle. And in my past works, I developed many infrastructure modules by Terraform and HCL.

Does DevStream interested in adding Terraform module support to manage the infrastructure lifecycle?

Backgound

Terraform is a tool to implement the IaC(Infrastructure as Code) pattern. I have also read many posts on Medium written by @IronCore864. Such as:

• 9 Extraordinary Terraform Best Practices That Will Change Your Infra World
• Security in Infrastructure as Code with Terraform — Everything You Need to Know

It's easy to read and understand. I enjoy reading it.

For example the Kubernetes Prow Module (pls ignore the tflint failed, I will fix it later~). Kubernetes Prow is an open-source CI/CD system supporting Chat-Ops slash style command.

Design

If I want to use DevStream to manage them, I should add a mechanism to call the Terraform root module.

I think there are three key points to archive it:

• How to download and manage the module
• How to call the Terraform modules
• How to manage the state of Terraform

Here is a simple solution, maybe only a toy. 😄

sequenceDiagram
    autonumber
    
    DevStream->>X Module: Call
    X Module->>Module Manager: Passthrough
    Module Manager->>TF Registry: Send download request
    TF Registry->>Module Manager: Download and save
    DevStream->>X Module: Call
    X Module->>Module Manager: Passthrough
    Module Manager->>TF StateManager: Save State & Statistics the state usage

Loading

The X is generic for any infrastructure workload, such as Prow. 😃

Two packages should be implemented:

• Module manager: like plugin manager, we also need a module manager to maintain the lifecycle of Terraform module. Download, init it to local and etc. Because we don't want to re-init it when applying every time.
• State Manager: I don't know the relationship between DevStream and Terraform state. If there are some tips that may be useful for this work.

[daniel-hutao]

Hi @yufeiminds, thank you for raising this discussion. We'll respond to you after an internal discussion this morning.

cc @IronCore864

[IronCore864]

Hi @yufeiminds, thanks for the great suggestion and detailed design!

At the moment, we are still discussing DevStream values and market positioning. After they are finalized, we will consider some great improvements for Helm/Terraform-related stuff, like your proposal. Please be patient at the moment. Thanks!

[yufeiminds]

OK, DevStream is a fantastic DevOps tool. I enjoy using it.