Implement pipeform for Terraform Operations

[osterman]

Describe the Feature

Improve the console DX of terraform operations by presenting the user with a friendly UI to show terrafom progress.

https://github.com/magodo/pipeform

Expected Behavior

Running atmos terraform plan and atmos terraform apply will launch the pipeform UI. Note, this does not require pipeform to be installed, as it will be used as a native Go module.

Use Case

On the command line, the output from terraform is overwhelming and doesn't convey the overall progress of the operation. Terraform will feel much friendlier if we present the user with a progress indicator.

Describe Ideal Solution

The UI would be enabled by default, but can be disabled via the atmos.yaml configuration.

# atmos.yaml
terraform:
  ui:
    enable: [plan, apply]

pipeform.mp4

Requirements:

• If no TTY is attached, the UI should be automatically disabled
• If the CI=true environment variable is set, then UI should be automatically disabled
• If the --no-ui flag is explicitly passed, then the UI should be disabled.

Alternatives Considered

No response

Additional Context

The pipeform library is licensed MPL2 and is not currently implemented as a Go module. We've contacted the @magodo (author) and he said he would accept a PR converting it to a library. As part of this implementation, we will need to first open a PR for that work.

[suzuki-shunsuke]

pipeform is interesting, but honestly I'm not sure if we should really use this by default.
I tried pipeform a bit.
When we use pipeform, the plan result doesn't output in the terminal.

$ terraform plan -json | pipeform
# No output

I think this is inconvenient.
I'm not sure if we can convert the output of terraform plan -json to the conventional terraform output.

Furthermore, seems like we can't show the detail of changes.
I expected that we can choose a resource and show its detail, but seems like we can't.

By the way, the final view of terraform plan -json | pipeform looks strange.
I think we can contribute to pipeform.

pipeform shows nothing. This confused me.
I found I needed to move page by h and l key.

[osterman]

pipeform is interesting, but honestly I'm not sure if we should really use this by default. I tried pipeform a bit. When we use pipeform, the plan result doesn't output in the terminal.

$ terraform plan -json | pipeform
# No output

This sounds like a bug in pipeform. Per the docs, it says it's supported.

the following terraform commands are supported:

terraform refresh -json
terraform plan -json
terraform apply -auto-approve -json

---

Furthermore, seems like we can't show the detail of changes. I expected that we can choose a resource and show its detail, but seems like we can't.

This would definitely be a welcome enhancement. It would likely benefit your tooling for terraform as well. We can sponsor this development in pipeform, if you're interested.

---

By the way, the final view of terraform plan -json | pipeform looks strange. I think we can contribute to pipeform.

Yes, this is not good. It should probably work the way you described above, allowing individual resources to be expanded. They shouldn't "disappear".

---

Ideal World

• When using pipeform, raw terraform logs are written to disk, allowing further inspection upon failure
• The UI would enable inspection of any resource
• A flag like --ui=false would disable the UI on a per-run basis.
• When a TTY is not detected, or we're in CI (env), then UI is disabled

[suzuki-shunsuke]

This sounds like a bug in pipeform. Per the docs, it says it's supported.

Really? I can't find the docs.
In my envrionment, TUI is shown, but after TUI quits, no output is shown in the terminal.

[osterman]

@suzuki-shunsuke here is what I am referring to. Are we talking about the same thing?

[suzuki-shunsuke]

I think it means pipeform can provide TUI for these commands, but it doesn't mean pipeform can output Terraform's conventional outoputs to stdout.

[suzuki-shunsuke]

For pipeform to output the conventional Terraform command output, pipeform needs to convert the output of terraform plan -json to the conventional Terraform command output.
This means the output of terraform plan -json needs to include all information of the conventional Terraform command output.

I checked the output of terraform plan -json, but seems like it doesn't include the information of resource attributes.

For instance, terraform plan -json outputs the data like this:

{
  "@level": "info",
  "@message": "github_issue_label.fooo: Plan to create",
  "@module": "terraform.ui",
  "@timestamp": "2025-01-15T22:33:03.343780+09:00",
  "change": {
    "resource": {
      "addr": "github_issue_label.fooo",
      "module": "",
      "resource": "github_issue_label.fooo",
      "implied_provider": "github",
      "resource_type": "github_issue_label",
      "resource_name": "fooo",
      "resource_key": null
    },
    "action": "create"
  },
  "type": "planned_change"
}

We can see the resource github_issue_label.fooo is created, but we can't see the attributes of the resource.
We can't convert this output to the conventional output like this:

  # github_issue_label.fooo will be created
  + resource "github_issue_label" "fooo" {
      + color       = "FF0000"
      + description = "foo"
      + etag        = (known after apply)
      + id          = (known after apply)
      + name        = "fooo"
      + repository  = "hoge"
      + url         = (known after apply)
    }

[suzuki-shunsuke]

So unfortunately, to provide both TUI and the conventional output, we need to run terraform plan twice.

[suzuki-shunsuke]

to provide both TUI and the conventional output, we need to run terraform plan twice.

TUI has no meaning if you run terraform plan twice 😅

[osterman]

@suzuki-shunsuke if the objective is to also support the regular terraform output, I think that could still be accomplished.

  # github_issue_label.fooo will be created
  + resource "github_issue_label" "fooo" {
      + color       = "FF0000"
      + description = "foo"
      + etag        = (known after apply)
      + id          = (known after apply)
      + name        = "fooo"
      + repository  = "hoge"
      + url         = (known after apply)
    }

Isn't the workaround as simple as:

terraform plan -json -out planfile.out | pipeform
terraform show planfile.out

[suzuki-shunsuke]

Oh, I see. Sounds good.

[suzuki-shunsuke]

We need to improve pipeform.

1. Pipeform can't show the detail of resources from TUI
2. terraform plan -json | pipeform finishes at a last page which doesn't show any resources

We need to discuss these problems with pipeform's maintainers.

[osterman]

We need to discuss these problems with pipeform's maintainers.

I'll get the conversation started by opening an issue in the project and linking to this topic.

[osterman]

@suzuki-shunsuke the terraform plan handling was just resolved in:

• Display information in the "Plan" stage magodo/pipeform#10

[suzuki-shunsuke]

I'm working on it.

[suzuki-shunsuke]

• terraform plan -json | pipeform doesn't show any resource magodo/pipeform#11

[suzuki-shunsuke]

When terraform plan finishes quickly, pipeform may annoy because users need to interact pipeform and need to exit it by Ctrl-C.

So I'm wondering if this should be enabled by default.

The UI would be enabled by default, but can be disabled via the atmos.yaml configuration.

[suzuki-shunsuke]

I've created a pull request.

• feat: run pipeform when running terraform plan #981

This is still work in progress.

[suzuki-shunsuke]

How to install pipeform

There are several options:

1. Install pipeform automatically somehow
2. Disable pipeform by default, and let users install it by themselves
3. Embed pipeform as Go library

You would prefer 3.

About 3, this is what you mentioned at magodo/pipeform#9 .

We would like to refactor portions of the code into a public package from internal/

A detailed issue describing the needs is welcome

[suzuki-shunsuke]

I tried to replace pipeform command with Go package.

• feat: Support pipeform (Go package) #983
• https://github.com/suzuki-shunsuke/pipeform/tree/feat-api

[suzuki-shunsuke]

#926 (comment)

This workaround is unavailable for terraform apply because it doesn't support -out option.

[suzuki-shunsuke]

1. What do you think?
   Implement pipeform for Terraform Operations #926 (comment)
   Implement pipeform for Terraform Operations #926 (comment)
2. You can build and try feat: Support pipeform (Go package) #983 . I really appreciate it if you try it and give me your feedback, whether the experience meets your expectation.

I tried it:

go install .
cd examples/quick-start-simple
~/go/bin/atmos tf plan station -s dev

[osterman]

@suzuki-shunsuke Yes, I agree with you. Something is off with this from the DX for terraform plan. Let me think what we can do to improve this.

Instead, maybe we should strictly focus on apply for now. Also, the two-key press for Ctrl+C to exit the UI, is not user friendly. If we're contributing to pipeform, this might be one thing to consider adding an alternative key stroke of just q to quit.

[osterman]

Also, looks like a bug during plan, it says apply:

[osterman]

Aha, interesting, so navigating with the ← and → keys, you can traverse between the various phases of the plan. Instead, it appears that the summary is "broken" as it doesn't summarize anything.

[suzuki-shunsuke]

maybe we should strictly focus on apply for now.

About apply, we need to think about this. #926 (comment)

This workaround is unavailable for terraform apply because it doesn't support -out option.

[suzuki-shunsuke]

• feat: Support pipeform (Go package) #983 (comment)

Using pipeform as Go library makes atmos depend on CGO.
In general, depending on CGO is undesirable.
This is a trade-off.

[suzuki-shunsuke]

I'm concerned that UI of pipeform may confuse users who aren't familiar with it.

[suzuki-shunsuke]

Problems (Discussion topics)

There are some problems, so let's manage the list in this comment.

• Improve UX of pipeform
  • Ctrl-C isn't user-friendly
    • Implement pipeform for Terraform Operations #926 (comment)
    • Implement pipeform for Terraform Operations #926 (comment)
  • UI of pipeform may confuse users who aren't familiar with it
    • Implement pipeform for Terraform Operations #926 (comment)
    • Implement pipeform for Terraform Operations #926 (comment)
• apply can't show the conventional output
  • Implement pipeform for Terraform Operations #926 (comment)
  • Implement pipeform for Terraform Operations #926 (comment)
• Go library
  • Need to add public API to the upstream
    • Implement pipeform for Terraform Operations #926 (comment)
  • pipeform requires CGO Implement pipeform for Terraform Operations #926 (comment)
    • Build failed in CI feat: Support pipeform (Go package) #983 (comment)
    • Solved. Implement pipeform for Terraform Operations #926 (comment)
• CLI
  • Need to install pipeform somehow Implement pipeform for Terraform Operations #926 (comment)

[osterman]

• feat: Support pipeform (Go package) #983 (comment)

Using pipeform as Go library makes atmos depend on CGO.
In general, depending on CGO is undesirable.

Ugh! Yes, I agree with the implications on long term maintainability for a noncore feature. I am surprised by this! I assumed it was just a thin charmbracelet veneer. What about it depends on CGO?

[osterman]

#926 (comment)

This workaround is unavailable for terraform apply because it doesn't support -out option.

Ah, that's unfortunate, I didn't think about that. 🤦‍♂️ so there are a few things though, and I am not sure this is a deal breaker by itself. In a CI context the UI would be disabled, and tools like tfcmt would work well.

You've raised some big concerns and the scope may be larger than initially anticipated, in which case will reevaluate what we do.

At this point let me do some more research and get back to you.

[suzuki-shunsuke]

I'm sorry but depending on CGO was my misunderstanding.
#983 failed to build due to the following error.

make build-linux

env GOOS=linux  go build -o build/atmos -v -ldflags "-X 'github.com/cloudposse/atmos/pkg/version.Version=test'"

clipboard_linux.c:15:10: fatal error: X11/Xlib.h: No such file or directory
   15 | #include <X11/Xlib.h>
      |      

In general, CGO is enabled by default.
And make build-linux didn't disable CGO, meaning CGO was enabled.
If CGO is enabled, pipeform depends on golang.design/x/clipboard.

https://github.com/magodo/pipeform/blob/3ddc4c24f8d64825684c75648a4bd3f1258cad04/internal/clipboard/clipboard_cgo.go#L1-L5

//go:build cgo


package clipboard


import cb "golang.design/x/clipboard"

golang.design/x/clipboard depends on the following things.

https://github.com/golang-design/clipboard?tab=readme-ov-file#dependency

Dependency
macOS: require Cgo, no dependency
Linux: require X11 dev package. For instance, install libx11-dev or xorg-dev or libX11-devel to access X window system.
Windows: no Cgo, no dependency

On Linux, it depends on X11 dev package.
On macOS it depends on CGO.

That's why build failed as X11 dev package wasn't installed.

To solve the issue, I tried to disable CGO in Makefile. b8c0cac
Then build succeeded.