Skip to content

Add testing components page #15458

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 2 commits into
base: master
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,7 @@ menu:
name: Components
parent: iac-concepts
weight: 70
identifier: iac-concepts-components
aliases:
- /docs/intro/concepts/resources/components/
- /docs/concepts/resources/components/
Expand Down
147 changes: 147 additions & 0 deletions content/docs/iac/concepts/components/testing-components.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,147 @@
---
title_tag: "Testing Pulumi Components"
meta_desc: "Learn strategies and tools for testing Pulumi Components during development and in CI/CD workflows."
title: Testing Pulumi Components
h1: Testing Pulumi Components
meta_image: /images/docs/meta-images/docs-meta.png
menu:
iac:
name: Testing Components
parent: iac-concepts-components
weight: 1
---

When authoring Pulumi components, it's critical to ensure changes won't break Pulumi programs that use them, or violate organizational policies. This page outlines different testing strategies and tools you can use to confidently update and maintain components.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
When authoring Pulumi components, it's critical to ensure changes won't break Pulumi programs that use them, or violate organizational policies. This page outlines different testing strategies and tools you can use to confidently update and maintain components.
When authoring Pulumi components, it's critical to ensure that changes won't unintentionally break Pulumi programs that consume your components, nor violate organizational policies. This page outlines different testing strategies and tools you can use to confidently update and maintain components.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I say "unintentionally" since a major version change would intentionally break consuming programs.


## Why Testing Matters: Blast Radius and Change Safety

When a component is updated, it's important to understand what other projects or teams might be affected. For example, if a platform engineering team maintains a shared component that encodes sensitive company security details, which then need to be updated in response to a security incident or policy change, before rolling that out, they will need to verify that the update won’t break downstream applications.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
When a component is updated, it's important to understand what other projects or teams might be affected. For example, if a platform engineering team maintains a shared component that encodes sensitive company security details, which then need to be updated in response to a security incident or policy change, before rolling that out, they will need to verify that the update won’t break downstream applications.
When a component is updated, it's important to understand what other projects or teams might be affected. For example, if a platform engineering team updates a shared component that encapsulates sensitive company security details in response to a security incident or a new regulatory requirement, the platform team will need to verify that the update won’t unintentionally break downstream Pulumi programs.


## Testing Strategies

### Scenario: Security Updates

In this scenario, a platform team owns a reusable IAM component used by many services across the organization. After a security event, they need to update a policy document or rotate a role.

To assess the impact of this change, two primary methods can be used:

### Testing Strategy: Use `pulumi preview`
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
### Testing Strategy: Use `pulumi preview`
### Testing Strategy: Link to a local copy and use `pulumi preview`


`pulumi preview` shows what changes will occur if a project consumes the updated version of the component. This helps identify if the changes are additive, destructive, or trigger replacements.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
`pulumi preview` shows what changes will occur if a project consumes the updated version of the component. This helps identify if the changes are additive, destructive, or trigger replacements.
[The `pulumi preview` command](/docs/iac/cli/commands/pulumi_preview/) shows what changes will occur if a project consumes the updated version of the component. This helps identify if the changes are additive, destructive, or trigger replacements.


This method works best when:

- You have access to consumer projects
- You can link the component locally

Running `pulumi preview` works great for existing projects to detect any unexpected behavior after an update. The best part is that it tests your new code in exactly the environments where it will be used, meaning they are very accurate and will show any issues, even ones you may not have considered ahead of time. Unfortunately, it may not scale well if you have a lot of projects or don't have direct access to the Pulumi project's code.

### Testing Strategy: Integration and Unit Testing Tools

Set up **integration tests** using tools like:

- Language-specific [unit testing](/docs/iac/concepts/testing/unit/) tools
- Local test benches ([see below](#yaml-test-benches))
- CI/CD workflows (like [GitHub Actions](/docs/iac/using-pulumi/continuous-delivery/github-actions/)) that validate downstream usage

These tests can assert that the updated component produces expected outputs and maintains compatibility. This works well when you don't have access to the end-user programs. However, there are limits to what tests can detect. It's often very difficult to write enough tests to have 100% test coverage of all inputs. Often there are environment-specific problems related to configuration, secrets, or other factors that are not able to be recreated in the testing environment. So, while these approaches give you *some* security, they are not as comprehensive as simply running `pulumi preview` and seeing what breaks.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
These tests can assert that the updated component produces expected outputs and maintains compatibility. This works well when you don't have access to the end-user programs. However, there are limits to what tests can detect. It's often very difficult to write enough tests to have 100% test coverage of all inputs. Often there are environment-specific problems related to configuration, secrets, or other factors that are not able to be recreated in the testing environment. So, while these approaches give you *some* security, they are not as comprehensive as simply running `pulumi preview` and seeing what breaks.
These tests can assert that the updated component produces expected outputs and maintains compatibility. This approach works well when you don't have access to the end-user programs. However, there are limits to what these types of tests can detect: It's often difficult to write enough tests to have 100% test coverage for all possible inputs. Often there are environment-specific problems related to configuration, secrets, or other factors that are not able to be recreated in the testing environment. So, while these approaches give you *some* security, they are not as comprehensive as simply running `pulumi preview` in a consuming program and seeing what breaks.


#### Integration Testing via the Pulumi Go Provider SDK

If you're working in the Go programming language, components can be tested using the [Pulumi Go Provider SDK](https://github.com/pulumi/pulumi-go-provider), via the built-in `integration` test framework.

Features:

- Can test components and Pulumi programs in any language
- Stand up ephemeral environments
- Assert on created resources and their outputs
- Validate behavior across versions and configs

See ["Integration Testing"](/docs/iac/concepts/testing/integration/) and [`pulumi-go-provider/integration`](https://github.com/pulumi/pulumi-go-provider/tree/main/integration) for usage examples.

## Prevention Strategies

While the following methods aren't strictly "testing", they do accomplish similar goals. It helps to limit and reduce the potential range of problems that a breaking change could cause. It's important to remember that the primary purpose of testing is not the testing itself, but the way that tests can reduce the scope of potential problems and prevent production problems.

### Scenario: Cost Control with Abstractions

Platform teams may also want to limit certain choices. For instance, restricting EC2 instance types to specific sizes to manage cost. Changing this inside of a component will help prevent new resources from being made that are too costly.

Instead of relying solely on tests:

### Prevention Strategy: Use Enums to Control Inputs

You can model allowed values directly in your component’s schema using enum types:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@julienp which of our languages currently support enums? We'll want to call it out

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I believe only Python does


```ts
instanceSize: pulumi.Input<"t3.micro" | "t3.small" | "t3.medium">
```

This makes invalid configurations impossible to compile or deploy, removing the need for some kinds of tests.

### Prevention Strategy: Use Policies as Guardrails

Pulumi CrossGuard policies can enforce input constraints dynamically at deployment time. For example:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we link to CrossGuard?


```ts
policy.resourceOfType("aws:ec2/instance:Instance", (args, reportViolation) => {
if (!["t3.micro", "t3.small"].includes(args.props.instanceType)) {
reportViolation("Only t3.micro and t3.small are allowed.");
}
});
```

Use policies to enforce security, cost, or compliance constraints that complement component-level modeling. Suppose a component author had changed the input enum accidentally. This policy would catch that at runtime, preventing the costly cloud resources from being created and alerting the end user to the issue. Policies can function as always-on unit tests in that way. Read more about about this in ["Property Testing"](/docs/iac/concepts/testing/property-testing/).

## Debugging Strategies

When things go wrong or you need to understand your component more deeply, these approaches can help:

### YAML Test Benches

For quick iteration and fast feedback, you can write minimal YAML programs that consume your component.

Example scaffold:

```yaml
name: test-component
runtime: yaml
resources:
myResource:
type: mycompany:platform:MyComponent
properties:
name: test
```

Run with:

```bash
pulumi up
```

This is especially useful for catching schema errors, input validation bugs, and unexpected property name casing issues.

We chose YAML for the test bench because it's the simplest language to consume a component in. This approach avoids any confusing issues with multi-language naming, syntax problems within the test bench itself, or issues with SDK generation.

That said, if you're running into problems in one target language but not the other, the same approach can be used with any Pulumi language.

### Inspect the schema directly

Use `pulumi package get-schema` to inspect the component’s generated schema directly, including:

- Inputs and outputs
- Required fields
- Enum types
- Documentation metadata

```bash
pulumi package get-schema ./my-component
```

This provides essential details for debugging interop issues and schema mismatches.

## Learn more

- [Build a Component](/docs/iac/using-pulumi/build-a-component/)
- [Testing Pulumi Programs](/docs/iac/concepts/testing/)
- [Pulumi Provider SDK](/docs/iac/extending-pulumi/pulumi-provider-sdk/)
Loading