-
Notifications
You must be signed in to change notification settings - Fork 245
Add testing components page #15458
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: master
Are you sure you want to change the base?
Add testing components page #15458
Changes from all commits
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Original file line number | Diff line number | Diff line change | ||||
---|---|---|---|---|---|---|
@@ -0,0 +1,147 @@ | ||||||
--- | ||||||
title_tag: "Testing Pulumi Components" | ||||||
meta_desc: "Learn strategies and tools for testing Pulumi Components during development and in CI/CD workflows." | ||||||
title: Testing Pulumi Components | ||||||
h1: Testing Pulumi Components | ||||||
meta_image: /images/docs/meta-images/docs-meta.png | ||||||
menu: | ||||||
iac: | ||||||
name: Testing Components | ||||||
parent: iac-concepts-components | ||||||
weight: 1 | ||||||
--- | ||||||
|
||||||
When authoring Pulumi components, it's critical to ensure changes won't break Pulumi programs that use them, or violate organizational policies. This page outlines different testing strategies and tools you can use to confidently update and maintain components. | ||||||
|
||||||
## Why Testing Matters: Blast Radius and Change Safety | ||||||
|
||||||
When a component is updated, it's important to understand what other projects or teams might be affected. For example, if a platform engineering team maintains a shared component that encodes sensitive company security details, which then need to be updated in response to a security incident or policy change, before rolling that out, they will need to verify that the update won’t break downstream applications. | ||||||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Suggested change
|
||||||
|
||||||
## Testing Strategies | ||||||
|
||||||
### Scenario: Security Updates | ||||||
|
||||||
In this scenario, a platform team owns a reusable IAM component used by many services across the organization. After a security event, they need to update a policy document or rotate a role. | ||||||
|
||||||
To assess the impact of this change, two primary methods can be used: | ||||||
|
||||||
### Testing Strategy: Use `pulumi preview` | ||||||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Suggested change
|
||||||
|
||||||
`pulumi preview` shows what changes will occur if a project consumes the updated version of the component. This helps identify if the changes are additive, destructive, or trigger replacements. | ||||||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Suggested change
|
||||||
|
||||||
This method works best when: | ||||||
|
||||||
- You have access to consumer projects | ||||||
- You can link the component locally | ||||||
|
||||||
Running `pulumi preview` works great for existing projects to detect any unexpected behavior after an update. The best part is that it tests your new code in exactly the environments where it will be used, meaning they are very accurate and will show any issues, even ones you may not have considered ahead of time. Unfortunately, it may not scale well if you have a lot of projects or don't have direct access to the Pulumi project's code. | ||||||
|
||||||
### Testing Strategy: Integration and Unit Testing Tools | ||||||
|
||||||
Set up **integration tests** using tools like: | ||||||
|
||||||
- Language-specific [unit testing](/docs/iac/concepts/testing/unit/) tools | ||||||
- Local test benches ([see below](#yaml-test-benches)) | ||||||
- CI/CD workflows (like [GitHub Actions](/docs/iac/using-pulumi/continuous-delivery/github-actions/)) that validate downstream usage | ||||||
|
||||||
These tests can assert that the updated component produces expected outputs and maintains compatibility. This works well when you don't have access to the end-user programs. However, there are limits to what tests can detect. It's often very difficult to write enough tests to have 100% test coverage of all inputs. Often there are environment-specific problems related to configuration, secrets, or other factors that are not able to be recreated in the testing environment. So, while these approaches give you *some* security, they are not as comprehensive as simply running `pulumi preview` and seeing what breaks. | ||||||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Suggested change
|
||||||
|
||||||
#### Integration Testing via the Pulumi Go Provider SDK | ||||||
|
||||||
If you're working in the Go programming language, components can be tested using the [Pulumi Go Provider SDK](https://github.com/pulumi/pulumi-go-provider), via the built-in `integration` test framework. | ||||||
|
||||||
Features: | ||||||
|
||||||
- Can test components and Pulumi programs in any language | ||||||
- Stand up ephemeral environments | ||||||
- Assert on created resources and their outputs | ||||||
- Validate behavior across versions and configs | ||||||
|
||||||
See ["Integration Testing"](/docs/iac/concepts/testing/integration/) and [`pulumi-go-provider/integration`](https://github.com/pulumi/pulumi-go-provider/tree/main/integration) for usage examples. | ||||||
|
||||||
## Prevention Strategies | ||||||
|
||||||
While the following methods aren't strictly "testing", they do accomplish similar goals. It helps to limit and reduce the potential range of problems that a breaking change could cause. It's important to remember that the primary purpose of testing is not the testing itself, but the way that tests can reduce the scope of potential problems and prevent production problems. | ||||||
|
||||||
### Scenario: Cost Control with Abstractions | ||||||
|
||||||
Platform teams may also want to limit certain choices. For instance, restricting EC2 instance types to specific sizes to manage cost. Changing this inside of a component will help prevent new resources from being made that are too costly. | ||||||
|
||||||
Instead of relying solely on tests: | ||||||
|
||||||
### Prevention Strategy: Use Enums to Control Inputs | ||||||
|
||||||
You can model allowed values directly in your component’s schema using enum types: | ||||||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. @julienp which of our languages currently support enums? We'll want to call it out There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I believe only Python does |
||||||
|
||||||
```ts | ||||||
instanceSize: pulumi.Input<"t3.micro" | "t3.small" | "t3.medium"> | ||||||
``` | ||||||
|
||||||
This makes invalid configurations impossible to compile or deploy, removing the need for some kinds of tests. | ||||||
|
||||||
### Prevention Strategy: Use Policies as Guardrails | ||||||
|
||||||
Pulumi CrossGuard policies can enforce input constraints dynamically at deployment time. For example: | ||||||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Should we link to CrossGuard? |
||||||
|
||||||
```ts | ||||||
policy.resourceOfType("aws:ec2/instance:Instance", (args, reportViolation) => { | ||||||
if (!["t3.micro", "t3.small"].includes(args.props.instanceType)) { | ||||||
reportViolation("Only t3.micro and t3.small are allowed."); | ||||||
} | ||||||
}); | ||||||
``` | ||||||
|
||||||
Use policies to enforce security, cost, or compliance constraints that complement component-level modeling. Suppose a component author had changed the input enum accidentally. This policy would catch that at runtime, preventing the costly cloud resources from being created and alerting the end user to the issue. Policies can function as always-on unit tests in that way. Read more about about this in ["Property Testing"](/docs/iac/concepts/testing/property-testing/). | ||||||
|
||||||
## Debugging Strategies | ||||||
|
||||||
When things go wrong or you need to understand your component more deeply, these approaches can help: | ||||||
|
||||||
### YAML Test Benches | ||||||
|
||||||
For quick iteration and fast feedback, you can write minimal YAML programs that consume your component. | ||||||
|
||||||
Example scaffold: | ||||||
|
||||||
```yaml | ||||||
name: test-component | ||||||
runtime: yaml | ||||||
resources: | ||||||
myResource: | ||||||
type: mycompany:platform:MyComponent | ||||||
properties: | ||||||
name: test | ||||||
``` | ||||||
|
||||||
Run with: | ||||||
|
||||||
```bash | ||||||
pulumi up | ||||||
``` | ||||||
|
||||||
This is especially useful for catching schema errors, input validation bugs, and unexpected property name casing issues. | ||||||
|
||||||
We chose YAML for the test bench because it's the simplest language to consume a component in. This approach avoids any confusing issues with multi-language naming, syntax problems within the test bench itself, or issues with SDK generation. | ||||||
|
||||||
That said, if you're running into problems in one target language but not the other, the same approach can be used with any Pulumi language. | ||||||
|
||||||
### Inspect the schema directly | ||||||
|
||||||
Use `pulumi package get-schema` to inspect the component’s generated schema directly, including: | ||||||
|
||||||
- Inputs and outputs | ||||||
- Required fields | ||||||
- Enum types | ||||||
- Documentation metadata | ||||||
|
||||||
```bash | ||||||
pulumi package get-schema ./my-component | ||||||
``` | ||||||
|
||||||
This provides essential details for debugging interop issues and schema mismatches. | ||||||
|
||||||
## Learn more | ||||||
|
||||||
- [Build a Component](/docs/iac/using-pulumi/build-a-component/) | ||||||
- [Testing Pulumi Programs](/docs/iac/concepts/testing/) | ||||||
- [Pulumi Provider SDK](/docs/iac/extending-pulumi/pulumi-provider-sdk/) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I say "unintentionally" since a major version change would intentionally break consuming programs.