Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Provide step by step guide to creating a native provider #39

Merged
merged 6 commits into from
Jun 14, 2022
Merged
Changes from 5 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
161 changes: 146 additions & 15 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,47 +1,178 @@
# Pulumi Native Provider Boilerplate

This repo is a boilerplate showing how to create a native Pulumi provider.
This repository is a boilerplate showing how to create a native Pulumi provider.

### Background

This repository is part of the [guide for authoring and publishing a Pulumi Package](https://www.pulumi.com/docs/guides/pulumi-packages/how-to-author).

Learn about the concepts behind [Pulumi Packages](https://www.pulumi.com/docs/guides/pulumi-packages/#pulumi-packages).

## Creating a Pulumi Native Provider
See where Resource Providers fit into [the Pulumi architecture](https://www.pulumi.com/docs/intro/concepts/how-pulumi-works/#how-pulumi-works)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What are we trying to say with this line?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A reminder of the architecture diagram may help those more visually inclined. Changing the wording to clarify.


You can search-replace `xyz` with the name of your desired provider as a starting point for creating a provider that manages resources in the target cloud.
A Pulumi Resource Provider:
- is a gRPC server which allows for the Pulumi engine to create resources in a specific cloud
- holds the lifecycle logic for these cloud resources
- holds a pulumi JSON schema that describes the provider
- provides language-specific SDKs so resources can be created in whichever language you prefer

Most of the code for the provider implementation is in `pkg/provider/provider.go`.
When we speak of a "native" provider, we mean that all implementation is native to Pulumi, as opposed
to [Terraform based providers](https://github.com/pulumi/pulumi-tf-provider-boilerplate).

An example of using the single resource defined in this example is in `examples/simple`.
## Authoring a Pulumi Native Provider

A code generator is available which generates SDKs in TypeScript, Python, Go and .NET which are also checked in to the `sdk` folder. The SDKs are generated from a schema in `provider/cmd/pulumi-resource-xyz/schema.json`. This file should be kept aligned with the resources, functions and types supported by the provider implementation.
The following instructions cover providers maintained by Pulumi (denoted with a "Pulumi Official" checkmark on the Pulumi registry).
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we can just say "this guide assumes that the provider is written for the pulumi organisation

In the future, we will add instruction for providers published and maintained by the Pulumi community, referred to as "third-party" providers.

Note that the generated provider plugin (`pulumi-resource-xyz`) must be on your `PATH` to be used by Pulumi deployments. If creating a provider for distribution to other users, you should ensure they install this plugin to their `PATH`.
This boilerplate creates a working Pulumi-owned provider named `xyz`.
It implements a random number generator that you can [build and test out for yourself](#test-against-the-example) and then replace the Random code with code specific to your provider.

## Pre-requisites

Install the `pulumictl` cli from the [releases](https://github.com/pulumi/pulumictl/releases) page or follow the [install instructions](https://github.com/pulumi/pulumictl#installation)
### Prerequisites

> NB: Usage of `pulumictl` is optional. If not using it, hard code the version in the [Makefile](Makefile) of when building explicitly pass version as `VERSION=0.0.1 make build`
Ensure the following tools are installed and present in your `$PATH`:

## Build and Test
* [`pulumictl`](https://github.com/pulumi/pulumictl#installation)
* [Go 1.17](https://golang.org/dl/) or 1.latest
* [NodeJS](https://nodejs.org/en/) 14.x. We recommend using [nvm](https://github.com/nvm-sh/nvm) to manage NodeJS installations.
* [Yarn](https://yarnpkg.com/)
* [TypeScript](https://www.typescriptlang.org/)
* [Python](https://www.python.org/downloads/) (called as `python3`). For recent versions of MacOS, the system-installed version is fine.
* [.NET](https://dotnet.microsoft.com/download)

```bash
# build and install the resource provider plugin
$ make build install

# test
### Creating and Initializing the Repository

Pulumi offers this repository as a [GitHub template repository](https://docs.github.com/en/repositories/creating-and-managing-repositories/creating-a-repository-from-a-template) for convenience. From this repository:

1. Click "Use this template".
1. Set the following options:
* Owner: pulumi
* Repository name: pulumi-xyz-native (replace "xyz" with the name of your provider)
* Description: Pulumi provider for xyz
* Repository type: Public
1. Clone the generated repository.

From the templated repository:

1. Search-replace `xyz` with the name of your desired provider.

#### Build the provider and install the plugin

```bash
$ make build install
```

This will:

1. Create the SDK codegen binary and place it in a `./bin` folder (gitignored)
2. Create the provider binary and place it in the `./bin` folder (gitignored)
3. Generate the dotnet, Go, Node, and Python SDKs and place them in the `./sdk` folder
4. Install the provider on your machine.

#### Test against the example

```bash
$ cd examples/simple
$ yarn link @pulumi/xyz
$ yarn install
$ pulumi stack init test
$ pulumi up
```

Now that you have completed all of the above steps, you have a working provider that generates a random string for you.

#### A brief repository overview

You now have:

1. A `provider/` folder containing the building and implementation logic
1. `cmd/`
1. `pulumi-gen-xyz/` - generates language SDKs from the schema
2. `pulumi-resource-xyz/` - holds the package schema, injects the package version, and starts the gRPC server
2. `pkg`
1. `provider` - holds the gRPC methods (and for now, the sample implementation logic) required by the Pulumi engine
2. `version` - semver package to be consumed by build processes
2. `deployment-templates` - a set of files to help you around deployment and publication
3. `sdk` - holds the generated code libraries created by `pulumi-gen-xyz/main.go`
4. `examples` a folder of Pulumi programs to try locally and/or use in CI.
5. A `Makefile` and this `README`.

### Writing the schema

The [JSON schema file](https://www.pulumi.com/docs/guides/pulumi-packages/schema) is used by `pulumi-gen-xyz` to create language-specific SDKs.
It is, therefore, a central requirement for any resource provider.
Provider schemas can be machine-generated from an API specification, or handwritten.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it's best to say that a schema can be generated from a combination of a 3rd party API Specification + Pulumi specific logic, or handwritten

This repository provides the [xyz example schema](./provider/cmd/pulumi-resource-xyz/schema.json) to get you started.
[The Azure Native Provider schema](https://raw.githubusercontent.com/pulumi/pulumi-azure-native/master/provider/cmd/pulumi-resource-azure-native/schema.json) provides a much larger example.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this is a monster to create - I suggest pointing to aws-native as it's not too big to digest?

Refer to the [package schema documentation](https://www.pulumi.com/docs/guides/pulumi-packages/schema/#pulumi-package-schema) for additional details.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Additional details about?


### Implementing the gRPC methods

Once you have a schema that describes all the resources for your provider, you will need to implement the desired gRPC methods.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

"all resources and metadata"

You will find a mostly blank implementation of these in `pkg/provider/provider.go`.
Note that these methods do not necessarily link 1:1 to the Pulumi CLI commands.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

they do not - it's not 'not necessarily'


#### Basic Functionality

The struct and creation of the provider are implemented already:

```go
// provider/pkg/provider.go
type xyzProvider struct {
host *provider.HostClient
name string
version string
schema []byte
}

func makeProvider(host *provider.HostClient, name, version string, pulumiSchema []byte) (pulumirpc.ResourceProviderServer, error) {
// Return the new provider
return &xyzProvider{
host: host,
name: name,
version: version,
schema: pulumiSchema,
}, nil
}
```

You need to provide the following methods _for each resource_:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think a small line to suggest what these do will help


1. Check
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is it for each resource? I can't see 100s of sets of these for pulumi-aws-native

2. Diff
3. Create
4. Update
5. Read
6. Delete

[Resource lifecycle methods are documented here](https://pulumi-developer-docs.readthedocs.io/en/stable/providers/implementers-guide.html#custom-resource-lifecycle).

The following methods are necessary for every provider and are already implemented:

1. GetPluginInfo
2. GetSchema


#### Additional Methods

The [resource provider service](https://github.com/pulumi/pulumi/blob/master/sdk/proto/provider.proto) includes a few more gRPC methods that you may need to implement and can read more about.

### Build Examples

Create an example program using the resources defined in your provider, and place it in the `examples/` folder.

You can now repeat the steps for [build, install, and test](#test-against-the-example).


## Documentation

Please [follow this guide to add documentation to your provider](https://www.pulumi.com/docs/guides/pulumi-packages/how-to-author/#write-documentation).

## Configuring CI and releases

We use [centralized CI management for our providers](https://github.com/pulumi/ci-mgmt/tree/master/native-provider-ci).
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We don't yet do this so I don't think this is fair to say

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

#41 so it doesn't fall off the radar


### Third-party providers

1. Follow the instructions laid out in the [deployment templates](./deployment-templates/README-DEPLOYMENT.md).
Expand Down