Proposal: JavaScript SDK

[svozza]

Background

Users would like to be able to interact with Workload Discovery without having to use the UI. The main use case for this is to allow for the scripting of importing accounts. However, it would also be possible to programmatically interact with the data in the Neptune database, e.g., listing all the S3 buckets and their related IAM policies and outputting the results to JSON for further processing.

Problem Statement

Currently the only way to interact with the Workload Discovery APIs is by using GraphQL queries. While this is possible, users need to sign the requests with sig4 and handwrite their queries to get the data they want.

Proposal

We will offer a JavaScript SDK that customers can use to to interact programmatically with the Workload Discovery API. This SDK will allow users to invoke queries and mutations with a simple JavaScript function call. The signing of requests will be handled internally by the SDK and will only require that the user runs their program with an IAM role that includes the appropriate permissions (see Appendix A).

Configuration

In order to issue requests to the API the SDK must have access to the URL of the GraphQL. There are two main ways we might do this:

• User provides the URL when initialising the client
• Automatically discover the URL by inspecting the output of the main WD CloudFormation stack

We may also want to provide a different set of APIs depending on whether the customer is using the AWS Organizations integration or not (see Supported GraphQL Operations), so this information would need to be provided too.

User Configuration

The user would provide the required information when creating the client.

const {createClient} = require('workload-discovery-sdk');

const client = createClient({
    apiUrl: '<api-url>',
    crossAccountDiscovery: 'AWS_ORGANIZATIONS' | 'SELF_MANAGED'
});

Pros:

• Explicit
• Simpler to implement

Cons:

• More work for consumer, you need to go to the AWS console or CLI to find this information
• Increased risk of misconfiguration
  • not a big problem for apiUrl but using the wrong crossAccountDiscovery value could cause issues (this issue can be mitigated by implementing the fix proposed in Accounts Removed From AWS Organization Still Appear in UI #396)

Automatic configuration

We would use the AWS SDK to query the WD CloudFormation stack to extract the information we need. This will require a change to the Outputs section of the main CloudFormation template to export the API URL and cross account discovery mode.

const {createClient} = require('workload-discovery-sdk');

const client = createClient({
    region: '<region>'
});

Pros:

• Consumer doesn’t need to find WD specific configuration information from the console or CLI
• User can’t misconfigure on initialisation so SDK can strictly enforce any restrictions based on cross account discovery mode

Cons

• More complex to implement as it adds an asynchronous initialisation step to client creation
  • Not a major issue as we will often need to make asynchronous calls to get credentials through the Provider chain
• Adds another npm dependency to the SDK
• Customer needs to add CloudFormation permissions to the IAM role they use to run their program (see Appendix A)

Supported GraphQL Operations

We must make a decision as to whether the SDK will support the invocation of all GraphQL operations or not.

All Operations

This option leaves it to users to refrain from using ‘unsafe’ APIs.

Pros:

• Simplest to implement
• Discovery process could use the SDK

Cons:

• Exposes APIs that allow users to write to Neptune and OpenSearch databases
  • Could cause consistency issues
  • Any changes made will be removed by the discovery process on its next run: any data that is in Neptune but not in either AWS Config or the resources discovered using the AWS SDKs is treated as deleted.
• Exposes APIs that allow users to add accounts in AWS_ORGANIZATION mode
  • Users do not manage account importing in this mode:
    • Any accounts added that are in the Organization will be overwritten whenever the discovery process runs
    • Accounts outside the org will not have any resources from AWS Config, which could cause confusion for customers (this issue can be addressed by implementing the fix proposed in Accounts Removed From AWS Organization Still Appear in UI #396)

Subset of Operations

In this case, we remove access to any APIs that can write to the Neptune and OpenSearch databases. We also remove access to the APIs that allow users to manage account discovery if WD is running in AWS_ORGANIZTIONS mode.

The following operations would not be available:

# These queries will be deprecated in v2.1.0
getLinkedNodesHierarchy 
batchGetLinkedNodesHierarchy

# The queries are use for account management and will only be
# unsupported in AWS_ORGANIZATION mode
getGlobalTemplate
getRegionalTemplate

# These mutations write to the Neptune database

deleteRelationships
deleteResources
addRelationships
addResources
updateResources

# These mutations write to OpenSearch
indexResources
deleteIndexedResources
updateIndexedResources

# The mutations are use for account management and will only be
# unsupported in AWS_ORGANIZATION mode
addAccounts
deleteAccounts
updateAccount
deleteRegions
addRegions
updateRegions

Pros:

• Prevents users from interfering with the Neptune and OpenSearch databases
• Prevents users from adding accounts that can’t be discovered properly when when running in AWS_ORGANIZATIONS mode

Cons:

• More complex to implement
• Can’t be used by discovery process as it requires access to most mutations
  • A possible compromise here would be to include the all generated GraphQL queries and mutations, which the discovery process could import and execute separately, but only expose functions in the SDK that use a subset of those operations.

Pagination

Several of the API queries are paginated, the SDK will provide a mechanism similar to the Amazon JS SDK v3 that allows users to page through the results returned using a for await...of loop. The Discovery process already implements this functionality and the way it is used is very similar to this proposal.

const {createClient, createPaginator} = require('wd-sdk');

// ..

const getResourcesPaginator = createPaginator(client.getResources, {pageSize: 1000});

const resources = [];

for await(const page of getResourcesPaginator({})) {
  resources.push(...page);
}      

Appendix A

Required AppSync policy to execute GraphQL operations

{
      "PolicyName": "access-appsync",
      "PolicyDocument": {
        "Version": "2012-10-17T00:00:00.000Z",
        "Statement": [
          {
            "Effect": "Allow",
            "Action": [
              "appsync:GraphQL"
            ],
            "Resource": "<wd-appsync-arn>/*"
          }
        ]
      }
} 

Users can further lock down what queries and mutations may be invoked by scoping the Resources field for the appsync:GraphQL action:

"Resource": [
    "<wd-appsync-arn>/types/Query/fields/getResources",
    "<wd-appsync-arn>/types/Mutation/fields/addAccounts"
]

Required CloudFormation policy for autoconfiguration

{
      "PolicyName": "wd-autoconfigure-policy",
      "PolicyDocument": {
        "Version": "2012-10-17",
        "Statement": [
          {
            "Effect": "Allow",
            "Action": [
              "cloudformation:DescribeStacks"
            ],
            "Resource": "*"
          }
        ]
      }
}   

[thebutler12]

This sounds great. A couple of things:

• Is it possible to see the operations that will be supported
• The metadata operations might be useful for getting high-level info from WD before making calls to get the full data set. Like it does in the UI. Could be used if consumers are using the SDK to build alternative UIs.

[svozza]

• Do you mean, have a function that lists the supported operations?
• Yeah, good point. It's easy to add them in

[thebutler12]

• That would be handy actually, i just meant in this proposal itself.

[svozza]

Yeah, I can add it to the relevant section.