HTTP API

<< Core API	Module Upload >>
* We are improving this documentation all the time. Please feel free to contribute!! *

Most Cloud based Apps provide a REST/HTTP based API for getting and creating "objects" in user accounts through an authenticated connection. WEAVE@cloud provides a [low level "API" for building component modules](Core API) to sync/migrate data to any type of API (i.e. it does not have to be a REST/HTTP based). On top of that lower level API, WEAVE@cloud provides a higher level API specifically for the purpose of making it easier to define component modules for REST/HTTP based APIs. This guide documents that higher level REST/HTTP API.

Before reading this guide, be sure to read the [lower level "API"](Core API) and, in particular, familiarize yourself with the details of the foxweave-components.json module descriptor file.

The primary goal of the REST/HTTP API is to save people writing code where possible. Our hope is that most Cloud Services can just configure the SaaS Input and Output connectors for their service (via the foxweave-components.json) and then [assemble and upload the component module](Module Upload).

The following are a few examples of existing modules that use the REST/HTTP API:

The basic steps in using the REST/HTTP API are as follows:

Clone (duplicate/copy) one of the above existing component modules. Make sure you are familiar with the standard module structure.
Create the SaaS Component Configurations
Configure Data Model
Configure HTTP Sync/Migrate Operations
Configure Authentication

Component Configurations

You'll need to configure the SaaS Input and Output Connectors. These are the WEAVE@cloud HTTP runtime components and are provided by the `foxweave-abstract-components-http` module.

First thing to do is include foxweave-abstract-components-http as a dependency in your module's build.gradle file. See MailChimp's build.gradle file.

Then add SaaSInputConnector and SaaSOutputConnector as components to your component module's foxweave-components.json. Let's use the Zendesk component module as an example of how to do this. Note in particular the runtime configuration on both components:

"components": [
    {
        "name": "zendesk",
        "displayName": "Zendesk",
        "whatitdoes": "Listen for events in Zendesk, the popular customer Help Desk Service.",
        "type": "InputConnector",
        "category": "customer_support",
        "runtime": "com.foxweave.connector.http.SaaSInputConnector",
        "smalllogo": "ui/images/favicon.png",
        "largelogo": "ui/images/largelogo.png",
        "help": "ui/help/input-connector-help.html",
        "configUI": {
            "html": "ui/connector-ui.html",
            "viewJs": "ui/connector.js"
        },
        "deps": [
            {
                "groupId": "com.foxweave.connector",
                "artifactId": "http",
                "version": "0.1",
                "fileName": "/lib/foxweave-abstract-components-http-0.1.jar"
            },
            {
                "groupId": "com.foxweave",
                "artifactId": "zendesk",
                "version": "0.1",
                "fileName": "/lib/foxweave-components-zendesk-0.1.jar"
            }
        ]
    },
    {
        "name": "zendesk",
        "displayName": "Zendesk",
        "whatitdoes": "Create objects in Zendesk, the popular customer Help Desk Service.",
        "type": "OutputConnector",
        "category": "customer_support",
        "runtime": "com.foxweave.connector.http.SaaSOutputConnector",
        "smalllogo": "ui/images/favicon.png",
        "largelogo": "ui/images/largelogo.png",
        "help": "ui/help/output-connector-help.html",
        "configUI": {
            "html": "ui/connector-ui.html",
            "viewJs": "ui/connector.js"
        },
        "deps": [
            {
                "groupId": "com.foxweave.connector",
                "artifactId": "http",
                "version": "0.1",
                "fileName": "/lib/foxweave-abstract-components-http-0.1.jar"
            },
            {
                "groupId": "com.foxweave",
                "artifactId": "zendesk",
                "version": "0.1",
                "fileName": "/lib/foxweave-components-zendesk-0.1.jar"
            }
        ]
    }
]

Other things to note about the above configuration snippet:

Check the foxweave-components.json and Component UI docs for descriptions of the smalllogo, largelogo and configUI configurations.
You'll also notice how both the components have deps sets (dependency sets) for com.foxweave.connector:http and com.foxweave:zendesk. You'll need to add equivalent dependencies in your component's descriptor.

API Data Model

You may need to provide `dataModel` definitions for the Objects in your API/App. See the top level [dataModel docs][dataModel] for details.

Take a look at the dataModel sections in the module descriptors of the following Apps:

Sync/Migrate Operations

The `httpDescriptor` section of the [foxweave-components.json][descriptor] provides all the details needed by WEAVE@cloud to connect to a REST/HTTP API and perform data sync and migration operations on that API. Before we describe the different parts of the `httpDescriptor` section, be sure to check out the descriptors of the following APIs/Apps and see how they specify the different HTTP operations they support:

The basic structure of the httpDescriptor section is as follows:

"httpDescriptor" : {
    "baseUrl" : "[the base url construction template]",
    "objOperations" : {
        // set of sub-structures defining the supported operations for the objects (ala the objects defined in the dataModel section - see above)
    }
}

Config Value Templating

Before looking at the individual config fields of the `httpDescriptor` section we need to talk a little bit about how we use **templating** when specifying the field values. For example, the value defined for `baseUrl` on the [Zendesk descriptor][zendesk-desc] is `https://${auth.zendeskSubdomain}.zendesk.com/`. As you can see, the value contains **${auth.zendeskSubdomain}**. This tells the WEAVE@cloud runtime to replace this part of the base URL with the `zendeskSubdomain` value of the `auth` object (authAccount) selected by the user (see the `authDescriptor` specification in the [Zendesk descriptor][zendesk-desc]).

WEAVE@cloud provides a number of templating objects that can be used within configuration values all through the httpDescriptor specifications:

auth: The selected authentication/connection parameter set selected by the user.
objectName: The name of the object in question i.e. same as the name of the object from the dataModel section.
config: The runtime component configuration, as supplied to the component implementation.
message: The current pipeline message instance (CREATE operations only).
lastMessage: The last message instance pushed through "this" pipeline instance.
lastEvals: The "evals" results for the last message pushed through "this" pipeline instance. See below.

Operation Specification

WEAVE@cloud handles API synchronizations and migrations through the execution of predefined objOperations (object operations). WEAVE@cloud supports a number of operations, calling them in a predefined sequence during the lifecycle of a sync/migration process:

Migration Read Operations:

READ_INITIAL_MIGRATE: The first operation to be executed on a migration task.
READ_CONTINUE_MIGRATE: Migration task operation executed after the READ_INITIAL_MIGRATE.

Sync Read Operations:

READ_INITIAL_SYNC: The first operation to be executed on a synchronization task.
READ_NEW: Synchronization task operation executed after the READ_INITIAL_SYNC.

Create/Write Operations:

CREATE: Operation to be executed when creating objects on the API (as opposed to the READ_* operations listed above).

The READ_* operations are expected to return lists of objects in reverse chronological order i.e. the newest objects coming first in the list. You need to configure the operations to make sure this happens.

Configuring the operations is not too difficult but if you're having trouble, please contact us and we'll help you get it right. The following is an example of the basic structure

"[operation name i.e. 'READ_INITIAL_MIGRATE', 'READ_CONTINUE_MIGRATE', 'READ_INITIAL_SYNC', 'READ_NEW', 'CREATE']" : {
    "method": "[http method/verb i.e. 'GET', 'POST', 'PUT']",
    "url": "[operation URL - added to 'baseUrl' - use templates to inject values from 'auth' object etc as shown above]",

    // Optional URL query parameters e.g....
    "params": {
        // Name and Value e.g.....
        "sort_by": "created_at",
        "sort_order": "desc",
        "per_page": 1
    },

    // Operation 'response' evaluation expressions. Details of 'response' follow this snippet.
    "evals": {
        "success": "[an expression evaluating whether or not the operation was a success]",
        "data": "[an expression extracting pipeline 'data' from the 'response']"
    }
}

You can use templating within the values of any of the fields in an operation descriptor e.g. "query": "type:${objectName} created>${lastMessage.created_at}" is one of the Zendesk operation query param values.

The response object referred to in the evals section in the above snippet refers to an object that can be used in expressions in the following way

response.statusCode: The HTTP response status code.
response.statusText: The HTTP response status text.
response.charset: The HTTP response character set.
response.contentLength: The HTTP response content length.
response.json: The HTTP response body/payload as an addressable JSON structure.
response.body: The HTTP response body/payload as a raw String.
response.ok: true if the response status code is 200, otherwise false.
response.fail: true if the response status code is >= 300, otherwise false.
response.notFail: true if the response status code is >= 200 and < 300, otherwise false.

Again, check out the httpDescriptor section of the Zendesk descriptor for examples of valid evals configs.

For a complete example of the objOperations section (from inside the httpDescriptor section), lets use the OnePageCRM descriptor:

"objOperations" : {
    "*" : {
        "READ_INITIAL_MIGRATE, READ_CONTINUE_MIGRATE, READ_INITIAL_SYNC, READ_NEW" : {
            "method": "GET",
            "url": "/api/${objectName}s.json",
            "params": {
                "newest": 1,
                "page": "${pageNum}"
            },
            "evals": {
                "success": "response.notFail && response.json.status == 0",
                "data": "response.json.data.${objectName}s"
            }
        },
        "CREATE" : {
            "method": "POST",
            "url": "/api/${objectName}s.json",
            "evals": {
                "success": "response.notFail"
            }
        }
    }
}

Authentication

All SaaS providers require some form of authentication to their service. This section describes how to configure a WEAVE@cloud REST/HTTP service to handle some of those mechanisms.

Shared Secret Parameter Auth

A common form of authentication is the use of a shared secret (e.g. an **apiKey**), supplied as a query parameter during the execution of the HTTP operation. Defining this for your WEAVE@cloud component is very simple and just requires the following steps.

Step 1: Specify the details of the shared secret in the params section of the authDescriptor in the component's foxweave-components.json descriptor file e.g.

"authDescriptor" : {
    "title": "MyFunkySaaS Account",
    "params" : {
        "accountName": {
            "label" : "Account Alias",
            "hint": "Account name/alias used to refer to the MyFunkySaaS account accessed by the API Key specified below."
        },
        "apiKey": {
            "label" : "API Key",
            "type": "password",
            "hint": "MyFunkySaaS API Key.  Check 'API Keys & Authorized Apps' in your MyFunkySaaS Account Settings."
        }
    }
}

Step 2: Specify ${auth.apiKey} as the value for one of query parameters in the params section of the relevant httpDescriptor object operations in the foxweave-components.json descriptor file e.g.

"httpDescriptor" : {
    "baseUrl" : "https://app.myfunkysaas.com/",
    "contentType": "application/json",
    "objOperations" : {
        "*" : {
            "READ_INITIAL_MIGRATE" : {
                "method": "GET",
                "url": "/api/${objectName}s.json",
                "params": {
                    "apiKey": "${auth.apiKey}",
                    // other query parameters...
                },
                "evals": {
                    // evals...
                }
            }
        }
    }
}

BASIC Auth

BASIC Authentication is a common [HTTP standard based][basicauth] authentication scheme used by many HTTP applications and containers. Defining BASIC Auth for your WEAVE@cloud component is very simple and just requires the following steps.

Step 1: Specify the authType and auth tokens to be captured and used in the BASIC Auth Authorization header. We specify these in the authDescriptor in the component's foxweave-components.json descriptor file.

Taking Zendesk as an example:

"authDescriptor" : {
    "title": "Zendesk Account",
    "authType" : "httpBASIC",
    "params" : {
        "accountName": {
            "displayIndex": 1,
            "label" : "Account Alias",
            "hint": "Account name/alias used to refer to the Zendesk account accessed by this set of security credentials."
        },
        "zendeskSubdomain": {
            "displayIndex": 2,
            "label" : "Zendesk Help Desk Sub-domain",
            "hint": "The Help Desk Sub-domain name.  For example, if your Zendesk Dashboard URL is 'https://acme.zendesk.com/agent/#/dashboard', then your sub-domain should be set to 'acme'."
        },
        "agentEmail": {
            "displayIndex": 3,
            "label" : "Agent Email Address",
            "hint": "Zendesk Agent Email Address associated with the API Token."
        },
        "apiToken": {
            "displayIndex": 4,
            "label" : "Agent API Token",
            "type": "password",
            "hint": "Zendesk Agent API Token. Find this in Settings > Channels > API (press the 'Manage' icon/button, located on bottom left toolbar)."
        }
    }
}

Step 2: Specify the httpBASICAuthCreds configuration in the httpDescriptor section of the foxweave-components.json descriptor file.

Again, from Zendesk:

"httpDescriptor" : {
    "baseUrl" : "https://${auth.zendeskSubdomain}.zendesk.com/",
    "httpBASICAuthCreds": "${auth.agentEmail}/token:${auth.apiToken}",
    "contentType": "application/json",
    "objOperations" : {
        // ...
    }
}

OAuth v1.0

WEAVE@cloud supports [OAuth v1.0](http://tools.ietf.org/html/rfc5849) as a REST/HTTP authentication scheme. Defining OAuth v1.0 for your WEAVE@cloud component is very simple and just requires the following steps.

Step 1: Specify the authType in the authDescriptor. Using Twitter as an example:

"authDescriptor" : {
    "title": "Twitter Account",
    "authType" : "OAuth1a"
}

Step 2: Register WEAVE@cloud with your service so as to allow it authenticate on behalf of users. Contact us with details of the consumerKey, consumerSecret, request_token_url etc.

OAuth v2.0

WEAVE@cloud supports [OAuth v2.0](http://oauth.net/2/) as a REST/HTTP authentication scheme. Defining OAuth v2.0 for your WEAVE@cloud component is very simple and just requires the following steps.

Step 1: Specify the authType in the authDescriptor. Using OnePageCRM as an example:

"authDescriptor" : {
    "title": "OnePage CRM Account",
    "authType" : "OAuth2"
}

Step 2: Register WEAVE@cloud with your service so as to allow it authenticate on behalf of users. Contact us with details of the auth_code_url, access_token_url, refresh_token_url (if used) etc, as well as the confidential auth details (client_id, client_secret etc).

Providing this information in the following JSON structure will help a lot:

{
    "client_id": "XXXXXXXXXX",
    "client_secret": "YYYYYYYYYYYYYYY",
    "auth_code_url": "https://app.acmeservice.com/api/oauth/authorize",
    "access_token_url": "https://app.acmeservice.com/api/oauth/access_token",
    "refresh_token_url": "https://app.acmeservice.com/api/oauth/refresh_token",
    "auth_code_params": {
        "response_type": "code",
        "approval_prompt": "force",
        "access_type": "offline"
    },
    "access_token_params": {
        "grant_type": "authorization_code"
    },
    "refresh_token_params": {
        "grant_type": "refresh_token"
    }
}

This JSON is eventually maintained securely in WEAVE@cloud's database but for testing purposes, you can add the above JSON to a file named 'AAAA-oauth.json' (where 'AAAA' is the component's name as defined in the component descriptor) and place this file in the component's resources directory.

Contributing to the wiki

If you spot a typo, please feel free to edit the wiki directly
If you have have a longer contribution, please clone the wiki repo and submit a pull request
For anything else... please contact Tom Fennelly

Provide feedback

Saved searches

Use saved searches to filter your results more quickly