tap-intercom

This is a Singer tap that produces JSON-formatted data following the Singer spec.

This tap:

• Pulls raw data from the Intercom v2.5 API
• Extracts the following resources:
  • Admins
  • Companies
  • Conversations
    • Conversation Parts
  • Data Attributes
    • Customer Attributes
    • Company Attributes
  • Leads
  • Segments
    • Company Segments
  • Tags
  • Teams
  • Users
• Outputs the schema for each resource
• Incrementally pulls data based on the input state

Streams

admins

• Endpoint: https://api.intercom.io/admins
• Primary key fields: id
• Foreign key fields: team_ids
• Replication strategy: FULL_TABLE
• Transformations: none

companies

• Endpoint: https://api.intercom.io/companies
• Primary key fields: id
• Foreign key fields: segments > id, tags > id
• Replication strategy: INCREMENTAL (query all, filter results)
  • Bookmark: updated_at (date-time)
• Transformations: de-nest segments, tags

company_attributes reference#list-customer-data-attributes)

• Endpoint: https://api.intercom.io/data_attributes/company
• Primary key fields: _sdc_record_hash ie. the hash of [id, name, description]
• Foreign key fields: none
• Replication strategy: FULL_TABLE
• Transformations: none

company_segments

• Endpoint: https://api.intercom.io/segments?type=company
• Primary key fields: id
• Foreign key fields: none
• Replication strategy: INCREMENTAL (query all, filter results)
  • Bookmark: updated_at (date-time)
• Transformations: none

conversations

• Endpoint: https://api.intercom.io/conversations
• Primary key fields: id
• Foreign key fields: assignee > id, author > id, customer > id, customers > id, teammate > id, tags > id, user > id
• Replication strategy: INCREMENTAL (query all, filter results)
  • Sort: updated_at asc
  • Bookmark: updated_at (date-time)
• Transformations: de-nest customers, tags

conversation_parts

• Endpoint: https://api.intercom.io/conversations/{conversation_id}
• Primary key fields: id
• Foreign key fields: conversation_id, author > id
• Replication strategy: FULL_TABLE (ALL for each changed parent Conversation)
• Transformations: Conversation parts with parent conversation_id

customer_attributes

• Endpoint: https://api.intercom.io/data_attributes/customer
• Primary key fields: _sdc_record_hash ie. the hash of [id, name, description]
• Foreign key fields: none
• Replication strategy: FULL_TABLE
• Transformations: none

leads

• Endpoint: https://api.intercom.io/contacts
• Primary key fields: id
• Foreign key fields: companies > id, segments > id, tags > id
• Replication strategy: INCREMENTAL (query all or filtered? TBD)
  • Sort: updated_at asc?
  • Bookmark query field: updated_since? TBD
  • Bookmark: updated_at (date-time)
• Transformations: de-nest companies, segments, social_profiles, tags

segments

• Endpoint: https://api.intercom.io/segments
• Primary key fields: id
• Foreign key fields: none
• Replication strategy: INCREMENTAL (query all, filter results)
  • Bookmark: updated_at (date-time)
• Transformations: none

tags

• Endpoint: https://api.intercom.io/tags
• Primary key fields: id
• Foreign key fields: none
• Replication strategy: FULL_TABLE
• Transformations: none

teams

• Endpoint: https://api.intercom.io/teams
• Primary key fields: id
• Foreign key fields: admin_ids
• Replication strategy: FULL_TABLE
• Transformations: none

users

• Endpoint: https://api.intercom.io/users
• Primary key fields: id
• Foreign key fields: companies > id, segments > id, tags > id
• Replication strategy: INCREMENTAL (query filtered)
  • Sort: updated_at asc
  • Bookmark query field: updated_since
  • Bookmark: updated_at (date-time)
• Transformations: de-nest companies, segments, social_profiles, tags

Authentication

Quick Start

1. Install

   Clone this repository, and then install using setup.py. We recommend using a virtualenv:

   > virtualenv -p python3 venv
   > source venv/bin/activate
   > python setup.py install
   OR
   > cd .../tap-intercom
   > pip install .

2. Dependent libraries The following dependent libraries were installed.

   > pip install singer-python
   > pip install singer-tools
   > pip install target-stitch
   > pip install target-json
   

   • singer-tools
   • target-stitch

3. Create your tap's config.json file. Intercom Authentication Types explains how to get an access_token. Make sure your OAuth Scope allows Read access to the endpoints above. Additionally, your App should use API Version v1.4. request_timeout is the time for which request should wait to get response. It is an optional parameter and default request_timeout is 300 seconds.

   {
       "access_token": "YOUR_API_ACCESS_TOKEN",
       "start_date": "2019-01-01T00:00:00Z",
       "request_timeout": 300,
       "user_agent": "tap-intercom <api_user_email@your_company.com>"
   }

   Optionally, also create a state.json file. currently_syncing is an optional attribute used for identifying the last object to be synced in case the job is interrupted mid-stream. The next run would begin where the last job left off.

   {
       "currently_syncing": "teams",
       "bookmarks": {
           "users": "2019-09-27T22:34:39.000000Z",
           "companies": "2019-09-28T15:30:26.000000Z",
           "conversations": "2019-09-28T18:23:53Z",
           "segments": "2019-09-27T22:40:30.000000Z",
           "company_segments": "2019-09-27T22:41:24.000000Z",
           "leads": "2019-09-27T22:16:30.000000Z"
       }
   }

4. Run the Tap in Discovery Mode This creates a catalog.json for selecting objects/fields to integrate:

   tap-intercom --config config.json --discover > catalog.json

   See the Singer docs on discovery mode here.

5. Run the Tap in Sync Mode (with catalog) and write out to state file

   For Sync mode:

   > tap-intercom --config tap_config.json --catalog catalog.json > state.json
   > tail -1 state.json > state.json.tmp && mv state.json.tmp state.json

   To load to json files to verify outputs:

   > tap-intercom --config tap_config.json --catalog catalog.json | target-json > state.json
   > tail -1 state.json > state.json.tmp && mv state.json.tmp state.json

   To pseudo-load to Stitch Import API with dry run:

   > tap-intercom --config tap_config.json --catalog catalog.json | target-stitch --config target_config.json --dry-run > state.json
   > tail -1 state.json > state.json.tmp && mv state.json.tmp state.json

6. Test the Tap

   While developing the intercom tap, the following utilities were run in accordance with Singer.io best practices: Pylint to improve code quality:

   > pylint tap_intercom -d missing-docstring -d logging-format-interpolation -d too-many-locals -d too-many-arguments

   Pylint test resulted in the following score:

   Your code has been rated at 9.73/10

   To check the tap and verify working:

   > tap-intercom --config tap_config.json --catalog catalog.json | singer-check-tap > state.json
   > tail -1 state.json > state.json.tmp && mv state.json.tmp state.json

   Check tap resulted in the following:

   The output is valid.
   It contained 127 messages for 10 streams.
   
       10 schema messages
       92 record messages
       25 state messages
   
   Details by stream:
   +---------------------+---------+---------+
   | stream              | records | schemas |
   +---------------------+---------+---------+
   | admins              | 12      | 1       |
   | company_attributes  | 226     | 1       |
   | customer_attributes | 61      | 1       |
   | segments            | 5       | 1       |
   | teams               | 5       | 1       |
   | tags                | 404     | 1       |
   | company_segments    | 3       | 1       |
   | conversations       | 2284    | 1       |
   | conversation_parts  | 52610   | 48      |
   | leads               | 480     | 1       |
   | users               | 6231    | 1       |
   | companies           | 7182    | 1       |
   +---------------------+---------+---------+

   To run unit tests run the following command:

   python -m nose
   

---

Copyright © 2019 Stitch