tap-twitter-ads

This is a Singer tap that produces JSON-formatted data following the Singer spec.

This tap:

Pulls raw data from the Twitter Ads API, version 7 using the Twitter Ads Python SDK.

Twitter Ads Tap: Sync Review: Summary of tap sync process and looping

Extracts the following normal GET endpoints:
- account_media
- accounts
- advertiser_business_categories
- bidding_rules
- campaigns
- cards:
- content_categories
- funding_instruments
- iab_categories
- line_item_apps
- line_items (aka ad_groups)
  - targeting_criteria (for each line_item)
- media_creatives
- preroll_call_to_actions
- promotable_users
- promoted_accounts
- promoted_tweets
- scheduled_promoted_tweets
- tailored_audiences
- tweets (scheduled and published, not draft)
Extracts the following targeting GET endpoints:
- targeting_conversations
- targeting_devices
- targeting_events (for countries)
- targeting_interests
- targeting_languages
- targeting_locations (for countries)
- targeting_network_operators (for countries)
- targeting_platform_versions
- targeting_platforms
- targeting_tv_markets
  - targeting_tv_shows (for each)
Extracts Asyncronous Reports using:
- Supports many reports, each with Report Config Settings:
  - Name: Name of report
  - Entity: 7 Entity Types
  - Segment: 20 Segmentation Types
  - Granularity: DAY, HOUR, or TOTAL
- Twitter Ads Entity Segmentation Rules (Google Sheet)
- Metrics and Segmentation Rules
- active_entities
- async_queued_jobs
- async_job_status
- async_results (download URL)
Outputs the schema for each resource
Incrementally pulls data based on the input state

Authentication

Twitter Ads requires authentication headers using OAuth 1.0a with an access token obtained via 3-legged OAuth flow. The access token, once generated, is permanent, but request tokens are short-lived without a documented expiry. The process is described in Obtaining Ads Account Credential.

Quick Start

Install

Clone this repository, and then install using setup.py. We recommend using a virtualenv:

> virtualenv -p python3 venv
> source venv/bin/activate
> python setup.py install
OR
> cd .../tap-twitter-ads
> pip install .

Dependent libraries The following dependent libraries were installed.

> pip install singer-python
> pip install singer-tools
> pip install target-stitch
> pip install twitter-ads (v7.0.0)

singer-tools
target-stitch
twitter-ads

Create your tap's config.json with the following parameters:

start_date: Absolute beginning date for bookmarked endpoints.
user_agent: Tap name and email address for API logging.
OAuth 1.0a credentials:
- consumer_key
- consumer_secret
- access_token
- access_token_secret
account_ids: Comma-delimited list of Twitter Ad Account IDs.
attribution_window: Number of days for latency look-back period to allow analytical reporting numbers to stabilize.
with_deleted: true or false; specifies whether to include logically deleted records in the results.
country_codes: Comma-delimited list of ISO 2-letter country codes for targeting and segmenttation.
reports: Object array of specified reports with name, entity, segment, and granularity.

{
    "start_date": "2019-01-01T00:00:00Z",
    "user_agent": "tap-twitter-ads <api_user_email@your_company.com>",
    "consumer_key": "YOUR_TWITTER_ADS_CONSUMER_KEY",
    "consumer_secret": "YOUR_TWITTER_ADS_CONSUMER_SECRET",
    "access_token": "YOUR_TWITTER_ADS_ACCESS_TOKEN",
    "access_token_secret": "YOUR_TWITTER_ADS_ACCESS_TOKEN_SECRET",
    "account_ids": "id1, id2, id3",
    "attribution_window": "14",
    "with_deleted": "true",
    "country_codes": "US, CA, MX, DE",
    "reports": [
        {
        "name": "campaigns_genders_hourly_report",
        "enitity": "CAMPAIGN",
        "segment": "GENDER",
        "granularity": "HOUR"
        },
        {
        "name": "line_items_regions_daily_report",
        "enitity": "LINE_ITEM",
        "segment": "REGIONS",
        "granularity": "DAY"
        }
    ]
}

Optionally, also create a state.json file. currently_syncing is an optional attribute used for identifying the last object to be synced in case the job is interrupted mid-stream. The next run would begin where the last job left off. Each bookmarked endpoint that supports INCREMENTAL syncs will be listed with its max last processed record based on updated_at, created_at, or end_time (depending on the endpoint).

{
    "currently_syncing": "creatives",
    "bookmarks": {
        "accounts": "2019-06-11T13:37:55Z",
        "account_media": "2019-06-19T19:48:42Z",
        "..."
    }
}

Run the Tap in Discovery Mode This creates a catalog.json for selecting objects/fields to integrate:
```
> tap-twitter-ads --config config.json --discover > catalog.json
```
See the Singer docs on discovery mode here.

Run the Tap in Sync Mode (with catalog) and write out to state file

For Sync mode:

> tap-twitter-ads --config tap_config.json --catalog catalog.json > state.json
> tail -1 state.json > state.json.tmp && mv state.json.tmp state.json

To load to json files to verify outputs:

> tap-twitter-ads --config tap_config.json --catalog catalog.json | target-json > state.json
> tail -1 state.json > state.json.tmp && mv state.json.tmp state.json

To pseudo-load to Stitch Import API with dry run:

> tap-twitter-ads --config tap_config.json --catalog catalog.json | target-stitch --config target_config.json --dry-run > state.json
> tail -1 state.json > state.json.tmp && mv state.json.tmp state.json

Test the Tap

While developing the twitter Ads tap, the following utilities were run in accordance with Singer.io best practices: Pylint to improve code quality:

> pylint tap_twitter_ads -d missing-docstring -d logging-format-interpolation -d too-many-locals -d too-many-arguments

Pylint test resulted in the following score:

Your code has been rated at 9.72/10.

To check the tap and verify working:

> tap-twitter-ads --config tap_config.json --catalog catalog.json | singer-check-tap > state.json
> tail -1 state.json > state.json.tmp && mv state.json.tmp state.json

Check tap resulted in the following:

  The output is valid.
  It contained 872667 messages for 62 streams.

      63 schema messages
  872432 record messages
      172 state messages

  Details by stream:
  +--------------------------------------------+---------+---------+
  | stream                                     | records | schemas |
  +--------------------------------------------+---------+---------+
  | account_media                              | 0       | 1       |
  | accounts                                   | 1       | 1       |
  | accounts_conversion_tags_hourly_report     | 0       | 1       |
  | accounts_daily_report                      | 87      | 1       |
  | accounts_metros_hourly_report              | 226240  | 1       |
  | advertiser_business_categories             | 16      | 1       |
  | bidding_rules                              | 64      | 1       |
  | campaigns                                  | 2       | 1       |
  | campaigns_conversion_tags_daily_report     | 0       | 1       |
  | campaigns_daily_report                     | 10      | 1       |
  | campaigns_genders_hourly_report            | 504     | 1       |
  | cards_image_app_download                   | 0       | 1       |
  | cards_image_conversation                   | 0       | 1       |
  | cards_image_direct_message                 | 0       | 1       |
  | cards_poll                                 | 0       | 1       |
  | cards_video_app_download                   | 0       | 1       |
  | cards_video_conversation                   | 0       | 1       |
  | cards_video_direct_message                 | 0       | 1       |
  | cards_video_website                        | 0       | 1       |
  | cards_website                              | 0       | 1       |
  | content_categories                         | 392     | 1       |
  | funding_instruments                        | 1       | 1       |
  | funding_instruments_age_daily_report       | 175     | 1       |
  | funding_instruments_devices_hourly_report  | 15288   | 1       |
  | funding_instruments_hourly_report          | 84      | 1       |
  | iab_categories                             | 15      | 1       |
  | line_item_apps                             | 0       | 1       |
  | line_items                                 | 2       | 1       |
  | line_items_conversion_tags_hourly_report   | 0       | 1       |
  | line_items_daily_report                    | 10      | 1       |
  | line_items_platform_versions_hourly_report | 3108    | 1       |
  | line_items_platforms_hourly_report         | 588     | 1       |
  | line_items_regions_daily_report            | 630     | 1       |
  | media_creatives                            | 0       | 1       |
  | media_creatives_daily_report               | 0       | 1       |
  | organic_tweets_daily_report                | 82      | 2       |
  | organic_tweets_hourly_report               | 1344    | 1       |
  | preroll_call_to_actions                    | 0       | 1       |
  | promotable_users                           | 0       | 1       |
  | promoted_accounts                          | 0       | 1       |
  | promoted_accounts_daily_report             | 0       | 1       |
  | promoted_tweets                            | 2       | 1       |
  | promoted_tweets_daily_report               | 10      | 1       |
  | promoted_tweets_interests_daily_report     | 3210    | 1       |
  | promoted_tweets_keywords_hourly_report     | 420     | 1       |
  | promoted_tweets_languages_daily_report     | 205     | 1       |
  | scheduled_promoted_tweets                  | 0       | 1       |
  | tailored_audiences                         | 0       | 1       |
  | targeting_app_store_categories             | 84      | 1       |
  | targeting_conversations                    | 41136   | 1       |
  | targeting_criteria                         | 10      | 1       |
  | targeting_devices                          | 355     | 1       |
  | targeting_events                           | 355     | 1       |
  | targeting_interests                        | 361     | 1       |
  | targeting_languages                        | 21      | 1       |
  | targeting_locations                        | 56682   | 1       |
  | targeting_network_operators                | 161     | 1       |
  | targeting_platform_versions                | 38      | 1       |
  | targeting_platforms                        | 4       | 1       |
  | targeting_tv_markets                       | 39      | 1       |
  | targeting_tv_shows                         | 520694  | 1       |
  | tweets                                     | 2       | 1       |
  +--------------------------------------------+---------+---------+

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

README.md

README.md

tap-twitter-ads

Authentication

Quick Start

Files

README.md

Latest commit

History

README.md

File metadata and controls

tap-twitter-ads

Authentication

Quick Start