Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Are there detailed tutorials or demos #27

Open
userxxj opened this issue Oct 28, 2022 · 4 comments
Open

Are there detailed tutorials or demos #27

userxxj opened this issue Oct 28, 2022 · 4 comments

Comments

@userxxj
Copy link

userxxj commented Oct 28, 2022

No description provided.

@jlloyd-widen
Copy link
Contributor

If there are any users of this tap that do have publicly available examples, please post them here.

Here is an "obfuscated" example of a meltano.yml file. I'm using it to hit an API internal to my company and another that is an API to datadog for which I couldn't find an acceptable tap and didn't want to develop one of my own (not an efficient solution). FYI, This doesn't even get close to exposing the full functionality of the tap; these examples are very simple.

  - name: tap-rest-api
    namespace: tap_rest_api_msdk
    pip_url: tap-rest-api-msdk
    executable: tap-rest-api-msdk
    capabilities:
    - state
    - catalog
    - discover
    settings:
    - name: api_url
    - name: next_page_token_path
    - name: pagination_request_style
    - name: pagination_response_style
    - name: pagination_page_size
    - name: streams
    - name: path
    - name: params
    - name: headers
    - name: records_path
    - name: primary_keys
    - name: replication_key
    - name: except_keys
    - name: num_inference_records
  - name: tap-***
    inherit_from: tap-rest-api
    config:
      api_url: https://***
      streams:
      - name: ***
        path: /
        primary_keys:
        - id
        replication_key: last_data_refresh
        except_keys:
        - cnames
        - cdn_embeds
        - cdn_hosts
        - collective_message_keys
        - collective_properties
        - brandings
        - nginx
        - properties
        - skins
        records_path: $.accounts[*]
        num_inference_records: 200
    select:
    - '*.*'
  - name: tap-datadog-wdt
    inherit_from: tap-rest-api
    config:
      api_url: https://api.datadoghq.com/api/v1
      streams:
      - name: ApiSummary
        path: /query
        headers:
          Content-Type: application/json
          DD-API-KEY: TAP_DD_API_KEY
          DD-APPLICATION-KEY: TAP_DD_APPLICATION_KEY
        params:
          query: sum:widen.api.count{env:prod}by{source,account_name,user_oauth_app,trace_span_operation,http_request_method,http_response_status}.rollup(86400)
          from: TAP_DD_FROM
          to: TAP_DD_TO
        primary_keys:
        - scope
        - start
        - end
        records_path: $.series[*]
    select:
    - '*.*'

@EJOOSTEROP
Copy link

EJOOSTEROP commented Nov 5, 2022

[edit]Never mind this - I now noticed you have the same/similar meltano.yml in your project already...

For the earthquakes example.

With:

  • meltano add extractor tap-rest-api-msdk
  • meltano add loader target-jsonl

meltano.yml as follows:

version: 1
default_environment: dev
project_id: 1f67e0b0-285b-4961-b71d-b825bce3dcc7
environments:
- name: dev
- name: staging
- name: prod
plugins:
  extractors:
  - name: tap-rest-api-msdk
    variant: widen
    pip_url: tap-rest-api-msdk
    capabilities:
    - state
    - catalog
    - discover
    settings:
    - name: api_url
    - name: next_page_token_path
    - name: pagination_request_style
    - name: pagination_response_style
    - name: pagination_page_size
    - name: streams
    - name: path
    - name: params
    - name: headers
    - name: records_path
    - name: primary_keys
    - name: replication_key
    - name: except_keys
    - name: num_inference_records
    config:
      api_url: https://earthquake.usgs.gov/fdsnws
      streams:
      - name: us_earthquakes
        params:
          format: geojson
          starttime: '2022-11-01'
          endtime: '2014-01-02'
          minmagnitude: 1
        path: /event/1/query
        primary_keys:
        - id
        records_path: $.features[*]
        num_inference_records: 200
    select:
    - '*.*'
  loaders:
  - name: target-jsonl
    variant: andyh1203
    pip_url: target-jsonl
schedules:
- name: rest-api-msdk-to-jsonl
  interval: '@hourly'
  extractor: tap-rest-api-msdk
  loader: target-jsonl
  transform: skip
  start_date: 2022-11-05 02:41:58.166575

@s7clarke10
Copy link
Contributor

Look at the documentation regarding this PR: #36. It may resolve you problem.

@vincentclaes
Copy link

vincentclaes commented Nov 9, 2023

version: 1
default_environment: dev
project_id: d63825fc-4b14-4051-a9f4-d96776f9fd39
environments:
- name: dev
- name: staging
- name: prod
plugins:
  extractors:
  - name: tap-rest-api-msdk
    variant: widen
    pip_url: tap-rest-api-msdk
    config:
      api_url: https://foo.elabjournal.com/api/v1
      streams:
      - name: foo
        path: /experiments
        headers:
          Authorization: <some api key>
        primary_keys:
        - experimentID
        records_path: $.data[*]

I want all the records inside the key "data"

{
  "recordCount": 1000,
  "currentPage": 0,
  "maxRecords": 1000,
  "totalRecords": 2036,
  "data": [
    {
      "usesSignatureWorkflow": false,
      "studyID": 45453,
      "projectID": 343434,,
      "experimentStatus": {
        "experimentStatusID": 23232,
        "groupID": 6445,
        "color": "#AA00AA",
        "status": "Completed",
        "experimentStatusType": "COMPLETED"
      },
    ...
]
}

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

5 participants