Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Feat: push up helpers #2

Open
wants to merge 1 commit into
base: new-parser-experiment
Choose a base branch
from

Conversation

z3z1ma
Copy link
Collaborator

@z3z1ma z3z1ma commented Aug 9, 2023

This is just a donation of some helper classes. Feel free to hack away.
I can also share how I have personally used them to simplify quite a few APIs.

@alex-harness
Copy link

alex-harness commented Aug 9, 2023

Example complete pipeline

import dlt

from continuous_data_flow.sdk.authenticators import BearerTokenAuthenticator
from continuous_data_flow.sdk.paginators import JSONPathPaginator
from continuous_data_flow.sdk.rest import RestAPIEndpoint, RestAPIEndpointFactory


class QualifiedEndpoint(RestAPIEndpoint[str]):
    def get_request_params(self, next_page_token):
        params = {"limit": 2000}
        if next_page_token:
            params["starting_after"] = next_page_token
        return params


class QualifiedAPI(
    RestAPIEndpointFactory[QualifiedEndpoint, JSONPathPaginator, BearerTokenAuthenticator]
):
    base_url = "https://api.qualified.com/v1/"
    endpoint_klass = QualifiedEndpoint
    default_paginator = JSONPathPaginator(jsonpath="$.meta.end_cursor")
    records_jsonpath = "$.data[*]"


@dlt.source
def qualified(api_token: str = dlt.secrets.value):
    """Qualified source for DLT"""
    api = QualifiedAPI(authenticator=BearerTokenAuthenticator(token=api_token))
    return (
        dlt.resource(
            iter(api.endpoint_factory(endpoint)),
            name=endpoint,
            write_disposition="replace",
            primary_key="id",
        )
        for endpoint in ("rep_conversations", "bot_conversations")
    )

@rudolfix
Copy link
Collaborator

rudolfix commented Aug 9, 2023

heh this is so good :) definitely should go into core library @z3z1ma thx! @steinitzu pls take a look

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants