Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

rest_api: Extend generic API source to allow for incremental path parameters #1880

Open
maxestorr opened this issue Aug 21, 2024 · 1 comment · May be fixed by #2210
Open

rest_api: Extend generic API source to allow for incremental path parameters #1880

maxestorr opened this issue Aug 21, 2024 · 1 comment · May be fixed by #2210
Assignees
Labels
community This issue came from slack community workspace question Further information is requested

Comments

@maxestorr
Copy link

maxestorr commented Aug 21, 2024

Source name

rest_api

Describe the data you'd like to see

I am using the generic API source to write my data pipeline declaratively, ingesting data from the ebird historical observations endpoint.

As you can see in its documentation linked above it's possible to incrementally load data from this endpoint, but not using traditional query parameters such as ?data_from=2024-08-01 but rather using path parameters, where each day has it's own endpoint path such as https://servername.com/2/data/obs/{{region-code}}/{{year}}/{{month}}/{{day}}.

Currently the generic API source allows you to query data incrementally using query parameters, using a config defined as so:

{
    "path": "posts",
    "data_selector": "results",  # Optional JSONPath to select the list of posts
    "params": {
        "created_since": {
            "type": "incremental",
            "cursor_path": "created_at", # The JSONPath to the field we want to track in each post
            "initial_value": "2024-01-25",
        },
    },
}

But I believe there's no such config that'd work for path parameters.

Are you a dlt user?

Yes, I'm already a dlt user.

Do you ready to contribute this extension?

No.

dlt destination

duck db

Additional information

No response

@maxestorr
Copy link
Author

I'm about to head out but can share the code I'm working with on my return, my wider project's context is I'm trying to ingest data from this API using dlt's generic API declarative config, as well as Airflow for orchestration, and ran into a number of issues (this being one of them) which has prevented me from achieving this.

@VioletM VioletM added the community This issue came from slack community workspace label Aug 22, 2024
@burnash burnash transferred this issue from dlt-hub/verified-sources Sep 26, 2024
@rudolfix rudolfix added the question Further information is requested label Oct 7, 2024
@burnash burnash changed the title Extend generic API source to allow for incremental path parameters rest_api: Extend generic API source to allow for incremental path parameters Oct 7, 2024
@burnash burnash linked a pull request Jan 26, 2025 that will close this issue
@burnash burnash moved this from Planned to In Progress in dlt core library Jan 26, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
community This issue came from slack community workspace question Further information is requested
Projects
Status: Planned
Status: In Progress
Development

Successfully merging a pull request may close this issue.

4 participants