rest_api: Enable resolve in query parameters

[jlopezpena]

Feature description

For REST APIs, it should be possible to dynamically populate query parameters based on values obtained from an existing resource.

This can be currently done only if the resolved parameter is part of the path, but not if it is used inside the query.

Eg, this works in a source resource definition:

{
    "name": "seasons",
    "endpoint": {
        "path": "competitions/{competition_id}/seasons",
        "params": {
            "competition_id": {
                "type": "resolve",
                "resource": "competitions",
                "field": "id",
            }
        }
    },
    "include_from_parent": ["id"],
}

but this doesn't:

{
    "name": "seasons",
    "endpoint": {
        "path": "competitions/seasons/?competition_id={competition_id}",
        "params": {
            "competition_id": {
                "type": "resolve",
                "resource": "competitions",
                "field": "id",
            }
        }
    },
    "include_from_parent": ["id"],
}

This forces creating a resource or transformer outside of the declarative API, using @dlt.transformer or @dlt.resource decorators, and makes for overall messy code

Are you a dlt user?

I am scoping it for our company's use case

Use case

It is a very common pattern to find REST APIs where the id's of documents coming from a query are used in subsequent queries. Sometimes the API defines this as a path parameter, and sometimes as a query parameter. This is just an implementation detail that should be abstracted away in dlt.

Proposed solution

The resolve mechanism should also work for query parameters. An ideal solution would work either with explicity inclusion of the query parameter in the path (as in the example above) or (even better!) in a declarative format, with something like this:

{
    "name": "seasons",
    "endpoint": {
        "path": "competitions/seasons/",
        "params": {
            "competition_id": {
                "type": "resolve",
                "resource": "competitions",
                "field": "id",
                "location": "query"  # <- This field would ensure the resolved parameter gets injected in the query
            }
        }
    },
    "include_from_parent": ["id"],
}

Related issues

None found

[joscha]

Use case

We have a similar use-case:

The Affinity CRM has three main entities: Persons, Companies and Opportunities.
The CRM allows to attach notes to each of these entities. They are always connected to at least one to many of each type of these entities, but never to different types. The way the API models this is described here:

Basically it's three different query parameters: person_id, organization_id and opportunity_id. You can pass none, then you get all notes or you can pass exactly one of the query parameters, then you get the notes related to that entity.
Now I want to fetch all persons and augment each person with their notes, same for organizations and opportunities.
For that I need to resolve a query parameter to another entity in my resources.

This forces creating a resource or transformer outside of the declarative API, using @dlt.transformer or @dlt.resource decorators, and makes for overall messy code

@jlopezpena do you have an example for this by any chance?

[francescomucio]

Another +1 for this

[francescomucio]

I didn't see the last mention and I went ahead implementing a PR for the query parameters and JSON