-
Notifications
You must be signed in to change notification settings - Fork 1.5k
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Guide for connecting to APIs (#23920)
## Summary & Motivation Guide for connecting to APIs using resources. I tried to overcorrect toward terseness, but very likely that this doesn't have enough explanation. Let me know and I can add more. ## How I Tested These Changes ## Changelog [New | Bug | Docs] `NOCHANGELOG`
- Loading branch information
1 parent
974064f
commit bd31eed
Showing
5 changed files
with
200 additions
and
1 deletion.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,4 +1,80 @@ | ||
--- | ||
title: Connecting to APIs | ||
sidebar_position: 20 | ||
--- | ||
--- | ||
|
||
When building a data pipeline, you'll likely need to connect to several external APIs, each with its own specific configuration and behavior. This guide demonstrates how to standardize your API connections and customize their configuration using Dagster resources. | ||
|
||
|
||
## What you'll learn | ||
|
||
- How to connect to an API using a Dagster resource | ||
- How to use that resource in an asset | ||
- How to configure a resource | ||
- How to source configuration values from environment variables | ||
|
||
<details> | ||
<summary>Prerequisites</summary> | ||
|
||
To follow the steps in this guide, you'll need: | ||
|
||
- Familiarity with [Asset definitions](/concepts/assets) | ||
- Familiarity with [resources](/concepts/resources) | ||
- Install the `requests` library: | ||
```bash | ||
pip install requests | ||
``` | ||
|
||
</details> | ||
|
||
## Step 1: Write a resource to connect to an API | ||
|
||
This example fetches the sunrise time for a given location from a REST API. | ||
|
||
Begin by defining a Dagster resource with a method to return the sunrise time for a location. In the first version of this resource, the location will be hard-coded to San Francisco International Airport. | ||
|
||
|
||
<CodeExample filePath="guides/external-systems/apis/minimal_resource.py" language="python" title="Resource to connect to the Sunrise API" /> | ||
|
||
|
||
## Step 2: Use the resource in an asset | ||
|
||
To use the resource written in Step 1, you can provide it as a parameter to an asset after including it in the Definitions object: | ||
|
||
<CodeExample filePath="guides/external-systems/apis/use_minimal_resource_in_asset.py" language="python" title="Use the SunResource in an asset" /> | ||
|
||
When you materialize `sfo_sunrise`, Dagster will provide an initialized `SunResource` to the `sun_resource` parameter. | ||
|
||
|
||
## Step 3: Configure your resource | ||
Many APIs have configuration you can set to customize your usage. Here is an updated version of the resource from Step 1 with configuration to allow for setting the query location: | ||
|
||
<CodeExample filePath="guides/external-systems/apis/use_configurable_resource_in_asset.py" language="python" title="Use the configurable SunResource in an asset" /> | ||
|
||
The configurable resource can be provided to an asset exactly as before. When the resource is initialized, you can pass values for each of the configuration options. | ||
|
||
When you materialize `sfo_sunrise`, Dagster will provide a `SunResource` initialized with the configuration values to the `sun_resource` parameter. | ||
|
||
|
||
## Step 4: Source configuration values from environment variables | ||
Resources can also be configured with environment variables. You can use Dagster's built-in `EnvVar` class to source configuration values from environment variables at materialization time. | ||
In this example, there is a new `home_sunrise` asset. Rather than hard-coding the location of your home, you can set it in environment variables, and configure the `SunResource` by reading those values: | ||
<CodeExample filePath="guides/external-systems/apis/env_var_configuration.py" language="python" title="Configure the resource with values from environment variables" /> | ||
When you materialize `home_sunrise`, Dagster will read the values set for the `HOME_LATITUDE`, `HOME_LONGITUDE`, and `HOME_TIMZONE` environment variables and initialize a `SunResource` with those values. | ||
The initialized `SunResource` will be provided to the `sun_resource` parameter. | ||
:::note | ||
You can also fetch environment variables using the `os` library. Dagster treats each approach to fetching environment variables differently, such as when they're fetched or how they display in the UI. Refer to the [Environment variables guide](/todo) for more information. | ||
::: | ||
|
||
|
||
## Next steps | ||
|
||
- [Authenticate to a resource](/guides/external-systems/authentication.md) | ||
- [Use different resources in different execution environments](/todo) | ||
- [Set environment variables in Dagster+](/todo) | ||
- Learn what [Dagster-provided resources](/todo) are available to use |
38 changes: 38 additions & 0 deletions
38
...cs_beta_snippets/docs_beta_snippets/guides/external-systems/apis/env_var_configuration.py
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,38 @@ | ||
import requests | ||
|
||
import dagster as dg | ||
|
||
|
||
class SunResource(dg.ConfigurableResource): | ||
latitude: str | ||
longitude: str | ||
time_zone: str | ||
|
||
@property | ||
def query_string(self) -> str: | ||
return f"https://api.sunrise-sunset.org/json?lat={self.latitude}&lng={self.longitude}&date=today&tzid={self.time_zone}" | ||
|
||
def sunrise(self) -> str: | ||
data = requests.get(self.query_string, timeout=5).json() | ||
return data["results"]["sunrise"] | ||
|
||
|
||
# highlight-start | ||
@dg.asset | ||
def home_sunrise(context: dg.AssetExecutionContext, sun_resource: SunResource) -> None: | ||
sunrise = sun_resource.sunrise() | ||
context.log.info(f"Sunrise at home is at {sunrise}.") | ||
|
||
|
||
defs = dg.Definitions( | ||
assets=[home_sunrise], | ||
resources={ | ||
"sun_resource": SunResource( | ||
latitude=dg.EnvVar("HOME_LATITUDE"), | ||
longitude=dg.EnvVar("HOME_LONGITUDE"), | ||
time_zone=dg.EnvVar("HOME_TIMEZONE"), | ||
) | ||
}, | ||
) | ||
|
||
# highlight-end |
16 changes: 16 additions & 0 deletions
16
...es/docs_beta_snippets/docs_beta_snippets/guides/external-systems/apis/minimal_resource.py
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,16 @@ | ||
import requests | ||
|
||
import dagster as dg | ||
|
||
|
||
class SunResource(dg.ConfigurableResource): | ||
@property | ||
def query_string(self) -> str: | ||
latittude = "37.615223" | ||
longitude = "-122.389977" | ||
time_zone = "America/Los_Angeles" | ||
return f"https://api.sunrise-sunset.org/json?lat={latittude}&lng={longitude}&date=today&tzid={time_zone}" | ||
|
||
def sunrise(self) -> str: | ||
data = requests.get(self.query_string, timeout=5).json() | ||
return data["results"]["sunrise"] |
41 changes: 41 additions & 0 deletions
41
...ets/docs_beta_snippets/guides/external-systems/apis/use_configurable_resource_in_asset.py
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,41 @@ | ||
import requests | ||
|
||
import dagster as dg | ||
|
||
|
||
class SunResource(dg.ConfigurableResource): | ||
# highlight-start | ||
latitude: str | ||
longitude: str | ||
time_zone: str | ||
|
||
@property | ||
def query_string(self) -> str: | ||
return f"https://api.sunrise-sunset.org/json?lat={self.latittude}&lng={self.longitude}&date=today&tzid={self.time_zone}" | ||
|
||
# highlight-end | ||
|
||
def sunrise(self) -> str: | ||
data = requests.get(self.query_string, timeout=5).json() | ||
return data["results"]["sunrise"] | ||
|
||
|
||
@dg.asset | ||
def sfo_sunrise(context: dg.AssetExecutionContext, sun_resource: SunResource) -> None: | ||
sunrise = sun_resource.sunrise() | ||
context.log.info(f"Sunrise in San Francisco is at {sunrise}.") | ||
|
||
|
||
# highlight-start | ||
defs = dg.Definitions( | ||
assets=[sfo_sunrise], | ||
resources={ | ||
"sun_resource": SunResource( | ||
latitude="37.615223", | ||
longitude="-122.389977", | ||
time_zone="America/Los_Angeles", | ||
) | ||
}, | ||
) | ||
|
||
# highlight-end |
28 changes: 28 additions & 0 deletions
28
...snippets/docs_beta_snippets/guides/external-systems/apis/use_minimal_resource_in_asset.py
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,28 @@ | ||
import requests | ||
|
||
import dagster as dg | ||
|
||
|
||
class SunResource(dg.ConfigurableResource): | ||
@property | ||
def query_string(self) -> str: | ||
latittude = "37.615223" | ||
longitude = "-122.389977" | ||
time_zone = "America/Los_Angeles" | ||
return f"https://api.sunrise-sunset.org/json?lat={latittude}&lng={longitude}&date=today&tzid={time_zone}" | ||
|
||
def sunrise(self) -> str: | ||
data = requests.get(self.query_string, timeout=5).json() | ||
return data["results"]["sunrise"] | ||
|
||
|
||
# highlight-start | ||
@dg.asset | ||
def sfo_sunrise(context: dg.AssetExecutionContext, sun_resource: SunResource) -> None: | ||
sunrise = sun_resource.sunrise() | ||
context.log.info(f"Sunrise in San Francisco is at {sunrise}.") | ||
|
||
|
||
defs = dg.Definitions(assets=[sfo_sunrise], resources={"sun_resource": SunResource()}) | ||
|
||
# highlight-end |
bd31eed
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Deploy preview for dagster-docs-beta ready!
✅ Preview
https://dagster-docs-beta-9ndhppuro-elementl.vercel.app
https://dagster-docs-beta.dagster-docs.io
Built with commit bd31eed.
This pull request is being automatically deployed with vercel-action