-
Notifications
You must be signed in to change notification settings - Fork 1.5k
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
- Loading branch information
1 parent
5d33620
commit 50dfe4a
Showing
3 changed files
with
83 additions
and
1 deletion.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,4 +1,41 @@ | ||
--- | ||
title: Connecting to APIs | ||
sidebar_position: 20 | ||
--- | ||
--- | ||
|
||
This guide describes how to connect to and interact with APIs in dagster. | ||
|
||
|
||
## What you'll learn | ||
|
||
- How to write a dagster Resource to connect to an API | ||
- How to use that Resource in an asset | ||
|
||
<details> | ||
<summary>Prerequisites</summary> | ||
|
||
To follow the steps in this guide, you'll need: | ||
|
||
- Familiarity with [Asset definitions](/concepts/assets) | ||
- Familiarity with [Resources](/concepts/resources) | ||
- Install the `requests` library: `pip install requests` | ||
|
||
</details> | ||
|
||
## Step 1: Write a Resource to connect to an API | ||
|
||
Here is a minimal Resource for connecting to the an API that returns data about the sunrise and sunset times. In this example, the request URL has been hardcoded to query for data at San Francisco International Airport. | ||
|
||
<CodeExample filePath="guides/external-systems/apis/minimal_resource.py" language="python" title="Simple Resource to connect to Sun API" /> | ||
|
||
|
||
## Step 2: Use the Resource in an asset | ||
|
||
To use the Resource written in step 1, you can provide it to an asset like this: | ||
|
||
<CodeExample filePath="guides/external-systems/apis/use_resource_in_asset.py" language="python" title="Use the XYZResource in an asset" /> | ||
|
||
## Next steps | ||
|
||
- [Customize resources with configuration](/todo) | ||
- Learn what [dagster-provided Resources](/todo) are available to use |
16 changes: 16 additions & 0 deletions
16
...es/docs_beta_snippets/docs_beta_snippets/guides/external-systems/apis/minimal_resource.py
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,16 @@ | ||
import requests | ||
|
||
import dagster as dg | ||
|
||
|
||
class SFOSunResource(dg.ConfigurableResource): | ||
@property | ||
def query_string(self) -> str: | ||
latittude = 37.615223 | ||
longitude = -122.389977 | ||
tzone = "America/Los_Angeles" | ||
return f"https://api.sunrise-sunset.org/json?lat={latittude}&lng={longitude}&date=today&tzid={tzone}" | ||
|
||
def sunrise(self) -> str: | ||
data = requests.get(self.query_string, timeout=5).json() | ||
return data["results"]["sunrise"] |
29 changes: 29 additions & 0 deletions
29
...cs_beta_snippets/docs_beta_snippets/guides/external-systems/apis/use_resource_in_asset.py
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,29 @@ | ||
import requests | ||
|
||
import dagster as dg | ||
|
||
|
||
class SFOSunResource(dg.ConfigurableResource): | ||
@property | ||
def query_string(self) -> str: | ||
latittude = 37.615223 | ||
longitude = -122.389977 | ||
tzone = "America/Los_Angeles" | ||
return f"https://api.sunrise-sunset.org/json?lat={latittude}&lng={longitude}&date=today&tzid={tzone}" | ||
|
||
def sunrise(self) -> str: | ||
data = requests.get(self.query_string, timeout=5).json() | ||
return data["results"]["sunrise"] | ||
|
||
|
||
@dg.asset | ||
def sfo_sunrise( | ||
context: dg.AssetExecutionContext, sun_resource: SFOSunResource | ||
) -> None: | ||
sunrise = sun_resource.sunrise() | ||
context.log.info(f"Sunrise in San Francisco is at {sunrise}.") | ||
|
||
|
||
defs = dg.Definitions( | ||
assets=[sfo_sunrise], resources={"sun_resource": SFOSunResource()} | ||
) |