Skip to content

Commit

Permalink
add configurable resource
Browse files Browse the repository at this point in the history
  • Loading branch information
jamiedemaria committed Aug 26, 2024
1 parent 50dfe4a commit 97faa69
Show file tree
Hide file tree
Showing 5 changed files with 76 additions and 8 deletions.
20 changes: 16 additions & 4 deletions docs/docs-beta/docs/guides/external-systems/apis.md
Original file line number Diff line number Diff line change
Expand Up @@ -26,16 +26,28 @@ To follow the steps in this guide, you'll need:

Here is a minimal Resource for connecting to the an API that returns data about the sunrise and sunset times. In this example, the request URL has been hardcoded to query for data at San Francisco International Airport.

Check failure on line 27 in docs/docs-beta/docs/guides/external-systems/apis.md

View workflow job for this annotation

GitHub Actions / runner / vale

[vale] reported by reviewdog 🐶 [Vale.Spelling] Did you really mean 'hardcoded'? Raw Output: {"message": "[Vale.Spelling] Did you really mean 'hardcoded'?", "location": {"path": "docs/docs-beta/docs/guides/external-systems/apis.md", "range": {"start": {"line": 27, "column": 153}}}, "severity": "ERROR"}

Check failure on line 27 in docs/docs-beta/docs/guides/external-systems/apis.md

View workflow job for this annotation

GitHub Actions / runner / vale

[vale] reported by reviewdog 🐶 [Dagster.spelling] Is 'hardcoded' spelled correctly? Raw Output: {"message": "[Dagster.spelling] Is 'hardcoded' spelled correctly?", "location": {"path": "docs/docs-beta/docs/guides/external-systems/apis.md", "range": {"start": {"line": 27, "column": 153}}}, "severity": "ERROR"}

<CodeExample filePath="guides/external-systems/apis/minimal_resource.py" language="python" title="Simple Resource to connect to Sun API" />
<CodeExample filePath="guides/external-systems/apis/minimal_resource.py" language="python" title="Resource to connect to Sun API" />


## Step 2: Use the Resource in an asset

Check failure on line 32 in docs/docs-beta/docs/guides/external-systems/apis.md

View workflow job for this annotation

GitHub Actions / runner / vale

[vale] reported by reviewdog 🐶 [Dagster.headings-casing] 'Step 2: Use the Resource in an asset' should be in sentence case Raw Output: {"message": "[Dagster.headings-casing] 'Step 2: Use the Resource in an asset' should be in sentence case", "location": {"path": "docs/docs-beta/docs/guides/external-systems/apis.md", "range": {"start": {"line": 32, "column": 4}}}, "severity": "ERROR"}

To use the Resource written in step 1, you can provide it to an asset like this:
To use the Resource written in Step 1, you can provide it to an asset like this:

<CodeExample filePath="guides/external-systems/apis/use_resource_in_asset.py" language="python" title="Use the XYZResource in an asset" />
<CodeExample filePath="guides/external-systems/apis/use_minimal_resource_in_asset.py" language="python" title="Use the SFOSunResource in an asset" />


## Step 3: Configure your Resource
Many APIs have configuration you can set to customize your usage. Here is an updated version of the Resource from Step 1 with configuration to allow for setting the query location:

<CodeExample filePath="guides/external-systems/apis/configurable_resource.py" language="python" title="Configurable Resource to connect to Sun API" />

## Step 4: Use the configurable Resource in an asset

Check failure on line 44 in docs/docs-beta/docs/guides/external-systems/apis.md

View workflow job for this annotation

GitHub Actions / runner / vale

[vale] reported by reviewdog 🐶 [Dagster.headings-casing] 'Step 4: Use the configurable Resource in an asset' should be in sentence case Raw Output: {"message": "[Dagster.headings-casing] 'Step 4: Use the configurable Resource in an asset' should be in sentence case", "location": {"path": "docs/docs-beta/docs/guides/external-systems/apis.md", "range": {"start": {"line": 44, "column": 4}}}, "severity": "ERROR"}

The configurable Resource written in Step 3 can be provided to an asset exactly as before. When the Resource is initialized, you can pass values for each of the configuration options.

<CodeExample filePath="guides/external-systems/apis/use_configurable_resource_in_asset.py" language="python" title="Use the configurable SunResource in an asset" />

## Next steps

- [Customize resources with configuration](/todo)
- [Authenticate to a resource](/guides/external-systems/authentication.md)
- Learn what [dagster-provided Resources](/todo) are available to use
Original file line number Diff line number Diff line change
@@ -0,0 +1,17 @@
import requests

import dagster as dg


class SunResource(dg.ConfigurableResource):
latitude: float
longitude: float
time_zone: str

@property
def query_string(self) -> str:
return f"https://api.sunrise-sunset.org/json?lat={self.latittude}&lng={self.longitude}&date=today&tzid={self.time_zone}"

def sunrise(self) -> str:
data = requests.get(self.query_string, timeout=5).json()
return data["results"]["sunrise"]
Original file line number Diff line number Diff line change
Expand Up @@ -8,8 +8,8 @@ class SFOSunResource(dg.ConfigurableResource):
def query_string(self) -> str:
latittude = 37.615223
longitude = -122.389977
tzone = "America/Los_Angeles"
return f"https://api.sunrise-sunset.org/json?lat={latittude}&lng={longitude}&date=today&tzid={tzone}"
time_zone = "America/Los_Angeles"
return f"https://api.sunrise-sunset.org/json?lat={latittude}&lng={longitude}&date=today&tzid={time_zone}"

def sunrise(self) -> str:
data = requests.get(self.query_string, timeout=5).json()
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,36 @@
import requests

import dagster as dg


class SunResource(dg.ConfigurableResource):
latitude: float
longitude: float
time_zone: str

@property
def query_string(self) -> str:
return f"https://api.sunrise-sunset.org/json?lat={self.latitude}&lng={self.longitude}&date=today&tzid={self.time_zone}"

def sunrise(self) -> str:
data = requests.get(self.query_string, timeout=5).json()
return data["results"]["sunrise"]


# highlight-start
@dg.asset
def sfo_sunrise(context: dg.AssetExecutionContext, sun_resource: SunResource) -> None:
sunrise = sun_resource.sunrise()
context.log.info(f"Sunrise in San Francisco is at {sunrise}.")


defs = dg.Definitions(
assets=[sfo_sunrise],
resources={
"sun_resource": SunResource(
latitude=37.615223, longitude=-122.389977, time_zone="America/Los_Angeles"
)
},
)

# highlight-end
Original file line number Diff line number Diff line change
Expand Up @@ -8,14 +8,15 @@ class SFOSunResource(dg.ConfigurableResource):
def query_string(self) -> str:
latittude = 37.615223
longitude = -122.389977
tzone = "America/Los_Angeles"
return f"https://api.sunrise-sunset.org/json?lat={latittude}&lng={longitude}&date=today&tzid={tzone}"
time_zone = "America/Los_Angeles"
return f"https://api.sunrise-sunset.org/json?lat={latittude}&lng={longitude}&date=today&tzid={time_zone}"

def sunrise(self) -> str:
data = requests.get(self.query_string, timeout=5).json()
return data["results"]["sunrise"]


# highlight-start
@dg.asset
def sfo_sunrise(
context: dg.AssetExecutionContext, sun_resource: SFOSunResource
Expand All @@ -27,3 +28,5 @@ def sfo_sunrise(
defs = dg.Definitions(
assets=[sfo_sunrise], resources={"sun_resource": SFOSunResource()}
)

# highlight-end

0 comments on commit 97faa69

Please sign in to comment.