Skip to content

Commit

Permalink
env vars
Browse files Browse the repository at this point in the history
  • Loading branch information
jamiedemaria committed Aug 26, 2024
1 parent cb0095e commit 6d4c695
Show file tree
Hide file tree
Showing 2 changed files with 48 additions and 1 deletion.
11 changes: 10 additions & 1 deletion docs/docs-beta/docs/guides/external-systems/apis.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,13 +3,15 @@ title: Connecting to APIs
sidebar_position: 20
---

This guide describes how to connect to and interact with APIs in dagster.
This guide describes how to connect to and interact with APIs in dagster. In this guide you will use dagster Resources to connect to an external API. Using a Resource allows you to standardize how you connect to an external API across your project, use configuration to customize your connections, and use difference implementations of a Resource for different environments.


## What you'll learn

- How to write a dagster Resource to connect to an API
- How to use that Resource in an asset
- How to configure a Resource
- How to source configuration values from environment variables

<details>
<summary>Prerequisites</summary>
Expand Down Expand Up @@ -47,6 +49,13 @@ The configurable Resource written in Step 3 can be provided to an asset exactly

<CodeExample filePath="guides/external-systems/apis/use_configurable_resource_in_asset.py" language="python" title="Use the configurable SunResource in an asset" />

## Step 5: Sourcing configuration values from environment variables

You can configure your Resource using values that are stored in environment variables using the `EnvVar` class. In this example, there is a new `home_sunrise` asset. Rather than hardcoding the location of your home, you can set it in environment variables, and configure the `SunResource` by reading those values:

Check failure on line 54 in docs/docs-beta/docs/guides/external-systems/apis.md

View workflow job for this annotation

GitHub Actions / runner / vale

[vale] reported by reviewdog 🐶 [Dagster.spelling] Is 'hardcoding' spelled correctly? Raw Output: {"message": "[Dagster.spelling] Is 'hardcoding' spelled correctly?", "location": {"path": "docs/docs-beta/docs/guides/external-systems/apis.md", "range": {"start": {"line": 54, "column": 179}}}, "severity": "ERROR"}

Check failure on line 54 in docs/docs-beta/docs/guides/external-systems/apis.md

View workflow job for this annotation

GitHub Actions / runner / vale

[vale] reported by reviewdog 🐶 [Vale.Spelling] Did you really mean 'hardcoding'? Raw Output: {"message": "[Vale.Spelling] Did you really mean 'hardcoding'?", "location": {"path": "docs/docs-beta/docs/guides/external-systems/apis.md", "range": {"start": {"line": 54, "column": 179}}}, "severity": "ERROR"}

<CodeExample filePath="guides/external-systems/apis/env_var_configuration.py" language="python" title="Configure the Resource with values from environment variables" />


## Next steps

- [Authenticate to a resource](/guides/external-systems/authentication.md)
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,38 @@
import requests

import dagster as dg


class SunResource(dg.ConfigurableResource):
latitude: float
longitude: float
time_zone: str

@property
def query_string(self) -> str:
return f"https://api.sunrise-sunset.org/json?lat={self.latitude}&lng={self.longitude}&date=today&tzid={self.time_zone}"

def sunrise(self) -> str:
data = requests.get(self.query_string, timeout=5).json()
return data["results"]["sunrise"]


# highlight-start
@dg.asset
def home_sunrise(context: dg.AssetExecutionContext, sun_resource: SunResource) -> None:
sunrise = sun_resource.sunrise()
context.log.info(f"Sunrise at home is at {sunrise}.")


defs = dg.Definitions(
assets=[home_sunrise],
resources={
"sun_resource": SunResource(
latitude=float(dg.EnvVar("HOME_LATITUDE")),
longitude=float(dg.EnvVar("HOME_LONGITUDE")),
time_zone=dg.EnvVar("HOME_TIMEZONE"),
)
},
)

# highlight-end

0 comments on commit 6d4c695

Please sign in to comment.