Guide for connecting to APIs #23920
Guide for connecting to APIs #23920
Conversation
This stack of pull requests is managed by Graphite. Learn more about stacking. Join @jamiedemaria and the rest of your teammates on |
jamiedemaria
requested review from
PedramNavid,
cmpadden,
benpankow and
erinkcochran87
|
Deploy preview for dagster-docs-beta ready! ✅ Preview Built with commit 1e94b9d. |
|
Deploy preview for dagster-docs ready! Preview available at https://dagster-docs-1rb85cqqe-elementl.vercel.app Direct link to changed pages: |
jamiedemaria
force-pushed
the
jamie/doc-329-write-using-api-connections
branch
from
b25a4f2 to
74d85dc
Compare
jamiedemaria
changed the title
jamiedemaria
force-pushed
the
jamie/doc-329-write-using-api-connections
branch
from
74d85dc to
26354b8
Compare
jamiedemaria
force-pushed
the
jamie/doc-329-write-using-api-connections
branch
from
26354b8 to
9593f31
Compare
|
@benpankow can i get a review for the technical content? it should be quick since it's a relatively straightforward guide |
jamiedemaria
force-pushed
the
jamie/doc-329-write-using-api-connections
branch
from
fe13e09 to
1e94b9d
Compare
| --- | ||
|
|
||
| When building a data pipeline, you'll likely need to connect to several external APIs, each with its own specific configuration and behavior. This guide demonstrates how to standardize your API connections and customize their configuration using Dagster resources. |
There was a problem hiding this comment.
@jamiedemaria This is great.
@erinkcochran87 @PedramNavid curious what you think in terms of in scope for a "how-to" guide. I just want it to be clear the resources are opt-in for this use case.
I think we should include (abridged) language about when it is good to use resources. E.g:
Accessing an API through a Dagster resource is useful if you want to:
- Parameterize how to access the service at runtime (either through API or UI) via config.
- Surface that configuration in Dagster.
- Centralize configuration and implementation of this API access.
- Plug in different implementations of resources in different environments (local dev versus production, for example).
If you don't want any of these features, you should just invoke the external service directly.
Summary & Motivation
Guide for connecting to APIs using resources. I tried to overcorrect toward terseness, but very likely that this doesn't have enough explanation. Let me know and I can add more.
How I Tested These Changes
Changelog [New | Bug | Docs]
NOCHANGELOG