Skip to content

Commit

Permalink
[dagster-sigma] Update Sigma documentation to showcase new API patter…
Browse files Browse the repository at this point in the history
…ns (#25598)

## Summary

Updates our Sigma docs to incorporate the new specs-building pattern and
norm of passing the resource to `Definitions`.
  • Loading branch information
maximearmstrong authored Oct 29, 2024
1 parent b82590b commit 42f5163
Show file tree
Hide file tree
Showing 5 changed files with 47 additions and 21 deletions.
36 changes: 24 additions & 12 deletions docs/content/integrations/sigma.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -38,37 +38,39 @@ pip install dagster dagster-sigma

To load Sigma assets into the Dagster asset graph, you must first construct a <PyObject module="dagster_sigma" object="SigmaOrganization" /> resource, which allows Dagster to communicate with your Sigma organization. You'll need to supply your client ID and client secret alongside the base URL. See [Identify your API request URL](https://help.sigmacomputing.com/reference/get-started-sigma-api#identify-your-api-request-url) in the Sigma documentation for more information on how to find your base URL.

Dagster can automatically load all datasets and workbooks ƒrom your Sigma workspace. Call the <PyObject module="dagster_sigma" object="SigmaOrganization" method="build_defs" /> function, which returns a `Definitions` object containing all the asset definitions for these Sigma assets.
Dagster can automatically load all datasets and workbooks from your Sigma workspace as asset specs. Call the <PyObject module="dagster_sigma" method="load_sigma_asset_specs" /> function, which returns list of <PyObject object="AssetSpec" />s representing your Sigma assets. You can then include these asset specs in your <PyObject object="Definitions" /> object:

```python file=/integrations/sigma/representing-sigma-assets.py
from dagster_sigma import SigmaBaseUrl, SigmaOrganization
from dagster_sigma import SigmaBaseUrl, SigmaOrganization, load_sigma_asset_specs

import dagster as dg

resource = SigmaOrganization(
sigma_organization = SigmaOrganization(
base_url=SigmaBaseUrl.AWS_US,
client_id=dg.EnvVar("SIGMA_CLIENT_ID"),
client_secret=dg.EnvVar("SIGMA_CLIENT_SECRET"),
)

defs = resource.build_defs()
sigma_specs = load_sigma_asset_specs(sigma_organization)
defs = dg.Definitions(assets=[*sigma_specs], resources={"sigma": sigma_organization})
```

### Customize asset definition metadata for Sigma assets

By default, Dagster will generate asset keys for each Sigma asset based on its type and name and populate default metadata. You can further customize asset properties by passing a custom <PyObject module="dagster_sigma" object="DagsterSigmaTranslator" /> subclass to the <PyObject module="dagster_sigma" object="SigmaOrganization" method="build_defs" /> function. This subclass can implement methods to customize the asset keys or specs for each Sigma asset type.
By default, Dagster will generate asset keys for each Sigma asset based on its type and name and populate default metadata. You can further customize asset properties by passing a custom <PyObject module="dagster_sigma" object="DagsterSigmaTranslator" /> subclass to the <PyObject module="dagster_sigma" method="load_sigma_asset_specs" /> function. This subclass can implement methods to customize the asset keys or specs for each Sigma asset type.

```python file=/integrations/sigma/customize-sigma-asset-defs.py
from dagster_sigma import (
DagsterSigmaTranslator,
SigmaBaseUrl,
SigmaOrganization,
SigmaWorkbook,
load_sigma_asset_specs,
)

import dagster as dg

resource = SigmaOrganization(
sigma_organization = SigmaOrganization(
base_url=SigmaBaseUrl.AWS_US,
client_id=dg.EnvVar("SIGMA_CLIENT_ID"),
client_secret=dg.EnvVar("SIGMA_CLIENT_SECRET"),
Expand All @@ -82,15 +84,18 @@ class MyCustomSigmaTranslator(DagsterSigmaTranslator):
return super().get_asset_spec(data)._replace(owners=["my_team"])


defs = resource.build_defs(dagster_sigma_translator=MyCustomSigmaTranslator)
sigma_specs = load_sigma_asset_specs(
sigma_organization, dagster_sigma_translator=MyCustomSigmaTranslator
)
defs = dg.Definitions(assets=[*sigma_specs], resources={"sigma": sigma_organization})
```

### Load Sigma assets from multiple organizations

Definitions from multiple Sigma organizations can be combined by instantiating multiple <PyObject module="dagster_sigma" object="SigmaOrganization" /> resources and merging their definitions. This lets you view all your Sigma assets in a single asset graph:
Definitions from multiple Sigma organizations can be combined by instantiating multiple <PyObject module="dagster_sigma" object="SigmaOrganization" /> resources and merging their specs. This lets you view all your Sigma assets in a single asset graph:

```python file=/integrations/sigma/multiple-sigma-organizations.py
from dagster_sigma import SigmaBaseUrl, SigmaOrganization
from dagster_sigma import SigmaBaseUrl, SigmaOrganization, load_sigma_asset_specs

import dagster as dg

Expand All @@ -106,9 +111,16 @@ marketing_team_organization = SigmaOrganization(
client_secret=dg.EnvVar("MARKETING_SIGMA_CLIENT_SECRET"),
)

defs = dg.Definitions.merge(
sales_team_organization.build_defs(),
marketing_team_organization.build_defs(),
sales_team_specs = load_sigma_asset_specs(sales_team_organization)
marketing_team_specs = load_sigma_asset_specs(marketing_team_organization)

# Merge the specs into a single set of definitions
defs = dg.Definitions(
assets=[*sales_team_specs, *marketing_team_specs],
resources={
"marketing_sigma": marketing_team_organization,
"sales_sigma": sales_team_organization,
},
)
```

Expand Down
2 changes: 2 additions & 0 deletions docs/sphinx/sections/api/apidocs/libraries/dagster-sigma.rst
Original file line number Diff line number Diff line change
Expand Up @@ -26,3 +26,5 @@ Assets (Sigma API)
.. autoclass:: SigmaDataset

.. autoclass:: SigmaWorkbook

.. autofunction:: load_sigma_asset_specs
Original file line number Diff line number Diff line change
Expand Up @@ -3,11 +3,12 @@
SigmaBaseUrl,
SigmaOrganization,
SigmaWorkbook,
load_sigma_asset_specs,
)

import dagster as dg

resource = SigmaOrganization(
sigma_organization = SigmaOrganization(
base_url=SigmaBaseUrl.AWS_US,
client_id=dg.EnvVar("SIGMA_CLIENT_ID"),
client_secret=dg.EnvVar("SIGMA_CLIENT_SECRET"),
Expand All @@ -21,4 +22,7 @@ def get_workbook_spec(self, data: SigmaWorkbook) -> dg.AssetSpec:
return super().get_asset_spec(data)._replace(owners=["my_team"])


defs = resource.build_defs(dagster_sigma_translator=MyCustomSigmaTranslator)
sigma_specs = load_sigma_asset_specs(
sigma_organization, dagster_sigma_translator=MyCustomSigmaTranslator
)
defs = dg.Definitions(assets=[*sigma_specs], resources={"sigma": sigma_organization})
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
from dagster_sigma import SigmaBaseUrl, SigmaOrganization
from dagster_sigma import SigmaBaseUrl, SigmaOrganization, load_sigma_asset_specs

import dagster as dg

Expand All @@ -14,7 +14,14 @@
client_secret=dg.EnvVar("MARKETING_SIGMA_CLIENT_SECRET"),
)

defs = dg.Definitions.merge(
sales_team_organization.build_defs(),
marketing_team_organization.build_defs(),
sales_team_specs = load_sigma_asset_specs(sales_team_organization)
marketing_team_specs = load_sigma_asset_specs(marketing_team_organization)

# Merge the specs into a single set of definitions
defs = dg.Definitions(
assets=[*sales_team_specs, *marketing_team_specs],
resources={
"marketing_sigma": marketing_team_organization,
"sales_sigma": sales_team_organization,
},
)
Original file line number Diff line number Diff line change
@@ -1,11 +1,12 @@
from dagster_sigma import SigmaBaseUrl, SigmaOrganization
from dagster_sigma import SigmaBaseUrl, SigmaOrganization, load_sigma_asset_specs

import dagster as dg

resource = SigmaOrganization(
sigma_organization = SigmaOrganization(
base_url=SigmaBaseUrl.AWS_US,
client_id=dg.EnvVar("SIGMA_CLIENT_ID"),
client_secret=dg.EnvVar("SIGMA_CLIENT_SECRET"),
)

defs = resource.build_defs()
sigma_specs = load_sigma_asset_specs(sigma_organization)
defs = dg.Definitions(assets=[*sigma_specs], resources={"sigma": sigma_organization})

0 comments on commit 42f5163

Please sign in to comment.