Merge branch 'dagster-io:master' into project-template-config-minimalism

CHANGES.md

@@ -1,5 +1,40 @@
 # Changelog
 
+## 1.8.12 (core) / 0.24.12 (libraries)
+
+### New
+
+- The `AutomationCondition.eager()`, `AutomationCondition.missing()`, and `AutomationCondition.on_cron` conditions are now compatible with asset checks.
+- Added `AssetSelection.materializable()`, which returns only assets that are materializable in an existing selection.
+- Added a new `AutomationCondition.all_deps_blocking_checks_passed` condition, which can be used to prevent materialization when any upstream blocking checks have failed.
+- Added a `code_version` parameter to the `@graph_asset` decorator.
+- If a `LaunchPartitionBackfill` mutation is submitted to GQL with invalid partition keys, it will now return an early `PartitionKeysNotFoundError`.
+- `AssetSelection.checks_for_assets` now accepts `AssetKey`s and string asset keys, in addition to `AssetsDefinition`s.
+- [ui] Added a search bar to partitions tab on the asset details page.
+- [ui] Restored docked left nav behavior for wide viewports.
+- [dagster-aws] `get_objects` now has a `since_last_modified` that enables only fetching objects modified after a given timestamp.
+- [dagster-aws] New AWS EMR Dagster Pipes client (`dagster_aws.pipes.PipesEMRCLient` ) for running and monitoring AWS EMR jobs from Dagster.
+- [dagster-looker] Pinned the looker-sdk dependency below 24.18.0 to avoid this issue: https://github.com/looker-open-source/sdk-codegen/issues/1518.
+
+### Bugfixes
+
+- Fixed an issue which could cause incorrect evaluation results when using self-dependent partition mappings with `AutomationConditions` that operate over dependencies.
+- [ui] Fixed an issue where the breadcumb on asset pages would flicker nonstop.
+- [dagster-embedded-elt] Fixed extraction of metadata for dlt assets whose source and destination identifiers differ.
+- [dagster-databricks] Fixed a permissioning gap that existed with the `DatabricksPySparkStepLauncher`, so that permissions are now set correctly for non-admin users.
+- [dagster-dbt] Fixed an issue where column metadata generated with `fetch_column_metadata` did not work properly for models imported through dbt dependencies.
+
+### Documentation
+
+- [dagster-k8s] `DagsterK8sPipesClient.run` now shows up in API docs.
+
+### Dagster Plus
+
+- [ui] Fixed a bug in the catalog UI where owners filters were not applied correctly.
+- [ui] Fixed width of the column lineage dropdown selector on the asset page.
+- [ui] Column lineage now correctly renders when set on asset definition metadata
+- [ui] Fixed Settings link on the list of deployments, for users in the legacy navigation flag.
+
 ## 1.8.11 (core) / 0.24.11 (libraries)
 
 ### New

docs/content/_apidocs.mdx

@@ -610,6 +610,16 @@ Dagster also provides a growing set of optional add-on libraries to integrate wi
         Includes implementations of run and event log storage built on Postgres.
       </td>
     </tr>
+    <tr>
+      <td>
+        <a href="/_apidocs/libraries/dagster-powerbi">PowerBI</a> (
+        <code>dagster-powerbi</code>)
+      </td>
+      <td>
+        Provides an integration to represent a PowerBI Workspace as a graph of
+        assets.
+      </td>
+    </tr>
     <tr>
       <td>
         <a href="/_apidocs/libraries/dagster-prometheus">Prometheus</a> (
@@ -631,6 +641,16 @@ Dagster also provides a growing set of optional add-on libraries to integrate wi
       </td>
       <td>Provides utilities for issuing shell commands from Dagster jobs.</td>
     </tr>
+    <tr>
+      <td>
+        <a href="/_apidocs/libraries/dagster-sigma">Sigma</a> (
+        <code>dagster-sigma</code>)
+      </td>
+      <td>
+        Provides an integration to represent a Sigma project as a graph of
+        assets.
+      </td>
+    </tr>
     <tr>
       <td>
         <a href="/_apidocs/libraries/dagster-slack">Slack</a> (
@@ -680,6 +700,13 @@ Dagster also provides a growing set of optional add-on libraries to integrate wi
         posting files via SFTP.
       </td>
     </tr>
+    <tr>
+      <td>
+        <a href="/_apidocs/libraries/dagster-tableau">Tableau</a> (
+        <code>dagster-tableau</code>)
+      </td>
+      <td>Provides a resource for integrating Tableau Workspaces</td>
+    </tr>
     <tr>
       <td>
         <a href="/_apidocs/libraries/dagster-twilio">Twilio</a> (

docs/content/_navigation.json

@@ -1532,6 +1532,10 @@
             "title": "PostgreSQL (dagster-postgres)",
             "path": "/_apidocs/libraries/dagster-postgres"
           },
+          {
+            "title": "PowerBI (dagster-powerbi)",
+            "path": "/_apidocs/libraries/dagster-powerbi"
+          },
           {
             "title": "Prometheus (dagster-prometheus)",
             "path": "/_apidocs/libraries/dagster-prometheus"
@@ -1568,6 +1572,10 @@
             "title": "SSH / SFTP (dagster-ssh)",
             "path": "/_apidocs/libraries/dagster-ssh"
           },
+          {
+            "title": "Tableau (dagster-tableau)",
+            "path": "/_apidocs/libraries/dagster-tableau"
+          },
           {
             "title": "Twilio (dagster-twilio)",
             "path": "/_apidocs/libraries/dagster-twilio"

docs/content/about/releases.mdx

@@ -23,12 +23,22 @@ The "experimental" marker allows us to offer new APIs to users and rapidly itera
 
 Experimental APIs may change or disappear within any release, but we try to avoid breaking them within minor releases if they have been around for a long time.
 
+### Superseded APIs
+
+The "superseded" marker indicates that we recommend avoiding an API, usually because there's a preferred option that should be used instead.
+
+Like non-deprecated public stable APIs, deprecated public stable APIs will not break within any major release after 1.0.
+
+Superseded APIs do not have a known scheduled removal point, but we recommend avoiding them in new code.
+
 ### Deprecated APIs
 
 The "deprecated" marker indicates that we recommend avoiding an API, usually because there's a preferred option that should be used instead.
 
 Like non-deprecated public stable APIs, deprecated public stable APIs will not break within any major release after 1.0.
 
+Unlike superseded APIs, deprecated APIs have a known scheduled removal point. We will provide guidance on when to expect the removal.
+
 ---
 
 ## Dagster integration libraries

docs/content/concepts/assets/external-assets.mdx

@@ -75,8 +75,6 @@ height={1654}
 
 ### External assets with dependencies
 
-External assets can depend only on other external assets.
-
 Dependencies are defined by using the `deps` argument of <PyObject object="AssetSpec" />. This enables Dagster to model entire graphs of assets scheduled and orchestrated by other systems.
 
 In the following example, we have two assets: `raw_logs` and `processed_logs`. The `processed_logs` asset is produced by a scheduled computation in another orchestration system. Using external assets allows you to model both assets in Dagster.

docs/content/deployment/run-launcher.mdx

@@ -26,7 +26,7 @@ The core abstraction in the launch process is the run launcher, which is configu
 
 ## Built-in run launchers
 
-The simplest run launcher is the built-in run launcher, <PyObject object="DefaultRunLauncher" />. This run launcher spawns a new process per run on the same node as the job's code location.
+The simplest run launcher is the built-in run launcher, <PyObject module="dagster._core.launcher" object="DefaultRunLauncher" />. This run launcher spawns a new process per run on the same node as the job's code location.
 
 Other run launchers include:
 

docs/content/integrations.mdx

@@ -285,4 +285,8 @@ Explore integration libraries that are maintained by the Dagster community.
     title="Open Metadata"
     href="https://docs.open-metadata.org/latest/connectors/pipeline/dagster"
   ></ArticleListItem>
+  <ArticleListItem
+    title="Perian"
+    href="https://github.com/Perian-io/dagster-perian"
+  ></ArticleListItem>
 </ArticleList>

docs/content/integrations/looker.mdx

@@ -27,27 +27,27 @@ This guide provides instructions for using Dagster with Looker. Your Looker asse
 
 ## Represent Looker assets in the asset graph
 
-To load Looker assets into the Dagster asset graph, you must first construct a `LookerResource`, which allows Dagster to communicate with your Looker instance. You'll need to supply your Looker instance URL and API credentials, which can be passed directly or accessed from the environment using `EnvVar`.
+To load Looker assets into the Dagster asset graph, you must first construct a <PyObject module="dagster_looker" object="LookerResource" />, which allows Dagster to communicate with your Looker instance. You'll need to supply your Looker instance URL and API credentials, which can be passed directly or accessed from the environment using <PyObject object="EnvVar" />.
 
-Dagster can automatically load all views, explores, and dashboards from your Looker instance. Call the `build_defs()` function, which returns a `Definitions` object containing all the asset definitions for these Looker assets.
+Dagster can automatically load all views, explores, and dashboards from your Looker instance. Call the <PyObject module="dagster_looker" object="LookerResource" method="build_defs" /> function, which returns a <PyObject object="Definitions" /> object containing all the asset definitions for these Looker assets.
 
 ```python file=/integrations/looker/representing-looker-assets.py
 from dagster_looker import LookerResource
 
-from dagster import EnvVar
+import dagster as dg
 
 resource = LookerResource(
-    client_id=EnvVar("LOOKERSDK_CLIENT_ID"),
-    client_secret=EnvVar("LOOKERSDK_CLIENT_SECRET"),
-    base_url=EnvVar("LOOKERSDK_HOST_URL"),
+    client_id=dg.EnvVar("LOOKERSDK_CLIENT_ID"),
+    client_secret=dg.EnvVar("LOOKERSDK_CLIENT_SECRET"),
+    base_url=dg.EnvVar("LOOKERSDK_HOST_URL"),
 )
 
 defs = resource.build_defs()
 ```
 
 ### Customize asset definition metadata for Looker assets
 
-By default, Dagster will generate asset keys for each Looker asset based on its type and name and populate default metadata. You can further customize asset properties by passing a custom `DagsterLookerAPITranslator` subclass to the `LookerResource.build_defs(...)` function. This subclass can implement methods to customize the asset keys or specs for each Looker asset type.
+By default, Dagster will generate asset keys for each Looker asset based on its type and name and populate default metadata. You can further customize asset properties by passing a custom <PyObject module="dagster_looker" object="DagsterLookerApiTranslator" /> subclass to the <PyObject module="dagster_looker" object="LookerResource" method="build_defs" /> function. This subclass can implement methods to customize the asset keys or specs for each Looker asset type.
 
 ```python file=/integrations/looker/customize-looker-assets.py
 from dagster_looker import (
@@ -57,17 +57,17 @@ from dagster_looker import (
     LookerStructureType,
 )
 
-from dagster import AssetSpec, EnvVar
+import dagster as dg
 
 resource = LookerResource(
-    client_id=EnvVar("LOOKERSDK_CLIENT_ID"),
-    client_secret=EnvVar("LOOKERSDK_CLIENT_SECRET"),
-    base_url=EnvVar("LOOKERSDK_HOST_URL"),
+    client_id=dg.EnvVar("LOOKERSDK_CLIENT_ID"),
+    client_secret=dg.EnvVar("LOOKERSDK_CLIENT_SECRET"),
+    base_url=dg.EnvVar("LOOKERSDK_HOST_URL"),
 )
 
 
 class CustomDagsterLookerApiTranslator(DagsterLookerApiTranslator):
-    def get_asset_spec(self, looker_structure: LookerStructureData) -> AssetSpec:
+    def get_asset_spec(self, looker_structure: LookerStructureData) -> dg.AssetSpec:
         asset_spec = super().get_asset_spec(looker_structure)
 
         # Add a team owner for all Looker assets
@@ -85,17 +85,17 @@ defs = resource.build_defs(dagster_looker_translator=CustomDagsterLookerApiTrans
 
 ### Materialize Looker PDTs from Dagster
 
-You can use Dagster to orchestrate the materialization of Looker PDTs. To model a PDT as an asset, pass its definition by passing `RequestStartPdtBuild` to `LookerResource.build_defs(...)` function.
+You can use Dagster to orchestrate the materialization of Looker PDTs. To model a PDT as an asset, pass its definition by passing <PyObject module="dagster_looker" object="RequestStartPdtBuild" /> to <PyObject module="dagster_looker" object="LookerResource" method="build_defs" /> function.
 
 ```python file=/integrations/looker/materializing-looker-pdts.py
 from dagster_looker import LookerResource, RequestStartPdtBuild
 
-from dagster import EnvVar
+import dagster as dg
 
 resource = LookerResource(
-    client_id=EnvVar("LOOKERSDK_CLIENT_ID"),
-    client_secret=EnvVar("LOOKERSDK_CLIENT_SECRET"),
-    base_url=EnvVar("LOOKERSDK_HOST_URL"),
+    client_id=dg.EnvVar("LOOKERSDK_CLIENT_ID"),
+    client_secret=dg.EnvVar("LOOKERSDK_CLIENT_SECRET"),
+    base_url=dg.EnvVar("LOOKERSDK_HOST_URL"),
 )
 
 defs = resource.build_defs(
@@ -107,3 +107,10 @@ defs = resource.build_defs(
     ]
 )
 ```
+
+### Related
+
+- [`dagster-looker` API reference](/\_apidocs/libraries/dagster-looker)
+- [Asset definitions](/concepts/assets/software-defined-assets)
+- [Resources](/concepts/resources)
+- [Using environment variables and secrets](/guides/dagster/using-environment-variables-and-secrets)