1.7.0 (core) / 0.23.0 (libraries)

Major Changes since 1.6.0 (core) / 0.22.0 (libraries)

• Asset definitions can now have tags, via the tags argument on @asset, AssetSpec, and AssetOut. Tags are meant to be used for organizing, filtering, and searching for assets.
• The Asset Details page has been revamped to include an “Overview” tab that centralizes the most important information about the asset – such as current status, description, and columns – in a single place.
• Assets can now be assigned owners.
• Asset checks are now considered generally available and will no longer raise experimental warnings when used.
• Asset checks can now be marked blocking, which causes downstream assets in the same run to be skipped if the check fails with ERROR-level severity.
• The new @multi_asset_check decorator enables defining a single op that executes multiple asset checks.
• The new build_last_updated_freshness_checks and build_time_partition_freshness_checks APIs allow defining asset checks that error or warn when an asset is overdue for an update. Refer to the Freshness checks guide for more info.
• The new build_column_schema_change_checks API allows defining asset checks that warn when an asset’s columns have changed since its latest materialization.
• In the asset graph UI, the “Upstream data”, “Code version changed”, and “Upstream code version” statuses have been collapsed into a single “Unsynced” status. Clicking on “Unsynced” displays more detailed information.
• I/O managers are now optional. This enhances flexibility for scenarios where they are not necessary. For guidance, see When to use I/O managers.
  • Assets with None or MaterializeResult return type annotations won't use I/O managers; dependencies for these assets can be set using the deps parameter in the @asset decorator.
• [dagster-dbt] Dagster’s dbt integration can now be configured to automatically collect metadata about column schema and column lineage.
• [dagster-dbt] dbt tests are now pulled in as Dagster asset checks by default.
• [dagster-dbt] dbt resource tags are now automatically pulled in as Dagster asset tags.
• [dagster-dbt] dbt owners from dbt groups are now automatically pulled in as Dagster owners.
• [dagster-snowflake] [dagster-gcp] The dagster-snowflake and dagster-gcp packages now both expose a fetch_last_updated_timestamps API, which makes it straightforward to collect data freshness information in source asset observation functions.

Changes since 1.6.14 (core) / 0.22.14 (libraries)

New

• Metadata attached during asset or op execution can now be accessed in the I/O manager using OutputContext.output_metadata.
• [experimental] Single-run backfills now support batched inserts of asset materialization events. This is a major performance improvement for large single-run backfills that have database writes as a bottleneck. The feature is off by default and can be enabled by setting the DAGSTER_EVENT_BATCH_SIZE environment variable in a code server to an integer (25 recommended, 50 max). It is only currently supported in Dagster Cloud and OSS deployments with a postgres backend.
• [ui] The new Asset Details page is now enabled for new users by default. To turn this feature off, you can toggle the feature in the User Settings.
• [ui] Queued runs now display a link to view all the potential reasons why a run might remain queued.
• [ui] Starting a run status sensor with a stale cursor will now warn you in the UI that it will resume from the point that it was paused.
• [asset-checks] Asset checks now support asset names that include ., which can occur when checks are ingested from dbt tests.
• [dagster-dbt] The env var DBT_INDIRECT_SELECTION will no longer be set to empty when executing dbt tests as asset checks, unless specific asset checks are excluded. dagster-dbt will no longer explicitly select all dbt tests with the dbt cli, which had caused argument length issues.
• [dagster-dbt] Singular tests with a single dependency are now ingested as asset checks.
• [dagster-dbt] Singular tests with multiple dependencies must have the primary dependency must be specified using dbt meta.

{{
    config(
        meta={
            'dagster': {
                'ref': {
                    'name': <ref_name>,
                    'package': ... # Optional, if included in the ref.
                    'version': ... # Optional, if included in the ref.
                },
            }
        }
    )
}}

...

• [dagster-dbt] Column lineage metadata can now be emitted when invoking dbt. See the documentation for details.
• [experimental][dagster-embedded-elt] Add the data load tool (dlt) integration for easily building and integration dlt ingestion pipelines with Dagster.
• [dagster-dbt][community-contribution] You can now specify a custom schedule name for schedules created with build_schedule_from_dbt_selection. Thanks @dragos-pop!
• [helm][community-contribution] You can now specify a custom job namespace for your user code deployments. Thanks @tmatthews0020!
• [dagster-polars][community-contribution] Column schema metadata is now integrated using the dagster-specific metadata key in dagster_polars. Thanks @danielgafni!
• [dagster-datadog][community-contribution] Added datadog.api module to the DatadogClient resource, enabling direct access to API methods. Thanks @shivgupta!

Bugfixes

• Fixed a bug where run status sensors configured to monitor a specific job would trigger for jobs with the same name in other code locations.
• Fixed a bug where multi-line asset check result descriptions were collapsed into a single line.
• Fixed a bug that caused a value to show up under “Target materialization” in the asset check UI even when an asset had had observations but never been materialized.
• Changed typehint of metadata argument on multi_asset and AssetSpec to Mapping[str, Any].
• [dagster-snowflake-pandas] Fixed a bug introduced in 0.22.4 where column names were not using quote identifiers correctly. Column names will now be quoted.
• [dagster-aws] Fixed an issue where a race condition where simultaneously materializing the same asset more than once would sometimes raise an Exception when using the s3_io_manager.
• [ui] Fixed a bug where resizable panels could inadvertently be hidden and never recovered, for instance the right panel on the global asset graph.
• [ui] Fixed a bug where opening a run with an op selection in the Launchpad could lose the op selection setting for the subsequently launched run. The op selection is now correctly preserved.
• [community-contribution] Fixed dagster-polars tests by excluding Decimal types. Thanks @ion-elgreco!
• [community-contribution] Fixed a bug where auto-materialize rule evaluation would error on FIPS-compliant machines. Thanks @jlloyd-widen!
• [community-contribution] Fixed an issue where an excessive DeprecationWarning was being issued for a ScheduleDefinition passed into the Definitions object. Thanks @2Ryan09!

Breaking Changes

• Creating a run with a custom non-UUID run_id was previously private and only used for testing. It will now raise an exception.
• [community-contribution] Previously, calling get_partition_keys_in_range on a MultiPartitionsDefinition would erroneously return partition keys that were within the one-dimensional range of alphabetically-sorted partition keys for the definition. Now, this method returns the cartesian product of partition keys within each dimension’s range. Thanks, @mst!
• Added AssetCheckExecutionContext to replace AssetExecutionContext as the type of the context param passed in to @asset_check functions. @asset_check was an experimental decorator.
• [experimental] @classmethod decorators have been removed from dagster-embedded-slt.sling DagsterSlingTranslator
• [dagster-dbt] @classmethod decorators have been removed from DagsterDbtTranslator.
• [dagster-k8s] The default merge behavior when raw kubernetes config is supplied at multiple scopes (for example, at the instance level and for a particluar job) has been changed to be more consistent. Previously, configuration was merged shallowly by default, with fields replacing other fields instead of appending or merging. Now, it is merged deeply by default, with lists appended to each other and dictionaries merged, in order to be more consistent with how kubernetes configuration is combined in all other places. See the docs for more information, including how to restore the previous default merge behavior.

Deprecations

• AssetSelection.keys() has been deprecated. Instead, you can now supply asset key arguments to AssetSelection.assets() .
• Run tag keys with long lengths and certain characters are now deprecated. For consistency with asset tags, run tags keys are expected to only contain alpha-numeric characters, dashes, underscores, and periods. Run tag keys can also contain a prefix section, separated with a slash. The main section and prefix section of a run tag are limited to 63 characters.
• AssetExecutionContext has been simplified. Op-related methods and methods with existing access paths have been marked deprecated. For a full list of deprecated methods see this GitHub Discussion.
• The metadata property on InputContext and OutputContext has been deprecated and renamed to definition_metadata .
• FreshnessPolicy is now deprecated. For monitoring freshness, use freshness checks instead. If you are using AutoMaterializePolicy.lazy(), FreshnessPolicy is still recommended, and will continue to be supported until an alternative is provided.

Documentation

• Lots of updates to examples to reflect the recent opt-in nature of I/O managers
• Dagster Cloud alert guides have been split up by alert type:
  • Managing alerts in the Dagster Cloud UI
  • Managing alerts using the dagster-cloud CLI
  • Email alerts
  • Microsoft Teams alerts
  • Slack alerts
• Added info about asset check-based-alerts to the Dagster Cloud alerting docs
• The Asset checks documentation got a face lift - info about defining and executing asset checks now lives in its own guide
• Added a new guide for using freshness checks to the Asset checks documentation
• Cleaned up the Getting help guide - it now includes a high-level summary of all Dagster support resources, making it easier to skim!
• [community-contribution] Fixed the indentation level of a code snippet in the dagster-polars documentation. Thanks @danielgafni!

Dagster Cloud

• The Dagster Cloud agent will now monitor the code servers that it spins to detect whether they have stopped serving requests, and will automatically redeploy the code server if it has stopped responding for an extended period of time.
• New additions and bugfixes in Insights:
  • Added per-metric cost estimation. Estimates can be added via the “Insights settings” button, and will appear in the table and chart for that metric.
  • Branch deployments are now included in the deployment filter control.
  • In the Deployments view, fixed deployment links in the data table.
  • Added support for BigQuery cost metrics.