1.1.21 (core) / 0.17.21 (libraries)

New

• Further performance improvements for build_asset_reconciliation_sensor.
• Dagster now allows you to backfill asset selections that include mapped partition definitions, such as a daily asset which rolls up into a weekly asset, as long as the root assets in your selection share a partition definition.
• Dagit now includes information about the cause of an asset’s staleness.
• Improved the error message for non-matching cron schedules in TimeWindowPartitionMappings with offsets. (Thanks Sean Han!)
• [dagster-aws] The EcsRunLauncher now allows you to configure the runtimePlatform field for the task definitions of the runs that it launches, allowing it to launch runs using Windows Docker images.
• [dagster-azure] Add support for DefaultAzureCredential for adls2_resource (Thanks Martin Picard!)
• [dagster-databricks] Added op factories to create ops for running existing Databricks jobs (create_databricks_run_now_op), as well as submitting one-off Databricks jobs (create_databricks_submit_run_op). See the new Databricks guide for more details.
• [dagster-duckdb-polars] Added a dagster-duckdb-polars library that includes a DuckDBPolarsTypeHandler for use with build_duckdb_io_manager, which allows loading / storing Polars DataFrames from/to DuckDB. (Thanks Pezhman Zarabadi-Poor!)
• [dagster-gcp-pyspark] New PySpark TypeHandler for the BigQuery I/O manager. Store and load your PySpark DataFrames in BigQuery using bigquery_pyspark_io_manager.
• [dagster-snowflake] [dagster-duckdb] The Snowflake and DuckDB IO managers can now load multiple partitions in a single step - e.g. when a non-partitioned asset depends on a partitioned asset or a single partition of an asset depends on multiple partitions of an upstream asset. Loading occurs using a single SQL query and returns a single DataFrame.

Bugfixes

• Previously, if an AssetSelection which matched no assets was passed into define_asset_job, the resulting job would target all assets in the repository. This has been fixed.
• Fixed a bug that caused the UI to show an error if you tried to preview a future schedule tick for a schedule built using build_schedule_from_partitioned_job.
• When a non-partitioned non-asset job has an input that comes from a partitioned SourceAsset, we now load all partitions of that asset.
• Updated the fs_io_manager to store multipartitioned materializations in directory levels by dimension. This resolves a bug on windows where multipartitioned materializations could not be stored with the fs_io_manager.
• Schedules and sensors previously timed out when attempting to yield many multipartitioned run requests. This has been fixed.
• Fixed a bug where context.partition_key would raise an error when executing on a partition range within a single run via Dagit.
• Fixed a bug that caused the default IO manager to incorrectly raise type errors in some situations with partitioned inputs.
• [ui] Fixed a bug where partition health would fail to display for certain time window partitions definitions with positive offsets.
• [ui] Always show the “Reload all” button on the code locations list page, to avoid an issue where the button was not available when adding a second location.
• [ui] Fixed a bug where users running multiple replicas of dagit would see repeated Definitions reloaded messages on fresh page loads.
• [ui] The asset graph now shows only the last path component of linked assets for better readability.
• [ui] The op metadata panel now longer capitalizes metadata keys
• [ui] The asset partitions page, asset sidebar and materialization dialog are significantly smoother when viewing assets with a large number of partitions (100k+)
• [dagster-gcp-pandas] The Pandas TypeHandler for BigQuery now respects user provided location information.
• [dagster-snowflake] ProgrammingError was imported from the wrong library, this has been fixed. Thanks @herbert-allium!

Experimental

• You can now set an explicit logical version on Output objects rather than using Dagster’s auto-generated versions.
• New get_asset_provenance method on OpExecutionContext allows fetching logical version provenance for an arbitrary asset key.
• [ui] - you can now create dynamic partitions from the partition selection UI when materializing a dynamically partitioned asset

Documentation

• Added an example of how to use dynamic asset partitions - in the examples/assets_dynamic_partitions folder
• New tutorial for using the BigQuery I/O manager.
• New reference page for BigQuery I/O manager features.
• New automating data pipelines guide

All Changes

1.1.20...1.1.21

See All Contributors

• 4343d59 - dagster-census api docs (#12413) by @yuhan
• 24b7e9b - graph_asset and graph_multi_asset decorators (#10152) by @sryza
• aa29161 - [dagster-snowflake-pyspark] fix bug loading partitions (#12472) by @jamiedemaria
• 2197dec - add graphql fields for querying run tags (#12409) by @prha
• 8889b48 - Add stale status causes (#11953) by @smackesey
• 323cdc8 - fix (#12477) by @salazarm
• 8e21900 - Update Contributing doc with instructions for ruff/pyright (#12481) by @smackesey
• c472edb - [bigquery] mark bigquery io manager experimental (#12479) by @jamiedemaria
• f2084d7 - add partial tag autocomplete for run filter input (#12410) by @prha
• 93e7cc1 - Support env valueFrom in Helm chart (#12425) by @johannkm
• 6601156 - Update GQL to expose StaleStatus and StaleStatusCause (#11952) by @smackesey
• 0ffe4a7 - remove timestamp comparisons of code location entries to reduce OSS dagit replica spam (#12407) by @prha
• 6af6f55 - Fix state status logical version test (#12484) by @smackesey
• a15d965 - fix ruff (#12486) by @alangenfeld
• 2c97adf - use opt_nullable_mapping for dagster library versions (#12487) by @alangenfeld
• 965152d - clarify error when op is missing argument for In (#12456) by @sryza
• 361b0ee - Remove existing RunConfig class (#12488) by @benpankow
• 75b6a5c - [pythonic resources] Clean up initialization of env vars, treat resource objects as immutable (#12445) by @benpankow
• e3c8825 - [structured config] Add support for Selectors w/ pydantic discriminated unions (#11280) by @benpankow
• 2d43d39 - Allow setting logical version inside op (#12189) by @smackesey
• 39f525a - Replace usages of nslookupwithnc for user deployments (#11033) by @michaeljguarino
• 2492317 - Revert "Replace usages of nslookupwithnc for user deployments (#11033)" by @johannkm
• b7fae37 - [pythonic resources] Last set of class renames (#12490) by @benpankow
• 763283f - [dagster-azure] Add support for DefaultAzureCredential for adls2_resource (#11309) by @mpicard
• 1d1f5ce - Add example of customizing task role and execution role arn to the ECS agent docs (#12491) by @gibsondan
• c4e3e87 - add dagster-duckdb-polars library (#12197) by @pzarabadip
• 3a05583 - [draft][pythonic config][docs] Introduce intro to Resources doc utilizing Pythonic resources (#12260) by @benpankow
• efed336 - [pythonic config] Add structured RunConfig object for specifying runtime, job config (#11965) by @benpankow
• 2f4a0e5 - [draft][pythonic config][docs] Introduce intro to Config doc utilizing Pythonic config (#12349) by @benpankow
• 4439129 - 1.1.20 changelog (#12506) by @benpankow
• 3dc1233 - refactor(databricks): lift polling methods up to the client (#12382) by @rexledesma
• edac939 - [fix] fix sphinx airflow version parsing (#12507) by @benpankow
• 215ae70 - Automation: versioned docs for 1.1.20 by @elementl-devtools
• da35211 - [dagit] Expose range-based asset health, use it for partition status rendering (#12302) by @bengotow
• e3fd740 - [dagit] Use range-based asset health for asset partitions / job partitions pages (#12434) by @bengotow
• 63b4273 - Fix 1.1.20 changelog codeblock (#12525) by @johannkm
• 2b1ad7c - [dagit] Delete generated GraphQL types before regenerating (#12518) by @hellendag
• 024db65 - [refactor] Delete build_solid_context (#12513) by @smackesey
• 18e7eb6 - [typing/static] serdes (#12522) by @smackesey
• 3e30026 - [refactor] make ResolvedRunConfig.to_dict use "ops" (#12514) by @smackesey
• 7e34083 - Improve multipartition performance for get_partition (#12431) by @clairelin135
• e798edf - feat(databricks): override user agent in resource (#12526) by @rexledesma
• 7036fca - Bump typing-extensions dep to >=4.4.0 (#12529) by @smackesey
• 781a8ea - [docs] Snowflake reference page fixes (#12455) by @jamiedemaria
• 2cc338b - [bugfix] Make assets downstream of partitions never stale in dagit (#12528) by @smackesey
• 65e92a1 - Enable internal testing for writing asset cached status data (#12497) by @clairelin135
• 281ec4b - [typing] fix typing in daemon tests (#12475) by @dpeng817
• f596b2f - dynamic partitions toy (#12533) by @sryza
• 7875f44 - pare down multi-partition runtime type checking in upath IO manager (#12508) by @sryza
• 102c0bc - [dagit] Upgrade to Jest 29, allow more time for coverage collection (#12534) by @bengotow
• a3b7dfc - Add description to invariant check (#12496) by @CodeMySky
• e00ba72 - [fix] Correctly resolve asset jobs with empty selections (#12531) by @OwenKephart
• 9d9b072 - [dagit] Support backfills on partition-mapped asset selections (#12458) by @bengotow
• 4fc6e3e - [dagit] Remove unnecessary usage of <TestProvider> (#12519) by @bengotow
• 31022e0 - Fix multipartitions w/ fs_io_manager on windows (#12414) by @clairelin135
• dd4ba07 - Fix master (#12550) by @clairelin135
• c93eb54 - Booleans that check permissions for specific objects and automatically incorporate per-location checks (#12548) by @gibsondan
• e867fcc - docs(dagster-dbt): add example to retrieve asset keys from dbt selection (#12323) by @rexledesma
• 704c638 - Fix IO manager doc snippets (#12545) by @clairelin135
• 6f2af95 - fix setup.cfg in create package (#12559) by @jamiedemaria
• 3a5ff3e - Fetch multipartition key from context methods (#12512) by @clairelin135
• 7d2b6be - Include partition in asset materialization planned event (#12333) by @johannkm
• 97fe5a0 - handle multiple non-time-window partitions in db io managers (#12517) by @sryza
• 848c6e8 - Add runtime_platform to EcsRunLauncher (#12566) by @gibsondan
• 2b22589 - Tag Selector Component (#12564) by @salazarm
• 1c3633c - Storybook for LaunchAssetChoosePartitionsDialog (#12567) by @salazarm
• 9ff5327 - Update setting-up-alerts.mdx (#12582) by @johannkm
• 2726401 - Move useViewport to uiformcore (#12578) by @salazarm
• 9a519d0 - [ui] Always show "Reload all" button (#12588) by @hellendag
• a6b91b3 - Bump minimist from 1.2.5 to 1.2.8 in /js_modules/dagit (#12595) by @dependabot[bot]
• 57830ae - Expose partition definition name (#12585) by @salazarm
• 6a1696b - add dynamic partitions example (#12494) by @sryza
• 2faf365 - [dagit] Do not capitalize tag keys in the Dagit op metadata panel (#12591) by @bengotow
• bc130b8 - [graphql] empty numPartitions and partitionNames for cross-partitioning backfills (#12462) by @sryza
• 14337e8 - [dagit] Use last path component of linked assets on the asset graph + add tooltips (#12590) by @bengotow
• 116ec37 - [dagster-snowflake] Fix ProgrammingError import in snowflake_io_manager (#12576) by @herbert-allium
• 9041d57 - remove dagster-cloud dep from dynamic partitions example (#12603) by @sryza
• c8bd24a - fix future ticks for partitioned schedules (#12601) by @sryza
• a41fecf - fix loading partitioned source assets in non-partitioned non-asset jobs (#12586) by @sryza
• c78992f - Update stale reasons to remove UNKNOWN, add MISSING (#12584) by @smackesey
• 216c9fc - GQL Add dynamic partition mutation (#12562) by @clairelin135
• 84201da - Add asset provenance access to OpExecutionContext (#12467) by @smackesey
• 7e3be46 - feat(databricks): add basic op implementations (#12492) by @rexledesma
• 2ffb21e - docs(databricks): add guide for Databricks integration (#12326) by @rexledesma
• d170c85 - migrate some tests away from AssetGroup (#12606) by @sryza
• 1a3019b - divest permissions for editing sensor from updating sensor cursor (#12589) by @dpeng817
• f532d6e - [dagit] Fix tests pinned to February (#12616) by @bengotow
• e403d39 - Move tests, fixtures, and storybooks into folders (#12614) by @salazarm
• 6ac4cff - Add root stale causes for assets (#12619) by @smackesey
• dab3831 - docs for graphs that depend on assets (#12597) by @sryza
• 122d46e - [UI] Dynamic partition creation and selection (#12615) by @salazarm
• e4418ee - [asset-reconciliation][perf] Do not get/set known_used_data in asset reconciliation loop (#12433) by @OwenKephart
• e4aeeee - [dagster-gcp-pyspark] Add BigQuery PySpark type handler (#12398) by @jamiedemaria
• b552e39 - Automating pipeline guide (#12547) by @odette-elementl
• a1b96c7 - Capture db timeout errors for tag key queries (#12596) by @prha
• e1f9d7f - Get partitions with planned but not completed materializations, the event log query (#12404) by @johannkm
• 83c875d - Add failures to partition status cache (#12599) by @johannkm
• ff7d080 - [bigquery-pandas] pipe gcp location through to type handler (#12587) by @jamiedemaria
• 08e5e0e - [docs, bigquery] tutorial and reference guide (#12452) by @jamiedemaria
• 5d9e03c - make freshness policies work with graph_asset and graph_multi_asset (#12630) by @sryza
• f8acfcd - Revert "Add failures to partition status cache (#12599)" by @johannkm
• 7f4676b - Revert "Get partitions with planned but not completed materializations, the event log query (#12404)" by @johannkm
• 445a134 - Modify get_first_partition_window to account for offset (#12504) by @clairelin135
• a1b4eea - [dagit] Switch to staleStatus for stale tags, display causes in tooltips (#12611) by @bengotow
• 349a0b0 - Change format of staleness root causes and remove old resolvers (#12628) by @smackesey
• c6e54a6 - Helm: Support valueFrom in user deployments (#12644) by @johannkm
• 69f751c - Update asset versioning guide (#12659) by @smackesey
• 6d17799 - update CHANGES.md for 1.1.21 (#12662) by @sryza
• ed9e77b - docs(databricks): update guide based on feedback (#12669) by @rexledesma
• c607767 - 1.1.21 by @elementl-devtools