Skip to content

Releases: dagster-io/dagster

1.0.7 (core) / 0.16.7 (libraries)

01 Sep 21:47
Compare
Choose a tag to compare

New

  • Several updates to the Dagit run timeline view: your time window preference will now be preserved locally, there is a clearer “Now” label to delineate the current time, and upcoming scheduled ticks will no longer be batched with existing runs.
  • [dagster-k8s] ingress.labels is now available in the Helm chart. Any provided labels are appended to the default labels on each object (helm.sh/chart, app.kubernetes.io/version, and app.kubernetes.io/managed-by).
  • [dagster-dbt] Added support for two types of dbt nodes: metrics, and ephemeral models
  • When constructing a GraphDefinition manually, InputMapping and OutputMapping objects should be directly constructed.
  • [dagit] The launchpad tab is no longer shown for Asset jobs. Asset jobs can be launched via the “Materialize All” button shown on the Overview tab. To provide optional configuration, hold shift when clicking “Materialize”.

Bugfixes

  • [dagster-snowflake] Pandas is no longer imported when dagster_snowflake is imported. Instead, it’s only imported when using functionality inside dagster-snowflake that depends on pandas.
  • Recent changes to run status sensors caused sensors that only monitored jobs in external repositories to also monitor all jobs in the current repository. This has been fixed.
  • Fixed an issue where "unhashable type" errors could be spawned from sensor executions.
  • [dagit] Clicking between assets in different repositories from asset groups and asset jobs now works as expected.
  • [dagit] The DAG rendering of composite ops with more than one input/output mapping has been fixed.
  • [dagit] Selecting a source asset in Dagit no longer produces a GraphQL error
  • [dagit] Viewing “Related Assets” for an asset run now shows the full set of assets included in the run, regardless of whether they were materialized successfully.
  • [dagit] The Asset Lineage view has been simplified and lets you know if the view is being clipped and more distant upstream/downstream assets exist.
  • Fixed erroneous experimental warnings being thrown when using with_resources alongside source assets.

Breaking Changes

  • The arguments to the (internal) InputMapping and OutputMapping constructors have changed.

Community Contributions

  • The ssh_resource can now accept configuration from environment variables. Thanks @cbini!
  • Spelling corrections in migrations.md. Thanks @gogi2811!

1.0.6

26 Aug 16:14
Compare
Choose a tag to compare

New

  • [dagit] nbconvert is now installed as an extra in Dagit.
  • Multiple assets can be monitored for materialization using the multi_asset_sensor (experimental).
  • Run status sensors can now monitor jobs in external repositories.
  • The config argument of define_asset_job now works if the job contains partitioned assets.
  • When configuring sqlite-based storages in dagster.yaml, you can now point to environment variables.
  • When emitting RunRequests from sensors, you can now optionally supply an asset_selection argument, which accepts a list of AssetKeys to materialize from the larger job.
  • [dagster-dbt] load_assets_from_dbt_project and load_assets_from_dbt_manifest now support the exlude parameter, allowing you to more precisely which resources to load from your dbt project (thanks @flvndh!)

Bugfixes

  • Previously, types for multi-assets would display incorrectly in Dagit when specified. This has been fixed.
  • In some circumstances, viewing nested asset paths in Dagit could lead to unexpected empty states. This was due to incorrect slicing of the asset list, and has been fixed.
  • Fixed an issue in Dagit where the dialog used to wipe materializations displayed broken text for assets with long paths.
  • [dagit] Fixed the Job page to change the latest run tag and the related assets to bucket repository-specific jobs. Previously, runs from jobs with the same name in different repositories would be intermingled.
  • Previously, if you launched a backfill for a subset of a multi-asset (e.g. dbt assets), all assets would be executed on each run, instead of just the selected ones. This has been fixed.
  • [dagster-dbt] Previously, if you configured a select parameter on your dbt_cli_resource , this would not get passed into the corresponding invocations of certain context.resources.dbt.x() commands. This has been fixed.

1.0.4

19 Aug 00:19
Compare
Choose a tag to compare

New

  • Assets can now be materialized to storage conditionally by setting output_required=False. If this is set and no result is yielded from the asset, Dagster will not create an asset materialization event, the I/O manager will not be invoked, downstream assets will not be materialized, and asset sensors monitoring the asset will not trigger.
  • JobDefinition.run_request_for_partition can now be used inside sensors that target multiple jobs (Thanks Metin Senturk!)
  • The environment variable DAGSTER_GRPC_TIMEOUT_SECONDS now allows for overriding the default timeout for communications between host processes like dagit and the daemon and user code servers.
  • Import time for the dagster module has been reduced, by approximately 50% in initial measurements.
  • AssetIn now accepts a dagster_type argument, for specifying runtime checks on asset input values.
  • [dagit] The column names on the Activity tab of the asset details page no longer reference the legacy term “Pipeline”.
  • [dagster-snowflake] The execute_query method of the snowflake resource now accepts a use_pandas_result argument, which fetches the result of the query as a Pandas dataframe. (Thanks @swotai!)
  • [dagster-shell] Made the execute and execute_script_file utilities in dagster_shell part of the public API (Thanks Fahad Khan!)
  • [dagster-dbt] load_assets_from_dbt_project and load_assets_from_dbt_manifest now support the exclude parameter. (Thanks @flvndh!)

Bugfixes

  • [dagit] Removed the x-frame-options response header from Dagit, allowing the Dagit UI to be rendered in an iframe.
  • [fully-featured project example] Fixed the duckdb IO manager so the comment_stories step can load data successfully.
  • [dagster-dbt] Previously, if a select parameter was configured on the dbt_cli_resource, it would not be passed into invocations of context.resources.dbt.run() (and other similar commands). This has been fixed.
  • [dagster-ge] An incompatibility between dagster_ge_validation_factory and dagster 1.0 has been fixed.
  • [dagstermill] Previously, updated arguments and properties to DagstermillExecutionContext were not exposed. This has since been fixed.

Documentation

  • The integrations page on the docs site now has a section for links to community-hosted integrations. The first linked integration is @silentsokolov’s Vault integration.

1.0.3

11 Aug 22:53
Compare
Choose a tag to compare

New

  • Failure now has an allow_retries argument, allowing a means to manually bypass retry policies.
  • dagstermill.get_context and dagstermill.DagstermillExecutionContext have been updated to reflect stable dagster-1.0 APIs. pipeline/solid referencing arguments / properties will be removed in the next major version bump of dagstermill.
  • TimeWindowPartitionsDefinition now exposes a get_cron_schedule method.

Bugfixes

  • In some situations where an asset was materialized and that asset that depended on a partitioned asset, and that upstream partitioned asset wasn’t part of the run, the partition-related methods of InputContext returned incorrect values or failed erroneously. This was fixed.
  • Schedules and sensors with the same names but in different repositories no longer affect each others idempotence checks.
  • In some circumstances, reloading a repository in Dagit could lead to an error that would crash the page. This has been fixed.

Community Contributions

Documentation

1.0.2

08 Aug 23:10
Compare
Choose a tag to compare

New

  • When the workpace is updated, a notification will appear in Dagit, and the Workspace tab will automatically refresh.

Bugfixes

  • Restored the correct version mismatch warnings between dagster core and dagster integration libraries
  • Field.__init__ has been typed, which resolves an error that pylance would raise about default_value
  • Previously, dagster_type_materializer and dagster_type_loader expected functions to take a context argument from an internal dagster import. We’ve added DagsterTypeMaterializerContext and DagsterTypeLoaderContext so that functions annotated with these decorators can annotate their arguments properly.
  • Previously, a single-output op with a return description would not pick up the description of the return. This has been rectified.

Community Contributions

  • Fixed the dagster_slack documentation examples. Thanks @ssingh13-rms!

Documentation

1.0.1

05 Aug 03:50
Compare
Choose a tag to compare

Bugfixes

  • Fixed an issue where Dagster libraries would sometimes log warnings about mismatched versions despite having the correct version loaded.

Documentation

  • The Dagster Cloud docs now live alongside all the other Dagster docs! Check them out by nagivating to Deployment > Cloud.

1.0.0 - Hello

05 Aug 02:54
Compare
Choose a tag to compare

Major Changes

  • A docs site overhaul! Along with tons of additional content, the existing pages have been significantly edited and reorganized to improve readability.
  • All Dagster examplesare revamped with a consistent project layout, descriptive names, and more helpful README files.
  • A new dagster project CLI contains commands for bootstrapping new Dagster projects and repositories
    • dagster project scaffold creates a folder structure with a single Dagster repository and other files such as workspace.yaml. This CLI enables you to quickly start building a new Dagster project with everything set up.
    • dagster project from-example downloads one of the Dagster examples. This CLI helps you to quickly bootstrap your project with an officially maintained example. You can find the available examples via dagster project list-examples.
    • Check out Create a New Project for more details.
  • A default_executor_def argument has been added to the @repository decorator. If specified, this will be used for any jobs (asset or op) which do not explicitly set an executor_def.
  • A default_logger_defs argument has been added to the @repository decorator, which works in the same way as default_executor_def.
  • A new execute_job function presents a Python API for kicking off runs of your jobs.
  • Run status sensors may now yield RunRequests, allowing you to kick off a job in response to the status of another job.
  • When loading an upstream asset or op output as an input, you can now set custom loading behavior using the input_manager_key argument to AssetIn and In.
  • In the UI, the global lineage graph has been brought back and reworked! The graph keeps assets in the same group visually clustered together, and the query bar allows you to visualize a custom slice of your asset graph.

Breaking Changes and Deprecations

Legacy API Removals

In 1.0.0, a large number of previously-deprecated APIs have been fully removed. A full list of breaking changes and deprecations, alongside instructions on how to migrate older code, can be found in MIGRATION.md. At a high level:

  • The solid and pipeline APIs have been removed, along with references to them in extension libraries, arguments, and the CLI (deprecated in 0.13.0).
  • The AssetGroup and build_asset_job APIs, and a host of deprecated arguments to asset-related functions, have been removed (deprecated in 0.15.0).
  • The EventMetadata and EventMetadataEntryData APIs have been removed (deprecated in 0.15.0).

Deprecations

  • dagster_type_materializer and DagsterTypeMaterializer have been marked experimental and will likely be removed within a 1.x release. Instead, use an IOManager.
  • FileManager and FileHandle have been marked experimental and will likely be removed within a 1.x release.

Other Changes

  • As of 1.0.0, Dagster no longer guarantees support for python 3.6. This is in line with PEP 494, which outlines that 3.6 has reached end of life.
  • [planned] In an upcoming 1.x release, we plan to make a change that renders values supplied to configured in Dagit. Up through this point, values provided to configured have not been sent anywhere outside the process where they were used. This change will mean that, like other places you can supply configuration, configured is not a good place to put secrets: You should not include any values in configuration that you don't want to be stored in the Dagster database and displayed inside Dagit.
  • fs_io_manager, s3_pickle_io_manager, and gcs_pickle_io_manager, and adls_pickle_io_manager no longer write out a file or object when handling an output with the None or Nothing type.
  • The custom_path_fs_io_manager has been removed, as its functionality is entirely subsumed by the fs_io_manager, where a custom path can be specified via config.
  • The default typing_type of a DagsterType is now typing.Any instead of None.

New since 0.15.8

  • [dagster-databricks] When using the databricks_pyspark_step_launcher the events sent back to the host process are now compressed before sending, resulting in significantly better performance for steps which produce a large number of events.
  • [dagster-dbt] If an error occurs in load_assets_from_dbt_project while loading your repository, the error message in Dagit will now display additional context from the dbt logs, instead of just DagsterDbtCliFatalRuntimeError.

Bugfixes

  • Fixed a bug that causes Dagster to ignore the group_name argument to AssetsDefinition.from_graph when a key_prefix argument is also present.
  • Fixed a bug which could cause GraphQL errors in Dagit when loading repositories that contained multiple assets created from the same graph.
  • Ops and software-defined assets with the None return type annotation are now given the Nothing type instead of the Any type.
  • Fixed a bug that caused AssetsDefinition.from_graph and from_op to fail when invoked on a configured op.
  • The materialize function, which is not experimental, no longer emits an experimental warning.
  • Fixed a bug where runs from different repositories would be intermingled when viewing the runs for a specific repository-scoped job/schedule/sensor.
  • [dagster-dbt] A regression was introduced in 0.15.8 that would cause dbt logs to show up in json format in the UI. This has been fixed.
  • [dagster-databricks] Previously, if you were using the databricks_pyspark_step_launcher, and the external step failed to start, a RESOURCE_DOES_NOT_EXIST error would be surfaced, without helpful context. Now, in most cases, the root error causing the step to fail will be surfaced instead.

Documentation

  • New guide that walks through seamlessly transitioning code from development to production environments.
  • New guide that demonstrates using Branch Deployments to test Dagster code in your cloud environment without impacting your production data.

0.15.8 (2022-07-28)

28 Jul 18:37
Compare
Choose a tag to compare

New

  • Software-defined asset config schemas are no longer restricted to dicts.
  • The OpDefinition constructor now accept ins and outs arguments, to make direct construction easier.
  • define_dagstermill_op accepts ins and outs in order to make direct construction easier.

Bugfixes

  • Fixed a bug where default configuration was not applied when assets were selected for materialization in Dagit.
  • Fixed a bug where RunRequests returned from run_status_sensors caused the sensor to error.
  • When supplying config to define_asset_job, an error would occur when selecting most asset subsets. This has been fixed.
  • Fixed an error introduced in 0.15.7 that would prevent viewing the execution plan for a job re-execution from 0.15.0 → 0.15.6
  • [dagit] The Dagit server now returns 500 http status codes for GraphQL requests that encountered an unexpected server error.
  • [dagit] Fixed a bug that made it impossible to kick off materializations of partitioned asset if the day_offset, hour_offset, or minute_offset parameters were set on the asset’s partitions definition.
  • [dagster-k8s] Fixed a bug where overriding the Kubernetes command to use to run a Dagster job by setting the dagster-k8s/config didn’t actually override the command.
  • [dagster-datahub] Pinned version of acryl-datahub to avoid build error.

Breaking Changes

  • The constructor of JobDefinition objects now accept a config argument, and the preset_defs argument has been removed.

Deprecations

  • DagsterPipelineRunMetadataValue has been renamed to DagsterRunMetadataValue. DagsterPipelineRunMetadataValue will be removed in 1.0.

Community Contributions

  • Thanks to @hassen-io for fixing a broken link in the docs!

Documentation

  • MetadataEntry static methods are now marked as deprecated in the docs.
  • PartitionMappings are now included in the API reference.
  • A dbt example and memoization example using legacy APIs have been removed from the docs site.

0.15.7 (2022-07-21)

27 Jul 00:53
Compare
Choose a tag to compare

New

  • DagsterRun now has a job_name property, which should be used instead of pipeline_name.
  • TimeWindowPartitionsDefinition now has a get_partition_keys_in_range method which returns a sequence of all the partition keys between two partition keys.
  • OpExecutionContext now has asset_partitions_def_for_output and asset_partitions_def_for_input methods.
  • Dagster now errors immediately with an informative message when two AssetsDefinition objects with the same key are provided to the same repository.
  • build_output_context now accepts a partition_key argument that can be used when testing the handle_output method of an IO manager.

Bugfixes

  • Fixed a bug that made it impossible to load inputs using a DagsterTypeLoader if the InputDefinition had an asset_key set.
  • Ops created with the @asset and @multi_asset decorators no longer have a top-level “assets” entry in their config schema. This entry was unused.
  • In 0.15.6, a bug was introduced that made it impossible to load repositories if assets that had non-standard metadata attached to them were present. This has been fixed.
  • [dagster-dbt] In some cases, using load_assets_from_dbt_manifest with a select parameter that included sources would result in an error. This has been fixed.
  • [dagit] Fixed an error where a race condition of a sensor/schedule page load and the sensor/schedule removal caused a GraphQL exception to be raised.
  • [dagit] The “Materialize” button no longer changes to “Rematerialize” in some scenarios
  • [dagit] The live overlays on asset views, showing latest materialization and run info, now load faster
  • [dagit] Typing whitespace into the launchpad Yaml editor no longer causes execution to fail to start
  • [dagit] The explorer sidebar no longer displays “mode” label and description for jobs, since modes are deprecated.

Community Contributions

  • An error will now be raised if a @repository decorated function expects parameters. Thanks @roeij!

Documentation

  • The non-asset version of the Hacker News example, which lived inside examples/hacker_news/, has been removed, because it hadn’t received updates in a long time and had drifted from best practices. The asset version is still there and has an updated README. Check it out here

All Changes

0.15.6...0.15.7

See All Contributors
  • 57acdd5 - Correct check for pickle s3 io manager (#8834) by @ripplekhera
  • 4114910 - [dagit] Always show “Materialize” instead of “Rematerialize” based on status (#8711) by @bengotow
  • 972274c - [dagster-dbt] make group configurable for load_assets_from_dbt (#8863) by @OwenKephart
  • 05fc596 - asset_partitions_def on InputContext and OutputContext (#8858) by @sryza
  • 2d86c76 - [dagster-dbt] refactor the dbt asset integration (#8793) by @OwenKephart
  • 150bc3e - PartitionMappings when non-partitioned assets depend on partitioned assets (#8866) by @sryza
  • 70a7dbf - [dagster-dbt] seeds and snapshots are assets when using dbt build (#8794) by @OwenKephart
  • e46b9a0 - Document valid names for asset keys (#8765) by @jamiedemaria
  • 2357a02 - [docs] Dagster + dbt guide (#8714) by @OwenKephart
  • dbaed58 - 0.15.6 changelog (#8876) by @yuhan
  • 5bb50c0 - provide description for MAP ConfigType (#8824) by @Jiafi
  • b1aa83a - Retrieve minimal set of asset records for assetsLatestInfo (#8835) by @bengotow
  • b8493f3 - error when duplicate asset keys on a repository (#8874) by @sryza
  • 084c66c - [docs] - Add Airflow Operator to Op Docs (#8875) by @clairelin135
  • 2f15fbf - dagster-datahub Rest and Kafka Emitter integration (#8764) by @Jiafi
  • 0988274 - Automation: versioned docs for 0.15.6 by @elementl-devtools
  • 0e83834 - [1.0] move solid to dagster.legacy (#8843) by @dpeng817
  • bc5e502 - Extract ECS task overrides (#8842) by @jmsanders
  • e3ea175 - [graphql] tolerate empty runConfigData (#8886) by @alangenfeld
  • 56c7023 - [dagit] Fix edge case where “ “ launchpad config is not coerced to an empty object (#8895) by @bengotow
  • ee2e977 - Fix ScheduleRootQuery typo (#8903) by @johannkm
  • 61c1c20 - unloadable shit (#8887) by @prha
  • 711b323 - Change base image for OSS release builds (#8902) by @gibsondan
  • c85e158 - change deprecation warnings to 1.0 (#8892) by @dpeng817
  • cd779b1 - update README for hacker news assets example (#8904) by @sryza
  • e657abd - [hacker news] add missing key prefix to activity analytics python assets (#8817) by @sryza
  • 4da2a9e - [buildkite] Specify internal branch used for compatibility pipeline (#8881) by @smackesey
  • 6c97c75 - [dagit] Remove “mode” label and description in explorer sidebar (#8907) by @bengotow
  • 4cefd84 - remove the non-asset version of the hacker news example (#8883) by @sryza
  • 23a9997 - Error when @repository-decorated function has arguments (#8913) by @roeij
  • f787d6d - [docs] - Correct snippets for dbt (#8923) by @erinkcochran87
  • 1961e51 - [bug] fix input loading regression (#8885) by @OwenKephart
  • ff87738 - [docs] - graph backed assets doc fix (#8927) by @jamiedemaria
  • 18f254d - silence system-originated experimental warning for PartitionMapping (#8931) by @sryza
  • a2df1de - Add partition key to build_output_context, add documentation for partition_key on build_op_context (#8774) by @dpeng817
  • 53287b9 - fix dimensions of screenshot on connecting ops tutorial page (#8908) by @sryza
  • c00de5b - ttv: remove undocumented/legacy example - user in loop (#8934) by @yuhan
  • 31f3283 - [docs] - Clean up graph-backed asset example, put under test (#8893) by @dpeng817
  • 7c60a46 - [docs] - Fix garbled sentence in ops.mdx (#8935) by @schrockn
  • c554461 - enable getting asset partitions def from op context (#8924) by @sryza
  • 7c13e28 - Increase test_docker_monitoring timeout (#8906) by @johannkm
  • 6365996 - PartitionsDefinition.get_partition_keys_in_range (#8933) by @sryza
  • b58d711 - Move pipeline to dagster.legacy (#8888) by @dpeng817
  • 7e11df2 - [dagit] Rename search open event (#8954) by @hellendag
  • f3caeae - [dagit] Adjust shift-selection behavior in asset graphs (#8950) by @bengotow
  • 65caf79 - [dagit] Clean up code around the graph sidebar (#8914) by @bengotow
  • 5bd5c8b - add a job_name property to PipelineRun (#8928) by @sryza
  • 9421f73 - remove partition entries from asset op config schema (#8951) by @sryza
  • 000d37a - avoid pipelines in run status sensor doc snippets (#8929) by @sryza
  • a9b25dd - [bug] Fix issue where 'invalid' asset metadata resulted in an error (#8947) by @OwenKephart
  • 4dadcd4 - [dagster-dbt] fix tagged source asset loading (#8943) by @OwenKephart
  • 065adbd - fix black in run status sensor docs example (#8974) by @sryza
  • 164c585 - [known state] fix build_for_reexecution bug (#8975) by @alangenfeld
  • cc70c88 - Document deprecation of MetadataEntry static constructors (#8984) by @smackesey
  • eed9277 - changelog (#8986) by @jamiedemaria
  • b283b8a - 0.15.7 by @elementl-devtools

0.15.6 (2022-07-15)

15 Jul 16:29
Compare
Choose a tag to compare

New

  • When an exception is wrapped by another exception and raised within an op, Dagit will now display the full chain of exceptions, instead of stopping after a single exception level.
  • A default_logger_defs argument has been added to the @repository decorator. Check out the docs on specifying default loggers to learn more.
  • AssetsDefinition.from_graph and AssetsDefinition.from_op now both accept a partition_mappings argument.
  • AssetsDefinition.from_graph and AssetsDefinition.from_op now both accept a metadata_by_output_name argument.
  • define_asset_job now accepts an executor_def argument.
  • Removed package pin for gql in dagster-graphql.
  • You can now apply a group name to assets produced with the @multi_asset decorator, either by supplying a group_name argument (which will apply to all of the output assets), or by setting the group_name argument on individual AssetOuts.
  • InputContext and OutputContext now each have an asset_partitions_def property, which returns the PartitionsDefinition of the asset that’s being loaded or stored.
  • build_schedule_from_partitioned_job now raises a more informative error when provided a non-partitioned asset job
  • PartitionMapping, IdentityPartitionMapping, AllPartitionMapping, and LastPartitionMapping are exposed at the top-level dagster package. They're currently marked experimental.
  • When a non-partitioned asset depends on a partitioned asset, you can now control which partitions of the upstream asset are used by the downstream asset, by supplying a PartitionMapping.
  • You can now set PartitionMappings on AssetIn.
  • [dagit] Made performance improvements to the loading of the partitions and backfill pages.
  • [dagit] The Global Asset Graph is back by popular demand, and can be reached via a new “View global asset lineage ”link on asset group and asset catalog pages! The global graph keeps asset in the same group visually clustered together and the query bar allows you to visualize a custom slice of your asset graph.
  • [dagit] Simplified the Content Security Policy and removed frame-ancestors restriction.
  • [dagster-dbt] load_assets_from_dbt_project and load_assets_from_dbt_manifest now support a node_info_to_group_name_fn parameter, allowing you to customize which group Dagster will assign each dbt asset to.
  • [dagster-dbt] When you supply a runtime_metadata_fn when loading dbt assets, this metadata is added to the default metadata that dagster-dbt generates, rather than replacing it entirely.
  • [dagster-dbt] When you load dbt assets with use_build_command=True, seeds and snapshots will now be represented as Dagster assets. Previously, only models would be loaded as assets.

Bugfixes

  • Fixed an issue where runs that were launched using the DockerRunLauncher would sometimes use Dagit’s Python environment as the entrypoint to launch the run, even if that environment did not exist in the container.
  • Dagster no longer raises a “Duplicate definition found” error when a schedule definition targets a partitioned asset job.
  • Silenced some erroneous warnings that arose when using software-defined assets.
  • When returning multiple outputs as a tuple, empty list values no longer cause unexpected exceptions.
  • [dagit] Fixed an issue with graph-backed assets causing a GraphQL error when graph inputs were type-annotated.
  • [dagit] Fixed an issue where attempting to materialize graph-backed assets caused a graphql error.
  • [dagit] Fixed an issue where partitions could not be selected when materializing partitioned assets with associated resources.
  • [dagit] Attempting to materialize assets with required resources now only presents the launchpad modal if at least one resource defines a config schema.

Breaking Changes

  • An op with a non-optional DynamicOutput will now error if no outputs are returned or yielded for that dynamic output.
  • If an Output object is used to type annotate the return of an op, an Output object must be returned or an error will result.

Community Contributions

  • Dagit now displays the path of the output handled by PickledObjectS3IOManager in run logs and Asset view. Thanks @danielgafni

Documentation

  • The Hacker News example now uses stable 0.15+ asset APIs, instead of the deprecated 0.14.x asset APIs.
  • Fixed the build command in the instructions for contributing docs changes.
  • [dagster-dbt] The dagster-dbt integration guide now contains information on using dbt with Software-Defined Assets.

All Changes

0.15.5...0.15.6

See All Contributors
  • 583ce34 - Fold asset_defs submodule into main dagster structure (#8446) by @smackesey
  • b4fa57e - Op that runs a kubernetes job (#8161) by @gibsondan
  • a3a5ccb - add validation for graph backed asset graphs (#8754) by @OwenKephart
  • 5d5dd71 - chore: mark snapshots as generated (#8758) by @rexledesma
  • 4a9718d - [op] fix empty list output bug (#8763) by @alangenfeld
  • 397ad03 - [dagster-dbt] Allow SDAs generated with load_assets_from_dbt* to be partitioned (#8725) by @OwenKephart
  • b007a0c - docs: update config schema descriptions for default executors (#8757) by @rexledesma
  • 58518d9 - Restore sensor daemon yielding when evaluating sensors synchronously (#8756) by @prha
  • c449c0f - bypass bucketed queries for mysql versions that do not support it (#8753) by @prha
  • b1ac8d6 - [dagster-dbt] Fix error that occurs when generating events for tests that depend on sources (#8775) by @OwenKephart
  • 2a237a0 - Specifying executor docs examples (#8530) by @dpeng817
  • eccdac2 - prevent multiple sensor evaluations from multithreaded race conditions (#8720) by @prha
  • 519d77f - Fix config case for default executor (#8777) by @dpeng817
  • daadb5a - Ensure graph inputs/outputs are included in all_dagster_types (#8736) by @smackesey
  • a94210a - improve error for build_schedule_from_partitioned_job with non-partitioned asset job (#8776) by @sryza
  • 0bab6fd - [dagit] Bring back the global asset graph as an “all asset groups” view (#8709) by @bengotow
  • f6987da - fix source asset regression (#8784) by @smackesey
  • 530c321 - fix issue with repos and partitioned scheduled asset jobs (#8779) by @sryza
  • b3ba40c - 0.15.5 Changelog (#8781) by @prha
  • 0e32054 - changelog (#8788) by @prha
  • 1452280 - Option to hide daemon heartbeat timestamp in Dagit (#8785) by @johannkm
  • 1e7691d - Fix bug with how resources are applied in materialize (#8790) by @dpeng817
  • ccd1893 - Add default_logger_defs arg to repository (#8512) by @dpeng817
  • e10b6f3 - update hackernews tests to use asset invocation and materialize_to_memory (#8592) by @dpeng817
  • 5bdd6cf - Add MetaDataEntry.path to PickledObjectS3IOManager (#8732) by @danielgafni
  • 6d19b1d - Automation: versioned docs for 0.15.5 by @elementl-devtools
  • f459da0 - add define_asset_job to __all__ (#8791) by @sryza
  • f1a4612 - eliminate incorrect SDA warnings (#8769) by @sryza
  • 738e7eb - update hacker news assets example for post-0.15.0 APIs (#7904) by @sryza
  • 5938830 - partition mappings on graph-backed assets (#8768) by @sryza
  • b721c70 - Snowflake IO Manager handles pandas timestamps (#8760) by @jamiedemaria
  • cb178b1 - Add Python 3.10 testing to BK and other image adjustments (#7700) by @smackesey
  • f4415aa - Option to skip daemon heartbeats with no errors (#8670) by @johannkm
  • 2d35b84 - Assorted type annotations (#8356) by @smackesey
  • 3e1d539 - Bump urllib3 (#8808) by @dependabot[bot]
  • de366db - Bump rsa (#8807) by @dependabot[bot]
  • 6771e78 - Bump pyyaml (#8806) by @dependabot[bot]
  • 44492db - Change default local grpc behavior to send back "dagster" as the entry point to use, rather than dagit's python environment (#8571) by @gibsondan
  • 776d701 - move Metadata and Tags concept page under jobs section (#8813) by @sryza
  • c142bd9 - updates to multi-assets docs page (#8814) by @sryza
  • 3eb0afd - Update asset ID to contain repository location and name (#8762) by @clairelin135
  • a05f9b1 - unpin gql (#8822) by @prha
  • c89d01e - remove unused old partitions ui (#8796) by @prha
  • 1539cfb - fix broken asset deps in hacker news example (#8809) by @sryza
  • 7f5cd8e - [dagit] Bump TS version (#8704) by @hellendag
  • 8da4644 - fix contributing docs (#8789) by @prha
  • 1714bc9 - Better support for nested causes in dagit (#8823) by @gibsondan
  • 641c707 - [easy] Fix docs link for RetryPolicy (#8830) by @gibsondan
  • 04fb5c1 - Remove the "cronjobs" permission from the helm chart (#8827) by @gibsondan
  • e7111f9 - fix gql resolver for graph-backed assets resources (#8825) by @smackesey
  • 418952d - refactor run storage to enable backfill status queries (#8695) by @prha
  • 3eae463 - refactor backfill / partition pages to stop run fetching (#8696) by @prha
  • 47238c2 - add multi_or_in_process_executor to __all__ (#8831) by @smackesey
  • a3ec60b - avoid apollo cache collision for partition/backfill status (#8841) by @prha
  • f741443 - distinguish between [] and None for asset queries (#8838) by @prha
  • 9906c4b - override batch loader to use asset records instead of legacy event materialization method (#8839) by @prha
  • e523dad - [dagit] Add analytics.group (#8832) by @hellendag
  • 52d4fdf - [dagster-io/ui] Fix disabled Button (#8844) by @hellendag
  • 633d6b4 - Fix issue where partitioned assets with resources fail materialization in dagit (#8837) by @smackesey
  • 652d12e - [dagit] Tweak analytics function sigs (#8851) by @hellendag
  • 7cd7de8 - [asset-defs] allow multi assets to have group names (#8847) by @OwenKephart
  • acf8c4d - Refactor op return checking code (#8755) by @dpeng817
  • df5833e - `[dagit] Remove frame-ancesto...
Read more