Skip to content

Latest commit

 

History

History
133 lines (107 loc) · 6.91 KB

CHANGES.md

File metadata and controls

133 lines (107 loc) · 6.91 KB

Changelog

0.6.2

  • Changed composition functions @pipeline and @composite_solid to automatically give solids aliases with an incrementing integer suffix when there are conflicts. This removes to the need to manually alias solid definitions that are used multiple times.
  • Add dagster schedule wipe command to delete all schedules and remove all schedule cron jobs
  • execute_solid test util now works on composite solids.
  • Docs and example improvements: https://dagster.readthedocs.io/
  • Added --remote flag to dagster-graphql for querying remote dagit servers.
  • Fixed issue with duplicate run tag autocomplete suggestions in dagit (#1839)
  • Fixed Windows 10 / py3.6+ bug causing pipeline execution failures

0.6.1

  • Fixed an issue where Dagster public images tagged latest on Docker Hub were erroneously published with an older version of Dagster (#1814)
  • Fixed an issue where the most recent scheduled run was not displayed in dagit (#1815)
  • Fixed a bug with the dagster schedule start --start-all command (#1812)
  • Added a new scheduler command to restart a schedule: dagster schedule restart. Also added a flag to restart all running schedules: dagster schedule restart --restart-all-running.

0.6.0

New

This major release includes features for scheduling, operating, and executing pipelines that elevate dagit and dagster from a local development tool to a deployable service.

  • DagsterInstance introduced as centralized system to control run, event, compute log, and local intermediates storage.
  • A Scheduler abstraction has been introduced along side an initial implementation of SystemCronScheduler in dagster-cron.
  • dagster-aws has been extended with a CLI for deploying dagster to AWS. This can spin up a Dagit node and all the supporting infrastructure—security group, RDS PostgreSQL instance, etc.—without having to touch the AWS console, and for deploying your code to that instance.
  • Dagit
    • Runs: a completely overhauled Runs history page. Includes the ability to Retry, Cancel, and Delete pipeline runs from the new runs page.
    • Scheduler: a page for viewing and interacting with schedules.
    • Compute Logs: stdout and stderr are now viewable on a per execution step basis in each run. This is available in real time for currently executing runs and for historical runs.
    • A Reload button in the top right in dagit restarts the web-server process and updates the UI to reflect repo changes, including DAG structure, solid names, type names, etc. This replaces the previous file system watching behavior.

Breaking Changes

  • --log and --log-dir no longer supported as CLI args. Existing runs and events stored via these flags are no longer compatible with current storage.
  • raise_on_error moved from in process executor config to argument to arguments in python API methods such as execute_pipeline

0.5.9

  • Fixes an issue using custom types for fan-in dependencies with intermediate storage.

0.5.8

  • Fixes an issue running some Dagstermill notebooks on Windows.
  • Fixes a transitive dependency issue with Airflow.
  • Bugfixes, performance improvements, and better documentation.

0.5.7

  • Fixed an issue with specifying composite output mappings (#1674)
  • Added support for specifying Dask worker resources (#1679)
  • Fixed an issue with launching Dagit on Windows

0.5.6

  • Execution details are now configurable. The new top-level ExecutorDefinition and @executor APIs are used to define in-process, multiprocess, and Dask executors, and may be used by users to define new executors. Like loggers and storage, executors may be added to a ModeDefinition and may be selected and configured through the execution field in the environment dict or YAML, including through Dagit. Executors may no longer be configured through the RunConfig.
  • The API of dagster-dask has changed. Pipelines are now executed on Dask using the ordinary execute_pipeline API, and the Dask executor is configured through the environment. (See the dagster-dask README for details.)
  • Added the PresetDefinition.from_files API for constructing a preset from a list of environment files (replacing the old usage of this class). PresetDefinition may now be directly instantiated with an environment dict.
  • Added a prototype integration with dbt.
  • Added a prototype integration with Great Expectations.
  • Added a prototype integration with Papertrail.
  • Added the dagster-bash library.
  • Added the dagster-ssh library.
  • Added the dagster-sftp library.
  • Loosened the PyYAML compatibility requirement.
  • The dagster CLI no longer takes a --raise-on-error or --no-raise-on-error flag. Set this option in executor config.
  • Added a MarkdownMetadataEntryData class, so events yielded from client code may now render markdown in their metadata.
  • Bug fixes, documentation improvements, and improvements to error display.

0.5.5

  • Dagit now accepts parameters via environment variables prefixed with DAGIT_, e.g. DAGIT_PORT.
  • Fixes an issue with reexecuting Dagstermill notebooks from Dagit.
  • Bug fixes and display improvments in Dagit.

0.5.4

  • Reworked the display of structured log information and system events in Dagit, including support for structured rendering of client-provided event metadata.
  • Dagster now generates events when intermediates are written to filesystem and S3 storage, and these events are displayed in Dagit and exposed in the GraphQL API.
  • Whitespace display styling in Dagit can now be toggled on and off.
  • Bug fixes, display nits and improvements, and improvements to JS build process, including better display for some classes of errors in Dagit and improvements to the config editor in Dagit.

0.5.3

  • Pinned RxPY to 1.6.1 to avoid breaking changes in 3.0.0 (py3-only).
  • Most definition objects are now read-only, with getters corresponding to the previous properties.
  • The valueRepr field has been removed from ExecutionStepInputEvent and ExecutionStepOutputEvent.
  • Bug fixes and dagit UX improvements, including SQL highlighting and error handling.

0.5.2

  • Added top-level define_python_dagster_type function.
  • Renamed metadata_fn to typecheck_metadata_fn in all runtime type creation APIs.
  • Renamed result_value and result_values to output_value and output_values on SolidExecutionResult
  • Dagstermill: Reworked public API now contains only define_dagstermill_solid, get_context, yield_event, yield_result, DagstermillExecutionContext, DagstermillError, and DagstermillExecutionError. Please see the new guide for details.
  • Bug fixes, including failures for some dagster CLI invocations and incorrect handling of Airflow timestamps.