Skip to content

Commit

Permalink
[docs] add StepLauncher supersession notes (#25335)
Browse files Browse the repository at this point in the history
## Summary & Motivation

`StepLauncher` is being superseded by Pipes. 

I added `@deprecated` to all relevant objects. 

Should [this](https://docs.dagster.io/integrations/spark#submitting-pyspark-jobs-on-emr) section now point to `PipesEMRClient`? It's kinda ready but not fully tested yet. 

## Changelog

Step Launchers are superseded by Dagster Pipes and will no longer be developed. Learn more here: <future-github-discussion>
  • Loading branch information
danielgafni authored Oct 22, 2024
1 parent c3e96ea commit 1b3e659
Show file tree
Hide file tree
Showing 8 changed files with 20 additions and 4 deletions.
Binary file modified docs/content/api/modules.json.gz
Binary file not shown.
Binary file modified docs/content/api/searchindex.json.gz
Binary file not shown.
Binary file modified docs/content/api/sections.json.gz
Binary file not shown.
Binary file modified docs/next/public/objects.inv
Binary file not shown.
4 changes: 2 additions & 2 deletions docs/sphinx/sections/api/apidocs/internals.rst
Original file line number Diff line number Diff line change
Expand Up @@ -214,8 +214,8 @@ Exception handling

----

Step Launchers (Experimental)
-----------------------------
Step Launchers (Superseded)
---------------------------

.. currentmodule:: dagster

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,7 @@
from typing import TYPE_CHECKING, Iterator, Mapping, NamedTuple, Optional

import dagster._check as check
from dagster._annotations import superseded
from dagster._core.definitions.reconstruct import ReconstructableJob
from dagster._core.execution.retries import RetryMode
from dagster._core.storage.dagster_run import DagsterRun
Expand Down Expand Up @@ -56,6 +57,13 @@ def __new__(
)


_step_launcher_supersession = superseded(
subject="StepLauncher",
additional_warn_text="Consider using Dagster Pipes instead. Learn more here: https://docs.dagster.io/concepts/dagster-pipes",
)


@_step_launcher_supersession
class StepLauncher(ABC):
"""A StepLauncher is responsible for executing steps, either in-process or in an external process."""

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@
resource,
)
from dagster._core.definitions.resource_definition import dagster_maintained_resource
from dagster._core.definitions.step_launcher import StepLauncher
from dagster._core.definitions.step_launcher import StepLauncher, _step_launcher_supersession
from dagster._core.errors import DagsterInvariantViolationError, raise_execution_interrupts
from dagster._core.execution.plan.external_step import (
PICKLED_EVENTS_FILE_NAME,
Expand Down Expand Up @@ -147,6 +147,7 @@
),
}
)
@_step_launcher_supersession
def emr_pyspark_step_launcher(context):
# Resolve legacy arguments
if context.resource_config.get("local_job_package_path") and context.resource_config.get(
Expand Down Expand Up @@ -216,6 +217,7 @@ def emr_pyspark_step_launcher(context):
)


@_step_launcher_supersession
class EmrPySparkStepLauncher(StepLauncher):
def __init__(
self,
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,11 @@
)
from dagster._core.definitions.metadata import MetadataValue, RawMetadataValue
from dagster._core.definitions.resource_definition import dagster_maintained_resource
from dagster._core.definitions.step_launcher import StepLauncher, StepRunRef
from dagster._core.definitions.step_launcher import (
StepLauncher,
StepRunRef,
_step_launcher_supersession,
)
from dagster._core.errors import DagsterInvariantViolationError, raise_execution_interrupts
from dagster._core.events import DagsterEvent, DagsterEventType, EngineEventData
from dagster._core.events.log import EventLogEntry
Expand Down Expand Up @@ -169,6 +173,7 @@
),
}
)
@_step_launcher_supersession
def databricks_pyspark_step_launcher(
context: InitResourceContext,
) -> "DatabricksPySparkStepLauncher":
Expand All @@ -188,6 +193,7 @@ def databricks_pyspark_step_launcher(
return DatabricksPySparkStepLauncher(**context.resource_config)


@_step_launcher_supersession
class DatabricksPySparkStepLauncher(StepLauncher):
def __init__(
self,
Expand Down

2 comments on commit 1b3e659

@github-actions
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Deploy preview for dagster-docs ready!

✅ Preview
https://dagster-docs-bath0216v-elementl.vercel.app
https://master.dagster.dagster-docs.io

Built with commit 1b3e659.
This pull request is being automatically deployed with vercel-action

@github-actions
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Deploy preview for dagster-docs-beta ready!

✅ Preview
https://dagster-docs-beta-njgackq6f-elementl.vercel.app

Built with commit 1b3e659.
This pull request is being automatically deployed with vercel-action

Please sign in to comment.