Skip to content

Commit

Permalink
[dagster-aws] delete dummy Pipes classes (#23441)
Browse files Browse the repository at this point in the history
## Summary & Motivation

Deleting dummy Pipes classes from AWS Pipes. These classes didn't
provide any functionality and their introduction was questionable from
the start.

Context:
-
#23353 (comment)
- #22968

I decided to keep `PipesGlueLambdaEventContextInjector`, because unlike
the the other classes, it's actually used for a unique purpose:
injecting variables into Lambda event input (it might be a bit confusing
because it inherits from `PipesEnvContextInjector` but doesn't actually
do anything with environment variables).

@schrockn let me know if you think if it makes sense 

## How I Tested These Changes

Nothing was really using these classes
  • Loading branch information
danielgafni committed Aug 8, 2024
1 parent 0ac7a50 commit 430d2bf
Show file tree
Hide file tree
Showing 6 changed files with 7 additions and 18 deletions.
Binary file modified docs/content/api/modules.json.gz
Binary file not shown.
Binary file modified docs/content/api/searchindex.json.gz
Binary file not shown.
Binary file modified docs/content/api/sections.json.gz
Binary file not shown.
5 changes: 2 additions & 3 deletions docs/content/concepts/dagster-pipes/aws-glue.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -107,8 +107,7 @@ Next, add the `PipesGlueClient` resource to your project's <PyObject object="Def

```python file=/guides/dagster/dagster_pipes/glue/dagster_code.py startafter=start_definitions_marker endbefore=end_definitions_marker
from dagster import Definitions # noqa
from dagster_aws.pipes import PipesGlueContextInjector, PipesS3MessageReader

from dagster_aws.pipes import PipesS3ContextInjector, PipesS3MessageReader

bucket = os.environ["DAGSTER_GLUE_S3_CONTEXT_BUCKET"]

Expand All @@ -118,7 +117,7 @@ defs = Definitions(
resources={
"pipes_glue_client": PipesGlueClient(
client=boto3.client("glue"),
context_injector=PipesGlueContextInjector(
context_injector=PipesS3ContextInjector(
client=boto3.client("s3"),
bucket=bucket,
),
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -24,8 +24,7 @@ def glue_pipes_asset(
# start_definitions_marker

from dagster import Definitions # noqa
from dagster_aws.pipes import PipesGlueContextInjector, PipesS3MessageReader

from dagster_aws.pipes import PipesS3ContextInjector, PipesS3MessageReader

bucket = os.environ["DAGSTER_GLUE_S3_CONTEXT_BUCKET"]

Expand All @@ -35,7 +34,7 @@ def glue_pipes_asset(
resources={
"pipes_glue_client": PipesGlueClient(
client=boto3.client("glue"),
context_injector=PipesGlueContextInjector(
context_injector=PipesS3ContextInjector(
client=boto3.client("s3"),
bucket=bucket,
),
Expand Down
15 changes: 3 additions & 12 deletions python_modules/libraries/dagster-aws/dagster_aws/pipes.py
Original file line number Diff line number Diff line change
Expand Up @@ -334,24 +334,15 @@ def run(
return PipesClientCompletedInvocation(session)


class PipesGlueContextInjector(PipesS3ContextInjector):
def no_messages_debug_text(self) -> str:
return "Attempted to inject context via Glue job Arguments"


class PipesGlueLogsMessageReader(PipesCloudWatchMessageReader):
pass


@experimental
class PipesGlueClient(PipesClient, TreatAsResourceParam):
"""A pipes client for invoking AWS Glue jobs.
Args:
context_injector (Optional[PipesContextInjector]): A context injector to use to inject
context into the Glue job, for example, :py:class:`PipesGlueContextInjector`.
context into the Glue job, for example, :py:class:`PipesS3ContextInjector`.
message_reader (Optional[PipesMessageReader]): A message reader to use to read messages
from the glue job run. Defaults to :py:class:`PipesGlueLogsMessageReader`.
from the glue job run. Defaults to :py:class:`PipesCloudWatchsMessageReader`.
client (Optional[boto3.client]): The boto Glue client used to launch the Glue job
"""

Expand All @@ -363,7 +354,7 @@ def __init__(
):
self._client = client or boto3.client("glue")
self._context_injector = context_injector
self._message_reader = message_reader or PipesGlueLogsMessageReader()
self._message_reader = message_reader or PipesCloudWatchMessageReader()

@classmethod
def _is_dagster_maintained(cls) -> bool:
Expand Down

0 comments on commit 430d2bf

Please sign in to comment.