[dagster-aws] add ECS Task executor (#26098)

## Summary & Motivation This PR ads a new ECS Tasks Executor. Currently a `ECSRunLauncher` is required in order to use the new `ecs_executor` and most of the ECS config comes from the run launcher. The executor can override a few important fields (such as resources and container overrides). Resolve #9671 ## How I Tested These Changes - manual testing in personal AWS account - adding tests with mocks. Heavily using the existing run launcher testing infra since the executor currently requires it anyway ## Changelog [dagster-aws] new `ecs_executor` which executes Dagster steps via AWS ECS tasks. Initially, we expect it to be used in conjunction with `ECSRunLauncher`.
dagster-io · Dec 17, 2024 · 82e2189 · 82e2189 · github-actions · Dec 17, 2024
1 parent 5499873
commit 82e2189
Show file tree

Hide file tree

Showing 11 changed files with 778 additions and 43 deletions.
diff --git a/docs/content/api/modules.json.gz b/docs/content/api/modules.json.gz
diff --git a/docs/content/api/searchindex.json.gz b/docs/content/api/searchindex.json.gz
diff --git a/docs/content/api/sections.json.gz b/docs/content/api/sections.json.gz
diff --git a/docs/next/public/objects.inv b/docs/next/public/objects.inv
diff --git a/docs/sphinx/sections/api/apidocs/libraries/dagster-aws.rst b/docs/sphinx/sections/api/apidocs/libraries/dagster-aws.rst
@@ -49,6 +49,9 @@ ECS
 .. autoconfigurable:: dagster_aws.ecs.EcsRunLauncher
   :annotation: RunLauncher
 
+.. autoconfigurable:: dagster_aws.ecs.ecs_executor
+  :annotation: ExecutorDefinition
+
 
 Redshift
 --------

diff --git a/python_modules/libraries/dagster-aws/dagster_aws/ecs/__init__.py b/python_modules/libraries/dagster-aws/dagster_aws/ecs/__init__.py
@@ -1,2 +1,3 @@
+from dagster_aws.ecs.executor import ecs_executor as ecs_executor
 from dagster_aws.ecs.launcher import EcsRunLauncher as EcsRunLauncher
 from dagster_aws.ecs.tasks import EcsEventualConsistencyTimeout as EcsEventualConsistencyTimeout