DLT Integration VS Deployment in an external environment

[BenameurZyad]

Hello All,

We would like to have your ideas on the best approach (might not even be listed here) to have the ingestion pipelines of DLT orchestrated by Dagster Cloud but hosted elsewhere, while having the best integration with our current code location on Dagster Cloud.

We are currently looking into deploying our DLT pipelines for ingestion in a different environment than in Dagster Cloud, for it's known limitations.

Here is what we are looking into as deployment solutions :

• Dagster Pipes + AWS Lambda
  Dagster + AWS Lambda integration
  Not ideal for long running processes as AWS Lambda has a 15 minutes timeout limit.
  Some DLT pipelines take way longer than that.

• Dagster Pipes + Kubernetes
  Is the Dagster DLT integration integration usable in this case ?

• Dagster Pipes + EC2 Instance + AWS Lambda
  The idea is that AWS Lambda will start the EC2 instance and launch the Main Python script for the DLT Pipeline.
  Lambda only acts as a trigger, and Dagster Pipes is integrated with the code in the EC2 instance.
  Maybe an EC2 instance is not the only option ? (EKS, ECS) ?
  Is there a way to use the Dagster DLT integration in this solution ?

A general question is : is the Dagster DLT integration usable in some way with the Dagster Pipes ?

We are exploring different options and would like to have your recommandations and also if possible, the future vision for the Dagster DLT integration from the Dagster Labs team if possible.

Thanks a bunch in advance !
We are looking forward to hear from you.

Best