Is it possible to separate task executions and orchestration on different machines?

[kumare3]

Hi all, I’m new here and wondering if it is possible to separate task executions and orchestration on different machines (or cloud providers) with Flyte. Something similar to Prefect’s Hybrid model (https://medium.com/the-prefect-blog/the-prefect-hybrid-model-1b70c7fd296) would be possible?

by [email protected]

(from slack)

[kumare3]

by @jeevb
i think this is already the case via a push model. flyte running on some k8s cluster within one cloud provider can farm tasks out to other providers like AWS Batch via flyte plugins / API calls. but it sounds like prefect hybrid uses a pull model where agents running on different infrastructure pull instructions from the orchestrator, execute, and push state updates back to the orchestrator. afaik this isn’t possible via the current OSS version of flyte.
@takan what’s the motivation, if you don’t mind me asking?

[kumare3]

@takan I will ad to what @jeev said,
So I will assume that you want to, to execute different tasks on different cloud environments.

Comment on Flyte’s design. It is designed to schedule out workflows to one K8s cluster. The tasks within a workflow are scheduled using backend plugins.
Thus,
As @jeev said you can have plugins that execute tasks on external services like AWS Batch, Sagemaker, BigQuery, Dataflow, other container orchestrators etc
We recently added support for executing tasks on external K8s clusters as well (can provide links to this - not really documented yet)
you can use externalized / virutal kubelets to scale out to different machines. For example use EKS Anywhere / GCP anthos etc.
@takan one interesting thing for us would be to understand - whats the usecase. How will you differentiate which tasks should be run on which clouds within one Workflow?

If the question is can the control plane and data plane be separated?
Then, they are already separated in the design

If the question is can we run different workflows on different cloud providers?
This is entire doable, except for the raw data references.