KEP-74: support argo workflow #2976

KunWuLuan · 2024-09-04T09:32:57Z

What type of PR is this?

/kind documentation
/kind feature

What this PR does / why we need it:

Which issue(s) this PR fixes:

Special notes for your reviewer:

Does this PR introduce a user-facing change?

NONE

k8s-ci-robot · 2024-09-04T09:33:04Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: KunWuLuan
Once this PR has been reviewed and has the lgtm label, please assign alculquicondor for approval. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

k8s-ci-robot · 2024-09-04T09:33:08Z

Hi @KunWuLuan. Thanks for your PR.

I'm waiting for a kubernetes-sigs member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

netlify · 2024-09-04T09:33:15Z

✅ Deploy Preview for kubernetes-sigs-kueue ready!

Name	Link
🔨 Latest commit	`fc0837d`
🔍 Latest deploy log	https://app.netlify.com/sites/kubernetes-sigs-kueue/deploys/670f1bb04733780008909230
😎 Deploy Preview	https://deploy-preview-2976--kubernetes-sigs-kueue.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify site configuration.

keps/74-support-argo-workflow/kep.yaml

kannon92 · 2024-09-05T18:49:33Z

/ok-to-test

tenzen-y · 2024-09-11T15:29:37Z

It seems that some sections are still empty. So, when you fill all sections, we will review this KEP.

KunWuLuan · 2024-09-13T01:15:35Z

@tenzen-y Hi, I filled in the empty sections. And some implement details will be added after we discuss and choose the road.

Co-authored-by: Kevin Hannon <[email protected]>

tenzen-y · 2024-09-17T17:55:50Z

@tenzen-y Hi, I filled in the empty sections. And some implement details will be added after we discuss and choose the road.

Thanks for the updates.
@KunWuLuan Could you resolve CI errors with "make toc-update"?

andreyvelich

Thank you for driving this @KunWuLuan!
I left my initial questions.

keps/74-support-argo-workflow/README.md

andreyvelich · 2024-09-24T13:31:24Z

keps/74-support-argo-workflow/README.md

+  // stage should be implemented as GenericJob
+  stages := wf.GetActiveStages()
+  for _, stage := range stages {
+    // 1. make sure there is only a single existing instance of the workload.


What do you mean by single existing instance of workload here ?

For a specific stage which may refer to a stageGroup in workflow's status in workflow, we create a kueue.workload to track the resource requests.

andreyvelich · 2024-09-24T13:33:00Z

keps/74-support-argo-workflow/README.md

+
+#### How to suspend by stages
+
+In my daily work, I primarily encounter workflow managers in the form of Argo and Tekton. Consequently, this section will focus on these two. Other workflow management systems can draw upon the discussions in this section for implementation guidance.


Do we have any significant differences between Argo/Tekton and Airflow when we execute workflows in Kubernetes?
cc @shravan-achar @akshaychitneni @bigsur0

It seems suspend action in Tekton is not supported.

andreyvelich · 2024-09-24T13:38:29Z

keps/74-support-argo-workflow/README.md

+spec:
+  entrypoint: loop-example-depth-1
+  templates:
+  - name: kueue-suspend


Does it mean that you are going to mutate the existing Workflow with kueue-suspend step ?
Why not just mutate the suspend: true parameter to every step ?

Does it mean that you are going to mutate the existing Workflow with kueue-suspend step ?

Yes

Why not just mutate the suspend: true parameter to every step ?

Change suspend to false will start all suspend steps. Adding suspend template before every leaf template allow us to control the start of specific steps. For example, for the DAG like this:

A | \ B C

I think step B and step C should wait for different workload admissions.

andreyvelich · 2024-09-24T13:46:04Z

keps/74-support-argo-workflow/README.md

+      args: ["3m"]
+```
+
+Option 3. Kueue Webhook Enhancement: A new webhook is added within Kueue to intercept pod creations in the cluster. 


How the Argo Workflow controller will reconcile steps if Pods will be in SchedulingGated status ?

AFAIK, SchedulingGated status is almost same with pending status, so it should not have any effect to Argo Workflow controller. Maybe @terrytangyuan can give us some more information.

Co-authored-by: Andrey Velichkevich <[email protected]>

tenzen-y · 2024-09-25T19:35:17Z

keps/74-support-argo-workflow/README.md

+}
+
+
+func (r *workflowReconciler)ReconcileGenericWorkflow(ctx context.Context, req ctrl.Request, wf GenericWorkflow) {


Why do we need this workflow dedicated Reconciler?
We want to keep using the ReconcileGenericJob. The CompsableJob is a good example of how to perform the special typed Job in the generic Job reconciler.

Could you propose an approach to extend the ReconcileGenericJob by WorkflowJob interface?

I will try to consider about this.

Updated the design.

KunWuLuan · 2024-10-11T01:44:20Z

@terrytangyuan Hi, do you have any comments on these options? I think we need the feedback of workflow managers to determine what Kueue can do in its framework.

terrytangyuan · 2024-10-11T01:46:46Z

@terrytangyuan Hi, do you have any comments on these options? I think we need the feedback of workflow managers to determine what Kueue can do in its framework.

Thanks for the ping. I'll take a look when I get a chance. Will you be willing to drive and implement the changes needed on Argo Workflows side too?

KunWuLuan · 2024-10-11T01:58:32Z

@terrytangyuan Oh, yes. I am willing to do this.

alculquicondor · 2024-10-11T15:00:59Z

Sorry folks. With the k/k enhancements freeze, I had little time to read this. I'll try to make some time soon.

cc @mimowo

hbelmiro · 2024-10-04T19:12:05Z

keps/74-support-argo-workflow/README.md

+
+In this section, we will discuss the advantages and disadvantages of queuing Workflows, Stages, and Tasks individually. Additionally, we will explore how each of these can be implemented within Kueue.
+
+### What are the Stages for Argo Workflows?


It's not clear to me what a stage is.
Is it an Argo concept that I'm missing? If so, can you add a link to its definition in the Argo documentation?
Otherwise, can you please elaborate a bit more?

Stage is not a concept in Argo. I try to use a new word to represent the process that will create a set of pods in workflow. In Argo this may be a TaskGroup, or a StepGroup.
You can see the section 'What are the Stages for Argo Workflows'. If there are still some question the concept of stage, I will try to explain more in the section.

mimowo · 2024-10-14T07:21:53Z

@KunWuLuan thank you for driving the KEP. I will try to get to it this week. cc @mwielgus

k8s-ci-robot · 2024-10-16T01:58:25Z

@KunWuLuan: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
pull-kueue-verify-main	`fc0837d`	link	true	`/test pull-kueue-verify-main`

Full PR test history. Your PR dashboard. Please help us cut down on flakes by linking to an open issue when you hit one in your PR.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

mimowo · 2024-10-17T14:35:51Z

Sorry folks, this is an important KEP, but I will not be able to review it before 0.9 release is concluded.

KunWuLuan · 2024-11-21T11:39:05Z

Hello friends, anyone have any comment?

tenzen-y · 2024-11-22T08:16:15Z

Hello friends, anyone have any comment?

We assume to aim for this feature after the next release (v0.11) since the v0.10 deadline will come soon, within 2 weeks.

kep-74: support argo workflow

e6fb7b6

k8s-ci-robot requested review from denkensk and trasc September 4, 2024 09:33

k8s-ci-robot added cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. labels Sep 4, 2024

k8s-ci-robot added the size/L Denotes a PR that changes 100-499 lines, ignoring generated files. label Sep 4, 2024

kannon92 reviewed Sep 5, 2024

View reviewed changes

keps/74-support-argo-workflow/kep.yaml Outdated Show resolved Hide resolved

k8s-ci-robot added ok-to-test Indicates a non-member PR verified by an org member that is safe to test. and removed needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. labels Sep 5, 2024

tenzen-y mentioned this pull request Sep 6, 2024

Support Argo/Tekton workflows #74

Open

KunWuLuan force-pushed the kep/kueue-workflow branch from 65feed7 to ac94cf8 Compare September 9, 2024 08:45

add kep

baa5ecd

KunWuLuan force-pushed the kep/kueue-workflow branch from ac94cf8 to baa5ecd Compare September 10, 2024 08:48

KunWuLuan changed the title ~~WIP: kep-74: support argo workflow~~ KEP-74: support argo workflow Sep 10, 2024

k8s-ci-robot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Sep 10, 2024

terrytangyuan mentioned this pull request Sep 10, 2024

Integration with Kueue argoproj/argo-workflows#12363

Open

update goals

0fac846

complete the doc

e6666be

KunWuLuan force-pushed the kep/kueue-workflow branch from aab364a to e6666be Compare September 12, 2024 02:32

Update keps/74-support-argo-workflow/kep.yaml

933c14b

Co-authored-by: Kevin Hannon <[email protected]>

k8s-ci-robot added do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. and removed size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels Sep 24, 2024

KunWuLuan changed the title ~~[WIP] KEP-74: support argo workflow~~ KEP-74: support argo workflow Sep 24, 2024

k8s-ci-robot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Sep 24, 2024

KunWuLuan force-pushed the kep/kueue-workflow branch from 7c71d1c to 9bc4d08 Compare September 24, 2024 09:23

update doc

2273de6

KunWuLuan force-pushed the kep/kueue-workflow branch from 9bc4d08 to 2273de6 Compare September 24, 2024 11:42

andreyvelich reviewed Sep 24, 2024

View reviewed changes

andreyvelich mentioned this pull request Sep 24, 2024

Support for the execution policy API in JobSet kubernetes-sigs/jobset#672

Open

3 tasks

KunWuLuan and others added 5 commits September 25, 2024 09:20

Update keps/74-support-argo-workflow/README.md

9486948

Co-authored-by: Andrey Velichkevich <[email protected]>

Update keps/74-support-argo-workflow/README.md

04379e1

Co-authored-by: Andrey Velichkevich <[email protected]>

Update keps/74-support-argo-workflow/README.md

43bd407

Co-authored-by: Andrey Velichkevich <[email protected]>

Update keps/74-support-argo-workflow/README.md

a0668ff

Co-authored-by: Andrey Velichkevich <[email protected]>

update toc

012eabe

tenzen-y reviewed Sep 25, 2024

View reviewed changes

update the design for workflow framework

1ed89ab

hbelmiro reviewed Oct 12, 2024

View reviewed changes

Merge branch 'kubernetes-sigs:main' into kep/kueue-workflow

fc0837d

This was referenced Oct 16, 2024

Self-nominate kunwuluan as reviewer #3239

Closed

REQUEST: New membership for kunwuluan kubernetes/org#5217

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

KEP-74: support argo workflow #2976

KEP-74: support argo workflow #2976

KunWuLuan commented Sep 4, 2024 •

edited by k8s-ci-robot

Loading

k8s-ci-robot commented Sep 4, 2024

k8s-ci-robot commented Sep 4, 2024

netlify bot commented Sep 4, 2024 •

edited

Loading

kannon92 commented Sep 5, 2024

tenzen-y commented Sep 11, 2024

KunWuLuan commented Sep 13, 2024

tenzen-y commented Sep 17, 2024

andreyvelich left a comment

andreyvelich Sep 24, 2024

KunWuLuan Sep 25, 2024

andreyvelich Sep 24, 2024

KunWuLuan Sep 25, 2024

andreyvelich Sep 24, 2024

KunWuLuan Sep 25, 2024

andreyvelich Sep 24, 2024

KunWuLuan Sep 25, 2024

tenzen-y Sep 25, 2024 •

edited

Loading

KunWuLuan Sep 30, 2024

KunWuLuan Sep 30, 2024

KunWuLuan commented Oct 11, 2024

terrytangyuan commented Oct 11, 2024

KunWuLuan commented Oct 11, 2024

alculquicondor commented Oct 11, 2024

hbelmiro Oct 4, 2024

KunWuLuan Oct 14, 2024

mimowo commented Oct 14, 2024

k8s-ci-robot commented Oct 16, 2024

mimowo commented Oct 17, 2024

KunWuLuan commented Nov 21, 2024

tenzen-y commented Nov 22, 2024


		#### How to suspend by stages

		In my daily work, I primarily encounter workflow managers in the form of Argo and Tekton. Consequently, this section will focus on these two. Other workflow management systems can draw upon the discussions in this section for implementation guidance.

		}


		func (r *workflowReconciler)ReconcileGenericWorkflow(ctx context.Context, req ctrl.Request, wf GenericWorkflow) {


		In this section, we will discuss the advantages and disadvantages of queuing Workflows, Stages, and Tasks individually. Additionally, we will explore how each of these can be implemented within Kueue.

		### What are the Stages for Argo Workflows?

KEP-74: support argo workflow #2976

Are you sure you want to change the base?

KEP-74: support argo workflow #2976

Conversation

KunWuLuan commented Sep 4, 2024 • edited by k8s-ci-robot Loading

What type of PR is this?

What this PR does / why we need it:

Which issue(s) this PR fixes:

Special notes for your reviewer:

Does this PR introduce a user-facing change?

k8s-ci-robot commented Sep 4, 2024

k8s-ci-robot commented Sep 4, 2024

netlify bot commented Sep 4, 2024 • edited Loading

✅ Deploy Preview for kubernetes-sigs-kueue ready!

kannon92 commented Sep 5, 2024

tenzen-y commented Sep 11, 2024

KunWuLuan commented Sep 13, 2024

tenzen-y commented Sep 17, 2024

andreyvelich left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tenzen-y Sep 25, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

KunWuLuan commented Oct 11, 2024

terrytangyuan commented Oct 11, 2024

KunWuLuan commented Oct 11, 2024

alculquicondor commented Oct 11, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mimowo commented Oct 14, 2024

k8s-ci-robot commented Oct 16, 2024

mimowo commented Oct 17, 2024

KunWuLuan commented Nov 21, 2024

tenzen-y commented Nov 22, 2024

KunWuLuan commented Sep 4, 2024 •

edited by k8s-ci-robot

Loading

netlify bot commented Sep 4, 2024 •

edited

Loading

tenzen-y Sep 25, 2024 •

edited

Loading