buildkite-agent pipeline upload should chunk uploads for the user

[jsoo1]

Is your feature request related to a problem? Please describe.
We regularly upload pipelines with step quantities > 500. It is frustrating to have to do the chunking as the user.

Describe the solution you'd like
buildkite-agent pipeline upload should chunk the steps for me, especially if the stdin stream can be made to support it.

Describe alternatives you've considered
GNU parallel does something like this but it is a papercut.

[moskyb]

hey @jsoo1! i get where you're coming from here - the 500 step limit is a little bit of intentional friction to give our servers a wee bit of a break in between large pipeline uploads. i agree that there may be better ways to do this - ie, chunking and waiting a bit between chunk uploads. we'll have a bit more of a chat internally and get back to you.

[jsoo1]

hey @jsoo1! i get where you're coming from here - the 500 step limit is a little bit of intentional friction to give our servers a wee bit of a break in between large pipeline uploads. i agree that there may be better ways to do this - ie, chunking and waiting a bit between chunk uploads. we'll have a bit more of a chat internally and get back to you.

Hey there! Totally understand! That would be appreciated!

[ozdenyilmaz]

Hi @jsoo1 , we did talk internally and this is great feedback. We unfortunately are unsure about when we'll get to it, but this will stay in our feedback backlog. If you need it urgently we would welcomes PRs for it. Just make sure that the chunk size is 100 to be safe.

[jsoo1]

Lovely! I may put in a patch but I wouldn't count on it.