Replicate (mostly) buildbot workflow for commits into master #1225

oxpa · 2024-04-19T21:51:54Z

Hello,

This pull request utilizes nginx self hosted runners to build unit and it's modules on a variety of distros and two archs (for the lack of others for now).

The proposed CI creates 252 jobs which can be a bit excessive and may require some optimisations: some jobs don't really do what they should do just skipping test. It's mostly true for python, java and php where each distro has a number of version available and I was too lazy to get a proper configuration for what plays where.

I also replicated configuration management system by creating setup.sh which installs packages from a number of lists. And I created those per-OS list. The list is a bit excessive and replicates current buildbot workers. So there is some room for optimisations there as well.

The existing CI is left active for pull requests as it provides relatively fast way of checking code for obvious problems.

.github/workflows/ci-pull.yml

ac000 · 2024-04-21T03:02:43Z

.github/scripts/setup.sh

+PACKAGE_MANAGER_INSTALL_yum="yum --setopt=skip_missing_names_on_install=False install -y"
+PACKAGE_MANAGER_REMOVE_yum="yum erase -y"


Once we drop CentOS 7 (EOL in a couple of months IIRC) we should replace yum with dnf (it replaced yum several years ago), I don't even have yum on my Fedora 39 boxes.

Looks like RedHat rebrands dnf to yum in RHEL 8: https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/8/html/configuring_basic_system_settings/managing-software-packages_configuring-basic-system-settings

Upstream documentation identifies the technology as DNF and the tool is referred to as DNF in the upstream.

We can fix this later, I guess.
As per fedora, in a default rawhide container yum is still a symlink to dnf, so I don't expect any problems with earlier releases either.

:~$ docker run --rm -it fedora:rawhide /usr/bin/yum --version 4.19.0 Installed: dnf-0:4.19.0-1.fc40.noarch at Mon Feb 19 08:10:37 2024 Built : Fedora Project at Thu Feb 8 16:33:42 2024 Installed: rpm-0:4.19.1.1-1.fc40.x86_64 at Mon Feb 19 08:10:37 2024 Built : Fedora Project at Wed Feb 7 15:55:53 2024

Looks like RedHat rebrands dnf to yum in RHEL 8: https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/8/html/configuring_basic_system_settings/managing-software-packages_configuring-basic-system-settings

Upstream documentation identifies the technology as DNF and the tool is referred to as DNF in the upstream.

Well, it's a version of yum that is compatible with dnf (and older yum).

We can fix this later, I guess. As per fedora, in a default rawhide container yum is still a symlink to dnf, so I don't expect any problems with earlier releases either.

:~$ docker run --rm -it fedora:rawhide /usr/bin/yum --version 4.19.0 Installed: dnf-0:4.19.0-1.fc40.noarch at Mon Feb 19 08:10:37 2024 Built : Fedora Project at Thu Feb 8 16:33:42 2024 Installed: rpm-0:4.19.1.1-1.fc40.x86_64 at Mon Feb 19 08:10:37 2024 Built : Fedora Project at Wed Feb 7 15:55:53 2024

Maybe I removed it...

$ yum bash: yum: command not found $ rpm -q yum package yum is not installed

ac000 · 2024-04-21T03:04:14Z

.github/workflows/ci-commit.yml

+          - ubuntu-22.04
+          - ubuntu-23.10
+          - debian-11
+          - debian-12
+          - alpine-3.19
+          - rhel-9
+          - amazonlinux-2023


There are more package lists than what's listed here, is that intentional?

Yes and no: there is alpine-3.18 list which I failed to use because there is no alpine 3.18 image for self hosted runners. So this one can be removed.
At the same time, having general lists (debian, alpine, ubuntu, rhel) is intentional: one can install a package for all versions of an OS family.

rhel-8 also seems to be in the lists but not the workflow file...

ac000 · 2024-04-21T03:07:26Z

Some observations

Does it make sense to still rename ci.yml?, seeing as it isn't limited to pull-requests.
The last patch should just be folded into the first or
Do we even need those commits (first & last) now?

callahad · 2024-04-22T09:49:20Z

I agree with Andrew's comments above; LGTM once those are addressed.

oxpa · 2024-04-22T10:48:12Z

@ac000

I'm happy to rename files further. See no reason to change their names but happy to rename it whatever you prefer.
I expect the purll request to be squash-merged. See no reason to keep this history of commits
Same as above: there should be a single commit in the repository.

callahad · 2024-04-22T12:24:47Z

Let's rename back to ci.yml and call this one ci-selfhosted.yml or something like that?

ac000 · 2024-04-22T13:23:55Z

.github/workflows/ci-pull.yml

@@ -1,10 +1,8 @@
 name: ci

 on:
+  workflow_dispatch:


Do we still need this change? Things seem to work without it...

This allows you to run a workflow manually on any branch. IMO it's useful and doesn't break anything.

Allows us to kick off builds without actually making a change to the repository, manually. Can be really useful!

OK, thanks for the explanation.

Then yeah, that should be done as a separate commit (it's not related to the other changes).

The file renaming part of this commit should be dropped.

ac000 · 2024-04-22T13:26:01Z

2. I expect the purll request to be squash-merged. See no reason to keep this history of commits

If you keep the ci.yml change, then that should at least be a separate commit (with an explanation of why it's needed).

arbourd

This looks great! Assuming it works tho, as the job wasn't able to kick off within the PR, obviously.

If there are too many jobs, the long-running self hosted environments could run all tests together at once, or at least, matrix against every module-lang rather than module-lang-version. But that can always be optimized later.

I'm happy to help with some of those optimizations if needed, as well.

oxpa · 2024-04-22T15:02:32Z

@arbourd Running too may tests is my main concern, really. I failed to find an easy way of pinning os-lang-version triplet while this will reduce the amount of jobs significantly. I'm happy to adopt any solution (maybe just hardcode it in?).
For now, I'm going to split the workflow into two parts, one per arch to avoid "no more than 256 jobs" limit.
We can also test multiple modules at once (say, go, php, java) on a single run. But it's more complicated with multiple versions of a single module (java-11, java-17, java-21). I'm happy to hear any proposals here as well.

thresheek · 2024-04-22T19:26:21Z

Is it a such a good idea to create a job for every "$OS-$ARCH-$LANG"? I think if we're to reuse the buildbot approach, we should be fine with "$OS-$ARCH" and then run steps for different $LANGs... this will save us quite some overhead in spinning up the machines.

ac000 · 2024-05-07T19:27:38Z

.github/workflows/ci-commit.yml

@@ -0,0 +1,345 @@
+name: ci


This needs a more descriptive name...

ac000 · 2024-05-07T19:28:50Z

.github/workflows/ci-commit.yml

+on:
+  workflow_dispatch:
+  push:
+    branches:
+      - master


I do sort of question this, as checking once things have hit master is a bit late...

We can't have self-hosted runners in a PR environment. Also, all of this should move to a separate repo with added protections on who can edit and call the CI steps on the workflow.

ac000 · 2024-05-07T19:31:32Z

.github/workflows/ci-selfhosted-or-smth-like-that.yml

Can we just name it correctly from the start?

ac000 · 2024-05-07T19:32:21Z

.github/workflows/ci.yml

I think this should be dropped. I.e this file should never have changed its name...

ac000 · 2024-05-07T19:34:04Z

.github/workflows/ci-selfhosted.yml

Again, can we name this correctly from the start?

thresheek · 2024-06-27T23:31:15Z

I've come up with a slightly (much) easier workflow, not too dissimilar from what we currently use for njs (https://github.com/nginx/njs/blob/master/.github/workflows/ci.yml and https://github.com/nginx/ci-self-hosted/blob/main/.github/workflows/njs-buildbot.yml).

I'll follow up with a PR for that soon-ish.

callahad · 2024-07-01T15:48:55Z

Thanks for the heads up; closing in lieu of the new approach.

oxpa added 3 commits April 19, 2024 22:45

Rename CI file and make it work for pull requests only

58e204e

Add package lists and a script to install them into workers

d4b218b

Replicate buildbot workflow for commits into master

acd515c

callahad reviewed Apr 19, 2024

View reviewed changes

.github/workflows/ci-pull.yml Outdated Show resolved Hide resolved

Re-enable GH based workflow for commit as well

b01c197

ac000 reviewed Apr 21, 2024

View reviewed changes

oxpa added 2 commits April 22, 2024 13:57

Update workflow file names

eeb8ea8

proper name for a ci file

b623a99

ac000 reviewed Apr 22, 2024

View reviewed changes

arbourd approved these changes Apr 22, 2024

View reviewed changes

ac000 reviewed May 7, 2024

View reviewed changes

.github/workflows/ci-commit.yml Outdated

@@ -0,0 +1,345 @@

name: ci

Copy link

Member

ac000 May 7, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This needs a more descriptive name...

ac000 reviewed May 7, 2024

View reviewed changes

.github/workflows/ci-selfhosted-or-smth-like-that.yml Outdated

Copy link

Member

ac000 May 7, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we just name it correctly from the start?

ac000 reviewed May 7, 2024

View reviewed changes

.github/workflows/ci.yml

Copy link

Member

ac000 May 7, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this should be dropped. I.e this file should never have changed its name...

ac000 reviewed May 7, 2024

View reviewed changes

.github/workflows/ci-selfhosted.yml

Copy link

Member

ac000 May 7, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Again, can we name this correctly from the start?

callahad closed this Jul 1, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Replicate (mostly) buildbot workflow for commits into master #1225

Replicate (mostly) buildbot workflow for commits into master #1225

oxpa commented Apr 19, 2024

ac000 Apr 21, 2024

callahad Apr 22, 2024

oxpa Apr 22, 2024

ac000 Apr 22, 2024

ac000 Apr 22, 2024

ac000 Apr 21, 2024

oxpa Apr 22, 2024

ac000 Apr 22, 2024

ac000 commented Apr 21, 2024

callahad commented Apr 22, 2024

oxpa commented Apr 22, 2024

callahad commented Apr 22, 2024

ac000 Apr 22, 2024

oxpa Apr 22, 2024

arbourd Apr 22, 2024 •

edited

Loading

ac000 Apr 22, 2024

ac000 May 7, 2024

ac000 commented Apr 22, 2024

arbourd left a comment

oxpa commented Apr 22, 2024

thresheek commented Apr 22, 2024

ac000 May 7, 2024

ac000 May 7, 2024

thresheek May 7, 2024

ac000 May 7, 2024

ac000 May 7, 2024

ac000 May 7, 2024

thresheek commented Jun 27, 2024

callahad commented Jul 1, 2024

		PACKAGE_MANAGER_INSTALL_yum="yum --setopt=skip_missing_names_on_install=False install -y"
		PACKAGE_MANAGER_REMOVE_yum="yum erase -y"

Replicate (mostly) buildbot workflow for commits into master #1225

Replicate (mostly) buildbot workflow for commits into master #1225

Conversation

oxpa commented Apr 19, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ac000 commented Apr 21, 2024

callahad commented Apr 22, 2024

oxpa commented Apr 22, 2024

callahad commented Apr 22, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

arbourd Apr 22, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ac000 commented Apr 22, 2024

arbourd left a comment

Choose a reason for hiding this comment

oxpa commented Apr 22, 2024

thresheek commented Apr 22, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

thresheek commented Jun 27, 2024

callahad commented Jul 1, 2024

arbourd Apr 22, 2024 •

edited

Loading