aarch64 stable repo only indexes latest RKE2 version, unlike x86_64

[keegandahm]

The x86_64 version of the rancher-rke2-1.28-stable repo indexes all versions of RKE2 for that channel, while the aarch64 repo only indexes the latest version.

Node(s) CPU architecture, OS, and Version:
Linux <hostname> 4.18.0-553.16.1.el8_10.aarch64 #1 SMP Thu Aug 1 08:17:57 UTC 2024 aarch64 aarch64 aarch64 GNU/Linux

Cluster Configuration:
Prior to installing RKE2

Describe the bug:
The aarch64 version of the stable repos (https://rpm.rancher.io/rke2/stable/<channel>/centos/8/<arch>) only indexes the latest release of RKE2 for a given channel. Meanwhile, the x86_64 repo indexes all releases of that channel.

Steps To Reproduce:

• Configure stable RKE2 repos on aarch64
• Try to install rke2-server v1.28.14, e.g. dnf install rke2-agent-1.28.14~rke2r1-0.el8.aarch64

Expected behavior:
RKE2 1.28.14 installs

Actual behavior:
Only version 1.28.15~rke2r1-0.el8 is available on aarch64

Additional context / logs:
The root problem is that repodata/<hash>-primary.xml.gz doesn't include the full set of releases on aarch64.

This can be most easily seen by listing packages with --showduplicates, e.g. dnf list rke2-server --showduplicates

On aarch64:

Last metadata expiration check: 0:00:22 ago on Tue 05 Nov 2024 12:44:39 AM UTC.
Available Packages
rke2-server.aarch64           1.28.15~rke2r1-0.el8           rancher-rke2-stable

On x86_64:

# dnf list rke2-server --showduplicates
Last metadata expiration check: 0:05:34 ago on Tue 05 Nov 2024 12:38:24 AM UTC.
Available Packages
rke2-server.x86_64           1.28.2~rke2r1-0.el8             rancher-rke2-stable
rke2-server.x86_64           1.28.3~rke2r1-0.el8             rancher-rke2-stable
rke2-server.x86_64           1.28.3~rke2r2-0.el8             rancher-rke2-stable
rke2-server.x86_64           1.28.4~rke2r1-0.el8             rancher-rke2-stable
rke2-server.x86_64           1.28.5~rke2r1-0.el8             rancher-rke2-stable
rke2-server.x86_64           1.28.6~rke2r1-0.el8             rancher-rke2-stable
rke2-server.x86_64           1.28.7~rke2r1-0.el8             rancher-rke2-stable
rke2-server.x86_64           1.28.8~rke2r1-0.el8             rancher-rke2-stable
rke2-server.x86_64           1.28.9~rke2r1-0.el8             rancher-rke2-stable
rke2-server.x86_64           1.28.10~rke2r1-0.el8            rancher-rke2-stable
rke2-server.x86_64           1.28.11~rke2r1-0.el8            rancher-rke2-stable
rke2-server.x86_64           1.28.12~rke2r1-0.el8            rancher-rke2-stable
rke2-server.x86_64           1.28.13~rke2r1-0.el8            rancher-rke2-stable
rke2-server.x86_64           1.28.14~rke2r1-0.el8            rancher-rke2-stable
rke2-server.x86_64           1.28.15~rke2r1-0.el8            rancher-rke2-stable

[davidgaster]

@brandond is someone tracking this issue? the same issue occurs on 1.30 -- the RPM for 1.30.5 have been deleted out from under us, and only 1.30.6 are available in the aarch64 repositories. this isn't the case for x86_64

[brandond]

the RPM for 1.30.5 have been deleted out from under us

What are you talking about? Nothing is ever deleted from the yum repo nor from the artifacts on GH.

[davidgaster]

the RPM for 1.30.5 have been deleted out from under us

What are you talking about? Nothing is ever deleted from the yum repo nor from the artifacts on GH.

Did you read the original post? We previously downloaded and installed 1.28.14. When 1.28.15 was released, the 1.28.14 RPMs were no longer indexed in the repo and we could not download them. Only 1.28.15 was available. We have the 1.28.14 rpms from a previous install to prove it. The same has happened with 1.30.5.

[brandond]

They're not deleted. They're just not in the repo metadata. You can still get them from the repo directly by filename, or from the releases page, as a workaround until we figure out what is going on with the s3 repo sync.

[brandond]

I believe this has something to do with rpm-s3 not working properly in a cross-arch environment. Unfortunately this tool is essentially unmaintained.
https://github.com/rancher/rke2-packaging/blob/master/rpm/centos7/scripts/upload-repo#L21

We should probably replace it with something else.