Docker builds failing for arm64 architecture since v1.50.7

[kayvanaarssen]

Here to report an issue with the latest docker tag (build)
See the error below, reverted back to tag 1.50 and no issues :-D

pwpush-1  | bundler: failed to load command: rake (/usr/local/bin/rake)
pwpush-1  | /usr/local/lib/ruby/3.4.0/bundler/source/git.rb:233:in 'Bundler::Source::Git#load_spec_files': https://github.com/pglombardo/version.git (at master@97678e0) is not yet checked out. Run `bundle install` first. (Bundler::GitError)
pwpush-1  |     from /usr/local/lib/ruby/3.4.0/bundler/source/path.rb:97:in 'Bundler::Source::Path#local_specs'
pwpush-1  |     from /usr/local/lib/ruby/3.4.0/bundler/source/git.rb:198:in 'Bundler::Source::Git#specs'
pwpush-1  |     from /usr/local/lib/ruby/3.4.0/bundler/lazy_specification.rb:145:in 'Bundler::LazySpecification#materialize_for_installation'
pwpush-1  |     from /usr/local/lib/ruby/3.4.0/bundler/lazy_specification.rb:134:in 'Bundler::LazySpecification#materialized_for_installation'
pwpush-1  |     from /usr/local/lib/ruby/3.4.0/bundler/gem_helpers.rb:73:in 'Array#each'
pwpush-1  |     from /usr/local/lib/ruby/3.4.0/bundler/gem_helpers.rb:73:in 'Enumerable#filter_map'
pwpush-1  |     from /usr/local/lib/ruby/3.4.0/bundler/gem_helpers.rb:73:in 'Bundler::GemHelpers.select_best_local_platform_match'
pwpush-1  |     from /usr/local/lib/ruby/3.4.0/bundler/materialization.rb:27:in 'Bundler::Materialization#specs'
pwpush-1  |     from /usr/local/lib/ruby/3.4.0/bundler/materialization.rb:18:in 'Bundler::Materialization#complete?'
pwpush-1  |     from /usr/local/lib/ruby/3.4.0/bundler/spec_set.rb:228:in 'block in Bundler::SpecSet#materialize_dependencies'
pwpush-1  |     from <internal:kernel>:168:in 'Kernel#loop'
pwpush-1  |     from /usr/local/lib/ruby/3.4.0/bundler/spec_set.rb:214:in 'Bundler::SpecSet#materialize_dependencies'
pwpush-1  |     from /usr/local/lib/ruby/3.4.0/bundler/spec_set.rb:110:in 'Bundler::SpecSet#materialize'
pwpush-1  |     from /usr/local/lib/ruby/3.4.0/bundler/definition.rb:640:in 'Bundler::Definition#materialize'
pwpush-1  |     from /usr/local/lib/ruby/3.4.0/bundler/definition.rb:232:in 'Bundler::Definition#specs'
pwpush-1  |     from /usr/local/lib/ruby/3.4.0/bundler/definition.rb:299:in 'Bundler::Definition#specs_for'
pwpush-1  |     from /usr/local/lib/ruby/3.4.0/bundler/runtime.rb:18:in 'Bundler::Runtime#setup'
pwpush-1  |     from /usr/local/lib/ruby/3.4.0/bundler.rb:167:in 'Bundler.setup'
pwpush-1  |     from /usr/local/lib/ruby/3.4.0/bundler/setup.rb:32:in 'block in <top (required)>'
pwpush-1  |     from /usr/local/lib/ruby/3.4.0/bundler/ui/shell.rb:173:in 'Bundler::UI::Shell#with_level'
pwpush-1  |     from /usr/local/lib/ruby/3.4.0/bundler/ui/shell.rb:119:in 'Bundler::UI::Shell#silence'
pwpush-1  |     from /usr/local/lib/ruby/3.4.0/bundler/setup.rb:32:in '<top (required)>'
pwpush-1  |     from /usr/local/lib/ruby/3.4.0/bundler/cli/exec.rb:57:in 'Kernel#require_relative'
pwpush-1  |     from /usr/local/lib/ruby/3.4.0/bundler/cli/exec.rb:57:in 'Bundler::CLI::Exec#kernel_load'
pwpush-1  |     from /usr/local/lib/ruby/3.4.0/bundler/cli/exec.rb:23:in 'Bundler::CLI::Exec#run'
pwpush-1  |     from /usr/local/lib/ruby/3.4.0/bundler/cli.rb:452:in 'Bundler::CLI#exec'
pwpush-1  |     from /usr/local/lib/ruby/3.4.0/bundler/vendor/thor/lib/thor/command.rb:28:in 'Bundler::Thor::Command#run'
pwpush-1  |     from /usr/local/lib/ruby/3.4.0/bundler/vendor/thor/lib/thor/invocation.rb:127:in 'Bundler::Thor::Invocation#invoke_command'
pwpush-1  |     from /usr/local/lib/ruby/3.4.0/bundler/vendor/thor/lib/thor.rb:538:in 'Bundler::Thor.dispatch'
pwpush-1  |     from /usr/local/lib/ruby/3.4.0/bundler/cli.rb:35:in 'Bundler::CLI.dispatch'
pwpush-1  |     from /usr/local/lib/ruby/3.4.0/bundler/vendor/thor/lib/thor/base.rb:584:in 'Bundler::Thor::Base::ClassMethods#start'
pwpush-1  |     from /usr/local/lib/ruby/3.4.0/bundler/cli.rb:29:in 'Bundler::CLI.start'
pwpush-1  |     from /usr/local/lib/ruby/gems/3.4.0/gems/bundler-2.6.2/exe/bundle:28:in 'block in <top (required)>'
pwpush-1  |     from /usr/local/lib/ruby/3.4.0/bundler/friendly_errors.rb:117:in 'Bundler.with_friendly_errors'
pwpush-1  |     from /usr/local/lib/ruby/gems/3.4.0/gems/bundler-2.6.2/exe/bundle:20:in '<top (required)>'
pwpush-1  |     from /usr/local/bin/bundle:25:in 'Kernel#load'
pwpush-1  |     from /usr/local/bin/bundle:25:in '<main>'
pwpush-1  | /usr/local/lib/ruby/3.4.0/bundler/source/path.rb:199:in 'Bundler::Source::Path#load_spec_files': The path `/usr/local/bundle/bundler/gems/version-97678e006854` does not exist. (Bundler::PathError)
pwpush-1  |     from /usr/local/lib/ruby/3.4.0/bundler/source/git.rb:230:in 'Bundler::Source::Git#load_spec_files'
pwpush-1  |     from /usr/local/lib/ruby/3.4.0/bundler/source/path.rb:97:in 'Bundler::Source::Path#local_specs'
pwpush-1  |     from /usr/local/lib/ruby/3.4.0/bundler/source/git.rb:198:in 'Bundler::Source::Git#specs'
pwpush-1  |     from /usr/local/lib/ruby/3.4.0/bundler/lazy_specification.rb:145:in 'Bundler::LazySpecification#materialize_for_installation'
pwpush-1  |     from /usr/local/lib/ruby/3.4.0/bundler/lazy_specification.rb:134:in 'Bundler::LazySpecification#materialized_for_installation'
pwpush-1  |     from /usr/local/lib/ruby/3.4.0/bundler/gem_helpers.rb:73:in 'Array#each'
pwpush-1  |     from /usr/local/lib/ruby/3.4.0/bundler/gem_helpers.rb:73:in 'Enumerable#filter_map'
pwpush-1  |     from /usr/local/lib/ruby/3.4.0/bundler/gem_helpers.rb:73:in 'Bundler::GemHelpers.select_best_local_platform_match'
pwpush-1  |     from /usr/local/lib/ruby/3.4.0/bundler/materialization.rb:27:in 'Bundler::Materialization#specs'
pwpush-1  |     from /usr/local/lib/ruby/3.4.0/bundler/materialization.rb:18:in 'Bundler::Materialization#complete?'
pwpush-1  |     from /usr/local/lib/ruby/3.4.0/bundler/spec_set.rb:228:in 'block in Bundler::SpecSet#materialize_dependencies'
pwpush-1  |     from <internal:kernel>:168:in 'Kernel#loop'
pwpush-1  |     from /usr/local/lib/ruby/3.4.0/bundler/spec_set.rb:214:in 'Bundler::SpecSet#materialize_dependencies'
pwpush-1  |     from /usr/local/lib/ruby/3.4.0/bundler/spec_set.rb:110:in 'Bundler::SpecSet#materialize'
pwpush-1  |     from /usr/local/lib/ruby/3.4.0/bundler/definition.rb:640:in 'Bundler::Definition#materialize'
pwpush-1  |     from /usr/local/lib/ruby/3.4.0/bundler/definition.rb:232:in 'Bundler::Definition#specs'
pwpush-1  |     from /usr/local/lib/ruby/3.4.0/bundler/definition.rb:299:in 'Bundler::Definition#specs_for'
pwpush-1  |     from /usr/local/lib/ruby/3.4.0/bundler/runtime.rb:18:in 'Bundler::Runtime#setup'
pwpush-1  |     from /usr/local/lib/ruby/3.4.0/bundler.rb:167:in 'Bundler.setup'
pwpush-1  |     from /usr/local/lib/ruby/3.4.0/bundler/setup.rb:32:in 'block in <top (required)>'
pwpush-1  |     from /usr/local/lib/ruby/3.4.0/bundler/ui/shell.rb:173:in 'Bundler::UI::Shell#with_level'
pwpush-1  |     from /usr/local/lib/ruby/3.4.0/bundler/ui/shell.rb:119:in 'Bundler::UI::Shell#silence'
pwpush-1  |     from /usr/local/lib/ruby/3.4.0/bundler/setup.rb:32:in '<top (required)>'
pwpush-1  |     from /usr/local/lib/ruby/3.4.0/bundler/cli/exec.rb:57:in 'Kernel#require_relative'
pwpush-1  |     from /usr/local/lib/ruby/3.4.0/bundler/cli/exec.rb:57:in 'Bundler::CLI::Exec#kernel_load'
pwpush-1  |     from /usr/local/lib/ruby/3.4.0/bundler/cli/exec.rb:23:in 'Bundler::CLI::Exec#run'
pwpush-1  |     from /usr/local/lib/ruby/3.4.0/bundler/cli.rb:452:in 'Bundler::CLI#exec'
pwpush-1  |     from /usr/local/lib/ruby/3.4.0/bundler/vendor/thor/lib/thor/command.rb:28:in 'Bundler::Thor::Command#run'
pwpush-1  |     from /usr/local/lib/ruby/3.4.0/bundler/vendor/thor/lib/thor/invocation.rb:127:in 'Bundler::Thor::Invocation#invoke_command'
pwpush-1  |     from /usr/local/lib/ruby/3.4.0/bundler/vendor/thor/lib/thor.rb:538:in 'Bundler::Thor.dispatch'
pwpush-1  |     from /usr/local/lib/ruby/3.4.0/bundler/cli.rb:35:in 'Bundler::CLI.dispatch'
pwpush-1  |     from /usr/local/lib/ruby/3.4.0/bundler/vendor/thor/lib/thor/base.rb:584:in 'Bundler::Thor::Base::ClassMethods#start'
pwpush-1  |     from /usr/local/lib/ruby/3.4.0/bundler/cli.rb:29:in 'Bundler::CLI.start'
pwpush-1  |     from /usr/local/lib/ruby/gems/3.4.0/gems/bundler-2.6.2/exe/bundle:28:in 'block in <top (required)>'
pwpush-1  |     from /usr/local/lib/ruby/3.4.0/bundler/friendly_errors.rb:117:in 'Bundler.with_friendly_errors'
pwpush-1  |     from /usr/local/lib/ruby/gems/3.4.0/gems/bundler-2.6.2/exe/bundle:20:in '<top (required)>'
pwpush-1  |     from /usr/local/bin/bundle:25:in 'Kernel#load'
pwpush-1  |     from /usr/local/bin/bundle:25:in '<main>'
pwpush-1 exited with code 1

[gadgetusaf]

I am seeing the same error

[graphixillusion]

yep i confirm. reverted to 1.50.6

[rdlmda]

Same issue on latest. Reverting to 1.50.6 works.

[pglombardo]

Damn - thanks for reporting. I'll fix today.

[pglombardo]

Fix in release v1.50.7. The fix works in a locally built container. I'll test again on the v1.50.7 container to confirm once it finishes building:

https://github.com/pglombardo/PasswordPusher/actions/runs/13098711932

[ivancarlosti]

The build failed :/

For now I am moving to

    image: docker.io/pglombardo/pwpush:stable
    image: docker.io/pglombardo/pwpush-worker:stable

[pglombardo]

I have another attempt at a fix building now. It passed in #3038/build.

@ivancarlosti is right - anyone who wishes to avoid issues like these that come up occasionally, use the stable tag. No need for you all to suffer my pain trying to resolve. :-)

I have some docs explaining each of the Docker tags here.

[kayvanaarssen]

I have another attempt at a fix building now. It passed in #3038/build.

@ivancarlosti is right - anyone who wishes to avoid issues like these that come up occasionally, use the stable tag. No need for you all to suffer my pain trying to resolve. :-)

I have some docs explaining each of the Docker tags here.

Will stay on the latest branch for our own use. So we can report issues and make the product even better! Can't wait for the self hosted pro version to come live. So we can get the request function💪

For our clients we indeed use the stable tag.

And we have close monitoring on it for uptime so if our Watchtower updates the docker container. We know when something is up. Like we did now. 👌

@pglombardo Keep up the good work!!

[pglombardo]

Thanks @kayvanaarssen you rock. And self-hosted Pro is my top priority now. (after this issue)

Just an update. The builds are getting random segfaults while compiling code. It seems an underlying library/image has introduced some issues. I'm trying some work-arounds...

[pglombardo]

After fixing the binfmt issues, bootsnap had other issues.

All of these problems were rooted in the QEMU cross arch build of arm64 so tldr; I've temporarily disabled arm64 Docker builds entirely. amd64 still builds without issue.

Details in release v1.50.11 (amd64 arch only).

Thanks all for the help.

I'll leave this issue open for when I re-introduce arm64 (once those bugs are fixed).

[pooley182]

On a couple of my own private projects I've managed to work around this bug in qemu by building arm64 and amd64 on their native architectures (now that github has ubuntu-arm runners available), then pushing a single manifest file to dockerhub that contains both architectures.

If you're happy to accept PR's for this issue I could probably put something together for you.

Nevermind, despite getting a proof of concept working doing builds on their native architecture, it seems there's an issue with using the public arm64 images as part of a matrix build as they'll also randomly fail with errors. It's proving just as unreliable to get consistent builds, so it's probably best to leave arm builds disabled for now.

[pglombardo]

Thanks for the offer in any case @pooley182. It's very much appreciated. I saw that option when I was working on this and I think I read that those arm64 images are in preview or something.

It's proving just as unreliable to get consistent builds, so it's probably best to leave arm builds disabled for now.

This is good to know.

I'll give it a month and then maybe we can just flip the switch back...