moon_app::systems::analyze Installing proto

[nhomble]

I am not sure if this an error yet, so I wanted to start a discussion first.

---

In my pipelines and local (with docker-compose), I frequently see errors after proto tool installation. The installation seems to happen as a side effect of running any of my tasks. When running with --log trace I see

#16 38.91           example:task | [DEBUG 19:03:10.221] moon_app::systems::analyze  Checking if proto is installed  proto="/root/.proto/tools/proto/0.44.1/proto"
#16 38.91           example:task | [DEBUG 19:03:10.221] moon_app::systems::analyze  Installing proto 
#16 38.91           example:task | [TRACE 19:03:10.221] starbase_utils::net  Checking for an internet connection  timeout=750
#16 38.91           example:task | [TRACE 19:03:10.221] starbase_utils::net::offline  Resolving 1.1.1.1:53 
#16 38.91           example:task | [TRACE 19:03:10.224] starbase_utils::net  Online! 
#16 38.91           example:task | [DEBUG 19:03:10.225] moon_app::systems::analyze  Downloading proto archive (aarch64-unknown-linux-gnu) 
#16 38.91           example:task | [TRACE 19:03:10.225] proto_installer  Downloading proto release from https://github.com/moonrepo/proto/releases/download/v0.44.1/proto_cli-aarch64-unknown-linux-gnu.tar.xz  version="0.44.1" triple="aarch64-unknown-linux-gnu"
#16 38.91           example:task | [TRACE 19:03:10.233] starbase_utils::net  Downloading file from remote URL to local file  source_url="https://github.com/moonrepo/proto/releases/download/v0.44.1/proto_cli-aarch64-unknown-linux-gnu.tar.xz" dest_file="/root/.proto/temp/proto_cli-aarch64-unknown-linux-gnu.tar.xz"
#16 38.91           example:task | [TRACE 19:03:10.548] starbase_utils::fs  Creating file  file="/root/.proto/temp/proto_cli-aarch64-unknown-linux-gnu.tar.xz"
#16 38.91           example:task | [DEBUG 19:03:13.086] moon_app::systems::analyze  Unpacking archive and installing proto 
#16 38.91           example:task | [TRACE 19:03:13.086] proto_installer  Unpacking downloaded and installing proto release  source="/root/.proto/temp/proto_cli-aarch64-unknown-linux-gnu.tar.xz" target="/root/.proto/temp/proto_cli-aarch64-unknown-linux-gnu"
#16 38.91           example:task | [TRACE 19:03:13.086] starbase_archive::archive  Unpacking archive  output_dir="/root/.proto/temp/proto_cli-aarch64-unknown-linux-gnu" input_file="/root/.proto/temp/proto_cli-aarch64-unknown-linux-gnu.tar.xz"
#16 38.91           example:task | [TRACE 19:03:13.086] starbase_archive::tree_differ  Creating a file tree differ for destination directory  dir="/root/.proto/temp/proto_cli-aarch64-unknown-linux-gnu"
#16 38.91           example:task | [TRACE 19:03:13.086] starbase_utils::fs  Opening file  file="/root/.proto/temp/proto_cli-aarch64-unknown-linux-gnu.tar.xz"
#16 38.91           example:task | [TRACE 19:03:13.086] starbase::app  Running shutdown phase (because another phase failed) 
#16 38.91           example:task | [TRACE 19:03:13.086] starbase::app  Error: Failed to read path ~/.proto/temp/proto_cli-aarch64-unknown-linux-gnu.tar.xz. 
#16 38.91           example:task | [TRACE 19:03:13.086] moon_console::console  Closing console and flushing buffered output 
#16 38.91           example:task | [TRACE 19:03:13.086] moon_console::buffer  Closing stderr stream 
#16 38.91           example:task | [TRACE 19:03:13.086] moon_console::buffer  Closing stdout stream 
#16 38.91           example:task | [TRACE 19:03:13.153] starbase::app  Setting exit code  code=1
#16 38.91           example:task | Error: fs::read
#16 38.91           example:task | 
#16 38.91           example:task |   × Failed to read path ~/.proto/temp/proto_cli-aarch64-unknown-linux-
#16 38.91           example:task |   │ gnu.tar.xz.
#16 38.91           example:task |   ╰─▶ No such file or directory (os error 2)
#16 38.91           example:task | 

Ultimately, the inability to read the proto_cli file seems to kill the task, but it's seems weird that I see multiple tasks do this download. Specifically I see:

#16 38.91           project1:task | [DEBUG 19:03:10.221] moon_app::systems::analyze  Installing proto 
#16 40.93           project2:task | [DEBUG 19:03:11.347] moon_app::systems::analyze  Installing proto 
#16 40.93           project3:task | [DEBUG 19:03:11.114] moon_app::systems::analyze  Installing proto 

If it helps, part of the execution of task is to query project metadata via moon project <id>. I wanted to try installing before executing this task across the projects, but moon setup doesn't seem to install proto.

[milesj]

@nhomble What's the full command you are running, is it piping a bunch of moon commands? Since pipes run in parallel, it may be a race condition.

[milesj]

Also, I'm guessing you're on like version 1.30.x? We have a race condition fix in in 1.31.2.

[nhomble]

I'm guessing you're on like version 1.30.x?

yep we're on 1.30.6

What's the full command you are running

:task delegates to a shell script that executes

moonPath=$(moon project "$moonId" --json | jq -r ".root")

the use case is that outside of our normal build (typescript) we are doing some processing on other known file (paths) in a moon project

[milesj]

Yeah give 1.31.2 a try, since that uses file locks to avoid race conditions, which should help for these multi-process collisions.

[nhomble]

nice yeah a file lock makes sense! I'll give it a shot and report back here.

[nhomble]

I haven't seen the issue since upgrading!

[milesj]

Awesome!