export itksnap singularity container: GUI fails

[dkp]

Problem:
Exported itksnap container won't run gui

Expectation:
itksnap container could run itksnap gui

Environment:
Ubuntu 22 with apptainer 1.2.5

Steps to recreate:
export container=itksnap_4.0.1_20230609
curl -X GET https://objectstorage.us-ashburn-1.oraclecloud.com/n/sd63xuke79z3/b/neurodesk/o/${container}.simg -O
singularity shell itksnap_4.0.1_20230609.simg

Singularity> which itksnap
/opt/itksnap-4.0.1/bin//itksnap
Singularity> itksnap
Launching ITK-SNAP
qt.qpa.plugin: Could not find the Qt platform plugin "wayland" in ""
This application failed to start because no Qt platform plugin could be initialized. Reinstalling the application may fix this problem.

Available platform plugins are: xcb.

Aborted (core dumped)

---

Note that command-line tools do seem to be available:
Singularity> c3d
PICSL convert3d tool - from the creators of ITK-SNAP
For full documentation and usage examples, see
http://www.itksnap.org/c3d
To get help on available commands, call
c3d -h

---

Any suggestions would be appreciated!

[stebo85]

Dear @dkp,

This seems to be related to your ubuntu version. It seems that you are running Wayland and this is not supported yet. It works on a Ubuntu 22 system without Wayland:

However, I can reproduce your problem when forcing wayland:

I see if I can somehow add Wayland support.

[stebo85]

Dear @dkp,

Can you try to run this before starting itksnap?

export QT_QPA_PLATFORM=xcb

does this help?

[dkp]

Like magic! That was perfect and it starts. Thank you!

[dema:~ () 2024] $ apptainer shell itksnap_4.0.1_20230609.simg
Singularity> export QT_QPA_PLATFORM=xcb
Singularity> itksnap
Launching ITK-SNAP
QStandardPaths: error creating runtime directory '/run/user/1001' (No such file or directory)

-Dianne

[dkp]

Although this worked perfectly on my local ubuntu machine, the signularity container fails on the HPC with a new message:
$ apptainer shell itksnap_4.0.1.sif
INFO: underlay of /etc/localtime required more than 50 (78) bind mounts
Singularity> itksnap
/opt/itksnap-4.0.1/lib/snap-4.0.1/ITK-SNAP: error while loading shared libraries: libQt6Core.so.6: cannot open shared object file: No such file or directory
Singularity> export QT_QPA_PLATFORM=xcb
Singularity> itksnap
/opt/itksnap-4.0.1/lib/snap-4.0.1/ITK-SNAP: error while loading shared libraries: libQt6Core.so.6: cannot open shared object file: No such file or directory
Singularity>

The HPC is centOS7. Any suggestions would be appreciated.

[stebo85]

Dear @dkp,

it sounds like your HPC sets an LD_LIBRARY_PATH. Can you start it the same way that neurocommand would install the container?

singularity exec --cleanenv --env DISPLAY=$DISPLAY  itksnap_4.0.1_20230609.simg itksnap

[dkp]

$ singularity exec --cleanenv DISPLAY=$DISPLAY itksnap_4.0.1.sif itksnap

FATAL:   While checking container encryption: could not open image /home/u7/dkp/DISPLAY=:1: failed to retrieve path for /home/u7/dkp/DISPLAY=:1: lstat /home/u7/dkp/DISPLAY=:1: no such file or directory
$ singularity exec --cleanenv itksnap_4.0.1.sif itksnap
INFO:    underlay of /etc/localtime required more than 50 (78) bind mounts
/opt/itksnap-4.0.1/lib/snap-4.0.1/ITK-SNAP: error while loading shared libraries: libQt6Core.so.6: cannot open shared object file: No such file or directory

and, since you mention it:

$ echo $LD_LIBRARY_PATH
/groups/dkp/neuroimaging/workbench/libs_rh_linux64_software_opengl:/groups/dkp/neuroimaging/workbench/libs_rh_linux64_software_opengl

[stebo85]

There is a typo in your command. Can yo please try:

$ singularity exec --cleanenv --env DISPLAY=$DISPLAY itksnap_4.0.1.sif itksnap

[dkp]

Sorry about that:

$ singularity exec --cleanenv --env DISPLAY=$DISPLAY itksnap_4.0.1.sif itksnap

INFO:    underlay of /etc/localtime required more than 50 (78) bind mounts
/opt/itksnap-4.0.1/lib/snap-4.0.1/ITK-SNAP: error while loading shared libraries: libQt6Core.so.6: cannot open shared object file: No such file or directory

[stebo85]

@Vbitz

https://github.com/NeuroDesk/neurocontainers/blob/master/recipes/itksnap/build.sh

[stebo85]

Dear @dkp,

I verified that libQt6Core.so.6 is in the neurodesk itksnap image and it is available there and working for me on my test systems. Could it be that you didn't download our itksnap image but got it from somewhere else? The reason I am asking is: our images do not end on .sif and they contain a date version in the filename. Maybe you renamed it, but currently it doesn't make 100% sense to me.

I rebuild ITK snap for you in a new version which also contains the fix from earlier on in this issue. Can you please try exactly that image and see if everything works for you there on both your systems?

curl -X GET https://objectstorage.us-ashburn-1.oraclecloud.com/n/sd63xuke79z3/b/neurodesk/o/temporary-builds/itksnap_4.0.2_20240108.simg -O
singularity shell itksnap_4.0.2_20240108.simg 
itksnap

Thank you
Steffen

[dkp]

I DID rename your image to shorten the name and change the extension to sif. But I tried it with the old name to ensure I hadn't introduced a problem (sorry I didn't explain).

$ curl -X GET https://objectstorage.us-ashburn-1.oraclecloud.com/n/sd63xuke79z3/b/neurodesk/o/temporary-builds/itksnap_4.0.2_20240108.simg -O
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  300M  100  300M    0     0  44.3M      0  0:00:06  0:00:06 --:--:-- 53.3M
$ singularity shell itksnap_4.0.2_20240108.simg 
INFO:    underlay of /etc/localtime required more than 50 (78) bind mounts
Singularity> itksnap
/opt/itksnap-4.0.2/lib/snap-4.0.2/ITK-SNAP: error while loading shared libraries: libQt6Core.so.6: cannot open shared object file: No such file or directory

😟

Thanks so much for working on it.

[stebo85]

Dear @dkp,

That's very strange then :( So far I haven't been able to reproduce this problem and it doesn't make sense yet. The library is there and should be found.

Which version of singularity or apptainer are you using?

Thank you
Steffen

[dkp]

The HPC reports the following:

$ which apptainer
/usr/bin/apptainer
$ apptainer --version
apptainer version 1.2.4-1.el7

I tried this, but it didn't help

$ singularity shell --cleanenv itksnap_4.0.2_20240108.simg 
INFO:    underlay of /etc/localtime required more than 50 (78) bind mounts
Singularity> itksnap
/opt/itksnap-4.0.2/lib/snap-4.0.2/ITK-SNAP: error while loading shared libraries: libQt6Core.so.6: cannot open shared object file: No such file or directory
Singularity> 

And this wasn't very informative:

Singularity> whereis libQt6Core.so.6
libQt6Core.so.6:
Singularity> which libQt6Core.so.6
Singularity> 

[stebo85]

I tested it with 1.2.3-1.el8 and can't reproduce the behaviour. Not sure what else it could be.

Would you have time for a quick interactive debug session?

[dkp]

Absolutely, I have the HPC desktop open now.

[stebo85]

https://uqz.zoom.us/j/84452277285

[stebo85]

conclusions for now:

• excluded any environment problems with cleanall, containall, no-home
• /opt/itksnap-4.0.2/bin//itksnap calls /opt/itksnap-4.0.2/lib/snap-4.0.2/ITK-SNAP, which then has the library problem. When directly calling it, it gives me that error:

/opt/itksnap-4.0.2/lib/snap-4.0.2/ITK-SNAP: error while loading shared libraries: libQt6OpenGLWidgets.so.6: cannot open shared object file: No such file or directory

[stebo85]

when I set

export LD_LIBRARY_PATH=/opt/itksnap-4.0.2/lib/snap-4.0.2

and then start /opt/itksnap-4.0.2/lib/snap-4.0.2/ITK-SNAP it works for me, but not for @dkp and I can also reproduce this problem now under centos 7, but it works perfectly fine in centos 8.

[dkp]

Here's what I see. I'm glad you are identifying the culprit here!

singularity shell --cleanenv itksnap_4.0.2_20240108.simg 
INFO:    underlay of /etc/localtime required more than 50 (78) bind mounts

Singularity> export LD_LIBRARY_PATH=/opt/itksnap-4.0.2/lib/snap-4.0.2

Singularity> itksnap
/opt/itksnap-4.0.2/lib/snap-4.0.2/ITK-SNAP: error while loading shared libraries: libQt6Core.so.6: cannot open shared object file: No such file or directory

[stebo85]

This seems to be the problem and solution: apptainer/apptainer#1698

[dkp]

Fascinating! Thank you!

Our HPC apptainer version is:

$ apptainer --version
apptainer version 1.2.4-1.el7

It sounds like our HPC needs to update apptainer...true? It isn't clear to me that a revised apptainer version is available. Thanks for your incredible sleuthing!

[stebo85]

@dkp - itksnap has to be build differently- there shouldn’t be a change necessary on your side. We will build a new container and I let you know when it’s ready for testing .

[stebo85]

Dear @dkp,

We haven't found a way of getting ITK snap 4.x to work on Centos 7. The problem is that the QT library is incompatible with the old kernel. This could theoretically be fixed through a compiler setting for this library, but so far, we haven't been successful.

Would the following workaround work for you? Could you use ITK snap 3.8? This version uses the older version of the QT library and still works in Centos 7:

curl -X GET https://objectstorage.us-ashburn-1.oraclecloud.com/n/sd63xuke79z3/b/neurodesk/o/itksnap_3.8.0_20201208.simg -O

Than you
Steffen

[dkp]

Thanks so much for all your work on this! I've learned a lot. It looks like itksnap 3.8 works fine:
$ apptainer shell itksnap_3.8.0_20201208.simg
Singularity> itksnap
(the gui starts)

I really appreciate your time and this project!

[stebo85]

Ok, great. Then I close this issue here for now. Should modern versions of itksnap work in any way on that old kernel I will post an update here.

[stebo85]

@dkp ,

I might have given up a few hours too early :)

Can you try this version?

curl -X GET https://objectstorage.us-ashburn-1.oraclecloud.com/n/sd63xuke79z3/b/neurodesk/o/temporary-builds-new/itksnap_4.0.2_20240117.simg -O

Here, I applied a fix to the libraries that works on my testsystem, but I am unsure if this breaks anything else:

find /opt/itksnap-4.0.2/ -name '*.so.*' | xargs strip --remove-section=.note.ABI-tag

[dkp]

Whoa, that is SO nice! I was able to launch it in two different ways:

$ singularity shell itksnap_4.0.2_20240117.simg 
INFO:    underlay of /etc/localtime required more than 50 (81) bind mounts
Apptainer> itksnap
Launching ITK-SNAP
QStandardPaths: XDG_RUNTIME_DIR not set, defaulting to '/tmp/runtime-dkp'
Return code : 0
Apptainer> c3d
PICSL convert3d tool - from the creators of ITK-SNAP 
For full documentation and usage examples, see
    http://www.itksnap.org/c3d
To get help on available commands, call
    c3d -h
Apptainer> exit
exit

and

$ singularity run itksnap_4.0.2_20240117.simg itksnap
INFO:    underlay of /etc/localtime required more than 50 (81) bind mounts
Launching ITK-SNAP
QStandardPaths: XDG_RUNTIME_DIR not set, defaulting to '/tmp/runtime-dkp'
Return code : 0

Not only that, but some weird text overlay problem from 3.8 no longer appears AND I can get access to the command line tools, like c3d from this new container (I could not find them in the 3.8 container).

I'm thrilled! This is absolutely perfect!

[stebo85]

Wonderful to hear @dkp - thank you for confirming that this works! I will release this version then to all people and consider this solved for now :)