CPack: Refactor AppImage and Apple DMG Generation #7252

tresf · 2024-05-12T01:17:21Z

Features:

Simplifies Mac and Linux packaging command to simply: make package

- make install && make appimage
- make install && make dmg
+ make package
# -- AppImage create: /__w/lmms/lmms/build/lmms-1.3.0-alpha.1.638+pr7252.gaaa9b4816-linux-x86_64.AppImage

Adds new WANT_DEBUG_CPACK flag to easily show detailed console messages about packaging

cmake .. -DWANT_DEBUG_CPACK=true

  Developer options
  -----------------------------------------
  * Debug FP exceptions               : Disabled
  * Debug using AddressSanitizer      : Disabled
  * Debug using ThreadSanitizer       : Disabled
  * Debug using MemorySanitizer       : Disabled
  * Debug using UBSanitizer           : Disabled
+ * Debug packaging commands          : Enabled

Preserves old CPack TGZ behavior through a hidden flag, WANT_CPACK_TARBALL. Does anyone use this?

Adds new CPACK_TOOL parameter to switch between .AppImage (default) and .run (experimental)

export CPACK_TOOL=makeself
make package
# -- Installer create: /__w/lmms/lmms/build/lmms-1.3.0-alpha.1.638+pr7252.gaaa9b4816-linux-x86_64.run

TODO:

Fix Carla crashing
Investigate possibly using the provided Mac DMG generators.
- Port the install_apple.sh script to CMake.
  It's called MacDeployQt.cmake now.
Investigate possibility using CPack for creating Linux AppImages
- Port the package_linux.sh script to Cmake
  It's called LinuxDeploy.cmake now.
Upgrade CMake for Linux runners

BUGS:

Fix background missing for DMG
Fix AppImages not building for Linux
WANT_DEBUG_CPACK is incorrectly toggled in CI. https://github.com/LMMS/lmms/actions/runs/9131345271/job/25110338288. Closed via 82a3aa6.

STRETCH GOALS:

Port project from linuxdeployqt to linuxdeploy
Examples:
- pbek/QOwnNotes (thank you!)
- wireshark/wireshark@34ce99f (thank you!)
Remove all references to linuxdeployqt (in favor of linuxdeploy)
~~[ ] Add ARM64 AppImage~~
- Can linuxdeploy support a cross-compilation environment ? linuxdeploy/linuxdeploy#258
- On hold until GitHub actions Linux ARM64 runners are released to the general public. https://github.com/orgs/community/discussions/73040#discussioncomment-9661482

Background:

Click to expand

The AppImage and Apple DMG installers historically (confusingly) require make install to be run prior to making packages. This PR is a proposal to remove that by switching to CPack.
- Utilizes make package now, just like Windows does
- Mac utilizes CPACK_GENERATOR of ~~DragNDrop~~ Bundle which will create a DMG without the need for node's appdmg package (still requires macdeployqt)
- Linux will utilize CPACK_GENERATOR of "External", which will use custom CMake commands to replace package_linux.sh (still requires linuxdeployqt).

tresf · 2024-05-12T04:37:58Z

Rule out possibly using the provided Mac DMG generators.

I was able to get the CPack DMG generator to work with macdeployqt however in order for us to brand our own DMG, we either need to leverage AppleScript (which is much slower and requires user interaction over appdmg), or we need to provide a premade .DS_Store file to bundle.

Edit: For simplicity, I've checked in the .DS_Store from our current DMG. Since it will be unobvious how to update this file in the event of a rebranding, I've kept basic support for appdmg for recreating it, although it's not used or needed unless we rebrand the installer.

tresf · 2024-05-14T03:35:15Z

Hmm... the AppImage is missing... investigating...

tresf · 2024-05-14T13:41:09Z

The AppImage is missing because the CPack feature for running custom scripts through variable CPACK_PRE_BUILD_SCRIPTS was not available in Ubuntu 20.04. There doesn't seem to be a replacement for this feature, which is alarming, I wonder how people used "External" packaging techniques prior to CMake 3.19. Hmm...

midi-pascal · 2024-05-14T14:03:53Z

Perhaps I have a hint for this:

I use Ubuntu 20.04 that comes with cmake version 3.16 BUT Qt (5.15.2 installed from Qt site) comes with cmake 3.27 in its tools directory. This is the one I use.

tresf · 2024-05-14T20:36:01Z

I use Ubuntu 20.04 that comes with cmake version 3.16 BUT Qt (5.15.2 installed from Qt site) comes with cmake 3.27 in its tools directory. This is the one I use.

Thanks! Any recommendation for getting this into a CI (command line?)

midi-pascal · 2024-05-14T21:06:31Z

Not sure of what I will suggest below since I build using Qt Creator(so all is set automagically) but there are environment variables related to cmake:

CMAKE_PREFIX_PATH

CMAKE_PREFIX_PATH
Semicolon-separated list of directories specifying installation
prefixes to be searched by the find_package(),
find_program(), find_library(), find_file(), and
find_path() commands. Each command will add appropriate
subdirectories (like bin, lib, or include) as specified in its own
documentation.
By default this is empty. It is intended to be set by the project.
There is also an environment variable CMAKE_PREFIX_PATH, which is used
as an additional list of search prefixes.
See also CMAKE_SYSTEM_PREFIX_PATH, CMAKE_INCLUDE_PATH,
CMAKE_LIBRARY_PATH, CMAKE_PROGRAM_PATH, and
CMAKE_IGNORE_PATH.

Hope this can help :-)

PhysSong · 2024-05-15T01:37:00Z

FYI, CMake homepage lists the official PPA and pip package as alternative installation methods.

messmerd · 2024-05-15T02:09:31Z

@tresf I could update the Linux image with a newer CMake using the official PPA if you want. That might be easiest and also won't impact build times

tresf · 2024-05-15T02:58:24Z

@tresf I could update the Linux image with a newer CMake using the official PPA if you want. That might be easiest and also won't impact build times

If parties agree on this CPack strategy, yes please (or from snap).

Tested on Ubuntu 20.04:

sudo snap install cmake --classic
alias cmake='snap run cmake'
# cmake --version
# cmake version 3.29.3
#
# CMake suite maintained and supported by Kitware (kitware.com/cmake).

... I had reservations about whether or not the snap version of cmake would play nicely... and it seems to work just fine.

tresf · 2024-05-15T06:19:33Z

@tresf I could update the Linux image with a newer CMake using the official PPA if you want. That might be easiest and also won't impact build times

@messmerd yeah, I'll take that offer. 🤣

Rossmaxx · 2024-05-15T06:22:26Z

can we use the PPA for mingw ci image too?

messmerd · 2024-05-15T14:09:24Z

yeah, I'll take that offer. 🤣

I'll try to do it sometime today

can we use the PPA for mingw ci image too?

Yes, I think so, though I don't think it will be needed for this PR

tresf · 2024-05-15T16:22:56Z

Yes, I think so, though I don't think it will be needed for this PR

Actually, we can justify it here since bad79a8.

tresf · 2024-05-16T04:46:39Z

I've marked this ready for review because I think it's in pretty good shape. (artifacts should start generating once cmake is updated).

ARM64.AppImage.mp4

- Refactor BashCompletion.cmake - Add initial support for Linux AppImages (untested) - Add debug flags for cpack

messmerd · 2024-10-23T16:35:57Z

On Linux, Carla Rack and Carla Patchbay crash when I try to use them, and this is not the case on master

EDIT: Here's the command-line output:

Carla appears to be installed on this system at /usr/lib[64]/carla so we'll use it.
Jack appears to be installed on this system, so we'll use it.
Lv2 plugin SUMMARY: 18 of 40 loaded in 80 msecs.
For details about not loaded plugins, please set
environment variable "LMMS_LV2_DEBUG" to nonempty.
/tmp/.mount_lmms-1XSszgf/AppRun.wrapped: line 24: 800103 Segmentation fault (core dumped) QT_X11_NO_NATIVE_MENUBAR=1 "$DIR"/usr/bin/lmms "$@"

JohannesLorenz · 2024-10-24T22:05:56Z

cmake/modules/BashCompletion.cmake

+		if(BASHCOMP_PKG_PATH)
+			# TODO: CMake 3.21 Use "file(COPY_FILE ...)"
+			install(CODE "
+					execute_process(COMMAND ${CMAKE_COMMAND} -E copy \"${SCRIPT_NAME}\" \"${BASHCOMP_PKG_PATH}\" ERROR_QUIET RESULT_VARIABLE result)


Why does it install SCRIPT_NAME twice, once in line 47 and once in line 51?
Also, where is the BASHCOMP_PKG_PATH variable from?

Why does it install SCRIPT_NAME twice, once in line 47 and once in line 51?

It's due to cmake's lack of conditional INSTALL(...) logic. For some reason, I removed the comment that describes this. It's here: 00d4ddd#diff-7960a523ea4d93951917fb0445e1145899f31c5a796e123ee06c805095e37221R59-R71.

In short:

Always install to user-space

Attempt to install system-wide, but don't fail the build if it doesn't work.

We did have some talks about this "system wide" install previously because this is the first time that we've had a component to install in a prefix that's outside of our own install prefix. I suppose we could instead detect a system-wide prefix and handle this in a more standard fashion. If that's the case, we can fix it as part of this PR if needed. I know some environments (such as Apple's Homebrew) ARE user-writable, so perhaps that was my motivation for this but that does NOT make sense to try when building a DMG or AppImage, so this logic is likely flawed.

Furthermore, I think there's a typo in the status message though.... I think this should say BASHCOMP_PKG_PATH not BASHCOMP_USER_PATH.

- message(STATUS "Unable to install bash-completion support system-wide: ${BASHCOMP_USER_PATH}/${SCRIPT_NAME}) + - message(STATUS "Unable to install bash-completion support system-wide: ${BASHCOMP_PKG_PATH}/${SCRIPT_NAME})

Also, where is the BASHCOMP_PKG_PATH variable from?

It's from #4604. This is not a newly introduced variable with this PR. I believe its intended to be the completionsdir component, found by pkg-config. My memory is foggy from back then and the conversations seem to have largely occurred on Discord so decisions are not all obvious. Anyway, we also re-use this variable name for cmake's find_package(bash-completion) portion as well. It's probably a bad name, but I do not plan on changing it as part of this PR but can if needed.

I believe its intended to be the completionsdir component, found by pkg-config.

Indeed! The top of the file actually explains it. You can even set this variable using -DBASHCOMP_PKG_PATH=....

Furthermore, I think there's a typo in the status message though.... I think this should say BASHCOMP_PKG_PATH not BASHCOMP_USER_PATH.

Agreeing. Please feel free to fix.

Attempt to install system-wide, but don't fail the build if it doesn't work.

IMO, the natural behavior would be: If the user does a local install, install bashcomp locally. If they do a system install ("root"), install it system-wide. I assume that back then, we thought: "An install should always try to be system-wide, because local installed bashcomp is not sourced by your bashrc". However, you could give the same argument for the PATH variable (a local install of bin/lmms will never be in your PATH by default). In both cases, you can extend your bashrc to source a local bashcomp or to extend your PATH to a local directory. All in all, I would favor to do the install exactly one, and always in the normal install tree. However, it is just my basic opinion, and also it might be considered out of scope of this PR. Tagging @PhysSong because they were also active in that bashcomp PR in 2018.

Thanks for the feedback. I'll put some time into fixing this.

tresf · 2024-10-25T03:52:26Z

On Linux, Carla Rack and Carla Patchbay crash when I try to use them, and this is not the case on master

Thanks! Should be fixed via aab4cc5.

tresf · 2024-10-25T04:13:46Z

On Linux, Carla Rack and Carla Patchbay crash when I try to use them, and this is not the case on master

Thanks! Should be fixed via aab4cc5.

Nevermind, still crashes. Investigating...

messmerd · 2024-10-25T06:05:15Z

@tresf
There are 119 files and 2 directories under squashfs-root/usr/lib in this PR, but just 92 files and 1 directory in master.

I'm not sure why there are so many more, but of those extra files, there were libcarlabase.so and libcarla_native-plugin.so. After I removed them, Carla now longer crashed LMMS.

tresf · 2024-10-25T06:26:56Z

@tresf There are 119 files and 2 directories under squashfs-root/usr/lib in this PR, but just 92 files and 1 directory in master.

I'm not sure why there are so many more, but of those extra files, there were libcarlabase.so and libcarla_native-plugin.so. After I removed them, Carla now longer crashed LMMS.

@messmerd thank you. I'm fully aware of the cause of the crash. linuxdeploy is run twice. First using a qt plugin, which we first use to crawl dependencies and create the .AppDir structure, but then we call it a second time passing the --output appimage flag which we call only after manual cleanup of these carla libraries is performed. Instead of being a replacement for the appimagetool command, it re-runs the dependency tree, so any manually removed files are copied back in. Here's the offending line:

lmms/cmake/linux/LinuxDeploy.cmake

Lines 244 to 250 in 22f17e0

    
               execute_process(COMMAND "${LINUXDEPLOY_BIN}" 
        
               	--appdir "${APP}" 
        
               	--output appimage 
        
               	--verbosity ${VERBOSITY} 
        
               	${OUTPUT_QUIET} 
        
               	COMMAND_ECHO ${COMMAND_ECHO} 
        
               	COMMAND_ERROR_IS_FATAL ANY)

There's a new --exclude-library flag which takes a globbing pattern that I'm experimenting with to see if it can replace the manual cleanup efforts.

tresf · 2024-10-25T07:06:02Z

Apologies for the whitespace changes in the latest commit 508f430, but spaces and tabs were inconsistent.

The relevant fix is the addition of ${EXCLUDES} variable which adds --exclude-library=pattern. It could be argued that the glob should be provided to both calls to linuxdeploy and the manual cleanup effort removed (rather than having two steps 1. cmake do the glob, 2. provide any found filenames to the --exclude-library flag) but at this point I'm afraid to change too much in fear of causing other issues.

Another big miss was never providing linuxdeploy the -executable= flags. We were building it but never passing it to linuxdeploy. This was would likely have caused some linking issues if not fixed prior to merging. (ignore the double dash change, that's just for readiblity).

There are 119 files and 2 directories under squashfs-root/usr/lib in this PR, but just 92 files and 1 directory in master.

I haven't done a comparison yet, but one thing that linuxdeploy does that linuxdeployqt did not is to attempt to fetch the licensing information for all dependencies. Most of these attempts fail when verbose logging is turned on, but it's one possible culprit. The second directory is a qt wrapper script exclusive to this tool.

I haven't tested the AppImage yet but I believe the Carla loading issue is at least resolved. I'm curious what the directory tree compare looks like. I used to use a commercial tool called "Beyond Compare" for comparing file trees, perhaps there's a FOSS equivalent.

messmerd · 2024-10-25T08:21:16Z

I've confirmed that Carla no longer crashes.

However, it looks like there are copies of all the LMMS plugins and some other .so files in both usr/bin and usr/lib/lmms instead of just the latter.
And for at least the past couple commits, the LMMS icon hasn't been showing up on the AppImage file.

As a side note, I just realized xpressive is a whopping 71 MB. It's also always last to finish compiling in our CI builds, so maybe something should be done about that. (Though not in this PR of course)

tresf · 2024-10-25T14:15:29Z

Although undocumented, it appears appimagetool is available as part of the linuxdeploy tool, as can be observed when verbose logging WANT_DEBUG_CPACK is enabled. I may attempt to use this directly. I'll have to put a few more hours into it.

tresf · 2024-12-26T20:35:11Z

Although undocumented, it appears appimagetool is available as part of the linuxdeploy tool, as can be observed when verbose logging WANT_DEBUG_CPACK is enabled. I may attempt to use this directly. I'll have to put a few more hours into it.

This is really the way forward because otherwise, linuxdeploy runs the tree shaking again which takes so much time to run. Starting with b8950f4, I've reverted the project to use appimagetool again. This is how we do it on master and I'm hopeful that it will make builds quicker and and better management of what we do and don't bundle.

it looks like there are copies of all the LMMS plugins and some other .so files in both usr/bin and usr/lib/lmms instead of just the latter.

@messmerd Thanks for this find! ~~The prophecy was foretold!~~ So with both Mac and Linux installers -- for some reason -- I had linking issues unless I added certain plugins to linuxdeployqt|macdeployqt respectively, however, I believe the behavior has changed slightly with linuxdeploy and actually copies these to bin. ~~Comically, this was already brought to question in this PR here:~~ (nevermind, I'm was lib/lmms with lib/lmms/ladspa which are unrelated), this mystery still exists.

When this happens with Zyn, I move the files and symlink so the binary can still be found, but for plugins I feel like I can just blindly remove them.

I have a sneaking suspicion that they're needed to fix linking against Qt libs, but since they're plugins they'd normally not get fixed.
I think the reason that they're copied to bin is because of the explicit use of the --executable= notation. Perhaps using the --library= notation will fix this, but then I still fear that these files will be copied to lib instead, when they actually belong in lib/lmms.

I'll take a swing at this new --library= option and see how well it works. If it does the same thing as before, I can workaround any misplaced files just before calling appimagetool, since we're back to using that now, it shouldn't be so hard to shuffle files around.

tresf · 2024-12-26T21:02:00Z

this mystery still exists.

It turns out that this is caused by the dependency of libfftw. I'll update the comments.

then I still fear that these files will be copied to lib instead, when they actually belong in lib/lmms.

This is exactly what happens, it just ignores our directory structure for items we've added using --library=. I'll have to introduce some more variables to track where these were so we can safely move them back to the correct location. This is a bit of a regression from linuxdeployqt, but it shouldn't be too hard to patch.

tresf · 2024-12-26T22:55:59Z

I'll have to introduce some more variables to track where these were so we can safely move them back to the correct location. This is a bit of a regression from linuxdeployqt, but it shouldn't be too hard to patch.

It is hard to patch. Since it uses relative paths when relinking, when we move them back, it breaks any dependant links.

For example, take libamplifier.so, when the lib is placed in <appimage>/usr/lib/libamplifier.so, it properly links to libQt5Gui.so.5 => LMMS.AppDir/usr/lib/libQt5Gui.so.5. The moment I move it to <appimage>/usr/lib/lmms, linking breaks.

Maybe I can find a flag to preserve the existing directory structure. If not, dumping all libs into one directory is going to get ugly (both cosmetically as well as in our code that searches for plugins).

tresf · 2024-12-27T01:45:47Z

Maybe I can find a flag to preserve the existing directory structure. If not, dumping all libs into one directory is going to get ugly (both cosmetically as well as in our code that searches for plugins).

I could not find one. I tried to reproduce the directory structure with a bunch of symbolic links, but they still fail to find the bundled libraries due to how the paths are loaded. More information here: https://stackoverflow.com/questions/34332571/how-is-the-rpath-resolved-when-the-application-is-started-via-a-symlink.

This is very bad and I'm at a bit of a crossroads. I may need to beg linuxdeploy to allow a structure preservation feature because this just clobbers all .so's into /usr/lib, which I'm not sure we can maintain.

messmerd · 2025-01-14T14:09:47Z

@tresf

I may need to beg linuxdeploy to allow a structure preservation feature because this just clobbers all .so's into /usr/lib, which I'm not sure we can maintain.

Could you open an issue upstream? Maybe it wouldn't be difficult for them to implement. It'd be better to have proper features/fixes upstream than any hacky workarounds here.

tresf · 2025-01-14T14:25:34Z

@tresf

I may need to beg linuxdeploy to allow a structure preservation feature because this just clobbers all .so's into /usr/lib, which I'm not sure we can maintain.

Could you open an issue upstream? Maybe it wouldn't be difficult for them to implement. It'd be better to have proper features/fixes upstream than any hacky workarounds here.

Yes, I had decided this was my next steps but hadn't updated this report. I'll crosslink when done.

tresf · 2025-01-14T17:02:56Z

Question raised upstream: linuxdeploy/linuxdeploy#305. If there's any wording there that's incorrect or can be improved, please let me know.

tresf · 2025-01-16T16:54:44Z

@tresf

I may need to beg linuxdeploy to allow a structure preservation feature because this just clobbers all .so's into /usr/lib, which I'm not sure we can maintain.

Could you open an issue upstream? Maybe it wouldn't be difficult for them to implement. It'd be better to have proper features/fixes upstream than any hacky workarounds here.

@messmerd no word from the upstream tracker or IRC (I hung out for the majority of a day, not one single reply). To move this PR forward, we may have to change the runtime code to find these plugins in /usr/lib. I don't love the idea of scanning unrelated files to populate our plugins listing, but we have to pick our battles.

tresf changed the title ~~CPack: Proof of concept for making Apple installers~~ CPack: Refactor AppImage and Apple DMG Generation May 14, 2024

tresf marked this pull request as ready for review May 16, 2024 04:40

tresf mentioned this pull request May 16, 2024

Update MinGW CI to 20.04 #7256

Closed

tresf added 13 commits May 16, 2024 17:02

CPack: Proof of concept for making Apple installers

aa13e0b

Formatting, wording

fefa368

Remove appdmg dependency

349a13f

Shellcheck

3a8174e

Shellcheck2

08143b4

Document .DS_Store better

87590ae

Try to fix tiff file

a6f50ff

Initial refactoring

615927c

Minor changes

5c4a986

Convert apple package script to cmake

2289655

Fix appdmg

ae049e5

Order of operations: Fix broken DMG

8443819

Initial Linux support:

d95d126

- Refactor BashCompletion.cmake - Add initial support for Linux AppImages (untested) - Add debug flags for cpack

tresf mentioned this pull request Jun 19, 2024

minor cleanup of MinGW-related scripts #7327

Merged

JohannesLorenz added this to the 1.3 milestone Aug 4, 2024

tresf added 6 commits September 16, 2024 13:20

Merge branch 'master' into cpack

7051ced

Remove unused CMAKE_INSTALL_PREFIX

fda485a

true -> TRUE, false -> FALSE

eb25a0d

Deobfuscate CPACK_DEBUG

cc9cde2

Don't strip Windows libs (for historic reasons)

7f217d8

Remove lingering linuxdeployqt workaround

a9fe1b3

JohannesLorenz reviewed Oct 24, 2024

View reviewed changes

Fix Carla on Linux

aab4cc5

Make DownloadBinary reproducible

22f17e0

Fix Carla, whitespace

508f430

Revert back to appimagetool for finer control on what we bundle

b8950f4

tresf mentioned this pull request Jan 14, 2025

Subfolder handling in /usr/lib linuxdeploy/linuxdeploy#305

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

CPack: Refactor AppImage and Apple DMG Generation #7252

CPack: Refactor AppImage and Apple DMG Generation #7252

tresf commented May 12, 2024 •

edited

Loading

tresf commented May 12, 2024 •

edited

Loading

tresf commented May 14, 2024

tresf commented May 14, 2024 •

edited

Loading

midi-pascal commented May 14, 2024

tresf commented May 14, 2024

midi-pascal commented May 14, 2024 •

edited by tresf

Loading

PhysSong commented May 15, 2024

messmerd commented May 15, 2024

tresf commented May 15, 2024 •

edited

Loading

tresf commented May 15, 2024

Rossmaxx commented May 15, 2024

messmerd commented May 15, 2024 •

edited

Loading

tresf commented May 15, 2024

tresf commented May 16, 2024 •

edited

Loading

messmerd commented Oct 23, 2024 •

edited

Loading

JohannesLorenz Oct 24, 2024

tresf Oct 24, 2024 •

edited

Loading

JohannesLorenz Oct 26, 2024 •

edited

Loading

tresf Oct 26, 2024

tresf commented Oct 25, 2024

tresf commented Oct 25, 2024

messmerd commented Oct 25, 2024

tresf commented Oct 25, 2024 •

edited

Loading

tresf commented Oct 25, 2024 •

edited

Loading

messmerd commented Oct 25, 2024

tresf commented Oct 25, 2024

tresf commented Dec 26, 2024 •

edited

Loading

tresf commented Dec 26, 2024

tresf commented Dec 26, 2024 •

edited

Loading

tresf commented Dec 27, 2024 •

edited

Loading

messmerd commented Jan 14, 2025

tresf commented Jan 14, 2025

tresf commented Jan 14, 2025

tresf commented Jan 16, 2025

CPack: Refactor AppImage and Apple DMG Generation #7252

Are you sure you want to change the base?

CPack: Refactor AppImage and Apple DMG Generation #7252

Conversation

tresf commented May 12, 2024 • edited Loading

tresf commented May 12, 2024 • edited Loading

tresf commented May 14, 2024

tresf commented May 14, 2024 • edited Loading

midi-pascal commented May 14, 2024

tresf commented May 14, 2024

midi-pascal commented May 14, 2024 • edited by tresf Loading

PhysSong commented May 15, 2024

messmerd commented May 15, 2024

tresf commented May 15, 2024 • edited Loading

tresf commented May 15, 2024

Rossmaxx commented May 15, 2024

messmerd commented May 15, 2024 • edited Loading

tresf commented May 15, 2024

tresf commented May 16, 2024 • edited Loading

messmerd commented Oct 23, 2024 • edited Loading

JohannesLorenz Oct 24, 2024

Choose a reason for hiding this comment

tresf Oct 24, 2024 • edited Loading

Choose a reason for hiding this comment

JohannesLorenz Oct 26, 2024 • edited Loading

Choose a reason for hiding this comment

tresf Oct 26, 2024

Choose a reason for hiding this comment

tresf commented Oct 25, 2024

tresf commented Oct 25, 2024

messmerd commented Oct 25, 2024

tresf commented Oct 25, 2024 • edited Loading

tresf commented Oct 25, 2024 • edited Loading

messmerd commented Oct 25, 2024

tresf commented Oct 25, 2024

tresf commented Dec 26, 2024 • edited Loading

tresf commented Dec 26, 2024

tresf commented Dec 26, 2024 • edited Loading

tresf commented Dec 27, 2024 • edited Loading

messmerd commented Jan 14, 2025

tresf commented Jan 14, 2025

tresf commented Jan 14, 2025

tresf commented Jan 16, 2025

tresf commented May 12, 2024 •

edited

Loading

tresf commented May 12, 2024 •

edited

Loading

tresf commented May 14, 2024 •

edited

Loading

midi-pascal commented May 14, 2024 •

edited by tresf

Loading

tresf commented May 15, 2024 •

edited

Loading

messmerd commented May 15, 2024 •

edited

Loading

tresf commented May 16, 2024 •

edited

Loading

messmerd commented Oct 23, 2024 •

edited

Loading

tresf Oct 24, 2024 •

edited

Loading

JohannesLorenz Oct 26, 2024 •

edited

Loading

tresf commented Oct 25, 2024 •

edited

Loading

tresf commented Oct 25, 2024 •

edited

Loading

tresf commented Dec 26, 2024 •

edited

Loading

tresf commented Dec 26, 2024 •

edited

Loading

tresf commented Dec 27, 2024 •

edited

Loading