Data Library's "Import from User Directory" interface's display is very slow for full directories

[vladvisan]

Description of the problem

(Tested on 23.2.2.dev0)

I'm referring to this interface:

• 
• (image taken from https://galaxyproject.org/data-libraries/#from-user-folder)

Before showing anything to the user, the page loads the whole folder structure recursively (it doesn't unfold it automatically, but it does load it)
Which, in our case (large sub-directory structure with many files, AND network-mounted ) leads to:

• the whole Galaxy instance being unresponsive for a few minutes, for all users
• eventually aborting because we reach "MAX_WALK_DIRS" (10K by default)

Relevant issues

• [performance] Library importing can be a bit slow #16112 ? I don't think so since it refers to the transition from data library to history, rather than the display of the user directory
• New Data Library Revision #16496 - Modernisation of the Data Library interface, although not specifically referencing this problem

Workaround
As a workaround, I did 2 things:

1. Lower "MAX_WALK_DIRS" drastically, so that at least the abort happens sooner so that other users are less impacted

• Maybe if the user closes his tab it would solve it, I haven't tested that. But either way, we can't assume they will do that.

2. Switch to the "Upload" interface, which, as you can see, only loads one depth level at a time, which is a lot more efficient:

• 
• 
• 

Remaining problems after the workaround

• Even the Upload interface still takes a long time for folders with alot of files directly (as opposed to in sub-folders)
• The Upload interface does not support linking files instead of copying
  • Although there is a very recent merged PR that might have dealt with it, I'm not sure Allow a posix file source to prefer linking. #19132

Potential solutions

• re-use the Upload tool's interface in the Data Library's "Import from User Directory" interface?
  • At least reduces the timeout pb to numerous files one-level, from numerous files recursively (much more common)
• modify the "recursive" option here? https://github.com/galaxyproject/galaxy/blob/release_23.2/lib/galaxy/managers/remote_files.py#L90
  • which calls -> lib/galaxy/files/sources/init.py / list -> /lib/galaxy/files/sources/posix.py / _list -> lib/galaxy/util/path/init.py / safe_walk
  • NB: the file has been modified since the 23.2 I am referencing (since it's my tested version)
• start with a call to the OS to get the amount of files (potentially recursively)
  • to be able to abort pre-emptively instead of trying for nothing
  • to be able to display a progress bar
• max amount of RAM per user instead of affecting other users