Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Max staging file size does not work as expected #502

Open
Pigrenok opened this issue Jun 26, 2024 · 1 comment
Open

Max staging file size does not work as expected #502

Pigrenok opened this issue Jun 26, 2024 · 1 comment

Comments

@Pigrenok
Copy link

If you use --max-staging-file-size=<size> you would expect that files larger than the given limit will not be synced.

Yet, I set 10 kB limit for the session (see below the mutagen sync list -l output), and yet it try to syncronise over 800 MB file.

--------------------------------------------------------------------------------
Name: test-split
Identifier: sync_tDEywqaNnSwijl8m0BTTWxn2xfxi9tsfVj5KLv755y3
Configuration:
	Synchronization mode: Two Way Safe
	Hashing algorithm: Default (SHA-1)
	Maximum allowed entry count: Default (2⁶⁴−1)
	Maximum staging file size: 10000 (10 kB)
	Symbolic link mode: Default (Portable)
	Ignore VCS mode: Default (Propagate)
	Ignores: None
	Permissions mode: Default (Portable)
Alpha:
	URL: /home/pigrenok/YandexDisk/QueenMary/Bioinformatics/ONT/test_split
	Configuration:
		Watch mode: Default (Portable)
		Watch polling interval: Default (10 seconds)
		Probe mode: Default (Probe)
		Scan mode: Default (Accelerated)
		Stage mode: Default (Mutagen Data Directory)
		File mode: Default (0600)
		Directory mode: Default (0700)
		Default file/directory owner: Default
		Default file/directory group: Default
	Connected: Yes
	Synchronizable contents:
		1 directory
		19 files (52 kB)
		0 symbolic links
	Transition problems:
		basecalls.bam: unable to swap file: unable to set staged file permissions: unable to set permission bits: chmod /home/pigrenok/.mutagen/staging/sync_tDEywqaNnSwijl8m0BTTWxn2xfxi9tsfVj5KLv755y3-alpha/a3/a318044517dcb210de625ce510a7d7a877251fdd2c2b98be3af4b85c040455245ae37a93: no such file or directory
		basecalls.bam.bai: unable to create file: unable to set staged file permissions: unable to set permission bits: chmod /home/pigrenok/.mutagen/staging/sync_tDEywqaNnSwijl8m0BTTWxn2xfxi9tsfVj5KLv755y3-alpha/31/313da2412c24a065816f72c054a3b96555a30f79ad56dc98edb105cab72e5e4b7f355e9a: no such file or directory
		report.PycoQC.html: unable to create file: unable to set staged file permissions: unable to set permission bits: chmod /home/pigrenok/.mutagen/staging/sync_tDEywqaNnSwijl8m0BTTWxn2xfxi9tsfVj5KLv755y3-alpha/e7/e70abea6c337802f43edb507242f0a9c8409af38401efbb1304489d15a2e3ee000ad1853: no such file or directory
Beta:
	URL: apocrita:/data/scratch/bty316/Turtles_test
	Configuration:
		Watch mode: Default (Portable)
		Watch polling interval: Default (10 seconds)
		Probe mode: Default (Probe)
		Scan mode: Default (Accelerated)
		Stage mode: Default (Mutagen Data Directory)
		File mode: Default (0600)
		Directory mode: Default (0700)
		Default file/directory owner: Default
		Default file/directory group: Default
		Compression: Default (DEFLATE)
	Connected: Yes
	Synchronizable contents:
		1 directory
		21 files (887 MB)
		0 symbolic links
Status: Staging files on alpha
Staging progress: 2/3 - 645 MB - 67%
Current file: basecalls.bam (639 MB/881 MB)
--------------------------------------------------------------------------------

Do I misunderstand the purpose of this option or is it not working correctly? It is binary file and any touching of it (even without any real changes) triggers full transfer of the file. If this option is not working, is there another reliable way to limit what files/dirs are being synchronized and which are not. I understand that scripts can be moved somewhere else, but that is not preferred way of doing in some cases.

I am aware of the issue #391 but that talks specifically about pre-rejection. I do not mind if first 10 kB (or whatever the limit) is transferred before the file is rejected for transfer, but it does not seem to work that way either.

Thanks.

@jd7h
Copy link

jd7h commented Nov 28, 2024

I have the same problem! maxStagingFileSize is set to 1 MB, but Mutagen is happily syncing large files according to mutagen sync monitor.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants