Baysor memory issues

[alipirani88]

Hi @VPetukhov

Is there a way to limit the max memory required by Baysor. It has been failing for one of the large datasets. Here is the stdout/stderr from the baysor run:

[13:24:23] Info: Run Rf9b005bd0
[13:24:23] Info: (2025-01-13) Run Baysor v0.6.2
[13:24:23] Info: Loading data...
[13:26:46] Info: Loaded 85251274 transcripts
[13:27:13] Info: Estimating noise level
[14:21:12] Info: Done
[14:29:08] Info: Clustering molecules...
[01:17:24] Info: Algorithm stopped after 1671 iterations. Error: 0.00639. Converged: true.
[01:17:26] Info: Done
[01:17:29] Info: Initializing algorithm. Scale: 5.0, scale std: 0.8, initial #components: 852512, #molecules: 85251274.
[01:23:17] Info: Using the following additional information about molecules: [:confidence, :cluster, :prior_segmentation]
[01:23:17] Info: Using 3D coordinates

Error message:

Progress: 100%|█████████████████████████████████████████| Time: 10:38:40
  Iteration:                  1671
  Max. difference:            0.00639
  Fraction of probs changed:  0.00691
Terminated 97%|███████████████████████████████████████▊ |  ETA: 0:38:43
  Iteration:        484
[3520852] signal (15): Terminated
in expression starting at none:1
sweep_page at /cache/build/default-amdci5-2/julialang/julia-release-1-dot-9/src/gc.c:1578 [inlined]
sweep_pool_page at /cache/build/default-amdci5-2/julialang/julia-release-1-dot-9/src/gc.c:1646 [inlined]
sweep_pool_pagetable0 at /cache/build/default-amdci5-2/julialang/julia-release-1-dot-9/src/gc.c:1666 [inlined]
sweep_pool_pagetable1 at /cache/build/default-amdci5-2/julialang/julia-release-1-dot-9/src/gc.c:1686 [inlined]
sweep_pool_pagetable at /cache/build/default-amdci5-2/julialang/julia-release-1-dot-9/src/gc.c:1716 [inlined]
gc_sweep_pool at /cache/build/default-amdci5-2/julialang/julia-release-1-dot-9/src/gc.c:1797
_jl_gc_collect at /cache/build/default-amdci5-2/julialang/julia-release-1-dot-9/src/gc.c:3540
ijl_gc_collect at /cache/build/default-amdci5-2/julialang/julia-release-1-dot-9/src/gc.c:3706
maybe_collect at /cache/build/default-amdci5-2/julialang/julia-release-1-dot-9/src/gc.c:1078 [inlined]
jl_gc_pool_alloc_inner at /cache/build/default-amdci5-2/julialang/julia-release-1-dot-9/src/gc.c:1443 [inlined]
jl_gc_pool_alloc_noinline at /cache/build/default-amdci5-2/julialang/julia-release-1-dot-9/src/gc.c:1504
jl_gc_alloc_ at /cache/build/default-amdci5-2/julialang/julia-release-1-dot-9/src/julia_internal.h:460 [inlined]
_new_array_ at /cache/build/default-amdci5-2/julialang/julia-release-1-dot-9/src/array.c:144 [inlined]
_new_array at /cache/build/default-amdci5-2/julialang/julia-release-1-dot-9/src/array.c:198 [inlined]
ijl_alloc_array_1d at /cache/build/default-amdci5-2/julialang/julia-release-1-dot-9/src/array.c:436
Array at ./boot.jl:477 [inlined]
make_alias_table! at /home/viktor_petukhov/.julia/packages/StatsBase/XgjIN/src/sampling.jl:618
alias_sample! at /home/viktor_petukhov/.julia/packages/StatsBase/XgjIN/src/sampling.jl:671
#sample!#213 at /home/viktor_petukhov/.julia/packages/StatsBase/XgjIN/src/sampling.jl:923
sample! at /home/viktor_petukhov/.julia/packages/StatsBase/XgjIN/src/sampling.jl:899 [inlined]
#sample#217 at /home/viktor_petukhov/.julia/packages/StatsBase/XgjIN/src/sampling.jl:938 [inlined]
sample at /home/viktor_petukhov/.julia/packages/StatsBase/XgjIN/src/sampling.jl:938 [inlined]
#sample#218 at /home/viktor_petukhov/.julia/packages/StatsBase/XgjIN/src/sampling.jl:941 [inlined]
sample at /home/viktor_petukhov/.julia/packages/StatsBase/XgjIN/src/sampling.jl:941 [inlined]
#sample_center!#255 at /home/viktor_petukhov/.julia/dev/Baysor/src/processing/bmm_algorithm/distribution_samplers.jl:12
sample_center! at /home/viktor_petukhov/.julia/dev/Baysor/src/processing/bmm_algorithm/distribution_samplers.jl:10 [inlined]
sample_position_params! at /home/viktor_petukhov/.julia/dev/Baysor/src/processing/bmm_algorithm/distribution_samplers.jl:4
sample_distribution! at /home/viktor_petukhov/.julia/dev/Baysor/src/processing/bmm_algorithm/distribution_samplers.jl:26
append_empty_component! at /home/viktor_petukhov/.julia/dev/Baysor/src/processing/bmm_algorithm/bmm_algorithm.jl:242 [inlined]
append_empty_components! at /home/viktor_petukhov/.julia/dev/Baysor/src/processing/bmm_algorithm/bmm_algorithm.jl:247
#bmm!#297 at /home/viktor_petukhov/.julia/dev/Baysor/src/processing/bmm_algorithm/bmm_algorithm.jl:348
bmm! at /home/viktor_petukhov/.julia/dev/Baysor/src/processing/bmm_algorithm/bmm_algorithm.jl:333 [inlined]
#run_segmentation#21 at /home/viktor_petukhov/.julia/dev/Baysor/src/processing/utils/cli_wrappers.jl:58
run_segmentation at /home/viktor_petukhov/.julia/dev/Baysor/src/processing/utils/cli_wrappers.jl:16
unknown function (ip: 0x1553c4db0b42)
_jl_invoke at /cache/build/default-amdci5-2/julialang/julia-release-1-dot-9/src/gf.c:2758 [inlined]
ijl_apply_generic at /cache/build/default-amdci5-2/julialang/julia-release-1-dot-9/src/gf.c:2940
#run#5 at /home/viktor_petukhov/.julia/dev/Baysor/src/cli/main.jl:123
run at /home/viktor_petukhov/.julia/dev/Baysor/src/cli/main.jl:51
unknown function (ip: 0x1553c4d3acbd)
_jl_invoke at /cache/build/default-amdci5-2/julialang/julia-release-1-dot-9/src/gf.c:2758 [inlined]
ijl_apply_generic at /cache/build/default-amdci5-2/julialang/julia-release-1-dot-9/src/gf.c:2940
command_main at /home/viktor_petukhov/.julia/packages/Comonicon/HDhA6/src/codegen/julia.jl:343
command_main at /home/viktor_petukhov/.julia/packages/Comonicon/HDhA6/src/codegen/julia.jl:90 [inlined]
julia_main at /home/viktor_petukhov/.julia/packages/Comonicon/HDhA6/src/frontend/cast.jl:481
unknown function (ip: 0x1553c4d3a84f)
_jl_invoke at /cache/build/default-amdci5-2/julialang/julia-release-1-dot-9/src/gf.c:2758 [inlined]
ijl_apply_generic at /cache/build/default-amdci5-2/julialang/julia-release-1-dot-9/src/gf.c:2940
#julia_main#2 at /home/viktor_petukhov/.julia/dev/Baysor/src/Baysor.jl:42
julia_main at /home/viktor_petukhov/.julia/dev/Baysor/src/Baysor.jl:42
jfptr_julia_main_152217 at /rsrch5/home/neuro_rsrch/apirani/baysor-x86_x64-linux-v0.6.2_build/bin/baysor/lib/julia/sys.so (unknown line)
_jl_invoke at /cache/build/default-amdci5-2/julialang/julia-release-1-dot-9/src/gf.c:2758 [inlined]
ijl_apply_generic at /cache/build/default-amdci5-2/julialang/julia-release-1-dot-9/src/gf.c:2940
jl_apply at /cache/build/default-amdci5-2/julialang/julia-release-1-dot-9/src/julia.h:1879 [inlined]
do_call at /cache/build/default-amdci5-2/julialang/julia-release-1-dot-9/src/interpreter.c:126
eval_value at /cache/build/default-amdci5-2/julialang/julia-release-1-dot-9/src/interpreter.c:226
eval_stmt_value at /cache/build/default-amdci5-2/julialang/julia-release-1-dot-9/src/interpreter.c:177 [inlined]
eval_body at /cache/build/default-amdci5-2/julialang/julia-release-1-dot-9/src/interpreter.c:624
jl_interpret_toplevel_thunk at /cache/build/default-amdci5-2/julialang/julia-release-1-dot-9/src/interpreter.c:762
jl_toplevel_eval_flex at /cache/build/default-amdci5-2/julialang/julia-release-1-dot-9/src/toplevel.c:912
jl_toplevel_eval_flex at /cache/build/default-amdci5-2/julialang/julia-release-1-dot-9/src/toplevel.c:856
ijl_toplevel_eval_in at /cache/build/default-amdci5-2/julialang/julia-release-1-dot-9/src/toplevel.c:971
ijl_eval_string at /cache/build/default-amdci5-2/julialang/julia-release-1-dot-9/src/jlapi.c:113
checked_eval_string at baysor (unknown line)
main at baysor (unknown line)
__libc_start_main at /lib64/libc.so.6 (unknown line)
_start at baysor (unknown line)
unknown function (ip: (nil))
Allocations: 406561687098 (Pool: 406147828712; Big: 413858386); GC: 481

I am running the job on hpc node with a single core and 700Gb:

Resource usage summary:

    CPU time :                                   113043.00 sec.
    Max Memory :                                 702 GB
    Average Memory :                             382.35 GB
    Total Requested Memory :                     700.00 GB
    Delta Memory :                               -2.00 GB
    Max Swap :                                   12 GB
    Max Processes :                              4
    Max Threads :                                45
    Run time :                                   113473 sec.
    Turnaround time :                            113468 sec.

Thanks,