ROOT installation from conda break HTCondor scaling in coffea-casa environment

[oshadura]

ROOT update HTCondor up to 23.3.x and this break whole Dask<-> HTCOndor scaling setup.
Currently UNL collector is running the HTCondor 10.x.x version.

cms-jovyan@jupyter-oksana-2eshadura-40cern-2ech:~$ mamba install -c conda-forge root

Looking for: ['root']

conda-forge/linux-64                                        Using cache
conda-forge/noarch                                          Using cache

Pinned packages:
  - python 3.9.*


Transaction

  Prefix: /opt/conda

  Updating specs:

   - root
   - ca-certificates
   - certifi
   - openssl


  Package                                   Version  Build                  Channel          Size
───────────────────────────────────────────────────────────────────────────────────────────────────
  Install:
───────────────────────────────────────────────────────────────────────────────────────────────────

  + gsl                                         2.7  he838d99_0             conda-forge       3MB
  + nlohmann_json                            3.11.2  h27087fc_0             conda-forge     115kB
  + cfitsio                                   4.3.1  hbdc6101_0             conda-forge     875kB
  + pythia8                                   8.310  py39h3d6467e_0         conda-forge      31MB
  + vector-classes                            1.4.4  h00ab1b0_0             conda-forge     250kB
  + gettext                                  0.21.1  h27087fc_0             conda-forge       4MB
  + fontconfig                               2.14.2  h14ed4e7_0             conda-forge     272kB
  + xorg-xextproto                            7.3.0  h0b41bf4_1003          conda-forge      30kB
  + xorg-libice                               1.1.1  hd590300_0             conda-forge      58kB
  + fftw                                     3.3.10  nompi_hc118613_108     conda-forge       2MB
  + giflib                                    5.2.1  h0b41bf4_3             conda-forge      77kB
  + libllvm13                                13.0.1  hf817b99_2             conda-forge      35MB
  + gtest                                    1.14.0  h00ab1b0_1             conda-forge     405kB
  + libcxxabi                                16.0.6  ha770c72_0             conda-forge     157kB
  + binutils                                   2.40  hdd6e379_0             conda-forge      30kB
  + pixman                                   0.43.0  h59595ed_0             conda-forge     387kB
  + portalocker                               2.8.2  py39hf3d152e_1         conda-forge      36kB
  + xorg-kbproto                              1.0.7  h7f98852_1002          conda-forge      27kB
  + xorg-xproto                              7.0.31  h7f98852_1007          conda-forge      75kB
  + xorg-renderproto                         0.11.1  h7f98852_1002          conda-forge      10kB
  + gl2ps                                     1.4.2  h0708190_0             conda-forge     150kB
  + vdt                                       0.4.4  h59595ed_0             conda-forge      30kB
  + fribidi                                  1.0.10  h36c2ea0_0             conda-forge     114kB
  + graphite2                                1.3.13  h58526e2_1001          conda-forge     105kB
  + libglib                                  2.78.3  h783c2da_0             conda-forge       3MB
  + libhwloc                                  2.9.3  default_h554bfaf_1009  conda-forge       3MB
  + davix                                     0.8.5  h3299534_0             conda-forge       1MB
  + xorg-fixesproto                             5.0  h7f98852_1002          conda-forge       9kB
  + xorg-libsm                                1.2.4  h7391055_0             conda-forge      27kB
  + libwebp                                   1.3.2  hdffd6e0_0             conda-forge      86kB
  + libcxx                                   16.0.6  h00ab1b0_0             conda-forge       1MB
  + xorg-libx11                               1.8.7  h8ee46fc_0             conda-forge     829kB
  + gts                                       0.7.6  h977cf35_4             conda-forge     318kB
  + atk-1.0                                  2.38.0  hd4edc92_1             conda-forge     552kB
  + gdk-pixbuf                              2.42.10  h6c15284_3             conda-forge     574kB
  + tbb                                   2021.11.0  h00ab1b0_0             conda-forge     194kB
  + xorg-libxfixes                            5.0.3  h7f98852_1004          conda-forge      18kB
  + xorg-libxrender                          0.9.11  hd590300_0             conda-forge      38kB
  + xorg-libxt                                1.3.0  hd590300_1             conda-forge     379kB
  + xorg-libxext                              1.3.4  h0b41bf4_2             conda-forge      50kB
  + xorg-libxcursor                           1.2.0  h0b41bf4_1             conda-forge      31kB
  + libglu                                    9.0.0  hac7e632_1003          conda-forge     331kB
  + xorg-libxpm                              3.5.17  hd590300_0             conda-forge      64kB
  + htcondor-cli                             23.3.0  py39hf3d152e_0         conda-forge      94kB
  + ftgl                                      2.4.0  hbcb1f35_0             conda-forge     115kB
  + glew                                      2.1.0  h9c3ff4c_2             conda-forge     663kB
  + font-ttf-dejavu-sans-mono                  2.37  hab24e00_0             conda-forge     397kB
  + font-ttf-inconsolata                      3.000  h77eed37_0             conda-forge      97kB
  + font-ttf-source-code-pro                  2.038  h77eed37_0             conda-forge     701kB
  + font-ttf-ubuntu                            0.83  h77eed37_1             conda-forge       2MB
  + ipyparallel                               8.6.1  pyhd8ed1ab_0           conda-forge     208kB
  + _sysroot_linux-64_curr_repodata_hack          3  h69a702a_13            conda-forge      21kB
  + fonts-conda-forge                             1  0                      conda-forge       4kB
  + metakernel                               0.30.1  pyhd8ed1ab_0           conda-forge     187kB
  + fonts-conda-ecosystem                         1  0                      conda-forge       4kB
  + cairo                                    1.18.0  h3faef2a_0             conda-forge     982kB
  + libgd                                     2.3.3  he9388d3_8             conda-forge     225kB
  + xorg-libxft                               2.3.8  hf69aa0a_0             conda-forge      59kB
  + binutils_linux-64                          2.40  hbdbef99_2             conda-forge      28kB
  + harfbuzz                                  8.3.0  h3d44ed6_0             conda-forge       2MB
  + gfortran_impl_linux-64                   12.3.0  hfcedea8_3             conda-forge      15MB
  + gcc_linux-64                             12.3.0  h76fc315_2             conda-forge      30kB
  + pango                                   1.50.14  ha41ecd1_2             conda-forge     444kB
  + gfortran_linux-64                        12.3.0  h7fe76b4_2             conda-forge      29kB
  + gxx_linux-64                             12.3.0  h8a814eb_2             conda-forge      29kB
  + gfortran                                 12.3.0  h499e0f7_2             conda-forge      27kB
  + c-compiler                                1.7.0  hd590300_0             conda-forge       6kB
  + gtk2                                    2.24.33  h7f000aa_3             conda-forge       7MB
  + librsvg                                  2.56.3  he3f83f7_1             conda-forge       6MB
  + fortran-compiler                          1.7.0  heb67821_0             conda-forge       6kB
  + cxx-compiler                              1.7.0  h00ab1b0_0             conda-forge       6kB
  + graphviz                                  9.0.0  h78e8752_1             conda-forge       2MB
  + afterimage                                 1.21  h28ea39c_1005          conda-forge     692kB
  + compilers                                 1.7.0  ha770c72_0             conda-forge       7kB
  + root_base                                6.30.2  py39hb30e9c3_1         conda-forge     202MB
  + root                                     6.30.2  py39hddac248_1         conda-forge      32kB

  Upgrade:
───────────────────────────────────────────────────────────────────────────────────────────────────

  - libsqlite                                3.43.0  h2797004_0             conda-forge     841kB
  + libsqlite                                3.44.2  h2797004_0             conda-forge     846kB
  - pcre2                                     10.40  hc3806b6_0             conda-forge       2MB
  + pcre2                                     10.42  hcad00b1_0             conda-forge       1MB
  - libxml2                                  2.12.2  h232c23b_0             conda-forge     705kB
  + libxml2                                  2.12.4  h232c23b_1             conda-forge     705kB
  - sqlite                                   3.43.0  h2c6b66d_0             conda-forge     831kB
  + sqlite                                   3.44.2  h2c6b66d_0             conda-forge     836kB
  - htcondor-classads                        10.8.0  he155508_1             conda-forge      11MB
  + htcondor-classads                        23.3.0  h1bc8f3f_0             conda-forge      11MB
  - libcondor_utils                          10.8.0  h1593839_1             conda-forge      24MB
  + libcondor_utils                          23.3.0  hf167c69_0             conda-forge      24MB
  - python-htcondor                          10.8.0  py39h3aa230b_1         conda-forge       8MB
  + python-htcondor                          23.3.0  py39h0fc6a71_0         conda-forge       8MB
  - htcondor-utils                           10.8.0  h841b71b_1             conda-forge      32MB
  + htcondor-utils                           23.3.0  h841b71b_0             conda-forge      33MB
  - htcondor                                 10.8.0  py39hf3d152e_1         conda-forge      22kB
  + htcondor                                 23.3.0  py39hf3d152e_0         conda-forge      22kB
  - kernel-headers_linux-64                  2.6.32  he073ed8_16            conda-forge     709kB
  + kernel-headers_linux-64                  3.10.0  h4a8ded7_13            conda-forge     976kB
  - sysroot_linux-64                           2.12  he073ed8_16            conda-forge      15MB
  + sysroot_linux-64                           2.17  h4a8ded7_13            conda-forge      36MB

  Downgrade:
───────────────────────────────────────────────────────────────────────────────────────────────────

  - libsanitizer                             13.2.0  h7e041cc_2             conda-forge       4MB
  + libsanitizer                             12.3.0  h0f45ef3_3             conda-forge       4MB
  - libgcc-devel_linux-64                    13.2.0  ha9c7c90_2             conda-forge       2MB
  + libgcc-devel_linux-64                    12.3.0  h8bca6fd_103           conda-forge       3MB
  - libstdcxx-devel_linux-64                 13.2.0  ha9c7c90_2             conda-forge       9MB
  + libstdcxx-devel_linux-64                 12.3.0  h8bca6fd_103           conda-forge      12MB
  - gcc_impl_linux-64                        13.2.0  h338b0a0_2             conda-forge      53MB
  + gcc_impl_linux-64                        12.3.0  he2b93b0_3             conda-forge      52MB
  - gxx_impl_linux-64                        13.2.0  h338b0a0_2             conda-forge      13MB
  + gxx_impl_linux-64                        12.3.0  he2b93b0_3             conda-forge      13MB
  - gcc                                      13.2.0  h574f8da_2             conda-forge      27kB
  + gcc                                      12.3.0  h8d2909c_2             conda-forge      27kB
  - gxx                                      13.2.0  h574f8da_2             conda-forge      27kB
  + gxx                                      12.3.0  h8d2909c_2             conda-forge      27kB

  Summary:

  Install: 76 packages
  Upgrade: 11 packages
  Downgrade: 7 packages

  Total download: 529MB

───────────────────────────────────────────────────────────────────────────────────────────────────

[oshadura]

After installing ROOT from conda HTCondor CLI shows an error:

cms-jovyan@jupyter-oksana-2eshadura-40cern-2ech:~$ condor_q
Error: Can't find address for schedd t3.unl.edu

Extra Info: You probably saw this error because the condor_schedd is not 
running on the machine you are trying to query. If the condor_schedd is not 
running, the Condor system will not be able to find an address and port to 
connect to and satisfy this request. Please make sure the Condor daemons are 
running and try again.
 
Extra Info: If the condor_schedd is running on the machine you are trying to 
query and you still see the error, the most likely cause is that you have 
setup a personal Condor, you have not defined SCHEDD_NAME in your 
condor_config file, and something is wrong with your SCHEDD_ADDRESS_FILE 
setting. You must define either or both of those settings in your config 
file, or you must use the -name option to condor_q. Please see the Condor 
manual for details on SCHEDD_NAME and SCHEDD_ADDRESS_FILE. 

[clundst]

It's upgrading htcondor to the 23 line, I don't know what effect that may have but I'll wager that's the reason.

[oshadura]

Yeah, it looks like HTCondor cli 23 requires different settings (?) by the way are we planning to upgrade HTCondor at some moment?

[oshadura]

Not anymore issue in current image, now root instead upgrading condor reducing its version (❓ ) but still everything works:

mamba install -c conda-forge root

Looking for: ['root']

conda-forge/noarch                                  17.7MB @  29.1MB/s  0.6s
conda-forge/linux-64                                40.6MB @  53.1MB/s  0.8s

Pinned packages:
  - python 3.10.*


Transaction

  Prefix: /usr/local

  Updating specs:

   - root
   - ca-certificates
   - certifi
   - openssl


  Package                            Version  Build               Channel          Size
─────────────────────────────────────────────────────────────────────────────────────────
  Install:
─────────────────────────────────────────────────────────────────────────────────────────

  + libgcc                            14.2.0  h77fa898_1          conda-forge     849kB
  + binutils_linux-64                   2.40  hb3c18ed_9          conda-forge      29kB
  + binutils                            2.40  h4852527_7          conda-forge      32kB
  + portalocker                        3.0.0  py310hff52083_0     conda-forge      40kB
  + libcxxabi                         18.1.8  hc655929_7          conda-forge     160kB
  + libstdcxx                         14.2.0  hc0a3c3a_1          conda-forge       4MB
  + libcxx                            18.1.8  h719d109_7          conda-forge     648kB
  + libtool                            2.4.7  he02047a_1          conda-forge     414kB
  + perl                              5.32.1  7_hd590300_perl5    conda-forge      13MB
  + xorg-libxft                        2.3.8  hf69aa0a_0          conda-forge      59kB
  + xorg-libxt                         1.3.0  hd590300_1          conda-forge     379kB
  + cfitsio                            4.4.1  hf8ad068_0          conda-forge     924kB
  + gsl                                  2.7  he838d99_0          conda-forge       3MB
  + vdt                                0.4.4  h59595ed_0          conda-forge      30kB
  + nlohmann_json                     3.11.3  he02047a_1          conda-forge     123kB
  + libllvm16                         16.0.6  hb3ce162_3          conda-forge      35MB
  + libglu                             9.0.0  hac7e632_1003       conda-forge     331kB
  + fftw                              3.3.10  nompi_hf1063bd_110  conda-forge       2MB
  + vector-classes                     1.4.5  h297d8ca_0          conda-forge     250kB
  + gtest                             1.15.2  h434a139_0          conda-forge     408kB
  + pcre                                8.45  h9c3ff4c_0          conda-forge     259kB
  + xorg-libxpm                       3.5.17  hd590300_0          conda-forge      64kB
  + gct                       6.2.1629922860  h9f628fe_3          conda-forge       3MB
  + davix                              0.8.6  h5f3b820_0          conda-forge       1MB
  + gl2ps                              1.4.2  hae5d5c5_1          conda-forge      74kB
  + ftgl                               2.4.0  hbcb1f35_0          conda-forge     115kB
  + glew                               2.1.0  h9c3ff4c_2          conda-forge     663kB
  + _x86_64-microarch-level                1  2_x86_64            conda-forge       8kB
  + ipyparallel                        9.0.0  pyhd8ed1ab_1        conda-forge     215kB
  + metakernel                        0.30.2  pyhd8ed1ab_0        conda-forge     186kB
  + pythia8                            8.312  py310hf19e369_100   conda-forge      42MB
  + gfortran_impl_linux-64            13.3.0  h10434e7_1          conda-forge      16MB
  + gcc_linux-64                      13.3.0  hc28eda2_7          conda-forge      32kB
  + gfortran_linux-64                 13.3.0  hb919d3a_7          conda-forge      30kB
  + gxx_linux-64                      13.3.0  h6834431_7          conda-forge      30kB
  + gfortran                          13.3.0  h9576a4e_1          conda-forge      53kB
  + c-compiler                         1.8.0  h2b85faf_1          conda-forge       6kB
  + root_base                         6.32.2  py310hcd03113_3     conda-forge     226MB
  + fortran-compiler                   1.8.0  h36df796_1          conda-forge       6kB
  + cxx-compiler                       1.8.0  h1a2810e_1          conda-forge       6kB
  + compilers                          1.8.0  ha770c72_1          conda-forge       7kB
  + root                              6.32.2  py310hced9424_3     conda-forge      27kB

  Change:
─────────────────────────────────────────────────────────────────────────────────────────

  - gtk2                             2.24.33  h90689f9_2          conda-forge       8MB
  + gtk2                             2.24.33  h6470451_5          conda-forge       7MB

  Upgrade:
─────────────────────────────────────────────────────────────────────────────────────────

  - ca-certificates                 2024.7.4  hbcca054_0          conda-forge     155kB
  + ca-certificates                2024.8.30  hbcca054_0          conda-forge     159kB
  - libgomp                           14.1.0  h77fa898_0          conda-forge     457kB
  + libgomp                           14.2.0  h77fa898_1          conda-forge     461kB
  - libgcc-ng                         14.1.0  h77fa898_0          conda-forge     842kB
  + libgcc-ng                         14.2.0  h69a702a_1          conda-forge      54kB
  - libstdcxx-ng                      14.1.0  hc0a3c3a_0          conda-forge       4MB
  + libstdcxx-ng                      14.2.0  h4852527_1          conda-forge      54kB
  - pcre2                              10.40  hc3806b6_0          conda-forge       2MB
  + pcre2                              10.44  hba22ea6_2          conda-forge     952kB
  - libsqlite                         3.46.0  hde9e2c9_0          conda-forge     865kB
  + libsqlite                         3.47.2  hee588c1_0          conda-forge     874kB
  - openssl                            3.3.1  hb9d3cd8_3          conda-forge       3MB
  + openssl                            3.4.0  hb9d3cd8_0          conda-forge       3MB
  - libpng                            1.6.43  h2797004_0          conda-forge     288kB
  + libpng                            1.6.44  hadc24fc_0          conda-forge     291kB
  - tbb                            2021.12.0  h434a139_3          conda-forge     193kB
  + tbb                            2021.13.0  h84d6215_0          conda-forge     176kB
  - libglib                           2.78.1  hebfc3b9_0          conda-forge       3MB
  + libglib                           2.82.2  h2ff4ddf_0          conda-forge       4MB
  - xrootd                             5.6.8  py310h4d51e2b_0     conda-forge       4MB
  + xrootd                             5.7.1  py310h4d9a3a3_1     conda-forge       4MB
  - harfbuzz                           8.3.0  h3d44ed6_0          conda-forge       2MB
  + harfbuzz                           9.0.0  hfac3d4d_0          conda-forge       2MB
  - gdk-pixbuf                       2.42.10  h829c605_4          conda-forge     572kB
  + gdk-pixbuf                       2.42.12  hb9ae30d_0          conda-forge     528kB
  - pango                            1.50.14  ha41ecd1_2          conda-forge     444kB
  + pango                             1.54.0  h4c5309f_1          conda-forge     447kB
  - librsvg                           2.56.3  he3f83f7_1          conda-forge       6MB
  + librsvg                           2.58.4  hc0ffecb_0          conda-forge       6MB
  - graphviz                           9.0.0  h78e8752_1          conda-forge       2MB
  + graphviz                          12.0.0  hba01fac_0          conda-forge       2MB
  - certifi                         2024.7.4  pyhd8ed1ab_0        conda-forge     159kB
  + certifi                        2024.8.30  pyhd8ed1ab_0        conda-forge     164kB

  Downgrade:
─────────────────────────────────────────────────────────────────────────────────────────

  - libsanitizer                      14.1.0  hcba0ae0_0          conda-forge       4MB
  + libsanitizer                      13.3.0  heb74ff8_1          conda-forge       4MB
  - htcondor-classads                 10.8.0  he155508_1          conda-forge      11MB
  + htcondor-classads                 9.0.20  h3b4a66b_0          conda-forge       7MB
  - libcondor_utils                   10.8.0  h1593839_1          conda-forge      24MB
  + libcondor_utils                   9.0.20  hdaf4444_0          conda-forge      17MB
  - python-htcondor                   10.8.0  py310hb6f69da_1     conda-forge       8MB
  + python-htcondor                   9.0.20  py310h77aee6a_0     conda-forge       7MB
  - htcondor-utils                    10.8.0  h841b71b_1          conda-forge      32MB
  + htcondor-utils                    9.0.20  h841b71b_0          conda-forge      15MB
  - htcondor                          10.8.0  py310hff52083_1     conda-forge      22kB
  + htcondor                          9.0.20  py310hff52083_0     conda-forge      24kB
  - libgcc-devel_linux-64             14.1.0  h5d3d1c9_100        conda-forge       3MB
  + libgcc-devel_linux-64             13.3.0  h84ea5a7_101        conda-forge       3MB
  - libstdcxx-devel_linux-64          14.1.0  h5d3d1c9_100        conda-forge      15MB
  + libstdcxx-devel_linux-64          13.3.0  h84ea5a7_101        conda-forge      14MB
  - gcc_impl_linux-64                 14.1.0  h3c94d91_0          conda-forge      72MB
  + gcc_impl_linux-64                 13.3.0  hfea6d02_1          conda-forge      67MB
  - gxx_impl_linux-64                 14.1.0  h8d00ecb_0          conda-forge      14MB
  + gxx_impl_linux-64                 13.3.0  hdbfa832_1          conda-forge      13MB
  - gcc                               14.1.0  h6f9ffa1_0          conda-forge      52kB
  + gcc                               13.3.0  h9576a4e_1          conda-forge      54kB
  - gxx                               14.1.0  h6f9ffa1_0          conda-forge      51kB
  + gxx                               13.3.0  h9576a4e_1          conda-forge      53kB