Issues with xcache

[bryates]

Hi experts,
I'm having an issue on Coffea-casa when processing a large number of files. I have a custom ttbar nanoAOD sample which includes the AK4 jet PF candidates, and I'm trying to run a simple processor over them on dask. I'm using the xcache redirector to load the files, and if I run over 100-200 of them, it runs fine most of the time. However, if I try to run over all the files (~700) it seems to time out around 20-30 minutes, and reports it cannot open a file. My notebook is in this git repo.

I spoke a bit with @oshadura about this, but we couldn't track it down.

[oshadura]

cc @clundst @jthiltges @Sam6734

Here in this setup it is used xcache1.unl.edu and maybe it is some issue with space? @bryates what is the side of the total dataset?

[bryates]

It could be. Each file is about 2GB, and there are 670 files. They currently have PFCands, which take up a lot of space. I'm no longer using those, so I might regenerate the files without them. Interestingly, I also ran this on lxplus with just a standard python file on condor. If I make a job for each file, it runs fine. However, when I submitted one job to run over all 670, it also failed. Since that was on lxplus, I used /eos/ directly, so maybe it's not an XRootD redirector issue at all.

[bryates]

I've reprocessed the samples without PFCands so now each file is ~400-800MB. I'm still seeing this issue. Interestingly, while playing with the chunk sizes, I found a chunk size of around 30,000 gets things to run up to ~90% in about 10 minutes, but then it hangs for about a half an hour before giving the errors we've seen in the past.

[oshadura]

@bryates probably it is extremely late, but in the last year we imporved stability of xcache and now ache image is using latest deployment:

cms-jovyan@jupyter-oksana-2eshadura-40cern-2ech:~$ env | grep xcache
XCACHE_HOST=xcache.cmsaf-dev.flatiron.hollandhpc.org

Task stream looks very intersting and it makes me think that maybe chunk size is wrong (?):

(this notebook in examople is using old coffea 0.7.X)

[bryates]

Thanks @oshadura. I'll give this a try when I have some time.