Ian's pfb tests #72

IanHeywood · 2022-12-05T14:29:49Z

IanHeywood
Dec 5, 2022
Collaborator

As per @landmanbester's request here's the log from my first cleaning run.

INFO      12:41:06 - CLEAN              | Initialising ThreadPool with 8 threads
INFO      12:41:06 - CLEAN              | Input Options:
INFO      12:41:06 - CLEAN              |                        epsilon = 1e-07
INFO      12:41:06 - CLEAN              |                         wstack = True
INFO      12:41:06 - CLEAN              |                   double_accum = True
INFO      12:41:06 - CLEAN              |                   host_address = None
INFO      12:41:06 - CLEAN              |                       nworkers = 8
INFO      12:41:06 - CLEAN              |            nthreads_per_worker = 1
INFO      12:41:06 - CLEAN              |                      nvthreads = 5
INFO      12:41:06 - CLEAN              |                       nthreads = 40
INFO      12:41:06 - CLEAN              |                      scheduler = threads
INFO      12:41:06 - CLEAN              |                          nband = 8
INFO      12:41:06 - CLEAN              |                output_filename = pfbtest
INFO      12:41:06 - CLEAN              |                        postfix =
INFO      12:41:06 - CLEAN              |                        product = I
INFO      12:41:06 - CLEAN              |                  residual_name = RESIDUAL
INFO      12:41:06 - CLEAN              |                     model_name = MODEL
INFO      12:41:06 - CLEAN              |                           mask = None
INFO      12:41:06 - CLEAN              |                           algo = clark
INFO      12:41:06 - CLEAN              |                       dirosion = 1
INFO      12:41:06 - CLEAN              |                       fits_mfs = True
INFO      12:41:06 - CLEAN              |                     fits_cubes = False
INFO      12:41:06 - CLEAN              |                       mop_flux = True
INFO      12:41:06 - CLEAN              |                         nmiter = 5
INFO      12:41:06 - CLEAN              |                      threshold = None
INFO      12:41:06 - CLEAN              |                 sigmathreshold = 2.0
INFO      12:41:06 - CLEAN              |                          gamma = 0.1
INFO      12:41:06 - CLEAN              |                    peak_factor = 0.15
INFO      12:41:06 - CLEAN              |                sub_peak_factor = 0.75
INFO      12:41:06 - CLEAN              |                   hogbom_maxit = 5000
INFO      12:41:06 - CLEAN              |                    clark_maxit = 50
INFO      12:41:06 - CLEAN              |                      sub_maxit = 1000
INFO      12:41:06 - CLEAN              |                        verbose = 1
INFO      12:41:06 - CLEAN              |                    report_freq = 10
INFO      12:41:06 - CLEAN              |                         cg_tol = 0.01
INFO      12:41:06 - CLEAN              |                       cg_maxit = 100
INFO      12:41:06 - CLEAN              |                       cg_minit = 1
INFO      12:41:06 - CLEAN              |                     cg_verbose = 1
INFO      12:41:06 - CLEAN              |                 cg_report_freq = 100
INFO      12:41:06 - CLEAN              |                      backtrack = True
INFO      12:41:06 - CLEAN              |                  memory_greedy = False
INFO      12:44:23 - CLEAN              | Iter 0: peak residual = 2.097e-01, rms = 3.380e-04
INFO      12:44:23 - CLEAN              | Running Clark
INFO      14:06:23 - CLEAN              | Getting residual
INFO      14:09:24 - CLEAN              | Iter 1: peak residual = 2.768e-02, rms = 1.521e-04
INFO      14:09:24 - CLEAN              | Updating results
INFO      14:09:52 - CLEAN              | Running Clark
INFO      16:11:43 - CLEAN              | Getting residual
INFO      16:14:50 - CLEAN              | Iter 2: peak residual = 3.714e-03, rms = 6.343e-05
INFO      16:14:50 - CLEAN              | Updating results
INFO      16:15:26 - CLEAN              | Running Clark
INFO      18:04:49 - CLEAN              | Getting residual
INFO      18:07:35 - CLEAN              | Iter 3: peak residual = 5.675e-04, rms = 3.137e-05
INFO      18:07:35 - CLEAN              | Updating results
INFO      18:08:12 - CLEAN              | Running Clark
INFO      23:30:11 - CLEAN              | Getting residual
INFO      23:32:59 - CLEAN              | Mopping flux at iter 4
INFO      09:16:39 - CLEAN              | Getting residual
INFO      09:19:25 - CLEAN              | Iter 4: peak residual = 6.773e-04, rms = 2.278e-05
INFO      09:19:25 - CLEAN              | Updating results
INFO      09:19:54 - CLEAN              | Running Clark
INFO      15:40:53 - CLEAN              | Getting residual
INFO      15:43:18 - CLEAN              | Mopping flux at iter 5
INFO      20:39:46 - CLEAN              | Getting residual
INFO      20:42:29 - CLEAN              | Iter 5: peak residual = 4.432e-03, rms = 8.017e-05
INFO      20:42:29 - CLEAN              | Updating results
INFO      20:42:59 - CLEAN              | Writing fits
INFO      20:43:17 - CLEAN              | All done here.

IanHeywood · 2022-12-05T14:39:02Z

IanHeywood
Dec 5, 2022
Collaborator Author

For some context: I'm imaging some short track (~10-15 min) MeerKAT data (L-band, 1024 channels), mainly for me to get to grips with how pfbclean works, but also hopefully to perform some useful debugging tests.

I have my first image out but I'm going to wind back to the init step to facilitate a more like-for-like comparison with my existing images, in particular I'd cached the DATA column to the zarr files, and I want to re-do this with CORRECTED_DATA.

Here is my reference image from wsclean:

The pfb residual seems very empty, but note that I did not use a cleaning mask, and it's obviously got into some psf-cleaning instability involving that bright off-axis source. As always I believe that masks are essential, so that could go some way to explaining it, coupled perhaps with my current lack of understanding about how to set pfb's thresholding and factor parameters:

It didn't seem to produce a restored image by default, so if that's possible please let me know, otherwise I can convolve the model and sum it to the residual.

I will post more findings and results here as I continue to experiement with the imager. I really like the philosophy of it and I'm looking forward to getting to grips with it.

Cheers.

0 replies

landmanbester · 2022-12-05T15:06:55Z

landmanbester
Dec 5, 2022
Maintainer

Ooh ouch, it looks like clean is doing alright (just slow) but then the flux mop step is unstable. This can sometimes happen if the PSF is not twice the size of the image but I've not seen it bomb out that badly. I have seen the peak flux increasing after a flux mop step but the rms should always decrease so I am not sure what went wrong. Does the dirty image come out looking similar to wsclean's? Did it produce a pre and post mop residual? If so, can you check if the residual looked alright before the flux mop step?

The pfb residual seems very empty, but note that I did not use a cleaning mask, and it's obviously got into some psf-cleaning instability involving that bright off-axis source.

A fits mask is supported. Let me know if you run into issues with it. The idea behind the flux mop is actually to provide some sort of auto-masking because it should extract all the flux inside the current mask defined as the pixels where the model is non-zero. Looks like I may still have some work to do to get that working though.

It didn't seem to produce a restored image by default, so if that's possible please let me know

You can make a restored image with the restore worker

pfb restore --help
Usage: pfb restore [OPTIONS]

  Create fits image data products (eg. restored images).

Options:
  -ha, --host-address TEXT        Address where the distributed client lives.
                                  Will use a local cluster if no address is
                                  provided and scheduler is set to
                                  distributed.
  -nw, --nworkers TEXT            Number of worker processes.
  -ntpw, --nthreads-per-worker INTEGER
                                  Number of threads per worker process.
  -nvthreads, --nvthreads INTEGER
                                  Number of threads used to scale vertically
                                  (eg. for FFTs).
  -nthreads, --nthreads INTEGER   Total number of threads to use between all
                                  processes. Will use all available resources
                                  by default
  -scheduler, --scheduler [threads|single-threaded|sync|distributed]
                                  Which dask scheduler to use.  [default:
                                  threads]
  -mname, --model-name TEXT       Name of model in mds  [default: MODEL]
  -rname, --residual-name TEXT    Name of residual in dds  [default: RESIDUAL]
  -nb, --nband INTEGER            Number of imaging bands  [required]
  -o, --output-filename BASENAME  Basename of output  [required]
  --postfix TEXT                  Can be used to specify a custom name for the
                                  image space data \ products
  -p, --product TEXT              Imaging products to produce. Options are I,
                                  Q, U, V. \ Only single Stokes products are
                                  currently supported  [default: I]
  --outputs TEXT                  Output products. (m)odel, (r)esidual,
                                  (i)mage, (c)lean beam. \ Captitals
                                  correspond to cubes.  [default: mMrRiI]
  --overwrite / --no-overwrite    Allow overwrite of output xds  [default: no-
                                  overwrite]
  --help                          Show this message and exit.

Note it currently writes out image cubes, I have not yet modified it to write a fits file per imaging band. I will modify that when I get a chance.

I will post more findings and results here as I continue to experiement with the imager. I really like the philosophy of it and I'm looking forward to getting to grips with it.

Thanks a lot for testing. I hope it will be worth your time. I am looking into why clean is so slow #70. I already have a number of optimisations in mind. In the meantime, can I suggest that you install ducc (https://gitlab.mpcdf.mpg.de/mtr/ducc) in development mode. This should speed up gridding and FFT's because it will compile for the specific hardware you are running on

0 replies

IanHeywood · 2022-12-06T20:19:59Z

IanHeywood
Dec 6, 2022
Collaborator Author

Reinstalled ducc and disabled flux mopping, and it was much faster this time.

INFO      12:50:47 - CLEAN              | Initialising ThreadPool with 8 threads
INFO      12:50:47 - CLEAN              | Input Options:
INFO      12:50:47 - CLEAN              |                        epsilon = 1e-07
INFO      12:50:47 - CLEAN              |                         wstack = True
INFO      12:50:47 - CLEAN              |                   double_accum = True
INFO      12:50:47 - CLEAN              |                   host_address = None
INFO      12:50:47 - CLEAN              |                       nworkers = 8
INFO      12:50:47 - CLEAN              |            nthreads_per_worker = 1
INFO      12:50:47 - CLEAN              |                      nvthreads = 5
INFO      12:50:47 - CLEAN              |                       nthreads = 40
INFO      12:50:47 - CLEAN              |                      scheduler = threads
INFO      12:50:47 - CLEAN              |                          nband = 8
INFO      12:50:47 - CLEAN              |                output_filename = pfbtest
INFO      12:50:47 - CLEAN              |                        postfix =
INFO      12:50:47 - CLEAN              |                        product = I
INFO      12:50:47 - CLEAN              |                  residual_name = RESIDUAL
INFO      12:50:47 - CLEAN              |                     model_name = MODEL
INFO      12:50:47 - CLEAN              |                           mask = None
INFO      12:50:47 - CLEAN              |                           algo = clark
INFO      12:50:47 - CLEAN              |                       dirosion = 1
INFO      12:50:47 - CLEAN              |                       fits_mfs = True
INFO      12:50:47 - CLEAN              |                     fits_cubes = False
INFO      12:50:47 - CLEAN              |                       mop_flux = False
INFO      12:50:47 - CLEAN              |                         nmiter = 5
INFO      12:50:47 - CLEAN              |                      threshold = None
INFO      12:50:47 - CLEAN              |                 sigmathreshold = 2.0
INFO      12:50:47 - CLEAN              |                          gamma = 0.1
INFO      12:50:47 - CLEAN              |                    peak_factor = 0.15
INFO      12:50:47 - CLEAN              |                sub_peak_factor = 0.75
INFO      12:50:47 - CLEAN              |                   hogbom_maxit = 5000
INFO      12:50:47 - CLEAN              |                    clark_maxit = 50
INFO      12:50:47 - CLEAN              |                      sub_maxit = 1000
INFO      12:50:47 - CLEAN              |                        verbose = 1
INFO      12:50:47 - CLEAN              |                    report_freq = 10
INFO      12:50:47 - CLEAN              |                         cg_tol = 0.01
INFO      12:50:47 - CLEAN              |                       cg_maxit = 100
INFO      12:50:47 - CLEAN              |                       cg_minit = 1
INFO      12:50:47 - CLEAN              |                     cg_verbose = 1
INFO      12:50:47 - CLEAN              |                 cg_report_freq = 100
INFO      12:50:47 - CLEAN              |                      backtrack = True
INFO      12:50:47 - CLEAN              |                  memory_greedy = False
INFO      12:52:20 - CLEAN              | Iter 0: peak residual = 2.128e-01, rms = 3.085e-04
INFO      12:52:20 - CLEAN              | Running Clark
INFO      13:31:39 - CLEAN              | Getting residual
INFO      13:34:01 - CLEAN              | Iter 1: peak residual = 2.982e-02, rms = 1.438e-04
INFO      13:34:01 - CLEAN              | Updating results
INFO      13:34:28 - CLEAN              | Running Clark
INFO      14:07:01 - CLEAN              | Getting residual
INFO      14:09:44 - CLEAN              | Iter 2: peak residual = 4.070e-03, rms = 6.144e-05
INFO      14:09:44 - CLEAN              | Updating results
INFO      14:10:17 - CLEAN              | Running Clark
INFO      14:44:48 - CLEAN              | Getting residual
INFO      14:47:13 - CLEAN              | Iter 3: peak residual = 6.463e-04, rms = 3.100e-05
INFO      14:47:13 - CLEAN              | Updating results
INFO      14:47:42 - CLEAN              | Running Clark
INFO      17:20:04 - CLEAN              | Getting residual
INFO      17:22:39 - CLEAN              | Iter 4: peak residual = 5.664e-04, rms = 2.315e-05
INFO      17:22:39 - CLEAN              | Updating results
INFO      17:23:10 - CLEAN              | Running Clark
INFO      19:44:00 - CLEAN              | Getting residual
INFO      19:46:18 - CLEAN              | Iter 5: peak residual = 1.539e-04, rms = 2.149e-05
INFO      19:46:18 - CLEAN              | Updating results
INFO      19:46:49 - CLEAN              | Writing fits
INFO      19:47:06 - CLEAN              | All done here.

Images pending...

0 replies

IanHeywood · 2022-12-06T20:50:49Z

IanHeywood
Dec 6, 2022
Collaborator Author

The residual isn't the deepest, but the cleaning ran wild over those sidelobes which are all in the model. I'll try next with a FITS mask (hopefully I have the original wsclean one somewhere) and re-enabling flux mop.

Using pfb restore --nband 8 --outputs i -o pfbtest didn't behave as I was expected. I thought the i would be the usual restored image (convolved model plus the residuals) but it only seems to have convolved the model:

0 replies

landmanbester · 2022-12-07T10:48:43Z

landmanbester
Dec 7, 2022
Maintainer

Ah, good catch. I introduced a double normalisation by wsum in the restore worker so the residuals are there they are just very tiny. I have pushed a fix that will go in with #74. This should siginificantly seed up your cleaning when using large (> 20k) images. As discussed in #70 there was some weirdness in how I was doing the padding and fftshifting. You will hopefully have a more stable memory profile now as well

0 replies

IanHeywood · 2024-04-08T11:41:39Z

IanHeywood
Apr 8, 2024
Collaborator Author

I'll resurrect this thread for recent adventures...

0 replies

IanHeywood · 2024-04-08T15:26:10Z

IanHeywood
Apr 8, 2024
Collaborator Author

On the ox-box:

(pfb) ianh@ndidi:/mnt/ndidi/ianh/GC/UHF_PFB$ ./hash_column.py 1590089458_sdp_l0_1024ch_SGRA.ms CORRECTED_DATA
1590089458_sdp_l0_1024ch_SGRA.ms CORRECTED_DATA
Successful readonly open of default-locked table 1590089458_sdp_l0_1024ch_SGRA.ms: 28 columns, 1851360 rows
Hash for column 'CORRECTED_DATA' is c5c2499c970a89111b5f0d419c7c5355ad79133b20faa655adee4c8c938cb990
(pfb) ianh@ndidi:/mnt/ndidi/ianh/GC/UHF_PFB$ python --version
Python 3.10.12

on nash:

(pfb) ianh@nash:~/GC$ ./hash_column.py 1590089458_sdp_l0_1024ch_SGRA.ms CORRECTED_DATA
1590089458_sdp_l0_1024ch_SGRA.ms CORRECTED_DATA
Successful readonly open of default-locked table 1590089458_sdp_l0_1024ch_SGRA.ms: 28 columns, 1851360 rows
Hash for column 'CORRECTED_DATA' is 64864ee80f6f5d46ffa3f486e25b93fc71e9f54029ecc6a9199225243ff441e7
(pfb) ianh@nash:~/GC$ python --version
Python 3.9.18

Mea culpa buddy. I'll try to figure out what went wrong here... Model restoration perhaps.

I still don't understand why I got two different runs on nash though, maybe some git pulls in between them. I'll keep trying.

2 replies

landmanbester Apr 8, 2024
Maintainer

Yeah maybe the combination of rsyncing and using different pfb versions. I would nuke the MS on nash (I have a backup on bbking anyway) and start the transfer afresh. Then just run the init step on nash so I can look at the resulting xds, it should be the same as the one you copied from the ox box. If not I have a bug to hunt down. I've also had issues with python3.9 so I would just go for 3.10 to keep things consistent. Thanks again for testing

landmanbester Apr 11, 2024
Maintainer

╭─bester@bbking ~/projects/GC/msdir
╰─➤  ./hash_column.py /net/nash/home/ianh/GC/1590089458_sdp_l0_1024ch_SGRA.ms CORRECTED_DATA
/net/nash/home/ianh/GC/1590089458_sdp_l0_1024ch_SGRA.ms CORRECTED_DATA
Successful readonly open of default-locked table /net/nash/home/ianh/GC/1590089458_sdp_l0_1024ch_SGRA.ms: 28 columns, 1851360 rows
Hash for column 'CORRECTED_DATA' is c5c2499c970a89111b5f0d419c7c5355ad79133b20faa655adee4c8c938cb990

The hash of the new ms on naah looks the same as the one on the ox box so hooray!

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Ian's pfb tests #72

{{title}}

Replies: 7 comments 2 replies

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Select a reply

Ian's pfb tests #72

IanHeywood Dec 5, 2022 Collaborator

Replies: 7 comments · 2 replies

IanHeywood Dec 5, 2022 Collaborator Author

landmanbester Dec 5, 2022 Maintainer

IanHeywood Dec 6, 2022 Collaborator Author

IanHeywood Dec 6, 2022 Collaborator Author

landmanbester Dec 7, 2022 Maintainer

IanHeywood Apr 8, 2024 Collaborator Author

IanHeywood Apr 8, 2024 Collaborator Author

landmanbester Apr 8, 2024 Maintainer

landmanbester Apr 11, 2024 Maintainer

IanHeywood
Dec 5, 2022
Collaborator

Replies: 7 comments 2 replies

IanHeywood
Dec 5, 2022
Collaborator Author

landmanbester
Dec 5, 2022
Maintainer

IanHeywood
Dec 6, 2022
Collaborator Author

IanHeywood
Dec 6, 2022
Collaborator Author

landmanbester
Dec 7, 2022
Maintainer

IanHeywood
Apr 8, 2024
Collaborator Author

IanHeywood
Apr 8, 2024
Collaborator Author

landmanbester Apr 8, 2024
Maintainer

landmanbester Apr 11, 2024
Maintainer