Updating WRF-DART tutorial obs converter doc and NCEP Prepbufr converter docs #823
Conversation
Generating quickstart instructions based on WRF-DART step 3 conversion instructions
Moving prepbufr converter instructions from WRF-DART tutorial to prepbufr docs. Add warning that instructions to reproduce the WRF-DART tutorial obs are not provided.
The previous WRF-DART tutorial obs conversion section was migrated to prep_bufr and provides full example of raw prepqm file conversion to obs_seq.
braczka
added
Documentation
|
@mjs2369 i added you as a reviewer because of your good work with github issue 634. This PR likely needs more warnings and links related to the compiling/software issues outlined there. Not intended as a fix for 634, but a good upgrade in NCEP prepbufr instructions. |
|
@braczka do you have someone who works with prepbufr data who can review this? Someone on the team who regularly uses prepbufr data in assimilations? |
models/wrf/tutorial/README.rst
Outdated
| See `prepbufr <../../../observations/obs_converters/NCEP/prep_bufr/prep_bufr.html>`__ | ||
| - Run PREPBUFR text conversion scripting (``prepbufr.csh``) | ||
| - Run text (ascii) to obs_seq executable (``create_real_obs``) | ||
| See `ascii_to_obs <../../../observations/obs_converters/NCEP/ascii_to_obs/create_real_obs.html>`__ |
There was a problem hiding this comment.
use :ref: or :doc: rather than .html links
| `create_real_obs <../../../observations/obs_converters/NCEP/ascii_to_obs/create_real_obs.html>`__ | ||
| page. |
There was a problem hiding this comment.
:ref: or :doc: rather than html.
|
@hkershaw-brown Not sure if 'regular' is the word, but @kdraeder or @mgharamti have done this in the past I think. |
|
I didn't merge in the latest changes from main, and I am getting a lint_for_svn_on_pr. Should I merge locally and re-push? |
| Use a similar approach to obtain the `RDA ds377 <https://rda.ucar.edu/datasets/d337000/>`__ | ||
| prepbufr files. Locate and download the GDAS PREPBUFR files of your choice. | ||
|
|
There was a problem hiding this comment.
Thanks for mentioning this. I just realized I blew off @kdraeder , request for comment on the RDA PR a couple weeks back. Sorry !! -- this was unintentional, I am trying to juggle too many things right now. I will restore the original link that Kevin generated.
| - It is helpful to also run the | ||
| `wrf_dart_obs_preprocess <../../../models/wrf/WRF_DART_utilities/wrf_dart_obs_preprocess.html>`__ | ||
| program, which can strip away observations not in the model domain, |
There was a problem hiding this comment.
This seems pretty wrf specific, is this the general advice for prep_bufr observations when using other models.
Yup stay up to date with main. You can merge locally and push (make sure your local main is up to date with NCAR/main)
The lint_for_svn_on_pr is not required to pass, it just flags any legacy svn info that is hanging around in files. |
|
Brett, thanks for working on this. |
Thanks @mgharamti, appreciate it. I also got a little loose with my use of the cosblocking tool -- I think I just took Nancy's and put it in my work directory, but maybe should have a better location for it. There is also still some redundancies between the 'Quickstart' section and the 'Prepbufr Overview' section, that I didn't fully address. |
|
some history on the cosconvert tool here: i talked to bob dattore in DSS about the tool because his name is on the source. he said it either has no license, or it could have a creative commons license, but we were free to do whatever we wanted with it. i have a copy in my home dir because i needed it for uncosblocking some older files. maybe a copy of it should go into the prep_bufr directory? it's probably not needed for more recent files, but if it is needed it might be hard to track down the source code again sometime in the future. |
There was a problem hiding this comment.
Brett, I appreciate all the efforts that went into this. This is beyond a doubt a very useful addition to the prepbuf obs saga. I have a bunch of comments and suggestions that will help clarify all of the obs preparation steps. I'd be happy to take another look after making the changes.
| cd $DART_DIR/observations/obs_converters/NCEP/prep_bufr | ||
| ./install.sh | ||
|
|
||
| Confirm the exe directory contains the executables ``prepbufr.x``, ``prepbufr_03z.x``, |
There was a problem hiding this comment.
make sure naming of the executable is correct: prepbufr_03Z.x (Z upper case)
| :: | ||
|
|
||
| cd $DART_DIR/observations/obs_converters/NCEP/prep_bufr/work | ||
| ./quickbuild.sh |
There was a problem hiding this comment.
I would add a sentence after running quickbuild telling the users what new executables they expect to find in the work directory. Is it only advance_time (and preprocess)?
|
|
||
|
|
||
| - Obtain the PREPBUFR observations for your desired time. You can go to the | ||
| the campaign collections directory as: |
There was a problem hiding this comment.
Add a sentence saying that in this example, we will download data from the month of June, 2016.
|
|
||
| :: | ||
|
|
||
| /glade/u/home/bmraczka/cosconvert -b A25366-201706prepqmB.tar |
There was a problem hiding this comment.
So, if I recall there was an issue with the cosconvert provided by RDA. I also have a copy of the same executable in home directory. For the sake of this documentation, I suggest having a copy on /glade/campaign/cisl/dares/Observations/bufr/progs. We can point the users to use the one in our team campaign directory rather than our own. So, I would re-write the cos-unblocking step as:
/glade/campaign/cisl/dares/Observations/bufr/progs/cosconvert -b A25366-201706prepqmB.tar
|
|
||
| :: | ||
|
|
||
| /glade/u/home/bmraczka/cosconvert -b A25366-201706prepqmB.tar |
There was a problem hiding this comment.
Didn't you rename the tar file to prepqm in the previous copying step?
cp A25366-201706prepqmB.tar $DART_DIR/observations/obs_converters/NCEP/prep_bufr/data/prepqm
The cosconvert won't work because the name of the file is different. I suggest keeping the original name. Unless you actually created a directory called prepqm inside observations/obs_converters/NCEP/prep_bufr/data. I would also tell the user to run cosconvert in the DART directory not the RDA directory. My suggestion is to be more verbose here.
|
|
||
| cd $DART_DIR/observations/obs_converters/NCEP/prep_bufr/work | ||
| cp ../exe/\*.x . | ||
| ./prepbufr.csh 2017 04 27 |
There was a problem hiding this comment.
Because, we unzipped the files in previous steps they have a .nr extension. Executing the ./prepburf.csh 2017 06 27 with result in an error:
MISSING INPUT FILE: cannot find either ../data/prepqm/prepqm17062706 or ../data/prepqm/prepqm17062706.gz Script will abort now.
Perhaps, edit this as follows:
set BUFR_in = ${BUFR_idir}/prepqm${sdtg}.nr
| format. Confirm that ``temp_obs.20170427*`` files within your | ||
| ``~/data/prepout directory`` exist. Please note that the script can function | ||
| with only the *06* prepqm input file, but will need the | ||
| *12* and *18* files to run to completion. |
There was a problem hiding this comment.
The script did the 06, 12, and 18Z files but then aborted because it asked for the obs at 2017062800 file. This could be fixed perhaps by changing files_per_day in prepbufr.csh but no big deal. I think the idea is clear.
There was a problem hiding this comment.
the scripting here is old and creaky. i think even if you don't ask for a "daily" file, that it still wants to process a day's worth of obs, which will output 4 intermediate files.
| max_num = 800000, | ||
| select_obs = 0, | ||
| ObsBase = '../../prepbufr/data/prepout/temp_obs.', | ||
| daily_file = .false., |
There was a problem hiding this comment.
It might be useful to describe this parameter. If this is set to true, then the program will look for temp_obs.{yyyy}{mm}{dd} files but since we do 6-hourly cycling we set this to .false.
| :: | ||
|
|
||
| cd $DART_DIR/observations/obs_converters/NCEP/ascii_to_obs/work | ||
| ./create_real_obs |
There was a problem hiding this comment.
I ran this and it created the obs sequence files for 06, 12, and 18Z. But it also gave me the following. Is it something we should look into? Or did I do something wrong?
ERROR FROM:
source : utilities_mod.f90
routine: open_file:
message: Cannot open file "../../prep_bufr/data/prepout/temp_obs.2017062724" for read
message: ... File may not exist or permissions may prevent the requested operation
message: ... Error code was 29
There was a problem hiding this comment.
hi moha - i think this may be related to the issue that observations directly on the boundary time between 6H windows can occur in either the previous or following file. i think the converter tries to open the next file to collect any obs which are on the initial time boundary. OR - it's related to the script trying to do 4 6H files at a time if you pick "daily=false".
There was a problem hiding this comment.
i just went through the prepbufr script again. if you say "daily=true" it makes a single 24 hour output file. if you say "daily=false" it still tries to convert a full day's worth of data but outputs 4 6-hour files. to finish up a day, it needs the 0Z and 6Z files from the following day to be available. (it creates the 6, 12, and 18 h files just fine, but ends with an error when it tries to open the 0Z and 6Z file from the next day.)
| into a readable tar file. On the NSF NCAR machine *yellowstone*, run: | ||
| | *> /glade/u/home/rdadata/bin/cosconvert -b rawfile data.tar* | ||
| into a readable tar file. On the NSF NCAR machine Derecho run: | ||
| | *> /glade/work/bmraczka/cosconvert -b data.tar* |
There was a problem hiding this comment.
See my previous comments on cosconvert
|
Based on standup conversation -- I put this on draft such that I can implement conversion instructions using RDA download and avoid use of cosconvert tool if possible. |
|
Latest commit includes fixes based on Helen/Moha comments. We may update the docs to switch from campaign to RDA download if we can avoid cosconvert. |
|
i'm sorry for the confusion. i had thought the files on the public RDA site were already unblocked and it was just because we asked for special versions that we had to do the extra step. but i just downloaded a couple files directly off the public RDA site and they also needed to be unblocked before they could be read. so i'm proposing we leave the derecho instructions as-is -- that people can just copy the files off campaign storage and unblock them before they continue. the instructions for how to download them off the RDA website are still there for people who aren't on derecho. |
|
Thank you Nancy -- I will keep the instructions as is, and add a few notes about the ds090, ds377 and CAM6 reanalysis observation differences. We should discuss at tomorrow's standup what we are OK to release to the user. Sorry - I am still a bit fuzzy on the discussion yesterday. Are only the newest ACAR obs restricted ? |
Description:
Removed the NCEP atmospheric prepbufr observation conversion (optional instructions) in Step 3 of the WRF-DART tutorial because they were not designed to replicate the tutorial obs_seq file and were incorrect. Condenses step 3 to only a summary of major conversion steps and provides links to prepbufr converter and create_real_obs 2 stage converters in the observation converter documentation.
I modified the step 3 instructions and inserted them into a 'Quickstart Section' of the prepbufr conversion section. This provides the users with all necessary steps in the 2 stage conversion process, including the download of the raw prepqm file from campaign storage and/or RDA download for ds0900 or ds370 archived data. This helps clarify the confusion of the 2 stage conversion process (prepqm --> ascii text --> obs_seq) by having all steps in one place. This does not resolve the software issues related to github issue 634, (gfortran issues etc) but is an improvement, because it provides a complete conversion example using Derecho and intel compiler.
Fixes issue
This fixes issue 792 and improves (but does not completely fix) issue 634
Types of changes
Documentation changes needed?
Tests
I have run the quickstart instructions on Derecho using the intel compiler.
Checklist for merging
Checklist for release
Testing Datasets
Instructions provided on how to obtain prepqm from campaign or from RDA archive.