Add parallel write of atlas fields on the root processor #84
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
* update field writer tests * add write server * add single proc and parallel write server tests * test ORCA indexing * standardise OrcaIndex x and y dim names * mpi safe assert, reduce OrcaIndex class * reduce pre-allocation and better const correctness
s-good
approved these changes
Mar 26, 2024
twsearle
force-pushed
the
feature/parallel-write-server
branch
2 times, most recently
from
da3c90d to
3792d07
Compare
twsearle
force-pushed
the
feature/parallel-write-server
branch
from
3792d07 to
a1b9681
Compare
Thanks very much for the review! It exposed a few things I missed. I will re-run mo-bundle to double check everything is OK before merging, but if you would rather re-review please can you press the 🔄 button next to your name in the reviewer section and add a new review. Thanks again! I am looking forward to getting this through to make testing the shelf model interface more easily, and I am really hoping I can use a similar approach to gather data in the ReadServer if we need to improve io performance further. |
s-good
approved these changes
Mar 27, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
Add parallel write of atlas fields for any partitioner. Data is written by the root processor only, with each processor sending its owned data to the root processor, where it is sorted before being written to the file. The ORCA logic is now abstracted out of the field writer to allow for the addition of non-orca grids. There is also some small refactoring to the reading classes to unify the naming and approaches and hopefully make things a little more clear.
Impact
This change ought to help with parallel 3DVar changes, when these are added by Dan Lea.
Checklist