-
Notifications
You must be signed in to change notification settings - Fork 4
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Collect and print/save timings #276
Conversation
This probably should have been deleted when the 2d branch was merged.
So far these timers time some MPI communication operations, quite a lot of operations in `nonlinear_solvers`, and preconditioner factorisation or application in `electron_kinetic_equation`.
Need to handle this specially, because NetCDF and/or NCDatasets.jl do not support it.
Do not need/use open files in this case (they are reopened as necessary), so actually close all the files.
One thing to be aware of with this PR - in order to save all the timers that get created to the output files, we have to gather a list of them, possibly update the list for the first couple of output steps, and MPI-communicate the list to all the MPI ranks that have to write to the output file. The code that does all that could be a little bit fragile if something happens that wasn't anticipated when it was written. Hopefully in getting all the tests to pass I've worked out those kinks, but if you get errors that come from |
Useful to monitor for performance regressions.
Timers are printed at the end of a run (this can be disabled by setting
display_timing_info=false
in the[output]
section), and saved to the output file. They can be plotted usingmakie_post_processing
, see the docs.Also, the
Profile
package seemed to be interacting badly with MPI in one case I tried - the run was much slower with profiling on, with a lot of time spent in MPI calls - so 'standard' profiling isn't necessarily representative and this feature provides an alternative. Hopefully this is a temporary bug in some external package(s) which will go away at some point, because sampling profiles are still pretty useful!Provides an option to switch on extra 'debug timers', see the docs.