Use a temp pass through errhandler for trivial errors causing abort

[islas]

TYPE: bug fix

KEYWORDS: mpi, quilting, comm

SOURCE: internal

DESCRIPTION OF CHANGES:
Problem:
PR #2157 added changes to match an appropriate MPI_Datatype to a specific typesize during col_on_comm() and dst_on_comm(). This relies on MPI_Type_match_size() to query MPI about the equivalent MPI definition for a particular datatype size. There are safety checks to query MPI_TYPECLASS_INTEGER if MPI_TYPECLASS_REAL fails.

However, when given a datatype size that does not match a possible MPI_TYPECLASS_REAL value (e.g. 1 byte where no real exists for single byte) instead of getting a failure via return code the query is treated as a critical failure and fully aborts the program. As the query does not rely on critical process handling and since there already exists adequate checks to abort if no sufficient value is found, this preemptive abort is unnecessary.

Solution:
Temporarily install a pass through errhandler that does not modify the return code but also does not abort. Allow the if statements of finding a correct MPI_Datatype to abort if deemed necessary. Additionally, once the checks are complete, reinstate any previous errhandler and free our pass through handle.

ISSUE:
#2225

TESTS CONDUCTED:

1. Tested the stability of this call to handle correct and incorrect types various times while constantly replacing the error handler.

RELEASE NOTE:
In collect_on_comm.c, use a temporary pass through errhandler to allow MPI_Type_match_size to fail correctly with error code rather than fully abort the program.

[mgduda]

If my understanding is correct, MPI_Errhandler_get and MPI_Errhandler_set have been deprecated, and MPI_Comm_get_errhandler and MPI_Comm_set_errhandler are the MPI-2 replacements. See https://www.mcs.anl.gov/research/projects/mpi/mpi-standard/mpi-report-2.0-sf/mpi2-report.htm#Node18 .

[islas]

Ah, yeah I see now. I found the same for MPI_Comm_create_errhandler() but had not looked closely at the get/set functions as I didn't get a deprecation warning on those. Interestingly there is no new call for the MPI_Errhandler_free()

[weiwangncar]

The Jenkins test results:

Test Type              | Expected  | Received |  Failed
= = = = = = = = = = = = = = = = = = = = = = = =  = = = =
Number of Tests        : 23           24
Number of Builds       : 60           57
Number of Simulations  : 158           150        0
Number of Comparisons  : 95           86        0

Failed Simulations are: 
None
Which comparisons are not bit-for-bit: 
None

[weiwangncar]

I passed the routine to another user who has problem with io_quilting on Derecho and it worked for him!