Getting Started

Third Party Libraries

MrHyDE is built upon Trilinos, which does require a few third-party libraries. Most of these libraries should be available on any HPC system through modules, or on a Mac using Homebrew. Before downloading MrHyDE and Trilinos, we suggest loading or installing these the following third party libraries: CMake, openmpi, zlib, netcdf, pnetcdf, Boost, HDF5 (probably mpi version), libx11, ninja. As of December 2024, Trilinos and MrHyDE are able to build on a Mac laptop using the Homebrew installed versions of these libraries. It might be possible to build Trilinos using older versions, but certain minimum versions are required for certain Trilinos packages. In addition, there are sometimes issues with HDF5 and HDF5-mpi both installed on the same machine and Trilinos failing to find the correct one. This can usually be mitigated by setting the appropriate library directory in the Trilinos configure script.

MrHyDE can be built using gcc, Intel, or Clang compilers. On a mac, we recommend Clang, but on the HPC systems there is little difference between the three. Some specialized algorithms in Trilinos may perform better using the Intel compilers, but MrHyDE does not often use these algorithms.

While MrHyDE uses Trilinos extensively, it treats Trilinos like another third-party library and does not use the tribits tools to build along-side Trilinos. This is by design and allows use to utilize other third-party packages if we need to.

Downloading Trilinos

Trilinos is an open source collection of packages stored on GitHub. The repository can be cloned using:

git clone https://github.com/trilinos/Trilinos.git

Downloading MrHyDE

MrHyDE is an open source software framework currently stored in a repository on Sandia’s external github site. The repository can be cloned using:

git clone [email protected]:sandialabs/MrHyDE.git

Building Trilinos

Creating an appropriate build of Trilinos is likely to be the biggest challenge for new users. On a personal machine, Trilinos will need to be built from source. While Trilinos/MrHyDE can be configured and optimized for a hybrid CPU/GPU environment, we strongly recommend starting with a standard build on either Mac or Linux machine first. At this time, building on Microsoft Windows is not supported.

Several examples of configure scripts are located under MrHyDE/scripts/configure-trilinos The most recent Mac build using the Apple M3 Max processor used configure-trilinos-mac-M3-serial Note that the paths will probably need to be changed for your system.

To build Trilinos, take the following steps:

1. Create a build directory someplace on your machine. The configure scripts probably assume that this will be under <YOUR PATH>/Trilinos/build but you could place this anywhere as long as the paths are correct in the configure script.
2. Go into the build directory.
3. Configure Trilinos (assuming your script is called configure): ./configure. This should generate quite a bit of Trilinos settings and tests for compilers and TPLs. This creates the CMake information required for a build. If the script fails, then you will need to fix the errors before Trilinos can build.
4. If Trilinos has been configured properly, then build using ninja: ninja
5. Once this completes, install using: ninja install. This will install into the directory given in the configure script. You will need to know this location for your MrHyDE build.

Building MrHyDE

Building MrHyDE follows a very similar process, and if your Trilinos build was successful, then it is highly likely that your MrHyDE will also build successfully. Several examples of configure scripts are located under MrHyDE/scripts/configure-mrhyde The most recent Mac build using the Apple M3 Max processor used configure-mrhyde-mac-M3-serial Note that the paths will probably need to be changed for your system, and your Trilinos install path needs to be wherever you installed Trilinos.

To build MrHyDE, take the following steps:

1. Create a build directory someplace on your machine. The configure scripts probably assume that this will be under <YOUR PATH>/MrHyDE/build but you could place this anywhere as long as the paths are correct in the configure script.
2. Go into the build directory.
3. Configure MrHyDE (assuming your script is called configure): ./configure. MrHyDE uses a few CMake flags, but this should be much faster than the Trilinos configure process.
4. If MrHyDE has been configured properly, then build using ninja: ninja
5. (Optional) You can install the MrHyDE executable on your machine, but this is usually not necessary. I typically create a soft link to the executable from wherever I am running from.

Testing MrHyDE

After MrHyDE builds, it is a good idea to run the regression tests to make sure everything works properly on your machine. Take the following steps:

1. Go into MrHyDE/regression
2. Create a soft link to the MrHyDE executable: ln -s ../build/src/mrhyde
3. Use python3 (not v2) to run the regression tests: python3 runtests.py This should generate a series of tests, over 100, that cover most of the capabilities in the code to some extent. All should pass. If you see any that fail, inspect why by comparing mrhyde.log with mrhyde.gold in the directory for the test that failed. Sometimes a test will fail due to precision errors, which is not a concern, but sometimes a larger issue may need to be addressed.

By design, MrHyDE is fairly quiet in terms of output to the screen. This is due to the emphasis on BFS capabilities, which typically require several forward model evaluations. Like most of the Trilinos packages, the amount of screen output is controlled via a verbosity flag. Setting the verbosity to at least 10 will output linear/nonlinear solver progress, time integration status, and the Teuchos timers. Another flag, shown above, is the debug level. This is useful for developers who want to dig into the code progress without having to recompile. Setting this flag to 1 will print a statement when starting and finishing most constructors and interface/manager functions that only get called once, typicall during the set up phase. Setting this flag to 2 will also print when entering/leaving functions that get called many times, typically during the solve phase. Finally, setting this flag to 3 will also print vectors, matrices and certain Kokkos views.