Unify FM over all grids #6981

tcclevenger · 2025-02-07T01:17:22Z

Store a single FieldManager for all fields over all grids.

This solves the issue in #6789 where having a group as a subset of another, and allocated on different grids, was causing subview indices mismatch between grids.

Also

Remove Derivation type. Now groups contain the union of all fields registered over all grids. I.e., requesting "tracers" on GLL grid without registering any tracers on GLL will give all tracers registered on PG2 grid.
Remove "Preferred" bundling. You either require bundling or not.

Also - remove devivation type - simplify group logic to ensure subfields over grids have same index

tcclevenger

A few comments highlight the important pieces since this is such a large PR. So many of the changes come from every instance of the FM has to be used differently (passing grid names instead of only one grid being stored).

tcclevenger · 2025-02-07T01:21:33Z

components/eamxx/src/control/atmosphere_driver.cpp

-  //   we register a corresponding version of the field on the "target" FM.
-
-  // Helper lambda to reduce code duplication
-  auto process_imported_groups = [&](const std::set<GroupRequest>& group_requests) {


I moved this logic to FieldManager::pre_process_groups().

tcclevenger · 2025-02-07T01:27:12Z

components/eamxx/src/share/field/field.cpp

+Field::clone(const std::string& name) const {
+  return clone(name, get_header().get_identifier().get_grid_name());
 }

 Field
-Field::clone(const std::string& name) const {
+Field::clone(const std::string& name, const std::string& grid_name) const {
  // Create new field
  const auto& my_fid = get_header().get_identifier();
  FieldIdentifier fid(name,my_fid.get_layout(),my_fid.get_units(),
-                      my_fid.get_grid_name(),my_fid.data_type());
+                      grid_name,my_fid.data_type());


I needed to add a grid argument to Field::clone() because, in many cases, we are cloning a grid (giving it a new name), constructing a FM(new_grid), but using a field whose ID gives a grid name from the old grid, and the new FM was complaining that the grid was not in the internal grids manager.

tcclevenger · 2025-02-07T01:28:26Z

components/eamxx/src/share/field/field.cpp

@@ -185,7 +190,7 @@ Field Field::subfield(const std::string& sf_name,
      "Error! Input field must be allocated in order to subview it.\n");

  auto sf_layout = lt.clone();
-  sf_layout.reset_dim(idim, index_end - index_beg);
+  sf_layout.reset_dim(idim, index_end+1 - index_beg);


Kokkos::subview(..., beg, end, ...) has inclusive bounds, but we were treating the subview bound as [bed, end).

tcclevenger · 2025-02-07T01:28:41Z

components/eamxx/src/share/field/field_alloc_prop.cpp

@@ -88,7 +88,7 @@ FieldAllocProp FieldAllocProp::subview(const int idim,
  EKAT_REQUIRE_MSG(index_beg < index_end,
                   "Error! Slice indices are invalid (non-increasing).\n");
  EKAT_REQUIRE_MSG(
-      index_beg >= 0 && index_end < m_layout.dim(idim),
+      index_beg >= 0 && index_end <= m_layout.dim(idim),


Same as comment above about Kokkos bounds

tcclevenger · 2025-02-07T01:29:38Z

components/eamxx/src/share/field/field_layout.hpp

@@ -117,6 +117,7 @@ class FieldLayout {
  // calls get_vector_component_idx()
  int get_vector_dim () const;
  FieldTag get_vector_tag () const;
+  std::vector<std::string> get_dim_names() const;


Needed by the new equivalent_layout() function.

tcclevenger · 2025-02-07T01:36:12Z

components/eamxx/src/share/field/field_manager.cpp

          info.m_subview_idx [*it] = std::distance(cluster_ordered_fields.begin(),it);
        }
        info.m_bundled = true;
-      }
-    }
-  }



Now indices info is independent of grid, guaranteeing each grid agrees.

tcclevenger · 2025-02-07T01:38:19Z

components/eamxx/src/share/field/field_manager.cpp

-  return it==m_fields.end() ? nullptr : it->second;
+FieldManager::get_field_ptr (const std::string& name, const std::string& grid_name) const {
+  auto it = m_fields.at(grid_name).find(name);
+  return it==m_fields.at(grid_name).end() ? nullptr : it->second;
 }

 void FieldManager::pre_process_group_requests () {


Important function for gathering which grids should register which group fields

tcclevenger · 2025-02-07T01:39:04Z

components/eamxx/src/share/field/field_manager.hpp

-  // the parent field. So at registration time, simply keep track of the subfields,
-  // and create them at registration_ends() time, after all other fields.
-  std::map<std::string,FieldRequest> m_subfield_requests;
-
  // The map group_name -> FieldGroupInfo
  group_info_map      m_field_groups;


Doesn't change, group info is the same for all grids.

tcclevenger · 2025-02-07T01:41:29Z

components/eamxx/src/share/grid/abstract_grid.cpp

@@ -115,6 +115,53 @@ AbstractGrid::get_3d_tensor_layout (const bool midpoints, const std::vector<int>
  return get_3d_tensor_layout(midpoints,cmp_dims,names);
 }

+FieldLayout
+AbstractGrid::equivalent_layout (const FieldLayout& template_layout) const


This is how we get the layout for tracer on a new grid.

tcclevenger · 2025-02-07T01:42:57Z

components/eamxx/src/share/grid/grids_manager.cpp

+std::set<std::string> GridsManager::
+get_grid_names () const {
+  std::set<std::string> names;
+  if (m_grids.size()==0) {
+    return names;
+  }
+  for (const auto& g : m_grids) {
+    names.emplace(g.second->name());
+  }
+  return names;
+}


I ended up using this function a lot, but I don't know that I love it. It doesn't take into account aliases. I could instead always loop over the grid manager's grid map and query the names from there.

tcclevenger · 2025-02-07T01:51:37Z

Quite a few fails in the CI, odd some didn't trigger on my workstation (rrtmgp standalone for instance)... Leaving Draft for now.

Unify FM over all grids

769af94

Also - remove devivation type - simplify group logic to ensure subfields over grids have same index

tcclevenger marked this pull request as draft February 7, 2025 01:17

tcclevenger commented Feb 7, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Unify FM over all grids #6981

Unify FM over all grids #6981

tcclevenger commented Feb 7, 2025

tcclevenger left a comment

tcclevenger Feb 7, 2025

tcclevenger Feb 7, 2025

tcclevenger Feb 7, 2025

tcclevenger Feb 7, 2025

tcclevenger Feb 7, 2025

tcclevenger Feb 7, 2025

tcclevenger Feb 7, 2025

tcclevenger Feb 7, 2025

tcclevenger Feb 7, 2025

tcclevenger Feb 7, 2025

tcclevenger commented Feb 7, 2025

Unify FM over all grids #6981

Are you sure you want to change the base?

Unify FM over all grids #6981

Conversation

tcclevenger commented Feb 7, 2025

tcclevenger left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tcclevenger commented Feb 7, 2025