Apps macro wps

source/Reviewers/howtocommit.rst

@@ -166,6 +166,14 @@ then you will need to upgrade the test-suite.
             where `vnX.Y_tZZZZ` is the `AFTER_TAG` of the latest upgrade macro.
             The upgrade is expected to fail for the `fab_jules`, `metadata_checker` and `umdp3_checker` apps.
 
+        .. tab-item:: LFRic Apps + Core
+
+            .. code-block:: RST
+
+                apply_macros.py vnX.Y_tZZZZ [--apps=/path/to/apps] [--core=/path/to/core] [--jules=/path/to/jules]
+
+            where `vnX.Y_tZZZZ` is the `AFTER_TAG` of the latest upgrade macro and the others are paths to the relevant sources. Apps defaults to the current location. Core and Jules default to reading the `dependencies.sh` file in the Apps source. A copy of `apply_macros.py` is available at `$UMDIR/SimSys_Scripts/lfric_macros`.
+
 .. dropdown:: New rose-stem app?
 
     If the ticket introduces a new rose-stem app, but doesn't otherwise have a macro
@@ -191,11 +199,12 @@ then you will need to upgrade the test-suite.
             rose macro --validate -M path/to/working_copy/rose-meta
 
     .. note::
-       For UM tickets, if there are linked `jules-shared
-       <https://code.metoffice.gov.uk/trac/jules/browser/main/trunk/rose-meta/jules-shared>`_
-       metadata changes these will need to be added to the metadata
-       path. Please see the :ref:`rose config-edit
-       example<metadata_changes>`.
+
+        LFRic Apps tickets will require an LFRic Core source to use. You can do this by checking out an appropriate working copy, and exporting the environment variable `ROSE_META_PATH=/path/to/core`.
+
+        For UM tickets, if there are linked `jules-shared
+        <https://code.metoffice.gov.uk/trac/jules/browser/main/trunk/rose-meta/jules-shared>`_
+        metadata changes then a suitable Jules source will need to be included in the `ROSE_META_PATH` as described above.
 
 
 .. dropdown:: Temporary Logical?
@@ -492,7 +501,6 @@ Supporting data is stored in the filesystems of our machines and changes to use
         .. note::
               The numbered run directory must be included in the suite name, eg. `name-of-suite/run1`.
 
-
         5. Verify the checksums updated properly by retriggering the failed checksums. First retrigger
         ``export-source``, and then when complete ``export-source_xc40`` if new checksums are present
         there (there is no need to retigger spice). You may need to change the maximum window extent

source/WorkingPractices/inputs.rst

@@ -53,7 +53,7 @@ metadata :ref:`is available <metadata_guidance>`.
   work with the model.
 
 Changes to the metadata which don't involve namelist changes may or may not
-require an upgrade macro. If you are unsure whether a UM change needs an
+require an upgrade macro. If you are unsure whether a change needs an
 upgrade macro, then run the following command on your branch:
 
 .. code-block::
@@ -63,10 +63,25 @@ upgrade macro, then run the following command on your branch:
 If all of the tests pass then there is no requirement to add an upgrade macro.
 If any of the metadata tests fail, then the developer should add a blank upgrade
 macro which contains no upgrade commands but simply points the rose stem suite
-to the new metadata.
+to the new metadata. The SSD team are also available to advise on whether an upgrade macro is necessary.
 
-..
-  The above should probably be extended to LFRic eventually.
+.. tip::
+
+  When editing metadata you should always check that the new metadata appears as expected in the gui, including testing that invalid settings raise appropriate warnings. The command to open the gui is in general:
+
+  .. code-block::
+
+    rose edit -C rose-stem/app/APP-NAME
+
+  For LFRic Apps a few extra changes are required:
+
+  .. code-block::
+
+    export ROSE_PYTHONPATH=$PYTHONPATH
+    export ROSE_META_PATH=/path/to/valid/core
+    rose edit -C rose-stem/app/APP-NAME
+
+  This requires an LFRic Core working copy at an appropriate revision to be available. It is also necessary to run from the top level of the Apps working copy to ensure rose metadata paths are valid.
 
 How to add an upgrade macro to your branch
 ------------------------------------------

source/WorkingPractices/macros.rst

@@ -3,10 +3,11 @@
 Upgrade Macros
 ==============
 
-To create an upgrade macro, the developer must edit a ``versions.py`` file which is used
-to update the various apps in the rose stem suite to accept the namelist changes. The upgrade
-macros also form the basis of the ``rose app-upgrade`` script applied by a user wishing to
-upgrade from one version of a model to the next.
+.. important::
+
+    When developing Upgrade Macros, they must be tested using a test branch (see :ref:`testing`).
+
+To create an upgrade macro, the developer must edit a ``versions.py`` file which is used to update the various apps in the rose stem suite to accept the namelist changes. The upgrade macros also form the basis of the ``rose app-upgrade`` script applied by a user wishing to upgrade from one version of a model to the next.
 
 The  ``versions.py`` file containing upgrade macros can be found in the following locations:
 
@@ -20,6 +21,11 @@ The  ``versions.py`` file containing upgrade macros can be found in the followin
 
         ``rose-meta/jules-standalone/versions.py``
 
+    .. tab-item:: LFRic Core + Apps
+
+        | ``applications/<APPLICATION>/rose-meta/lfric-<APPLICATION>/versions.py``
+        | Variations on this theme occur, eg. LFRic Apps science sections or Components in LFRic Core
+
 
 Within the file a blank upgrade macro will typically look like this:
 
@@ -73,3 +79,39 @@ This command can then be run on a **test** branch (see :ref:`testing`).
   namelists and changing the value that a particular variable takes.
   A `tutorial <http://metomi.github.io/rose/doc/html/tutorial/rose/furthertopics/upgrading.html>`_
   is also available.
+
+
+Upgrade Macros in LFRic
+-----------------------
+
+.. warning::
+
+    Namelist files in application example directories are not currently updated by the Apply Macros script. This feature is intended to be introduced, but for now, developers still need to manually update those files.
+
+The organisation of LFRic metadata is different from other repositories (UM + Jules) as the metadata is stored with the Science or Application section it is associated with and is then imported by other apps that require it. This helps modularise the LFRic code but complicates macro chains as different apps may require different chains depending on the metadata they import. A simple implementation of macros in this case might require duplication of a particular macro if its metadata is imported multiple times. This is unsatisfactory as it makes mistakes more likely and requires more effort.
+
+To solve this, macros in LFRic Apps are applied using a wrapper script which will read the added macros and combine them in versions.py files where the metadata is imported. Therefore when adding macros, the macro should be added in the versions.py file in the same metadata directory as the metadata it is applying. It will then be shared as appropriate by the ``apply_macros.py`` script. For example, if a change to metadata is made in ``science/gungho/rose-meta/lfric-gungho``, the macro should be added to the ``versions.py`` file in that directory. This will then be copied to other ``versions.py`` files that import gungho metadata, eg. lfric_atm, transport etc.
+
+.. important::
+
+    Some complex macro commands may be dependent on the order in which they are applied. If macros are copied by the wrapper script, the order they are applied will always be determined by the reverse metadata import order. For example, lfric_atm imports gungho metadata, which itself imports components/driver. If all 3 sections have an associated macro, then the macro commands would be applied in the order: components/driver, gungho, lfric_atm.
+
+.. tip::
+
+    The wrapper script will read the ``dependencies.sh`` file in your LFRic Apps working copy and will checkout a temporary copy of the LFRic Core source if required. Some Core metadata changes will also modify the Core rose apps. In this case make sure to also commit these changes back to the core branch.
+
+To add upgrade macros to LFRic the following steps can be followed:
+
+1. Checkout an LFRic Apps working copy and update the core source in ``dependencies.sh`` if required.
+2. Add your upgrade macros. These **must** be added to the versions.py file associated with the metadata they are changing.
+3. Run the Upgrade Macro script - this **must** be done in a test branch (see :ref:`testing`). This is located in the `SimSys_Scripts github repo <https://github.com/MetOffice/SimSys_Scripts>`_ (at meto an up to date clone is available in $UMDIR/SimSys_Scripts). The syntax for running is:
+
+.. code-block::
+
+    SimSys_Scripts/lfric_macros/apply_macros.py vnXX.Y_tTTTT -a Apps -c Core -j Jules
+
+The Apps, Core and Jules options are paths to sources for each of these. Apps will default to the present location (so it is recommended to launch from an Apps working copy). Core and Jules will default to reading the ``dependencies.sh`` file in the Apps source if not provided.
+
+.. tip::
+
+    The apply_macros script requires python >= 3.9. At the Met Office this can be achieved by ``module load scitools``.

source/WorkingPractices/testing.rst

@@ -79,12 +79,12 @@ commands, noting that ``--jules-path`` is only required if you have
 +-------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
 | JULES | ``./bin/upgrade_jules_test_apps vnX.X_tXXXX``                                                                                                                                |
 +-------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+| LFRic | ``apply_macros.py vnX.Y_tZZZZ [--apps=/path/to/apps] [--core=/path/to/core] [--jules=/path/to/jules]``                                                                       |
++-------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
 
-..
-    | LFRic | *Macro capabilities are currently in development*                                                                                              |
-    +-------+------------------------------------------------------------------------------------------------------------------------------------------------+
+.. tip::
 
-.. todo: add an LFRic upgrade macro command... and make these more consistent?
+    The `apply_macros.py` script is located in the `SimSys_Scripts github repo <https://github.com/MetOffice/SimSys_Scripts>`_ (at meto an up to date clone is available in $UMDIR/SimSys_Scripts).
 
 .. warning::
    Please ensure that Cylc7 is used with `update_all.py` @vn13.5. This is fixed at HoT and either Cylc7 or Cylc8 can be used.