DAS-2254: wrap gridding code with harmony_service (#3)

.github/workflows/mypy.yml

@@ -13,7 +13,7 @@ jobs:
         python-version: ['3.12']
 
     steps:
-    - name: Check out smap-l2-gridder code
+    - name: Check out harmony-SMAP-L2-gridding-service code
       uses: actions/checkout@v4
 
     - name: Set up Python ${{ matrix.python-version }}
@@ -25,7 +25,7 @@ jobs:
       run: |
         python -m pip install --upgrade pip
         pip install mypy
-        pip install -r pip_requirements.txt -r pip_dev_requirements.txt
+        pip install -r pip_requirements.txt -r tests/pip_test_requirements.txt
 
     - name: Run mypy
       run: mypy .

.gitignore

@@ -1 +1,3 @@
 **/__pycache__
+/.coverage
+/reports/

.pre-commit-config.yaml

@@ -10,13 +10,10 @@ repos:
       - id: check-yaml
       - id: check-added-large-files
   - repo: https://github.com/astral-sh/ruff-pre-commit
-    rev: v0.7.1
+    rev: v0.8.0
     hooks:
       - id: ruff
-        args: ["--fix", "--show-fixes", "--extend-select", "I"]
-  - repo: https://github.com/psf/black-pre-commit-mirror
-    rev: 24.10.0
-    hooks:
-      - id: black-jupyter
-        args: ["--skip-string-normalization"]
-        language_version: python3.12
+        args: ["--fix", "--show-fixes"]
+        types_or: [python, jupyter]
+      - id: ruff-format
+        types_or: [python, jupyter]

CHANGELOG.md

@@ -10,3 +10,4 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 ### Added
 
 - Initial codebase that transforms SPL2SMP_E granules into NetCDF4-CF grids. [#1](https://github.com/nasa/harmony-SMAP-L2-gridding-service/pull/1)
+- Code and configuration to wrap gridding logic into a Harmony Service [#3](https://github.com/nasa/harmony-SMAP-L2-gridding-service/pull/3 )

CONTRIBUTING.md

@@ -0,0 +1,47 @@
+# Contributing to SMAP-L2-Gridding-Service
+
+Thanks for contributing!
+
+## Making Changes
+
+To allow us to incorporate your changes, please use the
+[Fork-and-Pull](https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/getting-started/about-collaborative-development-models#fork-and-pull-model)
+development model:
+
+1. Fork this repository to your personal account.
+2. Create a branch and make your changes.
+3. Test the changes locally/in your personal fork.
+4. Submit a pull request to open a discussion about your proposed changes.
+5. The maintainers will talk with you about it and decide to merge or request
+   additional changes.
+
+For larger items, consider contacting the maintainers first to coordinate
+development efforts.
+
+## Commits
+
+Our ticketing and CI/CD tools are configured to sync statuses amongst each
+other. Commits play an important role in this process. Please start all commits
+with the Jira ticket number associated with your feature, task, or bug. All
+commit messages should follow the format
+"[Jira Project]-XXXX - [Your commit message here]"
+
+## General coding practices:
+
+This repository adheres to Python coding style recommendations from
+[PEP8](https://peps.python.org/pep-0008/). Additionally, type hints are
+required in all function signatures.
+
+When adding or updating functionality, please ensure unit tests are added to
+an appropriate module in the `tests` sub directories, which cover each branch
+of the code.
+
+## Disclaimer
+
+SMAP-L2-Gridding-Service maintainers will review all pull requests submitted. Only requests that
+meet the standard of quality set forth by existing code, following the patterns
+set forth by existing code, and adhering to existing design patterns will be
+considered and/or accepted.
+
+For general tips on open source contributions, see
+[Contributing to Open Source on GitHub](https://guides.github.com/activities/contributing-to-open-source/).

README.md

@@ -1,20 +1,88 @@
-# smap-l2-gridder
+# SMAP L2 Gridding Service
 
-This is a python service to transform NASA level 2 Grid trajectory data into gridded NetCDF4-CF output files.
+This repository contains the code for the SMAP-L2-Gridding-Service, which is a python service that transforms NASA level 2 gridded trajectory data into gridded NetCDF4-CF output files.
 
+This code currently works on `SPL2SMP_E` data and will be adapted for other SMAP collections of gridded trajectory data.
 
-## Transform Data
 
-To run the regridder you can create an isolated python 3.12 environment installing packages from the `pip_requirements.txt` file.
+## Transforming Data
 
-From the commandline run:
+The logic of transforming data is contained in the `smap_l2_gridder` directory. It reads NASA L2 Gridded trajectory data and writes output NetCDF-CF files with the trajecotry style data correctly populated into EASE2 grids.
+
+### Commandline invocation
+To run the regridder on an input file.  Create an isolated python 3.12 environment using packages from the `pip_requirements.txt` file and then from the commandline run:
 
 ```python
 python -m smap_l2_gridder --input path/to/granule.h5 --output path/to/output_granule.nc
 ```
 
+`smap_l2_gridder/__main__.py` is the entrypoint to the science logic module and can be used for testing and development.
+
+## Directory structure
+
+```
+📁
+├── CHANGELOG.md
+├── CONTRIBUTING.md
+├── LICENSE
+├── README.md
+├── 📁 bin
+├── 📁 docker
+├── 📁 harmony_service
+├── pip_requirements.txt
+├── pyproject.toml
+├── 📁 smap_l2_gridder
+└── 📁 tests
+```
+
+* `CHANGELOG.md` -   Contains a record of changes applied to each new release of the SMAP-L2-Gridding-Service.
+* `CONTRIBUTING.md` -  Instructions on how to contribute to the repository.
+* `LICENSE` - Required for distribution under NASA open-source approval. Details conditions for use, reproduction and distribution.
+* `README.md` - This file, containing guidance on developing the library and service.
+* `bin` - A directory containing utility scripts to build the service and test images. A script to extract the release notes for the most recent version, as contained in `CHANGELOG.md` is also in this directory.
+* `docker` - A directory containing the Dockerfiles for the service and test images. It also contains `service_version.txt`, which contains the semantic version number of the library and service image. Update this file with a new version to trigger a release.
+*  `harmony_service` - A directory containing the Harmony Service specific python code. `adapter.py` contains the `SMAPL2GridderAdapter` class that is invoked by calls to the Harmony service.
+* `pip_requirements.txt` - Contains a list of python packages needed to run the service.
+* `pyproject.toml` - Configuration file used by packaging tools, and other tools such as linters, type checkers, etc.
+* `smap_l2_gridder` - Python package containing the logic for reformatting L2G data.
+* `tests` -  Contains the pytest test suite.
+
+
+## Local development
+
+Local testing of service functionality can be achieved via a local instance of
+[Harmony](https://github.com/nasa/harmony) aka Harmony-In-A-Box. Please see instructions there
+regarding creation of a local Harmony instance.
+
+For local development and testing of library modifications or small functions independent of the main Harmony application:
+
+1. Create a Python virtual environment
+1. Install the dependencies in `pip_requirements.txt`, and `tests/pip_test_requirements.txt`
+1. Install the pre-commit hooks ([described below](#pre-commit-hooks)).
+
+
+## Tests
+
+This service utilises the Python `pytest` package to perform unit tests on
+classes and functions in the service. After local development is complete, and
+test have been updated, they can be run in Docker via:
+
+```bash
+$ ./bin/build-image
+$ ./bin/build-test
+$ ./bin/run-test
+```
+
+It is also possible to run the test scripts directly (without docker) by just running the `run_tests.sh` script with a proper python environment. Do note that the `reports` directory will appear in the directory you call the script from.
 
-## pre-commit hooks:
+The `tests/run_tests.sh` script will also generate a coverage report, rendered
+in HTML, and scan the code with `pylint`.
+
+Currently, the `pytest` suite is run automatically within a GitHub workflow
+as part of a CI/CD pipeline. These tests are run for all changes made in a PR
+against the `main` branch. The tests must pass in order to merge the PR.
+
+## `pre-commit` hooks
 
 This repository uses [pre-commit](https://pre-commit.com/) to enable pre-commit
 checks that enforce coding standard best practices. These include:
@@ -35,3 +103,23 @@ pip install pre-commit
 # Install the git hook scripts:
 pre-commit install
 ```
+
+## Versioning:
+
+Docker service images for the `smap_l2_gridder` adhere to [semantic
+version](https://semver.org/) numbers: major.minor.patch.
+
+* Major increments: These are non-backwards compatible API changes.
+* Minor increments: These are backwards compatible API changes.
+* Patch increments: These updates do not affect the API to the service.
+
+## CI/CD:
+
+The CI/CD for SMAP-L2-Gridding-Service is run on github actions with the workflows in the
+`.github/workflows` directory:
+
+* [TODO: complete this section when the above statement is true]
+
+## Releasing
+
+* [TODO: complete when implemented]

bin/build-image

@@ -0,0 +1,27 @@
+#!/bin/bash
+###############################################################################
+#
+# Build a Docker image of the Harmony-SMAP-L2-Gridder service.
+#
+###############################################################################
+
+image="ghcr.io/nasa/harmony-smap-l2-gridder"
+
+# Retrieve the tag from the script arguments, or default to "latest". Bamboo
+# will pass the contents of `docker/service_version.txt` as this argument,
+# which contains a semantic version number.
+#
+tag=${1:-latest}
+
+
+# Remove old versions of: ghcr.io/nasa/harmony-smap-l2-gridder and
+# ghcr.io/nasa/harmony-smap-l2-gridder-test images
+./bin/clean-images
+
+# Build the image
+# This command tags the image with two tags:
+# - The one supplied as a script argument. Bamboo will provide the semantic
+#   version number from `docker/service_version.txt`.
+# - "latest", so the test Dockerfile can use the service image as a base image.
+#
+docker build  -t ${image}:${tag} -t ${image}:latest -f docker/service.Dockerfile .

bin/build-test

@@ -0,0 +1,23 @@
+#!/bin/bash
+###############################################################################
+#
+# Build a Docker container to run the test suite for the SMAP-L2-Gridding-Service
+#
+# To build the ghcr.io/nasa/harmony-smap-l2-gridder-test image, the
+# ghcr.io/nasa/harmony-smap-l2-gridder image must also be present
+# locally, as it is used as the base image in `docker/tests.Dockerfile`.
+#
+###############################################################################
+
+image="ghcr.io/nasa/harmony-smap-l2-gridder-test"
+tag=${1:-latest}
+
+
+# Look for old version of image and remove
+old=$(docker images | grep "$image" | grep "$tag" | awk '{print $3}')
+if [ ! -z "$old" ] && [ "$2" != "--no-delete" ]; then
+    docker rmi "$old"
+fi
+
+# Build the image
+docker build -t ${image}:${tag} -f docker/tests.Dockerfile .

bin/clean-images

@@ -0,0 +1,24 @@
+#!/bin/bash
+###############################################################################
+#
+# remove all images containing the string
+# "ghcr.io/nasa/harmony-smap-l2-gridder" and remove them.
+# This is used for clean-up after development.
+#
+###############################################################################
+remove_image_by_name() {
+    echo "Removing images for: $1"
+    matching_images=$(docker images --filter reference="$1" --format="{{.ID}}")
+
+    if [ ! -z "${matching_images}" ]; then
+        docker rmi "${matching_images}"
+    fi
+}
+
+image_base_name="ghcr.io/nasa/harmony-smap-l2-gridder"
+
+# First remove test images:
+remove_image_by_name "${image_base_name}-test"
+
+# Next remove service images:
+remove_image_by_name "${image_base_name}"

bin/run-test

@@ -0,0 +1,24 @@
+#!/bin/bash
+###############################################################################
+#
+# Execute the ghcr.io/nasa/harmony-smap-l2-gridder-test Docker image
+#
+###############################################################################
+
+set -ex
+
+# Remove cached bytecode Python files, to ensure latest code is used
+find . | grep -E "(__pycache__|\.pyc|\.pyo$)" | xargs rm -rf
+
+# Make the directory into which XML format test reports will be saved
+mkdir -p reports/test-reports
+
+# Make the directory into which coverage reports will be saved
+mkdir -p reports/coverage
+
+# Run the tests in a Docker container with mounted volumes for XML report
+# output and test coverage reporting
+docker run  --rm \
+	-v $(pwd)/reports/test-reports:/home/reports/test-reports \
+	-v $(pwd)/reports/coverage:/home/reports/coverage \
+	ghcr.io/nasa/harmony-smap-l2-gridder-test "$@"

docker/service.Dockerfile

@@ -0,0 +1,29 @@
+###############################################################################
+#
+# Service image for ghcr.io/nasa/harmony-smap-l2-gridder
+
+# Harmony-SMAP-L2-Gridder backend service that transforms L2G (gridded
+# trajectory) data into actual gridded data.
+#
+# This image installs dependencies via Pip. The service code is then copied
+# into the Docker image.
+#
+###############################################################################
+FROM python:3.12-slim-bookworm
+
+WORKDIR "/home"
+
+RUN apt-get update
+
+# Install Pip dependencies
+COPY pip_requirements.txt /home/
+
+RUN pip install --no-input --no-cache-dir \
+    -r pip_requirements.txt
+
+# Copy service code.
+COPY ./harmony_service harmony_service
+COPY ./smap_l2_gridder smap_l2_gridder
+
+# Configure a container to be executable via the `docker run` command.
+ENTRYPOINT ["python", "-m", "harmony_service"]

docker/service_version.txt

@@ -0,0 +1 @@
+0.0.0

docker/tests.Dockerfile

@@ -0,0 +1,20 @@
+###############################################################################
+#
+# Test image for the SMAP-L2-Gridding-Service. This test image uses the main
+# service image as a base layer for the tests. This ensures that the contents
+# of the service image are tested, preventing discrepancies between the service
+# and test environments.
+###############################################################################
+FROM ghcr.io/nasa/harmony-smap-l2-gridder
+
+# Install additional Pip requirements (for testing)
+COPY tests/pip_test_requirements.txt .
+
+RUN pip install --no-input --no-cache-dir \
+	-r pip_test_requirements.txt
+
+# Copy test directory containing Python unittest suite, test data and utilities
+COPY ./tests tests
+
+# Configure a container to be executable via the `docker run` command.
+ENTRYPOINT ["/home/tests/run_tests.sh"]

harmony_service/__init__.py

@@ -0,0 +1 @@
+"""Initialize the harmony_service package."""

harmony_service/__main__.py

@@ -0,0 +1,26 @@
+"""Run the Harmony-SMAP-L2-Gridder via the Harmony CLI."""
+
+from argparse import ArgumentParser
+from sys import argv
+
+from harmony_service_lib import is_harmony_cli, run_cli, setup_cli
+
+from harmony_service.adapter import SMAPL2GridderAdapter
+from harmony_service.exceptions import SERVICE_NAME
+
+
+def main(arguments: list[str]):
+    """Parse command line arguments and invoke the appropriate method."""
+    parser = ArgumentParser(prog=SERVICE_NAME, description='Run SMAP L2 Gridder.')
+
+    setup_cli(parser)
+    harmony_arguments, _ = parser.parse_known_args(arguments[1:])
+
+    if is_harmony_cli(harmony_arguments):
+        run_cli(parser, harmony_arguments, SMAPL2GridderAdapter)
+    else:
+        parser.error('Only --harmony CLIs are supported')
+
+
+if __name__ == '__main__':
+    main(argv)