Skip to content

Commit

Permalink
Merge pull request #44 from nasa/harmony-864
Browse files Browse the repository at this point in the history
Harmony 864 - Update README to reflect removal of run commands
  • Loading branch information
vinnyinverso authored Jan 16, 2024
2 parents 7c3e8e4 + 13924fa commit efe0f7d
Show file tree
Hide file tree
Showing 8 changed files with 24 additions and 222 deletions.
13 changes: 2 additions & 11 deletions Makefile
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
.PHONY: install test lint build-image build-test-image test-in-docker run-in-docker
.PHONY: install test lint build-image

install:
pip install -r requirements/core.txt -r requirements/dev.txt
Expand All @@ -10,13 +10,4 @@ lint:
flake8 --ignore=W503 harmony_netcdf_to_zarr

build-image:
LOCAL_SVCLIB_DIR=${LOCAL_SVCLIB_DIR} bin/build-image

build-test-image:
bin/build-test-image

test-in-docker:
LOCAL_SVCLIB_DIR=${LOCAL_SVCLIB_DIR} bin/test-in-docker

run-in-docker:
LOCAL_SVCLIB_DIR=${LOCAL_SVCLIB_DIR} bin/run-in-docker example/harmony-operation.json
LOCAL_SVCLIB_DIR=${LOCAL_SVCLIB_DIR} bin/build-image
85 changes: 20 additions & 65 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,113 +11,70 @@ using the `h5netcdf` driver. This includes some HDF5 EOSDIS datasets.
Individual collections must be tested to ensure compatibility.


## Development
## Development Setup

### Harmony Instance

It is recommended that the NetCDF-to-Zarr service is tested and developed using
a local Harmony instance. This can be established following the instructions in
the [Harmony repository](https://github.com/nasa/harmony).

### Setup

#### Docker

It is possible to develop and run this service locally using only Docker. This
is the recommended option for validation and small changes. Install
[Docker](https://www.docker.com/get-started) on your development machine.

#### Environment file
### Environment File

This service uses the
[harmony-service-lib-py](https://github.com/nasa/harmony-service-lib-py),
and requires that certain environment variables be set, as shown in the Harmony
Service Lib README. For example, `STAGING_BUCKET` and `STAGING_PATH` are
required, and `EDL_USERNAME` and `EDL_PASSWORD` are required for any
data behind Earthdata Login. For local testing (not integrated into Harmony in
data behind Earthdata Login. For automated testing (not integrated into Harmony in
a dev environment or AWS deployment), use the example `.env` file in this repo:

$ cp example/dotenv .env

and update the `.env` with the correct values.

#### Python & Project Dependencies (Optional)
### Python & Project Dependencies

If you would like to do local development outside of Docker, install Python (3.7.4), and create a Python virtual environment.
In order to be able to run the automated tests, install Python (3.9), and create a Python virtual environment.

Install project dependencies:
Activate the newly created environment and install the project dependencies:

$ python -m pip install --upgrade pip
$ make install

### Development with Docker

If you'd rather not build the image locally (as instructed below), you can simply pull the latest image:

$ docker pull harmonyservices/netcdf-to-zarr

Some of the [Makefile](./Makefile) targets referenced below include an optional argument that allows us to use a local copy of
`harmony-service-lib-py` (which is useful for concurrent development):

$ make target-name LOCAL_SVCLIB_DIR=../harmony-service-lib-py

#### Testing & Running the Service Independently

To run unit tests, coverage reports, or run the service on a sample message _outside_ of the
entire Harmony stack, start by building new runtime and test images:

*IMPORTANT*: If Minikube is installed, be sure to do these steps in a shell in which has *not* been updated to point to
the Minikube Docker daemon. This is usually done via a shell `eval` command. Doing so will
cause tests and the service to fail due to limitations in Minikube.

$ make build-image
$ make build-test-image

Run unit tests and generate overage reports. This will mount the local directory into the
container and run the unit tests. So all tests will reflect local changes to the service.

$ make test-in-docker

Finally, run the service using an example Harmony operation request
([example/harmony-operation.json](example/harmony-operation.json)) as input. This will reflect
local changes to this repo, but will not include local changes to the Harmony Service Lib.
$ make install

$ make run-in-docker

#### Testing & Running the Service in Harmony

*Without local Harmony Service Lib changes*:
## Running & Testing the Service

If using Minikube, be sure your environment is pointed to the Minikube Docker daemon:

$ eval $(minikube docker-env)

Build the image:
Build the Docker image:

$ make build-image

or build the image using a local copy of `harmony-service-lib-py` (useful for concurrent development):

$ make build-image LOCAL_SVCLIB_DIR=../harmony-service-lib-py

You can now run a workflow in your local Harmony stack and it will execute using this image.

Restart the services in your local Harmony instance (the script below is
contained in the Harmony repository):

$ bin/restart-services

### Development without Docker
Run through these steps again (build image, restart services) in order to pick up any new changes.

#### Testing & running the Service Independently
If you'd rather not build the image locally, you can simply pull the latest image:

$ docker pull ghcr.io/nasa/harmony-netcdf-to-zarr

This will require credentials for the Harmony Sandbox NGAPShApplicationDeveloper
to be present in your `~/.aws/credentials` file.
## Automated Tests

Run tests with coverage reports:

$ make test

Run an example:

$ dotenv run python3 -m harmony_netcdf_to_zarr --harmony-action invoke --harmony-input "$(bin/replace.sh example/harmony-operation.json)"

#### Installing `harmony-service-lib-py` in Development Mode

You may be concurrently developing on this service as well as the `harmony-service-lib-py`. If so, and you
want to test changes to it along with this service, install the `harmony-service-lib-py` in 'development mode'.
Install it using pip and the path to the local clone of the service library:
Expand All @@ -128,8 +85,6 @@ pip install -e ../harmony-service-lib-py

Now any changes made to that local repo will be visible in this project when you run tests, etc.

Finally, you can test & run the service in Harmony just as shown in the `Development with Docker` section above.

## Contributions:

Developers working on the NetCDF-to-Zarr service will need to create a feature
Expand Down
21 changes: 0 additions & 21 deletions bin/replace.sh

This file was deleted.

42 changes: 0 additions & 42 deletions bin/run-in-docker

This file was deleted.

25 changes: 0 additions & 25 deletions bin/run-test

This file was deleted.

35 changes: 0 additions & 35 deletions bin/test-in-docker

This file was deleted.

3 changes: 2 additions & 1 deletion example/dotenv
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,8 @@ USE_LOCALSTACK=true
SHARED_SECRET_KEY=_THIS_IS_MY_32_CHARS_SECRET_KEY_

# Set to 'true' if running Docker in Docker and the docker daemon is somewhere other than the current context
DIND=true
# Leave blank, i.e. DIND= if not running Docker in Docker
DIND=

# Indicates where docker commands should find the docker daemon
DOCKER_DAEMON_ADDR=host.docker.internal:2375
22 changes: 0 additions & 22 deletions example/harmony-operation.json

This file was deleted.

0 comments on commit efe0f7d

Please sign in to comment.