Feature/swot expert (#27)

* /deploy sandbox * /deploy sandbox * /deploy sandbox * add in generate tig config script * update generate config scripts * fix pylint * added in code to generate ssha_karin_2 images for swot expert * update to python 3.10 * update python version to 3.10 * /deploy sit * /deploy sit * /deploy sit * /deploy sit * /deploy sit * /deploy sit * /deploy sit * /deploy sit * /deploy sit * update libraries * /deploy sit * removed unused code * update readme and changelog * revert tig.py * fix tig code and update changelog * fix pylint * update swot expert generation algorithm * - Added "helper" script to generate thumbnails for each collection - Updated dependency versions * Added help info * removed extra space * Update helper script docs * revert line * update docs * update docs --------- Co-authored-by: James Wood <[email protected]>
podaac · Oct 31, 2023 · 521a9ad · 521a9ad
1 parent bdc6f24
commit 521a9ad
Show file tree

Hide file tree

Showing 16 changed files with 1,478 additions and 801 deletions.
diff --git a/.github/workflows/.release-created.yml b/.github/workflows/.release-created.yml
@@ -21,7 +21,7 @@ jobs:
           ref: 'refs/heads/develop'
       - uses: actions/setup-python@v2
         with:
-          python-version: 3.9
+          python-version: '3.10'
       - name: Install Poetry
         uses: abatilo/[email protected]
         with:

diff --git a/.github/workflows/build.yml b/.github/workflows/build.yml
@@ -29,7 +29,7 @@ jobs:
       - uses: actions/checkout@v2
       - uses: actions/setup-python@v2
         with:
-          python-version: 3.9
+          python-version: '3.10'
       - name: Install Poetry
         uses: abatilo/[email protected]
         with:
@@ -546,7 +546,7 @@ jobs:
           account_id: ${{ secrets[format('AWS_ACCOUNT_ID_SERVICES_{0}', env.TARGET_ENV_UPPERCASE)] }}
           repo: podaac/tig
           region: us-west-2
-          tags: ${{ env.the_version }}
+          tags: ${{ steps.meta.outputs.version }}
           create_repo: true
           dockerfile: ./docker/lambdaDockerfile
           extra_build_args: --build-arg DIST_PATH="dist/" --build-arg SOURCE=${{ env.local_tig }}
@@ -615,12 +615,13 @@ jobs:
           ls -al bin/
           which python3
           python3 --version
-          python3 override.py https://github.com/podaac/tig/releases/download/${{ env.the_version }}/tig-terraform-${{ env.the_version }}.zip ${{ secrets[format('AWS_ACCOUNT_ID_SERVICES_{0}', env.TARGET_ENV_UPPERCASE)] }}.dkr.ecr.us-west-2.amazonaws.com/podaac/tig:${{ env.the_version }}
+          python3 override.py https://github.com/podaac/tig/releases/download/${{ env.the_version }}/tig-terraform-${{ env.the_version }}.zip ${{ secrets[format('AWS_ACCOUNT_ID_SERVICES_{0}', env.TARGET_ENV_UPPERCASE)] }}.dkr.ecr.us-west-2.amazonaws.com/podaac/tig:${{ steps.meta.outputs.version }}
           ls -al
           echo "Show override contents"
           cat override.tf.json
           echo "End show override contents"
           terraform --version
-          source bin/config.sh ${{ env.venue }}
-          terraform plan -var-file=tfvars/"${{ env.venue }}".tfvars -var="app_version=${{ env.the_version }}" -out="tfplan"
+          source bin/config.sh ${{ env.TARGET_ENV_LOWERCASE }}
+          terraform plan -var-file=tfvars/"${{ env.TARGET_ENV_LOWERCASE }}".tfvars -var="app_version=${{ env.the_version }}" -out="tfplan"
           terraform apply -auto-approve tfplan
+
diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -7,6 +7,10 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 ## [unreleased]
 
 ### Added
+- ** Add Hitide Generate Configuration **
+  - [issue/26](https://github.com/podaac/tig/issues/26): Added cli command to generate hitide configurations for tig/forge
+- ** Added in swot expert image generation **
+  - [issue/18] (https://github.com/podaac/hitide/issues/18): Added in code to generate ssha_karin_2 in swot expert 
 ### Changed
 ### Deprecated
 ### Removed

diff --git a/README.md b/README.md
@@ -33,6 +33,23 @@ Use cli to test thumbnail image generation for a granule with configuration file
 tig --input_file <granule> --output_dir <output_dir> --config_file <config_file> --palette_dir <palette_dir>
 ```
 
+Use cli to create a tig configuration for collections 
+```
+generate_hitide_config --granule <granule_file> -dataset-id <collection short name> --include-image-variables <csv file image variables> --longitude <lon variable> --latitude <lat variable> --time <time variable> --footprint_strategy <footprint strategy>
+```
+
+granule: a sample granule file to generate the configuration for
+datset-id: collection short name 
+include-image-variables: csv file of with image variable names and min max setting for each variable
+longitude: longitude variable include the group if they're in group defaults to longitude
+latitude: latitude variable include the group if they're in a group defaults to latitude
+time: time variable include the group if they're in a group defaults to time
+footprint_strategy: strategy to generate footprint will default to None options should be ["periodic", "linestring", "polar", "swot_linestring", "polarsides", "smap"]
+
+Use cli helper script to generate thumbnails for each collection
+```
+Go to dir `podaac/tig` and look at generate_thumbnails.sh script.  Follow directions at top to setup.
+```
 
 ## How to load and use tig module
 Project using tig can include/use the tig as following:

diff --git a/docker/Dockerfile b/docker/Dockerfile
@@ -1,4 +1,4 @@
-FROM python:3.8-slim
+FROM python:3.10-slim
 
 RUN apt-get update \
     && DEBIAN_FRONTEND=noninteractive apt-get upgrade -y \

diff --git a/docker/lambdaDockerfile b/docker/lambdaDockerfile
@@ -1,6 +1,6 @@
 ARG FUNCTION_DIR="/function"
 
-FROM python:3.8-slim as build-image
+FROM python:3.10-slim as build-image
 
 # Include global arg in this stage of the build
 ARG FUNCTION_DIR

diff --git a/example_vars.csv b/example_vars.csv
@@ -0,0 +1,3 @@
+variable,min,max
+data_01/ku/ssha,-0.2,0.2
+data_01/ku/swh_ocean,0,30
diff --git a/podaac/lambda_handler/lambda_handler.py b/podaac/lambda_handler/lambda_handler.py
@@ -355,6 +355,7 @@ def image_generate(self, file_, config_file, palette_dir, granule_id):
         processes = []
 
         var_list = [variables_config[:len(variables_config)//2], variables_config[len(variables_config)//2:]]
+        var_list = [variables_config]
 
         for variables in var_list:
             if variables:

diff --git a/podaac/tig/generate_hitide_config.py b/podaac/tig/generate_hitide_config.py
@@ -0,0 +1,170 @@
+"""Python script used to generate hitide config for forge and tig"""
+
+import json
+import csv
+import click
+import netCDF4 as nc
+
+# sample = {
+#   "shortName": "CYGNSS_L3_V3.0",
+#   "latVar": "lat",
+#   "lonVar": "lon",
+#   "timeVar": "time",
+#   "is360": true,
+#   "tiles": {
+#     "steps": [
+#       30,
+#       14
+#     ]
+#   },
+#   "footprint": {
+#     "findValid": true,
+#     "strategy": "fixed",
+#     "geospatial_lat_min": -43,
+#     "geospatial_lat_max": 43,
+#     "geospatial_lon_min": -180,
+#     "geospatial_lon_max": 180,
+#     "t": "0:*",
+#     "s1": "0:*",
+#     "b": "0:*",
+#     "s2": "0:*"
+#   },
+#   "imgVariables": [
+#     {
+#       "id": "wind_speed",
+#       "title": "Wind speed",
+#       "units": "m s-1",
+#       "min": "-5.0",
+#       "max": "100.0",
+#       "palette": "paletteMedspirationIndexed"
+#     }
+#   ],
+#   "image": {
+#     "ppd": 4,
+#     "res": 8
+#   },
+#   "variables": [
+#     "time"
+#   ]
+# }
+
+
+def get_variables_with_paths(group, current_path=""):
+    """Function to recursively get variables with group paths"""
+
+    variables_with_paths = []
+    for var_name, var_obj in group.variables.items():  # pylint: disable=unused-variable
+        variable_path = current_path + "/" + var_name if current_path else var_name
+        variables_with_paths.append(variable_path)
+    for subgroup_name, subgroup in group.groups.items():
+        subgroup_path = current_path + "/" + subgroup_name if current_path else subgroup_name
+        subgroup_variables = get_variables_with_paths(subgroup, subgroup_path)
+        variables_with_paths.extend(subgroup_variables)
+    return variables_with_paths
+
+
+def read_min_max_csv(filename):
+    """ Function to read csv with variables, min, max values """
+
+    data = {}
+    with open(filename, newline='') as csvfile:
+        reader = csv.reader(csvfile)
+        headers = next(reader)  # Read the column headers
+        for row in reader:
+            key = row[0].replace("'", "").replace(",", "")  # Use the first column as the key
+            values = {}
+            for i in range(1, len(headers)):
+                values[headers[i]] = row[i]
+            data[key] = values
+    return data
+
+
+@click.command()
+@click.option('-g', '--granule', help='Sample granule file', required=True)
+@click.option('-d', '--dataset-id', help='Collection short name', required=True)
+@click.option('-i', '--include-image-variables', help='CSV file with variables, min, max settings', required=False)
+@click.option('--longitude', required=False, help='longitude variable', default="longitude")
+@click.option('--latitude', required=False, help='latitude variable', default="latitude")
+@click.option('--time', required=False, help='time variable', default="time")
+@click.option('--footprint-strategy', help='forge footprint strategy', required=False)
+def generate_hitide_config(granule, dataset_id, include_image_variables, longitude, latitude, time, footprint_strategy):
+    """Function to generate hitide configuration"""
+
+    dataset_config = {
+        'shortName': dataset_id,
+        'latVar': latitude,
+        'lonVar': longitude,
+        'timeVar': time,
+        'is360': True,
+        'tiles': {'steps': [30, 14]},
+        'footprint': {},
+        'imgVariables': [],
+        'image': {'ppd': 16, 'res': 8},
+        'variables': []
+    }
+
+    if footprint_strategy:
+        dataset_config['footprint'] = {
+            'strategy': footprint_strategy,
+            't': '0:0,0:*',
+            's1': '0:*,0:0',
+            'b': '*:*,0:*',
+            's2': '0:*,*:*'
+        }
+
+    vars_min_max = {}
+    if include_image_variables:
+        vars_min_max = read_min_max_csv(include_image_variables)
+
+    with nc.Dataset(granule, 'r') as dataset:  # pylint: disable=no-member
+
+        data_var_names = get_variables_with_paths(dataset)
+        dataset_config['variables'] = data_var_names
+
+        try:
+            for data_var in data_var_names:
+                if vars_min_max and data_var in vars_min_max:
+                    variable = dataset[data_var]
+
+                    units = variable.units if 'units' in variable.ncattrs() else ''
+                    long_name = variable.long_name if 'long_name' in variable.ncattrs() else ''
+
+                    if data_var in vars_min_max:
+                        min_max = vars_min_max[data_var]
+                        min_val = float(min_max['min'])
+                        max_val = float(min_max['max'])
+                    else:
+                        min_val = variable.valid_min if 'valid_min' in variable.ncattrs() else ''
+                        max_val = variable.valid_max if 'valid_max' in variable.ncattrs() else ''
+
+                    dataset_config['imgVariables'].append({
+                        'id': data_var,
+                        'title': long_name,
+                        'units': units,
+                        'min': min_val,
+                        'max': max_val,
+                        'palette': 'paletteMedspirationIndexed'
+                    })
+
+        except Exception as ex:  # pylint: disable=broad-exception-caught
+            print(f"Error: Failed on variable {data_var}, exception: " + str(ex))
+
+    print(json.dumps(dataset_config, indent=4))
+
+    # Specify the file path where you want to save the JSON data
+    file_path = f"{dataset_id}.cfg"
+
+    # Open the file in write mode and write the JSON data to it
+    with open(file_path, "w") as json_file:
+        json.dump(dataset_config, json_file, indent=4)
+
+    return dataset_config
+
+
+if __name__ == '__main__':
+    generate_hitide_config()  # pylint: disable=no-value-for-parameter
+
+# Example runs:
+# python gen_dataset_config.py -g SWOT_L2_LR_SSH_Basic_001_001_20160901T000000_20160901T005126_DG10_01.nc -d SWOT_L2_LR_SSH_BASIC_1.0 -i SWOT_L2_LR_SSH_BASIC_1.0.csv
+# python gen_dataset_config.py -g SWOT_L2_LR_SSH_WindWave_001_001_20160901T000000_20160901T005126_DG10_01.nc -d SWOT_L2_LR_SSH_WINDWAVE_1.0 -i SWOT_L2_LR_SSH_WINDWAVE_1.0.csv
+# python gen_dataset_config.py -g SWOT_L2_LR_SSH_Expert_001_001_20160901T000000_20160901T005126_DG10_01.nc -d SWOT_L2_LR_SSH_EXPERT_1.0 -i SWOT_L2_LR_SSH_EXPERT_1.0.csv
diff --git a/podaac/tig/generate_thumbnails.sh b/podaac/tig/generate_thumbnails.sh
@@ -0,0 +1,61 @@
+#!/usr/bin/env bash
+
+# Prerequisites
+#
+# 1. Clone forge-tig-configuration into the parent dir of this repo
+# 2. Setup poetry and run poetry install
+
+# HOW-TO Use
+#
+# Setup Collection Short Name Directories
+#
+#  1. Create a directory in the tig root dir named "configs"
+#  2. Create directories under "configs" dir with the collection short name
+#  3. Copy this script to each short name dir
+#  4. Copy file "example_vars.csv" (in repo root dir) to a file called "vars.csv" in each short name dir
+#  5. Update vars.csv file in each collection short name specifically for that collection (i.e. update var names and min/maxes)
+#  6. Download a "representative" .nc granule file for that collection short name and move it into each short name dir
+#  7. Update this script in each short name dir specifically for its collection short name (e.g. update input parameters)
+
+# Run Script
+#
+#  1. Change dir to the configs/<shortname> dir
+#  2. Run command "./generate_thumbnails.sh" to automatically generate the config, run tig, and open the thumbnails
+#  3. The collection <shortname>.cfg file will be saved in each short name dir
+#  4. Verify .cfg file and all thumbnail images in each short name dir
+#  5. Tweak var.csv or this script in short name dir if needed and re-run
+#
+#  NOTE: In rare cases, the generate_hitide_config.py file may need tweaking to generate a correct config file
+
+#  Publish config files to forge-tig-configuration repo
+#
+#  1. Copy each short name .cfg file to the forge-tig-configuration repo under "config-files"
+#  2. Commit the new cfg files in forge-tig-configuration to a new branch named "feature/<name>"
+#  3. Open a PR in forge-tig-configuration for this new branch
+
+collection=$(basename "$(pwd)")
+
+echo Generating configuration for collection $collection...
+
+if [ -e "$collection.cfg" ]; then
+    rm "$collection.cfg"
+    echo "Removed existing $collection.cfg"
+fi
+
+if [ -d "images" ]; then
+    rm -rf "images"
+    echo "Removed existing 'images' directory"
+fi
+
+cd ../..
+
+poetry run generate_hitide_config -g configs/$collection/*.nc -d $collection -i configs/$collection/vars.csv --latitude lat --longitude lon --time time --footprint-strategy periodic
+
+echo "Running tig to generate thumbnail images..."
+
+poetry run tig --input_file configs/$collection/*.nc --output_dir configs/$collection/images --config_file "$collection".cfg --palette_dir ../forge-tig-configuration/palettes
+
+mv "$collection".cfg configs/$collection
+
+cd configs/$collection
+open images/*