Skip to content
Permalink

Comparing changes

Choose two branches to see what’s changed or to start a new pull request. If you need to, you can also or learn more about diff comparisons.

Open a pull request

Create a new pull request by comparing changes across two branches. If you need to, you can also . Learn more about diff comparisons here.
base repository: nasa/harmony-swath-projector
Failed to load repositories. Confirm that selected base ref is valid, then try again.
Loading
base: 1.1.0
Choose a base ref
...
head repository: nasa/harmony-swath-projector
Failed to load repositories. Confirm that selected head ref is valid, then try again.
Loading
compare: main
Choose a head ref
  • 14 commits
  • 15 files changed
  • 4 contributors

Commits on Sep 18, 2024

  1. Copy the full SHA
    b4b17e7 View commit details

Commits on Oct 10, 2024

  1. [pre-commit.ci] pre-commit autoupdate (#15)

    Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
    pre-commit-ci[bot] authored Oct 10, 2024
    Copy the full SHA
    842d224 View commit details

Commits on Oct 30, 2024

  1. Copy the full SHA
    5c97edb View commit details

Commits on Oct 31, 2024

  1. [pre-commit.ci] pre-commit autoupdate (#20)

    Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
    Co-authored-by: Matt Savoie <github@flamingbear.com>
    pre-commit-ci[bot] and flamingbear authored Oct 31, 2024
    Copy the full SHA
    d181ea4 View commit details

Commits on Nov 16, 2024

  1. [pre-commit.ci] pre-commit autoupdate (#21)

    Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
    pre-commit-ci[bot] authored Nov 16, 2024
    Copy the full SHA
    e6e08e1 View commit details

Commits on Nov 27, 2024

  1. [pre-commit.ci] pre-commit autoupdate (#22)

    updates:
    - [github.com/astral-sh/ruff-pre-commit: v0.7.3 → v0.8.0](astral-sh/ruff-pre-commit@v0.7.3...v0.8.0)
    
    Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
    pre-commit-ci[bot] authored Nov 27, 2024
    Copy the full SHA
    dbc5e2e View commit details

Commits on Dec 3, 2024

  1. [pre-commit.ci] pre-commit autoupdate (#23)

    updates:
    - [github.com/astral-sh/ruff-pre-commit: v0.8.0 → v0.8.1](astral-sh/ruff-pre-commit@v0.8.0...v0.8.1)
    
    Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
    pre-commit-ci[bot] authored Dec 3, 2024
    Copy the full SHA
    ddb607f View commit details

Commits on Dec 9, 2024

  1. [pre-commit.ci] pre-commit autoupdate

    updates:
    - [github.com/astral-sh/ruff-pre-commit: v0.8.1 → v0.8.2](astral-sh/ruff-pre-commit@v0.8.1...v0.8.2)
    pre-commit-ci[bot] authored Dec 9, 2024
    Copy the full SHA
    fbc0819 View commit details

Commits on Dec 11, 2024

  1. Merge pull request #24 from nasa/pre-commit-ci-update-config

    updates:
    - [github.com/astral-sh/ruff-pre-commit: v0.8.1 → v0.8.2](astral-sh/ruff-pre-commit@v0.8.1...v0.8.2)
    
    Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
    flamingbear and pre-commit-ci[bot] authored Dec 11, 2024
    Copy the full SHA
    8d9b0f7 View commit details
  2. Copy the full SHA
    6253559 View commit details

Commits on Dec 18, 2024

  1. [pre-commit.ci] pre-commit autoupdate (#26)

    updates:
    - [github.com/astral-sh/ruff-pre-commit: v0.8.2 → v0.8.3](astral-sh/ruff-pre-commit@v0.8.2...v0.8.3)
    
    Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
    pre-commit-ci[bot] authored Dec 18, 2024
    Copy the full SHA
    debd5f8 View commit details

Commits on Jan 6, 2025

  1. [pre-commit.ci] pre-commit autoupdate (#28)

    Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
    pre-commit-ci[bot] authored Jan 6, 2025
    Copy the full SHA
    2b2fb28 View commit details

Commits on Jan 7, 2025

  1. [pre-commit.ci] pre-commit autoupdate (#29)

    Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
    pre-commit-ci[bot] authored Jan 7, 2025
    Copy the full SHA
    83d10cc View commit details

Commits on Jan 21, 2025

  1. [pre-commit.ci] pre-commit autoupdate (#30)

    Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
    pre-commit-ci[bot] authored Jan 21, 2025
    Copy the full SHA
    c84d624 View commit details
4 changes: 2 additions & 2 deletions .github/workflows/run_tests.yml
Original file line number Diff line number Diff line change
@@ -30,13 +30,13 @@ jobs:
run: ./bin/run-test

- name: Archive test results
uses: actions/upload-artifact@v3
uses: actions/upload-artifact@v4
with:
name: Test results
path: test-reports/

- name: Archive coverage report
uses: actions/upload-artifact@v3
uses: actions/upload-artifact@v4
with:
name: Coverage report
path: coverage/*
6 changes: 3 additions & 3 deletions .pre-commit-config.yaml
Original file line number Diff line number Diff line change
@@ -2,20 +2,20 @@ ci:
autofix_prs: false
repos:
- repo: https://github.com/pre-commit/pre-commit-hooks
rev: v4.6.0
rev: v5.0.0
hooks:
- id: trailing-whitespace
- id: end-of-file-fixer
- id: check-json
- id: check-yaml
- id: check-added-large-files
- repo: https://github.com/astral-sh/ruff-pre-commit
rev: v0.6.2
rev: v0.9.2
hooks:
- id: ruff
args: ["--fix", "--show-fixes", "--select", "I"]
- repo: https://github.com/psf/black-pre-commit-mirror
rev: 24.8.0
rev: 24.10.0
hooks:
- id: black-jupyter
args: ["--skip-string-normalization"]
34 changes: 28 additions & 6 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,26 @@
# Changelog

## [v1.2.0] - 2024-10-10

### Changed

- [[DAS-2216](https://bugs.earthdata.nasa.gov/browse/DAS-2216)]
The Swath Projector has been updated with quick fixes to add support for TEMPO level 2 data. These changes include optional transposing of arrays based on dimension sizes, addition of rows_per_scan parameter for ewa interpolation, and updates to the configuration file for TEMPO_O3TOT_L2 to correctly locate coordinate variables and exclude science variables with dimensions that do no match those of the coordinate variables.

## [v1.1.1] - 2024-09-16

### Changed

- [[TRT-558](https://bugs.earthdata.nasa.gov/browse/TRT-558)]
The Swath Projector has been updated to use `earthdata-varinfo` version 3.0.0.
This update primarily involves the streamlining of the configuration file
schema. Please see the
[earthdata-varinfo release notes](https://github.com/nasa/earthdata-varinfo/releases/tag/3.0.0)
for more information. The configuration file used by the Swath Projector has
also been renamed to `earthdata_varinfo_config.json`.

## [v1.1.0] - 2024-08-29

### Changed

- [[DAS-1934](https://bugs.earthdata.nasa.gov/browse/DAS-1934)]
@@ -26,13 +46,15 @@ include updated documentation and files outlined by the

Repository structure changes include:

* Migrating `pymods` directory to `swath_projector`.
* Migrating `swotrepr.py` to `swath_projector/adapter.py`.
* Addition of `swath_projector/main.py`.
- Migrating `pymods` directory to `swath_projector`.
- Migrating `swotrepr.py` to `swath_projector/adapter.py`.
- Addition of `swath_projector/main.py`.

For more information on internal releases prior to NASA open-source approval,
see legacy-CHANGELOG.md.

[v1.1.0]:(https://github.com/nasa/harmony-swath-projector/releases/tag/1.0.1)
[v1.0.1]:(https://github.com/nasa/harmony-swath-projector/releases/tag/1.0.1)
[v1.0.0]:(https://github.com/nasa/harmony-swath-projector/releases/tag/1.0.0)
[v1.2.0]: (https://github.com/nasa/harmony-swath-projector/releases/tag/1.2.0)
[v1.1.1]: (https://github.com/nasa/harmony-swath-projector/releases/tag/1.1.1)
[v1.1.0]: (https://github.com/nasa/harmony-swath-projector/releases/tag/1.1.0)
[v1.0.1]: (https://github.com/nasa/harmony-swath-projector/releases/tag/1.0.1)
[v1.0.0]: (https://github.com/nasa/harmony-swath-projector/releases/tag/1.0.0)
6 changes: 3 additions & 3 deletions bin/project_local_granule.py
Original file line number Diff line number Diff line change
@@ -125,8 +125,8 @@ def project_granule(
{
'url': local_file_path,
'temporal': {
'start': '2021-01-03T23:45:00.000Z',
'end': '2020-01-04T00:00:00.000Z',
'start': '2020-01-03T23:45:00.000Z',
'end': '2025-01-04T00:00:00.000Z',
},
'bbox': [-180, -90, 180, 90],
}
@@ -141,5 +141,5 @@ def project_granule(

reprojector = SwathProjectorAdapter(message, config=config(False))

with patch('swotrepr.shutil.rmtree', side_effect=rmtree_side_effect):
with patch('swath_projector.adapter.shutil.rmtree', side_effect=rmtree_side_effect):
reprojector.invoke()
2 changes: 1 addition & 1 deletion docker/service_version.txt
Original file line number Diff line number Diff line change
@@ -1 +1 @@
1.1.0
1.2.0
2 changes: 1 addition & 1 deletion pip_requirements.txt
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
# Open source packages available from PyPI:
earthdata-varinfo ~= 1.0.0
earthdata-varinfo ~= 3.0.0
harmony-service-lib~=1.0.22
netCDF4 ~= 1.6.4
numpy ~= 1.24.2
26 changes: 0 additions & 26 deletions swath_projector/cf_config.json

This file was deleted.

63 changes: 63 additions & 0 deletions swath_projector/earthdata_varinfo_config.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,63 @@
{
"Identification": "Swath Projector VarInfo configuration",
"Version": 4,
"CollectionShortNamePath": [
"ShortName",
"collection_shortname"
],
"Mission": {
"VNP10": "VIIRS",
"TEMPO_O3TOT_L2": "TEMPO"
},
"ExcludedScienceVariables": [
{
"Applicability": {
"Mission": "TEMPO",
"ShortNamePath": "TEMPO_O3TOT_L2"
},
"VariablePattern": [
"/support_data/a_priori_layer_o3",
"/support_data/cal_adjustment",
"/support_data/dNdR",
"/support_data/layer_efficiency",
"/support_data/lut_wavelength",
"/support_data/N_value",
"/support_data/N_value_residual",
"/support_data/ozone_sensitivity_ratio",
"/support_data/step_1_N_value_residual",
"/support_data/step_2_N_value_residual",
"/support_data/temp_sensitivity_ratio"
]
}
],
"MetadataOverrides": [
{
"Applicability": {
"Mission": "VIIRS",
"ShortNamePath": "VNP10",
"VariablePattern": "/SnowData/.*"
},
"Attributes": [
{
"Name": "coordinates",
"Value": "/GeolocationData/latitude, /GeolocationData/longitude"
}
],
"_Description": "VNP10 SnowData variables have incorrect relative paths for coordinates."
},
{
"Applicability": {
"Mission": "TEMPO",
"ShortNamePath": "TEMPO_O3TOT_L2",
"VariablePattern": "^/product/.*|^/support_data/.*|^/geolocation/(?!time$).*"
},
"Attributes": [
{
"Name": "coordinates",
"Value": "/geolocation/latitude, /geolocation/longitude"
}
],
"_Description": "TEMPO_O3TOT_L2 variables only contain basenames for coordinates, which are found in sibling hierarchical groups. This rule fully qualifies the paths to these coordinates. Some variables in these groups are excluded via 'ExcludedScienceVariables'"
}
]
}
2 changes: 2 additions & 0 deletions swath_projector/interpolation.py
Original file line number Diff line number Diff line change
@@ -25,6 +25,7 @@
from swath_projector.utilities import (
create_coordinates_key,
get_coordinate_variable,
get_rows_per_scan,
get_scale_and_offset,
get_variable_file_path,
get_variable_numeric_fill_value,
@@ -268,6 +269,7 @@ def get_ewa_results(
ewa_information['target_area'],
variable['values'],
maximum_weight_mode=maximum_weight_mode,
rows_per_scan=get_rows_per_scan(variable['values'].shape[0]),
)

if variable['fill_value'] is not None:
2 changes: 1 addition & 1 deletion swath_projector/reproject.py
Original file line number Diff line number Diff line change
@@ -20,7 +20,7 @@
)
CRS_DEFAULT = '+proj=longlat +ellps=WGS84'
INTERPOLATION_DEFAULT = 'ewa-nn'
CF_CONFIG_FILE = 'swath_projector/cf_config.json'
CF_CONFIG_FILE = 'swath_projector/earthdata_varinfo_config.json'


def reproject(
16 changes: 7 additions & 9 deletions swath_projector/swath_geometry.py
Original file line number Diff line number Diff line change
@@ -13,7 +13,7 @@


def get_projected_resolution(
projection: Proj, longitudes: Variable, latitudes: Variable
projection: Proj, longitudes: np.ma.MaskedArray, latitudes: np.ma.MaskedArray
) -> Tuple[float]:
"""Find the resolution of the target grid in the projected coordinates, x
and y. First the perimeter points are found. These are then projected
@@ -40,7 +40,7 @@ def get_projected_resolution(


def get_extents_from_perimeter(
projection: Proj, longitudes: Variable, latitudes: Variable
projection: Proj, longitudes: np.ma.MaskedArray, latitudes: np.ma.MaskedArray
) -> Tuple[float]:
"""Find the swath extents in the target CRS. First the perimeter points of
unfilled valid pixels are found. These are then projected to the target
@@ -59,19 +59,17 @@ def get_extents_from_perimeter(
def get_projected_coordinates(
coordinates_mask: np.ma.core.MaskedArray,
projection: Proj,
longitudes: Variable,
latitudes: Variable,
longitudes: np.ma.MaskedArray,
latitudes: np.ma.MaskedArray,
) -> Tuple[np.ndarray]:
"""Get the required coordinate points projected in the target Coordinate
Reference System (CRS).
"""
if len(longitudes.shape) == 1:
coordinates = get_all_coordinates(longitudes[:], latitudes[:], coordinates_mask)
coordinates = get_all_coordinates(longitudes, latitudes, coordinates_mask)
else:
coordinates = get_perimeter_coordinates(
longitudes[:], latitudes[:], coordinates_mask
)
coordinates = get_perimeter_coordinates(longitudes, latitudes, coordinates_mask)

return reproject_coordinates(coordinates, projection)

@@ -135,7 +133,7 @@ def get_absolute_resolution(polygon_area: float, n_pixels: int) -> float:


def get_valid_coordinates_mask(
longitudes: Variable, latitudes: Variable
longitudes: np.ma.MaskedArray, latitudes: np.ma.MaskedArray
) -> np.ma.core.MaskedArray:
"""Get a `numpy` N-d array containing boolean values (0 or 1) indicating
whether the elements of both longitude and latitude are valid at that
61 changes: 52 additions & 9 deletions swath_projector/utilities.py
Original file line number Diff line number Diff line change
@@ -31,7 +31,8 @@ def get_variable_values(
As the variable data are returned as a `numpy.ma.MaskedArray`, the will
return no data in the filled pixels. To ensure that the data are
correctly handled, the fill value is applied to masked pixels using the
`filled` method.
`filled` method. The variable values are transposed if the `along-track`
dimension size is less than the `across-track` dimension size.
"""
# TODO: Remove in favour of apply2D or process_subdimension.
@@ -42,27 +43,38 @@ def get_variable_values(
if len(variable[:].shape) == 1:
return make_array_two_dimensional(variable[:])
elif 'time' in input_file.variables and 'time' in variable.dimensions:
# Assumption: Array = (1, y, x)
return variable[0][:].filled(fill_value=fill_value)
# Assumption: Array = (time, along-track, across-track)
return transpose_if_xdim_less_than_ydim(variable[0][:]).filled(
fill_value=fill_value
)
else:
# Assumption: Array = (y, x)
return variable[:].filled(fill_value=fill_value)
# Assumption: Array = (along-track, across-track)
return transpose_if_xdim_less_than_ydim(variable[:]).filled(
fill_value=fill_value
)


def get_coordinate_variable(
dataset: Dataset, coordinates_tuple: Tuple[str], coordinate_substring
) -> Optional[Variable]:
) -> Optional[np.ma.MaskedArray]:
"""Search the coordinate dataset names for a match to the substring,
which will be either "lat" or "lon". Return the corresponding variable
from the dataset. Only the base variable name is used, as the group
path may contain either of the strings as part of other words.
data from the dataset. Only the base variable name is used, as the group
path may contain either of the strings as part of other words. The
coordinate variables are transposed if the `along-track`dimension size is
less than the `across-track` dimension size.
"""
for coordinate in coordinates_tuple:
if coordinate_substring in coordinate.split('/')[-1] and variable_in_dataset(
coordinate, dataset
):
return dataset[coordinate]
# QuickFix (DAS-2216) for short and wide swaths
if dataset[coordinate].ndim == 1:
return dataset[coordinate][:]

return transpose_if_xdim_less_than_ydim(dataset[coordinate][:])

raise MissingCoordinatesError(coordinates_tuple)


@@ -216,3 +228,34 @@ def make_array_two_dimensional(one_dimensional_array: np.ndarray) -> np.ndarray:
"""
return np.expand_dims(one_dimensional_array, 1)


def transpose_if_xdim_less_than_ydim(
variable_values: np.ma.MaskedArray,
) -> np.ma.MaskedArray:
"""Return transposed variable when variable is wider than tall.
QuickFix (DAS-2216): We presume that a swath has more rows than columns and
if that's not the case we transpose it so that it does.
"""
if len(variable_values.shape) != 2:
raise ValueError(
f'Input variable must be 2 dimensional, but got {len(variable_values.shape)} dimensions.'
)
if variable_values.shape[0] < variable_values.shape[1]:
return np.ma.transpose(variable_values).copy()

return variable_values


def get_rows_per_scan(total_rows: int) -> int:
"""
Finds the smallest divisor of the total number of rows. If no divisor is
found, return the total number of rows.
"""
if total_rows < 2:
return 1
for row_number in range(2, int(total_rows**0.5) + 1):
if total_rows % row_number == 0:
return row_number
return total_rows
Loading