PB-1169 Search forecast properties #500

benschs · 2025-01-23T09:46:08Z

spec

Add forecast properties to search filter for POST. Refactor
forecast:variable and forecast:perturbed descriptions into schema properties to
be reused.

New spec for POST /search

In the GET parameters I decided to use underscores(_) instead of colons(:) for the parameter names as colons need to be url encoded. The parameter names are quite long, but I think it's ok to be more descriptive, otherwise we could also shorten forecast to fc (e.g. fc_reference_datetime). In the POST parameters i kept the colon in the names to stay consistent.
Decided to only support filtering of forecast properties in the POST /search request.

Implementation

Add filters for forecast properties to search endpoint.
Can filter for forecast:reference_datetime either by exact value or by a range.

I did not add any indexes for the forecast fields. Do you think we should add them already on all fields? I can't really tell how the search will be used and if it makes sense to just add an index as it will make writes slower.
Decided to add indexes on all forecast fields.

Refactor

Linter fails as the models.py file is larger than 1000 lines. The last to commit move the model into a new directory and split it into multiple files.

asteiner-swisstopo

Looks good to me thanks 👍 A few suggestions on the code comments and one suggestion for less code duplication.

asteiner-swisstopo · 2025-01-23T12:34:04Z

app/stac_api/managers.py

+            queryset:
+                 A django queryset (https://docs.djangoproject.com/en/3.0/ref/models/querysets/)


That refers to self, right? I would not list that here.

Same on line 131.

I removed the "queryset" comment

asteiner-swisstopo · 2025-01-23T12:34:59Z

app/stac_api/managers.py

+            queryset:
+                 A django queryset (https://docs.djangoproject.com/en/3.0/ref/models/querysets/)
+            date_time:
+                A string


Maybe add an example of how this would typically look like ('2020-10-28T13:05:10Z').

Added more info for date_time

asteiner-swisstopo · 2025-01-23T12:38:39Z

app/stac_api/managers.py

+            queryset:
+                A django queryset (https://docs.djangoproject.com/en/3.0/ref/models/querysets/)
+            start_datetime:
+                A string with the start datetime


I would add a note that this can also be ".."

Added information about open start and open end in the comment

asteiner-swisstopo · 2025-01-23T12:50:47Z

app/stac_api/managers.py

+    def filter_by_forecast_horizon(self, duration):
+        return self.get_queryset().filter_by_forecast_horizon(duration)
+
+    def filter_by_forecast_duration(self, duration):
+        return self.get_queryset().filter_by_forecast_duration(duration)
+
+    def filter_by_forecast_variable(self, val):
+        return self.get_queryset().filter_by_forecast_variable(val)
+
+    def filter_by_forecast_perturbed(self, pert):
+        return self.get_queryset().filter_by_forecast_perturbed(pert)
+


Couldn't all these be replaced by something less redundant?

For example

def filter_by_field(self, field_name, value): return self.get_queryset().filter(**{field_name: value})

I had a deeper look at this, but am not sure it makes sense to just change this. I prefer to keep it consistent with the existing field filters. Maybe in the future we could refactor all of the search filters.

asteiner-swisstopo · 2025-01-23T13:29:26Z

app/tests/tests_10/test_search_endpoint.py

+        cls.items = cls.factory.create_item_samples(
+            [
+                'item-forecast-1',
+                'item-forecast-2',
+                'item-forecast-3',
+                'item-forecast-4',
+                'item-forecast-5'
+            ],
+            cls.collection,
+            db_create=True,
+        )


Consider creating these items individually with create_item_sample such that you can set the name of the samples. Otherwise in the tests you refer to the automatically set names ("item-1", "item-2", ...) instead of the ones on the item ("item-forecast-1", "item-forecast-2", ...).

Changed to use the create_item_sample and check against the item forecast names

Add forecast properties to search filter for GET and POST. Refactor forecast:variable and forecast:perturbed descriptions into schema properties to be reused.

Add filters for forecast properties to search endpoints. Can filter for forecast_reference_datetime either by exact value or by a range.

Decided to remove the forecast parameters because the variable name differs to the post variable names (using underscore instead of colon) and colons need url encoding.

Decided to remove the forecast filtering options from the GET /search request due to colons in the parameter names. Also fixed the POST request body to actually use the parameters that contain a colon.

Fix code comments. Refactor create sample data names.

Add index for the item forecast properties as they can be used as filters in the item search.

Move models file into folder /models and rename file to general.py. Fix all imports that referenced the models file.

Move collection related models into separate file. Move item related models into separate file. Fix import statements across all files.

github-actions bot added the feature label Jan 23, 2025

benschs requested review from asteiner-swisstopo and boecklic January 23, 2025 09:48

asteiner-swisstopo approved these changes Jan 23, 2025

View reviewed changes

benschs force-pushed the feat-pb-1169-search-forecast branch from 8c248f9 to 3e0d8fc Compare January 23, 2025 14:54

benschs added 5 commits January 27, 2025 08:57

PB-1169: Spec search forecast fields

2f1d51d

Add forecast properties to search filter for GET and POST. Refactor forecast:variable and forecast:perturbed descriptions into schema properties to be reused.

PB-1169: Search forecast fields

ad90dab

Add filters for forecast properties to search endpoints. Can filter for forecast_reference_datetime either by exact value or by a range.

PB-1169: Spec remove forecast params from Get search

b29937c

Decided to remove the forecast parameters because the variable name differs to the post variable names (using underscore instead of colon) and colons need url encoding.

PB-1169: Remove forecast filter from GET search

757c46d

Decided to remove the forecast filtering options from the GET /search request due to colons in the parameter names. Also fixed the POST request body to actually use the parameters that contain a colon.

Fix for review comments

d2492aa

Fix code comments. Refactor create sample data names.

benschs force-pushed the feat-pb-1169-search-forecast branch from 3e0d8fc to d2492aa Compare January 27, 2025 07:58

benschs added 3 commits January 27, 2025 10:23

PB-1169: Index item forecast fields

e5aa63f

Add index for the item forecast properties as they can be used as filters in the item search.

Refactor models - move file

18d698b

Move models file into folder /models and rename file to general.py. Fix all imports that referenced the models file.

Refactor models - split into files

e606b08

Move collection related models into separate file. Move item related models into separate file. Fix import statements across all files.

benschs merged commit 1e2d766 into develop Jan 27, 2025
5 checks passed

benschs deleted the feat-pb-1169-search-forecast branch January 27, 2025 15:48

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

PB-1169 Search forecast properties #500

PB-1169 Search forecast properties #500

benschs commented Jan 23, 2025 •

edited

Loading

asteiner-swisstopo left a comment

asteiner-swisstopo Jan 23, 2025

benschs Jan 27, 2025

asteiner-swisstopo Jan 23, 2025

benschs Jan 27, 2025

asteiner-swisstopo Jan 23, 2025

benschs Jan 27, 2025

asteiner-swisstopo Jan 23, 2025

benschs Jan 27, 2025

asteiner-swisstopo Jan 23, 2025

benschs Jan 27, 2025

		queryset:
		A django queryset (https://docs.djangoproject.com/en/3.0/ref/models/querysets/)

PB-1169 Search forecast properties #500

PB-1169 Search forecast properties #500

Conversation

benschs commented Jan 23, 2025 • edited Loading

spec

Implementation

Refactor

asteiner-swisstopo left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

benschs commented Jan 23, 2025 •

edited

Loading