Improve caching.

* Move the caching module to core. Add an in memory cache that for caching function and method results during an energy management run (optimization run). Two decorators are provided for methods and functions. * Improve the file cache store by load and save functions. Make EOS load the cache file store on startup and save it on shutdown. Add a cyclic task that cleans the cache file store from outdated cache files. * Improve startup of EOSdash by EOS Make EOS starting EOSdash adhere to path configuration given in EOS. The whole environment from EOS is now passed to EOSdash. Should also prevent test errors due to unwanted/ wrong config file creation. * Allow to check server process startup by health status Both servers now provide a health endpoint that can be used to detect whether the server is running. This is also used for testing now. * Add administration endpoints to control caching from REST API. * Improve test support (to be able to test caching) - Add system test option to pytest for running tests with "real" resources - Add new test fixture to start server for test class Signed-off-by: Bobby Noelte <[email protected]>
Akkudoktor-EOS · Feb 7, 2025 · 24e1bf6 · 24e1bf6
1 parent da4994c
commit 24e1bf6
Show file tree

Hide file tree

Showing 35 changed files with 2,855 additions and 764 deletions.
diff --git a/docs/_generated/config.md b/docs/_generated/config.md
@@ -27,12 +27,10 @@ Validators:
 | ---- | -------------------- | ---- | --------- | ------- | ----------- |
 | data_folder_path | `EOS_GENERAL__DATA_FOLDER_PATH` | `Optional[pathlib.Path]` | `rw` | `None` | Path to EOS data directory. |
 | data_output_subpath | `EOS_GENERAL__DATA_OUTPUT_SUBPATH` | `Optional[pathlib.Path]` | `rw` | `output` | Sub-path for the EOS output data directory. |
-| data_cache_subpath | `EOS_GENERAL__DATA_CACHE_SUBPATH` | `Optional[pathlib.Path]` | `rw` | `cache` | Sub-path for the EOS cache data directory. |
 | latitude | `EOS_GENERAL__LATITUDE` | `Optional[float]` | `rw` | `52.52` | Latitude in decimal degrees, between -90 and 90, north is positive (ISO 19115) (°) |
 | longitude | `EOS_GENERAL__LONGITUDE` | `Optional[float]` | `rw` | `13.405` | Longitude in decimal degrees, within -180 to 180 (°) |
 | timezone | | `Optional[str]` | `ro` | `N/A` | Compute timezone based on latitude and longitude. |
 | data_output_path | | `Optional[pathlib.Path]` | `ro` | `N/A` | Compute data_output_path based on data_folder_path. |
-| data_cache_path | | `Optional[pathlib.Path]` | `ro` | `N/A` | Compute data_cache_path based on data_folder_path. |
 | config_folder_path | | `Optional[pathlib.Path]` | `ro` | `N/A` | Path to EOS configuration directory. |
 | config_file_path | | `Optional[pathlib.Path]` | `ro` | `N/A` | Path to EOS configuration file. |
 :::
@@ -46,7 +44,6 @@ Validators:
        "general": {
            "data_folder_path": null,
            "data_output_subpath": "output",
-           "data_cache_subpath": "cache",
            "latitude": 52.52,
            "longitude": 13.405
        }
@@ -62,18 +59,41 @@ Validators:
        "general": {
            "data_folder_path": null,
            "data_output_subpath": "output",
-           "data_cache_subpath": "cache",
            "latitude": 52.52,
            "longitude": 13.405,
            "timezone": "Europe/Berlin",
            "data_output_path": null,
-           "data_cache_path": null,
            "config_folder_path": "/home/user/.config/net.akkudoktoreos.net",
            "config_file_path": "/home/user/.config/net.akkudoktoreos.net/EOS.config.json"
        }
    }
 ```
 
+## Cache Configuration
+
+:::{table} cache
+:widths: 10 20 10 5 5 30
+:align: left
+
+| Name | Environment Variable | Type | Read-Only | Default | Description |
+| ---- | -------------------- | ---- | --------- | ------- | ----------- |
+| subpath | `EOS_CACHE__SUBPATH` | `Optional[pathlib.Path]` | `rw` | `cache` | Sub-path for the EOS cache data directory. |
+| cleanup_intervall | `EOS_CACHE__CLEANUP_INTERVALL` | `float` | `rw` | `300` | Intervall in seconds for EOS file cache cleanup. |
+:::
+
+### Example Input/Output
+
+```{eval-rst}
+.. code-block:: json
+
+   {
+       "cache": {
+           "subpath": "cache",
+           "cleanup_intervall": 300.0
+       }
+   }
+```
+
 ## Logging Configuration
 
 :::{table} logging
@@ -889,10 +909,13 @@ Attributes:
        "general": {
            "data_folder_path": null,
            "data_output_subpath": "output",
-           "data_cache_subpath": "cache",
            "latitude": 52.52,
            "longitude": 13.405
        },
+       "cache": {
+           "subpath": "cache",
+           "cleanup_intervall": 300.0
+       },
        "logging": {
            "level": "INFO"
        },

diff --git a/docs/_generated/openapi.md b/docs/_generated/openapi.md
@@ -166,6 +166,93 @@ Note:
 
 ---
 
+## GET /v1/admin/cache
+
+**Links**: [local](http://localhost:8503/docs#/default/fastapi_admin_cache_get_v1_admin_cache_get), [eos](https://petstore3.swagger.io/?url=https://raw.githubusercontent.com/Akkudoktor-EOS/EOS/refs/heads/main/openapi.json#/default/fastapi_admin_cache_get_v1_admin_cache_get)
+
+Fastapi Admin Cache Get
+
+```
+Current cache management data.
+
+Returns:
+    data (dict): The management data.
+```
+
+**Responses**:
+
+- **200**: Successful Response
+
+---
+
+## POST /v1/admin/cache/clear
+
+**Links**: [local](http://localhost:8503/docs#/default/fastapi_admin_cache_clear_post_v1_admin_cache_clear_post), [eos](https://petstore3.swagger.io/?url=https://raw.githubusercontent.com/Akkudoktor-EOS/EOS/refs/heads/main/openapi.json#/default/fastapi_admin_cache_clear_post_v1_admin_cache_clear_post)
+
+Fastapi Admin Cache Clear Post
+
+```
+Clear the cache from expired data.
+
+Deletes expired cache files.
+
+Args:
+    clear_all (Optional[bool]): Delete all cached files. Default is False.
+
+Returns:
+    data (dict): The management data after cleanup.
+```
+
+**Parameters**:
+
+- `clear_all` (query, optional): No description provided.
+
+**Responses**:
+
+- **200**: Successful Response
+
+- **422**: Validation Error
+
+---
+
+## POST /v1/admin/cache/load
+
+**Links**: [local](http://localhost:8503/docs#/default/fastapi_admin_cache_load_post_v1_admin_cache_load_post), [eos](https://petstore3.swagger.io/?url=https://raw.githubusercontent.com/Akkudoktor-EOS/EOS/refs/heads/main/openapi.json#/default/fastapi_admin_cache_load_post_v1_admin_cache_load_post)
+
+Fastapi Admin Cache Load Post
+
+```
+Load cache management data.
+
+Returns:
+    data (dict): The management data that was loaded.
+```
+
+**Responses**:
+
+- **200**: Successful Response
+
+---
+
+## POST /v1/admin/cache/save
+
+**Links**: [local](http://localhost:8503/docs#/default/fastapi_admin_cache_save_post_v1_admin_cache_save_post), [eos](https://petstore3.swagger.io/?url=https://raw.githubusercontent.com/Akkudoktor-EOS/EOS/refs/heads/main/openapi.json#/default/fastapi_admin_cache_save_post_v1_admin_cache_save_post)
+
+Fastapi Admin Cache Save Post
+
+```
+Save the current cache management data.
+
+Returns:
+    data (dict): The management data that was saved.
+```
+
+**Responses**:
+
+- **200**: Successful Response
+
+---
+
 ## GET /v1/config
 
 **Links**: [local](http://localhost:8503/docs#/default/fastapi_config_get_v1_config_get), [eos](https://petstore3.swagger.io/?url=https://raw.githubusercontent.com/Akkudoktor-EOS/EOS/refs/heads/main/openapi.json#/default/fastapi_config_get_v1_config_get)
@@ -238,14 +325,14 @@ Returns:
 
 ---
 
-## PUT /v1/config/reset
+## POST /v1/config/reset
 
-**Links**: [local](http://localhost:8503/docs#/default/fastapi_config_update_post_v1_config_reset_put), [eos](https://petstore3.swagger.io/?url=https://raw.githubusercontent.com/Akkudoktor-EOS/EOS/refs/heads/main/openapi.json#/default/fastapi_config_update_post_v1_config_reset_put)
+**Links**: [local](http://localhost:8503/docs#/default/fastapi_config_reset_post_v1_config_reset_post), [eos](https://petstore3.swagger.io/?url=https://raw.githubusercontent.com/Akkudoktor-EOS/EOS/refs/heads/main/openapi.json#/default/fastapi_config_reset_post_v1_config_reset_post)
 
-Fastapi Config Update Post
+Fastapi Config Reset Post
 
 ```
-Reset the configuration to the EOS configuration file.
+Reset the configuration.
 
 Returns:
     configuration (ConfigEOS): The current configuration after update.
@@ -321,6 +408,22 @@ Returns:
 
 ---
 
+## GET /v1/health
+
+**Links**: [local](http://localhost:8503/docs#/default/fastapi_health_get_v1_health_get), [eos](https://petstore3.swagger.io/?url=https://raw.githubusercontent.com/Akkudoktor-EOS/EOS/refs/heads/main/openapi.json#/default/fastapi_health_get_v1_health_get)
+
+Fastapi Health Get
+
+```
+Health check endpoint to verify that the EOS server is alive.
+```
+
+**Responses**:
+
+- **200**: Successful Response
+
+---
+
 ## PUT /v1/measurement/data
 
 **Links**: [local](http://localhost:8503/docs#/default/fastapi_measurement_data_put_v1_measurement_data_put), [eos](https://petstore3.swagger.io/?url=https://raw.githubusercontent.com/Akkudoktor-EOS/EOS/refs/heads/main/openapi.json#/default/fastapi_measurement_data_put_v1_measurement_data_put)
@@ -537,6 +640,56 @@ Merge the measurement of given key and value into EOS measurements at given date
 
 ---
 
+## PUT /v1/prediction/import/{provider_id}
+
+**Links**: [local](http://localhost:8503/docs#/default/fastapi_prediction_import_provider_v1_prediction_import__provider_id__put), [eos](https://petstore3.swagger.io/?url=https://raw.githubusercontent.com/Akkudoktor-EOS/EOS/refs/heads/main/openapi.json#/default/fastapi_prediction_import_provider_v1_prediction_import__provider_id__put)
+
+Fastapi Prediction Import Provider
+
+```
+Import prediction for given provider ID.
+
+Args:
+    provider_id: ID of provider to update.
+    data: Prediction data.
+    force_enable: Update data even if provider is disabled.
+        Defaults to False.
+```
+
+**Parameters**:
+
+- `provider_id` (path, required): Provider ID.
+
+- `force_enable` (query, optional): No description provided.
+
+**Request Body**:
+
+- `application/json`: {
+  "anyOf": [
+    {
+      "$ref": "#/components/schemas/PydanticDateTimeDataFrame"
+    },
+    {
+      "$ref": "#/components/schemas/PydanticDateTimeData"
+    },
+    {
+      "type": "object"
+    },
+    {
+      "type": "null"
+    }
+  ],
+  "title": "Data"
+}
+
+**Responses**:
+
+- **200**: Successful Response
+
+- **422**: Validation Error
+
+---
+
 ## GET /v1/prediction/keys
 
 **Links**: [local](http://localhost:8503/docs#/default/fastapi_prediction_keys_get_v1_prediction_keys_get), [eos](https://petstore3.swagger.io/?url=https://raw.githubusercontent.com/Akkudoktor-EOS/EOS/refs/heads/main/openapi.json#/default/fastapi_prediction_keys_get_v1_prediction_keys_get)
@@ -580,7 +733,7 @@ Args:
 
 - `end_datetime` (query, optional): Ending datetime (exclusive).
 
-- `interval` (query, optional): Time duration for each interval.
+- `interval` (query, optional): Time duration for each interval. Defaults to 1 hour.
 
 **Responses**:
 

diff --git a/docs/akkudoktoreos/prediction.md b/docs/akkudoktoreos/prediction.md
@@ -141,9 +141,12 @@ The prediction key for the electricity price forecast data is:
 - `elecprice_marketprice_wh`: Electricity market price per Wh (€/Wh).
 
 The electricity proce forecast data must be provided in one of the formats described in
-<project:#prediction-import-providers>. The data source must be given in the
+<project:#prediction-import-providers>. The data source can be given in the
 `import_file_path` or `import_json` configuration option.
 
+The data may additionally or solely be provided by the
+**PUT** `/v1/prediction/import/ElecPriceImport` endpoint.
+
 ## Load Prediction
 
 Prediction keys:
@@ -184,9 +187,12 @@ The prediction keys for the load forecast data are:
 - `load_mean_adjusted`: Predicted load mean value adjusted by load measurement (W).
 
 The load forecast data must be provided in one of the formats described in
-<project:#prediction-import-providers>. The data source must be given in the `loadimport_file_path`
+<project:#prediction-import-providers>. The data source can be given in the `loadimport_file_path`
 or `loadimport_json` configuration option.
 
+The data may additionally or solely be provided by the
+**PUT** `/v1/prediction/import/LoadImport` endpoint.
+
 ## PV Power Prediction
 
 Prediction keys:
@@ -362,9 +368,12 @@ The prediction keys for the PV forecast data are:
 - `pvforecast_dc_power`: Total AC power (W).
 
 The PV forecast data must be provided in one of the formats described in
-<project:#prediction-import-providers>. The data source must be given in the
+<project:#prediction-import-providers>. The data source can be given in the
 `import_file_path` or `import_json` configuration option.
 
+The data may additionally or solely be provided by the
+**PUT** `/v1/prediction/import/PVForecastImport` endpoint.
+
 ## Weather Prediction
 
 Prediction keys:
@@ -460,7 +469,7 @@ The `WeatherImport` provider is designed to import weather forecast data from a
 string. An external entity should update the file or JSON string whenever new prediction data
 becomes available.
 
-The prediction keys for the PV forecast data are:
+The prediction keys for the weather forecast data are:
 
 - `weather_dew_point`: Dew Point (°C)
 - `weather_dhi`: Diffuse Horizontal Irradiance (W/m2)
@@ -486,5 +495,8 @@ The prediction keys for the PV forecast data are:
 - `weather_wind_speed`: Wind Speed (kmph)
 
 The PV forecast data must be provided in one of the formats described in
-<project:#prediction-import-providers>. The data source must be given in the
+<project:#prediction-import-providers>. The data source can be given in the
 `import_file_path` or `import_json` configuration option.
+
+The data may additionally or solely be provided by the
+**PUT** `/v1/prediction/import/WeatherImport` endpoint.