Improve EOSdash.

Make EOSdash use UI components from MonsterUI to ease further development.

- Add a first menu with some dummy pages and the configuration page.
- Make the configuration scrollable.
- Add markdown component that uses markdown-it-py (same as used by
  the myth-parser for documentation generation).
- Add bokeh (https://docs.bokeh.org/) component for charts
- Added several prediction charts to demo
- Add a footer that displays connection status with EOS server
- Add logo and favicon

Update EOS server:

- Provide health endpoint for alive checking.
- Move error message generation to extra module

Signed-off-by: Bobby Noelte <[email protected]>

docs/_generated/openapi.md

@@ -332,7 +332,7 @@ Returns:
 Fastapi Config Reset Post
 
 ```
-Reset the configuration.
+Reset the configuration to the EOS configuration file.
 
 Returns:
     configuration (ConfigEOS): The current configuration after update.
@@ -640,6 +640,42 @@ Merge the measurement of given key and value into EOS measurements at given date
 
 ---
 
+## GET /v1/prediction/dataframe
+
+**Links**: [local](http://localhost:8503/docs#/default/fastapi_prediction_dataframe_get_v1_prediction_dataframe_get), [eos](https://petstore3.swagger.io/?url=https://raw.githubusercontent.com/Akkudoktor-EOS/EOS/refs/heads/main/openapi.json#/default/fastapi_prediction_dataframe_get_v1_prediction_dataframe_get)
+
+Fastapi Prediction Dataframe Get
+
+```
+Get prediction for given key within given date range as series.
+
+Args:
+    key (str): Prediction key
+    start_datetime (Optional[str]): Starting datetime (inclusive).
+        Defaults to start datetime of latest prediction.
+    end_datetime (Optional[str]: Ending datetime (exclusive).
+
+Defaults to end datetime of latest prediction.
+```
+
+**Parameters**:
+
+- `keys` (query, required): Prediction keys.
+
+- `start_datetime` (query, optional): Starting datetime (inclusive).
+
+- `end_datetime` (query, optional): Ending datetime (exclusive).
+
+- `interval` (query, optional): Time duration for each interval. Defaults to 1 hour.
+
+**Responses**:
+
+- **200**: Successful Response
+
+- **422**: Validation Error
+
+---
+
 ## PUT /v1/prediction/import/{provider_id}
 
 **Links**: [local](http://localhost:8503/docs#/default/fastapi_prediction_import_provider_v1_prediction_import__provider_id__put), [eos](https://petstore3.swagger.io/?url=https://raw.githubusercontent.com/Akkudoktor-EOS/EOS/refs/heads/main/openapi.json#/default/fastapi_prediction_import_provider_v1_prediction_import__provider_id__put)

openapi.json

@@ -3279,7 +3279,7 @@
         },
         "/v1/config/reset": {
             "post": {
-                "description": "Reset the configuration.\n\nReturns:\n    configuration (ConfigEOS): The current configuration after update.",
+                "description": "Reset the configuration to the EOS configuration file.\n\nReturns:\n    configuration (ConfigEOS): The current configuration after update.",
                 "operationId": "fastapi_config_reset_post_v1_config_reset_post",
                 "responses": {
                     "200": {
@@ -3859,6 +3859,108 @@
                 ]
             }
         },
+        "/v1/prediction/dataframe": {
+            "get": {
+                "description": "Get prediction for given key within given date range as series.\n\nArgs:\n    key (str): Prediction key\n    start_datetime (Optional[str]): Starting datetime (inclusive).\n        Defaults to start datetime of latest prediction.\n    end_datetime (Optional[str]: Ending datetime (exclusive).\n\nDefaults to end datetime of latest prediction.",
+                "operationId": "fastapi_prediction_dataframe_get_v1_prediction_dataframe_get",
+                "parameters": [
+                    {
+                        "description": "Prediction keys.",
+                        "in": "query",
+                        "name": "keys",
+                        "required": true,
+                        "schema": {
+                            "description": "Prediction keys.",
+                            "items": {
+                                "type": "string"
+                            },
+                            "title": "Keys",
+                            "type": "array"
+                        }
+                    },
+                    {
+                        "description": "Starting datetime (inclusive).",
+                        "in": "query",
+                        "name": "start_datetime",
+                        "required": false,
+                        "schema": {
+                            "anyOf": [
+                                {
+                                    "type": "string"
+                                },
+                                {
+                                    "type": "null"
+                                }
+                            ],
+                            "description": "Starting datetime (inclusive).",
+                            "title": "Start Datetime"
+                        }
+                    },
+                    {
+                        "description": "Ending datetime (exclusive).",
+                        "in": "query",
+                        "name": "end_datetime",
+                        "required": false,
+                        "schema": {
+                            "anyOf": [
+                                {
+                                    "type": "string"
+                                },
+                                {
+                                    "type": "null"
+                                }
+                            ],
+                            "description": "Ending datetime (exclusive).",
+                            "title": "End Datetime"
+                        }
+                    },
+                    {
+                        "description": "Time duration for each interval. Defaults to 1 hour.",
+                        "in": "query",
+                        "name": "interval",
+                        "required": false,
+                        "schema": {
+                            "anyOf": [
+                                {
+                                    "type": "string"
+                                },
+                                {
+                                    "type": "null"
+                                }
+                            ],
+                            "description": "Time duration for each interval. Defaults to 1 hour.",
+                            "title": "Interval"
+                        }
+                    }
+                ],
+                "responses": {
+                    "200": {
+                        "content": {
+                            "application/json": {
+                                "schema": {
+                                    "$ref": "#/components/schemas/PydanticDateTimeDataFrame"
+                                }
+                            }
+                        },
+                        "description": "Successful Response"
+                    },
+                    "422": {
+                        "content": {
+                            "application/json": {
+                                "schema": {
+                                    "$ref": "#/components/schemas/HTTPValidationError"
+                                }
+                            }
+                        },
+                        "description": "Validation Error"
+                    }
+                },
+                "summary": "Fastapi Prediction Dataframe Get",
+                "tags": [
+                    "prediction"
+                ]
+            }
+        },
         "/v1/prediction/import/{provider_id}": {
             "put": {
                 "description": "Import prediction for given provider ID.\n\nArgs:\n    provider_id: ID of provider to update.\n    data: Prediction data.\n    force_enable: Update data even if provider is disabled.\n        Defaults to False.",
@@ -4213,19 +4315,33 @@
                         "name": "force_update",
                         "required": false,
                         "schema": {
+                            "anyOf": [
+                                {
+                                    "type": "boolean"
+                                },
+                                {
+                                    "type": "null"
+                                }
+                            ],
                             "default": false,
-                            "title": "Force Update",
-                            "type": "boolean"
+                            "title": "Force Update"
                         }
                     },
                     {
                         "in": "query",
                         "name": "force_enable",
                         "required": false,
                         "schema": {
+                            "anyOf": [
+                                {
+                                    "type": "boolean"
+                                },
+                                {
+                                    "type": "null"
+                                }
+                            ],
                             "default": false,
-                            "title": "Force Enable",
-                            "type": "boolean"
+                            "title": "Force Enable"
                         }
                     }
                 ],

requirements.txt

@@ -4,6 +4,10 @@ numpydantic==1.6.7
 matplotlib==3.10.0
 fastapi[standard]==0.115.7
 python-fasthtml==0.12.0
+MonsterUI==0.0.29
+markdown-it-py==3.0.0
+mdit-py-plugins==0.4.2
+bokeh==3.6.3
 uvicorn==0.34.0
 scikit-learn==1.6.1
 timezonefinder==6.5.8

src/akkudoktoreos/core/dataabc.py

@@ -1845,6 +1845,88 @@ def key_to_array(
 
         return array
 
+    def keys_to_dataframe(
+        self,
+        keys: list[str],
+        start_datetime: Optional[DateTime] = None,
+        end_datetime: Optional[DateTime] = None,
+        interval: Optional[Any] = None,  # Duration assumed
+        fill_method: Optional[str] = None,
+    ) -> pd.DataFrame:
+        """Retrieve a dataframe indexed by fixed time intervals for specified keys from the data in each DataProvider.
+
+        Generates a pandas DataFrame using the NumPy arrays for each specified key, ensuring a common time index..
+
+        Args:
+            keys (list[str]): A list of field names to retrieve.
+            start_datetime (datetime, optional): Start date for filtering records (inclusive).
+            end_datetime (datetime, optional): End date for filtering records (exclusive).
+            interval (duration, optional): The fixed time interval. Defaults to 1 hour.
+            fill_method (str, optional): Method to handle missing values during resampling.
+                - 'linear': Linearly interpolate missing values (for numeric data only).
+                - 'ffill': Forward fill missing values.
+                - 'bfill': Backward fill missing values.
+                - 'none': Defaults to 'linear' for numeric values, otherwise 'ffill'.
+
+        Returns:
+            pd.DataFrame: A DataFrame where each column represents a key's array with a common time index.
+
+        Raises:
+            KeyError: If no valid data is found for any of the requested keys.
+            ValueError: If any retrieved array has a different time index than the first one.
+        """
+        # Ensure datetime objects are normalized
+        start_datetime = to_datetime(start_datetime, to_maxtime=False) if start_datetime else None
+        end_datetime = to_datetime(end_datetime, to_maxtime=False) if end_datetime else None
+        if interval is None:
+            interval = to_duration("1 hour")
+        if start_datetime is None:
+            # Take earliest datetime of all providers that are enabled
+            for provider in self.enabled_providers:
+                if start_datetime is None:
+                    start_datetime = provider.min_datetime
+                elif (
+                    provider.min_datetime
+                    and compare_datetimes(provider.min_datetime, start_datetime).lt
+                ):
+                    start_datetime = provider.min_datetime
+        if end_datetime is None:
+            # Take latest datetime of all providers that are enabled
+            for provider in self.enabled_providers:
+                if end_datetime is None:
+                    end_datetime = provider.max_datetime
+                elif (
+                    provider.max_datetime
+                    and compare_datetimes(provider.max_datetime, end_datetime).gt
+                ):
+                    end_datetime = provider.min_datetime
+            if end_datetime:
+                end_datetime.add(seconds=1)
+
+        # Create a DatetimeIndex based on start, end, and interval
+        reference_index = pd.date_range(
+            start=start_datetime, end=end_datetime, freq=interval, inclusive="left"
+        )
+
+        data = {}
+        for key in keys:
+            try:
+                array = self.key_to_array(key, start_datetime, end_datetime, interval, fill_method)
+
+                if len(array) != len(reference_index):
+                    raise ValueError(
+                        f"Array length mismatch for key '{key}' (expected {len(reference_index)}, got {len(array)})"
+                    )
+
+                data[key] = array
+            except KeyError as e:
+                raise KeyError(f"Failed to retrieve data for key '{key}': {e}")
+
+        if not data:
+            raise KeyError(f"No valid data found for the requested keys {keys}.")
+
+        return pd.DataFrame(data, index=reference_index)
+
     def provider_by_id(self, provider_id: str) -> DataProvider:
         """Retrieves a data provider by its unique identifier.
 

src/akkudoktoreos/core/pydantic.py

@@ -437,6 +437,10 @@ def to_dataframe(self) -> pd.DataFrame:
         index = pd.Index([to_datetime(dt, in_timezone=self.tz) for dt in df.index])
         df.index = index
 
+        # Check if 'date_time' column exists, if not, create it
+        if "date_time" not in df.columns:
+            df["date_time"] = df.index
+
         dtype_mapping = {
             "int": int,
             "float": float,

src/akkudoktoreos/prediction/elecpriceimport.py

@@ -63,6 +63,9 @@ def provider_id(cls) -> str:
         return "ElecPriceImport"
 
     def _update_data(self, force_update: Optional[bool] = False) -> None:
+        if self.config.elecprice.provider_settings is None:
+            logger.debug(f"{self.provider_id()} data update without provider settings.")
+            return
         if self.config.elecprice.provider_settings.import_file_path:
             self.import_from_file(
                 self.config.elecprice.provider_settings.import_file_path,

src/akkudoktoreos/prediction/loadimport.py

@@ -62,6 +62,9 @@ def provider_id(cls) -> str:
         return "LoadImport"
 
     def _update_data(self, force_update: Optional[bool] = False) -> None:
+        if self.config.load.provider_settings is None:
+            logger.debug(f"{self.provider_id()} data update without provider settings.")
+            return
         if self.config.load.provider_settings.import_file_path:
             self.import_from_file(self.config.provider_settings.import_file_path, key_prefix="load")
         if self.config.load.provider_settings.import_json:

src/akkudoktoreos/prediction/pvforecastimport.py

@@ -63,6 +63,9 @@ def provider_id(cls) -> str:
         return "PVForecastImport"
 
     def _update_data(self, force_update: Optional[bool] = False) -> None:
+        if self.config.pvforecast.provider_settings is None:
+            logger.debug(f"{self.provider_id()} data update without provider settings.")
+            return
         if self.config.pvforecast.provider_settings.import_file_path is not None:
             self.import_from_file(
                 self.config.pvforecast.provider_settings.import_file_path,

src/akkudoktoreos/prediction/weatherimport.py

@@ -63,6 +63,9 @@ def provider_id(cls) -> str:
         return "WeatherImport"
 
     def _update_data(self, force_update: Optional[bool] = False) -> None:
+        if self.config.weather.provider_settings is None:
+            logger.debug(f"{self.provider_id()} data update without provider settings.")
+            return
         if self.config.weather.provider_settings.import_file_path:
             self.import_from_file(
                 self.config.weather.provider_settings.import_file_path, key_prefix="weather"

src/akkudoktoreos/server/dash/assets/favicon/site.webmanifest

@@ -0,0 +1 @@
+{"name":"","short_name":"","icons":[{"src":"/android-chrome-192x192.png","sizes":"192x192","type":"image/png"},{"src":"/android-chrome-512x512.png","sizes":"512x512","type":"image/png"}],"theme_color":"#ffffff","background_color":"#ffffff","display":"standalone"}

src/akkudoktoreos/server/dash/bokeh.py

@@ -0,0 +1,38 @@
+# Module taken from https://github.com/koaning/fh-altair
+# MIT license
+
+from typing import Optional
+
+from bokeh.embed import components
+from bokeh.models import Plot
+from monsterui.franken import H4, Card, NotStr, Script
+
+BokehJS = [
+    Script(src="https://cdn.bokeh.org/bokeh/release/bokeh-3.6.3.min.js", crossorigin="anonymous"),
+    Script(
+        src="https://cdn.bokeh.org/bokeh/release/bokeh-widgets-3.6.3.min.js",
+        crossorigin="anonymous",
+    ),
+    Script(
+        src="https://cdn.bokeh.org/bokeh/release/bokeh-tables-3.6.3.min.js", crossorigin="anonymous"
+    ),
+    Script(
+        src="https://cdn.bokeh.org/bokeh/release/bokeh-gl-3.6.3.min.js", crossorigin="anonymous"
+    ),
+    Script(
+        src="https://cdn.bokeh.org/bokeh/release/bokeh-mathjax-3.6.3.min.js",
+        crossorigin="anonymous",
+    ),
+]
+
+
+def Bokeh(plot: Plot, header: Optional[str] = None) -> Card:
+    """Converts an Bokeh plot to a FastHTML FT component."""
+    script, div = components(plot)
+    if header:
+        header = H4(header, cls="mt-2")
+    return Card(
+        NotStr(div),
+        NotStr(script),
+        header=header,
+    )