Interpolate At Time update to ignore null values in calculation (rtdi…

…p#792)

* Inital Commit

Signed-off-by: GBBBAS <[email protected]>

* Interpolate At Time Update

Signed-off-by: GBBBAS <[email protected]>

* Update Tests

Signed-off-by: GBBBAS <[email protected]>

* Remove Uni

Signed-off-by: GBBBAS <[email protected]>

---------

Signed-off-by: GBBBAS <[email protected]>

docs/assets/extra.css

@@ -15,13 +15,13 @@
  */
 
 :root {
-    --md-primary-fg-color: #4e08c7 !important;
-    --md-primary-mg-color: #d445a3 !important;
-    --md-accent-fg-color: #bb1fa4 !important;
-    --md-primary-bg-color: white !important;
-    --md-primary-text-slate: white !important;
-    --md-primary-bg-slate: #2f303e !important;
-  }
+  --md-primary-fg-color: #4e08c7 !important;
+  --md-primary-mg-color: #d445a3 !important;
+  --md-accent-fg-color: #bb1fa4 !important;
+  --md-primary-bg-color: white !important;
+  --md-primary-text-slate: white !important;
+  --md-primary-bg-slate: #2f303e !important;
+}
 
 /* header font colour */
 .md-header {
@@ -41,7 +41,7 @@
 }
 
 .md-nav__item .md-nav__link--active {
-  color:#d445a3;
+  color: #d445a3;
 }
 
 .image-center {

docs/getting-started/installation.md

@@ -12,6 +12,8 @@ This article provides a guide on how to install the RTDIP SDK. Get started by en
 
 ## Prerequisites
 
+<!-- --8<-- [start:prerequisites] -->
+
 ### Python
 
 There are a few things to note before using the RTDIP SDK. The following prerequisites will need to be installed on your local machine.
@@ -55,6 +57,8 @@ Installing the RTDIP can be done using a package installer, such as [Pip](https:
        
         micromamba self-update
 
+<!-- --8<-- [end:prerequisites] -->
+
 ### ODBC
 To use pyodbc or turbodbc python libraries, ensure it is installed as per the below and the ODBC driver is installed as per these [instructions](https://docs.microsoft.com/en-us/azure/databricks/integrations/bi/jdbc-odbc-bi#download-the-odbc-driver).
 
@@ -87,7 +91,7 @@ To use RTDIP Pipelines components in your own environment that leverages [pyspar
         - defaults
     dependencies:
         - python==3.11
-        - pip==23.0.1
+        - pip
         - openjdk==11.0.15
         - pip:
             - rtdip-sdk
@@ -108,6 +112,8 @@ To use RTDIP Pipelines components in your own environment that leverages [pyspar
 
 ## Installing the RTDIP SDK
 
+<!-- --8<-- [start:installation] -->
+
 RTDIP SDK is a PyPi package that can be found [here](https://pypi.org/project/rtdip-sdk/). On this page you can find the **project description**,  **release history**, **statistics**, **project links** and **maintainers**.
 
 Features of the SDK can be installed using different extras statements when installing the **rtdip-sdk** package:
@@ -128,7 +134,7 @@ Features of the SDK can be installed using different extras statements when inst
         pip install "rtdip-sdk[pipelines,pyspark]"
 
     !!! note "Java"
-        Ensure that Java is installed prior to installing the rtdip-sdk with the **[pipelines,pyspark]**. See [here](#java) for more information.
+        Ensure that Java is installed prior to installing the rtdip-sdk with the **[pipelines,pyspark]**. See [here](https://www.rtdip.io/getting-started/installation/#java) for more information.
 
 The following provides examples of how to install the RTDIP SDK package with Pip, Conda or Micromamba. Please note the section above to update any extra packages to be installed as part of the RTDIP SDK.
 
@@ -153,7 +159,7 @@ The following provides examples of how to install the RTDIP SDK package with Pip
         - defaults
     dependencies:
         - python==3.11
-        - pip==23.0.1
+        - pip
         - pip:
             - rtdip-sdk
     ```
@@ -177,7 +183,7 @@ The following provides examples of how to install the RTDIP SDK package with Pip
         - defaults
     dependencies:
         - python==3.11
-        - pip==23.0.1
+        - pip
         - pip:
             - rtdip-sdk
     ```
@@ -190,6 +196,9 @@ The following provides examples of how to install the RTDIP SDK package with Pip
 
         micromamba update -f environment.yml
 
+
+<!-- --8<-- [end:installation] -->
+
 ## Next steps
 Once the installation is complete you can learn how to use the SDK [here.](../sdk/overview.md)
 

docs/sdk/authentication/azure.md

@@ -14,10 +14,12 @@ The RTDIP SDK includes several Azure AD authentication methods to cater to the p
 
 ## Authentication
 
-The following section describes authentication using [Azure Active Directory.](../code-reference/authentication/azure.md).
+<!-- --8<-- [start:azuread] -->
+
+The following section describes authentication using [Azure Active Directory.](https://www.rtdip.io/sdk/code-reference/authentication/azure/).
 
 !!! note "Note"
-        </b>If you are using the SDK directly in Databricks please note that DefaultAuth will not work.<br />
+       If you are using the SDK directly in Databricks please note that DefaultAuth will not work.
 
 1\. Import **rtdip-sdk** authentication methods with the following:
 
@@ -43,5 +45,8 @@ Once authenticated, it is possible to retrieve tokens for specific Azure Resourc
 === "Databricks"
         access_token = DefaultAzureCredential.get_token("2ff814a6-3304-4ab8-85cb-cd0e6f879c1d/.default").token
 
+<!-- --8<-- [end:azuread] -->
+
 !!! note "Note"
-    </b>RTDIP are continuously adding more to this list so check back regularly!<br />
+    RTDIP are continuously adding more to this list so check back regularly!
+

docs/sdk/authentication/databricks.md

@@ -1,5 +1,7 @@
 # Databricks 
 
+<!-- --8<-- [start:databrickspat] -->
+
 Databricks supports authentication using Personal Access Tokens (PAT) and information about this authentication method is available [here.](https://docs.databricks.com/dev-tools/api/latest/authentication.html)
 
 ## Authentication
@@ -22,4 +24,6 @@ access_token = "dbapi......."
 connection = DatabricksSQLConnection(server_hostname, http_path, access_token)
 ```
 
-Replace **server_hostname**, **http_path** with your own information and specify your Databricks PAT token for the **access_token**. 
+Replace **server_hostname**, **http_path** with your own information and specify your Databricks PAT token for the **access_token**. 
+
+<!-- --8<-- [end:databrickspat] -->

docs/sdk/code-reference/query/functions/metadata.md

@@ -3,7 +3,7 @@
 
 ## Example
 ```python
---8<-- "https://raw.githubusercontent.com/rtdip/samples/main/queries/Metadata/metadata.py"
+--8<-- "https://raw.githubusercontent.com/rtdip/samples/main/queries/TimeSeriesQueryBuilder/Metadata/metadata.py"
 ```
 
 This example is using [```DefaultAuth()```](../../authentication/azure.md) and [```DatabricksSQLConnection()```](../connectors/db-sql-connector.md) to authenticate and connect. You can find other ways to authenticate [here](../../authentication/azure.md). The alternative built in connection methods are either by [```PYODBCSQLConnection()```](../connectors/pyodbc-sql-connector.md), [```TURBODBCSQLConnection()```](../connectors/turbodbc-sql-connector.md) or [```SparkConnection()```](../connectors/spark-connector.md).

docs/sdk/code-reference/query/functions/time_series/circular-average.md

@@ -3,7 +3,7 @@
 
 ## Example
 ```python
- --8<-- "https://raw.githubusercontent.com/rtdip/samples/main/queries/Circular-Average/circular_average.py"
+ --8<-- "https://raw.githubusercontent.com/rtdip/samples/main/queries/TimeSeriesQueryBuilder/Circular-Average/circular_average.py"
 ```
 
 This example is using [```DefaultAuth()```](../../../authentication/azure.md) and [```DatabricksSQLConnection()```](../../connectors/db-sql-connector.md) to authenticate and connect. You can find other ways to authenticate [here](../../../authentication/azure.md). The alternative built in connection methods are either by [```PYODBCSQLConnection()```](../../connectors/pyodbc-sql-connector.md), [```TURBODBCSQLConnection()```](../../connectors/turbodbc-sql-connector.md) or [```SparkConnection()```](../../connectors/spark-connector.md).

docs/sdk/code-reference/query/functions/time_series/circular-standard-deviation.md

@@ -3,7 +3,7 @@
 
 ## Example
 ```python
- --8<-- "https://raw.githubusercontent.com/rtdip/samples/main/queries/Circular-Standard-Deviation/circular_standard_deviation.py"
+ --8<-- "https://raw.githubusercontent.com/rtdip/samples/main/queries/TimeSeriesQueryBuilder/Circular-Standard-Deviation/circular_standard_deviation.py"
 ```
 
 This example is using [```DefaultAuth()```](../../../authentication/azure.md) and [```DatabricksSQLConnection()```](../../connectors/db-sql-connector.md) to authenticate and connect. You can find other ways to authenticate [here](../../../authentication/azure.md). The alternative built in connection methods are either by [```PYODBCSQLConnection()```](../../connectors/pyodbc-sql-connector.md), [```TURBODBCSQLConnection()```](../../connectors/turbodbc-sql-connector.md) or [```SparkConnection()```](../../connectors/spark-connector.md).

docs/sdk/code-reference/query/functions/time_series/interpolate.md

@@ -3,7 +3,7 @@
 
 ## Example
 ```python
---8<-- "https://raw.githubusercontent.com/rtdip/samples/main/queries/Interpolate/interpolate.py"
+--8<-- "https://raw.githubusercontent.com/rtdip/samples/main/queries/TimeSeriesQueryBuilder/Interpolate/interpolate.py"
 ```
 
 This example is using [```DefaultAuth()```](../../../authentication/azure.md) and [```DatabricksSQLConnection()```](../../connectors/db-sql-connector.md) to authenticate and connect. You can find other ways to authenticate [here](../../../authentication/azure.md). The alternative built in connection methods are either by [```PYODBCSQLConnection()```](../../connectors/pyodbc-sql-connector.md), [```TURBODBCSQLConnection()```](../../connectors/turbodbc-sql-connector.md) or [```SparkConnection()```](../../connectors/spark-connector.md).

docs/sdk/code-reference/query/functions/time_series/interpolation-at-time.md

@@ -3,7 +3,7 @@
 
 ## Example
 ```python
---8<-- "https://raw.githubusercontent.com/rtdip/samples/main/queries/Interpolation-at-Time/interpolation_at_time.py"
+--8<-- "https://raw.githubusercontent.com/rtdip/samples/main/queries/TimeSeriesQueryBuilder/Interpolation-at-Time/interpolation_at_time.py"
 ```
 
 This example is using [```DefaultAuth()```](../../../authentication/azure.md) and [```DatabricksSQLConnection()```](../../connectors/db-sql-connector.md) to authenticate and connect. You can find other ways to authenticate [here](../../../authentication/azure.md). The alternative built in connection methods are either by [```PYODBCSQLConnection()```](../../connectors/pyodbc-sql-connector.md), [```TURBODBCSQLConnection()```](../../connectors/turbodbc-sql-connector.md) or [```SparkConnection()```](../../connectors/spark-connector.md).

docs/sdk/code-reference/query/functions/time_series/latest.md

@@ -3,7 +3,7 @@
 
 ## Example
 ```python
---8<-- "https://raw.githubusercontent.com/rtdip/samples/main/queries/Latest/latest.py"
+--8<-- "https://raw.githubusercontent.com/rtdip/samples/main/queries/TimeSeriesQueryBuilder/Latest/latest.py"
 ```
 
 This example is using [```DefaultAuth()```](../../../authentication/azure.md) and [```DatabricksSQLConnection()```](../../connectors/db-sql-connector.md) to authenticate and connect. You can find other ways to authenticate [here](../../../authentication/azure.md). The alternative built in connection methods are either by [```PYODBCSQLConnection()```](../../connectors/pyodbc-sql-connector.md), [```TURBODBCSQLConnection()```](../../connectors/turbodbc-sql-connector.md) or [```SparkConnection()```](../../connectors/spark-connector.md).

docs/sdk/code-reference/query/functions/time_series/plot.md

@@ -3,7 +3,7 @@
 
 ## Example
 ```python
---8<-- "https://raw.githubusercontent.com/rtdip/samples/main/queries/Plot/plot.py"
+--8<-- "https://raw.githubusercontent.com/rtdip/samples/main/queries/TimeSeriesQueryBuilder/Plot/plot.py"
 ```
 
 This example is using [```DefaultAuth()```](../../../authentication/azure.md) and [```DatabricksSQLConnection()```](../../connectors/db-sql-connector.md) to authenticate and connect. You can find other ways to authenticate [here](../../../authentication/azure.md). The alternative built in connection methods are either by [```PYODBCSQLConnection()```](../../connectors/pyodbc-sql-connector.md), [```TURBODBCSQLConnection()```](../../connectors/turbodbc-sql-connector.md) or [```SparkConnection()```](../../connectors/spark-connector.md).

docs/sdk/code-reference/query/functions/time_series/raw.md

@@ -3,7 +3,7 @@
 
 ## Example
 ```python
---8<-- "https://raw.githubusercontent.com/rtdip/samples/main/queries/Raw/raw.py"
+--8<-- "https://raw.githubusercontent.com/rtdip/samples/main/queries/TimeSeriesQueryBuilder/Raw/raw.py"
 ```
 
 This example is using [```DefaultAuth()```](../../../authentication/azure.md) and [```DatabricksSQLConnection()```](../../connectors/db-sql-connector.md) to authenticate and connect. You can find other ways to authenticate [here](../../../authentication/azure.md). The alternative built in connection methods are either by [```PYODBCSQLConnection()```](../../connectors/pyodbc-sql-connector.md), [```TURBODBCSQLConnection()```](../../connectors/turbodbc-sql-connector.md) or [```SparkConnection()```](../../connectors/spark-connector.md).

docs/sdk/code-reference/query/functions/time_series/resample.md

@@ -3,7 +3,7 @@
 
 ## Example
 ```python
---8<-- "https://raw.githubusercontent.com/rtdip/samples/main/queries/Resample/resample.py"
+--8<-- "https://raw.githubusercontent.com/rtdip/samples/main/queries/TimeSeriesQueryBuilder/Resample/resample.py"
 ```
 
 This example is using [```DefaultAuth()```](../../../authentication/azure.md) and [```DatabricksSQLConnection()```](../../connectors/db-sql-connector.md) to authenticate and connect. You can find other ways to authenticate [here](../../../authentication/azure.md). The alternative built in connection methods are either by [```PYODBCSQLConnection()```](../../connectors/pyodbc-sql-connector.md), [```TURBODBCSQLConnection()```](../../connectors/turbodbc-sql-connector.md) or [```SparkConnection()```](../../connectors/spark-connector.md).

docs/sdk/code-reference/query/functions/time_series/summary.md

@@ -3,7 +3,7 @@
 
 ## Example
 ```python
---8<-- "https://raw.githubusercontent.com/rtdip/samples/main/queries/Summary/summary.py"
+--8<-- "https://raw.githubusercontent.com/rtdip/samples/main/queries/TimeSeriesQueryBuilder/Summary/summary.py"
 ```
 
 This example is using [```DefaultAuth()```](../../../authentication/azure.md) and [```DatabricksSQLConnection()```](../../connectors/db-sql-connector.md) to authenticate and connect. You can find other ways to authenticate [here](../../../authentication/azure.md). The alternative built in connection methods are either by [```PYODBCSQLConnection()```](../../connectors/pyodbc-sql-connector.md), [```TURBODBCSQLConnection()```](../../connectors/turbodbc-sql-connector.md) or [```SparkConnection()```](../../connectors/spark-connector.md).

docs/sdk/code-reference/query/functions/time_series/time-weighted-average.md

@@ -3,7 +3,7 @@
 
 ## Example
 ```python
---8<-- "https://raw.githubusercontent.com/rtdip/samples/main/queries/Time-Weighted-Average/time_weighted_average.py"
+--8<-- "https://raw.githubusercontent.com/rtdip/samples/main/queries/TimeSeriesQueryBuilder/Time-Weighted-Average/time_weighted_average.py"
 ```
 
 This example is using [```DefaultAuth()```](../../../authentication/azure.md) and [```DatabricksSQLConnection()```](../../connectors/db-sql-connector.md) to authenticate and connect. You can find other ways to authenticate [here](../../../authentication/azure.md). The alternative built in connection methods are either by [```PYODBCSQLConnection()```](../../connectors/pyodbc-sql-connector.md), [```TURBODBCSQLConnection()```](../../connectors/turbodbc-sql-connector.md) or [```SparkConnection()```](../../connectors/spark-connector.md).

docs/sdk/code-reference/query/functions/weather/latest.md

@@ -4,55 +4,13 @@
 ## Example get_point
 
 ```python
-from rtdip_sdk.authentication.azure import DefaultAuth
-from rtdip_sdk.queries.weather.latest import get_point
-from rtdip_sdk.connectors import DatabricksSQLConnection
-
-auth = DefaultAuth().authenticate()
-token = auth.get_token("2ff814a6-3304-4ab8-85cb-cd0e6f879c1d/.default").token
-connection = DatabricksSQLConnection("{server_hostname}", "{http_path}", token)
-
-params = {
-    "forecast": "mock_forecast",
-    "forecast_type": "mock_weather",
-    "region": "mock_region",
-    "data_security_level": "mock_security",
-    "data_type": "mock_data_type",
-    "lat": 1.1,
-    "lon": 1.1,
-}
-
-x = get_point(connection, params)
-
-print(x)
+--8<-- "https://raw.githubusercontent.com/rtdip/samples/main/queries/WeatherQueryBuilder/Latest-Point/latest_point.py"
 ```
 
 ## Example get_grid
 
 ```python
-from rtdip_sdk.authentication.azure import DefaultAuth
-from rtdip_sdk.queries.weather.latest import get_point
-from rtdip_sdk.connectors import DatabricksSQLConnection
-
-auth = DefaultAuth().authenticate()
-token = auth.get_token("2ff814a6-3304-4ab8-85cb-cd0e6f879c1d/.default").token
-connection = DatabricksSQLConnection("{server_hostname}", "{http_path}", token)
-
-params = {
-    "forecast": "mock_forecast",
-    "forecast_type": "mock_weather",
-    "region": "mock_region",
-    "data_security_level": "mock_security",
-    "data_type": "mock_data_type",
-    "min_lat": 36,
-    "max_lat": 38,
-    "min_lon": -109.1,
-    "max_lon": -107.1,
-}
-
-x = get_grid(connection, params)
-
-print(x)
+--8<-- "https://raw.githubusercontent.com/rtdip/samples/main/queries/WeatherQueryBuilder/Latest-Grid/latest_grid.py"
 ```
 
 These examples are using [```DefaultAuth()```](../../../authentication/azure.md) and [```DatabricksSQLConnection()```](../../connectors/db-sql-connector.md) to authenticate and connect. You can find other ways to authenticate [here](../../../authentication/azure.md). The alternative built in connection methods are either by [```PYODBCSQLConnection()```](../../connectors/pyodbc-sql-connector.md), [```TURBODBCSQLConnection()```](../../connectors/turbodbc-sql-connector.md) or [```SparkConnection()```](../../connectors/spark-connector.md).

docs/sdk/code-reference/query/functions/weather/raw.md

@@ -4,67 +4,13 @@
 ## Example get_point
 
 ```python
-from rtdip_sdk.authentication.azure import DefaultAuth
-from rtdip_sdk.queries.weather.raw import get_point
-from rtdip_sdk.connectors import DatabricksSQLConnection
-
-auth = DefaultAuth().authenticate()
-token = auth.get_token("2ff814a6-3304-4ab8-85cb-cd0e6f879c1d/.default").token
-connection = DatabricksSQLConnection("{server_hostname}", "{http_path}", token)
-
-params = {
-    "forecast": "mock_forecast",
-    "forecast_type": "mock_weather",
-    "region": "mock_region",
-    "data_security_level": "mock_security",
-    "data_type": "mock_data_type",
-    "lat": 1.1,
-    "lon": 1.1,
-    "start_date": "2020-01-01",
-    "end_date": "2020-01-02",
-    "forecast_run_start_date": "2020-01-01",
-    "forecast_run_end_date": "2020-01-02",
-    "timestamp_column": "EventTime",
-    "forecast_run_timestamp_column": "EnqueuedTime",
-}
-
-x = get_point(connection, params)
-
-print(x)
+--8<-- "https://raw.githubusercontent.com/rtdip/samples/main/queries/WeatherQueryBuilder/Raw-Point/raw_point.py"
 ```
 
 ## Example get_grid
 
 ```python
-from rtdip_sdk.authentication.azure import DefaultAuth
-from rtdip_sdk.queries.weather.raw import get_grid
-from rtdip_sdk.connectors import DatabricksSQLConnection
-
-auth = DefaultAuth().authenticate()
-token = auth.get_token("2ff814a6-3304-4ab8-85cb-cd0e6f879c1d/.default").token
-connection = DatabricksSQLConnection("{server_hostname}", "{http_path}", token)
-
-params = {
-    "forecast": "mock_forecast",
-    "forecast_type": "mock_weather",
-    "region": "mock_region",
-    "data_security_level": "mock_security",
-    "data_type": "mock_data_type",
-    "min_lat": 36,
-    "max_lat": 38,
-    "min_lon": -109.1,
-    "max_lon": -107.1,
-    "start_date": "2020-01-01",
-    "end_date": "2020-01-02",
-    "forecast_run_start_date": "2020-01-01",
-    "forecast_run_end_date": "2020-01-02",
-    "timestamp_column": "EventTime",
-    "forecast_run_timestamp_column": "EnqueuedTime",
-}
-
-x = get_grid(connection, params)
-
-print(x)
+--8<-- "https://raw.githubusercontent.com/rtdip/samples/main/queries/WeatherQueryBuilder/Raw-Grid/raw_grid.py"
 ```
 
 These examples are using [```DefaultAuth()```](../../../authentication/azure.md) and [```DatabricksSQLConnection()```](../../connectors/db-sql-connector.md) to authenticate and connect. You can find other ways to authenticate [here](../../../authentication/azure.md). The alternative built in connection methods are either by [```PYODBCSQLConnection()```](../../connectors/pyodbc-sql-connector.md), [```TURBODBCSQLConnection()```](../../connectors/turbodbc-sql-connector.md) or [```SparkConnection()```](../../connectors/spark-connector.md).

docs/sdk/examples/query/Circular-Average.md

@@ -1 +1 @@
---8<-- "https://raw.githubusercontent.com/rtdip/samples/main/queries/Circular-Average/README.md"
+--8<-- "https://raw.githubusercontent.com/rtdip/samples/main/queries/TimeSeriesQueryBuilder/Circular-Average/README.md"

docs/sdk/examples/query/Circular-Standard-Deviation.md

@@ -1 +1 @@
---8<-- "https://raw.githubusercontent.com/rtdip/samples/main/queries/Circular-Standard-Deviation/README.md"
+--8<-- "https://raw.githubusercontent.com/rtdip/samples/main/queries/TimeSeriesQueryBuilder/Circular-Standard-Deviation/README.md"