Partial pipeline example

[ajeklund]

Description:

Example script, showing how to use partial pipelines. Converted from original Jupyter notebook demo version.

Closes issue #151.

Type of change:

• Bug fix.
• New feature.
• Documentation update.

Checklist for the reviewer:

This checklist should be used as a help for the reviewer.

• Is the change limited to one issue?
• Does this PR close the issue?
• Is the code easy to read and understand, including clearly named variables?
• Do all new feature have an accompanying new test?
• Has the documentation been updated as necessary?

[ajeklund]

Printout of the 'siemens' error from the oteapi-services container, likely caused by the string 'S-3400' being passed to a pint.Quantity in

dlite/mappings.py", line 720, in instance_routes:

oteapi_1  | [2023-07-12 14:59:00 +0000] [49] [ERROR] Error in ASGI Framework
oteapi_1  | Traceback (most recent call last):
oteapi_1  |   File "/usr/local/lib/python3.9/site-packages/hypercorn/asyncio/task_group.py", line 23, in _handle
oteapi_1  |     await app(scope, receive, send, sync_spawn, call_soon)
oteapi_1  |   File "/usr/local/lib/python3.9/site-packages/hypercorn/app_wrappers.py", line 33, in __call__
oteapi_1  |     await self.app(scope, receive, send)
oteapi_1  |   File "/usr/local/lib/python3.9/site-packages/fastapi/applications.py", line 282, in __call__
oteapi_1  |     await super().__call__(scope, receive, send)
oteapi_1  |   File "/usr/local/lib/python3.9/site-packages/starlette/applications.py", line 122, in __call__
oteapi_1  |     await self.middleware_stack(scope, receive, send)
oteapi_1  |   File "/usr/local/lib/python3.9/site-packages/starlette/middleware/errors.py", line 184, in __call__
oteapi_1  |     raise exc
oteapi_1  |   File "/usr/local/lib/python3.9/site-packages/starlette/middleware/errors.py", line 162, in __call__
oteapi_1  |     await self.app(scope, receive, _send)
oteapi_1  |   File "/usr/local/lib/python3.9/site-packages/starlette/middleware/exceptions.py", line 79, in __call__
oteapi_1  |     raise exc
oteapi_1  |   File "/usr/local/lib/python3.9/site-packages/starlette/middleware/exceptions.py", line 68, in __call__
oteapi_1  |     await self.app(scope, receive, sender)
oteapi_1  |   File "/usr/local/lib/python3.9/site-packages/fastapi/middleware/asyncexitstack.py", line 20, in __call__
oteapi_1  |     raise e
oteapi_1  |   File "/usr/local/lib/python3.9/site-packages/fastapi/middleware/asyncexitstack.py", line 17, in __call__
oteapi_1  |     await self.app(scope, receive, send)
oteapi_1  |   File "/usr/local/lib/python3.9/site-packages/starlette/routing.py", line 718, in __call__
oteapi_1  |     await route.handle(scope, receive, send)
oteapi_1  |   File "/usr/local/lib/python3.9/site-packages/starlette/routing.py", line 276, in handle
oteapi_1  |     await self.app(scope, receive, send)
oteapi_1  |   File "/usr/local/lib/python3.9/site-packages/starlette/routing.py", line 66, in app
oteapi_1  |     response = await func(request)
oteapi_1  |   File "/usr/local/lib/python3.9/site-packages/fastapi/routing.py", line 241, in app
oteapi_1  |     raw_response = await run_endpoint_function(
oteapi_1  |   File "/usr/local/lib/python3.9/site-packages/fastapi/routing.py", line 167, in run_endpoint_function
oteapi_1  |     return await dependant.call(**values)
oteapi_1  |   File "/app/app/routers/function.py", line 83, in get_function
oteapi_1  |     session_update = function_strategy.get(session=session_data)
oteapi_1  |   File "/oteapi-dlite/oteapi_dlite/strategies/interoperability.py", line 121, in get
oteapi_1  |     print(list(instances))
oteapi_1  |   File "/usr/local/lib/python3.9/site-packages/dlite/dlite.py", line 2194, in get_instances
oteapi_1  |     for inst in instantiate_all(
oteapi_1  |   File "/usr/local/lib/python3.9/site-packages/dlite/mappings.py", line 878, in instantiate_all
oteapi_1  |     routes = instance_routes(
oteapi_1  |   File "/usr/local/lib/python3.9/site-packages/dlite/mappings.py", line 720, in instance_routes
oteapi_1  |     sources[f'{inst.meta.uri}#{key}'] = quantity(value, props[key].unit)
oteapi_1  |   File "/usr/local/lib/python3.9/site-packages/pint/facets/plain/quantity.py", line 200, in __new__
oteapi_1  |     inst = ureg.parse_expression(value)
oteapi_1  |   File "/usr/local/lib/python3.9/site-packages/pint/facets/plain/registry.py", line 1309, in parse_expression
oteapi_1  |     return build_eval_tree(gen).evaluate(_define_op)
oteapi_1  |   File "/usr/local/lib/python3.9/site-packages/pint/pint_eval.py", line 146, in evaluate
oteapi_1  |     return bin_op[op_text](
oteapi_1  |   File "/usr/local/lib/python3.9/site-packages/pint/facets/plain/quantity.py", line 840, in __sub__
oteapi_1  |     return self._add_sub(other, operator.sub)
oteapi_1  |   File "/usr/local/lib/python3.9/site-packages/pint/facets/plain/quantity.py", line 100, in wrapped
oteapi_1  |     return f(self, *args, **kwargs)
oteapi_1  |   File "/usr/local/lib/python3.9/site-packages/pint/facets/plain/quantity.py", line 727, in _add_sub
oteapi_1  |     raise DimensionalityError(self._units, "dimensionless")
oteapi_1  | pint.errors.DimensionalityError: Cannot convert from 'siemens' to 'dimensionless'

[ajeklund]

The 'siemens' error gives an ApiError in the Jupyter notebook:

---------------------------------------------------------------------------
ApiError                                  Traceback (most recent call last)
Cell In[6], line 6
      3 pipeline = cansMprDataParser>>cansMprDataMappings>>generatorMappings>>generatorFilter>>createCollection
      5 # Execute the pipeline
----> 6 pipeline.get()

File ~/.venv/ote/lib/python3.10/site-packages/otelib/backends/strategies.py:94, in AbstractBaseStrategy.get(self, session_id)
     92 self.initialize(session_id)
     93 if self.input_pipe:
---> 94     self.input_pipe.get(session_id)
     95 return self.fetch(session_id)

File ~/.venv/ote/lib/python3.10/site-packages/otelib/pipe.py:18, in Pipe.get(self, session_id)
     16 def get(self, session_id: "Optional[str]" = None) -> bytes:
     17     """Call the input strategy's `get()` method."""
---> 18     return self.input.get(session_id)

File ~/.venv/ote/lib/python3.10/site-packages/otelib/backends/strategies.py:95, in AbstractBaseStrategy.get(self, session_id)
     93 if self.input_pipe:
     94     self.input_pipe.get(session_id)
---> 95 return self.fetch(session_id)

File ~/.venv/ote/lib/python3.10/site-packages/otelib/backends/services/base.py:89, in BaseServicesStrategy.fetch(self, session_id)
     83     return response.content
     84 strategy_name = (
     85     self.strategy_name[len("data") :]
     86     if self.strategy_name.startswith("data")
     87     else self.strategy_name
     88 )
---> 89 raise ApiError(
     90     f"Cannot fetch {self.strategy_name}: session_id={session_id!r} "
     91     f"{strategy_name}_id={self.strategy_id!r}"
     92     f"{' content=' + str(response.content) if self.debug else ''}",
     93     status=response.status_code,
     94 )

ApiError: ApiError: status=500 Cannot fetch function: session_id='session-c9a38462-a1b4-4ef3-b8d6-7688566008fe' function_id='function-a7402487-1f30-4e87-b4b0-109a668da7b7'

[ajeklund]

This PR currently has its own oteapi-dlite sub-directory with code changes that probably should be added to the repository. These code changes are handled separately in #156. This PR (i.e., the example) is thus blocked by #156.

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 89.80%. Comparing base (52533ad) to head (3c4e0f2).
Report is 90 commits behind head on master.

Additional details and impacted files

@@             Coverage Diff             @@
##           master     #152       +/-   ##
===========================================
+ Coverage   68.73%   89.80%   +21.06%     
===========================================
  Files          15       15               
  Lines         403      451       +48     
===========================================
+ Hits          277      405      +128     
+ Misses        126       46       -80     

Flag	Coverage Δ
linux	89.80% <ø> (+21.06%)	⬆️
windows	89.80% <ø> (+21.30%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[ajeklund]

The 'siemens' error arises because the data value is S-3400 and when a pint.Quantity is created with this as input, pint (by design) interprets this value as the mathematical expression '1 siemens - 3400' which results in a DimensionalityError since the two terms indeed have different dimensionalities.

Code for reproducing the behavior:
from pint import Quantity
q = Quantity('S-3400')
or q = Quantity('S-3400', None)

But this "works": q = Quantity('S-3400S') (it results in <Quantity(-3399, 'siemens')>)