Partial pipeline example #152

ajeklund · 2023-07-11T09:23:54Z

Description:

Example script, showing how to use partial pipelines. Converted from original Jupyter notebook demo version.

Closes issue #151.

Type of change:

Bug fix.
New feature.
Documentation update.

Checklist for the reviewer:

This checklist should be used as a help for the reviewer.

Is the change limited to one issue?
Does this PR close the issue?
Is the code easy to read and understand, including clearly named variables?
Do all new feature have an accompanying new test?
Has the documentation been updated as necessary?

ajeklund · 2023-07-12T15:14:34Z

Printout of the 'siemens' error from the oteapi-services container, likely caused by the string 'S-3400' being passed to a pint.Quantity in

dlite/mappings.py", line 720, in instance_routes:

oteapi_1  | [2023-07-12 14:59:00 +0000] [49] [ERROR] Error in ASGI Framework
oteapi_1  | Traceback (most recent call last):
oteapi_1  |   File "/usr/local/lib/python3.9/site-packages/hypercorn/asyncio/task_group.py", line 23, in _handle
oteapi_1  |     await app(scope, receive, send, sync_spawn, call_soon)
oteapi_1  |   File "/usr/local/lib/python3.9/site-packages/hypercorn/app_wrappers.py", line 33, in __call__
oteapi_1  |     await self.app(scope, receive, send)
oteapi_1  |   File "/usr/local/lib/python3.9/site-packages/fastapi/applications.py", line 282, in __call__
oteapi_1  |     await super().__call__(scope, receive, send)
oteapi_1  |   File "/usr/local/lib/python3.9/site-packages/starlette/applications.py", line 122, in __call__
oteapi_1  |     await self.middleware_stack(scope, receive, send)
oteapi_1  |   File "/usr/local/lib/python3.9/site-packages/starlette/middleware/errors.py", line 184, in __call__
oteapi_1  |     raise exc
oteapi_1  |   File "/usr/local/lib/python3.9/site-packages/starlette/middleware/errors.py", line 162, in __call__
oteapi_1  |     await self.app(scope, receive, _send)
oteapi_1  |   File "/usr/local/lib/python3.9/site-packages/starlette/middleware/exceptions.py", line 79, in __call__
oteapi_1  |     raise exc
oteapi_1  |   File "/usr/local/lib/python3.9/site-packages/starlette/middleware/exceptions.py", line 68, in __call__
oteapi_1  |     await self.app(scope, receive, sender)
oteapi_1  |   File "/usr/local/lib/python3.9/site-packages/fastapi/middleware/asyncexitstack.py", line 20, in __call__
oteapi_1  |     raise e
oteapi_1  |   File "/usr/local/lib/python3.9/site-packages/fastapi/middleware/asyncexitstack.py", line 17, in __call__
oteapi_1  |     await self.app(scope, receive, send)
oteapi_1  |   File "/usr/local/lib/python3.9/site-packages/starlette/routing.py", line 718, in __call__
oteapi_1  |     await route.handle(scope, receive, send)
oteapi_1  |   File "/usr/local/lib/python3.9/site-packages/starlette/routing.py", line 276, in handle
oteapi_1  |     await self.app(scope, receive, send)
oteapi_1  |   File "/usr/local/lib/python3.9/site-packages/starlette/routing.py", line 66, in app
oteapi_1  |     response = await func(request)
oteapi_1  |   File "/usr/local/lib/python3.9/site-packages/fastapi/routing.py", line 241, in app
oteapi_1  |     raw_response = await run_endpoint_function(
oteapi_1  |   File "/usr/local/lib/python3.9/site-packages/fastapi/routing.py", line 167, in run_endpoint_function
oteapi_1  |     return await dependant.call(**values)
oteapi_1  |   File "/app/app/routers/function.py", line 83, in get_function
oteapi_1  |     session_update = function_strategy.get(session=session_data)
oteapi_1  |   File "/oteapi-dlite/oteapi_dlite/strategies/interoperability.py", line 121, in get
oteapi_1  |     print(list(instances))
oteapi_1  |   File "/usr/local/lib/python3.9/site-packages/dlite/dlite.py", line 2194, in get_instances
oteapi_1  |     for inst in instantiate_all(
oteapi_1  |   File "/usr/local/lib/python3.9/site-packages/dlite/mappings.py", line 878, in instantiate_all
oteapi_1  |     routes = instance_routes(
oteapi_1  |   File "/usr/local/lib/python3.9/site-packages/dlite/mappings.py", line 720, in instance_routes
oteapi_1  |     sources[f'{inst.meta.uri}#{key}'] = quantity(value, props[key].unit)
oteapi_1  |   File "/usr/local/lib/python3.9/site-packages/pint/facets/plain/quantity.py", line 200, in __new__
oteapi_1  |     inst = ureg.parse_expression(value)
oteapi_1  |   File "/usr/local/lib/python3.9/site-packages/pint/facets/plain/registry.py", line 1309, in parse_expression
oteapi_1  |     return build_eval_tree(gen).evaluate(_define_op)
oteapi_1  |   File "/usr/local/lib/python3.9/site-packages/pint/pint_eval.py", line 146, in evaluate
oteapi_1  |     return bin_op[op_text](
oteapi_1  |   File "/usr/local/lib/python3.9/site-packages/pint/facets/plain/quantity.py", line 840, in __sub__
oteapi_1  |     return self._add_sub(other, operator.sub)
oteapi_1  |   File "/usr/local/lib/python3.9/site-packages/pint/facets/plain/quantity.py", line 100, in wrapped
oteapi_1  |     return f(self, *args, **kwargs)
oteapi_1  |   File "/usr/local/lib/python3.9/site-packages/pint/facets/plain/quantity.py", line 727, in _add_sub
oteapi_1  |     raise DimensionalityError(self._units, "dimensionless")
oteapi_1  | pint.errors.DimensionalityError: Cannot convert from 'siemens' to 'dimensionless'

ajeklund · 2023-07-12T15:56:54Z

The 'siemens' error gives an ApiError in the Jupyter notebook:

---------------------------------------------------------------------------
ApiError                                  Traceback (most recent call last)
Cell In[6], line 6
      3 pipeline = cansMprDataParser>>cansMprDataMappings>>generatorMappings>>generatorFilter>>createCollection
      5 # Execute the pipeline
----> 6 pipeline.get()

File ~/.venv/ote/lib/python3.10/site-packages/otelib/backends/strategies.py:94, in AbstractBaseStrategy.get(self, session_id)
     92 self.initialize(session_id)
     93 if self.input_pipe:
---> 94     self.input_pipe.get(session_id)
     95 return self.fetch(session_id)

File ~/.venv/ote/lib/python3.10/site-packages/otelib/pipe.py:18, in Pipe.get(self, session_id)
     16 def get(self, session_id: "Optional[str]" = None) -> bytes:
     17     """Call the input strategy's `get()` method."""
---> 18     return self.input.get(session_id)

File ~/.venv/ote/lib/python3.10/site-packages/otelib/backends/strategies.py:95, in AbstractBaseStrategy.get(self, session_id)
     93 if self.input_pipe:
     94     self.input_pipe.get(session_id)
---> 95 return self.fetch(session_id)

File ~/.venv/ote/lib/python3.10/site-packages/otelib/backends/services/base.py:89, in BaseServicesStrategy.fetch(self, session_id)
     83     return response.content
     84 strategy_name = (
     85     self.strategy_name[len("data") :]
     86     if self.strategy_name.startswith("data")
     87     else self.strategy_name
     88 )
---> 89 raise ApiError(
     90     f"Cannot fetch {self.strategy_name}: session_id={session_id!r} "
     91     f"{strategy_name}_id={self.strategy_id!r}"
     92     f"{' content=' + str(response.content) if self.debug else ''}",
     93     status=response.status_code,
     94 )

ApiError: ApiError: status=500 Cannot fetch function: session_id='session-c9a38462-a1b4-4ef3-b8d6-7688566008fe' function_id='function-a7402487-1f30-4e87-b4b0-109a668da7b7'

ajeklund · 2023-07-13T14:20:45Z

This PR currently has its own oteapi-dlite sub-directory with code changes that probably should be added to the repository. These code changes are handled separately in #156. This PR (i.e., the example) is thus blocked by #156.

Remove cached files from folders. Add to .gitignore. Make the partial-pipelines demo an installable package based around "oteutils". Move ote strategies under oteutils as demo-specific strategies.

hardcoding

codecov · 2024-02-26T13:03:41Z

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 89.80%. Comparing base (52533ad) to head (3c4e0f2).
Report is 90 commits behind head on master.

Additional details and impacted files

@@             Coverage Diff             @@
##           master     #152       +/-   ##
===========================================
+ Coverage   68.73%   89.80%   +21.06%     
===========================================
  Files          15       15               
  Lines         403      451       +48     
===========================================
+ Hits          277      405      +128     
+ Misses        126       46       -80

Flag	Coverage Δ
linux	`89.80% <ø> (+21.06%)`	⬆️
windows	`89.80% <ø> (+21.30%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

ajeklund · 2024-02-26T14:48:22Z

The 'siemens' error arises because the data value is S-3400 and when a pint.Quantity is created with this as input, pint (by design) interprets this value as the mathematical expression '1 siemens - 3400' which results in a DimensionalityError since the two terms indeed have different dimensionalities.

Code for reproducing the behavior:
from pint import Quantity
q = Quantity('S-3400')
or q = Quantity('S-3400', None)

But this "works": q = Quantity('S-3400S') (it results in <Quantity(-3399, 'siemens')>)

Initial Jupyter notebook version.

e5b61a3

ajeklund linked an issue Jul 11, 2023 that may be closed by this pull request

Add partial pipeline example #151

Open

ajeklund marked this pull request as draft July 11, 2023 09:46

ajeklund added 5 commits July 11, 2023 13:59

Standard pip install for otelib (bugfix released)

6254ba7

Update README and requirements.txt.

a88be54

Update requirements-txt.

9ba4a0d

Data files for local access.

d441eee

matchmaker.ipynb runs + recreates 'siemens' error.

6b7a85d

Created script version of the matchmaker notebook.

054c107

jesper-friis and others added 4 commits July 13, 2023 18:54

Simple formatting cleanup

1cbc79d

Simple formatting cleanup

b2cc13f

Clean up of example

4612dd7

Remove cached files from folders. Add to .gitignore. Make the partial-pipelines demo an installable package based around "oteutils". Move ote strategies under oteutils as demo-specific strategies.

Mention the use of the mapping ttl file instead of

3c4e0f2

hardcoding

CasperWA closed this Feb 26, 2024

CasperWA deleted the 151-partial-pipeline-example branch February 26, 2024 14:19

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Partial pipeline example #152

Partial pipeline example #152

ajeklund commented Jul 11, 2023 •

edited

Loading

ajeklund commented Jul 12, 2023 •

edited

Loading

ajeklund commented Jul 12, 2023

ajeklund commented Jul 13, 2023

codecov bot commented Feb 26, 2024

ajeklund commented Feb 26, 2024

Partial pipeline example #152

Partial pipeline example #152

Conversation

ajeklund commented Jul 11, 2023 • edited Loading

Description:

Type of change:

Checklist for the reviewer:

ajeklund commented Jul 12, 2023 • edited Loading

ajeklund commented Jul 12, 2023

ajeklund commented Jul 13, 2023

codecov bot commented Feb 26, 2024

Codecov Report

ajeklund commented Feb 26, 2024

ajeklund commented Jul 11, 2023 •

edited

Loading

ajeklund commented Jul 12, 2023 •

edited

Loading