Store session data in the pipeline lifecycle

[raproth]

I'd like to write some metadata of our pipeline to a database (e.g. start and end timestamps and pipeline execution duration). To do so, I created a before_pipeline_run and after_pipeline_run hooks.

I'm struggeling to exchange data between the two hooks. Is there a way to store the start-timestamp in a session state (in before_pipeline_run) and read this value in after_pipeline_run in order to calculate the duration without persisting this value to a file?

It would also be nice to gather some intermediate informations (in the node logic) in a session-object

[datajoely]

I think hooks are not aware of each other by design (Kedro uses pluggy by the pytest folks behind the scenes)

I think persisting to a file (possibly SQLite so you get some concurrency protection) may be your best bet, but I'm interested hear what other people say

[raproth]

Implemented my own session-state (singleton). Gathering session start timestamp in a hook

[datajoely]

Would love if you posted your hook here for others to discover!