You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Hi all! Starting using dagster for a data acquisition pipeline, to process products data, do some transformations, normalization, etc.
This pipeline will be shared between all the brands we get data from. Right now, We're going for a graph asset containing assets and ops, and every brand is going to have it's own config to control the flow inside the job, also it's own schedule.
Is there a way to use class-based assets, ops, graphs so I can make every brand have it's own graph inheriting assets and ops from a base class? I know that for Config and Resources are class-based. But for assets, ops, graphs, all the examples in the docs use the decorators, so I don't know if dagster should be used like that, since I didn't find examples for that. I'm asking before it's to late to change, without rewriting a lot of code, if it's possible, the team would prefer to go this way. Also any thoughts on the approach I choose to go, is welcome.
area: pipeline-designIssues or discussions related to how to model a pipeline in Dagster
1 participant
Heading
Bold
Italic
Quote
Code
Link
Numbered list
Unordered list
Task list
Attach files
Mention
Reference
Menu
reacted with thumbs up emoji reacted with thumbs down emoji reacted with laugh emoji reacted with hooray emoji reacted with confused emoji reacted with heart emoji reacted with rocket emoji reacted with eyes emoji
-
Hi all! Starting using dagster for a data acquisition pipeline, to process products data, do some transformations, normalization, etc.
This pipeline will be shared between all the brands we get data from. Right now, We're going for a graph asset containing assets and ops, and every brand is going to have it's own config to control the flow inside the job, also it's own schedule.
Is there a way to use class-based assets, ops, graphs so I can make every brand have it's own graph inheriting assets and ops from a base class? I know that for Config and Resources are class-based. But for assets, ops, graphs, all the examples in the docs use the decorators, so I don't know if dagster should be used like that, since I didn't find examples for that. I'm asking before it's to late to change, without rewriting a lot of code, if it's possible, the team would prefer to go this way. Also any thoughts on the approach I choose to go, is welcome.
thank you!
Beta Was this translation helpful? Give feedback.
All reactions