Replies: 1 comment
-
Op Jobs are appropriate for ad hoc tasks such as this. And they can be triggered to run automatically via Schedules or Sensors! |
Beta Was this translation helpful? Give feedback.
0 replies
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
-
So I have assets for a pipeline which downloads data from a url using a daily partition, stores the data in what should be a temporary staging table in DuckDB, aggregates the data, writes reports, etc. My question is that at some point in this pipeline I would like to clean up the staging tables (eg drop them). My first thought was "Oh I'll just make an asset to drop tables". But my second thought was, wait, that doesn't seem very "dagsterish" because that's an action or task, not a "noun" so to speka. So as the title of the discussion suggests, I'm wondering what is the best or at least most agreed upon way to do clean up like this?
Beta Was this translation helpful? Give feedback.
All reactions