Ingestion of multiple sling assets in parallel

[cody-scott]

Hi!

Looking for some feedback on the approach of loading multiple sling assets in parallel. Right now we are targeting the entire set of assets for a given database and ingesting that in one job.

What we see on the backend is that only one table is running at a time when loading.

One thought we had was to split the larger tables into their own jobs to run in parallel, but is there a cleaner approach in dagster to accomplish this right off a sling_asset?

The tables, unfortunately, cannot be incremental in this case.

[cmpadden]

Hi @cody-scott ,

There is a limitation with the way the Sling integration has been implemented, in that one step and command runs for all of the assets in the @sling_assets definition.

So to accomplish what you need, you will have to separate them into their own configurations. You may find how we are using Sling internally in dagster-open-platform a helpful reference:

• https://github.com/dagster-io/dagster-open-platform/blob/main/dagster_open_platform/sling/assets.py#L48-L77

[cody-scott]

Thanks for the quick reply Colton.

If I am understanding correctly, you have split the low/high volume feeds into their own configs with their own sling_assets for each, which is then triggered by using the automaterialization policies instead of manually defining jobs?

Low volume (cloud_product_main_low_volume) running every 2 hours, high volume (cloud_product_main_high_volume) every 5 minutes?

[cmpadden]

That's exactly right.

And we used auto-materialization policies in this case, but you could just as well use jobs and schedules.

[cody-scott]

Perfect! Thanks!

Probably a bit off topic here, but is there a generally recommended approach to auto vs not? I'm seeing your DBT project for example is every 3 hours, why not simply have the assets trigger downstream from your sling load?

[nicklausroach]

Hey Cody! Dagster's internal data engineer here, great to meet you. That's actually exactly where we're headed! We've been testing out all the different forms of automation within our internal data platform, but I think where we'll land (likely in the coming weeks) is that most of our assets will be on automations instead of jobs. That said, we'll need to be careful that we're not over-executing and needlessly running up our Snowflake spend (which is one of the reasons for the less-frequent cadence in our dbt job).

But to answer your original question about auto vs jobs, I find that if I have a large group of assets that I want to have a lot of visibility and control over, it's easier to put those in a job right now since those jobs can be monitored as a whole more easily than many assets individually right now, just my take! I think over time though automations will be the right call for most situations as visibility and monitoring improve. Hope this helps!