Running SQL queries concurrently

[yograjpshinde]

Hi,
I am new to Dagster and just started reading about it. I have been using Apache NIFI for few years. In NIFI i have a flow which execute 10 SQL in parallel against SAP Hana and load the output into Snowflake table. I have total 120 queries for 120 fiscal quarters. I run 10 queries in parallel and rest of the queries wait in queue. When any of the executor finishes it picks up next query from queue and output keep copying into a Snowflake table. I need a help to implement the same in Dagster.

[matthias-Q]

If I understand you correctly, the query is more or less the same, but the where clause changes (different fiscal year). If so, I would write a partitioned asset and use the partition key in your query as a parameter. Set the allowed maximum of concurrent queries per asset to 10 and force a full refresh. Then dagster should start 10 queries until all 120 partitions are done.

[yograjpshinde]

Thank you for your guidance on this. Yes that is right i have exact same SELECT query with only change in where condition for individual quarter. I will give it a try to implement my solution using a partitioned asset

[yograjpshinde]

I tried but no luck in making it work. Could you please provide a sample code so that i can build on it ?
Thank you!