Use thread parameter for asset

[dridk]

I'd like to be able to limit the number of materializations per asset directly from the asset definition. Indeed, I execute SQL queries, and depending on the database, I want to avoid making too many queries at the same time.

For example, inspired by snakemake :

@asset(thread=10)
def myasset():
    df = pd.read_sql("SELECT * FROM .. ")
   

This setting will prevent more than 10 instances running at the same time.

[jamiedemaria]

You can set concurrency limits for assets using tags https://docs.dagster.io/guides/limiting-concurrency-in-data-pipelines#limiting-opasset-concurrency-across-runs

on your asset:

@asset(op_tags={"dagster/concurrency_key": "sql"})
def myasset():
    df = pd.read_sql("SELECT * FROM .. ")

And then run the CLI command

dagster instance concurrency set sql 10

And then dagster will never have more than 10 assets with the tag sql running at once