Is there a limit to the number of assets you can have? The number of dynamic partitions?

[dagsir[bot]]

How many assets are too many assets? Currently I use dynamic partitions but wonder if it would make more sense to just add an additional item to the asset key prefix. Anyone have experienced that? With dynamic partitions it seems it wouldn’t take a code change but if it was an asset, I would need to either load some factory built assets from some file/configuration somewhere I assume

---

The question was originally asked in Dagster Slack.

[jamiedemaria]

Adding an item to the key prefix is probably a bad idea. It could interfere with how the assets are stored/retrieved by the I/O manager and could have impacts on how dependencies are determined if you have your assets in different groups.

We've had users get up to around 10,000 dynamic partitions before seeing some slowdowns. I don't have exact numbers for how many assets you can support before seeing slowdowns, but i think there are users out there with thousands of assets, (maybe even tens of thousands?).

Some of the performance will depend a bit on how many resources you give the dagster deployment, but as long as you're not running on a super small compute you should be fine i think