Question about CSVs as assets

[dagsir[bot]]

Hello, I have a best practice question. If I have CSVs in an S3 bucket, should authenticating to S3 and fetching those CSVs be @op functions and each row from the many CSV files be an @asset?

---

The question was originally asked in Dagster Slack.

[jamiedemaria]

The "best practice" way to do this depends a bit on how the CSVs get into S3.

• You can probably use the S3 Resource to authenticate and connect with S3
• If the CSVs are put in the bucket by an external service and you do not intend to move the logic to put the CSVs in S3 to dagster, then the CSVs are probably modeled best using ExternalAssets . You'd have one ExternalAsset per CSV file
• If you want to be able to re-generate a particular CSV file using dagster (for example, if you want to be able to use dagster to fetch new data from an API and replace the existing CSV file with the new data). Then you would want to have an @asset per CSV file. The function decorated by @asset would fetch the new data and replace the CSV file in S3

[kasteph]

Thanks @jamiedemaria!

• CSVs are an upstream resource not managed by me, so ExternalAssets looks good. It seems that I can use a Sensor with it as well, which is perfect for my use case (CSVs updated at irregular intervals).