Passing credentials as an Asset in Dagster which orchestrates dlt pipeline

[bruk0]

Hi, community. I have a question about passing credentials to dlt pipeline dynamically in Dagster.

I want to create an Asset which will send a OAuth2 creds to dlt source and GCP service account creds to dlt destination. How to implement that? And how to not use the secrets.toml file in dlt?

Thanks in advance for your time and effort.

[garethbrickman]

Check out this guide for using environment variables and secrets. I recommend setting up a Dagster resource for handling that.

[bruk0]

@garethbrickman thanks for your response and sorry for a late response from my end.
I have a question and I'd like to know if it's possible.
I have a dummy Dagster asset with this code:

@asset(
    group_name='Source_Secret'
)
def source_secret(context: AssetExecutionContext):
    json_cred_source = {
       //JSON DICT WITH CREDS
    }
    json_string = json.dumps(json_cred_source)
    json_dict = json.loads(json_string)
    context.log.info(json_string)
    context.log.info(json_dict['client_id'])
    return json_dict`

I want to access Creds values in my @dlt_assets method.

I use this approach now, but it doesn't work in my case:

//Code of @dlt_assets method
upstream_value = context.output_for_asset_key(AssetKey("source_secret"))

The output of upstream_value is 'result' and that's it.

[garethbrickman]

I don't think output_for_asset_key is intended to provide the return value of an asset- its doc string says: "Return the output name for the corresponding asset key."

Why would these secrets need to be returned from an upstream asset? In the dagster-dlt docs there is a section for how the integration manages secrets.

[bruk0]

@garethbrickman because I retrieve GCP creds from api where we store them and I need setup those creds for different clients which have different GCP creds.