How to use dataframes from asset to asset?

[JamarWhitfield]

If I create an asset that creates a data frame and then performs some processes on the data frame how would I go about using the processed data frame in another asset?

[garethbrickman]

You can return the dataframe in the upstream asset and use it as an input in a dependent downstream asset via the @asset()'s ins parameter (docs). Working example using a Pandas DataFrame:

import pandas as pd
from dagster import asset, Definitions, AssetIn, AssetKey

@asset
def create_dataframe(context):
    data = {
        'A': [1, 2, 3, 4],
        'B': [5, 6, 7, 8]
    }
    df = pd.DataFrame(data)
    context.log.info(f"Raw dataframe: {df}")
    return df

@asset(ins={"df": AssetIn(key=AssetKey("create_dataframe"))})
def process_dataframe(context, df: pd.DataFrame):
    # Perform some operations on the DataFrame
    df['C'] = df['A'] + df['B']
    context.log.info(f"Processed dataframe: {df}")
    return df

defs = Definitions(
    assets=[create_dataframe, process_dataframe]
)

[JamarWhitfield]

So how would it differ if create_dataframe created multiple (lets say 3) dataframes and process_dataframe took in those dataframes? Also thank you.

[garethbrickman]

In this example, we'll return multiple assets in the create_dataframe function and they'll be used as inputs for the process_dataframe asset:

import pandas as pd
from dagster import asset, multi_asset, Definitions, AssetIn, AssetKey, AssetOut

@multi_asset(
    outs={
        "df1": AssetOut(),
        "df2": AssetOut()
    }
)
def create_dataframe(context):
    data1 = {
        'A': [1, 2, 3, 4],
        'B': [5, 6, 7, 8]
    }
    data2 = {
        'X': [9, 10, 11, 12],
        'Y': [13, 14, 15, 16]
    }
    df1 = pd.DataFrame(data1)
    df2 = pd.DataFrame(data2)
    context.log.info(f"Raw dataframe 1: {df1}")
    context.log.info(f"Raw dataframe 2: {df2}")
    return df1, df2  # Must return a tuple if not yielding each output

@asset(
    ins={
        "df1": AssetIn(key=AssetKey("df1")),
        "df2": AssetIn(key=AssetKey("df2"))
    }
)
def process_dataframe(context, df1: pd.DataFrame, df2: pd.DataFrame):
    # Perform some operations on the DataFrames
    df1['C'] = df1['A'] + df1['B']
    df2['Z'] = df2['X'] + df2['Y']
    context.log.info(f"Processed dataframe 1: {df1}")
    context.log.info(f"Processed dataframe 2: {df2}")
    return df1, df2

defs = Definitions(
    assets=[create_dataframe, process_dataframe]
)

[JamarWhitfield]

Thank you!