How can I serialize all of my asset definitions?

[dagsir[bot]]

Has anyone attempted to serialize all asset definitions to a file upon running dagster? I've attempted pickle/cloudpickle/dill, and also some functions in dagster._serdes , but i've having issues pickling dagster objects.

---

The question was originally asked in Dagster Slack.

[alangenfeld]

As discovered, the Dagster definitions objects are not serializable. "External" representations of the definitions are used to communicate information about definitions across processes. Those are currently serialized with dagster._serdes .

[fridiculous]

Thanks @alangenfeld

With large DAGs and assets, the startup time may take minutes. As a result, I'm interested to know how to serialize Dagster AssetDefinitions and assets in general.

As a workaround, it would be really nice to be able to compile the Definitions once and then persist it during Docker build. Then the definitions could be loaded directly from a file, if that file exists. This would also decrease bootstrapping time on the server.

As a feature request - could it be possible to make Dagster Definitions or AssetDefinitions externally serializable to a file?

I attempted to serialize and deserialize AssetDefinitions using dill. To make the pickling and unpickling work, I had to add custom reduce methods to internal dagster objects.

class DailyPartitionsDefinition(TimeWindowPartitionsDefinition):
    
    def __reduce__(self):  # must match __init__ signature
        return DailyPartitionsDefinition, (
            self.start,
            self.timezone,
            self.end,
            self.fmt,
            self.end_offset,
            self.minute_offset,
            self.hour_offset
        )

And similarly...

class ConfigType:
    """The class backing DagsterTypes as they are used processing configuration data."""

    def __reduce__(self):
        self.get_snapshot()
        return ConfigType, (self.key, self.kind, self.given_name, self._description, self.type_params, self._snap)
        
    def __init__(
        self,
        key: str,
        kind: ConfigTypeKind,
        given_name: Optional[str] = None,
        description: Optional[str] = None,
        type_params: Optional[Sequence["ConfigType"]] = None,
        _snap: Optional["ConfigTypeSnap"] = None,
    ):
        self.key: str = check.str_param(key, "key")
        self.kind: ConfigTypeKind = check.inst_param(kind, "kind", ConfigTypeKind)
        self.given_name: Optional[str] = check.opt_str_param(given_name, "given_name")
        self._description: Optional[str] = check.opt_str_param(description, "description")
        self.type_params: Optional[Sequence[ConfigType]] = (
            check.sequence_param(type_params, "type_params", of_type=ConfigType)
            if type_params
            else None
        )

        # memoized snap representation
        self._snap: Optional["ConfigTypeSnap"] = _snap

[alangenfeld]

With large DAGs and assets, the startup time may take minutes

Can you share more details about how you are building your Definitions? Do you have any profiling information you can share? We do not expect it to be the case that instantiating the definitions objects themselves would take minutes even for a large number of them.

Some patterns I have observed from working through this type of problems with users before have been:

• They generated their definitions programmatically, sometimes based on information fetched from external services. Caching an intermediate representation that is used to construct the final definitions objects worked well for this case.
• The bodies of assets/ops required libraries that were very expensive to import. Moving these imports from module scope to the body of the asset/op can prevent incurring this cost for producing the definition objects.

Profiling will reveal the specific code that is taking time. I recommend https://github.com/benfred/py-spy ie
py-spy record -f speedscope -- python <def entry point>
and for import time specifically tuna and the built-in import profiling ie
python -X importtime <def entry point> &> /tmp/import.txt && tuna /tmp/import.txt

Its certainly possible you are hitting some slow code we are unaware of and need to optimize, which profiler results would help identify.