[module-loaders] [rfc] Include specs in asset module loaders

[dpeng817]

Summary & Motivation

This PR is the tablestakes approach for adding spec support to our asset module loaders. It enables asset spec loading in the code path, and gates it behind a bool so as to not change behavior for any existing users.

One intricacy here is that the loading the following module will error:

import dagster as dg

spec = dg.AssetSpec("my_asset")

@dg.multi_asset(specs=[spec])
def uses_spec():
    ...

At this point in time, I think this is fine. It will just basically push users towards a pattern like this:

import dagster as dg

def specs():
 return [dg.AssetSpec("my_asset")]

@dg.multi_asset(specs=specs())
def uses_spec():
    ...

How I tested this

Added additional testing for spec loading case, ensured existing test behavior did not change.

Changelog

Our asset loading functions can now load AssetSpec objects when called with the include_specs parameter. This includes load_assets_from_package_name, load_assets_from_modules, load_assets_from_modules, and load_assets_from_current_module.

[dpeng817]

• [module-loaders] [rfc] Include specs in asset module loaders #26524 👈 (View in Graphite)
• [module-loaders] Use new loader code in checks loaders #26514
• [module-loaders] Delete extra_source_assets param from assets_from_modules #26494 : 9 other dependent PRs (#26484 , #26498 , #26527 and 6 others)
• master

This stack of pull requests is managed by Graphite. Learn more about stacking.

[yuhan]

open to ideas - i wonder which option is better:

1. this PR - add an additional param include_specs to existing APIs
2. or brand new load_asset_specs_from_x

hmmm 🤔 actually as im typing out this, i think this PR does feel cleaner and less of a cognitive load to remembering both load_asset_ and load_asset_specs.

[yuhan]

OK I think this is a good step forward and it's not too risky or irreversible.

Could you update the relevant docstrings to communicate this addition in a non-confusing way? (btw the current docstring is pretty busted 😓 https://docs-preview.dagster.io/api/python-api/assets#code-locations)

[dpeng817]

Yea I think that ideally we don't introduce the cross product of these APIs - make the existing ones work sensibly, and then eventually introduce a centralized defs loader

[OwenKephart]

nice

[dpeng817]

Closing because all the commits from this PR are now contained within #26494 - ideally this would be merged, but gt fold failed to resolve things properly for me, and now this PR is in an unresolveable state.