How to test all Definitions are loadable?

[HynekBlaha]

TLDR:
I support over 20 Dagster repositories in my company (package updates, CI/CD, etc..)
Is there any easy generic test to check my update didn't break anything?

Something like dagster dev to resolve all definitions, but in pytest?

I tried to validate Definitions with this test, but it passes even for incorrect Definitions (eg. all assets are missing).

import re
import importlib

from dagster import Definitions

def test_definitions(pytestconfig):
    # Get list of Dagster modules
    workspace_yaml = (pytestconfig.rootpath / "workspace.yaml").read_text()
    dagster_modules = re.findall(r'python_module:\s+(\w+)', workspace_yaml)

    # Get all Definitions
    for dagster_module in dagster_modules:
        # Dynamically import the module
        try:
            module = importlib.import_module(dagster_module)
        except ModuleNotFoundError:
            print(f"Module {dagster_module} not found")
            continue

        try:
            defs = getattr(module, 'defs')
        except AttributeError:
            print(f"Module {dagster_module} has no defs")
            continue

        # Validate Definitions
        Definitions.validate_loadable(defs)

[garethbrickman]

This has been requested here: #23498

We're developing such a feature in this draft PR: #23692