feat(cli.project): enable optional minimalist configuration in templates #25245

dbrtly · 2024-10-12T02:08:58Z

PEP 621 (Nov 2020) introduced pyproject.toml.
Setuptools is fully compatible with pyroject.
All the config in setup.cfg and setup.py will also be in pyproject.toml project template if this is merged.
Minimalist config to make adopting dagster easy.

Summary & Motivation

See #25244

How I Tested These Changes

Last time I tried the run the test suite it failed because my macbook has an intel chipset.

Changelog

(CLI)
dagster project scaffold: Add option to create dagster projects from templates with excluded files/filepaths
dagster project scaffold-code-location: Refactored to leverage scaffold command with excludes flag. Deprecated.

python_modules/dagster/dagster/_generate/generate.py

dbrtly · 2024-10-15T07:53:07Z

Note that excludes is flexible.

dagster project scaffold --name qs
--excludes "asset"
--excludes "test"
--excludes "setup*"

PedramNavid · 2024-10-15T17:45:25Z

@dbrtly instead of adding an excludes, could we just remove the setup.cfg and setup.py from the templates? is there any reason to keep them?

cc: @cmpadden who thinks about this stuff more than i do these days

dbrtly · 2024-10-15T19:36:11Z

To me no that would be even better. I assumed the comms to remove them would be involved. My baggage.

PedramNavid · 2024-10-15T21:32:37Z

Might be worth doing — something we'll look into!

…

On Tue, Oct 15, 2024 at 12:36 PM, Daniel Bartley < ***@***.*** > wrote: To me no that would be even better. I assumed the comms to remove them would be involved. My baggage. — Reply to this email directly, view it on GitHub ( #25245 (comment) ) , or unsubscribe ( https://github.com/notifications/unsubscribe-auth/AAH7LZU4WTCTLD6GXANOKDDZ3VVEFAVCNFSM6AAAAABPZ5VJEKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDIMJUHA2TEOJUGU ). You are receiving this because your review was requested. Message ID: <dagster-io/dagster/pull/25245/c2414852945 @ github. com>

dbrtly · 2024-10-15T23:50:40Z

I'll make a new branch to remove those setup files completely. I still think excludes is valid to enable using the scaffold feature in the docs.

Example the current docs suggest a reluctance to include the tests directory in hello_dagster.

I'm pro aggressively remove setup files if you can handle it.

dbrtly · 2024-10-16T01:33:12Z

companion PR to remove setup.cfg and setup.py from the templates as proposed by Pedram
#25303

cmpadden · 2024-10-16T15:20:43Z

I think this looks great, @dbrtly - thanks!

If you could address the pyright and ruff issues, then I believe this is good to merge.

pyright:

[2024-10-15T19:11:30Z] /workdir/python_modules/dagster/dagster/_generate/download.py:
[2024-10-15T19:11:30Z]   90:15: Argument missing for parameter "excludes" (reportCallIssue)

ruff:

[2024-10-15T18:47:34Z] python_modules/dagster/dagster/_generate/generate.py:1:1: I001 [*] Import block is un-sorted or un-formatted
[2024-10-15T18:47:34Z]    |
[2024-10-15T18:47:34Z]  1 | / from __future__ import annotations
[2024-10-15T18:47:34Z]  2 | | import os
[2024-10-15T18:47:34Z]  3 | | import posixpath
[2024-10-15T18:47:34Z]  4 | |
[2024-10-15T18:47:34Z]  5 | | import click
[2024-10-15T18:47:34Z]  6 | | import jinja2
[2024-10-15T18:47:34Z]  7 | |
[2024-10-15T18:47:34Z]  8 | | from dagster.version import __version__ as dagster_version
[2024-10-15T18:47:34Z]  9 | |
[2024-10-15T18:47:34Z] 10 | | IGNORE_PATTERN_LIST = ["__pycache__", ".pytest_cache", "*.egg-info", ".DS_Store", "tox.ini"]
[2024-10-15T18:47:34Z]    | |_^ I001
[2024-10-15T18:47:34Z]    |
[2024-10-15T18:47:34Z]    = help: Organize imports
[2024-10-15T18:47:34Z]
[2024-10-15T18:47:34Z] python_modules/dagster/dagster/_generate/generate.py:1:24: TID251 `__future__.annotations` is banned: Directly quote annotations instead.
[2024-10-15T18:47:34Z]   |
[2024-10-15T18:47:34Z] 1 | from __future__ import annotations
[2024-10-15T18:47:34Z]   |                        ^^^^^^^^^^^ TID251
[2024-10-15T18:47:34Z] 2 | import os
[2024-10-15T18:47:34Z] 3 | import posixpath
[2024-10-15T18:47:34Z]   |
[2024-10-15T18:47:34Z]
[2024-10-15T18:47:34Z] Found 2 errors.

And it appears some the CLI unit tests have failed:

2024-10-15T18:52:39Z] =================================== FAILURES ===================================
[2024-10-15T18:52:39Z] _________________ test_scaffold_code_location_command_succeeds _________________
[2024-10-15T18:52:39Z]
[2024-10-15T18:52:39Z]     def test_scaffold_code_location_command_succeeds():
[2024-10-15T18:52:39Z]         runner = CliRunner()
[2024-10-15T18:52:39Z]         with runner.isolated_filesystem():
[2024-10-15T18:52:39Z]             result = runner.invoke(scaffold_code_location_command, ["--name", "my_dagster_code"])
[2024-10-15T18:52:39Z] >           assert result.exit_code == 0
[2024-10-15T18:52:39Z] E           assert 1 == 0
[2024-10-15T18:52:39Z] E            +  where 1 = <Result TypeError('can only concatenate list (not "tuple") to list')>.exit_code
[2024-10-15T18:52:39Z]
[2024-10-15T18:52:39Z] dagster_tests/cli_tests/test_project_commands.py:67: AssertionError
[2024-10-15T18:52:39Z] ______________________ test_from_example_command_succeeds ______________________
[2024-10-15T18:52:39Z]
[2024-10-15T18:52:39Z]     def test_from_example_command_succeeds():
[2024-10-15T18:52:39Z]         runner = CliRunner()
[2024-10-15T18:52:39Z]         with runner.isolated_filesystem():
[2024-10-15T18:52:39Z]             result = runner.invoke(
[2024-10-15T18:52:39Z]                 from_example_command,
[2024-10-15T18:52:39Z]                 ["--name", "my_dagster_project", "--example", "assets_dbt_python"],
[2024-10-15T18:52:39Z]             )
[2024-10-15T18:52:39Z] >           assert result.exit_code == 0
[2024-10-15T18:52:39Z] E           assert 1 == 0
[2024-10-15T18:52:39Z] E            +  where 1 = <Result TypeError("_should_skip_file() missing 1 required positional argument: 'excludes'")>.exit_code
[2024-10-15T18:52:39Z]
[2024-10-15T18:52:39Z] dagster_tests/cli_tests/test_project_commands.py:96: AssertionError
[2024-10-15T18:52:39Z] _________________ test_from_example_command_versioned_succeeds _________________
[2024-10-15T18:52:39Z]
[2024-10-15T18:52:39Z]     def test_from_example_command_versioned_succeeds():
[2024-10-15T18:52:39Z]         runner = CliRunner()
[2024-10-15T18:52:39Z]         with runner.isolated_filesystem():
[2024-10-15T18:52:39Z]             result = runner.invoke(
[2024-10-15T18:52:39Z]                 from_example_command,
[2024-10-15T18:52:39Z]                 [
[2024-10-15T18:52:39Z]                     "--name",
[2024-10-15T18:52:39Z]                     "my_dagster_project",
[2024-10-15T18:52:39Z]                     "--example",
[2024-10-15T18:52:39Z]                     "assets_dbt_python",
[2024-10-15T18:52:39Z]                     "--version",
[2024-10-15T18:52:39Z]                     "1.3.11",
[2024-10-15T18:52:39Z]                 ],
[2024-10-15T18:52:39Z]             )
[2024-10-15T18:52:39Z] >           assert result.exit_code == 0
[2024-10-15T18:52:39Z] E           assert 1 == 0
[2024-10-15T18:52:39Z] E            +  where 1 = <Result TypeError("_should_skip_file() missing 1 required positional argument: 'excludes'")>.exit_code
[2024-10-15T18:52:39Z]
[2024-10-15T18:52:39Z] dagster_tests/cli_tests/test_project_commands.py:118: AssertionError
[2024-10-15T18:52:39Z] ____________________ test_from_example_command_default_name ____________________
[2024-10-15T18:52:39Z]
[2024-10-15T18:52:39Z]     def test_from_example_command_default_name():
[2024-10-15T18:52:39Z]         runner = CliRunner()
[2024-10-15T18:52:39Z]         with runner.isolated_filesystem():
[2024-10-15T18:52:39Z]             result = runner.invoke(
[2024-10-15T18:52:39Z]                 from_example_command,
[2024-10-15T18:52:39Z]                 ["--name", "assets_dbt_python", "--example", "assets_dbt_python"],
[2024-10-15T18:52:39Z]             )
[2024-10-15T18:52:39Z] >           assert result.exit_code == 0
[2024-10-15T18:52:39Z] E           assert 1 == 0
[2024-10-15T18:52:39Z] E            +  where 1 = <Result TypeError("_should_skip_file() missing 1 required positional argument: 'excludes'")>.exit_code
[2024-10-15T18:52:39Z]
[2024-10-15T18:52:39Z] dagster_tests/cli_tests/test_project_commands.py:133: AssertionError
[2024-10-15T18:52:39Z] _____________ test_available_examples_in_sync_with_example_folder ______________
[2024-10-15T18:52:39Z]
[2024-10-15T18:52:39Z]     def test_available_examples_in_sync_with_example_folder():
[2024-10-15T18:52:39Z]         # ensure the list of AVAILABLE_EXAMPLES is in sync with the example folder minus EXAMPLES_TO_IGNORE
[2024-10-15T18:52:39Z]         example_folder = file_relative_path(__file__, "../../../../examples")
[2024-10-15T18:52:39Z] >       available_examples_in_folder = [
[2024-10-15T18:52:39Z]             e
[2024-10-15T18:52:39Z]             for e in os.listdir(example_folder)
[2024-10-15T18:52:39Z]             if (
[2024-10-15T18:52:39Z]                 os.path.isdir(os.path.join(example_folder, e))
[2024-10-15T18:52:39Z]                 and e not in EXAMPLES_TO_IGNORE
[2024-10-15T18:52:39Z]                 and not _should_skip_file(e)
[2024-10-15T18:52:39Z]             )
[2024-10-15T18:52:39Z]         ]
[2024-10-15T18:52:39Z]
[2024-10-15T18:52:39Z] dagster_tests/cli_tests/test_project_commands.py:144:
[2024-10-15T18:52:39Z] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
[2024-10-15T18:52:39Z]
[2024-10-15T18:52:39Z] .0 = <list_iterator object at 0x7f3bb493a320>
[2024-10-15T18:52:39Z]
[2024-10-15T18:52:39Z]     available_examples_in_folder = [
[2024-10-15T18:52:39Z]         e
[2024-10-15T18:52:39Z]         for e in os.listdir(example_folder)
[2024-10-15T18:52:39Z]         if (
[2024-10-15T18:52:39Z]             os.path.isdir(os.path.join(example_folder, e))
[2024-10-15T18:52:39Z]             and e not in EXAMPLES_TO_IGNORE
[2024-10-15T18:52:39Z] >           and not _should_skip_file(e)
[2024-10-15T18:52:39Z]         )
[2024-10-15T18:52:39Z]     ]
[2024-10-15T18:52:39Z] E   TypeError: _should_skip_file() missing 1 required positional argument: 'excludes'
[2024-10-15T18:52:39Z]
[2024-10-15T18:52:39Z] dagster_tests/cli_tests/test_project_commands.py:150: TypeError

cmpadden

See comment above.

cmpadden · 2024-10-18T21:15:00Z

@dbrtly - I wasn't able to make a PR merging into this one, so it's merging into master. But my findings are that the multiple=True args can be a Tuple or a List, which is what was causing some tests to fail. This updates the annotation and forces it to a list.

https://github.com/dagster-io/dagster/pull/25379/files#diff-bb06c1741d6917084a15032c17164582227a166023df6fd0a94f2525de04df72R127-R128

And the wildcard, setup.* test was failing, but it does work when they are explicitly excluded. There might be a bug in the glob logic.

https://github.com/dagster-io/dagster/pull/25379/files#diff-4932d2cd14cf27e7fcb4c7f7b8b42de335737db862be22eb7263c1aaac569f8dR85-R93

And I also had to remove the from __future__ import annotations import. Not entirely sure what that's about--but by doing that I may be breaking older version. Need to run the full test suite.

Feel free to make these changes here, and I can close out my PR.

dbrtly · 2024-10-19T01:53:49Z

@cmpadden fwiw Colton, I'm ok with you committing directly to this branch. Not sure what was happening there.

dbrtly · 2024-10-19T13:05:39Z

Apologies for so many changes.

Based on my experimental observation, there is no practical difference between these snippets:
dagster project scaffold --name dagster_<3 --excludes README.md
dagster project scaffold-code-location --name dagster_<3

Given this PR adds the excludes flag, I took the liberty of deleting the duplicated code and deprecating the scaffold-code-location command.

I tried to break the new code with this test but nothing weird happened. Only a mess.

def test_project_scaffold_command_nested_succeeds():
    """Documenting behaviour and expecting that nobody would ever want to do this."""
    runner = CliRunner()
    with runner.isolated_filesystem():
        result = runner.invoke(scaffold_command, ["--name", "dagster_package/dagster_module"])
        assert result.exit_code == 0
        assert os.path.exists("dagster_package/dagster_module/dagster_package/dagster_module")
        assert os.path.exists("dagster_package/dagster_module/README.md")
        assert os.path.exists("dagster_package/dagster_module/pyproject.toml")

cmpadden · 2024-10-21T18:11:46Z

Apologies for so many changes.

Based on my experimental observation, there is no practical difference between these snippets: dagster project scaffold --name dagster_<3 --excludes README.md dagster project scaffold-code-location --name dagster_<3

Given this PR adds the excludes flag, I took the liberty of deleting the duplicated code and deprecating the scaffold-code-location command.

I tried to break the new code with this test but nothing weird happened. Only a mess.
def test_project_scaffold_command_nested_succeeds():
    """Documenting behaviour and expecting that nobody would ever want to do this."""
    runner = CliRunner()
    with runner.isolated_filesystem():
        result = runner.invoke(scaffold_command, ["--name", "dagster_package/dagster_module"])
        assert result.exit_code == 0
        assert os.path.exists("dagster_package/dagster_module/dagster_package/dagster_module")
        assert os.path.exists("dagster_package/dagster_module/README.md")
        assert os.path.exists("dagster_package/dagster_module/pyproject.toml")

No worries at all, @dbrtly. I pushed a couple of commits to fix ruff and pyright.

However, we're still getting a few unit test failures. For a faster feedback loop, I'd suggest running it locally via:

$ pytest -v dagster_tests/cli_tests/test_project_commands.py

[2024-10-21T18:10:23Z] FAILED dagster_tests/cli_tests/test_project_commands.py::test_project_scaffold_command_fails_on_package_conflict - AssertionError: assert 1 == 0
[2024-10-21T18:10:23Z]  +  where 1 = <Result FileNotFoundError(2, 'No such file or directory')>.exit_code
[2024-10-21T18:10:23Z] FAILED dagster_tests/cli_tests/test_project_commands.py::test_project_scaffold_command_succeeds - AssertionError: assert 1 == 0
[2024-10-21T18:10:23Z]  +  where 1 = <Result FileNotFoundError(2, 'No such file or directory')>.exit_code
[2024-10-21T18:10:23Z] FAILED dagster_tests/cli_tests/test_project_commands.py::test_project_scaffold_command_excludes_succeeds - AssertionError: assert 1 == 0
[2024-10-21T18:10:23Z]  +  where 1 = <Result FileNotFoundError(2, 'No such file or directory')>.exit_code
[2024-10-21T18:10:23Z] FAILED dagster_tests/cli_tests/test_project_commands.py::test_scaffold_code_location_command_succeeds - AssertionError: assert 1 == 0
[2024-10-21T18:10:23Z]  +  where 1 = <Result FileNotFoundError(2, 'No such file or directory')>.exit_code

dbrtly · 2024-10-27T01:21:25Z

got the tests passing. refactored some of them because flakey

PEP 621 (Nov 2020) introduced pyproject.toml. Setuptools is fully compatible with pyroject Completes dagster config in project template. Minimalist config to make adopting dagster easy.

dbrtly · 2024-10-28T23:51:55Z

I added the excludes validation step at the last minute. Fixed now.

 ========================================================================== test session starts ==========================================================================
platform linux -- Python 3.12.3, pytest-8.3.3, pluggy-1.5.0 -- /home/dbrtly00/dagster/.venv/bin/python3
cachedir: .pytest_cache
rootdir: /home/dbrtly00/dagster
configfile: pyproject.toml
plugins: rerunfailures-14.0, buildkite-test-collector-0.1.9, syrupy-4.7.2, xdist-3.6.1, mock-3.14.0, asyncio-0.24.0, cov-5.0.0
asyncio: mode=Mode.STRICT, default_loop_scope=None
collected 17 items                                                                                                                                                      

dagster/python_modules/dagster/dagster_tests/cli_tests/test_project_commands.py::test_project_scaffold_command_fails_when_dir_path_exists PASSED                  [  5%]
dagster/python_modules/dagster/dagster_tests/cli_tests/test_project_commands.py::test_project_scaffold_command_fails_on_package_conflict PASSED                   [ 11%]
dagster/python_modules/dagster/dagster_tests/cli_tests/test_project_commands.py::test_project_scaffold_command_succeeds PASSED                                    [ 17%]
dagster/python_modules/dagster/dagster_tests/cli_tests/test_project_commands.py::test_project_scaffold_command_excludes_succeeds PASSED                           [ 23%]
dagster/python_modules/dagster/dagster_tests/cli_tests/test_project_commands.py::test_project_scaffold_command_excludes_fails_on_required_files PASSED            [ 29%]
dagster/python_modules/dagster/dagster_tests/cli_tests/test_project_commands.py::test_from_example_command_fails_when_example_not_available PASSED                [ 35%]
dagster/python_modules/dagster/dagster_tests/cli_tests/test_project_commands.py::test_from_example_command_succeeds PASSED                                        [ 41%]
dagster/python_modules/dagster/dagster_tests/cli_tests/test_project_commands.py::test_from_example_command_versioned_succeeds PASSED                              [ 47%]
dagster/python_modules/dagster/dagster_tests/cli_tests/test_project_commands.py::test_from_example_command_default_name PASSED                                    [ 52%]
dagster/python_modules/dagster/dagster_tests/cli_tests/test_project_commands.py::test_available_examples_in_sync_with_example_folder PASSED                       [ 58%]
dagster/python_modules/dagster/dagster_tests/cli_tests/test_project_commands.py::test_scaffold_code_location_deprecation PASSED                                   [ 64%]
dagster/python_modules/dagster/dagster_tests/cli_tests/test_project_commands.py::test_scaffold_code_location_scaffold_command_fails_when_dir_path_exists PASSED   [ 70%]
dagster/python_modules/dagster/dagster_tests/cli_tests/test_project_commands.py::test_scaffold_code_location_command_succeeds PASSED                              [ 76%]
dagster/python_modules/dagster/dagster_tests/cli_tests/test_project_commands.py::test_scaffold_repository_deprecation PASSED                                      [ 82%]
dagster/python_modules/dagster/dagster_tests/cli_tests/test_project_commands.py::test_scaffold_repository_scaffold_command_fails_when_dir_path_exists PASSED      [ 88%]
dagster/python_modules/dagster/dagster_tests/cli_tests/test_project_commands.py::test_scaffold_repository_command_succeeds PASSED                                 [ 94%]
dagster/python_modules/dagster/dagster_tests/cli_tests/test_project_commands.py::test_versioned_download PASSED                                                   [100%]

========================================================================== 17 passed in 48.03s ==========================================================================

cmpadden · 2024-10-29T02:42:17Z

Turns out this method was used elsewhere, and has caused some failures in other test suites, and our docs build, due to the renaming of this method:

[2024-10-29T02:34:35Z] /workdir/docs/sphinx/sections/api/apidocs/libraries/dagster-dbt.rst:19: ERROR: Failed to import "project_app_typer_click_object" from "dagster_dbt.cli.app". The following exception was raised:
[2024-10-29T02:34:35Z] Traceback (most recent call last):
[2024-10-29T02:34:35Z]   File "/workdir/docs/.tox/sphinx/lib/python3.11/site-packages/sphinx_click/ext.py", line 444, in _load_module
[2024-10-29T02:34:35Z]     mod = __import__(module_name, globals(), locals(), [attr_name])
[2024-10-29T02:34:35Z]           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[2024-10-29T02:34:35Z]   File "/workdir/python_modules/libraries/dagster-dbt/dagster_dbt/cli/app.py", line 8, in <module>
[2024-10-29T02:34:35Z]     from dagster._cli.project import check_if_pypi_package_conflict_exists
[2024-10-29T02:34:35Z] ImportError: cannot import name 'check_if_pypi_package_conflict_exists' from 'dagster._cli.project' (/workdir/python_modules/dagster/dagster/_cli/project.py) [docutils]
[2024-10-29T02:34:35Z] looking for now-outdated files... none found

Apologies that this one is taking so long to get over the finish line, but it turns out it's pretty intertwined with everything else!

dbrtly · 2024-10-29T04:11:15Z

Where are the failing tests?
Might revert the name.

Fixed flakey tests by refactor `os.path` to pathlib.Path.

cmpadden · 2024-10-29T13:58:23Z

Where are the failing tests? Might revert the name.

Looks to be used in dagster-dbt and dagster-sdf:

[2024-10-29T02:35:11Z] _ ERROR collecting python_modules/libraries/dagster-dbt/dagster_dbt_tests/cli/test_prepare_and_package.py _
[2024-10-29T02:35:11Z] ImportError while importing test module '/workdir/python_modules/libraries/dagster-dbt/dagster_dbt_tests/cli/test_prepare_and_package.py'.
[2024-10-29T02:35:11Z] Hint: make sure your test modules/packages have valid Python names.
[2024-10-29T02:35:11Z] Traceback:
[2024-10-29T02:35:11Z] .tox/py311-dbt17-cloud/lib/python3.11/site-packages/_pytest/python.py:493: in importtestmodule
[2024-10-29T02:35:11Z]     mod = import_path(
[2024-10-29T02:35:11Z] .tox/py311-dbt17-cloud/lib/python3.11/site-packages/_pytest/pathlib.py:582: in import_path
[2024-10-29T02:35:11Z]     importlib.import_module(module_name)
[2024-10-29T02:35:11Z] /usr/local/lib/python3.11/importlib/__init__.py:126: in import_module
[2024-10-29T02:35:11Z]     return _bootstrap._gcd_import(name[level:], package, level)
[2024-10-29T02:35:11Z] <frozen importlib._bootstrap>:1204: in _gcd_import
[2024-10-29T02:35:11Z]     ???
[2024-10-29T02:35:11Z] <frozen importlib._bootstrap>:1176: in _find_and_load
[2024-10-29T02:35:11Z]     ???
[2024-10-29T02:35:11Z] <frozen importlib._bootstrap>:1147: in _find_and_load_unlocked
[2024-10-29T02:35:11Z]     ???
[2024-10-29T02:35:11Z] <frozen importlib._bootstrap>:690: in _load_unlocked
[2024-10-29T02:35:11Z]     ???
[2024-10-29T02:35:11Z] .tox/py311-dbt17-cloud/lib/python3.11/site-packages/_pytest/assertion/rewrite.py:184: in exec_module
[2024-10-29T02:35:11Z]     exec(co, module.__dict__)
[2024-10-29T02:35:11Z] dagster_dbt_tests/cli/test_prepare_and_package.py:11: in <module>
[2024-10-29T02:35:11Z]     from dagster_dbt.cli.app import app
[2024-10-29T02:35:11Z] dagster_dbt/cli/app.py:8: in <module>
[2024-10-29T02:35:11Z]     from dagster._cli.project import check_if_pypi_package_conflict_exists
[2024-10-29T02:35:11Z] E   ImportError: cannot import name 'check_if_pypi_package_conflict_exists' from 'dagster._cli.project' (/workdir/python_modules/dagster/dagster/_cli/project.py)
[2024-10-29T02:35:11Z] _ ERROR collecting python_modules/libraries/dagster-dbt/dagster_dbt_tests/cli/test_scaffold.py _

[2024-10-29T02:40:14Z] _ ERROR collecting python_modules/libraries/dagster-sdf/dagster_sdf_tests/cli/test_scaffold.py _
[2024-10-29T02:40:14Z] ImportError while importing test module '/workdir/python_modules/libraries/dagster-sdf/dagster_sdf_tests/cli/test_scaffold.py'.
[2024-10-29T02:40:14Z] Hint: make sure your test modules/packages have valid Python names.
[2024-10-29T02:40:14Z] Traceback:
[2024-10-29T02:40:14Z] .tox/py311/lib/python3.11/site-packages/_pytest/python.py:493: in importtestmodule
[2024-10-29T02:40:14Z]     mod = import_path(
[2024-10-29T02:40:14Z] .tox/py311/lib/python3.11/site-packages/_pytest/pathlib.py:582: in import_path
[2024-10-29T02:40:14Z]     importlib.import_module(module_name)
[2024-10-29T02:40:14Z] /usr/local/lib/python3.11/importlib/__init__.py:126: in import_module
[2024-10-29T02:40:14Z]     return _bootstrap._gcd_import(name[level:], package, level)
[2024-10-29T02:40:14Z] <frozen importlib._bootstrap>:1204: in _gcd_import
[2024-10-29T02:40:14Z]     ???
[2024-10-29T02:40:14Z] <frozen importlib._bootstrap>:1176: in _find_and_load
[2024-10-29T02:40:14Z]     ???
[2024-10-29T02:40:14Z] <frozen importlib._bootstrap>:1147: in _find_and_load_unlocked
[2024-10-29T02:40:14Z]     ???
[2024-10-29T02:40:14Z] <frozen importlib._bootstrap>:690: in _load_unlocked
[2024-10-29T02:40:14Z]     ???
[2024-10-29T02:40:14Z] .tox/py311/lib/python3.11/site-packages/_pytest/assertion/rewrite.py:184: in exec_module
[2024-10-29T02:40:14Z]     exec(co, module.__dict__)
[2024-10-29T02:40:14Z] dagster_sdf_tests/cli/test_scaffold.py:9: in <module>
[2024-10-29T02:40:14Z]     from dagster_sdf.cli.app import app
[2024-10-29T02:40:14Z] dagster_sdf/cli/app.py:8: in <module>
[2024-10-29T02:40:14Z]     from dagster._cli.project import check_if_pypi_package_conflict_exists
[2024-10-29T02:40:14Z] E   ImportError: cannot import name 'check_if_pypi_package_conflict_exists' from 'dagster._cli.project' (/workdir/python_modules/dagster/dagster/_cli/project.py)

cmpadden

We're running green!

Thanks for sticking it out with this one, and thanks for your contribution.

python_modules/dagster/dagster/_cli/project.py

dbrtly force-pushed the project-template-config-minimalism branch 4 times, most recently from 1495a52 to 75ca860 Compare October 12, 2024 07:35

graphite-app bot reviewed Oct 12, 2024

View reviewed changes

python_modules/dagster/dagster/_generate/generate.py Outdated Show resolved Hide resolved

dbrtly mentioned this pull request Oct 12, 2024

Simplify scaffold project with option to exclude files from the template #25244

Open

dbrtly force-pushed the project-template-config-minimalism branch from 75ca860 to 7c75421 Compare October 12, 2024 11:40

garethbrickman requested a review from PedramNavid October 14, 2024 14:18

PedramNavid requested a review from cmpadden October 15, 2024 17:44

dbrtly mentioned this pull request Oct 16, 2024

refactor: consolidate config to pyproject.toml only #25303

Open

cmpadden requested changes Oct 16, 2024

View reviewed changes

dbrtly force-pushed the project-template-config-minimalism branch from 7c75421 to 360df7d Compare October 17, 2024 10:30

cmpadden mentioned this pull request Oct 18, 2024

[cli] fix ruff and unit tests for --excludes #25379

Closed

dbrtly force-pushed the project-template-config-minimalism branch from e916af9 to 2ec9386 Compare October 19, 2024 12:48

dbrtly force-pushed the project-template-config-minimalism branch from 1e96ccb to a0f13e9 Compare October 27, 2024 01:20

dbrtly requested a review from cmpadden October 27, 2024 01:32

dbrtly force-pushed the project-template-config-minimalism branch 2 times, most recently from 06d2320 to 8178a4e Compare October 27, 2024 01:42

dbrtly force-pushed the project-template-config-minimalism branch 2 times, most recently from 91ecf1c to d3d97c5 Compare October 28, 2024 23:48

dbrtly and others added 6 commits October 28, 2024 23:50

feat(cli.project): enable optional minimalist configuration in templates

270b21c

PEP 621 (Nov 2020) introduced pyproject.toml. Setuptools is fully compatible with pyroject Completes dagster config in project template. Minimalist config to make adopting dagster easy.

resolve make ruff

b9529d8

appease pyright

b798158

use context to invoke scaffold command

c7701a8

include excludes in generate_project

2086480

rm unneeded project list entry

a54ac52

dbrtly force-pushed the project-template-config-minimalism branch from d3d97c5 to a12248f Compare October 28, 2024 23:50

eat(cli): add dagster project scaffold --excludes foo option

4340fa8

Fixed flakey tests by refactor `os.path` to pathlib.Path.

dbrtly force-pushed the project-template-config-minimalism branch from f203e99 to 4340fa8 Compare October 29, 2024 10:26

fix pyright

4f99f5b

cmpadden approved these changes Oct 29, 2024

View reviewed changes

python_modules/dagster/dagster/_cli/project.py Outdated Show resolved Hide resolved

cmpadden merged commit 45c6f1a into dagster-io:master Oct 29, 2024
1 check passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(cli.project): enable optional minimalist configuration in templates #25245

feat(cli.project): enable optional minimalist configuration in templates #25245

dbrtly commented Oct 12, 2024 •

edited

Loading

dbrtly commented Oct 15, 2024

PedramNavid commented Oct 15, 2024

dbrtly commented Oct 15, 2024

PedramNavid commented Oct 15, 2024 via email

dbrtly commented Oct 15, 2024

dbrtly commented Oct 16, 2024

cmpadden commented Oct 16, 2024

cmpadden left a comment

cmpadden commented Oct 18, 2024 •

edited

Loading

dbrtly commented Oct 19, 2024 •

edited

Loading

dbrtly commented Oct 19, 2024

cmpadden commented Oct 21, 2024

dbrtly commented Oct 27, 2024

dbrtly commented Oct 28, 2024

cmpadden commented Oct 29, 2024

dbrtly commented Oct 29, 2024 •

edited

Loading

cmpadden commented Oct 29, 2024

cmpadden left a comment

feat(cli.project): enable optional minimalist configuration in templates #25245

feat(cli.project): enable optional minimalist configuration in templates #25245

Conversation

dbrtly commented Oct 12, 2024 • edited Loading

Summary & Motivation

How I Tested These Changes

Changelog

dbrtly commented Oct 15, 2024

PedramNavid commented Oct 15, 2024

dbrtly commented Oct 15, 2024

PedramNavid commented Oct 15, 2024 via email

dbrtly commented Oct 15, 2024

dbrtly commented Oct 16, 2024

cmpadden commented Oct 16, 2024

cmpadden left a comment

Choose a reason for hiding this comment

cmpadden commented Oct 18, 2024 • edited Loading

dbrtly commented Oct 19, 2024 • edited Loading

dbrtly commented Oct 19, 2024

cmpadden commented Oct 21, 2024

dbrtly commented Oct 27, 2024

dbrtly commented Oct 28, 2024

cmpadden commented Oct 29, 2024

dbrtly commented Oct 29, 2024 • edited Loading

cmpadden commented Oct 29, 2024

cmpadden left a comment

Choose a reason for hiding this comment

dbrtly commented Oct 12, 2024 •

edited

Loading

cmpadden commented Oct 18, 2024 •

edited

Loading

dbrtly commented Oct 19, 2024 •

edited

Loading

dbrtly commented Oct 29, 2024 •

edited

Loading