[dagster-ge] Drop support for old APIs (#25540)

## Summary & Motivation Update `dagster-ge` to support newer versions and drop support for older versions. Why? - Prior to this PR, the most recent version of `great_expectations` supported by `dagster_ge` is `0.17.11`, released 2023-08-17 (so over a year old). There have been many releases (including 1.x series) since then. - Pydantic 2 is only supported in 0.17.15+, so we can't update `dagster` to Pydantic 2-only until `dagster-ge` supports it. - The later versions of `great_expectations` we want to support drop some of the old APIs this integration was still supporting. - Due to `dagster-ge` supporting only old `great_expectations`, `dagster-ge` was a PITA for environment management, with nth-order dependencies capped at old versions conflicting with the newer versions we'd like to use elsewhere. What was done here: - Prior to this PR, `dagster-ge` exposed two versions of its sole public API: `ge_validation_op_factory` and `ge_validation_op_factory_v3`. `ge_validation_op_factory_v3` uses APIs that are supported in more recent GE versions. `ge_validation_op_factory` used ancient APIs. Also `ge_validation_op_factory_v3` wasn't even exported from the top-level. I simply renamed `ge_validation_op_factory_v3` to `ge_validation_op_factory` and deleted the old `ge_validation_op_factory`. - General consolidation refactor of tests for clarity and concision. - Adapt `examples/with_great_expectations` to use the new version. - Update dev install script to install `dagster-ge` again. It was skipping it due to the aforementioned nth-order dependency conflicts. ## How I Tested These Changes Revamped tests. ## Changelog [dagster-ge] `dagster-ge` now only supports `great_expectations>=0.17.15`. The `ge_validation_op_factory` API has been replaced with the API previously called `ge_validation_op_factory_v3`. Now there is only one API, `ge_validation_op_factory`.
dagster-io · Oct 25, 2024 · 7f47c5f · 7f47c5f
1 parent ccd0567
commit 7f47c5f
Show file tree

Hide file tree

Showing 52 changed files with 474 additions and 4,283 deletions.
diff --git a/docs/content/api/modules.json.gz b/docs/content/api/modules.json.gz
diff --git a/docs/content/api/searchindex.json.gz b/docs/content/api/searchindex.json.gz
diff --git a/docs/content/api/sections.json.gz b/docs/content/api/sections.json.gz
diff --git a/docs/next/public/objects.inv b/docs/next/public/objects.inv
diff --git a/examples/with_great_expectations/setup.py b/examples/with_great_expectations/setup.py
@@ -6,7 +6,6 @@
     install_requires=[
         "dagster",
         "dagster-ge",
-        "great_expectations>=0.14.12",  # pinned because pip is using the cached wheel for 0.13.14
     ],
     extras_require={"dev": ["dagster-webserver", "pytest"]},
 )
diff --git a/examples/with_great_expectations/with_great_expectations/ge_demo.py b/examples/with_great_expectations/with_great_expectations/ge_demo.py
@@ -32,7 +32,12 @@ def postprocess_payroll(numrows, expectation):
 
 # start_ge_demo_marker_factory
 payroll_expectations = ge_validation_op_factory(
-    name="ge_validation_op", datasource_name="getest", suite_name="basic.warning"
+    name="ge_validation_op",
+    datasource_name="getest",
+    data_connector_name="my_runtime_data_connector",
+    data_asset_name="test_asset",
+    suite_name="basic.warning",
+    batch_identifiers={"foo": "bar"},
 )
 # end_ge_demo_marker_factory
 

diff --git a/...expectations/with_great_expectations/great_expectations/expectations/.ge_store_backend_id b/...expectations/with_great_expectations/great_expectations/expectations/.ge_store_backend_id
diff --git a/...with_great_expectations/with_great_expectations/great_expectations/great_expectations.yml b/...with_great_expectations/with_great_expectations/great_expectations/great_expectations.yml
diff --git a/...s/with_great_expectations/great_expectations/notebooks/pandas/validation_playground.ipynb b/...s/with_great_expectations/great_expectations/notebooks/pandas/validation_playground.ipynb