[SPARK-36989][TESTS][PYTHON] Add type hints data tests #34296

zero323 · 2021-10-15T16:36:13Z

What changes were proposed in this pull request?

This PR:

Adds basic data test runner to dev/lint-python, using typeddjango/pytest-mypy-plugins
Migrates data test cases from pyspark-stubs

In case of failure, a message similar to the following one

starting mypy annotations test...
annotations passed mypy checks.

starting mypy data test...
annotations failed data checks:
============================= test session starts ==============================
platform linux -- Python 3.9.7, pytest-6.2.5, py-1.10.0, pluggy-1.0.0
rootdir: /path/to/spark/python, configfile: pyproject.toml
plugins: mypy-plugins-1.9.2
collected 37 items

python/pyspark/ml/tests/typing/test_classification.yml ..                [  5%]
python/pyspark/ml/tests/typing/test_evaluation.yml .                     [  8%]
python/pyspark/ml/tests/typing/test_feature.yml .                        [ 10%]
python/pyspark/ml/tests/typing/test_param.yml .                          [ 13%]
python/pyspark/ml/tests/typing/test_readable.yml .                       [ 16%]
python/pyspark/ml/tests/typing/test_regression.yml ..                    [ 21%]
python/pyspark/sql/tests/typing/test_column.yml F                        [ 24%]
python/pyspark/sql/tests/typing/test_dataframe.yml .......               [ 43%]
python/pyspark/sql/tests/typing/test_functions.yml .                     [ 45%]
python/pyspark/sql/tests/typing/test_pandas_compatibility.yml ..         [ 51%]
python/pyspark/sql/tests/typing/test_readwriter.yml ..                   [ 56%]
python/pyspark/sql/tests/typing/test_session.yml .....                   [ 70%]
python/pyspark/sql/tests/typing/test_udf.yml .......                     [ 89%]
python/pyspark/tests/typing/test_context.yml .                           [ 91%]
python/pyspark/tests/typing/test_core.yml .                              [ 94%]
python/pyspark/tests/typing/test_rdd.yml .                               [ 97%]
python/pyspark/tests/typing/test_resultiterable.yml .                    [100%]

=================================== FAILURES ===================================
______________________________ colDateTimeCompare ______________________________
/path/to/spark/python/pyspark/sql/tests/typing/test_column.yml:39: 
E   pytest_mypy_plugins.utils.TypecheckAssertionError: Invalid output: 
E   Actual:
E     main:20: note: Revealed type is "pyspark.sql.column.Column" (diff)
E   Expected:
E     main:20: note: Revealed type is "datetime.date*" (diff)
E   Alignment of first line difference:
E     E: ...ote: Revealed type is "datetime.date*"
E     A: ...ote: Revealed type is "pyspark.sql.column.Column"
E                                  ^
=========================== short test summary info ============================
FAILED python/pyspark/sql/tests/typing/test_column.yml::colDateTimeCompare - 
======================== 1 failed, 36 passed in 56.13s =========================

will be displayed.

Why are the changes needed?

Currently, type annotations are tested primarily for integrity and, to lesser extent, against actual API. Testing against examples is work in progress (SPARK-36997). Data tests allow us to improve coverage and test negative cases (code, that should fail type checker validation).

Does this PR introduce any user-facing change?

No.

How was this patch tested?

Running linter tests with additions proposed in this PR

zero323 · 2021-10-15T16:38:11Z

cc @HyukjinKwon @itholic @ueshin @xinrong-databricks FYI

SparkQA · 2021-10-15T16:40:16Z

Test build #144306 has finished for PR 34296 at commit 736a91a.

This patch fails RAT tests.
This patch merges cleanly.
This patch adds no public classes.

.github/workflows/build_and_test.yml

SparkQA · 2021-10-15T17:27:37Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48785/

SparkQA · 2021-10-15T18:06:50Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48786/

SparkQA · 2021-10-15T18:30:25Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48785/

SparkQA · 2021-10-15T18:47:24Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48786/

SparkQA · 2021-10-15T19:26:05Z

Test build #144307 has finished for PR 34296 at commit 6f55c43.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

HyukjinKwon · 2021-10-16T01:55:02Z

thanks @zero323 for working on this!

SparkQA · 2021-10-17T11:28:52Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48810/

SparkQA · 2021-10-17T12:10:17Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48810/

SparkQA · 2021-10-17T14:03:17Z

Test build #144331 has finished for PR 34296 at commit 149996e.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
class Foo(Params):
class Bar(Params):

SparkQA · 2021-10-18T13:32:23Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48846/

SparkQA · 2021-10-18T14:15:47Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48846/

SparkQA · 2021-10-18T15:13:29Z

Test build #144371 has finished for PR 34296 at commit 62018ce.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
class Foo(Params):
class Bar(Params):

SparkQA · 2021-10-18T16:24:39Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48848/

zero323 · 2021-10-18T17:04:22Z

Quick update:

All test files from pyspark-stubs where migrated and dev scripts are prepared.

However, I hit typeddjango/pytest-mypy-plugins#83, which has serious impact on test performance, when numpy is installed (the whole suite requires > 20 minutes, compared to ~50 seconds when numpy is not present).

SparkQA · 2021-10-18T17:23:06Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48848/

SparkQA · 2021-10-18T18:07:51Z

Test build #144374 has finished for PR 34296 at commit 13ad5e3.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2021-10-20T10:44:23Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48930/

SparkQA · 2021-10-20T11:29:54Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48930/

SparkQA · 2021-10-20T12:35:24Z

Test build #144457 has finished for PR 34296 at commit d3d3755.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

zero323 · 2021-10-20T19:32:31Z

dev/lint-python

+    echo "starting mypy annotations test..."
+    PYTEST_REPORT=$( ($MYPY_BUILD \
+      --config-file python/mypy.ini \
+      --cache-dir /tmp/.mypy_cache/ \


If we set --cache-dir here explicitly, we can reuse cached for the data tests.

Alternatively, we can use $PWD as the --mypy-testing-base in the data tests, but I'd prefer to avoid that, because temporary files are written there.

SparkQA · 2021-10-23T16:44:14Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49028/

…ing package

SparkQA · 2021-10-23T17:45:02Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49028/

SparkQA · 2021-10-23T17:52:37Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49029/

SparkQA · 2021-10-23T18:27:23Z

Test build #144557 has finished for PR 34296 at commit b3e3f25.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2021-10-23T18:52:09Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49029/

SparkQA · 2021-10-23T19:06:58Z

Test build #144558 has finished for PR 34296 at commit 1ae0494.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2021-10-23T19:37:57Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49031/

SparkQA · 2021-10-23T20:16:04Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49031/

SparkQA · 2021-10-23T21:34:24Z

Test build #144560 has finished for PR 34296 at commit e052edc.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

zero323 · 2021-10-26T08:34:00Z

Merged to master.

github-actions bot added BUILD INFRA PYTHON labels Oct 15, 2021

zero323 commented Oct 15, 2021

View reviewed changes

.github/workflows/build_and_test.yml Outdated Show resolved Hide resolved

github-actions bot added ML SQL labels Oct 17, 2021

zero323 force-pushed the SPARK-36989 branch from 149996e to 62018ce Compare October 18, 2021 12:19

zero323 force-pushed the SPARK-36989 branch from 13ad5e3 to d3d3755 Compare October 20, 2021 09:15

zero323 commented Oct 20, 2021

View reviewed changes

zero323 added 12 commits October 23, 2021 19:07

Add data test runner and example tests

34baef1

Add license to test_context.yml

57bef80

Prepare yaml files

1c1ee25

Replace * import from pyspark.sql.types

3ea06b9

Update pytest-mypy-plugins hash

d87ff75

Move data tests to typing subdirectory of the tests of the correspond…

0a88bc7

…ing package

Set explicit cache dir for base mypy run

c410d86

Add future tests for overloaded varargs functions

f2694e9

Address @ueshin's requests

2a99c4e

Drop unused variables

77094ae

Check pytest plugin through imports

af7a34a

Use PYTHON_EXECUTABLE

1ae0494

zero323 force-pushed the SPARK-36989 branch from b3e3f25 to 1ae0494 Compare October 23, 2021 17:07

Update pytest-mypy-plugin hash

e052edc

zero323 marked this pull request as ready for review October 23, 2021 21:05

zero323 requested review from HyukjinKwon and ueshin October 24, 2021 11:49

ueshin approved these changes Oct 25, 2021

View reviewed changes

zero323 closed this in 6d3cfed Oct 26, 2021

zero323 deleted the SPARK-36989 branch October 26, 2021 08:34

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[SPARK-36989][TESTS][PYTHON] Add type hints data tests #34296

[SPARK-36989][TESTS][PYTHON] Add type hints data tests #34296

zero323 commented Oct 15, 2021 •

edited

Loading

zero323 commented Oct 15, 2021

SparkQA commented Oct 15, 2021

SparkQA commented Oct 15, 2021

SparkQA commented Oct 15, 2021

SparkQA commented Oct 15, 2021

SparkQA commented Oct 15, 2021

SparkQA commented Oct 15, 2021

HyukjinKwon commented Oct 16, 2021

SparkQA commented Oct 17, 2021

SparkQA commented Oct 17, 2021

SparkQA commented Oct 17, 2021

SparkQA commented Oct 18, 2021

SparkQA commented Oct 18, 2021

SparkQA commented Oct 18, 2021

SparkQA commented Oct 18, 2021

zero323 commented Oct 18, 2021

SparkQA commented Oct 18, 2021

SparkQA commented Oct 18, 2021

SparkQA commented Oct 20, 2021

SparkQA commented Oct 20, 2021

SparkQA commented Oct 20, 2021

zero323 Oct 20, 2021

SparkQA commented Oct 23, 2021

SparkQA commented Oct 23, 2021

SparkQA commented Oct 23, 2021

SparkQA commented Oct 23, 2021

SparkQA commented Oct 23, 2021

SparkQA commented Oct 23, 2021

SparkQA commented Oct 23, 2021

SparkQA commented Oct 23, 2021

SparkQA commented Oct 23, 2021

zero323 commented Oct 26, 2021

[SPARK-36989][TESTS][PYTHON] Add type hints data tests #34296

[SPARK-36989][TESTS][PYTHON] Add type hints data tests #34296

Conversation

zero323 commented Oct 15, 2021 • edited Loading

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

zero323 commented Oct 15, 2021

SparkQA commented Oct 15, 2021

SparkQA commented Oct 15, 2021

SparkQA commented Oct 15, 2021

SparkQA commented Oct 15, 2021

SparkQA commented Oct 15, 2021

SparkQA commented Oct 15, 2021

HyukjinKwon commented Oct 16, 2021

SparkQA commented Oct 17, 2021

SparkQA commented Oct 17, 2021

SparkQA commented Oct 17, 2021

SparkQA commented Oct 18, 2021

SparkQA commented Oct 18, 2021

SparkQA commented Oct 18, 2021

SparkQA commented Oct 18, 2021

zero323 commented Oct 18, 2021

SparkQA commented Oct 18, 2021

SparkQA commented Oct 18, 2021

SparkQA commented Oct 20, 2021

SparkQA commented Oct 20, 2021

SparkQA commented Oct 20, 2021

zero323 Oct 20, 2021

Choose a reason for hiding this comment

SparkQA commented Oct 23, 2021

SparkQA commented Oct 23, 2021

SparkQA commented Oct 23, 2021

SparkQA commented Oct 23, 2021

SparkQA commented Oct 23, 2021

SparkQA commented Oct 23, 2021

SparkQA commented Oct 23, 2021

SparkQA commented Oct 23, 2021

SparkQA commented Oct 23, 2021

zero323 commented Oct 26, 2021

zero323 commented Oct 15, 2021 •

edited

Loading