kedro-datasets: Investigate spark.SparkHiveDataset flaky unit tests

[ankatiyar]

Description

The tests for spark.SparkHiveDataset have been failing intermittently during nightly builds and checks for PRs. Usually re-running them solves the problem but we should investigate the root cause.

Context

Eg for a failed run - https://github.com/kedro-org/kedro-plugins/actions/runs/9807280152/job/27080744636
Seems to be related to the SparkSession setup.

[felipemonroy]

Hi @ankatiyar , I have opened a PR with a possible solution. Could you check it please?. I am not sure on how to re-run the actions to test it again.

[felipemonroy]

I am not sure how to fix it. I assume some tests use an unintended spark session (like the ones in test_spark_hive_dataset using one without hive support). Do we have the option of running tests/spark without xdist as the ones in tests/tensorflow/test_tensorflow_model_dataset.py?

[ankatiyar]

@felipemonroy we can investigate this at our end, if in the meantime your tests are failing on other PRs with this error, tag someone from the team (or me) we can re-run the tests for the PR. Locally, you can run make test-no-spark I think to run all the other unit tests