[hotfix] Fixing flaky tests in flink-table-planner /hints

[mihirgune]

What is the purpose of the change

This fix is intended to fix flaky tests in the following classes -

• org.apache.flink.table.planner.hint.ClearJoinHintsWithCapitalizeQueryHintsShuttleTest
• org.apache.flink.table.planner.hint.ClearLookupJoinHintsWithInvalidPropagationShuttleTest

For example, when the following command is run on one of these tests using the Nondex plugin -
mvn -pl flink-table/flink-table-planner edu.illinois:nondex-maven-plugin:2.1.7:nondex -Dtest=org.apache.flink.table.planner.hint.ClearJoinHintsWithCapitalizeQueryHintsShuttleTest
This error is thrown.

[ERROR]   ClearJoinHintsWithCapitalizeQueryHintsShuttleTest.testClearCaseInsensitiveLookupHint:133->ClearQueryHintsWithInvalidPropagationShuttleTestBase.verifyRelPlan:142 afterPropagatingHints ==> expected: <
LogicalProject(a=[$0], hints=[[[ALIAS options:[t1]]]]), rowType=[RecordType(BIGINT a)]
+- LogicalCorrelate(correlation=[$cor0], joinType=[inner], requiredColumns=[{0, 1}], joinHints=[[[lookUp inheritPath:[0] options:{async=true, output-mode=allow_unordered, table=d, timeout=300 s, capacity=1000}]]], hints=[[[ALIAS inheritPath:[0] options:[t1]]]]), rowType=[RecordType(BIGINT a, TIMESTAMP_LTZ(3) *PROCTIME* pts, BIGINT a0)]
   :- LogicalProject(a=[$0], pts=[PROCTIME()]), rowType=[RecordType(BIGINT a, TIMESTAMP_LTZ(3) *PROCTIME* pts)]
   :  +- LogicalTableScan(table=[[builtin, default, src]]), rowType=[RecordType(BIGINT a)]
   +- LogicalFilter(condition=[=($cor0.a, $0)]), rowType=[RecordType(BIGINT a)]
      +- LogicalSnapshot(period=[PROCTIME()]), rowType=[RecordType(BIGINT a)]
         +- LogicalTableScan(table=[[builtin, default, lookup]]), rowType=[RecordType(BIGINT a)]
> but was: <
LogicalProject(a=[$0], hints=[[[ALIAS options:[t1]]]]), rowType=[RecordType(BIGINT a)]
+- LogicalCorrelate(correlation=[$cor0], joinType=[inner], requiredColumns=[{0, 1}], joinHints=[[[lookUp inheritPath:[0] options:{table=d, timeout=300 s, capacity=1000, output-mode=allow_unordered, async=true}]]], hints=[[[ALIAS inheritPath:[0] options:[t1]]]]), rowType=[RecordType(BIGINT a, TIMESTAMP_LTZ(3) *PROCTIME* pts, BIGINT a0)]
   :- LogicalProject(a=[$0], pts=[PROCTIME()]), rowType=[RecordType(BIGINT a, TIMESTAMP_LTZ(3) *PROCTIME* pts)]
   :  +- LogicalTableScan(table=[[builtin, default, src]]), rowType=[RecordType(BIGINT a)]
   +- LogicalFilter(condition=[=($cor0.a, $0)]), rowType=[RecordType(BIGINT a)]
      +- LogicalSnapshot(period=[PROCTIME()]), rowType=[RecordType(BIGINT a)]
         +- LogicalTableScan(table=[[builtin, default, lookup]]), rowType=[RecordType(BIGINT a)]

This flakiness presents itself because the test is doing a direct string comparison over String forms of objects. Since the hintOptions required for these tests are stored in a map (org.apache.flink.table.planner.plan.nodes.exec.spec.LookupJoinHintTestUtil#getLookupJoinHintOptions), their ordering is non-deterministic, and thus leads to these tests intermittently failing. Furthermore, the examples that were used to compare these tests with (in files flink-table/flink-table-planner/src/test/resources/org/apache/flink/table/planner/hint/ClearJoinHintsWithCapitalizeQueryHintsShuttleTest.xml and flink-table/flink-table-planner/src/test/resources/org/apache/flink/table/planner/hint/ClearJoinHintsWithCapitalizeQueryHintsShuttleTest.xml do not have consistent ordering for hintOptions.

The following tests have been fixed with this change -

• org.apache.flink.table.planner.hint.ClearJoinHintsWithCapitalizeQueryHintsShuttleTest#testClearCaseInsensitiveLookupHint
• org.apache.flink.table.planner.hint.ClearLookupJoinHintsWithInvalidPropagationShuttleTest#testNoNeedToClearLookupHint
• org.apache.flink.table.planner.hint.ClearLookupJoinHintsWithInvalidPropagationShuttleTest#testClearLookupHintWithInvalidPropagationToSubQuery
• org.apache.flink.table.planner.hint.ClearLookupJoinHintsWithInvalidPropagationShuttleTest#testNoNeedToClearLookupHintWhileJoinWithUnnest
• org.apache.flink.table.planner.hint.ClearLookupJoinHintsWithInvalidPropagationShuttleTest#testNoNeedToClearLookupHintWhileJoinWithUDTF

Brief change log

• To fix the flakiness in these tests, the hint options (org.apache.flink.table.planner.plan.nodes.exec.spec.LookupJoinHintTestUtil#getLookupJoinHintOptions) have been changed to use a LinkedHashMap instead of a HashMap, which will preserve insertion order.
• The test files flink-table/flink-table-planner/src/test/resources/org/apache/flink/table/planner/hint/ClearJoinHintsWithCapitalizeQueryHintsShuttleTest.xml and flink-table/flink-table-planner/src/test/resources/org/apache/flink/table/planner/hint/ClearLookupJoinHintsWithInvalidPropagationShuttleTest.xml have been amended to have a consistent ordering for hint options.

Verifying this change

Please make sure both new and modified tests in this PR follow the conventions for tests defined in our code quality guide.

This change added tests and can be verified as follows:

• Fixed test cases that relied on non-deterministic aspects of a data structure - the ordering within a map and made them reliable.

Does this pull request potentially affect one of the following parts:

• Dependencies (does it add or upgrade a dependency): (no)
• The public API, i.e., is any changed class annotated with @Public(Evolving): (no)
• The serializers: (no)
• The runtime per-record code paths (performance sensitive): (no)
• Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: (no)
• The S3 file system connector: (no)

Documentation

• Does this pull request introduce a new feature? (no)
• If yes, how is the feature documented? (not applicable)

[flinkbot]

CI report:

• a992bed Azure: SUCCESS

Bot commands

The @flinkbot bot supports the following commands:

• @flinkbot run azure re-run the last Azure build

[snuyanzin]

Is there a way to reproduce it without 3rd party plugins which are not part of dependencies?
I guess this 3rd party is a reason

This flakiness presents itself because the test is doing a direct string comparison over String forms of objects. Since the hintOptions required for these tests are stored in a map

in fact we use embedded jackson functionality which sorts properties in abc order in case of maps

[mihirgune]

Hi, thanks for the response!

If you want to execute NonDex without using it as a plugin, we also have steps to use it directly over command line locally - https://github.com/TestingResearchIllinois/NonDex?tab=readme-ov-file#use-command-line

in fact we use embedded jackson functionality which sorts properties in abc order in case of maps

Would it be possible for you to redirect me to where this implementation is located? I'll try to understand if it can be extended to cover these testcases. Alternatively, I'd also be happy to provide further insights into how NonDex introduces non-determinism to induce flakiness, if required.

Thanks