Skip to content

Commit 630dfdf

Browse files
committed
[SPARK-28101][DSTREAM][TEST] Fix Flaky Test: InputStreamsSuite.Modified files are correctly detected in JDK9+
## What changes were proposed in this pull request? It seems that https://bugs.openjdk.java.net/browse/JDK-8068730 makes `InputStreamsSuite` very flaky. <img width="903" alt="error" src="https://user-images.githubusercontent.com/9700541/59727067-017eb780-91e9-11e9-8bb0-ac5f4c1bc44d.png"> As we can see the Jenkins result, this can be reproduced frequently with JDK9+. ``` $ build/sbt "streaming/testOnly *.InputStreamsSuite" [info] - Modified files are correctly detected. *** FAILED *** (134 milliseconds) [info] Set("renamed") did not equal Set() (InputStreamsSuite.scala:312) [info] org.scalatest.exceptions.TestFailedException: ``` The reason is the `renamed.txt`'s modification time becomes greater than the clock in JDK9+ and Spark ignored it with **not selected** message. In JDK8, the modification time generated by this test case doesn't have `milliseconds` part. ``` Getting new files for time 1560896662000, ignoring files older than 1560896659679 file:/.../streaming/subdir/renamed.txt not selected as mod time 1560896662679 > current time 1560896662000 file:/.../streaming/subdir/existing ignored as mod time 1560896657679 <= ignore time 1560896659679 Finding new files took 0 ms New files at time 1560896662000 ms: ``` ## How was this patch tested? Pass the Jenkins and manually repeat the following with JDK11 10 times. ``` $ build/sbt "streaming/testOnly *.InputStreamsSuite" ``` Closes apache#24904 from dongjoon-hyun/SPARK-28101. Authored-by: Dongjoon Hyun <[email protected]> Signed-off-by: Dongjoon Hyun <[email protected]>
1 parent 36b327d commit 630dfdf

File tree

1 file changed

+1
-2
lines changed

1 file changed

+1
-2
lines changed

streaming/src/test/scala/org/apache/spark/streaming/InputStreamsSuite.scala

+1-2
Original file line numberDiff line numberDiff line change
@@ -293,8 +293,7 @@ class InputStreamsSuite extends TestSuiteBase with BeforeAndAfter {
293293
val textPath = new Path(generatedSubDir, "renamed.txt")
294294
write(textPath, "renamed\n")
295295
val now = clock.getTimeMillis()
296-
val modTime = now + durationMs / 2
297-
fs.setTimes(textPath, modTime, modTime)
296+
fs.setTimes(textPath, now, now)
298297
val textFilestatus = fs.getFileStatus(existingFile)
299298
assert(textFilestatus.getModificationTime < now + durationMs)
300299

0 commit comments

Comments
 (0)