Swift test suite doesn't honour LIT_FILTER_OUT environment variable

[hjyamauchi]

Description

https://ci-external.swift.org/job/swift-main-windows-toolchain/1155/

********************
Failed Tests (1):
  Swift(android-aarch64) :: ClangImporter/availability_custom_domains.swift

********************
Unexpectedly Passed Tests (3):
  Swift(android-aarch64) :: Driver/parseable_output.swift
  Swift(android-aarch64) :: Driver/parseable_output_unicode.swift
  Swift-validation(android-aarch64) :: compiler_crashers_fixed/28795-inprotocol-isrequirementsignaturecomputed-missing-signature.swift

Reproduction

swift-main-windows-toolchain

Expected behavior

No failures

Environment

swift-main-windows-toolchain

Additional information

No response

[hjyamauchi]

Based on the change history #80040 might be cause.

[tshortli]

I marked that test unsupported on Windows, but apparently the Windows CI is testing Android:

# .---command stderr------------
# | <unknown>:0: warning: libc not found for 'aarch64-unknown-linux-android'; C stdlib may be unavailable
# | Assertion failed: Val && "isa<> used on a null pointer", file C:\Users\swift-ci\jenkins\workspace\swift-main-windows-toolchain\llvm-project\llvm\include\llvm/Support/Casting.h, line 109
# | Please submit a bug report (https://swift.org/contributing/#reporting-bugs) and include the crash backtrace.
# | Stack dump:
# | 0.	Program arguments: t:\\5\\bin\\swift-frontend.exe -target aarch64-unknown-linux-android -sdk T:/android-ndk-r27c/toolchains/llvm/prebuilt/windows-x86_64/sysroot -resource-dir T:/aarch64-unknown-linux-android28/Runtime/./lib/swift -module-cache-path T:\\aarch64-unknown-linux-android28\\Runtime\\swift-test-results\\aarch64-unknown-linux-android\\clang-module-cache -swift-version 4 -define-availability "SwiftStdlib 9999:macOS 9999, iOS 9999, watchOS 9999, tvOS 9999" -define-availability "SwiftStdlib 5.0:macOS 10.14.4, iOS 12.2, watchOS 5.2, tvOS 12.2" -define-availability "SwiftStdlib 5.1:macOS 10.15, iOS 13.0, watchOS 6.0, tvOS 13.0" -define-availability "SwiftStdlib 5.2:macOS 10.15.4, iOS 13.4, watchOS 6.2, tvOS 13.4" -define-availability "SwiftStdlib 5.3:macOS 11.0, iOS 14.0, watchOS 7.0, tvOS 14.0" -define-availability "SwiftStdlib 5.4:macOS 11.3, iOS 14.5, watchOS 7.4, tvOS 14.5" -define-availability "SwiftStdlib 5.5:macOS 12.0, iOS 15.0, watchOS 8.0, tvOS 15.0" -define-availability "SwiftStdlib 5.6:macOS 12.3, iOS 15.4, watchOS 8.5, tvOS 15.4" -define-availability "SwiftStdlib 5.7:macOS 13.0, iOS 16.0, watchOS 9.0, tvOS 16.0" -define-availability "SwiftStdlib 5.8:macOS 13.3, iOS 16.4, watchOS 9.4, tvOS 16.4" -define-availability "SwiftStdlib 5.9:macOS 14.0, iOS 17.0, watchOS 10.0, tvOS 17.0" -define-availability "SwiftStdlib 5.10:macOS 14.4, iOS 17.4, watchOS 10.4, tvOS 17.4, visionOS 1.1" -define-availability "SwiftStdlib 6.0:macOS 15.0, iOS 18.0, watchOS 11.0, tvOS 18.0, visionOS 2.0" -define-availability "SwiftStdlib 6.1:macOS 9999, iOS 9999, watchOS 9999, tvOS 9999, visionOS 9999" -define-availability "SwiftStdlib 6.2:macOS 9999, iOS 9999, watchOS 9999, tvOS 9999, visionOS 9999" -typo-correction-limit 10 -enable-source-import -sdk C:\\\\Users\\\\swift-ci\\\\jenkins\\\\workspace\\\\swift-main-windows-toolchain\\\\swift\\\\test\\\\Inputs\\\\clang-importer-sdk -I C:\\\\Users\\\\swift-ci\\\\jenkins\\\\workspace\\\\swift-main-windows-toolchain\\\\swift\\\\test\\\\Inputs\\\\clang-importer-sdk\\\\swift-modules -typecheck -verify -import-objc-header C:\\Users\\swift-ci\\jenkins\\workspace\\swift-main-windows-toolchain\\swift\\test\\ClangImporter/Inputs/availability_domains_bridging_header.h -I C:\\Users\\swift-ci\\jenkins\\workspace\\swift-main-windows-toolchain\\swift\\test\\ClangImporter/Inputs/custom-modules/availability-domains -enable-experimental-feature CustomAvailability C:\\Users\\swift-ci\\jenkins\\workspace\\swift-main-windows-toolchain\\swift\\test\\ClangImporter\\availability_custom_domains.swift C:\\Users\\swift-ci\\jenkins\\workspace\\swift-main-windows-toolchain\\swift\\test\\ClangImporter/Inputs/availability_custom_domains_other.swift

[tshortli]

@hjyamauchi are you able to run the test and gather a backtrace I could use to understand the problem further? I don't have a way to reproduce Windows test failures and unfortunately the crash in the log from the bot is completely unsymbolicated so I have nothing to work with. I'll probably mark the test unsupported for Android, but it would be nice to see if I can reproduce the crash another way and fix it.

[tshortli]

Test disabled here #80060.

[hjyamauchi]

CC @weliveindetail

[hjyamauchi]

I think some Android tests are/were expected to fail, right? And the CI should pass now after disabling? I'd be happy to get a backtrace if it's still needed.

[tshortli]

If you can get a backtrace I'd appreciate it, but no rush.

[weliveindetail]

My PR this morning was supposed to fix that: #80053

[compnerd]

I do think that the revert makes sense - the test is expecting failure because it is testing something in the driver not the host that the generated code is for. I think that it makes sense to limit it to // REQUIRES: OS=windows-msvc || OS=apple-macosx || OS=linux-gnu and expect it to pass for all the targets.

[finagolfin]

The ClangImporter test only fails on Windows, not on the community Android CI, which runs on linux: PASS: Swift(android-aarch64) :: ClangImporter/availability_custom_domains.swift (441 of 18726).

@weliveindetail, you'll want to inform all your TBC people of the Windows-specific test blocklist you've added for your Android tests, so these tests aren't disabled for Android because of Windows-specific issues. You can check the results of the community Android CI to see if the failure is specific to Windows or not, and then inform Swift devs like Allan of where the problem is. Please revert #80060, as that is not necessary.

[weliveindetail]

@finagolfin Yes, I agree with both:

1. Android tests shouldn't be disabled on all build platforms, only because they fail on Windows. We might want to wait with the revert until the Windows bot is green again and until I have a bit more time to look into it.
2. The Windows-specific test blocklist is a bad solution, because it adds a parallel structure that people don't know about. Unfortunately, there was no other way to get the regression tests ready. The set of tests that fail for Windows and Linux is different and it's like not going away soon. There is no way to specify the HOST_OS in an XFAIL or UNSUPPORTED clause in LIT. It might be added, but it's a big change and won't be very intuitive either.

[weliveindetail]

For the moment, they should go back to green after #80148. Seems like https://ci-external.swift.org/job/swift-main-windows-toolchain/1162/ doesn't have this change, but the next one hopefully will 🙏

In the end, the build failed only due to unexpectedly passing tests. And these were intended to be skipped as per the filter:

Unexpectedly Passed Tests (3):
  Swift(android-aarch64) :: Driver/parseable_output.swift
  Swift(android-aarch64) :: Driver/parseable_output_unicode.swift
  Swift-validation(android-aarch64) :: compiler_crashers_fixed/28795-inprotocol-isrequirementsignaturecomputed-missing-signature.swift

So, I am afraid the remaining issue here is investigate why the Swift test suite doesn't honour the LIT_FILTER_OUT environment variable. We load it here and the same mechanism works well for LIT tests in LLDB:
https://github.com/weliveindetail/swift/blob/117b95498a70526f53fa9b5ba161c820d8bc306a/utils/build.ps1#L1679