Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Promote exit tests to API #324

Open
wants to merge 11 commits into
base: main
Choose a base branch
from
Open

Promote exit tests to API #324

wants to merge 11 commits into from
+283 −174

Conversation

grynspan
Copy link
Contributor

@grynspan grynspan commented Apr 2, 2024 •
edited
Loading

This PR promotes exit tests to API, pending approval of the proposal at swiftlang/swift-evolution#2718.

View the full proposal here.

Checklist:

  • Code and documentation should follow the style of the Style Guide.
  • If public symbols are renamed or modified, DocC references should be updated.

@grynspan grynspan added tools integration Integration of swift-testing into tools/IDEs windows 🪟 Windows support linux 🐧 Linux support (all distros) darwin 🍎 macOS, iOS, watchOS, tvOS, and visionOS support public-api Affects public API swift-6 labels Apr 2, 2024
@grynspan grynspan self-assigned this Apr 2, 2024
@grynspan grynspan force-pushed the jgrynspan/exit-tests-proposal branch from fe21f14 to 988ddfe Compare April 2, 2024 17:57
@grynspan grynspan added the enhancement New feature or request label Apr 2, 2024
@grynspan grynspan marked this pull request as draft April 2, 2024 22:10
@grynspan grynspan force-pushed the jgrynspan/exit-tests-proposal branch from 988ddfe to 697123a Compare April 10, 2024 17:40
@grynspan grynspan changed the title [SWT-0001] Exit tests [SWT-NNNN] Exit tests Apr 10, 2024
@grynspan grynspan force-pushed the jgrynspan/exit-tests-proposal branch 3 times, most recently from 19f37c3 to 4825821 Compare April 12, 2024 13:30
@grynspan grynspan force-pushed the jgrynspan/exit-tests-proposal branch 2 times, most recently from feb425b to e84b014 Compare April 30, 2024 14:23
@grynspan grynspan added the api-proposal API proposal PRs (documentation only) label May 1, 2024
@grynspan grynspan force-pushed the jgrynspan/exit-tests-proposal branch 2 times, most recently from 6f0e7f0 to 875b870 Compare May 16, 2024 14:41
@grynspan grynspan force-pushed the jgrynspan/exit-tests-proposal branch from 875b870 to 4365611 Compare May 24, 2024 20:47
@grynspan grynspan force-pushed the jgrynspan/exit-tests-proposal branch from 4365611 to 7d25e07 Compare July 12, 2024 15:43
@grynspan
Copy link
Contributor Author

Rebased.

@grynspan grynspan removed the swift-6 label Jul 25, 2024
@grynspan
Copy link
Contributor Author

@swift-ci test

@grynspan grynspan changed the title [SWT-NNNN] Exit tests Promote exit tests to API Mar 3, 2025
@grynspan grynspan removed the api-proposal API proposal PRs (documentation only) label Mar 3, 2025
@grynspan grynspan force-pushed the jgrynspan/exit-tests-proposal branch from 417cb17 to 5705163 Compare March 3, 2025 21:32
@grynspan
Copy link
Contributor Author

grynspan commented Mar 3, 2025

@swift-ci test

1 similar comment
@grynspan
Copy link
Contributor Author

grynspan commented Mar 3, 2025

@swift-ci test

@grynspan grynspan force-pushed the jgrynspan/exit-tests-proposal branch from ecda40c to 59ec9f5 Compare March 20, 2025 22:27
@grynspan
Copy link
Contributor Author

@swift-ci test

@grynspan grynspan added the api-proposal API proposal PRs (documentation only) label Mar 20, 2025
grynspan added 4 commits March 24, 2025 15:25
@grynspan
[SWT-NNNN] Exit tests
aa0b32f 
One of the first enhancement requests we received for swift-testing was the
ability to test for precondition failures and other critical failures that
terminate the current process when they occur. This feature is also frequently
requested for XCTest. With swift-testing, we have the opportunity to build such
a feature in an ergonomic way.

Read the full proposal [here](https://github.com/apple/swift-testing/blob/jgrynspan/exit-tests-proposal/Documentation/Proposals/NNNN-exit-tests.md).
@grynspan
Remove proposal doc (moving to SE)
ddfe351
@grynspan
Try to work around compiler crash
c947c4b
@grynspan
Create an Exit Testing DocC article
4c0d958
@grynspan grynspan force-pushed the jgrynspan/exit-tests-proposal branch from 59ec9f5 to 4c0d958 Compare March 24, 2025 19:54
@grynspan
Copy link
Contributor Author

@swift-ci test

@grynspan
Copy link
Contributor Author

@swift-ci test

@grynspan
Copy link
Contributor Author

@swift-ci test

@grynspan
Copy link
Contributor Author

@swift-ci test

@grynspan
Copy link
Contributor Author

@swift-ci test

iamleeg
iamleeg requested changes Mar 26, 2025
View reviewed changes
Copy link
Contributor

@iamleeg iamleeg left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for writing this article, it's really helpful for developers!

Sources/Testing/ExitTests/ExitTest.Condition.swift
/// See https://en.cppreference.com/w/c/program/EXIT_status for more
/// information about exit codes defined by the C standard.
/// }
///
/// On macOS, FreeBSD, OpenBSD, and Windows, the full exit code reported by
/// the process is yielded to the parent process. Linux and other POSIX-like
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is "yielded" the correct word here? I associate it with coroutines, but this documentation is about a function. "…the function returns the full value of the process's exit code."

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

"Reported" may be better. The mechanism that gets this value into the parent process involves the kernel, so the terminology is more vague than usual.

Sources/Testing/ExitTests/ExitTest.Result.swift
public struct Result: Sendable {
/// The exit condition the exit test exited with.
/// The status of the process hosting the exit test at the time it exits.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How about "The exit status reported by the process hosting the exit test"? Just trying to reduce the number of clauses here.

Sources/Testing/Testing.docc/exit-testing.md

### Constraints on exit tests

#### State cannot be captured
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
#### State cannot be captured
#### Don't capture state in the test body

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This isn't an instruction. It's physically impossible to do it. There is no "do".

Sources/Testing/Testing.docc/exit-testing.md

#### State cannot be captured

Exit tests cannot capture any state originating in the parent process or from
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
Exit tests cannot capture any state originating in the parent process or from
Exit tests can't capture any state originating in the parent process or from

Sources/Testing/Testing.docc/exit-testing.md
#### State cannot be captured

Exit tests cannot capture any state originating in the parent process or from
the enclosing lexical context. For example, the following exit test will fail to
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
the enclosing lexical context. For example, the following exit test will fail to
the enclosing lexical context. For example, the following exit test fails to

Sources/Testing/Testing.docc/exit-testing.md
}
```

#### Exit tests cannot be nested
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
#### Exit tests cannot be nested
#### Don't nest exit tests

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This isn't an instruction. It's physically impossible to do it. There is no "do".

Sources/Testing/Testing.docc/exit-testing.md

#### Exit tests cannot be nested

An exit test cannot run within another exit test.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
An exit test cannot run within another exit test.
An exit test can't run another exit test.

It means the same to say "a test can't run a child test" as "a test can't run in a parent test", and it's simpler.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@amartini51 amartini51 amartini51 left review comments

@iamleeg iamleeg iamleeg requested changes

@dennisweissmann dennisweissmann Awaiting requested review from dennisweissmann

@briancroom briancroom Awaiting requested review from briancroom briancroom is a code owner

@SeanROlszewski SeanROlszewski Awaiting requested review from SeanROlszewski SeanROlszewski is a code owner

@suzannaratcliff suzannaratcliff Awaiting requested review from suzannaratcliff suzannaratcliff is a code owner

@stmontgomery stmontgomery Awaiting requested review from stmontgomery stmontgomery is a code owner

Assignees

@grynspan grynspan

Labels
api-proposal API proposal PRs (documentation only) darwin 🍎 macOS, iOS, watchOS, tvOS, and visionOS support enhancement New feature or request exit-tests ☠️ Work related to exit tests freebsd 😈 FreeBSD support linux 🐧 Linux support (all distros) openbsd 🐡 OpenBSD support public-api Affects public API tools integration Integration of swift-testing into tools/IDEs windows 🪟 Windows support
Projects
None yet
Milestone
Swift 6.2
Development

Successfully merging this pull request may close these issues.
6 participants
@grynspan @dabrahams @maartene @stmontgomery @iamleeg @amartini51