Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Implement soft assertions #2750

Open
wants to merge 3 commits into
base: main
Choose a base branch
from
Open

Implement soft assertions #2750

wants to merge 3 commits into from

Conversation

mbroton
Copy link

@mbroton mbroton commented Feb 18, 2025

Related to #1272

This PR adds support for soft assertions, allowing multiple assertion failures to be collected before raising them together. I implemented this using a context manager pattern since it feels more natural and Pythonic:

with expect.soft() as soft_expect:
    await soft_expect(page).to_have_title("wrong title")
    await soft_expect(page.locator("#div1")).to_have_text("wrong")
    response = await page.request.get("...")
    await soft_expect(response).to_be_ok()

Example error report:

>       with expect.soft() as soft_expect:
E       AssertionError: Soft assertion failures
E       1. Page title expected to be 'wrong title'
E       Actual value: Page Title
E       Call log:
E         - PageAssertions.to_have_title with timeout 5000ms
E         -   - waiting for locator(":root")
E         -     9 × locator resolved to <html>…</html>
E         -       - unexpected value "Page Title"
E
E       2. Locator expected to have text 'wrong'
E       Actual value: Text1
E       Call log:
E         - LocatorAssertions.to_have_text with timeout 5000ms
E         -   - waiting for locator("#div1")
E         -     9 × locator resolved to <div id="div1">Text1</div>
E         -       - unexpected value "Text1"
E
E       3. Response status expected to be within [200..299] range, was '404'
E       Call log:
E       → GET http://localhost:2137/non-existent
E         -   user-agent: (...)
E         -   accept: */*
E         -   accept-encoding: gzip,deflate,br
E         - ← 404 Not Found
E         -   content-type: text/plain
E         -   transfer-encoding: chunked

Implementation Details

  • The implementation maintains backward compatibility with all existing assertion methods
  • Both sync and async APIs are fully supported with identical behavior
  • Error messages preserve the order of failures and include clear numbering (1., 2., etc.)
  • Custom error messages are properly handled and included in the final error report
  • The implementation handles nested errors correctly (when other exceptions occur within the soft assertion block)
  • Timeout settings are respected both at the global and individual assertion level

Initially, I used contextlib.contextmanager for expect.soft() as it was a simpler implementation. However, this approach included some unrelevant information in the traceback when assertions failed. To address this, I created a custom SoftAssertionContextManager class that gives us more control over the error handling and traceback presentation.

This is just one way to add soft assertions in Python Playwright. I think the context manager pattern works well with Python, but I’m totally open to other ideas if they fit the project better.

@mbroton
Copy link
Author

mbroton commented Feb 18, 2025

@microsoft-github-policy-service agree

@mbroton
Copy link
Author

mbroton commented Mar 5, 2025

I reviewed the failing checks and also tried to reproduce some of them locally without success. I couldn't find any indication that these failures are connected to my changes, so it seems they might be flaky or not related? Please let me know if you think further investigation is needed on my end.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
1 participant
@mbroton