Skip to content

Charlie Freestone Blog Post - Some Best Practices for Accessible Automation Testing #317

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 7 commits into
base: gh-pages
Choose a base branch
from

Conversation

cfreestone-ScottLogic
Copy link

@cfreestone-ScottLogic cfreestone-ScottLogic commented May 14, 2025

Please add a direct link to your post here:

https://cfreestone-scottlogic.github.io/blog/2025/05/14/some-best-practices-for-accessible-automation.html

Have you (please tick each box to show completion):

  • Added your blog post to a single category?
  • Added a brief summary for your post? Summaries should be roughly two sentences in length and give potential readers a good idea of the contents of your post.
  • Checked that the build passes?
  • Checked your spelling (you can use npm install followed by npx mdspell "**/{FILE_NAME}.md" --en-gb -a -n -x -t if that's your thing)
  • Ensured that your author profile contains a profile image, and a brief description of yourself? (make it more interesting than just your job title!)
  • Optimised any images in your post? They should be less than 100KBytes as a general guide.

Posts are reviewed / approved by your Regional Tech Lead.

Copy link
Member

@chriswilty chriswilty left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Overall, plenty of good advice.

I was surprised that an article about testing with "Accessible" in the title only lightly touched on accessibility concerns - using roles to locate elements. I feel this is somewhat misleading, as the rest of the article concerns your thoughts on naming and tagging tests to make them more informative, plus the part which I think is about not worrying too much about specificity, not testing in too much detail the things which are not all that important, none of which is about accessibility.

I would advise changing the title to reflect this.

While on my most recent project I had the unique experience of working closely with many testers and test minded individuals. This allowed me to learn some much-needed lessons about how to best implement automation testing with accessibility in mind, a sometimes-overlooked area of test automation.

## Introduction
What I hope to share with you is how simple it can be to both think about and implement the approach to make sure your tests are accessible. This can be because you have strict reasons on the project to ensure compliance with certain metrics, because this can lead to improved readability of your code and its longevity, or because you want to read some wonderful thoughts from certain Testers perspective.
Copy link
Member

@chriswilty chriswilty May 19, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

certain Testers perspective

Not sure what you are trying to say... Do you mean "from a certain tester's perspective", i.e. you? Or did you mean plural "testers"?

I think this introductory paragraph highlights my main question about your article: what does it mean for a test to be accessible, and what do metrics or code readability have to do with that? Are you concerned about making your test code accessible to testers who need to use a screen reader? If so, I think you need to make that clear, and give more examples of how you can achieve that.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Certain testers perspective was meaning myself.

What I wanted to share was skills I learnt which anyone can write in their tests which is in line which accessibility testing. Its not aimed at testers who need a screen reader. Its lessons I have learnt from doing Accessibility testing, which can make your automation tests more accessible.

Copy link
Member

@chriswilty chriswilty May 20, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Certain testers perspective was meaning myself.

Then you want "from a certain tester's perspective" (need apostrophe)

Accessibility testing is about testing that a UI is accessible to all users, whether they are keyboard-only users, have vision or motor impairments, dyslexia, epilepsy, etc. The only part of this article which is about accessibility is locating elements by role. You are not really making your automation tests more accessible, you are making them more readable.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

By the way, how did you do Accessibility testing, what tool(s) did you use? That would also make an interesting blog post.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Made the adjustment to add the apostrophes.

The application of these methods was what I picked up while learning about accessibility testing on the project. Readability does allign itself into accessibility. If something is more readable it is inherantly more accessable. The lessons I learnt about accessibility testing I have led to creating tests which are more accessible to others, readability is one of those areas.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

By the way, how did you do Accessibility testing, what tool(s) did you use? That would also make an interesting blog post.

I shall keep this in mind for a future blog

Copy link
Member

@chriswilty chriswilty Jun 6, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Made the adjustment to add the apostrophes.

Ah no, you've added quotation marks around the phrase. I mean this:

from a certain tester's perspective

I still take issue with your use of the word accessible here. You need to be really careful with that word, because accessibility testing (how accessible is your UI) is a totally different ballgame to writing tests in a way that makes them easier for other people to read, modify and maintain - that is not accessibility testing.

Readability or understandability is one very small part of making test code accessible, and has little to do with the UI itself being accessible. Did you use the axe plugin for playwright? https://playwright.dev/docs/accessibility-testing

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok if you still think it needs an overhaul then just tell me what you would like me to change so its no longer an issue please.

Copy link
Author

@cfreestone-ScottLogic cfreestone-ScottLogic Jun 11, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The legitmate reason for myself making this blog post was by myself doing some Accessibility Testing and then through that I learnt some best practicies to implement into my automation tests.


Things like filters and drop-downs can be difficult to locate but they can be found. A good tip for finding this is by entering the webpage and open up DevTools, then clicking near the item you wish to find the selector of, but not on it. After that press tab and open up the console within DevTools. If you then type the command: `document.activeElement` you will find the info you need.

By the same accessible reasoning it is good practice to not use `.getbyTestID`.This is due to the fact that names of those same ID’s will not be useful to someone with an accessibility issue, as it will sound like Jargon when navigated to.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't believe screen readers will announce data-testid attributes, so I don't know what you mean here.


This is because there is an interface that runs behind the scenes of the UI and Playwright can pick selectors which will not be seen by someone who needs to use the keyboard to navigate around the system. For these users, the elements they have access to are either tabbed to or navigated to via arrow keys. If using a screen reader these selectors need to have a consistent and readable naming convention. Otherwise, any software that reads it aloud would read out jargon which will only confuse the User.

Things like filters and drop-downs can be difficult to locate but they can be found. A good tip for finding this is by entering the webpage and open up DevTools, then clicking near the item you wish to find the selector of, but not on it. After that press tab and open up the console within DevTools. If you then type the command: `document.activeElement` you will find the info you need.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You will not find the information you need if the page has not been implemented using semantic html or aria attributes.

## Don’t just getBy, Role with it
After correctly measuring our expectations, we want to roll into selecting all the necessary parameters we can get by with. Therefore, it comes as no surprise we use the `.getByRole function`, but in order for us to make sense of it then it's all about those pesky selectors.

This is because there is an interface that runs behind the scenes of the UI and Playwright can pick selectors which will not be seen by someone who needs to use the keyboard to navigate around the system. For these users, the elements they have access to are either tabbed to or navigated to via arrow keys. If using a screen reader these selectors need to have a consistent and readable naming convention. Otherwise, any software that reads it aloud would read out jargon which will only confuse the User.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What is this consistent and readable naming convention that you mention here? You do not elaborate on it. Are you thinking about semantic html? Ensuring elements have an accessible name?

Element locators are only tangentially related to what a screen reader will announce, and both are fully dependent on the underlying HTML.


This is because there is an interface that runs behind the scenes of the UI and Playwright can pick selectors which will not be seen by someone who needs to use the keyboard to navigate around the system. For these users, the elements they have access to are either tabbed to or navigated to via arrow keys. If using a screen reader these selectors need to have a consistent and readable naming convention. Otherwise, any software that reads it aloud would read out jargon which will only confuse the User.

Things like filters and drop-downs can be difficult to locate but they can be found. A good tip for finding this is by entering the webpage and open up DevTools, then clicking near the item you wish to find the selector of, but not on it. After that press tab and open up the console within DevTools. If you then type the command: `document.activeElement` you will find the info you need.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why are filters and dropdowns difficult to locate?

If you want to go the extra mile it's even better to use annotations for ticket numbers. That way tags can be used exclusively to filter what tests are run, filtering test results. This is what is documented in the Playwright documentatio.

## Conclusion
Well, there have been some odd analogies and puns along the way but hopefully this peek into a tester’s mind has helped you understand the importance of, and some ways to implement some good accessible skills in your automation tests while also keeping it simple to implement. Maybe there are even more simple techniques out there so go out and find them, or better yet bring them up with others and let's get everything even more accessible!
Copy link
Member

@chriswilty chriswilty Jun 16, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As I keep trying to point out, the only "accessible skills" you elaborate on in this post is locating elements by role, which is one tiny part of accessibility. None of the remaining tips are related to accessibility testing, which is my primary point of contention in this post.

While on my most recent project I had the unique experience of working closely with many testers and test minded individuals. This allowed me to learn some much-needed lessons about how to best implement automation testing with accessibility in mind, a sometimes-overlooked area of test automation.

## Introduction
What I hope to share with you is how simple it can be to both think about and implement the approach to make sure your tests are accessible. This can be because you have strict reasons on the project to ensure compliance with certain metrics, because this can lead to improved readability of your code and its longevity, or because you want to read some wonderful thoughts from certain testers perspective.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Typo

from a certain tester's perspective.

I am not sure what you mean by tests being accessible, as that is not the same as testing the UI is accessible.


This neat little function is the bread-and-butter basics of automating in playwright, it's what we want to see when we place our test into a certain configuration. This doesn’t mean it has to do handstands or backflips, but just like any gymnastics routine we do wait on it with bated breath, hoping for results we will cheer for. This anticipation makes it easy for us to write something which makes us wait for that big finale in our tests, even if the tests took the literal definition of breaking a leg.

And therein lies the issue; during a test we usually assert for confirmation to ensure we are where we expect. That means we can `expect(response.status()).toBe(200)` which is a wonderful successful request that has found the requested resource of a page. Instead we get a `201 response`, which is just as equally a successful request but it led to the creation of a resource. Test results can’t always be perfect and that means that this stopping point during the test, before we check the larger critical parameters later on, will skip the test. Instead, we have to accept that not every response will be exciting and just `expect(response.status()).toBe(OK)` which allows us to still capture data from a test even if the status code is different.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This fix appears to have slipped through the net.

expect(response.status()).toBeOK()

https://playwright.dev/docs/api/class-apiresponseassertions

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants