Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

'appium:nativeWebTap':true only works for the first click #2510

Open
craitose opened this issue Jan 16, 2025 · 20 comments
Open

'appium:nativeWebTap':true only works for the first click #2510

craitose opened this issue Jan 16, 2025 · 20 comments

Comments

@craitose
Copy link

craitose commented Jan 16, 2025

version: 2.15.0
target environment: iOS
app: react
context: webView
devices: real mobiles
issue: calling the click() command on an element that is visible & clickable does not trigger the click event and does not throw any errors in the logs.

Having issues clicking on elements when using XCUITest in the capabilities, it was suggested to use the flag 'appium:nativeWebTap' and that worked for the 1st element but once the test moved to the next element the issue returned.

I Understand that coordinates calculated may be incorrect and that is the reason for the click event not being triggered but as there is no way to see the location where the click occurred - I cannot know for sure if the click happened.

A good way to troubleshoot would be to have a flag that highlights the element that was clicked, I have tried to write a script in the onCommand tag but that does not work:

`onCommand: async function (commandName, args) {
if (commandName === 'elementClick') { // also tried 'onClick' , 'click' , 'tap' , 'Click Element'
try {
const browser = global.broswer;
const element = await browser.$(args[0]);

			//highlight the element
			await browser.execute((el) => {
				el.style.border = '9px solid green';
				el.style.transition = 'border 0.3s';
			}, element);

			//pause to make the highlight visible
			await browser.pause(500);

			//remove highlight after click
			await browser.execute((el) => {
				el.style.border = '';
			}, element);
		} catch (error) {
			console.error('Error during highlighting', error.message);
		}
	}`

Since I am only interested in the safari browser , it would be good to utilise the appium-safari-driver so that we could use it on a windows machine.

Any suggestions would be welcome.

@KazuCocoa
Copy link
Member

Do you have the full appium server log? I think it has coordinate-related logs to send a click with coordinate action.

@craitose
Copy link
Author

I do but it does not show any error and I have no coordinates to compare with those in the logs

@KazuCocoa
Copy link
Member

Then, it could mean NATIVE_APP level tap action (nativeWebTap does the tap action in the NATIVE_APP context) succeeded, but the coordinate was wrong or Safari/browser content didn't get any events because of iOS internal something.

Browser's tab bar affects the coordinate calculation, for example. nativeWebTapTabBarVisibility in https://appium.github.io/appium-xcuitest-driver/latest/reference/settings/ explains such a case.

Usually, coordinate info is in the appium server log such as

this.log.debug(`Converted coordinates: ${JSON.stringify(newCoords)}`);

I thought such logs were in your appium server log as well.

@craitose
Copy link
Author

Thanks @KazuCocoa I missed this flag,
I will give 'appium:nativeWebTapTabBarVisibility':set

a try and get back to you.

@craitose
Copy link
Author

craitose commented Jan 20, 2025

@KazuCocoa What are the acceptable values of 'appium: nativeWebTapTabBarVisibility'?
I have tried 'DETECT','VISIBLE','SET' but it had no effect.

@craitose
Copy link
Author

craitose commented Jan 20, 2025

I have extracted this snippet from the log and what is interesting is
1.the 6th line 'Additional native web tap offset computed: 0', even though the tabBar was detected.
2.the 2nd to-last line 'Responding to client with driver.click() result:null'

2025-01-20 10:42:14:308 - [XCUITestDriver@926c] tabBarPosition and topOffset: top, 95
2025-01-20 10:42:14:308 - [XCUITestDriver@926c] Waiting up to 0 ms for condition
2025-01-20 10:42:14:310 - [XCUITestDriver@926c] Matched '/elements' to command name 'findElements'
2025-01-20 10:42:14:310 - [XCUITestDriver@926c] Proxying [POST /elements] to [POST http://localhost:8100/session/EADBC945-30F4-4DC5-986F-D338888EA79F/elements] with body: {"using":"accessibility id","value":"Close app download offer"}
2025-01-20 10:42:14:549 - [XCUITestDriver@926c] Got response with status 200: {"value":[],"sessionId":"EADBC945-30F4-4DC5-986F-D338888EA79F"}
2025-01-20 10:42:14:549 - [XCUITestDriver@926c] Additional native web tap offset computed: 0
2025-01-20 10:42:14:549 - [XCUITestDriver@926c] Set implicit wait to 0ms
2025-01-20 10:42:14:549 - [XCUITestDriver@926c] Converted coordinates: {"x":365,"y":703}
2025-01-20 10:42:14:549 - [XCUITestDriver@926c] rect: {"y":0,"x":0,"width":390,"height":844}
2025-01-20 10:42:14:549 - [XCUITestDriver@926c] wvPos: {"x":0,"y":95}
2025-01-20 10:42:14:550 - [XCUITestDriver@926c] realDims: {"w":390,"h":700}
2025-01-20 10:42:14:550 - [XCUITestDriver@926c] wvDims: {"w":390,"h":664}
2025-01-20 10:42:14:550 - [XCUITestDriver@926c] xRatio: 1
2025-01-20 10:42:14:550 - [XCUITestDriver@926c] yRatio: 1.0542168674698795
2025-01-20 10:42:14:550 - [XCUITestDriver@926c] Converted web coords {"x":365.34375,"y":577.05908203125} into real coords {"x":365,"y":703}
2025-01-20 10:42:14:555 - [XCUITestDriver@926c] Proxying to WDA with an unknown route: POST /wda/tap/0
2025-01-20 10:42:14:556 - [XCUITestDriver@926c] Proxying [POST /wda/tap/0] to [POST http://localhost:8100/session/EADBC945-30F4-4DC5-986F-D338888EA79F/wda/tap/0] with body: {"x":365,"y":703}
2025-01-20 10:42:14:948 - [XCUITestDriver@926c] Got response with status 200: {"value":null,"sessionId":"EADBC945-30F4-4DC5-986F-D338888EA79F"}
2025-01-20 10:42:14:949 - [XCUITestDriver@926c] Responding to client with driver.click() result: null
2025-01-20 10:42:14:949 - [HTTP] <-- POST /session/ec3b69bc-5437-4bf3-bbb0-d5304d595610/element/:wdc:1737369718116/click 200 959 ms - 14

@KazuCocoa
Copy link
Member

Oh, sorry. My initial intention was safariTabBarPosition, not nativeWebTapTabBarVisibility. nativeWebTapTabBarVisibility is for tab bar/smart app banner related in iOS term. So they might not help.

Converted web coords {"x":365.34375,"y":577.05908203125} into real coords {"x":365,"y":703}

This means the element was {"x":365.34375,"y":577.05908203125} in the Safari/WebView context. It was {"x":365,"y":703} as the native app context (native app context means as the entire screen, including non-Safari/WebView components such as tab bar etc)

If {"x":365,"y":703} was expected and no explicit error response came from Appium/WDA, tapping the coordinate as the iOS system might work with no issues. I would check the Web app side to see if an expected touch event occurred. This is not via appium. You would need to work with the developer to add event logger etc.

@craitose
Copy link
Author

So safariTabBarPosition won't help either? Asking the Devs to add event logging is a long shot.

Is there a way to highlight elements being clicked?

@craitose
Copy link
Author

craitose commented Jan 21, 2025

Here is a picture of the screen and it seems that there a 2 bars [top being the Tab Bar] and the other [being nav bar].
Does the coordinate calculation cater for the nav bar ?

Image

@KazuCocoa
Copy link
Member

2025-01-20 10:42:14:308 - [XCUITestDriver@926c] tabBarPosition and topOffset: top, 95

https://github.com/appium/appium-xcuitest-driver/blob/08418ed9bd49ff159728552b703985c9f41490f0/lib/commands/web.js#L575C23-L575C51

@craitose
Copy link
Author

craitose commented Jan 21, 2025

This is the element :

<div class="MenuBtn_lobbymenu_btn__8CDOr" style="display: block;"> </div>

could it be because this element does not have the button tag that the driver does not click/tap it?

I pointed the click command to another element on the same page that had the <button> tag and it was clicked just fine.

@KazuCocoa
Copy link
Member

I don't know since I have little knowledge of Web UI. It is how the Web App handles events rather than if Appium/WDA clicks a coordinate via XCTest framework (NATIVE_APP).

@craitose
Copy link
Author

As a work around , can you think of a way to highlight the position that is being clicked?

@craitose
Copy link
Author

craitose commented Jan 23, 2025

I don't think my webapp is the issue, there is definitely something wrong with xcuitest - here is a generic use case to try:

using w3school site to click on a paragraph that has a click event to change colour to red.

On an android device the paragraph is clicked but not on the iOS device.

    it('should make text red by clicking on it',async function() {
        await browser.pause(2000);

        const AcceptAll = $('#accept-choices');
        const TextChange = $('p=Click me to change my text color.');

        console.log('navigating to url');
        await browser.url('https://www.w3schools.com/tags/tryit.asp?filename=tryhtml5_ev_onclick2');
        console.log('done navigating to url');
        await AcceptAll.waitForDisplayed();
        await AcceptAll.click();
        await browser.switchFrame($('iframe'));
        await TextChange.waitForDisplayed();
        await TextChange.click();
        await browser.pause(2000);
        await browser.switchFrame(null);
}

@KazuCocoa
Copy link
Member

If the coordinate in Appium XCUITest driver was correct in the log but nothing happened, it could be XCTest framework internal or Safari engine related. If so, nothing could be done on the appium side, unfortunately. As Appium, we can tune up the coordination calculation only in this case.

Android is via chromedriver. They work in different mechanisms. In terms of this, 'appium:nativeWebTap':false could behave similarly in Appium XCUITest driver.

@craitose
Copy link
Author

craitose commented Jan 24, 2025

I understand please try to tune up the calculation , in the mean-time in order to spot if the correct element was being targeted - I have found a way to highlight the element using the below JS:

import { browser } from '@wdio/globals'

export async function highlight() {
    await browser.addCommand('highlight', async function() {
    await this.waitForExist();
    await browser.execute((el) => {
        el.style.border = '5px solid #00ff22';
    }, await this);
    await browser.pause(1000);
    await browser.execute((el) => {
        el.style.border = 'none'
    }, await this);
    await browser.pause(500);
}, true);
}

If you could add a flag that does this on the click coordinates, it could help with troubleshooting the clicks

@craitose
Copy link
Author

Update I have the iOS log and it seems to show the tap was at the xcuitest calculated coordinates, please have a look to see if I missed an error:

iOSLog.txt

@KazuCocoa
Copy link
Member

KazuCocoa commented Jan 24, 2025

Perhaps these are relevant logs about such a tap action. As they have Single tap identified logs in MobileSafari, probably the action itself identified by the system (and Safari/WebKit engine)

Jan 21 12:21:05 testmanagerd(FrontBoardServices)[268] <Notice>: [FBSDisplaySource 4-4896] did invalidate
Jan 21 12:21:05 testmanagerd[268] <Notice>: normalized {195, 398} to {0.5, 0.47156398104265401}
Jan 21 12:21:05 testmanagerd[268] <Notice>: Waiting up to 5.0s for TouchEventsCompleted to confirm 
<XCSynthesizedEventRecord 'tap display 1'>
Path 1:
	Touch down at 195.0, 398.0, offset=0.00s
	Touch up at 195.0, 398.0, offset=0.05s.
Jan 21 12:21:05 MobileSafari(WebKit)[454] <Notice>: Single tap identified. Request details on potential zoom. (0x10585b000)
Jan 21 12:21:05 testmanagerd[268] <Notice>: Handling kAXUserTestingNotification from AX element pid: 62, elementOrHash.elementID: 0.1 (0x1056243a0)
Jan 21 12:21:05 testmanagerd[268] <Notice>: Got user testing notification with payload <private>
Jan 21 12:21:05 testmanagerd[268] <Notice>: Calling handler for AX notification kAXUserTestingNotification from AX element pid: 62, elementOrHash.elementID: 0.1 with payload <private>
Jan 21 12:21:05 testmanagerd[268] <Notice>: Event 
<XCSynthesizedEventRecord 'tap display 1'>
Path 1:
	Touch down at 195.0, 398.0, offset=0.00s
	Touch up at 195.0, 398.0, offset=0.05s confirmed by TouchEventsCompleted.
Jan 21 12:21:05 WebDriverAgentRunner-Runner(WebDriverAgentLib)[423] <Notice>: Waiting up to 10s until com.apple.mobilesafari is in idle state (including animations)
Jan 21 12:21:05 nearbyd[110] <Error>: onHIDEvent HIDServiceClient is nil
Jan 21 12:21:09 testmanagerd(FrontBoardServices)[268] <Notice>: [FBSDisplaySource 3-4903] did invalidate
Jan 21 12:21:09 testmanagerd(FrontBoardServices)[268] <Notice>: [FBSDisplaySource 4-4904] did invalidate
Jan 21 12:21:09 testmanagerd[268] <Notice>: normalized {270, 120} to {0.69230769230769229, 0.14218009478672985}
Jan 21 12:21:09 testmanagerd[268] <Notice>: Waiting up to 5.0s for TouchEventsCompleted to confirm 
<XCSynthesizedEventRecord 'tap display 1'>
Path 1:
	Touch down at 270.0, 120.0, offset=0.00s
	Touch up at 270.0, 120.0, offset=0.05s.
Jan 21 12:21:09 MobileSafari(WebKit)[454] <Notice>: Single tap identified. Request details on potential zoom. (0x10585b000)
Jan 21 12:21:09 testmanagerd[268] <Notice>: Handling kAXUserTestingNotification from AX element pid: 62, elementOrHash.elementID: 0.1 (0x1056243a0)
Jan 21 12:21:09 testmanagerd[268] <Notice>: Got user testing notification with payload <private>
Jan 21 12:21:09 testmanagerd[268] <Notice>: Calling handler for AX notification kAXUserTestingNotification from AX element pid: 62, elementOrHash.elementID: 0.1 with payload <private>
Jan 21 12:21:09 MobileSafari(WebKit)[454] <Notice>: Ending potential tap. (0x10585b000)
Jan 21 12:21:09 MobileSafari(WebKit)[454] <Notice>: Single tap recognized - commit potential tap (0x10585b000)
Jan 21 12:21:09 testmanagerd[268] <Notice>: Event 
<XCSynthesizedEventRecord 'tap display 1'>
Path 1:
	Touch down at 270.0, 120.0, offset=0.00s
	Touch up at 270.0, 120.0, offset=0.05s confirmed by TouchEventsCompleted.
Jan 21 12:21:09 testmanagerd[268] <Notice>: Handling kAXUserTestingNotification from AX element pid: 454, elementOrHash.elementID: 0.1 (0x1056243a0)
Jan 21 12:21:09 testmanagerd[268] <Notice>: Got user testing notification with payload <private>
Jan 21 12:21:09 testmanagerd[268] <Notice>: Calling handler for AX notification kAXUserTestingNotification from AX element pid: 454, elementOrHash.elementID: 0.1 with payload <private>
Jan 21 12:21:09 WebDriverAgentRunner-Runner(WebDriverAgentLib)[423] <Notice>: Waiting up to 10s until com.apple.mobilesafari is in idle state (including animations)
Jan 21 12:21:09 MobileSafari(XCTAutomationSupport)[454] <Notice>: Received request to notify when the main run loop is idle
Jan 21 12:21:09 MobileSafari(XCTAutomationSupport)[454] <Notice>: Idle notifier finished setting up run loop observer
Jan 21 12:21:09 MobileSafari(XCTAutomationSupport)[454] <Notice>: Idle notifier run loop observer fired
Jan 21 12:21:12 testmanagerd(FrontBoardServices)[268] <Notice>: [FBSDisplaySource 4-4912] did invalidate
Jan 21 12:21:12 testmanagerd[268] <Notice>: normalized {178, 199} to {0.4564102564102564, 0.23578199052132701}
Jan 21 12:21:12 testmanagerd[268] <Notice>: Waiting up to 5.0s for TouchEventsCompleted to confirm 
<XCSynthesizedEventRecord 'tap display 1'>
Path 1:
	Touch down at 178.0, 199.0, offset=0.00s
	Touch up at 178.0, 199.0, offset=0.05s.
Jan 21 12:21:12 MobileSafari(WebKit)[454] <Notice>: Single tap identified. Request details on potential zoom. (0x10585b000)
Jan 21 12:21:12 testmanagerd[268] <Notice>: Handling kAXUserTestingNotification from AX element pid: 62, elementOrHash.elementID: 0.1 (0x1056243a0)
Jan 21 12:21:12 testmanagerd[268] <Notice>: Got user testing notification with payload <private>
Jan 21 12:21:12 testmanagerd[268] <Notice>: Calling handler for AX notification kAXUserTestingNotification from AX element pid: 62, elementOrHash.elementID: 0.1 with payload <private>
Jan 21 12:21:12 testmanagerd[268] <Notice>: Event 
<XCSynthesizedEventRecord 'tap display 1'>
Path 1:
	Touch down at 178.0, 199.0, offset=0.00s
	Touch up at 178.0, 199.0, offset=0.05s confirmed by TouchEventsCompleted.
Jan 21 12:21:12 MobileSafari(WebKit)[454] <Notice>: Ending potential tap. (0x10585b000)
Jan 21 12:21:12 MobileSafari(WebKit)[454] <Notice>: Single tap recognized - commit potential tap (0x10585b000)
Jan 21 12:21:16 testmanagerd(FrontBoardServices)[268] <Notice>: [FBSDisplaySource 4-4920] did invalidate
Jan 21 12:21:16 testmanagerd[268] <Notice>: normalized {109, 247} to {0.27948717948717949, 0.29265402843601895}
Jan 21 12:21:16 testmanagerd[268] <Notice>: Waiting up to 5.0s for TouchEventsCompleted to confirm 
<XCSynthesizedEventRecord 'tap display 1'>
Path 1:
	Touch down at 109.0, 247.0, offset=0.00s
	Touch up at 109.0, 247.0, offset=0.05s.
Jan 21 12:21:16 MobileSafari(WebKit)[454] <Notice>: Single tap identified. Request details on potential zoom. (0x10585b000)
Jan 21 12:21:16 testmanagerd[268] <Notice>: Handling kAXUserTestingNotification from AX element pid: 62, elementOrHash.elementID: 0.1 (0x1056243a0)
Jan 21 12:21:16 testmanagerd[268] <Notice>: Got user testing notification with payload <private>
Jan 21 12:21:16 testmanagerd[268] <Notice>: Calling handler for AX notification kAXUserTestingNotification from AX element pid: 62, elementOrHash.elementID: 0.1 with payload <private>
Jan 21 12:21:16 testmanagerd[268] <Notice>: Event 
<XCSynthesizedEventRecord 'tap display 1'>
Path 1:
	Touch down at 109.0, 247.0, offset=0.00s
	Touch up at 109.0, 247.0, offset=0.05s confirmed by TouchEventsCompleted.
Jan 21 12:21:16 CacheDeleteAppContainerCaches[443] <Notice>: Processing [<private>: <private>]...
Jan 21 12:21:16 WebDriverAgentRunner-Runner(WebDriverAgentLib)[423] <Notice>: Waiting up to 10s until com.apple.mobilesafari is in idle state (including animations)
Jan 21 12:21:16 MobileSafari(XCTAutomationSupport)[454] <Notice>: Received request to notify when the main run loop is idle
Jan 21 12:21:16 MobileSafari(XCTAutomationSupport)[454] <Notice>: Idle notifier finished setting up run loop observer
Jan 21 12:21:16 MobileSafari(XCTAutomationSupport)[454] <Notice>: Received request to notify when animations are idle
Jan 21 12:21:22 testmanagerd[268] <Notice>: normalized {365, 703} to {0.9358974358974359, 0.83293838862559244}
Jan 21 12:21:22 testmanagerd[268] <Notice>: Waiting up to 5.0s for TouchEventsCompleted to confirm 
<XCSynthesizedEventRecord 'tap display 1'>
Path 1:
	Touch down at 365.0, 703.0, offset=0.00s
	Touch up at 365.0, 703.0, offset=0.05s.
Jan 21 12:21:22 MobileSafari(WebKit)[454] <Notice>: Single tap identified. Request details on potential zoom. (0x10585b000)
Jan 21 12:21:22 MobileSafari(WebKit)[454] <Notice>: Ending potential tap. (0x10585b000)
Jan 21 12:21:22 MobileSafari(WebKit)[454] <Notice>: Single tap recognized - commit potential tap (0x10585b000)
Jan 21 12:21:22 testmanagerd[268] <Notice>: Handling kAXUserTestingNotification from AX element pid: 454, elementOrHash.elementID: 0.1 (0x1056243a0)
Jan 21 12:21:22 testmanagerd[268] <Notice>: Got user testing notification with payload <private>
Jan 21 12:21:22 testmanagerd[268] <Notice>: Calling handler for AX notification kAXUserTestingNotification from AX element pid: 454, elementOrHash.elementID: 0.1 with payload <private>
Jan 21 12:21:22 testmanagerd[268] <Notice>: Event 
<XCSynthesizedEventRecord 'tap display 1'>
Path 1:
	Touch down at 365.0, 703.0, offset=0.00s
	Touch up at 365.0, 703.0, offset=0.05s confirmed by TouchEventsCompleted.
Jan 21 12:21:22 testmanagerd[268] <Notice>: Handling kAXUserTestingNotification from AX element pid: 62, elementOrHash.elementID: 0.1 (0x1056243a0)
Jan 21 12:21:22 testmanagerd[268] <Notice>: Got user testing notification with payload <private>
Jan 21 12:21:22 testmanagerd[268] <Notice>: Calling handler for AX notification kAXUserTestingNotification from AX element pid: 62, elementOrHash.elementID: 0.1 with payload <private>

@craitose
Copy link
Author

Thanks , the coordinates of the last tap corresponds with that in the appium logs and I saw no error messages in the rest of the iOS logs.All I have left is my button tag theory, I will modify the w3schools example to use a button as the click target instead of a paragraph and see if it works.

@craitose
Copy link
Author

craitose commented Jan 29, 2025

After further investigation , I can conclude the following:

  1. clicks on elements with <button> tag work fine.
  2. clicks on elements without the tag are occurring slightly below the element target.

How many pixels below the target is variable but the coordinate calculations need to be tweaked.

To help with trouble shooting by highlighting the target followed by the actual clicking of the target element I have have come up with the following script.
(as said before this would help others in trouble shooting the click issues if made in to a built-in method that can be activated in a capability flag i.e 'appium:highlightWebClick': true)

export async function highlightWebClick() {
    await browser.addCommand('highlightWebClick', async function() {
    await this.waitForExist();
    await browser.execute((el) => {
        el.style.border = '5px solid #00ff22'; // highlight target element
    }, await this);
    await browser.execute((el) => { // click target element
      el.click();
    }, await this);
    await browser.pause(1000);
    await browser.execute((el) => { // remove highlight from target element
        el.style.border = 'none'
    }, await this);
    await browser.pause(500);
}, true);
}

Usage:

await this.menuIcon.highlightWebClick();

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants