Description
Environment (please complete the following information):
- Node.js version: 22.14.0
- NPM version: yarn 4.4.0
- Browser name and version: Chrome 117
- Platform name and version: Linux CI
- WebdriverIO version: 9.12.2
@wdio/visual-serviceversion: 6.4.0
Config of WebdriverIO + @wdio/visual-service
const imageServiceOpts: VisualServiceOptions = {
autoSaveBaseline: false,
baselineFolder: path.join("storybook-visual-tests", "storybookTestBaseline"),
screenshotPath: "storybook-visual-tests",
waitForFontsLoaded: true,
disableCSSAnimation: true,
disableBlinkingCursor: true,
clearRuntimeFolder: true,
ignoreAntialiasing: true,
storybook: {
url: baseUrlDomain,
additionalSearchParams: new URLSearchParams({ globals: `cssVariables:${THEMES[themeId]}` }),
getStoriesBaselinePath: (category, component) => `${category}/${component}/${themeId}`,
numShards: 32,
skipStories: createRegex(),
},
};
const config: WebdriverIO.Config = {
runner: "local",
specs: [],
exclude: [],
maxInstances: 20,
capabilities: [],
waitforTimeout: 10_000,
logLevel: "error",
services: [["visual", imageServiceOpts]],
framework: "mocha",
reporters: ["spec"],
hostname: "localhost",
port: 4444,
path: "/wd/hub",
};Describe the bug
Since the update to webdriver-image-comparison v6+, we noticed flakiness in our visual tests.
The differences are usually a few bytes on the baselines (0.001% in the logs, always less than 0.1%, depends on the resolution).
We tried many configuration, eliminating potential external factors and debugging, but our conclusion is that it must be related to this change.
To Reproduce
- Compare tests before after
webdriver-image-comparisonmajor update with 100+ runs
Expected behavior
No flakiness in test due to a very low percentage difference
Additional context
Should we try pixelmatch instead of RessembleJS? It has less options, but it is already included in Jimp (check the node_modules) and used by other testing framework (Playwright and Cypress plugins). Note that this will be a breaking change due to less options.