ARTICLE AD BOX
We are working on automating a web application built with Flutter, using WebView2 in a desktop application.
Since Flutter Web renders the UI using a canvas-based approach, standard DOM elements are not accessible. This makes traditional automation methods (e.g., locating elements by ID, class, or XPath) ineffective.
We do not have the ability to modify the application to add test hooks or identifiers.
Looking for suggestions or approaches to handle automation in such scenarios.
Our Workaround
To overcome this limitation, we found a workaround by enabling accessibility features in the Flutter app. This exposes aria-label attributes, allowing partial interaction with elements. We are triggering accessibility mode using the following script:
document.querySelector('[aria-label="Enable accessibility"]').click();Once accessibility is enabled, we attempt to interact with elements like this:
let button = document.querySelector('[aria-label=""Login""]'); if (button) { button.click(); }This works for some visible elements such as buttons and basic controls.
Problem
While the accessibility workaround partially works, we are facing several limitations:
Not all elements expose aria-labels, especially: Popups Dropdowns / submenus Dynamic overlays No access to unique identifiers like id or stable selectors for many elements. Elements rendered after interactions (like opening menus) are often: Not accessible via querySelector The DOM structure remains minimal due to Flutter’s canvas rendering, limiting automation capabilitiesQuestions
We are looking for guidance or best practices on the following:
Is there a reliable way to automate Flutter Web apps using WebView2 or similar tools? How can we interact with dynamically rendered elements like popups and submenus? Are there any alternative approaches (e.g., DevTools protocol, accessibility tree inspection, or Flutter-specific hooks)? Would using tools like Playwright, Puppeteer, or WinAppDriver provide better results in this scenario?Any suggestions, workarounds, or insights would be greatly appreciated.
