Virtual Display is one of several modes for displaying platform views on Android. See Android Platform Views for an overview of modes.
AndroidView widgets need to be composited within the Flutter UI and interleaved between Flutter widgets. However the entire Flutter UI is rendered to a single texture.
In order to solve this problem, Flutter inflates and renders AndroidView widgets inside of VirtualDisplays instead of attempting to add it to the Android View hierarchy alongside Flutter‘s texture directly. The VirtualDisplay renders its output to a raw graphical buffer (accessed through getSurface()), and not to any actual real display(s) of the device. This allows Flutter to graphically interleave the Android View inside of its own Flutter widget tree by taking the texture from the VirtualDisplay output and treating it as a texture associated with any other Flutter widget in its internal hierarchy. Then the VirtualDisplay's Surface output is composited with the rest of the Flutter widget hierarchy and rendered as part of Flutter’s larger texture output on Android.
While this approach unblocks the core problem with embedding Android views in a Flutter UI visually, the VirtualDisplay choice causes a long tail of issues that we've needed to work around.
The heart of the problem with this approach is that the Android View inside of the VirtualDisplay is, as far as Android is concerned, inside of its own View hierarchy in a completely invisible Display and completely unconnected and unrelated to the View hierarchy that contains the Flutter texture output.
Much of the internal functionality to Android depends on walking the View hierarchy and querying information about the given View in its current hierarchy and window in order to function. Since the information for the embedded view is “wrong” here this internal functionality tends to break down. In addition to that this logic can change based on Android version, so we've needed to case some of our logic based on the runtime version of the OS.
Some of our issues with this approach and extended workarounds are described below.
To see all known issues specific to this mode, search for the vd-only label.
By default touch events would not work with PlatformViews. The Android View is inflated inside of a VirtualDisplay. Whenever the user taps what they see as the Android View on their primary display, they‘re “really” tapping the part of Flutter’s texture output that we‘re rendering to look like the actual Android View. The touch event created by their input is getting sent straight to Flutter’s View, not to the Android View they're actually trying to tap on.
AndroidView Flutter widget using hit testing logic within the Flutter Framework.VirtualDisplay. We then create a new MotionEvent describing the touch and forward it to the real Android View inside of its VirtualDisplay.MotionEvent directly to the known Android View embedded by the user. In cases where the View spawns other Views this dispatch may be sent to the wrong place.MotionEvent based on the information handed to us by the Flutter framework. It’s possible that there's data being lost and not totally carried over to our new MotionEvent, or some other general problem with synthesizing MotionEvents.Explaining our problems with accessibility (or “a11y”) requires a brief detour into explaining a11y itself on Android, and how a11y typically works on Flutter.
Android itself builds up a tree of a11y nodes for each Android View rendered to the screen. Each a11y node contains semantic information about what information is represented by a UI element: for example, whether it's a button, and what the text on button is. Typically the OS builds this information itself by directly walking the View hierarchy. A11y services on the device then use this tree of a11y nodes to deliver the semantic information in an accessible way to the user, by example drawing highlights over interactable UI elements and reading the semantic information described by the UI aloud to the user. Users can then use a11y services to interact with a UI, for example by using it to activate the button without needing to press once directly on where the button is graphically rendering on screen.
Some UIs, like Flutter and WebViews, don't have a tree of Android Views to describe their UIs. For these Android has a concept of a “virtual” a11y node hierarchy. Instead of walking the View hierarchy, Android Views may implement an AccessibilityNodeProvider that returns a tree of a11y nodes to describe whatever is being rendered within its own View subtree. The Flutter Android embedding uses this concept to generate an a11y node tree that matches the Semantics tree created by the Flutter framework.
Theoretically we should have been able to attach the a11y tree from the embedded Android View to our AccessibilityNodeProvider in Flutter, but in reality we've found that any attempts to do so directly fail. The reality of the situation appears to be that a11y code in Android frequently relies on the View hierarchy for state management and querying extra information, so attempting to parent the child’s hierarchy from the embedded view to our virtual hierarchy fails to really correctly associate the underlying view hierarchy in a functional way for a11y. In other words, reparenting the a11y node trees only “fixes” the a11y node tree itself, but there are still multiple places where Android a11y code relies on being able to query the “real” view hierarchy for required info and ends up failing with bugs in our reparented case.
AccessibilityNodeProvider.Button are not.Tracked in flutter/flutter#19418. Better support for reparenting a11y hierarchies is also an Android SDK issue.
Normally the embedded Android view wouldn’t be able to get any text input because it‘s inside of a VirtualDisplay that is always reported as being unfocused. Android doesn’t offer any APIs to dynamically set or change the focus of a Window. The focused Window for a Flutter app would normally be the one actually holding the “real” Flutter texture and UI, and directly visible to the user. InputConnections (how text is entered in Android) to unfocused Views are normally discarded.
InputConnection when the embedded Android View wants one.InputMethodManager (IMM) to be instantiated per Window instead of being a global singleton. So our naive attempt at setting up a proxy no longer worked on Q. To work around this further we created a subclass of Context that returned the same IMM as the Flutter View instead of the real IMM for the window when getSystemService is queried. This means whenever Android tries to use the IMM in our VirtualDisplay it still sees and uses the IMM from the real display that has been set to use the Flutter View as a proxy.InputConnection, it checks to see if an embedded Android View is “really” the target of the input. If so it internally goes to the embedded Android View, gets the InputConnection from there, and returns it to Android as if the embedded Android View's InputConnection is its own.InputConnection that's ultimately from the embedded Android View and uses it successfully.WebViews running on Android versions pre N have additional complications because they have their own internal logic for creating and setting up input connections that don‘t completely defer to Android. Within the flutter_webview plugin we’ve also needed to add extra workarounds in order to enable text input there.
InputConnection calls without the Flutter View proxy ever being notified.