When discussing containerized solution development, resource consumption and performance are considered some of the biggest challenges alongside security. GPU acceleration is a key factor in enabling high performance with reasonable resource consumption, ensuring seamless content loading and responsiveness. This is especially important for solutions like mARTini, since the Android system is highly driven by UX factors. This article explores the necessity and methodology of integrating the Linux host graphics pipeline, based on Wayland, with the Android graphics pipeline. The integration must achieve zero-copy operations to ensure efficient management and distribution of graphics data. The article outlines the current state of both pipelines, the challenges of their coexistence, various approaches to resolving these challenges, and their impact on the performance and resource consumption.

Currently, the mARTini solution is proven to support GPU acceleration on NXP i.MX8QM, NVIDIA Jetson AGX Xavier, and Jetson Nano platforms.

Problem Statement

To embrace GPU acceleration within implemented solution, enabling the coexistence of the platform’s host graphics pipeline based on Wayland with the containerized Android graphics pipeline is mandatory. Achieving this through a zero-copy approach ensures that all graphics operations are performed on the same graphics data, which are allocated once and distributed with handles to all relevant parties. This is the only way to achieve performance levels worthy of the high-quality embedded world.

The main problem is that Wayland-based Linux and Android graphics paradigm differ and are not compatible by default. Therefore, it is essential to adopt the Android container graphics into typical Wayland workflow available on the host platform.

Linux Host Graphics Pipeline

The Linux host graphics pipeline is built around Wayland, a protocol that specifies communication between clients and the display server. It is designed to be a simpler and more efficient replacement for the X Window System. Key components of Linux graphics are presented on the illustration below, and they include:

• Wayland Protocol: Wayland is a display server protocol that facilitates direct communication between a display server (compositor) and its clients. Unlike X11, Wayland does not perform drawing operations itself but allows clients to manage their window contents directly. Wayland uses a minimalistic approach, providing mechanisms for buffer sharing and input event delivery, which significantly reduces the complexity and overhead of the display server.
• OpenGL ES API: OpenGL for Embedded Systems (OpenGL ES) is a subset of the OpenGL API designed for embedded devices, such as smartphones and tablets. It provides a set of standard APIs for rendering 2D and 3D graphics, allowing for hardware-accelerated graphics operations. OpenGL ES is essential for rendering complex scenes with high performance and low power consumption, crucial for embedded and mobile platforms.
• EGL: EGL is a widely adopted interface between rendering APIs like OpenGL ES and the native windowing system. It handles context management, surface binding, and synchronization. EGL supports the Direct Rendering Infrastructure (DRI) and hardware acceleration, enabling direct rendering to the screen without going through the window system. This reduces overhead and latency, providing a more responsive graphics experience.
• Proprietary Wayland-enabled EGL Implementations: These are vendor-specific implementations that bind Wayland objects with EGL objects, allowing the creation of rendering contexts from Wayland connections. This enables content producers to use EGL contexts to access OpenGL APIs without direct memory allocation, encapsulated within the Wayland-enabled EGL stack. These implementations ensure optimal performance and compatibility with specific hardware.

Wayland-based Linux host graphics stack

Android Graphics Pipeline

Unlike Linux, the Android graphics pipeline is not based on the Wayland protocol, but handles graphics buffers in its own  custom way. Android graphics also supports EGL and OpenGL paradigms in a manner very similar to Linux. However, it binds its entities to Android-specific ones instead of using Wayland. Key components are illustrated below, and they include:

• EGL Contexts: Used by content producers to access OpenGL APIs, encapsulated within the Android-enabled EGL stack similarly to how it is done in the Wayland-enabled EGL stack on Linux. EGL contexts manage all the states related to rendering operations, including textures, shaders, and framebuffers. This encapsulation allows efficient resource management and sharing between different components of the graphics pipeline.
• BufferQueue/ANativeWindow: BufferQueue is a key component that manages the exchange of graphic buffers between producers and consumers. It ensures that buffers are queued and dequeued in the correct order, maintaining synchronization between different parts of the pipeline. ANativeWindow is an abstraction layer that allows native applications to interact with BufferQueue, providing a consistent API for rendering surfaces.
• SurfaceFlinger: SurfaceFlinger is the central system service responsible for compositing buffers from different applications. It collects buffers from BufferQueue, orchestrates composition operations, and cooperates with HWComposer in sending the final image for display. SurfaceFlinger uses double buffering to ensure smooth rendering and to prevent visual artifacts. It manages layers, transparencies, and animations, providing a seamless user experience.
• HWComposer: HWComposer is a hardware abstraction layer that interacts directly with the display hardware. It receives commands on handling layers, target buffers for rendering, and similar from SurfaceFlinger and performs the final steps of displaying the image on the screen. HWComposer leverages hardware acceleration to optimize performance and power consumption. It is responsible for tasks such as scaling, rotation, and color conversion, ensuring that the output matches the display’s capabilities.
• Gralloc: Gralloc is the graphics memory allocator in Android. It manages the allocation and deallocation of graphic buffers, ensuring efficient use of memory resources. Gralloc is crucial for zero-copy operations, as it allows buffers to be shared between different components without copying the data. This minimizes latency and improves performance, especially in high-resolution and high-frame-rate scenarios.

Android graphics stack

Challenges

Enabling GPU acceleration implies addressing two major challenges: the existence of the Android-enabled EGL stack and the adoption of Wayland-Android entities. How these two challenges are approached is tightly dependent. Therefore, the rationale on how we will address these challenges will be analyzed and discussed in more detail in the following paragraphs.

Existence of EGL/OpenGL ES Stack in Android

As previously mentioned, the primary challenge is integrating a vendor-specific EGL/OpenGL ES stack within the mARTini Android device. While the EGL stack is usually provided by the chip vendor, this is not always the case, particularly if the vendor does not supply a graphically accelerated Android device. This stack is crucial to the Android graphics pipeline, as it relies on proprietary graphics driver implementations. Effective utilization of these drivers is essential to achieving high-quality rendering performance. Therefore, this stack is critical for managing graphics and needs to be aligned with the existing systems. The proprietary nature of EGL/OpenGL ES stacks can lead to compatibility issues and requires careful handling to ensure performance and stability.

Wayland and Android Adaptation

Another significant challenge is aligning the Android graphics pipeline with the host pipeline using Wayland, ensuring seamless operation and zero-copy efficiency. This requires adapting the buffer management and rendering mechanisms within the mARTini container solution to be compatible with both, Android and Linux systems. The goal is to achieve efficient data sharing between Weston and Android without duplicating resources, which would otherwise lead to increased memory usage and reduced performance.

Possible Approaches

Existence of EGL/OpenGL ES Stack in Android

The first challenge of enabling a GPU accelerated EGL stack within the Android system can be addressed in four different ways:

• Recycling Native Android
• Mesa Approach
• Patched or re-built EGL stack from host
• IPC-based Approach

The following sections will provide more details on each approach and include a comparison, which is presented in the table below.

Recycling Native Android

This approach involves integrating a prebuilt Android-enabled EGL stack within the mARTini device. The approach implies the existence of Native Android device provided by the chip vendor from which EGL stack can be extracted. It is necessary to carefully match the available kernel implementations and configurations between the Linux host platform and the Native Android device.

Mesa-3D Approach

Mesa is an open-source implementation of the OpenGL specification – a system for rendering interactive 3D graphics. It provides hardware acceleration for rendering, which, as we have already emphasized, is crucial for maintaining performance. The Mesa library includes various modules which abstract GPU driver development, making it easier to support multiple hardware platforms.

This abstraction provided by the Mesa solution is only possible if it relies on a specific set of open-source graphics driver implementations. If the open-source drivers supported by Mesa are enabled on the host, Mesa can be used to bridge the Wayland-Android gap. This approach involves:

1. Building Mesa within the mARTini Android device.
2. Including the proper Mesa modules, a compatible EGL stack, HWComposer, and Gralloc.
3. Disabling the proprietary driver on the host.
4. Enabling the open-source driver on the host.

However, this approach is not viable for platforms using proprietary GPU drivers, which is usually the case, and it can lead to mismatches due to the insufficient quality of open-source driver implementations.

Patched or Re-built EGL Stack from Host

If Mesa cannot be applied and there is no available Android-enabled EGL stack, consider using a patched or re-built EGL stack from the host platform to run it within the Android environment. This approach involves:

1. Patching Wayland-enabled EGL stack libraries to make it compatible with the Android environment (if the EGL stack on the host is provided without source code).
2. Rebuilding Wayland-enabled EGL stack libraries to make them compatible with the Android environment (if the EGL stack on the host is provided with source code).
3. Integration in the Android environment.

IPC-Based Approach

If none of the above approaches are viable, an IPC-based approach should be considered as the last alternative. This involves:

1. Definition of a suitable IPC protocol, considering the nature of communication mechanisms (request-response and/or event triggered) and low latency as key performance indicators.
2. Translate entities expected by Wayland and Android graphics pipelines.
3. Push OpenGL commands from Android to the Linux host.
4. Execute commands on the Linux host and return handles, pointers, and operation results to Android.

This approach requires the implementation of a new IPC-based EGL stack and integration with SurfaceFlinger on the Android side, as well as the implementation of a Graphics Container Manager on the Linux host to support communication and execute commands provided over the defined IPC. This EGL stack would not interface GPU in a typical way but will push commands through the IPC to be executed natively on the host platform. The results of the operations will be returned to the Android container and integrated within the typical Android pipeline. This will still require Wayland and Android adoption, which will be described in the following paragraphs.

Wayland and Android Adaptation

Besides providing a vendor-proprietary EGL/OpenGL ES stack implementation (whether coming from the host platform or Android natively) within the mARTini device, it is crucial to address the differences between Wayland and Android paradigms and create needed adaptations.

Integrating the EGL stack involves ensuring that the EGL contexts created by Android can interface directly with Wayland surfaces, which is not supported by default within the Android environment. Implementing a Wayland-Android adaptation layer is essential to align graphics buffer allocation and management processes between the two environments.

Input flinger also requires modification for seamless input event communication via the Wayland pipe.

Combination with Integration of Existing Android Stack

When integrating an existing Android-enabled EGL stack into the mARTini containerized Android device,  the Android system can be considered as Wayland client (it is certainly more than that, but this perspective is simplified for the purpose of demystification of the approach). The typical Android graphics pipeline will perform its usual operations to compose the final image from multiple applications and components. This final image will then be provided to the Wayland compositor as if it came from a simple Wayland client, with the goal to be displayed like all other applications. This way, the Wayland compositor (Weston) will remain agnostic about the operations performed within Android to compose the final image.

The Android component which is responsible for sending a command to post graphics data to display is HWComposer.

Therefore, HWComposer modification is mandatory to adapt the Android graphics pipeline to the Wayland compositor on the host platform. Modifications to HWComposer would include:

• Establishing the connection with Weston and creating the required Wayland entities (wl_display, wl_surface, etc.)
• Implementation which creates Wayland-compatible buffers from graphics data and commits them to the Wayland surface instead of displaying them.

Rendered graphics data are shared with HWComposer through buffer management mechanisms, most commonly using DRM handles. The structure of these handles depends on the proprietary implementation but always derives from Android’s native handle, following a specific order and data types. This allows the determination of DMA-BUF properties and file descriptors necessary for creating a Wayland-compatible buffer.

HWComposer uses ‘set()’ functions to specify rendering targets for clients. This target object containing composed graphics data will be used with the ‘zwp’ Linux protocol functions to extract DMA-BUF handles and create a Wayland-compatible buffer. The actual modification of HWComposer will depend on its interface version. For HWC1, the end of the ‘set()’ function will be the target scope of interaction with Wayland. In HWC2, this will not be done in the ‘set()’ function, but in the ‘post()’ function, by removing the code for presenting the graphics buffer to display and providing it to Weston instead.

Combination with Host Stack Patching or IPC

This approach implies bypassing Android graphics components and relying on OpenGL API equivalence. No buffers from the Android stack will be used; instead, Wayland-enabled EGL context handling will be integrated directly with Surface Flinger.

This approach requires creating a custom implementation of an EGL adaptation layer that interfaces directly with both Wayland and Android components. The EGL context would manage OpenGL ES state and command execution, while Wayland protocols would handle buffer management and display. Additionally, this adaptation layer would translate between Wayland-compatible EGL entities and buffers on one side and Android-compatible EGL entities and buffers on the other. This requires deep integration with both Wayland and Android graphics subsystems.

NXP i.MX8QM, NVIDIA Jetson AGX Xavier, and Jetson Nano

The containerized Android solution, mARTini, has successfully enabled GPU acceleration on the NXP i.MX8QM, NVIDIA Jetson AGX Xavier, and Jetson Nano platforms. For these platforms, native Android devices which already contain an Android-enabled EGL stack utilizing GPU acceleration are avilable, allowing for extraction of the graphics stack. This ensured the presence of a GPU-accelerated EGL stack within the Android environment. The mARTini solution supports two approaches – integration of the graphics stack solely within the RT-RK custom and optimized mARTini Android device or an adaptation of the entire native Android device (including the graphics stack) to be deployable within an LXC container.

As previously explained, the implementation of the HWComposer module with Wayland expansion goes hand in hand with the integration of already existing graphics stack from the Android environment. Therefore, for the NXP i.MX8QM,  NVIDIA Jetson AGX Xavier, and Jetson Nano platforms, the HWComposer module was modified to support such Wayland integration. The mARTini solution enabled the integration of a Wayland adaptation layer by providing either our custom HWComposer module or within the HWComposer module provided by the chip vendor, system integrator, or a third party.

Impact on Performance

Achieved performance and resource consumption can differ widely based on hardware components and the quality of the system software itself. To emphasize the significant impact of GPU acceleration on overall system performance, consider the following example:

Running premium video content streaming services such as Netflix without GPU acceleration – using software rendering on the CPU and shared memory for buffer management – can lead to approximately:

~350-400% of CPU usage

~900MB of RAM consumed per Netflix processes

With GPU acceleration, these factors are significantly reduced, resulting in a higher quality user experience:

~ 50% of CPU usage

~ 550MB of RAM consumed per Netflix processes

Conclusion

Integrating the Linux host graphics pipeline with Wayland and the Android graphics pipeline is a complex but essential task. Achieving zero-copy operations ensures efficient graphics data management and distribution. Various approaches – including Mesa, native Android solutions, patched or re-built EGL stacks, and IPC-based solutions – offer potential paths forward. Each approach has its own pros, cons, risks, and contingency plans, and the best solution depends on the specific platform requirements and available resources.

Expertise in development of such tasks is needed to refine these approaches and ensure seamless integration, ultimately enhancing the performance and efficiency of graphics data handling in hybrid Linux-Android environments.