As stationary technologies and means of communication take their place in the rising world of mobility, new preferences and favourites emerge. Since the communication speed over PCIe supersedes communication over Ethernet, leading companies in the world of mobility redesign their hardware and software architectures to enable interhost communication over PCIe. The main hardware component in a such design is a PCIe switch connected to all hosts participating in the communication network.

The challenge

The PCIe protocol was originally designed to enable fast communication between a CPU and PCIe devices on the same host. This means that in the whole system there must be only one CPU which is denoted as the root complex. All other devices are enumerated by the root complex in the start-up phase. In a multi-host environment, there are multiple hosts and multiple CPUs meaning there must be a mechanism to enable their communication and overcome the usual enumeration process of all other devices. The second big challenge was to design an architecture to notify the hosts of data arrival. The third big challenge was to offload the CPU for big data transfers and use the DMA switch instead. In embedded systems, DMA is crucial for communication performance to fully make use of the maximum theoretical protocol speed. DMA shouldn’t just be enabled for communication between the CPU and the PCIe device, but also for host-to-host data transfers. The fourth big challenge was to design a comprehensive and simple API for user applications. Finally, the existing system drivers which can be used as they are or at least as a good reference point were implemented in Linux. In the industry of mobility, a safer operating system must be used which, in this case, was QNX.

Solution

To enable PCIe communication in the system with multiple root complexes an NTB (Non-Transparent Bridge) mechanism must be used. In that way, a host sees another host as any other regular PCIe device. NTB is defined as a part of the switch configuration. To enable the notifying mechanism between hosts upon data arrival, standard PCIe interrupt mechanisms were configured and used. This was both configured as a part of the switch configuration and implemented in the corresponding drivers. To enable DMA data transfers between two hosts, an NTB-based Remote DMA (NTRMD) concept was used and extended for multiple hosts. As a user application API we opted for an existing well-known API used in the generic NTRDMA concept. All drivers were developed in QNX. Some of them from scratch, some of them were ported from another operating systems, and some of them were ported and extended to enable multi-host support.

Benefits

With our solution the mobility industry can easily redesign hardware and software architecture to use faster PCIe based interhost communication instead of Ethernet. Fast data transfers are a must at big data processing between hosts (e.g. video processing and sensor data processing). This is achieved by using a state-of-the-art NTB based Remote DMA concept. User applications are developed using well-known and well documented APIs. In case of a request to use Ethernet based APIs like sockets, this can be enabled without changing lower layers. If a different switch vendor is selected, one implementation layer can be adapted to support it with minimum effort.