Introduction

Automotive industry has been rapidly rising in the last couple of years and fully autonomous driving is the final goal for many car manufacturers. Advanced Driver Assistance Systems (ADAS) are becoming increasingly available in many vehicles, in order to improve safety and comfort. In addition, the automotive industry is investing a lot in order to reach level 4 and level 5 of Autonomous Driving (AD). Before starting a mass production, all these vision-based systems must be thoroughly tested, presuming conduction of many tests with real-life scenarios. Different driving scenarios, with different conditions, must be considered. All these tests are time-consuming, and therefore expensive to be performed in real traffic. Most of these tests can be replaced with tailored HIL simulations.

The challenge

The customer Denso approached us with requirement to develop a video logger capable of capturing video streams from automotive cameras with resolution of up to 2 M pixels with frame rates of 30 and 60 fps. In addition to capturing video data, the logger should be capable of logging CAN and CAN FD data. All logged data should be time stamped by using time from global positioning system (GPS) network. Captured data should be recorded to permanent storage (SSD) or forwarded to PC over PCIe interface. In addition to data logging, the device should be able to reproduce logged data in the same manner as it was captured – time alignment should be respected. Additionally, it should be possible to use several devices at the same time with synchronized time stamps of captured data.

Logger

The developed video logger is a TDA2P based device that captures raw video data from one or more cameras, performs image processing, and makes the frames along with CAN data stored for further analysis or algorithm development, testing and optimization. The device supports logging data on a SSD device, playback of recorded data, and live feed of real image and CAN data to algorithms running on a PC side. Additionally, it is possible to send recorded data to a server via 10Gbps Ethernet interface.

TDA2P is a SoC in which Linux runs on ARM Cortex A15 processor and RTOS runs on ARM Cortex M4, c66 DSPs and EVE cores. To have the capturing process on Linux and RTOS side aligned, it was necessary to synchronize their system time. That was done by getting the PPS (Pulse Per Second signal from GPS) signal on both Linux and RTOS to identify the start of a second, and exchange system time to determine which second has started at a pulse edge. In a similar manner, it possible to combine several devices with synchronized capturing of video from all cameras in the system.

To allow live evaluation of video processing algorithms on a PC we supported live data-forwarding over PCIe interface from device to a PC. As TDA2P and PC are acting as RootComplex devices, it was necessary to use PCIe switch configuration for RootComplex-to-RootComplex data transfer with NTP Linux driver. It was possible to connect up to three devices to one PC over PCIe interfaces.

Example of data forwarding from two synchronous devices is shown in the following image. One device acting as a clock master extracts time information and the PPS from a GPS device (stratum-0 source) and synchronizes its system time with it. The other device synchronizes its system time with the master device (stratum-1 source). Besides the system time, Frame sync signal is used as a reference point for capturing of video frames. Therefore, all cameras from both devices capture frames at the same time – having same time stamp (unique across devices). Captured video frames from each device are being sent to the PC via PCIe interface, where they can be processed in real time by an algorithm that is a candidate to be ported on the target embedded system (ECU).

The software architecture of the video logger is shown in the following figure – with a clear distribution of functionalities between Linux and RTOS operating systems.