It is usual in modern cameras to have a single image sensor overlaid with a colour filter array (CFA). The image from camera is usually put out in a raw image format and converted to a visible image format (RGB or YUV) in a real-time by a SoC image signal processor or offline when raw image data are saved. An example of colour filter array on a sensor (Bayer pattern) and original image and captured image (Bayer pattern coloured) are presented in the following images.

The challenge

The customer Texas Instruments approached us with a requirement to develop a solution on their TDA2P SoC that will enable conversion of video streams recorded in a raw image format. They have customers that were recording camera video streams in the raw image format and later performing fine tuning of parameters used in a process of RAW to YUV conversion by a software running on TDA2P’s image signal processor (ISP). The initial option to read the raw image data from the SD card and write data converted to YUV format back to the SD card was not acceptable due to the large amount of data in the video streams recorded in the raw image format. Furthermore, possibly, data could be stored in any standard or even custom container format. Therefore, it was necessary to provide a way for the potential user to handle data in a preferred container format, send only raw image frames to TDA2P for conversion, and receive image frames converted to YUV image format. The proposal was to use PCIe interface to transfer raw image data to TDA2P and to get converted image frames from TDA2P. To enable maximum flexibility, a PC was chosen as the origin of the files with raw image data in a specific container format. Therefore, the application of the potential user (Texas Instruments’ customer) could be executed on a standard PC with possibility to convert raw image frames on the external TDA2P connected via PCIe interface – on request – frame by frame. In such configuration, TDA2P acts as a coprocessor for the PC.
The anticipated configuration for automated conversion of raw image frames to YUV frames is presented in the following figure.

The application running on a PC is sending up to 4 frames of raw image format via PCIe interface to the Texas Instruments TDA2P evaluation board. Received raw image frames are transferred from the Linux domain (where PCIe driver resides) to the RTOS domain (where video data processing is being performed). Then, raw image data are processed on the internal image signal processor (ISP) and sent back to the Linux domain. The converted image frames are then transferred from Linux domain on TDA2P to the host PC. Since the PC is always acting as a Root Complex on PCIe bus, TDA2P was configured to act as an End Point device.
The software application consists of two parts:

1. TDA2P SoC software

• Micro Linux kernel running a PCIe driver (as Endpoint PCIe device)
• Interface between the SYS/BIOS RTOS and the Linux driver
• Link&Chains framework utilized to implement image-processing pipeline
• RAW to YUV conversion integrated on ISP block

2. PC side

• Linux Ubuntu with custom PCIe driver (as Root Complex PCIe device)
• Application that uses this PCIe driver to send RAW, receive YUV data, and record on SSD
• SDK and library of functions, for Texas Instruments’ customers to implement their own SW tools to access the YUV streams from PCIe

A photograph of the hardware setup is shown below. The Texas Instruments TDA2P evaluation board is connected via PCIe cable to a standard PC PCIe slot.