Look behind you! Intelligent ADAS rear-view cameras
12 October 2017
With cars becoming more intelligent, ADAS (Advanced Driver Assistance Systems) can now offer drivers additional support – utilising more and more camera systems. The latest camera-based in-car ADAS systems monitor drivers’ fatigue and distraction levels.
For the digital edition of this piece, visit this link – or click here to register for EPDT's magazine.
Cameras mounted on the exterior of the vehicle deliver a full 360-degree view around the car – invaluable when manoeuvring in tight spaces or parking in modern, crowded cities.
This article from Toshiba Electronics Europe looks at the key challenges facing designers of these systems – and reviews some of the latest technology that facilitates the necessary miniaturisation and image processing.
According to the National Highway Traffic Safety Administration (NHTSA), reversing cameras will substantially reduce injury and death, especially among children. Several hundred people are killed each year, and thousands injured, when drivers reverse over someone without noticing them. The vast majority of victims are children, mainly because their small size and unpredictable movements can make them harder for drivers to be prepared for, and to see and judge from the driver’s seat. However, rear-view technology is changing: ultrasonic sensors are complemented by video feeds, and these early rear-view video feeds are being augmented by intelligent systems that can identify and flag up potential obstructions.
Rear-view requirements: driven by legislation
Since 2007, it has been known by the US Congress that rear-view cameras can reduce accidents significantly. In 2014, the NHTSA defined a precise rule that manufacturers must implement by May 2018, stating that all vehicles under 10,000 pounds (4,535kg) must include a reversing camera. Furthermore, the camera must cover a 10-foot by 20-foot (3m x 6m) zone behind the vehicle. Other countries are also considering or implementing their own legislation around rear-view cameras – and clearly the US stipulation means that manufacturers globally must address the technology regardless.
In fact, the automotive industry has proved proactive in this respect, acting ahead of this legislation, and a significant number of new cars sold today already include such a rear-view camera. In fact, even without this legislation, it is predicted that the availability of rear-view cameras in new cars will have increased to 75% by 2018.
The key challenge is detecting and warning the driver about obstructions. These fall into two main categories: the detection of fixed/static and moving objects, and their classification (for example, as pedestrian, cyclists or other vehicles). Both require intelligent processing of the camera feed in real time – regardless of whether the car is stationary or moving.
Figure 1. Block diagram of the smart rear-view camera reference design
This will of course have an impact on the cost of the vehicle. The impact will be lower on vehicles that already feature a suitable infotainment screen (circa US$40); for entry-level models and small cars, both a camera and screen will be required, increasing the cost to around US$140. Achieving these price points puts substantial pressure on automotive designers and camera system suppliers to deliver a low cost, high performance solution.
In future, the most basic systems will have to comply with the legislation by simply switching a video feed from the rear-view camera to the in-cabin screen – within two seconds when reverse gear is selected. Many of the current implementations of rear-view cameras are already more sophisticated than this and incorporate a processing module that overlays trajectory lines on the screen for manoeuvring and indicating obstacles, whereby the information for the obstacle detection is provided by available ultrasound sensors.
The challenge here is to improve object recognition through image-based processing and include it within the space confinements of the camera module. The new module needs to be the same size as current camera modules, and of course be hermetically sealed against moisture ingress. This vastly increases the requirements for system suppliers, as a balance has to be struck between processing power and dissipation losses. It must be taken into account that CMOS image sensors typically exhibit an increased noise performance at temperatures of >85°C, which in addition to reducing image quality, will also limit object detection.
Dedicated image processing
Figure 2a & 2b. The reference design demonstrates how a CMOS sensor and image recognition processor can fit into a small space.
Toshiba have developed, since the first release of its original image recognition processor in 2004, a dedicated product line-up. This includes the 11mm x 11mm LFBGA324 packaged TMPV7502, which is primarily aimed at ADAS applications where the available space is restricted.
The TMPV7502 is a three-core RISC-based processor with on-board RAM and a total of five hardware-based accelerators, for high performance object recognition processing and low current consumption. The device marries high performance with low power dissipation of <1W, through a highly parallelised approach – as opposed to serial processing with a high clock frequency. Along with the on-board CAN-bus interface, the TMPV7502 is utilised in single controller systems within rear camera modules.
Single module camera reference design
In order to demonstrate the capability of the processor and to ease the design burden on customers, TEE has also developed a comprehensive reversing camera system as a reference design.
Figure 3: The multi-board smart camera/image-processing system
The system consists of two main elements: the rear-view camera, including processing hardware; and a display unit that enables designers to evaluate and test the system.
The camera part of the reference system is based on four folded flexible circuit boards, occupying a small space.
The reference design is compatible with CMOS sensors from a range of manufacturers, leaving the options open for designers to specify a sensor module that meets the exact needs of the application. The imager is mounted on the first board, together with the associated EEPROM. The second board houses the main processor (TMPV7502) and the necessary working memory.
The third board houses the flash memory, along with DIP switches to define boot options and a manual reset switch, to assist during system debugging. The final board provides communications with the receiver board via an FPD-link III interface and an HSD connector. This board also provides a micro USB interface, which allows a direct connection to the camera system for software development, monitoring and debugging.
Figure 4a & 4b. The receiver board provides multiple connectivity options.
The second part of the reference design is the double-sided receiver board. In a production vehicle this functionality would be incorporated into the central infotainment system, but in the reference design it provides an easy way for developing and evaluating a rear-view camera system.
Specifically, the receiver board contains an FPD-link III deserialiser. There is an LVDS interface and bare header, which enables the ability to connect the reference design to other systems for further signal processing or added functionality. The board also allows engineers to directly connect a DVI monitor to access the data from the image processor visually.
Image processing, software and calibration
Rear-view cameras use fisheye lenses to achieve an over 180-degree field of view, in order to capture any pedestrians or objects that could impede the vehicle. This wide-angle introduces some extreme image distortion that needs to be corrected, in order to both receive an attractive image and ensure the object processing works effectively. The automated software calibration ensures all vertical objects are, indeed, vertical.
Figure 5. The software structure for the reference design.
The TMPV7502 processor uses Toshiba’s CoHOG (Co-occurrence Histograms of Oriented Gradients) hardware accelerators to classify such hazards as pedestrians. Other accelerators on the chip support computer vision algorithms with efficient edge and circle formation detection, thus defining the outer boundaries of any obstructions. By processing locally in the camera module, the central infotainment system processor capacity is retained for other processing tasks.
Within the reference design toolkit, Toshiba provides designers with both the low-level driver and sl_lib that connects into the accelerators within the TMPV7502 processor. The middleware is also included. Customers can then easily build their own applications upon this stack.
The Toshiba smart rear-view camera reference toolkit represents a significant step forward in rear-view camera design. By using a state-of-the-art image processor SoC (system on a chip), which relies on a parallel hardware accelerator-based processing system, power requirements are kept below 1W. And this allows the whole solution to be fitted in a small weather-sealed enclosure.
The system offers the flexibility to use CMOS imagers from many different manufacturers, and provides all of the low level code and debugging tools – to allow automotive customers to quickly and confidently design and debug intelligent rear-view camera systems.
Contact Details and Archive...