News

Sensor Fusion

Active safety systems promise to deliver huge advances in road safety.

Sensor fusion, the technique of combining radar and vision systems and using advanced data fusion techniques, is one of the big reasons why. Dr. Andreas Teuner, Delphi's chief safety engineer, Europe, explains the latest developments.

When it comes to vehicle safety systems, there are few ideas quite as appealing as a car that can avoid accidents on its own. Though radar-based Adaptive Cruise Control (ACC) has been with us for a number of years, new technology has enabled Delphi to take the next step with collision detection and mitigation systems. Available on vehicles including the Volvo S80 and Ford Mondeo, these collision mitigation systems warn the driver of an impending collision and if a collision is unavoidable, will automatically pre-charge the brakes for fast brake activation to mitigate the accident.

The second generation collision mitigation system was launched in 2007 and is available on the Volvo S80, V70 and XC70 vehicles. This system provides a warning and can automatically brake to reduce the impact speed if a collision is unavoidable. This system utilizes a combination of radar and vision sensors. In addition to moving vehicles, it also activates for stationary vehicles which requires a higher level of system robustness.

	Leveraging its radar and vision system expertise, Delphi fuses radar and vision technologies to provide advanced safety functions.

Both the Volvo and Ford systems were developed with Delphi. These systems are just a part of collision detection and mitigation systems that are the beginning of an active safety revolution that is expected to grow exponentially over the next few years. As active safety systems become more robust and intelligent, they will play an even more important role in helping drivers avoid collisions altogether. And, one of the key enablers of these active safety systems is data fusion; literally, the fusion and analysis of data derived from more than one type of sensor.

Data fusion
Radar sensors continue to serve the industry well; however, because of their inability to classify an object, they limit what active safety systems can do. To achieve optimal performance for systems like collision mitigation, radar needs to be used in conjunction with other sensing systems. When fused with a vision system (high-quality video camera, image processing modules and complex algorithms), the results are remarkable. And, with high-definition, solid-state camera technology entering mass production in other industries, system cost is becoming more affordable for automotive applications.

Further out into the future, we can expect to see integration of ESC (electronic stability control) and GPS (global positioning system) data, too. ESC adds vehicle dynamics data to the safety equation, while GPS-enabled map matching gives us location information, such as whether a vehicle is approaching a curve or a bridge. When combined with stability control information, the system has a more accurate picture of the car's dynamic state and how that state is likely to change over the next few seconds — even milliseconds. And, the combination of ESC and GPS data will enhance the safety system's ability to decide precisely what action the system should take.

The next generation radar sensor

	Delphi Electronically Scanning Radar

Delphi's newest safety systems use an Electronically Scanned Radar or ESR as the prime obstacle detection sensor. While electronic scanning techniques have been used by the military for some time, until recently, they have been too expensive for automotive use.

Electronic scanning is achieved by using a process called receive digital beam forming (RDBF). In this process, a modulated high-frequency signal used with a specially designed antenna provides the ability to shape the radar's beam pattern to scan or focus on an object of interest. Additionally, Delphi's ESR uses a simultaneous transmit and receive-pulse doppler (STAR PD) waveform, providing true simultaneous range and range rate measurements, mitigating ambiguities inherent with frequency modulated continuous wave (FMCW) radars. The RDBF antenna, coupled with STAR PD waveform, provides multi-target discrimination in range, range rate and angle for optimal performance in dense, multi-object scenarios.

Delphi's forward-looking ESR also provides simultaneous long range (200m) and medium range (60m) coverage ahead of the vehicle. The medium range mode provides a wide field of view, making it ideal for managing object detection in curves. This added benefit means that a single sensor can replace multiple radars while achieving the same frontal coverage and system performance at a reduced system cost.

Fusing radar and vision sensing
To further enhance active safety system functionality, Delphi uses a complementary metal oxide semi-conductor, high definition range (CMOS HDR) camera in conjunction with the radar.

	Delphi High Definition Range (CMOS HDR) Camera

Fusing a vision system with radar adds the capability of object classification. With a dynamic range of up to 120dB, the CMOS HDR camera's capability is greater than a standard consumer CCD (charge coupled device) camera at around 60dB. As a result, it can handle both bright and dark conditions as well as the transition between the two which makes it particularly useful in classic motorway situations such as a vehicle entering or exiting a tunnel. It also has a wide field of view, 'seeing' an area 50° horizontally and 35° vertically. Millions of cameras are manufactured annually for things like cell phones, seemingly providing a wide range of camera suppliers. However, the camera, like the radar, must operate reliably in harsh environments for the life of the vehicle. Requirements specify a 10-15-year life plus a substantial thermal operating range — between -40°C and +85°C.

Radar — vision data fusion
The next step is data fusion itself, accomplished by using the sensors (a radar and a camera) we just talked about plus several software modules. Data from each sensor is correlated so that observations from the radar are in sync with those of the camera. The synchronized data then passes to an object refinement module where the data is used to identify and classify the objects ahead of the vehicle. Obstacle features are classified by comparing data parameters and range. This process also helps filter out environmental clutter and noise.

From the object refinement module, the data then passes to the 'situation refinement' module. This step essentially takes all the objects identified in the previous processes and attempts to make a picture out of them. At this point, the system will have assembled a zone of regard populated with the classified objects of interest and is ready to enter the decision-making process. These objects may be cars, trucks or immovable objects such as poles.

Threat Assessment
The safety system objective at this stage is to take appropriate countermeasures. Countermeasures can range from throttle retarding to assisted braking options (brake pre-fill to panic brake assist to autonomous braking) and can also include passive safety system pre-arming and activation, as well as rolling up windows, or actively repositioning seat backs and head rests.

This process includes risk assessment, action planning and decision making. Risk assessment involves establishing confidence levels for the data so that appropriate countermeasures can be defined. For example; confidence levels must be extremely high to signal autonomous emergency braking, but issuing a driver warning signaling an imminent collision does not require that the confidence level be quite as high. What is important is that the driver not be bombarded with annoying false alarms.

Once risk has been assessed, the algorithms define an appropriate mitigation strategy (countermeasure) and develop an action plan before moving to the final stage — the decision-making process. At this point, the strategies of the vehicle manufacturer (VM) are incorporated, because ultimately, it is the VM that decides what action their product will take in any given set of circumstances.

Multiple benefits
What are the benefits of data fusion? Clearly they are immense. When information regarding a target's range, position and velocity obtained from the vehicle's radar is combined with shape and edge information obtained from its vision system, optimal safety system performance is more likely to be achieved. Data fusion produces reliable target selection resulting from vision-based lane assignment and helps avoid false alarms generated by stationary on-road objects which may not be an obstacle.

Let me use an example to clarify. Let's say that a car is approaching a curve on a two lane road. Traffic is moving in both directions. As the car approaches the curve, the safety sensors will detect relevant moving objects in the car's lane. Because of the curve, however, it will also detect irrelevant objects in the opposite lane. What's more, it may even 'see' a road sign directly in its path.

By using the data from both the radar and vision systems, the system can cope with this scenario because it can see the road curving away from the sign. With radar alone, it cannot. The described scenario could also be enhanced by incorporating GPS-based map matching to tell the system about the curve ahead. Vision systems can also help distinguish between objects like a vehicle and a bridge-supporting pillar, for example.

What are the difficulties in robustly designing these systems? Perhaps one of the most important is the variability of real life. It will never be possible to model every possible traffic scenario. For the same reason, system validation is a huge challenge. It requires vast amounts of data acquisition, analysis and offline simulation. Algorithm capability can also be a challenge; not in terms of software development, but in finding sufficient processing power at low enough cost.

Defining the future
Even though we already have safety systems in production that demonstrate the effectiveness of data fusion, there is still more innovation ahead. The technology could be used to enhance safety in side impacts, too. Used to enhance safety in side impacts, data fusion techniques could make it possible to confidently trigger side airbags sooner, providing a greater level of protection for the occupants. Perhaps even more exciting is the prospect that more sophisticated versions of the system capable of recognizing pedestrians are only two or three years away. We have already shown that advanced sensor systems combined with data fusion can be immensely effective and there can be little doubt as to the huge contribution that the technology will make to road safety worldwide.

Note: This article originally appeared in Vision Zero.