The Problem with Test Driving Autonomous Vehicles

A Google self-driving car like this one collided with a bus in February. Although no one was injured, this incident raised fresh concerns about the reliability of autonomous vehicles and emphasized the need for further testing. (Image courtesy of Google.)

A recent report by the RAND Corporation suggests that self-driving cars may require hundreds of years of test driving to proven safe.

RAND researchers analysed the ability to demonstrate the failure rate of autonomous vehicles to a certain confidence level using the number of miles driven without a failure and a statistical analysis based on binomial distribution.

Test-Driving Autonomous Cars for Safety

In order to demonstrate a failure rate of 1.09 fatalities per 100 million miles with 95 percent confidence, the researchers found that autonomous vehicles would need to be test-driven for 275 million miles without failure. That’s equivalent to a fleet of 100 self-driving cars being driven continuously for 12.5 years.

The four colored lines show results for different levels of confidence. The five dashed vertical reference lines indicate the failure rates of human drivers in terms of fatalities (1.09), reported injuries (77), estimated total injuries (103), reported crashes (190) and estimated total crashes (382). (Image courtesy of RAND Corporation.)

A second analysis sought to discern the failure rate to a 20 percent error tolerance within a two-sided confidence interval of 95 percent on a standard distribution. This analysis found that autonomous vehicles would need to drive over 8.8 billion miles—the equivalent of 100 self-driving cars being test-driven continuously for 400 years.

The three colored lines show results of different levels of precision δ, defined as the side of the CI as a percent of the failure-rate estimate. The five dashed vertical reference lines indicate the failure rates of human drivers in terms of fatalities (1.09), reported injuries (77), estimated total injuries (103), reported crashes (190) and estimated total crashes (382). (Image courtesy of RAND Corporation.)

Comparing Self-Driving Cars with Human Drivers

Practically, however, the benchmark for autonomous vehicle safety is set relative to the safety of human drivers. The researchers found that 500 billion miles of test-driving, or 225 years, would be required to demonstrate with 95 percent confidence that the failure rate of autonomous vehicles is 20 percent lower than an assumed benchmark of 1.09 fatalities per 100 million miles.

The results depend upon the estimated failure rate of autonomous vehicles. This is shown on the horizontal axis and defined as a percent improvement over the human driver failure rate. The comparison can be made to the human driver fatality rate (blue line), reported injury rate (purple line), estimated total injury rate (green line), reported crash rate (red line) or estimated total crash rate (orange line). (Image courtesy of RAND Corporation.)

In order to prove this with a power of 80 percent—that is, an 80 percent probability of rejecting the null hypothesis—a fleet of 100 autonomous cars would need to be continuously test-driven for 11 billion miles, or 518 years.

Can Self-Driving Cars Be Proven Safe?

It’s worth noting that these estimates are optimistic, since they assume that the performance of human drivers is a known and constant value, rather than a statistical conundrum of its own—which is actually the case. The technology for self-driving vehicles is also still evolving, creating a similarly moving target.

Additionally, 95 percent confidence is much lower than level typically required by industry and policy markers, which is upwards of 99.9 percent. All things considered, the findings of the report indicate that it is prohibitively impractical to attempt to demonstrate the reliability of autonomous vehicles through test driving alone.

These results suggest that the need for new approaches to testing, such as simulations, mathematical modelling and human behavioral testing—each of which have their own benefits and challenges. They also indicate the need for new policies from all parties affected by the implementation of these vehicles as well as formal definitions of what constitutes a “safe” self-driving car.

Still, as RAND justice policy distributor James M. Anderson writes, “promise outweighs peril.” The National Highway Traffic Safety Administration reports that 90 percent of crashes are caused by human error, including drunk, distracted, aggressive and impaired driving. These are problems that self-driving cars will never have. Autonomous vehicles have the potential to significantly improve road safety.

We just need to prove it.

The full RAND report is available here.