The Fuse

Autonomous Cars Are Getting Better, But Standard Testing Metrics Needed

by Matt Piotrowski | February 07, 2017

It’s still the early days of autonomous vehicle development, but data is showing significant progress for the technology. The disengagement report for self-driving cars released last week by the California DMV reflects substantial improvements by a number of automakers in this space. But although the California data provides some insight to industry gains, the real story is the fact that the industry needs consistent monitoring and metrics nationwide to drive a deeper understanding of the technology and its capabilities.

It’s still the early days of autonomous vehicle development, but data is showing significant progress for the technology.

Disengagement reports reveal how often the human driver inside the car has to take over, whether due to operational glitches or an unforeseen problem. Based on the California numbers, Google (whose autonomous program is now called Waymo) was by far the leader in testing with 636,000 autonomous miles driven in 2016 on California roads and a disengagement rate of one for every 5,128 miles. That is a vast improvement from disengaging once in every 1,244 miles the year before. Vehicles for Nissan and Ford also traveled for long stretches without disengaging, while GM logged just under 10,000 miles of autonomous driving. Software discrepancy, technology evaluation management, traffic light detection, and aborted lane change were, among others, reasons listed for disengagements.

While the data gives us some insight how far certain companies are in their testing, the picture given by California’s data must be understood as incomplete. The numbers are not fully descriptive for a variety of reasons. For one, the data collection is occurring only in California, but testing is also taking place in other states. Some companies, wanting to avoid competitors observing their data, have incentive to test more experimental methods outside of California.

Other factors make the California report incomplete, particularly since companies are in different stages of AV development. Tesla, for instance, has only reported 550 miles of autonomous testing in California, based on the DMV records, but most of the company’s data for autonomy and semi-autonomy comes from its customers already using vehicles with that capability.

“The broad definition of ‘disengagements’ limit[s] its utility as a metric, as it allows companies to choose and report different interpretations.”

Another big issue is how little context is given about why the drivers have to take over when they do. “The broad definition of ‘disengagements’ limit[s] its utility as a metric, as it allows companies to choose and report different interpretations,” according to the Commission on Autonomous Vehicle Testing and Safety, a project of SAFE. A rise in disengagements could be a sign of larger problems, but there is not necessarily a connection between reliability and higher disengagement rates. It’s unclear if disengagements occurred because of companies setting low thresholds for human takeover, weather impacts, the fault of another driver on the road, or some other factor. Crucially, companies do not report whether disengagements would have led to collisions had the backup driver not intervened.

The best way forward in measuring autonomous safety testing is for industry to set up an independent data consortium to formulate objective metrics to quantify progress.

According to the Commission, the best way forward in measuring autonomous safety testing is for industry to set up an independent data consortium to formulate objective metrics to quantify progress. In doing so, the industry can have nationwide standards based on industry input instead of a fragmented regulatory framework with laws, such as the ones in California, that provide an inadequate picture. This can help guide best practices for the industry and provide it with needed flexibility but also avoid unnecessary risks from a hasty rollout that could set back AVs.

ADD A COMMENT