The Fuse

Autonomous Vehicles Require Less Human Intervention, But Federal Metrics Necessary

by Matt Piotrowski | February 15, 2018

There are more signs autonomous vehicle (AV) technology is making important progress.

The recently released disengagement report from the California DMV provides an indicator of improvement for manufacturers of AVs. While the new data shows advancement in the volume and quality of AV activity, disengagement reports are an imperfect measure and the reporting does not include activities on private roads and outside California. Several journalistic analyses indicate that a more robust process with better metrics would be needed to better measure safety performance and advance innovation. In other words, the required disengagement reporting has given the public only limited insight into the state of AV technology and the industry will need to develop better protocols and metrics before AV safety can be confidently measured.

The recently released disengagement report from the California DMV provides an indicator of improvement for manufacturers of AVs.

That said, the broad trends in disengagement reporting is indicative of high level trends. Since 2015, the California DMV has required companies to report how many disengagements occur on the state’s roads. Disengagement reports show how often a human driver inside the AV has to take over, whether due to operational glitches or for safety precautions.

Waymo and GM had the lowest disengagement rates, and improved sharply versus 2016. Waymo vehicles drove more than 350,000 miles and logged just 63 disengagements, one for every 5,600 miles. GM’s Cruise said its vehicles traveled 131,676 miles in 2017 with 105 disengagements, or one for every 1,254 miles. See the graphic below from the San Francisco Chronicle.

SFAVgraphic

Not only are vehicles traveling farther without the need for human intervention; a much larger group of companies reported last year. There were 20 companies that submitted disengagement information for 2017, up from seven for 2015. The state has issued AV testing permits to 50 companies as of January. This shows the growth of capital being invested in self-driving technology throughout the industry and companies’ increasing interest in seizing market opportunities.

Reports on disengagements may be one of the best metrics we currently have for AV performance, but they are limited. For one, they do not account for how challenging developers make their AV tests. Second, the data collection is taking place only in California, but testing is occurring on a large scale in other states. Waymo, for instance, is now driving AVs more in Arizona than in California. Third, companies perform testing beyond public roads, using virtual simulation and on test-tracks. Lastly, we do not know exactly why the drivers intervened on the reported disengagements. “The problem is that licensed companies have considerable leeway in interpreting these terms, meaning certain disengagement events might be reported by one company but not another,” Edward Niedermeyer wrote in Bloomberg View.

According to the Commission on Autonomous Vehicle Testing and Safety, a project of SAFE, “The broad definition of ‘disengagements’ limit[s] its utility as a metric, as it allows companies to choose and report different interpretations.” Disengagements could be a sign of major problems, but not necessarily: Companies do not report whether disengagements would have led to an accident if the backup driver did not take over.

“Disengagement or other performance metric reporting would be useful if they would be attached to a process to either evaluate AV safety or facilitate further progress.”

The shortcomings of disengagement data reinforce the need for greater transparency on a federal level. “Disengagement or other performance metric reporting would be useful if they would be attached to a process to either evaluate AV safety or facilitate further progress,” said SAFE’s Vice President, Autonomous Vehicle Amitai Bin-Nun. “However, with disengagements, there’s no next step and the data isn’t good enough to support one. It’s important to get everyone on the same page and start developing objective measures that are useful in AV safety assurance and regulation.”

As Niedermeyer pointed out, the problem of the inadequacy of disengagement data hinders AV advancement for the longer run. “This horse race between some of the biggest companies in the world makes for gripping entertainment, but there is mounting evidence that it does so at the cost of important long-term perspective.”

The Commission has argued that the establishment of an independent data group to develop metrics to quantify progress can provide certainty for the industry and help regulators avoid a fragmented framework with laws, such as those in California, that are insufficient.

Given the growth in self-driving investments, the need for more data is becoming greater as companies move closer to commercialization.

Given the growth in self-driving investments, the need for more data is becoming greater as companies move closer to commercialization of AVs and public opinion is trending in the right directions. It is important that regulators seize the current opportunity to establish best practices for the industry and provide it with the necessary data, information, and flexibility to roll out AVs quickly, efficiently, and safely.

ADD A COMMENT