Tesla's autopilot, on the road

[GRA]

I like your solution about stricter licensing requirements. It would be very effective, but also impossible to implement without causing a major uproar. And all the other tech solutions just won't be as effective as taking the 4-D drivers out of the loop. So we're back to the disagreement on timing and how many lives would benefit from the aggressive approach.

As for the quality of the accident statistics, it is self-consistent, since Tesla's numbers are only for Teslas (search for "Tesla autopilot quarterly safety report"). The difference between the numbers is attributable strictly to autopilot. The past 3 autopilot safety reports have been trending lower (3.34 per-million in Q3, 2.91 per-million in Q4, and 2.87 per-million in Q1), but still consistent fewer accidents than miles driven with-out A/P engaged (1.92 per-million in Q3, 1.58 per-million in Q4, and 1.76 per-million in Q1). The next quarterly report is due in 2 months. I predict it will stay the same, or trend down slightly, as more first-time A/P owners learn first hand what A/P is capable of. As the ratio of new Tesla owners to existing ones grow lower, the statistic should improve.

Again, until Tesla allows some independent agency to examine all the data, it's just Tesla hot air. They need to put up or shut up, voluntarily or by getting sued under truth in advertising laws and be forced to do so. BTW, the fact that it's only for Teslas is part of the problem with their numbers, as it has to be compared with specifically comparable types, demographics, conditions etc. This was one of the problems cited with Elon's A/P safety claims a couple of years ago, and again last fall, e.g.:

How safe is Tesla Autopilot? Parsing the statistics (as suggested by Elon Musk)

https://www2.greencarreports.com/news/1 ... -elon-musk

and

TESLA'S AUTOPILOT REPORT MAKES BIG SAFETY CLAIMS WITH LITTLE CONTEXT

https://www.wired.com/story/tesla-autop ... tatistics/

. . . The safety report compares that 1.92 million miles per incident figure to data from the National Highway Traffic Safety Administration. It says NHTSA figures show “there is an automobile crash every 492,000 miles." (Tesla apparently used the NHTSA’s public database to derive this number.) That indicates drivers in other manufacturers’ cars crash nearly seven times more often than drivers using Autopilot.

But again, a closer look raises questions. A broad comparison of Tesla with everyone else on the road doesn’t account for the type of car, or driver demographics, just for starters. A more rigorous statistical analysis could separate daytime versus nighttime crashes, drunk drivers versus sober, clear skies versus snow, new cars versus clunkers, and so on. More context, more insight.

“It’s silly to call it a vehicle safety report,” says David Friedman, a former NHTSA official who now directs advocacy for Consumer Reports. “It’s a couple of data points which are clearly being released in order to try to back up previous statements, but it’s missing all the context and detail that you need.”

Tesla’s one-page report comes the day after Consumer Reports published its comparison of “semiautonomous” systems that let drivers take their hands off the wheel but require them to keep their eyes on the road. That ranking put Cadillac’s Super Cruise in first place and Autopilot in second, followed by Nissan’s Pro Pilot Assist and Volvo’s Pilot Assist. It evaluated each on how it ensures the human is monitoring the car as well as its driving. . . .

. . . it could be that its Autopilot system is making highway driving safer, perhaps by reducing driver fatigue or reducing rear-end collisions. But this report isn’t enough to show that. Friedman says he was hoping for more. He wants Tesla to give its data to an academic, who can do a rigorous, independent, statistical analysis. “If the data shows that Autopilot is delivering a safety benefit, then that’s great. . . .”

Tesla has always moved faster than the mainstream auto industry and deserves credit for acceleration the adoption of electric driving, software updates, and self-driving features. But if it wants to be congratulated for making roads safer, it has to cough up more data.

[EVDrive]

@GRA, you have to be the biggest troll in the history of this site.

You have all these nonsensical opinions about everything. I've owned 4 electric cars. How many have you owned or leased? What's that silence? You've never even had an electric car. Your opinions are based on your own imagination and not grounded in experience or fact. Why don't you stop trolling this site with your ludicrous opinions. Speaking of hot air, you have more hot air than a blow dryer.

I have autopilot on my tesla. It's awesome. You have to watch it because it has issues detecting median walls, road debri etc. It does a fantastic job of making stop and go traffic un-stressful. So happy to have a car with autopilot and 250kwh fast charging. Go Tesla go.

[LeftieBiker]

He is posting links, not just writing his opinions. That's hardly unusual here.

[Oils4AsphaultOnly]

I like your solution about stricter licensing requirements. It would be very effective, but also impossible to implement without causing a major uproar. And all the other tech solutions just won't be as effective as taking the 4-D drivers out of the loop. So we're back to the disagreement on timing and how many lives would benefit from the aggressive approach.

As for the quality of the accident statistics, it is self-consistent, since Tesla's numbers are only for Teslas (search for "Tesla autopilot quarterly safety report"). The difference between the numbers is attributable strictly to autopilot. The past 3 autopilot safety reports have been trending lower (3.34 per-million in Q3, 2.91 per-million in Q4, and 2.87 per-million in Q1), but still consistent fewer accidents than miles driven with-out A/P engaged (1.92 per-million in Q3, 1.58 per-million in Q4, and 1.76 per-million in Q1). The next quarterly report is due in 2 months. I predict it will stay the same, or trend down slightly, as more first-time A/P owners learn first hand what A/P is capable of. As the ratio of new Tesla owners to existing ones grow lower, the statistic should improve.

Again, until Tesla allows some independent agency to examine all the data, it's just Tesla hot air. They need to put up or shut up, voluntarily or by getting sued under truth in advertising laws and be forced to do so. BTW, the fact that it's only for Teslas is part of the problem with their numbers, as it has to be compared with specifically comparable types, demographics, conditions etc. This was one of the problems cited with Elon's A/P safety claims a couple of years ago, and again last fall, e.g.:

How safe is Tesla Autopilot? Parsing the statistics (as suggested by Elon Musk)

https://www2.greencarreports.com/news/1 ... -elon-musk

and

TESLA'S AUTOPILOT REPORT MAKES BIG SAFETY CLAIMS WITH LITTLE CONTEXT

https://www.wired.com/story/tesla-autop ... tatistics/

. . . The safety report compares that 1.92 million miles per incident figure to data from the National Highway Traffic Safety Administration. It says NHTSA figures show “there is an automobile crash every 492,000 miles." (Tesla apparently used the NHTSA’s public database to derive this number.) That indicates drivers in other manufacturers’ cars crash nearly seven times more often than drivers using Autopilot.

But again, a closer look raises questions. A broad comparison of Tesla with everyone else on the road doesn’t account for the type of car, or driver demographics, just for starters. A more rigorous statistical analysis could separate daytime versus nighttime crashes, drunk drivers versus sober, clear skies versus snow, new cars versus clunkers, and so on. More context, more insight.

“It’s silly to call it a vehicle safety report,” says David Friedman, a former NHTSA official who now directs advocacy for Consumer Reports. “It’s a couple of data points which are clearly being released in order to try to back up previous statements, but it’s missing all the context and detail that you need.”

Tesla’s one-page report comes the day after Consumer Reports published its comparison of “semiautonomous” systems that let drivers take their hands off the wheel but require them to keep their eyes on the road. That ranking put Cadillac’s Super Cruise in first place and Autopilot in second, followed by Nissan’s Pro Pilot Assist and Volvo’s Pilot Assist. It evaluated each on how it ensures the human is monitoring the car as well as its driving. . . .

. . . it could be that its Autopilot system is making highway driving safer, perhaps by reducing driver fatigue or reducing rear-end collisions. But this report isn’t enough to show that. Friedman says he was hoping for more. He wants Tesla to give its data to an academic, who can do a rigorous, independent, statistical analysis. “If the data shows that Autopilot is delivering a safety benefit, then that’s great. . . .”

Tesla has always moved faster than the mainstream auto industry and deserves credit for acceleration the adoption of electric driving, software updates, and self-driving features. But if it wants to be congratulated for making roads safer, it has to cough up more data.

Both of those reports are quibbling about the details and lack of transparency. Neither articles counter my self-consistency claim. Tesla knows how many miles their cars were driven with autopilot versus without. Those were the numbers that I pointed out, as they were unambiguous. I specifically didn't point out the NHTSA statistics, because I agree that it doesn't seem to be directly comparable. So focusing on just the A/P vs Non-A/P Tesla drivers, regardless of how an "incident" was defined, there was roughly a 40% reduction in those incidents with autopilot enabled versus not. You can't refute that even if you don't trust the method used for defining an "incident".

And just to quibble about the quibbling. Mr. Noland's mistake about the "small" sample size of # of number of deaths is that he was looking at the numerator and not the denominator. Although smaller than the data sets for other vehicles, 222 million miles of A/P driving (by July 2018) isn't a small sample size. The A/P miles driven had ballooned to 1billion miles by Nov 28, 2018. Even if you include Gao Yaning and Walter Huang (individuals who abused A/P just like Joshua Brown) to the statistics, the stat is now 333 million miles between deaths. Jeremy Banner's death came in Mar '19, after an additional 1B miles of A/P driving. Is 4 A/P driver deaths enough to be statistically relevant now? How about this stat - there are now 5x more fatalities (21 versus 4 per ElonBachman) from wreckless Tesla drivers than there were from A/P. Note that all the passenger/pedestrian/cyclist deaths were due to humans driving behind the wheel, not A/P.

[Evoforce]

You have to wonder why GRA or anyone else without a Tesla wouldn't consult Tesla owners on this site... Maybe most of us have first hand experience and would completely disagree.

[EVDrive]

@Evoforce, what you are saying makes sense. I’ll trust firsthand experience with autopilot providing lane keep assist, lane changes and auto breaking. In fact I upgraded to full self driving recently so I can get a better computer vision computer in my Tesla. I put my money where by beliefs are. I’m not going to trust my life to autopilot at the moment, but I will let it help me steer along the freeway.

I’m sure I could web link some stuff with my comment, aparently posting links here makes you believable.

[Evoforce]

@Evoforce, what you are saying makes sense. I’ll trust firsthand experience with autopilot providing lane keep assist, lane changes and auto breaking. In fact I upgraded to full self driving recently so I can get a better computer vision computer in my Tesla. I put my money where by beliefs are. I’m not going to trust my life to autopilot at the moment, but I will let it help me steer along the freeway.

I’m sure I could web link some stuff with my comment, aparently posting links here makes you believable.

Agreed! But I give GRA a lot of credit because he does do a good job of finding content and posting it to support his beliefs...

[GRA]

Both of those reports are quibbling about the details and lack of transparency. Neither articles counter my self-consistency claim. Tesla knows how many miles their cars were driven with autopilot versus without. Those were the numbers that I pointed out, as they were unambiguous. I specifically didn't point out the NHTSA statistics, because I agree that it doesn't seem to be directly comparable. So focusing on just the A/P vs Non-A/P Tesla drivers, regardless of how an "incident" was defined, there was roughly a 40% reduction in those incidents with autopilot enabled versus not. You can't refute that even if you don't trust the method used for defining an "incident".[/.quote]
But the question is where were those miles driven, by whom, and when? You say the articles are 'quibbling' about the details, but statistics is all about the details - you can't ignore them if you are to have something other than meaningless numbers.

And just to quibble about the quibbling. Mr. Noland's mistake about the "small" sample size of # of number of deaths is that he was looking at the numerator and not the denominator. Although smaller than the data sets for other vehicles, 222 million miles of A/P driving (by July 2018) isn't a small sample size. The A/P miles driven had ballooned to 1billion miles by Nov 28, 2018. Even if you include Gao Yaning and Walter Huang (individuals who abused A/P just like Joshua Brown) to the statistics, the stat is now 333 million miles between deaths. Jeremy Banner's death came in Mar '19, after an additional 1B miles of A/P driving. Is 4 A/P driver deaths enough to be statistically relevant now? How about this stat - there are now 5x more fatalities (21 versus 4 per ElonBachman) from wreckless Tesla drivers than there were from A/P. Note that all the passenger/pedestrian/cyclist deaths were due to humans driving behind the wheel, not A/P.

We all know that humans do stupid things, the question is does A/P cause more accidents than it prevents in the same situations compared to humans, or the reverse? Only a rigorous statistical analysis can determine that, and Tesla's not noted for its rigor with numbers, which is one reason why independent analysis is required.

[GRA]

You have to wonder why GRA or anyone else without a Tesla wouldn't consult Tesla owners on this site... Maybe most of us have first hand experience and would completely disagree.

I'm happy to consult Tesla or any other car's owners for their experience and opinions where relevant, but their evidence is inevitably anecdotal and suffers from self-selection bias, as is the case with owners of any other make or model of car. Actual safety as opposed to perceived safety is all about statistics, e.g. the example of flying versus driving which I cited back a few posts.

As another example, let's look at the level of reliability I mentioned as needed or required for AVs to be acceptably safer than humans. I said I consider six nines* a bare minimum, but seven or eight nines as is typical of aviation safety-of-life critical systems is probably what's needed. There are something over 260 million light duty vehicles in the U.S. fleet, and Americans take an average of 1.1 billion car trips/day. The forecast is that AVs may well increase that number, although many of the trips would be without occupants. If the entire fleet were AVs, reliablity was six nines, and there was only a single potential interaction where a failure would cause an accident per trip, then we could expect AVs to cause 1,100 accidents per day. In reality, the average trip involves multiple failure opportunities, so you can multiply that 1,100 by whatever figure you think is likely.

With human drivers, auto accidents currently kill a little over 100 people/day in the U.S. Most accidents don't involve fatalities or serious injuries, so the question then is are the accidents that AVs will get into on average more or less severe than those involving human drivers?


*If you think six nines is excessive, don't take my word for it:

This is the true problem of autonomy: getting a machine learning system to be 99% correct is relatively easy, but getting it to be 99.9999% correct*, which is where it ultimately needs to be, is vastly more difficult. One can see this with the annual machine vision competitions, where the computer will properly identify something as a dog more than 99% of the time, but might occasionally call it a potted plant. Making such mistakes at 70 mph would be highly problematic.
https://www.tesla.com/support/correctio ... riving-car

*i.e. six nines

[Oils4AsphaultOnly]

Both of those reports are quibbling about the details and lack of transparency. Neither articles counter my self-consistency claim. Tesla knows how many miles their cars were driven with autopilot versus without. Those were the numbers that I pointed out, as they were unambiguous. I specifically didn't point out the NHTSA statistics, because I agree that it doesn't seem to be directly comparable. So focusing on just the A/P vs Non-A/P Tesla drivers, regardless of how an "incident" was defined, there was roughly a 40% reduction in those incidents with autopilot enabled versus not. You can't refute that even if you don't trust the method used for defining an "incident".[/.quote]
But the question is where were those miles driven, by whom, and when? You say the articles are 'quibbling' about the details, but statistics is all about the details - you can't or ignore them if you are to have something other than meaningless numbers.

And just to quibble about the quibbling. Mr. Noland's mistake about the "small" sample size of # of number of deaths is that he was looking at the numerator and not the denominator. Although smaller than the data sets for other vehicles, 222 million miles of A/P driving (by July 2018) isn't a small sample size. The A/P miles driven had ballooned to 1billion miles by Nov 28, 2018. Even if you include Gao Yaning and Walter Huang (individuals who abused A/P just like Joshua Brown) to the statistics, the stat is now 333 million miles between deaths. Jeremy Banner's death came in Mar '19, after an additional 1B miles of A/P driving. Is 4 A/P driver deaths enough to be statistically relevant now? How about this stat - there are now 5x more fatalities (21 versus 4 per ElonBachman) from wreckless Tesla drivers than there were from A/P. Note that all the passenger/pedestrian/cyclist deaths were due to humans driving behind the wheel, not A/P.

We all know that humans do stupid things, the question is does A/P cause more accidents than it prevents in the same situations compared to humans, or the reverse? Only a rigorous statistical analysis can determine that, and Tesla's not noted for is rigor with numbers, which is one reason why independent analysis is required.

So you're arguing that the A/P numbers are based on road situations that have been pre-selected to be easier for the system to handle, and that the human drivers have accepted responsibility for the trickier, and thus more accident prone situations, thereby skewing the statistic?

I can agree with that criticism. But the fault of that "lack of rigor" in the data isn't entirely with Tesla's stats (which would be heavily skewed towards highway-only miles), but with the human driven stat. Does the NHTSA data differentiate between surface street accidents versus highway accidents? You might get more comparable results that way instead.