Now Revealed: Why Teslas Have Only Camera-Based Vision

Walter Isaacson shadowed Elon Musk for 2 years to get the material for his bestselling (#2 on the New York Times Best Sellers List) biography. He was granted almost limitless access to the hyperactive mind and businesses of the world’s richest—and perhaps busiest—man. We find out what happened behind the scenes at Twitter, his latest acquisition, and the ruthless cuts in the number of employees. We get a glimpse into why he decides to override recommendations from his best engineers, despite touting engineering as the most noble of professions.

We have been scratching our heads as to why Musk would go exclusively with camera systems since we wrote about it here, after a New York Times investigation in which 19 people who worked at Tesla, speaking only on condition of anonymity out of fear of reprisal, expressed the need for radar. 

Musk was a firm believer in “returning to first principles” to make products simple. One of those principles was human eyesight. It mattered not to Musk that the state-of-the-art approach to sensing in self-driving employed by every other self-driving car manufacturer was multi-sensor.

“Vision became so good that radar actually reduced SNR [signal to noise ratio], so radar was turned off,” said Musk in a tweet in October of 2021. “Humans drive with eyes & biological neural nets, so makes sense that cameras & silicon neural nets are only way to achieve generalized solution to self-driving.”

So, Tesla remains the only car with Level 2 or Level 3 autonomy that uses only camera vision. Driver assistance systems with every other auto manufacturer use a combination of sensor technologies such as radar and ultrasonic  along with cameras. Companies seeking Level 5 autonomy (associated with robotaxis) add LiDAR (more on that later).

Musk was interested in using radar after a fatal collision of a Tesla Model S (equipped with radar and cameras) with a tractor trailer in 2016 when the vision system was fooled by the white color of the trailer and the sky behind it. A team was assembled to deploy radar in production vehicles, but the team leader left the team after 16 months and the rest of the radar team disintegrated soon afterwards, according to the New York Times.

Isaacson reveals the “tug of war” between Tesla’s engineers and Musk that occurred over the years. Sometimes the engineers would pull Musk over to their side and he would relent to using multiple sensors, such as when he allowed the Model S to have 8 cameras plus 12 ultrasonic cameras and 1 forward-facing radar unit. This allowed Tesla’s high-end sedan to “see” in darkness, rain, fog and smoke.

But Musk would eventually pull back and insist on camera-only vision, ordering his autopilot team to improve it. He would use the Los Angeles freeways as his testing grounds.

One part of Musk’s commute from his 9,300-square-foot modern mansion in Bel-Air¹ to SpaceX in Hawthorne, Calif. was bothering him. A stretch of I-405 had faded lane markings that would send his self-driving Tesla Model S into the other lane. Unable to fix this for Musk in the software, Tesla engineers resorted to what Isaacson likens to a Keystone Cops comedy although it more closely resembles a Seinfeld episode². The engineers were about to commandeer a line painting machine, go out at 3 am to close the interstate and repaint the lane markings until a Musk fan in the state’s Department of Transportation agreed to have the painting done in exchange for a tour of SpaceX.

Musk initially considered using the same sensing system as Waymo, Alphabet’s (formerly Google) self-driving car company, but after a disagreement with Larry Page, one of Google’s founders, Musk dropped the idea and went back to camera-only vision.

Underlying the “first principle” approach was an ongoing urge to make products—from cars to rockets—as simple and cheap as possible. In all his endeavors, Musk was to wage war against what he considered complexity, whether it be extra lines of code, extra parts in a vehicle, extra pounds in a rocket, or as we’ll see later, superfluous regulation.

The cost cutting could be bizarre and extreme. A factor of 10 was often invoked.

But Musk had an existential reason for the cutting cost. Having a full sensor stack, the LiDAR, radar, ultrasonics … such as Waymo’s robotaxis drive around with, would add well over $10,000 to the production cost of a vehicle.

Of all the sensors on Waymos, or any of the other robotaxis commonly seen in San Francisco, the LiDAR is the most visible and most expensive. It is the coffee-can-size device with a spinning mirror you see on top of Waymo’s Jaguar I-Pace all-electric SUVs (above). Solid state LiDAR units with no moving parts, aka LiDAR on a chip, promise to bring the price of adding LiDAR way down. Analog Photonics has one such design at the prototype stage, reports the September 2023 issue of IEEE Spectrum, but its mass production is scheduled to start in 2025. But Elon Musk does not have patience.

Cameras had to be good enough. The software would be able to recognize images and calculate distances without the expense of hardware.

Although Musk relented to keep or add radar on some cars in some model years, he would eventually give into his “first principle” approach and “delete radar” from every Tesla model. 

An undated “Vision Update” on the Tesla site announces the return to exclusively camera vision for the Model 3 and Model Y beginning in 2021 and the same for the remaining models, the Model S and Model X, in 2022.

Fight every regulation, said Musk repeatedly to Tesla engineers, even if it is mine. There was no better way to get Musk mad, and sometimes lose your job on the spot, than to quote a regulation and be unable to name the originator of the regulation or the reason for its existence. When questioned on why there were three antennas on a satellite, for example, an engineer who said they were following regulations would be attacked with pointed questions:

“Who made this regulation?”

“Why does it exist?”

“What happens if you ignore it?”

During “surges,” the all-hands-on-deck alarms that Musk would often ring, he would insist that regulations be dismissed wholesale. You could add 10 percent of the regulations back, leaving you with only the necessary regulations.

The only regulations Musk would allow without question would be those imposed by the laws of physics. All others, including regulations meant to ensure safety, such as launch windows that opened between bad weather systems, could be ignored.

Musk had studied physics and materials at the University of Pennsylvania. He has no engineering degree but positions himself as the lead engineer in his companies. He believes that a knowledge of physics supplies all the principles of engineering.

It was not uncommon for Musk to grill engineers on the cost of single parts and suggest alternative materials. His favorite material was stainless steel, which he preferred over carbon fiber on rocket fuel tanks. He would refer to the final cost of a part over the cost of its material as its idiot factor, as in “only an idiot would specify a $1,000 for a part made of $10 worth of aluminum.” One engineer grilled on which parts had the highest idiot factor in a design could not do it on the spot. He suffered from symptoms of PTSD for years after this encounter. 

It may be strange to think of the richest man in the world being so obsessed with the cost of cars and rockets that he would pinch pennies, but the penny-pinching die was cast when both Tesla and SpaceX were in the throes of heavy losses and facing bankruptcy. SpaceX’s first few rockets blew up on the launchpad or shortly thereafter. The Tesla Fremont factory was unable to reach breakeven production rates. It was not until years later that Musk was able to cash in on stock prices that had jumped to 10 times their COVID-19 lows that he rocketed past Jeff Bezos as the world’s richest man.