I heard that Musk was the one that insisted on using video instead of lidar, even though that is clearly wrong. This was a while ago, but now looking back on it, it all makes sense.
It was a supply chain problem. No one could get lidar during Covid. Musk’s solution was to lie and say Tesla didn’t need lidar.
I’m pretty sure this is not true at all.
https://insideevs.com/news/718859/tesla-luminar-top-customer-2024/
“What we’re going to explain to you today is that lidar is a fool’s errand,” said Musk during a 2019 investors’ event. “Anyone relying on lidar is doomed. Doomed. Expensive sensors that are unnecessary.”
It’s just him being crazy. Apparently he does this: He just decides that some particular thing is the right decision, and doubles down on it to the ends of the earth just because he decided it. It apparently really fucks up the engineering at companies he’s involved in, because that is about the worst possible way you could do your engineering.
Because he’s a raging narcicist who thinks he’s the most qualified to run the world and anyone who disagrees must be wrong because how could someone so rich be wrong?
Yes, that was the lie to cover for their part sourcing problem.
That was in May of 2019. It is fairly unlikely that they were having Covid-related supply chain problems at that point that they needed to lie to cover up.
I’m willing to bet that the cars are computationally constrained and it is not about the sensors under the surface. I would not trust a small model and last I checked, Tesla is not running server class GPUs in cars. This has to be a super optimised small AI model in an agentic framework and the results must be faster than realtime. That sounds very much like the bleeding edge for a mass produced device that is several years old. The workflow cannot be safely offloaded and must run locally. The choice does not sound like one of ideals but one of physical constraints. The old hardware is just not up to the task so they pick the lesser evil with the most potential. It won’t matter though. Tesla will flop to bankruptcy in less than 4 years.
There were toys with good enough lidar to do rudimentary navigation 20 years ago. The first and most computationally expensive step of video-based navigation is often turning the image into a depth map. Meanwhile, LiDAR just gives you the depth map directly.
Tensors can’t be interrupted though. They tend to push the cache at the limits of I/O. Processing even a small amount of interrupted sensor data is not like pausing an execution thread in traditional serialized code. It will wind up poisoning part of the cache and cause much slower RAM to get called. Plus, that additional data likely does not fit into the running tensor math anyways.
Just armchair speculating here, but I get the impression compute constraints have always been the big issue causing underperformance and delays with Tesla. The issue is not just computation but also speed.
Toys could use traditional serialized code in a loop. The two workflows are likely incompatible.
If that really is the bottleneck I’d expect them to use FPGAs or customized ASICs instead of a common CPU.
These options do not scale for transformers workloads. I can’t say that I fully understand the reason, but if you search on YT (and get lucky), I have seen someone that was working on this specifically within Altera/Intel that explained why FPGAs do not work for AI. I seem to recall it has to do with power and something else about how scaling works.
The AI tensor bottleneck is the bus width between L2 to L1. You can find info for that one on the Chips and Cheese blog. That bus can’t be made larger without pulling down the core. The processor speeds are just too high and optimized far too much for serial code execution.
For very large AI purposes people use GPU clusters, and a GPU is already very close to a tensor-product-optimized ASIC. There also has been recent work on even more specialized AI chips.
For applications that need to run on a smaller scale, people can and do run things like neural networks on FPGAs.
Go put money on FPGAs for AI like no one else and make your fortune.
The GPU is only a hack as well. It is the best hack at the moment.
Specialized AI hardware will fail too. The CPU itself will be redesigned and satisfy all workloads because splitting workloads across divergent architectures has been done already in the 286-386 era and failed. Any design that is good enough to handle both workloads will absolutely dominate in data centers and therefore everywhere else too. CPU speeds are only for marketing nonsense. This has no real value. Slowing things down will make parallelism easier and allow for a wider cache bus width. Improving cache access will also pay dividends. There are many ways to improve the ALU too. We are still 8 years away from the real fully redesigned hardware that could have started shortly after the Llama model weights dropped. That was the critical moment on the timeline. All edge hardware has a 10 year lead time.
In AI land, programmability has proven to be king so far.
Nvidia GPUs are so dominant because everything is prototyped, then deployed, in PyTorch. I think Google TPUs are a good counter example where a big entity throws tons of money at this issue, and even releases some models optimal for their hardware, yet Flax and TPUs themself gain very little traction and are still incompatible with the new architectures that come out every other day because no one has bothered to port them.
FPGAs take this problem and make it an order of magnitude worse. Very few people know how to port and optimize to them… So they don’t.
There’s definitely some limitations to video as your only input for self driving. There’s simply some information you can’t get from just video alone. Lidar can also give much more detailed ranging information, but its also expensive. So a lot of companies were trying to figure out how to do it without it.
