I’m willing to bet that the cars are computationally constrained and it is not about the sensors under the surface. I would not trust a small model and last I checked, Tesla is not running server class GPUs in cars. This has to be a super optimised small AI model in an agentic framework and the results must be faster than realtime. That sounds very much like the bleeding edge for a mass produced device that is several years old. The workflow cannot be safely offloaded and must run locally. The choice does not sound like one of ideals but one of physical constraints. The old hardware is just not up to the task so they pick the lesser evil with the most potential. It won’t matter though. Tesla will flop to bankruptcy in less than 4 years.
There were toys with good enough lidar to do rudimentary navigation 20 years ago. The first and most computationally expensive step of video-based navigation is often turning the image into a depth map. Meanwhile, LiDAR just gives you the depth map directly.
Tensors can’t be interrupted though. They tend to push the cache at the limits of I/O. Processing even a small amount of interrupted sensor data is not like pausing an execution thread in traditional serialized code. It will wind up poisoning part of the cache and cause much slower RAM to get called. Plus, that additional data likely does not fit into the running tensor math anyways.
Just armchair speculating here, but I get the impression compute constraints have always been the big issue causing underperformance and delays with Tesla. The issue is not just computation but also speed.
Toys could use traditional serialized code in a loop. The two workflows are likely incompatible.
These options do not scale for transformers workloads. I can’t say that I fully understand the reason, but if you search on YT (and get lucky), I have seen someone that was working on this specifically within Altera/Intel that explained why FPGAs do not work for AI. I seem to recall it has to do with power and something else about how scaling works.
The AI tensor bottleneck is the bus width between L2 to L1. You can find info for that one on the Chips and Cheese blog. That bus can’t be made larger without pulling down the core. The processor speeds are just too high and optimized far too much for serial code execution.
For very large AI purposes people use GPU clusters, and a GPU is already very close to a tensor-product-optimized ASIC. There also has been recent work on even more specialized AI chips.
For applications that need to run on a smaller scale, people can and do run things like neural networks on FPGAs.
Go put money on FPGAs for AI like no one else and make your fortune.
The GPU is only a hack as well. It is the best hack at the moment.
Specialized AI hardware will fail too. The CPU itself will be redesigned and satisfy all workloads because splitting workloads across divergent architectures has been done already in the 286-386 era and failed. Any design that is good enough to handle both workloads will absolutely dominate in data centers and therefore everywhere else too. CPU speeds are only for marketing nonsense. This has no real value. Slowing things down will make parallelism easier and allow for a wider cache bus width. Improving cache access will also pay dividends. There are many ways to improve the ALU too. We are still 8 years away from the real fully redesigned hardware that could have started shortly after the Llama model weights dropped. That was the critical moment on the timeline. All edge hardware has a 10 year lead time.
In AI land, programmability has proven to be king so far.
Nvidia GPUs are so dominant because everything is prototyped, then deployed, in PyTorch. I think Google TPUs are a good counter example where a big entity throws tons of money at this issue, and even releases some models optimal for their hardware, yet Flax and TPUs themself gain very little traction and are still incompatible with the new architectures that come out every other day because no one has bothered to port them.
FPGAs take this problem and make it an order of magnitude worse. Very few people know how to port and optimize to them… So they don’t.
I really wish I had saved the reference of the guy from Altera explaining why FPGAs simply will not work for models. I don’t have a ton of interest in FPGAs in general… It may have been on Lex Friedman.
The FPGA is ultimately anything, and they work for smaller stuff obviously, but there is some specific reason why they do not scale to work with current models. It might have been the way rotations are done in transformers or something like that. The person was on papers I have seen and skimmed on arxiv. Names and details outside of my curiosity do not stick in my abstractive functional thought and memory.
There’s definitely some limitations to video as your only input for self driving. There’s simply some information you can’t get from just video alone. Lidar can also give much more detailed ranging information, but its also expensive. So a lot of companies were trying to figure out how to do it without it.
I’m willing to bet that the cars are computationally constrained and it is not about the sensors under the surface. I would not trust a small model and last I checked, Tesla is not running server class GPUs in cars. This has to be a super optimised small AI model in an agentic framework and the results must be faster than realtime. That sounds very much like the bleeding edge for a mass produced device that is several years old. The workflow cannot be safely offloaded and must run locally. The choice does not sound like one of ideals but one of physical constraints. The old hardware is just not up to the task so they pick the lesser evil with the most potential. It won’t matter though. Tesla will flop to bankruptcy in less than 4 years.
There were toys with good enough lidar to do rudimentary navigation 20 years ago. The first and most computationally expensive step of video-based navigation is often turning the image into a depth map. Meanwhile, LiDAR just gives you the depth map directly.
Tensors can’t be interrupted though. They tend to push the cache at the limits of I/O. Processing even a small amount of interrupted sensor data is not like pausing an execution thread in traditional serialized code. It will wind up poisoning part of the cache and cause much slower RAM to get called. Plus, that additional data likely does not fit into the running tensor math anyways.
Just armchair speculating here, but I get the impression compute constraints have always been the big issue causing underperformance and delays with Tesla. The issue is not just computation but also speed.
Toys could use traditional serialized code in a loop. The two workflows are likely incompatible.
If that really is the bottleneck I’d expect them to use FPGAs or customized ASICs instead of a common CPU.
These options do not scale for transformers workloads. I can’t say that I fully understand the reason, but if you search on YT (and get lucky), I have seen someone that was working on this specifically within Altera/Intel that explained why FPGAs do not work for AI. I seem to recall it has to do with power and something else about how scaling works.
The AI tensor bottleneck is the bus width between L2 to L1. You can find info for that one on the Chips and Cheese blog. That bus can’t be made larger without pulling down the core. The processor speeds are just too high and optimized far too much for serial code execution.
For very large AI purposes people use GPU clusters, and a GPU is already very close to a tensor-product-optimized ASIC. There also has been recent work on even more specialized AI chips.
For applications that need to run on a smaller scale, people can and do run things like neural networks on FPGAs.
Go put money on FPGAs for AI like no one else and make your fortune.
The GPU is only a hack as well. It is the best hack at the moment.
Specialized AI hardware will fail too. The CPU itself will be redesigned and satisfy all workloads because splitting workloads across divergent architectures has been done already in the 286-386 era and failed. Any design that is good enough to handle both workloads will absolutely dominate in data centers and therefore everywhere else too. CPU speeds are only for marketing nonsense. This has no real value. Slowing things down will make parallelism easier and allow for a wider cache bus width. Improving cache access will also pay dividends. There are many ways to improve the ALU too. We are still 8 years away from the real fully redesigned hardware that could have started shortly after the Llama model weights dropped. That was the critical moment on the timeline. All edge hardware has a 10 year lead time.
In AI land, programmability has proven to be king so far.
Nvidia GPUs are so dominant because everything is prototyped, then deployed, in PyTorch. I think Google TPUs are a good counter example where a big entity throws tons of money at this issue, and even releases some models optimal for their hardware, yet Flax and TPUs themself gain very little traction and are still incompatible with the new architectures that come out every other day because no one has bothered to port them.
FPGAs take this problem and make it an order of magnitude worse. Very few people know how to port and optimize to them… So they don’t.
I really wish I had saved the reference of the guy from Altera explaining why FPGAs simply will not work for models. I don’t have a ton of interest in FPGAs in general… It may have been on Lex Friedman.
The FPGA is ultimately anything, and they work for smaller stuff obviously, but there is some specific reason why they do not scale to work with current models. It might have been the way rotations are done in transformers or something like that. The person was on papers I have seen and skimmed on arxiv. Names and details outside of my curiosity do not stick in my abstractive functional thought and memory.
There’s definitely some limitations to video as your only input for self driving. There’s simply some information you can’t get from just video alone. Lidar can also give much more detailed ranging information, but its also expensive. So a lot of companies were trying to figure out how to do it without it.