Meta AI will be available on Quest headsets in the US and Canada as an experimental feature in August, and will include the Vision capability on Quest 3. Meta AI is the company’s conversational AI assistant powered by its Llama series of open-source large language models (LLMs). In the US & Canada, Meta AI is currently available in text form on the web at meta.ai or in the WhatsApp and Messenger phone apps, and in audio form on Ray-Ban Meta smart glasses via saying “Hey Meta”.
My thoughts on this are WOW this is going to be an incredible invasion of privacy. Meta is going to have an AI that is able to see your surroundings and understanding them. It would seem they could ask the AIs of everyone using a Quest 3 and find out almost any personal detail about the space they are in.
Running locally a LLM is quite the task, and the Quest 3 is way underspecced to do what’s being added. It’ll be the usual sending a picture with a request, having the servers process it and then receiving a response… so while I don’t have any doubt they’ll use all data they receive, the device itself isn’t going to do anything aside from sending pictures.
Very glad to hear this.
If you happen to have an iPhone and want to get a sense of how difficult to run are LLM on a mobile device, there’s a free app https://apps.apple.com/app/id6497060890 that allows just that. If your device has at least 8gb of memory then it will even show the most basic version of LLaMa (just text there), and since everything is done on device, you can even try it in airplane mode. Less meaningful would be running it on a computer, for that I suggest https://lmstudio.ai/ that is very easy to use.
I would think the quest 3 could probably run a light LLM if that was all it was doing, no pass through, tracking or rendering of any kind.
Sounds about right. But a multimodal one? Ehh… sticking with Meta, their smallest LLaMa is a 7b, and as such without any multimodal features it’s already going to use most of the Quest’s 8gb and it would be slow enough that people wouldn’t like it. Going smaller is fun, for example I like (in the app I linked) to use a 1.6b model, it’s virtually useless but it sure can summarize text. And to be fair, there are multimodal ones that could run on the Quest (not fast), but going small means lower quality. For example the best one I can run on my phone takes… maybe 20 seconds? To generate this description “ The image shows three high-performance sports cars racing on a track. The first car is a white Lamborghini, the second car is a red Ferrari, and the third car is a white Bugatti. The cars are positioned in a straight line, with the white Lamborghini in the lead, followed by the red Ferrari, and then the white Bugatti. The background is blurred, but it appears to be a race track, indicating that the cars are racing on a track.” and it’s not bad. But
I’m not sure I’d call it trustworthy :D
Yeah, it would not be at all practical to run an LLM on device, it would just be theoretically possible. Realistically, they want you to have a constant internet connection anyways to use the device so Meta would probably want a cloud reliant solution.