On device is great, hopefully it will allow for automation scripting that is as fast as running scripts and doesn't require one to upload something to a server for processing.
Got a tip for us?
Let us know
Become a MacRumors Supporter for $50/year with no ads, ability to filter front page stories, and private forums.
Apple Releases Open Source AI Models That Run On-Device
- Thread starter MacRumors
- Start date
- Sort by reaction score
Great idea for this magically AI future everyone is allegedly clamoring for is locked in technically to one vendor. I guess the EU is going to step in? 😉
Nvidia is dominating AI just because they are using Nvidia based ecosystem, software, technology and more. CUDA is a great example. And Nvidia invested AI for a decade. Yes, AI chip or NPU is much more efficient but not powerful enough to beat Nvidia GPU for now and even if they are powerful, they are only limited by their own AI models.
It's better than nothing but it's a long way to go especially since Mac is only limited for 2D based software like video, music, photo, and more.
Hey now I paid for those scratches; I heard they used Steve Jobs personal finger bones to scratch test each screen
And you came to this stunning insight by … counting parameters? 🙄
I guess faster clock speeds automatically means better chips.
1.76 trillion parameters is a little like turbo boost on a Hummer. Wasteful in power, costly and given to delusions.
Apple seems to be staking out ground in the on-device AI model domain as this article demonstrates. Despite the cynical comments and attempt to cast shade on OpenELM, advancements like MLX and Ferret demonstrate that Apple is focused on building infrastructure not just models. Apple has an audience of billions of devices and privacy oriented users that are by themself a huge market. If Apple can deliver on-device, ecosystem-optimized tools, the CUDA advantage could possibly become less of and perhaps even a non-issue. I’m really hoping to see some consequential advancement in Apple ecosystem AI development tools at WWDC. 🙏🏽
Did you miss the part about this being an AI than can run locally on your device? Not everything is about how big your parameters are.
Yes if your phone is three inches thick, has a fan and one terabyte RAM.
You can’t have a full on LLM on a smart phone. It has to be ‘family friendly’ and needs guard rails to prevent it going off the rails. It has to tightly integrate with the device hardware and software without wasting user time.
When I ask Siri for SE Asia travel recommendations it should say ‘You should stay away from Myanmar right now because tourists are being kidnapped and forced to work as scammers on Facebook’.
Siri should not go off the rails and reply ‘don’t go there bro Myanmar is a steaming mountain of **** and I hope someone nukes it soon ohhhh boyyyyy Ima send my my ****ing robot drone army to start a war.’
I've worked on iOS AI Apps for several years now and I've looked long and hard at "on device".
What stops anyone with a decent, unique, intelligence model from putting it on device is having it stolen, re-packaged and resold.
iOS is pretty hardened but not unbreakable.
I asked Apple peeps, on the dev forums, and the answer I got was "yeah, that's DRM, it's not perfect"
Apple's own ML API got an encryption feature to protect proprietary models a couple of years ago but it was limited to only what ML supported.
So I don't expect a mad rush to use any on device LLM's unless there's an impressively secure way of locking in and protecting what they contain.
If the new Apple silicon (M4) has an AI focus then it'll no doubt be encased within a pretty strong silicon encryption. Something that Apple has been doing for things like biometrics, banking and ML.
It's not just about how good is your AI silicon is, it's as much about how well secured your AI model is on silicon for on device to succeed.
What stops anyone with a decent, unique, intelligence model from putting it on device is having it stolen, re-packaged and resold.
iOS is pretty hardened but not unbreakable.
I asked Apple peeps, on the dev forums, and the answer I got was "yeah, that's DRM, it's not perfect"
Apple's own ML API got an encryption feature to protect proprietary models a couple of years ago but it was limited to only what ML supported.
So I don't expect a mad rush to use any on device LLM's unless there's an impressively secure way of locking in and protecting what they contain.
If the new Apple silicon (M4) has an AI focus then it'll no doubt be encased within a pretty strong silicon encryption. Something that Apple has been doing for things like biometrics, banking and ML.
It's not just about how good is your AI silicon is, it's as much about how well secured your AI model is on silicon for on device to succeed.
AI is better than me at making MacRumors posts.
Que? They open sourced webkit and that gave Google a chance to dethrone Internet Explorer with Chrome.
Using the term "AI" in the title for clickbait when you use the correct term (LLM) in the post body 🙄
Well, not a great example as WebKit was originally the open source KHTML component from KDE before Apple got their paws on it...
My experience with my basic M1pro macbook is quite good using llama3 or mistral, so I would expect that an 8 GB iPhone with an optimised model could be quite ok.
From what I remember Apple’s model will stream in and out of memory dynamically instead of hogging all the memory like typical language models do.
Food for thought....what makes you any different than these so called "experts" you speak of? If Apple can't even get siri to do miniscule type functions without screwing that up, that what makes you think they'll get AI right?