So was there ever a deal with OpenAI? Nothing in the keynote mentioned them or needs them. If there isn’t a deal, I’d love to know how everyone claiming it was signed on the dotted line was led so far down that garden path.
Siri reviews the request and decided if it can respond on its own or if it needs ChatGPT. It then pops up a dialog asking if it is OK ti send the request to ChatGPT. It will not be the default LLM.
That's also my question. What exactly is apple custom LLM, and what is openAI tech ?
I'm quite confident in the ability of openAI to provide a great usable LLM tech, but much less so of apple. All the demo they've shown in the WWDC could just fall flat if the tech really isn't working well enough in practice. I guess we'll just have to wait and see..
OpenAI is an option when making a query, but Apple made it sound like the first deal they're making, not the tight collaboration everybody was expecting.
They said the models can scale to "private cloud compute" based on Apple Silicon which will be ensured by your device to run "publicly verifiable software" in order to guarantee no misuse of your data.
I wonder if their server-side code will be open-source? That'd be positively surprising. Curious to see how this evolves.
Anyway, overall looks really really cool. If it works as marketed, then it will be an easy "shut up and take my money". Siri seems to finally be becoming what it was meant to be (I wonder if they're piggy-backing on top of the Shortcuts Actions catalogue to have a wide array of possible actions right away), and the image and emoji generation features that integrate with Apple Photos and other parts of the system look _really_ cool.
It seems like it will require M1+ on Macs/iPads, or an iPhone 15 Pro.
You don't even have to buy a new device since it's backwards compatible with A17 Pro and M1, M2, M3 and M4. It feels like the integration of the services are using existing models and integrating the API used traditionally originally from AppleScript but, extending it to LLM or stable diffusion systems. It seems that they want the M4 as soon as possible though for the gaming and cloud pushes.
For those curious, there is in fact a ChatGPT integration.
The way it works is that when the on-device model decides "this could better be answered by chatgpt" then it will ask you if it should use that. They described it in a way which seems to indicate that it will be pluggable for other models too over time. Notably, ChatGPT 4o will be available for free without creating an OpenAI account.
I don't think that 4o will actually be available for free. It seemed like they were quite careful in choosing their words. My guess is 3.5 is free without an account, and accessing 4o requires linking your OpenAI account.
They only mentioned 4o, but they mentioned it explicitly at the start, well before they mentioned one can also tie in to their openAI account, if you have one, at the end of the presentation.
To me that implies 4o by default, but I guess we'll find out.
I'm really curious about this. Framing it as "running a large language model in the cloud" is almost burying the lede for me. Is this saying that in general the client will be able to cryptographically ascertain somehow the code that the server is running? That sounds incredibly interesting and useful outside of this.
It seems like this is an orchestration layer that runs on Apple Silicon, given that ChatGPT integration looks like an API call from that. It's not clear to me what is being computed on the "private cloud compute"?
If I understand correctly there's three things here:
- on-device models, which will power any tasks it's able to, including summarisation and conversation with Siri
- private compute models (still controlled by apple), for when it wants to do something bigger, that requires more compute
- external LLM APIs (only chatgpt for now), for when the above decide that it would be better for the given prompt, but always asks the user for confirmation
The second point makes sense. It gives Apple optionality to cut off the external LLMs at a later date if they want to. I wonder what % of requests will be handled by the private cloud models vs. local. I would imagine TTS and ASR is local for latency reasons. Natural language classifiers would certainly run on-device. I wonder if summarization and rewriting will though - those are more complex and definitely benefit from larger models.
I'm glad that they finally brought a better window tiling system. If that sherlocks a couple of dozen utility apps, I'm OK with that. it has been long enough. Some of those apps will be fine as they will probably provide more controls.
Siri runs on device and on Apple's cloud which should be more private than ChatGPT. ChatGPT integration is a separate feature, and will include other providers in the future too.
I'm so happy about this. Siri has great voice recognition and voice synthesis, but it really struggled with intent, context, and understanding what I wanted it to do.
Combining the existing aspects of Siri with an LLM will, I expect, make it the best voice assistant available.
.. and none of it is open. It's a closed source OS with a proprietary dev cloud service using proprietary model that's only accessible with proprietary sdks.
When does this roll out exactly? And exactly which inference actually is on-device?
I think people have been fooled by marketing for this one and the new Co-Pilot PCs into thinking that most of the AI really is running on-device. The models that run fast locally are still fairly limited compared to what runs in the cloud.
Specifically, the iOS update comes out with the new iPhones (usually dropping the same day the new devices are available) and the other OSes are usually timed to be released with it since features are shared across the OSes (so they want to release them at the same time) and the beta periods are the same.
The public betas will be available later this month. The official OS releases are usually in Sept and Oct. Some of the AI stuff should be available right away but rumors say that some of the more advanced Siri features (like app integration) might not launch until after the first of the year.
At some point I wonder if Apple might share compute across your devices so you can do things on low-power hardware like the Watch while leveraging the CPU on the phone, etc. If Apple's dedicated to on-device compute, this ends up being your own "private cloud" of sorts.
Too many of their devices in the wild are battery powered, and one really nice thing about them is that their sleep-state battery use is incredibly low while maintaining quick wake-up.
Plus it’d be weird UX for all your devices to get worse because the iPad in a drawer somewhere finally ran out of power.
Apple (I reckon): Most people don’t know how to use AI well. It’s our responsibility to pare it all down and release features one by one selling each new software feature as part of next year’s hardware.
I am pretty unhappy with Apple doing the image generation, was really hoping that just would not happen.
But a lot of the other features actually seem useful without feeling shoehorned in. At least so far.
I am hoping that I can turn off the ability to use a server while keeping local processing, but curious what that would actually look like. Would it just say "sorry can't do that" or something? Is it that there is too much context and it can't happen locally or entire features that only work in the cloud?
Edit:
OK how they handle the ChatGPT integration I am happy with. Asks me each time if I want to do it.
However... using recipe generation as an example... is a choice.
Emoji, Memoji, stickers, now gen images. Can’t wait to start receiving them from my dad and my mother-in-law in the most absurd of contexts. Like honestly, I like how much the older relatives enjoy weird, tacky stuff like this.
People have ethical concerns about all the public images that were scraped. Regardless of whether or not we agree with them, it is a pretty popular stance to take.
I don't know why it's that fascinating. Lots of people think some amount of copyright is reasonable but that life of the author + 70 years is far too long.
Exactly, nuance is important. Short term protections are good, but companies shouldn’t be able to keep works from entering public domain for decades on end.
Adobe didn't generate them. Someone uploaded them to their stock photos site.
Nor was their issue that the images were in the style of Ansel Adams, but rather that they used his name. That's not a copyright issue. It's a trademark one.
I wonder if those are to avoid two of the big image generation controversies:
1. Imitation of artists' styles (Make an image in the style of...). The restricted styles are pretty generic, so harder to pin down as being a copy of or imitation of some artist.
2. It's cartoony, which avoids photorealistic but fake images being generated of real people.
It's great that Apple is capitalizing so well on everyone else's inventions, but couldn't they at least pretend they will give something back to the ecosystem?
I wish someone somewhere creates something like intents for the web browser
I would imagine the end goal here is to develop their own internal models (or partner with one of the companies doing open source models) that would be hosted on the Apple Silicon-based cloud they mentioned and then they would not be dependent on anyone else's compute.
microsoft was always going to take SMBs. Data is what makes them useful, so Microsoft keeps their SMBs, Apple gets consumers, Google gets their slice of productivity and android, where their preachy models will let you know if you did a harassment
On the contrary this is probably good for MS. Lots of people just don’t want to buy into the Apple ecosystem. MS is dumping R&D money into this ML stuff, apparently without thinking of an actual product or application first. So, now they can just copy Apple.
It seems like this is what Rabbit's LAM was supposed to be. It is interesting to see it work, and I wonder how it will work in practice. I'm not sold on using voice for interacting with things still.
Image Generation is gross, I really didn't want this. I am not excited to start seeing how many horrible AI images I'm going to get sent.
I like Semantic Search in my photos.
This does seem like the typical Apple polish.I think this might be the first main stream application of Gen AI that I can see catching on.
I like that they finally brought typing interaction to Siri. You won't always need to use voice.
This does look like a real-world implementation of the concept promoted by Rabbit. Apple already had the App Intents API mechanism in place to give them the hooks into the apps. They have also publish articles about their Ferret UI LLM that can look at an app's UI and figure out how to interact with it, if there are no available intents. This is pretty exciting.
It has, but it’s presently an accessibility affordance you have to enable first.
It’s found under device Settings > Accessibility > Siri > “Type to Siri” On/Off
Oh, go figure. Let that be a lesson: if you don't check out the accessibility options, you're missing at least half the cool stuff your phone can do to actually make your life easier.
I had similar reactions, a couple add-ons to make:
1. Yes, App Intents feel like the best version of a LAM we'll ever get. With each developer motivated to log their own actions for Siri to hook into, it seems like a solid experience.
2. Image Gen - yeah, they're pretty nasty, BUT their focus on a "emoji generator" is great. Whatever model they made is surprisingly good at that. It's really niche but really fun. The lifelessness of the generations doesn't matter so much here.
3. Polish - there's so much polish, I'm amazed. Across the board, they've given the "Intelligence" features a unique and impressive look.
I wonder how they will extend this to business processes that are not in their training set.
At https://openadapt.ai we rely on users to demonstrate tasks, then have the model analyze these demonstrations in order to automate them. The goal is similar to Rabbit's "teach mode", except it's desktop only and open source.
I was looking for homepod updates as well - I want to get rid of my Amazon Echo devices with homepods, but siri is #1 Slow #2 Dumb #3 messes up my grocery list. Grocery list and timers are the main use case for Amazon Echos - I really hope apple fixes it soon.
Really excited about semantic index. This should allow for google knowledge graph like features grounded in reality for their llm. However it really depends how well it works.
"To run more complex requests that require more processing power, Private Cloud Compute extends the privacy and security of Apple devices into the cloud to unlock even more intelligence.
With Private Cloud Compute, Apple Intelligence can flex and scale its computational capacity and draw on larger, server-based models for more complex requests."
I thought privacy was really well handled for a high-level overview. Basically it seems like anything thing it can't do on device uses ephemeral private compute with no stored data.
Any data sent to 3rd party AI models requests your consent first.
The details will need to emerge on how they live up to this vision, but I think it's the best AI privacy model so far. I only wish they'd go further and release the Apple Intelligence models as open source.
Considering they spent the first half of that segment throwing shade at people who claim privacy guarantees without any way to verify them, Apple hopefully will provide a very robust verification process.
if the servers are so private, why is on-device such a win? here are some irrelevant distractions:
- the cpu arch of the servers
- mentioning that you have to trust vendors not to keep your data, then announcing a cloud architecture where you have to trust them not to keep your data
- pushing the verifiability of the phone image, when all we ever cared about was what they sent to servers
- only "relevant" data is sent, which over time is everything, and since they never give anyone fine-grained control over anything, the llm will quietly determine what's relevant
- the mention that the data is encrypted, which of course it isn't, since they couldn't inference. They mean in flight, which hopefully _everything_ is, so it's irrelevant
Except the Intent API is completely crippled. Maybe the next big OS will just let the AI parse existing menus and figure out all the potential actions an app can take. Some actions need complex objects, so we need a new general mechanism for AIs to connect to 'exported functions'.
Some general OS rethinking is overdue. Or maybe Android is ready for this? Haven't looked into it since they made development impossible via gradle.
Despite this negativity the announcements were better than expected, rebranding AI is bold and funny. But the future will belong to general Agents, not a hardcoded one as presented.
Apple trying to rebrand AI = "Apple Intelligence" is a totally Apple thing to do.
I'll be curious to see if Apple gets caught with some surprise or scandal from unexpected behavior in their generative models. I expect that the rollout will be smoother than Bing or Google's first attempts, but I don't think even Apple will prove capable of identifying and mitigating all possible edge cases.
I noticed during the livestream that the re-write feature generated some pretty bad text, with all the hallmarks of gen AI content. That's not a good sign for this feature, especially considering they thought it was good enough to include in the presentation.
I noticed this too. I quickly skimmed a rewritten email and it was totally wooden. It's the one where they ask the AI to reword their rant that includes tons of all-caps words.
The output is almost worse, dripping in a passive aggressive tone.
It's bad branding because they can't use the "AI" abbreviation. It's too commonly used to be appropriated by Apple. Honestly calling it "Apple Intelligence" just feels a little lazy.
> the re-write feature generated some pretty bad text, with all the hallmarks of gen AI content
This is the curse of small-language models. They are better suited for constrained output like categorization. Using them for email generation takes... well, that takes courage.
Thankfully, there is an option to use GPT-4o for many of the text generation tasks.
Apple's phones have, for several years now, been on a tick/tock sort of pattern. iPhone N Pro has the new CPU (15 Pro got the A17) and iPhone N uses the prior gen pro CPU (15 got the A16, which was in the iPhone 14 Pro). So this is probably an A17+ feature. If they stick to form, the iPhone 16 will have the A17 processor and the iPhone 16 Pro will have the A18.
That's part of the differentiation between (on the phones) the Pro and non-Pro these days with the Pro getting all the new stuff, and non-Pro getting a partially improved (and partially degraded, usually wrt the camera) Pro from the prior year.
Actually 15 Pro has the A17 “Pro” SoC, while the regular 15 uses the last gen A16 Bionic (Bionic being the regular version). We don’t know whether the next regular iPhone will get the last-gen A17 Pro (which would be a little confusing), or a “last-gen-but-new” A17 Bionic or if they’ll go with 18 Bionic + Pro.
I thought it was underwhelming. The fact that integration with ChatGPT is not seamless pours cold water over it. Siri will seek your permission each time before passing the question to ChatGPT. I can avoid that step by using ChatGPT directly.
I think this is good. For folks like you, that will always be an option. For people that have yet to touch ChatGPT and "still don't know how to access AI" (I've heard this sentiment from many people that couldn't care less about it all) it's a perfect balance. Siri will operate as you expect, until one day it prompts you to pass your question over to ChatGPT. You can opt out or give it a try.
I do agree that the extra tap is a bummer for anyone that wanted ChatGPT baked into the OS, even easier to access than it is in the ChatGPT app.
Personally I feel less and less comfortable giving OpenAI access to my private data tho, so I’m really happy there’s a divide. As you said, if you really just need ChatGPT for something you can open that app. But I’m happy the default isn’t to send all Apple users requests to OpenAI all the time.
On the opposite side, it not being seamless is entirely why I would actually trust using the new Siri with any sensitive data.
I am not entirely sure I will ever actually allow it to connect to ChatGPT for privacy reasons, but having the option when it can't be handled another way is nice.
I imagine this is more a stopgap until more and more of this can happen locally anyways. Especially since it sounds like Siri determins when it should reach out.
1,283 comments
[ 8.2 ms ] story [ 703 ms ] threadhttps://x.com/markgurman/status/1800198524031906258?ref_src=...
Siri reviews the request and decided if it can respond on its own or if it needs ChatGPT. It then pops up a dialog asking if it is OK ti send the request to ChatGPT. It will not be the default LLM.
I'm quite confident in the ability of openAI to provide a great usable LLM tech, but much less so of apple. All the demo they've shown in the WWDC could just fall flat if the tech really isn't working well enough in practice. I guess we'll just have to wait and see..
They gave more space and reverence to Ubisoft.
Sam: https://x.com/sama/status/1800237314360127905
They said the models can scale to "private cloud compute" based on Apple Silicon which will be ensured by your device to run "publicly verifiable software" in order to guarantee no misuse of your data.
I wonder if their server-side code will be open-source? That'd be positively surprising. Curious to see how this evolves.
Anyway, overall looks really really cool. If it works as marketed, then it will be an easy "shut up and take my money". Siri seems to finally be becoming what it was meant to be (I wonder if they're piggy-backing on top of the Shortcuts Actions catalogue to have a wide array of possible actions right away), and the image and emoji generation features that integrate with Apple Photos and other parts of the system look _really_ cool.
It seems like it will require M1+ on Macs/iPads, or an iPhone 15 Pro.
The way it works is that when the on-device model decides "this could better be answered by chatgpt" then it will ask you if it should use that. They described it in a way which seems to indicate that it will be pluggable for other models too over time. Notably, ChatGPT 4o will be available for free without creating an OpenAI account.
To me that implies 4o by default, but I guess we'll find out.
- on-device models, which will power any tasks it's able to, including summarisation and conversation with Siri
- private compute models (still controlled by apple), for when it wants to do something bigger, that requires more compute
- external LLM APIs (only chatgpt for now), for when the above decide that it would be better for the given prompt, but always asks the user for confirmation
No API keys, no prompt engineering or switching between AI models.
It. just. works.
Combining the existing aspects of Siri with an LLM will, I expect, make it the best voice assistant available.
I think people have been fooled by marketing for this one and the new Co-Pilot PCs into thinking that most of the AI really is running on-device. The models that run fast locally are still fairly limited compared to what runs in the cloud.
(Sharing because I had trouble finding it).
Plus it’d be weird UX for all your devices to get worse because the iPad in a drawer somewhere finally ran out of power.
But a lot of the other features actually seem useful without feeling shoehorned in. At least so far.
I am hoping that I can turn off the ability to use a server while keeping local processing, but curious what that would actually look like. Would it just say "sorry can't do that" or something? Is it that there is too much context and it can't happen locally or entire features that only work in the cloud?
Edit:
OK how they handle the ChatGPT integration I am happy with. Asks me each time if I want to do it.
However... using recipe generation as an example... is a choice.
Regardless, even if it wasn't ChatGPT, given the recent problems I would not have used that as one example given that regardless of who it came from.
but my biggest concern is that I think they look tacky, and putting it right in the messaging apps is gonna be ... irritating.
Apple may have actually nailed this one.
Edit: except for image generation. That one sucks
Followed by maybe search engines once it gets to a certain level of quality (which we seem to be a bit far from).
Then either desktop or home(alexa).
Say more? it's just the media thing about intellectual property rights?
Nor was their issue that the images were in the style of Ansel Adams, but rather that they used his name. That's not a copyright issue. It's a trademark one.
1. Imitation of artists' styles (Make an image in the style of...). The restricted styles are pretty generic, so harder to pin down as being a copy of or imitation of some artist.
2. It's cartoony, which avoids photorealistic but fake images being generated of real people.
When they pay us sufficiently
It's great that Apple is capitalizing so well on everyone else's inventions, but couldn't they at least pretend they will give something back to the ecosystem?
I wish someone somewhere creates something like intents for the web browser
Microsoft are so boned. They don't even have a mobile proposition.
It seems like this is what Rabbit's LAM was supposed to be. It is interesting to see it work, and I wonder how it will work in practice. I'm not sold on using voice for interacting with things still.
Image Generation is gross, I really didn't want this. I am not excited to start seeing how many horrible AI images I'm going to get sent.
I like Semantic Search in my photos.
This does seem like the typical Apple polish.I think this might be the first main stream application of Gen AI that I can see catching on.
This does look like a real-world implementation of the concept promoted by Rabbit. Apple already had the App Intents API mechanism in place to give them the hooks into the apps. They have also publish articles about their Ferret UI LLM that can look at an app's UI and figure out how to interact with it, if there are no available intents. This is pretty exciting.
1. Yes, App Intents feel like the best version of a LAM we'll ever get. With each developer motivated to log their own actions for Siri to hook into, it seems like a solid experience.
2. Image Gen - yeah, they're pretty nasty, BUT their focus on a "emoji generator" is great. Whatever model they made is surprisingly good at that. It's really niche but really fun. The lifelessness of the generations doesn't matter so much here.
3. Polish - there's so much polish, I'm amazed. Across the board, they've given the "Intelligence" features a unique and impressive look.
At https://openadapt.ai we rely on users to demonstrate tasks, then have the model analyze these demonstrations in order to automate them. The goal is similar to Rabbit's "teach mode", except it's desktop only and open source.
Looking forward to my house gaining a few IQ points.
I don’t see anything that mentions HomePod specifically but hopefully the updates will come.
“Added ‘start the chronometer’ to your reminders”
"To run more complex requests that require more processing power, Private Cloud Compute extends the privacy and security of Apple devices into the cloud to unlock even more intelligence. With Private Cloud Compute, Apple Intelligence can flex and scale its computational capacity and draw on larger, server-based models for more complex requests."
Any data sent to 3rd party AI models requests your consent first.
The details will need to emerge on how they live up to this vision, but I think it's the best AI privacy model so far. I only wish they'd go further and release the Apple Intelligence models as open source.
They talk about "independent experts" a bit, which I remember being hindered (and sued?) by them rather than supported.
- the cpu arch of the servers
- mentioning that you have to trust vendors not to keep your data, then announcing a cloud architecture where you have to trust them not to keep your data
- pushing the verifiability of the phone image, when all we ever cared about was what they sent to servers
- only "relevant" data is sent, which over time is everything, and since they never give anyone fine-grained control over anything, the llm will quietly determine what's relevant
- the mention that the data is encrypted, which of course it isn't, since they couldn't inference. They mean in flight, which hopefully _everything_ is, so it's irrelevant
If we assume AI will get even 3-4x better, at a certain point, I can't help but think this is the future of computing.
Most users on mobile won't even need to open other apps.
We really are headed for agents doing mostly everything for us.
Some general OS rethinking is overdue. Or maybe Android is ready for this? Haven't looked into it since they made development impossible via gradle.
Despite this negativity the announcements were better than expected, rebranding AI is bold and funny. But the future will belong to general Agents, not a hardcoded one as presented.
I'll be curious to see if Apple gets caught with some surprise or scandal from unexpected behavior in their generative models. I expect that the rollout will be smoother than Bing or Google's first attempts, but I don't think even Apple will prove capable of identifying and mitigating all possible edge cases.
I noticed during the livestream that the re-write feature generated some pretty bad text, with all the hallmarks of gen AI content. That's not a good sign for this feature, especially considering they thought it was good enough to include in the presentation.
The output is almost worse, dripping in a passive aggressive tone.
- HN Guidelines.
This is the curse of small-language models. They are better suited for constrained output like categorization. Using them for email generation takes... well, that takes courage.
Thankfully, there is an option to use GPT-4o for many of the text generation tasks.
I assume it will come to all the iPhone 16’s this fall? Or is Apple Intelligence a Pro Feature?
Either way, my first reaction is that this is going to sell a lot of iPhones.
That's part of the differentiation between (on the phones) the Pro and non-Pro these days with the Pro getting all the new stuff, and non-Pro getting a partially improved (and partially degraded, usually wrt the camera) Pro from the prior year.
I do agree that the extra tap is a bummer for anyone that wanted ChatGPT baked into the OS, even easier to access than it is in the ChatGPT app.
I am not entirely sure I will ever actually allow it to connect to ChatGPT for privacy reasons, but having the option when it can't be handled another way is nice.
I imagine this is more a stopgap until more and more of this can happen locally anyways. Especially since it sounds like Siri determins when it should reach out.