If it would also show the written text of what it said, faded in as soon as the word is spoken, overlaid on the object it describes, that would help a lot.
Word of warning, if you lock your phone and you're not familiar with the voice over input it can be really annoying to unlock your phone...Seems to turn off face id as well?
You can ask Siri to turn off voice over and all is well.
The YouTube channel by Kristy Viers has been helpful in understanding how they use an iPhone and other devices. Here’s a video on VoiceOver in the Camera app: https://youtu.be/8CAafjodkyE
If you get stuck in VoiceOver, you can ask Siri to "turn off VoiceOver". The touch gestures are completely different to normal iPhone usage, with it turned on.
Quick guide: Single-tap to highlight a widget, double-tap to click, swipe with three fingers to scroll (after first highlighting the part of the screen you want to scroll). To go home, swipe up from the bottom and hold until it vibrates.
The cautions not to use it in navigation or circumstances where you might be harmed ("yellow car driving directly at you at a high rate of speed!") or diagnoses of medical conditions ("might want to get that wart looked at, dude") made me laugh.
The descriptions are shockingly detailed. I wish there was a way you could add these descriptions to the metadata of all the photos in your library, it would be an amazing way to add rich searchable data.
In particular, asking Siri to "show me photos of <x>" where x is a thing that the photo descriptions does recognize, doesn't always work.
Microsoft Seeing AI has similar features; might be of interest to visually impaired people as it’s made more specifically for identifying objects or reading documents
In my anecdotal experience it makes no difference on my 11 Pro. It presumably uses the same hardware that’s already awake all the time listening for “Hey Siri”.
Haha I remember having to test apps, and being completely confused by it in the beginning too. First thing I did was add VoiceOver toggle to Control Center.
Once you get used to it, it's actually kinda amazing. Apple has put so much effort into iOS to make this work well, and there are a ton of features packed into VoiceOver mode.
Most have no idea this even exists, if you're bored you should play around with it a bit more.
Interesting, I pointed it at my son and it immediately accurately described "a child sitting at a white table doing a jigsaw puzzle". I pointed at one of his Lego creations, and immediately "a multi-colour lego construction sitting on a wooden shelf". I was blown away.
Neat. Is my phone classifying all this stuff, or is it sending all my photos to some Apple server to do it? I watched the video in the OP but it didn't go into any technical details.
The ML part is local for the voice transcription but the questions almost always need to be answered from the server. But if you ask to play music or set a timer for example that is all local. Also works in Airplane mode.
All ML on device definitely seems like a goal for Apple[1]. However, right now Siri doesn't work in airplane mode.
Maybe its still too power hungry to run locally? Maybe they still need the data from the requests server side to improve the model?
I hope both Siri and Maps, and especially their integration, i.e. "Siri navigate to ____." "Siri find restaurants nearby (or along the way)" works on device in the near future. Its annoying for the CarPlay integration to stop working when you're in a remote area and need the navigational help.
Siri’s voice translation works in your phone but the answers it needs are all calculated from the server (on the whole). Maps, for example, are all stored server side and route calculation are server side processes as well, as are the locations of restaurants.
There’s a difference been decoding voice requests and being able to answer any questions you may have; you wouldn’t expect Google web search or Wikipedia to work in offline mode, so why do you expect it to work for Siri?
I expect that it will one day soon work offline. Likely not a full web search.
By 202X, while offline my device should have:
1. Enough storage for voice to text models, translation models, local Maps data auto-downloaded when I have reception (including restaurants, reviews, image thumbnails, etc), Wikipedia, wallets (car keys, credit cards, concert/movie tickets, email, daily news, music/podcasts/YouTube that I’ve subscribed to. All of this should be automatically synced when I have reception (podcasts and music already do es this).
2. Enough compute to translate voice to text (and between languages), set a simple timer/alarm, unlock my car, search through Wikipedia, search for restaurants, directions between points on the map (without using traffic data), open an App or play downloaded music/podcasts/YouTube (including the option to chrome-cast without wifi - using BlueTooth (or similar with high bandwidth), NFC/UWB for simple pairing).
3. Auto-upgrade to use the network when it’s available, like use current traffic data for directions, hi-res images/audio/video.
Bonus: When offline, Apps like Spotify, Gmail, etc should also run some local search options for downloaded data.
Bonus: When offline/online, restaurants should have NFC/UWB servers for their menu. My iPhone should just “download” the menu from the POS system. No need for “signal” or wifi.
Bonus 1: you can search your offline library in spotify.
Bonus 2: this used to be common back in the 00s but it was generally decided to be mostly spam.
Bonus: local search for downloaded spotify stuff works without a connection.
It runs on the phone, no doubt using whatever model they use to enable the keyword searching in the camera-roll. Just verified it all works in airplane mode.
It seems to basically be using an image captioning model like this one: https://deepai.org/machine-learning-model/neuraltalk , but possibly trained on more data? I wonder what dataset they’ve used, or if they created it themselves and won’t share it.
51 comments
[ 3.1 ms ] story [ 108 ms ] threadIf it would also show the written text of what it said, faded in as soon as the word is spoken, overlaid on the object it describes, that would help a lot.
You can ask Siri to turn off voice over and all is well.
Right down the bottom of Settings > Accessibility.
Edit: you can also enable screen curtain in iOS (and watchOS!), which means your phone can operate with the screen completely off when in VoiceOver.
Good for privacy / battery!
https://support.apple.com/en-ca/HT201443
Quick guide: Single-tap to highlight a widget, double-tap to click, swipe with three fingers to scroll (after first highlighting the part of the screen you want to scroll). To go home, swipe up from the bottom and hold until it vibrates.
In particular, asking Siri to "show me photos of <x>" where x is a thing that the photo descriptions does recognize, doesn't always work.
"Waterloo patios with friendname" - me looking for when I last saw that person remembering it was on a patio.
"Driveway or sidewalk"
"cottage docks"
"receipts"
"Memes with Ted Cruz fleeing to Mexico"
Though I'm not sure if it comes at the cost of increased battery usage.
Once you get used to it, it's actually kinda amazing. Apple has put so much effort into iOS to make this work well, and there are a ton of features packed into VoiceOver mode.
Most have no idea this even exists, if you're bored you should play around with it a bit more.
Maybe its still too power hungry to run locally? Maybe they still need the data from the requests server side to improve the model?
I hope both Siri and Maps, and especially their integration, i.e. "Siri navigate to ____." "Siri find restaurants nearby (or along the way)" works on device in the near future. Its annoying for the CarPlay integration to stop working when you're in a remote area and need the navigational help.
[1] https://appleinsider.com/articles/18/11/15/apple-considering...
There’s a difference been decoding voice requests and being able to answer any questions you may have; you wouldn’t expect Google web search or Wikipedia to work in offline mode, so why do you expect it to work for Siri?
I expect that it will one day soon work offline. Likely not a full web search.
By 202X, while offline my device should have:
1. Enough storage for voice to text models, translation models, local Maps data auto-downloaded when I have reception (including restaurants, reviews, image thumbnails, etc), Wikipedia, wallets (car keys, credit cards, concert/movie tickets, email, daily news, music/podcasts/YouTube that I’ve subscribed to. All of this should be automatically synced when I have reception (podcasts and music already do es this).
2. Enough compute to translate voice to text (and between languages), set a simple timer/alarm, unlock my car, search through Wikipedia, search for restaurants, directions between points on the map (without using traffic data), open an App or play downloaded music/podcasts/YouTube (including the option to chrome-cast without wifi - using BlueTooth (or similar with high bandwidth), NFC/UWB for simple pairing).
3. Auto-upgrade to use the network when it’s available, like use current traffic data for directions, hi-res images/audio/video.
Bonus: When offline, Apps like Spotify, Gmail, etc should also run some local search options for downloaded data.
Bonus: When offline/online, restaurants should have NFC/UWB servers for their menu. My iPhone should just “download” the menu from the POS system. No need for “signal” or wifi.
Bonus 1: you can search your offline library in spotify. Bonus 2: this used to be common back in the 00s but it was generally decided to be mostly spam. Bonus: local search for downloaded spotify stuff works without a connection.
Does VO in general not work for non-English?