Whenever I take a picture with the camera button on the left, it shows a loading indicator and the app crashes. Not a great first impression. Coming from a company the size of Microsoft, such trivial crashes should have been caught.
I figured that was the issue, which is specifically why I included that information. I am guessing that the commenter I replied to may have also been running a beta OS. And it turns out I was correct! But thank you for your snark anyway.
Anirudh and team: Excellent product release. Thanks for doing such important work for marginalized people! We need more tech like this and less like that.
Hopefully this can be expedited by translating from the New Zealand version which is released. Just lower the prior probability of sheep and you're done!
I really hope not. I'm not saying localization isn't important, but holding back an English app because of this is something I don't understand. A huge part of the world speaks English just fine. Release first, localize later if it's not possible right away.
This is probably not the case here, but I hate it when devs forget that the US isn't the entire world that they're releasing their app to.
Also some of us don't even like to have the software localized. English is not my mother tongue and it's not an official language where I live either but I always set the system language to American English on my computers and devices.
You are right. Internationalization might be the word for what I'm talking about, though some might argue that that's not correct either because they might say that l10n is a subset of i18n. Not quite sure. But anyways, yes, l10n in the sense that it works in my part of the world is desirable. I18n if only taken to mean translation (and things like right-to-left and such which arise from supporting certain languages) is what doesn't matter to me.
It's not about just language, of course. Otherwise "localization" would be called "translation". But a machine vision AI app isn't very useful if it cannot recognize local signs, brands, whatever, or pronounce them correctly. It would be simply bad publicity.
It's unlikely to be localisation, because you'd expect to see it in the UK store as well.
Some companies only release iOS applications in certain countries to begin with because they want a gradual rollout rather than doing a global launch all at once. The App Store doesn't give you many options in this respect.
make it available for other countries as well in a machine 'Language' learning mode. The seeing crowd could train it the local terms, pointing at the fridge and dictating Kühlschrank.
This would be a cool project to help others. I would enjoy doing that, maybe gamify it.
I work with a school for the visually impaired in the UK and would definitely like to trial this - also with a team that supports VI children in mainstream schools. Is there any way to get an alert when it appears in the UK appstore?
You can still download it from the US app store if you create an account and link in your phone. Switching between accounts is painful (a lot of password typing is involved) but allow apps from multiples countries on the same iPhone.
Yes, YES, this is what I'm talking about Microsoft. I'm surprised how muted the reaction is from HN here.
On the technical side, this is a perfect example of how AI can be used effectively, and is a (very obvious in hindsight) application of the cutting edge in scene understanding and HCI. There are quite a few recent and advanced techniques rolled into one product here, and although I haven't tried it out yet it seems fairly polished from the video. A whitepaper of how this works from the technical side would be fascinating, because even though I'm familiar with the relevant papers it's a long jump between the papers and this product.
On the social side, I think this is a commendable effort, and a fairly low hanging fruit to demonstrate the positive power of new ML techniques. On a site where every other AI article is full of comments (somewhat rightfully) despairing about the negative social aspects of AI and the associated large scale data collection, we should be excited about a tool that exists to improve lives most of us don't even realize need improving. This is the kind of thing I hope can inspire more developers to take the reins on beneficial AI applications.
These are all definitely applications of ML we've seen around in different places, but this is the first package I've seen to put them together in an integrated way on a smartphone.
I'm also impressed with Microsoft's effort here. It looks pretty nice, especially the different 'channels' metaphor for the various ML processing tasks (Scene, barcode, people, short text etc) for the visually impaired.
Also on the technical side, the fact that Microsoft (Microsoft!) is releasing an iOS-only app, even if it comes from its research department, is maybe more shocking than its open-source releases of late. Who are you, Microsoft?!
Ah the company behind Pocket Casts (love that app). Well I searched through their blog and could only find Amazon App Store drama.
I'm going to assume the original statement is FUD as accused. It does seem incredibly unlikely a dev could draw such a link even if Apple were doing that kind of penalisation (which makes literally no sense to me).
My headphones are out of juice so you have settle for this summary:
> Coincidentally, Russell Ivanovic is a case in point for what could happen if you defy Apple and launch on Android first. Ivanovic had initially been very lucky to have been assigned an Apple Developer Relations representative who gave him exclusive promotional opportunities. Few developers get assigned these representatives. Among the benefits Ivanovic received was the privilege to have Shifty Jelly's apps preloaded on iPads in Apple stores in Australia, a major marketing boost.
> Things went south In 2012, when Ivanovic launched a new version of the Pocket Casts app on the Android Play Store first, rather than Apple's App Store. The launch was a real success, and he publicly shared the good news. Before he knew it, his Apple Developer Relations representative stopped all contact. The representative would not even answer his emails. Ivanovic had been completely shut out.
Ah okay, so at most it's a withdrawal of special treatment, NOT punishing a regular app's position in the app store because they launched in the Android store first, which was how the original accusation sounded.
But thanks for the info, I can see it certainly would stink to be treated that way, losing your benefits and being ignored by your contact.
So a single app developer experienced unknown delays and attributed that to the fact that they launched on Android first... How's that anecdotal evidence going?
I'm sure the part about Apple punishing you for launching on Android first is rubbish, but the second claim is true - App Store rules are that you can't mention the word 'Android'.
Rule 2.3.1:
"don’t include names, icons, or imagery of other mobile platforms in your app or metadata"
And here is a case where they actually enforced it:
I just used my best google-fu and can find no evidence for this. If you're going to make controversial claims like that in the future, include a link to the evidence or you will mostly just end up spreading FUD.
Yep. The iPhone has close to 100% market share among visually impaired users, because the accessibility features are so far in advance of Android. Google have been playing catch-up for years, but they just don't have the obsessive single-minded focus on UI that Apple are famous/notorious for.
When the iPhone originally launched, blindness charities were protesting that the touch screen interface would exclude visually impaired users. Less than a year later, the iPhone was being hailed as one of the greatest advances for visually impaired people since braille. Say what you like about Jobs and Ive, but they set the bar for accessibility.
At this stage it's a long list of small details rather than any specific feature. Apple place an extremely high value on UI consistency, which is disproportionately important for visually impaired users. Accessibility is an intrinsic part of their design culture, rather than something they retrofit on later. Subtle things like menu items being in a logical order make a really big difference when you rely on a screenreader. Apple also work far harder to promote and support accessibility for third-party developers. Accessibility is always a major focus at WWDC and is deeply woven into the Human Interface Guidelines. UIKit is highly accessible by default.
By way of metaphor: Visually impaired people need to be extremely neat and tidy, because they can't easily navigate a messy environment. If you don't put things back on the shelves in exactly the same place every time, you're in deep trouble. Google have a really poor track record of putting things in the same place every time.
For users with other accessibility needs, the size and consistency of the iOS ecosystem is often crucial. iOS gets all the good apps and peripherals first, because it's a much easier development target.
In addition to the brilliant accessibility features, there's a limited set of devices running iOS, all with known camera specs and picture quality, which can definitely help in maintaining some consistency in results by reducing variation in input.
Didn't they release an iOS-only photo app years ago that was also very clever? IIRC it collated thousands of user photos of a single piece of architecture into a 3D model or something
Edit: Just checked App Store. Microsoft has literally hundreds of apps on there. You should probably go look. But I can't find the app I'm talking about lol
To be honest I guess it's most likely a combiation of their bing speech and vision apis, so as there are android sdks for all of them, you could easily reproduce this as an android app.
I have a technical question here: Can this be called a true AI? Because as far as I'm concerned, most companies just use AI and ML interchangeably. I guess these are only applied AI tailored to do a plethora of tasks, but at the end of the day, they're not conscious and aware of their existence.
If I remember correctly, this came out of a OneWeek project - Microsoft's company-wide weeklong hackathon. Very cool to see a final published version of this!
You are right, this project evolved from the week long 2015 company wide Hackathon, with a participation of close to 16,000 employees, where people are allowed to take a week off to build anything of interest. Many accessibility projects have come out, off the top of my head:
(1) Eye Controlled Wheelchair for people with ALS
(2) Color Binoculars - App for people who are colorblind
(3) Hearing AI - App including live speech recognition and sounds recognition for people with profound hearing loss
If I have a friend that is visually impaired and is using this, I have to consent to their phone recording me and analyzing me and sending all of that data off to who knows where.
And this is just from my perspective - someone who is not visually impaired. For the person who is, every single thing they look at and read is going to be recorded and used.
It's an unfortunate situation for people to put in, and I'm sure everyone will choose using improvements like this over not using them. As much as I would love to see a focus on privacy for projects like this, I don't imagine it happening any time soon, given how powerful the data involved is.
I imagine a future where AI assistants like this are commonplace, and there is no escaping them.
As far as public use goes, it's always been considered legal / ethical to photograph people in public places. One does not know where the "information" in a photograph will end up, but since they are already technically in public view, it's something of a moot point.
In the future, most AI will happen locally, not remote, because of privacy, latency and cost. There is a lot of research in adapting models to fit phones (such as reducing the size of the NN by 20x-100x and still having almost all the accuracy).
Just as an extra piece of information for people, the person presenting the videos in that page is Saqib Shaikh, who is a developer at Microsoft. Earlier on HN, there was a really interesting video of him giving a talk about how he can use the accessibility features in visual studio to help him code. https://www.youtube.com/watch?v=iWXebEeGwn0
For years I've wanted a pair of glasses that overlays the world with captions, putting names and "you know them from" next to people, address numbers next to buildings, etc. Google glass made me hopeful, but it fizzled out.
Apparently there is some research and development taking place into a cane that has a sonar or some similar sensor embedded into its tip, and transmits haptic feedback to the user via the handle.
These assistive technologies are fantastic, but I wonder whether a vision-impaired person who has adapted to life before they were available would be weary about adopting them, on the basis that if it breaks or becomes unavailable that they would have maybe lost the skills and sharpness of other senses to compensate?
I'm pretty blown away by this. I took a picture of myself in the mirror with the scene description feature, and it said "probably a man standing in front of a mirror posing for the camera". I took a picture of the room in front of me and it said "probably a living room". Think I'll be experimenting with this for days.
One of my friends from college has limited vision and the feature to read text aloud will be a game changing convenience.
He has a magnifier in his home, but it isn't portable and is limited to working only with documents and images that can lie flat.
edit: After speaking with my friend, he already uses a popular app called KNFB Reader that works very well on short text and documents, but costs $100. On the plus side, it works on Android or iOS.
The use of AI and ML for application purposes is starting to get to a point that it can really be used for problem solving, we did a demo app similar use of this technology, https://zyanya.tech/hashtagger-android
I am going to give Seeing AI a try as well, but I totally understand why a research department would like to have a demo as an Application available for public.
Unimpressed. Took a picture of a ceiling fan and it said, "probably a chair sitting in front of a mirror." Took a pic of a dresser, and it said "a bedroom with a wooden floor." Tried the ceiling fan again and got an equally absurd answer.
138 comments
[ 2.7 ms ] story [ 192 ms ] threadI am guessing you are using iOS 11 Beta. We are currently working on support for iOS 11 beta, and will shortly be updating the app. Stay tuned!
Anirudh from Seeing AI team
It's a really commendable effort from MS. An immense benefit for people who have vision challenges.
Are you guys hiring? If yes then what are the skills you are looking for?
edit: I'm wrong. it's in other stores as well, but not in the Australian app store, which is the one that I tried.
Anirudh from Seeing AI team
I do however prefer a German locale as in number, time, units, etc.
Some companies only release iOS applications in certain countries to begin with because they want a gradual rollout rather than doing a global launch all at once. The App Store doesn't give you many options in this respect.
The process in image: https://www.imore.com/how-download-pokemon-go-canada-uk-and-... I did that to get some Japanese apps.
Things like this make articles like this one seem silly: https://www.madebymany.com/stories/what-if-ai-is-a-failed-dr...
On the technical side, this is a perfect example of how AI can be used effectively, and is a (very obvious in hindsight) application of the cutting edge in scene understanding and HCI. There are quite a few recent and advanced techniques rolled into one product here, and although I haven't tried it out yet it seems fairly polished from the video. A whitepaper of how this works from the technical side would be fascinating, because even though I'm familiar with the relevant papers it's a long jump between the papers and this product.
On the social side, I think this is a commendable effort, and a fairly low hanging fruit to demonstrate the positive power of new ML techniques. On a site where every other AI article is full of comments (somewhat rightfully) despairing about the negative social aspects of AI and the associated large scale data collection, we should be excited about a tool that exists to improve lives most of us don't even realize need improving. This is the kind of thing I hope can inspire more developers to take the reins on beneficial AI applications.
This is the direction that needs to be undertaken with as much or more fervour than business automation applications.
I'm inspired!
I'm also impressed with Microsoft's effort here. It looks pretty nice, especially the different 'channels' metaphor for the various ML processing tasks (Scene, barcode, people, short text etc) for the visually impaired.
Also, Apple will ban your app if it has the word "android" anywhere in it.
Do you have a link?
I'm going to assume the original statement is FUD as accused. It does seem incredibly unlikely a dev could draw such a link even if Apple were doing that kind of penalisation (which makes literally no sense to me).
https://www.relay.fm/inquisitive/31
(I'll probably listen anyway since I like those guys, but still ...)
> Coincidentally, Russell Ivanovic is a case in point for what could happen if you defy Apple and launch on Android first. Ivanovic had initially been very lucky to have been assigned an Apple Developer Relations representative who gave him exclusive promotional opportunities. Few developers get assigned these representatives. Among the benefits Ivanovic received was the privilege to have Shifty Jelly's apps preloaded on iPads in Apple stores in Australia, a major marketing boost.
> Things went south In 2012, when Ivanovic launched a new version of the Pocket Casts app on the Android Play Store first, rather than Apple's App Store. The launch was a real success, and he publicly shared the good news. Before he knew it, his Apple Developer Relations representative stopped all contact. The representative would not even answer his emails. Ivanovic had been completely shut out.
Source: http://www.elischiff.com/blog/2015/3/24/fear-of-apple
But thanks for the info, I can see it certainly would stink to be treated that way, losing your benefits and being ignored by your contact.
I am not sure if this is "withdrawal of special treatment". These guys made great ios apps, both before and after they started making Android apps.
And it was those things which they lost right, rather than some kind of search position punishment?
Rule 2.3.1:
"don’t include names, icons, or imagery of other mobile platforms in your app or metadata"
And here is a case where they actually enforced it:
http://bgr.com/2015/05/21/app-store-rejection-android-ios/
/conspiracyfallacysarcasm :)
https://blog.shiftyjelly.com
When the iPhone originally launched, blindness charities were protesting that the touch screen interface would exclude visually impaired users. Less than a year later, the iPhone was being hailed as one of the greatest advances for visually impaired people since braille. Say what you like about Jobs and Ive, but they set the bar for accessibility.
(Said as a former nexus 5 owner)
By way of metaphor: Visually impaired people need to be extremely neat and tidy, because they can't easily navigate a messy environment. If you don't put things back on the shelves in exactly the same place every time, you're in deep trouble. Google have a really poor track record of putting things in the same place every time.
For users with other accessibility needs, the size and consistency of the iOS ecosystem is often crucial. iOS gets all the good apps and peripherals first, because it's a much easier development target.
Lol
No it is not
Edit: Just checked App Store. Microsoft has literally hundreds of apps on there. You should probably go look. But I can't find the app I'm talking about lol
EDIT 2: Found a story on it! 11 years ago!! https://arstechnica.com/uncategorized/2006/07/7381/
It was called Photosynth and it seems to have disappeared from the Internet :(
EDIT 3: crap they just shut it down THIS YEAR! https://blogs.msdn.microsoft.com/photosynth/2017/02/06/micro... It was cool as hell!
Found a 10 year old demo of PhotoSynth here: https://www.ted.com/talks/blaise_aguera_y_arcas_demos_photos...
Reminder: This is a Microsoft marketing video, they are never like the actual product. See Kinect and Hololens teasers for more.
This will help so many people.
(1) Eye Controlled Wheelchair for people with ALS
(2) Color Binoculars - App for people who are colorblind
(3) Hearing AI - App including live speech recognition and sounds recognition for people with profound hearing loss
(4) Learning Tools for OneNote - https://www.onenote.com/learningtools
(5) Dictate - Speech based keyboard control to type emails/documents ( http://dictate.ms), built originally for people with hand dexterity issues
A few of my peers now work full time on projects originated at the hackathon.
- Anirudh from Seeing AI team
And this is just from my perspective - someone who is not visually impaired. For the person who is, every single thing they look at and read is going to be recorded and used.
It's an unfortunate situation for people to put in, and I'm sure everyone will choose using improvements like this over not using them. As much as I would love to see a focus on privacy for projects like this, I don't imagine it happening any time soon, given how powerful the data involved is.
I imagine a future where AI assistants like this are commonplace, and there is no escaping them.
Also, check this blog from a blind user and some interesting comments: http://chrishofstader.com/seeing-ai-first-impressions/
Screw Siri, thats a real AI assistant :)
These assistive technologies are fantastic, but I wonder whether a vision-impaired person who has adapted to life before they were available would be weary about adopting them, on the basis that if it breaks or becomes unavailable that they would have maybe lost the skills and sharpness of other senses to compensate?
http://imgur.com/jxsWrEq
http://www.orcam.com/
I would hope it could get this one right, there's a massive amount of training data to recognize it.
He has a magnifier in his home, but it isn't portable and is limited to working only with documents and images that can lie flat.
edit: After speaking with my friend, he already uses a popular app called KNFB Reader that works very well on short text and documents, but costs $100. On the plus side, it works on Android or iOS.
I am going to give Seeing AI a try as well, but I totally understand why a research department would like to have a demo as an Application available for public.
Deleted app.