Launch HN: Mirror (YC W17) – turn a selfie into emoji that look like you
We are launching an app (Mirror) that uses a single selfie to create ~250 emoji/stickers that look like you and your friends.
We previously started a large online travel company in Russia (Ostrovok, >$500m in sales this year) and while there got very interested in AI as a tool to solve business objectives like predicting cancelations or conversion rates to inform marketing spend. And when we started something new, we wanted a product that had extremely high-frequency usage to build feedback loops into + something that a lot of humans really want to do, but that AI can do better.
Emoji keyboards sound like trivial toys, but this is actually a path to teaching AI one of the most complex human social behaviors. Offline, our faces communicate the majority of information that passes between us; far more than words do. In fact there is a whole piece of our brain (fusiform gyrus) dedicated to facial analysis and if lesioned humans lose ability to read faces while losing nothing else. Basically humans pay closer attention to faces than to any other object in the universe – from the day we are born.
AI needs data that informs it whether its decisions are right. And emoji are by far the largest structured source of facial data + usage in the world – >15 billion sent a day. Getting that ocean of data will help train AI that can solve all kinds of future problems – from VR characters that look like you in the gameworld, to personalized advertising that has your face in it the way you like, and a lot of other things. Everyone asks us why after starting a big “serious" company, we decided to work with emoji, and this is the reason.
On to the actual product. We think custom emoji apps like Bitmoji are a good step towards bringing our real, unique faces into messaging. We take this further and let you:
- Get your emoji in 3 seconds with one tap.
- Not spend mental energy wondering what your head shape/nose shape etc. are
- Add your friends/family (soon your pets and more) to your own emoji in seconds. Manually constructing 10+ friends is just too hard.
- Use all text-containing emoji in 8 languages (will expand to >25 shortly). We really have no idea why Bitmoji and other such apps tend to be English-only.
- Get emoji that become more and more recognizable/liked with time by learning from data (explicit choices + implicit metrics). For example we see that people want larger-than proportional eyes and darker skin. Except in Japan where it seems users want lighter skin. Really excited to find more like this with usage data.
Google Allo (and some others) tried doing this but got very little done. The reason is that while the idea is extremely obvious, the tech is quite hard. Here's why:
- You have to get a lot of facial features right. Some of these (e.g. hair, facial hair) are very diverse across age/sex/ethnicity. Some (e.g. head shape) are 3d but need to be inferred from shadows in a 2d image. Some (e.g. lower eyelids or wrinkles) can be detected but have a tradeoff of authenticity vs beauty. Some (colors of skin, hair, lips, eyes, eyebrows) are highly dependent on lighting conditions. All are must-haves. And this is still not enough – you have to detect and place wrinkles, birthmarks, sunspots etc. too
- You have to compensate for a lot of variability in selfie-taking – lighting/shadows, turn/pitch/tilt, camera distance, device camera type, background, whether the user is smiling or not etc.
- You have to use one photo to infer and reconstruct how that person looks with different emotions.
- There’s a whole dedicated part of the brain for this task, with a lot of neural machinery (fusiform gyrus). Humans can’t formally explain how they recognize other humans, but are very demanding to result quality. Humans take our exceptional skill at face recognition for granted, although for us a face is THE most well-perceived object in the universe. Think about it: can you instantly recognize a piece of a random wall that looks...
80 comments
[ 1.9 ms ] story [ 157 ms ] threadMaybe clarify that, or open it up to the rest of us :)
I installed it but wasn't too impressed with the quality of the emoji. (This could be because I look kind of weird; eg I have pink hair.) It's really easy to take photos and see how the emoji look; I encourage other people to try that out.
I'm excited to see whether this technology ends up working well in future.
yea we don't have pink hair – basically it was really hard to get the color palettes working right in all kinds of different lighting conditions, so we didn't include rare/custom hair.
we'll get better at it :) thanks for using the product!!
Very interesting problem set. I look forward to seeing what you do with it.
it does tap into a lot of human emotions. one of the reasons we decided to pursue this area.
While the idea is admirable, there is very little scope for error in a space as personal as thjs
this is a beta, we will improve quickly.
Not really what I was expecting. Glad I wasn't showing him the screen at the time, at least. But you might want to fix that.
thank you, very useful. and sorry you had that experience.
> AI needs data that informs it whether its decisions are right
Well, aren't the emojis all out there for you to analyze? Isn't there a lot of videos with real faces for you to analyze?
> VR characters that look like you in the gameworld
XBox does this today, doesn't it?
> personalized advertising that has your face in it the way you like
I couldn't care less about advertising. Maybe you're pitching to investors?
> Everyone asks us why after starting a big “serious" company, we decided to work with emoji, and this is the reason.
What's the reason? It's not clear.
As an user of this, it looks like a gimmick (like Tom The Talking Cat).
– we tweak eyes to be larger-than-real and observe the reaction by letting users pick from several options or just observing usage frequency. that way we find the optimal point and then slice it down by age, sex, ethnic etc. groups; as well as to personal preference. - do that with a large amount of data + genetic algos that do much less-obvious things than eye size. - when a user wants a World of Warcraft orc avatar, we can use the data to make the orc look both like an orc and like he wants to see himself.
you can't do this with generic emoji – you actually have to keep tweaking and seeing user reactions to understand how people truly want to see themselves and others. and then you can use this to other contexts.
thanks for the feedback, i agree this is not explained clearly enough in the post.
thanks for mentioning, we'll improve!
if you mean in general, right now we are focused on building a great, highly engaging product that people enjoy. and once that is there and at large scale, we'll start thinking about other applications.
impossible to do anything in life well without obsessive focus...
- bitmoji takes a long time and a lot of mental effort to set up. i have no idea what my face shape is and had to ask my ex-gf whether my eyes are green or brown.
- with bitmoji you cannot add other people to your emoji because even if you DO go through mental effort and time to create yourself, you won't do a bunch of other people. and often you really want this – e.g. i want to send an image of YOU as Captain Obvious if the situation calls for it, not of ME.
- over time this will get better and better in terms of quality, while Bitmoji will not.
- bitmoji has no market share in any other language than English, and little market share in non-snapchat messengers.
Anyway, it's a cool app. I'm not trying to be a downer, just wanted to give you honest feedback.
i remember that at our hotel booking company, only 10-15% of people used any filters/sorts for hotels, and >80% just picked from the top list of what we gave them. to me this is shocking given a hotel is a big and rare purchase. but it makes me feel the majority of the world want the fewest taps and the fewest mental barriers.
plus we kind of feel like we want to force ourselves to do a good job out of automatic creation, and a lot of manual customization is defocusing.
(also see comment below about hotels, same discussion)
what do you think?
we do send your images to our servers at this point, and we do store them + standard usage data about how you interact with app flows (conversions, retention etc).
in the future we'll try to have ML on the device and not even send your images to the cloud. both better for privacy and saves us money.
thanks for bringing this up. i have zero sense of privacy for myself so it helps when others point out how they feel about this.
I'm not sure what could be done to alleviate that concern, and perhaps I'm being paranoid, but for now, that has stopped me from using the app. Obviously a 10 year old wouldn't have that concern, and perhaps that is a different problem.
I hope you understand--nothing personal.
if we kept it in Russia latency would suck + we'd have to comply with local regulations we don't want to deal with.
will do this soon :)
also i'm very fortunate in that i already have one company that was bought, plus another that will hopefully go public. so want to build something bigger. plus in Silicon Valley nobody truly respects you if your only goal is to be bought by another company.
But it is a classic case of that for a large company it is hard to execute on many priorities at once and dedicate a lot of organizational resources and attention. Plus they seem to have bet on going directly to VR.
We are laser-focused on doing this particular usecase extremely well, have a large/growing/strong engineering team, and have business leaders that don't have any other priorities beyond making this product excellent.
longer-term it is about the data and using it for other technologies.
> longer-term it is about the data and using it for other technologies.
I'm not sure what this means. Better adtech? Or is there something else unique about this data that I'm not seeing?
thanks for testing
To top all of that off, I was immediately presented with profanity thats completely out of line with the store rating.
Nice try, but bitmoji was more fun and way more accurate.
regarding profanity – thinking how to both let people who want to use it use it (as many do) while not annoying anyone with it.
thanks for trying the product, sorry it didn't work for you
If you're going to have them, put them in a "you gotta look for them corner" of the app, not front and center.
And thanks for making an R-rated option. I will use it.
I'm impressed
thanks for giving us a try! if at some point you have other feedback pls let us know.
Your payoff says I could turn a selfie into an emoji that looks like me, but I got stickers instead of emoji, and they looked nothing like me.
thanks for your feedback!
- very easy to install and use - took a pic (I am not that good looking) and my emoji looks great! - I sent it to a couple of friends who liked it
Couple of ideas, you may already have considered:
1) Use a color-correction algo to help with tinted hair. Then use something like MASK-RCNN to get pixels of hair and get dominate cluster of color. 2) If lighting/shadows/environment causes so much trouble maybe have user rotate/tilt their head in circle-motion (like Apple has users do to train FaceID) then these run ~30 images through your algos and average the results? Maybe have torch on during capture to make the data more homogenous. (Or do once with torch, once without and train two different networks for both types of captures.) 3) If user manually overrides hair color show them your algos next 5 best predictions as other color choices so you can more easily retrain if user selects one of these other predictions. Maybe have them manually adjust bounding polygon of hair detection and even adjust image color correction sliders to show "what my hair actually looks like" (so you can train color correction).
Just some ideas. Cool product! Good luck!
unfortunately full access is the only way to make image pasting work. Bitmoji and all other apps that utilize this mechanic do the same thing.
For what it's worth, Apple would brutalize us if we were offloading customer keystrokes to our servers or something like that.