- It's not IP address. Facebook successfully identified a number of specific locations (bars, theaters, etc) even though I had uploaded the photos from my home
- It's not geo-tagging. All of my photos were taken with a camera that does not geo-tag (Nikon d700).
- It's not contextual tagging. There were no people tagged in the photos, no comments in a lot of them, no words or phrases or names in the captions that could have given clues
- It's not image recognition. One set of photos was taken at Cafe du Nord in SF, CA and every single shot was of the performer onstage, with no identifying characteristics or clues to be had.
"
I would really like to know as this is very interesting and none of the reddit comments (as of now, 12 hours after submission) really answer this question. What technology or methods are they using to suggest (accurate?) locations where pictures have been taken from?
Even more strangely, I have never used my mobile phone with Facebook, but when I uploaded a photo just now of a place from my childhood to which I haven't ever been since using Facebook, Facebook correctly suggested the location.
Purely speculation but perhaps he is carrying his phone with him and it is correlating his location with the timestamps on the images, or perhaps he marked himself as attending an event which is at the venue.
From an image recognition standpoint, if anyone else was at the event and took similar sets of photos which they then tagged, that could also be used.
This is how it can be done without any EXIF/GPS data with a fairly dumb algorithm.
It's a picture of a baby at a hospital.
There are probably hundreds of baby photos being taken per day at that hospital being posted to FB. Some have GPS and some don't.
The images that do have GPS, get recognized and lumped together as the same "entity" since they all look the same.
FB then looks up where she lives, then looks for any photos that don't have a location (like her baby photo), then tries to match it with any photos within a 50 mi proximity to her hometown. Bingo finds 90% match on the baby pic in the hospital, and asks user for verification. She says YES, then this pic gets lumped together with all the other entities. FB also asks more often since it's "right".
A slightly more sophisticated approach would be to use the all /other/ data they have on her to predict what hospital she is likely to choose. Why stop with just her hometown?
Sure let's speculate they have a function that returns a guessed gps coordinate at time t. Again there are dumb implementations for this. Can take a time decaying and time-of-day/day-of-week weighted mode of your recognized locations. I imagine it'd be fairly accurate.
Could be as simple as figuring out that the picture is a baby, get the user's location and then find near-by hospitals. Then ask if the guess is right.
Redditor has facebook app installed on their smartphone (or just uses the website), sets status to "OMG wife is going into labour, at the hospital now". Facebook now knows roughly where redditor was at the specified time based on the ip, they can narrow this down further by looking at keywords in the status message and check it against a list of addresses in the local area and select the best match.
When the redditor comes to upload their photos days or weeks later, facebook just checks the photo timestamp against the user's location +/- X hours and makes a guess at where the photos were taken.
The logic doesn't hold up. I uploaded a bunch of photos taken in Christchurch, New Zealand. The pics were over two years old and I now live in the UK and uploaded them from here. The photos were also taken on a cheap, point & shoot camera, not a smartphone.
right, but assuming the timestamp is on the photos, facebook still knows where you were at the time they were taken. The time you upload them is immaterial
In your opinion, how would Facebook know your location if you never logged in from a GPS-enabled smart phone? (and never approved an HTML5 geo permission request on your laptop).
well they can do a best guess based on the location of your IP, even if you only sign into facebook once while you're travelling. The really interesting question is: can they do this even if you have not signed into facebook at all on your travels, or for totally new accounts with no location history at all.
If they can, then they must be doing image recognition in some way. Perhaps someone who has more time / energy than me could try this by creating a new account and uploading some photos of recognisable objects (but without exif geo data).
> In your opinion, how would Facebook know your location if you never logged in from a GPS-enabled smart phone? (and never approved an HTML5 geo permission request on your laptop).
No real trick to that; if the user signs in from an IP leased to a Christchurch, NZ ISP on March 5th, 2007, then it's a safe bet that any photos taken that day were taken near Christchurch (and you get date taken from the EXIF data of virtually every camera on the planet).
Guess: your friends also took pictures of the same events from similar angles and uploaded them, fb worked out their location from one of the other suggested methods, and then matched it to your photos through simple computer vision algorithms.
Or, you used facebook at some point while in that city, facebook recognised that your photo was of a baby just born, by comparing it with the millions of other photos tagged with "OMG MY BABY IS BEING BORN!," it knows that babies are born in hospitals, and it knows that there is only one hospital in that city, so it guesses that the photo was taken in that hospital.
It doesn't need to be taken on a smartphone. It's sufficient to open facebook on a smartphone only once (app or web) in roughly the same timeframe that the picture was taken. It can then look where one has been in the timeframe when the picture was taken by comparing the picture timestamp with the list of checkin ip addresses, and do a geolookup of the ip address against a location database.
Again, that doesn't align with my experience. The first smartphone I owned was when I arrived in the UK. In NZ, I was restricted to an old Nokia work phone.
Also, does the location info purely say "Christchurch, New Zealand" or is it more granular? I'm guessing that when facebook only has sparse information about your location, as in your case, it "zooms out" to the city, state, country etc, but the more information it can collate, the more accurate the suggestion it gives is.
I reckon they use image recognition too. I have a gash digital camera that gets a lot of use; no exif data beyond camera make/model.
Anyway; in 2005 I went on a skiing trip to Chamonix. Pics were uploaded about a month later, no Facebook activity at the time (or while I was there). The other day FB asked me if they had been taken in Chamonix. So... something else used there at the very least.
How does Facebook use image recognition to recognize that an image of a baby on a blanket with no surroundings visible was taken inside a certain hospital? I doubt it.
Regarding your Chamonix images, perhaps your friends uploaded photos taken at roughly the same time, with geo-tags? Perhaps you are tagged in some of them? That would make for pretty simple logic:
- Facebook knows that you were at a certain place a certain time through the geo-tagging of those photos, as your user is tagged in one of them
- You upload photos taken at the same time as the photos you were tagged in
Conclusion: You must have taken those photos where the photos you were tagged in were taken.
Regarding your Chamonix images, perhaps your friends uploaded photos taken at roughly the same time, with geo-tags?
Nope. Family holiday. Of those there I am the only one with photos on Facebook - and in fact at the time was the only family member with a Facebook account :)
I am intrigued because if it is image recognition there isn't much for them to have gone on. But I am stumped for what else they could have used.
maybe Facebook just randomly guesses locations. those who get matches write comments on reddit wondering how the hell it works and those that get weird suggestions ignore them :)
Wow. Maybe they have a bunch of people looking at pictures, manually making suggestions? Google "similar images" search-like tool? It does sound scary if they really had nothing to go on other than the image data.
It's probably a combination of techniques to be honest. They do have some unprecedented access to contextually heavy data, as well as an unusually large base of free labor to supervise the learning...
My creepy moment came when It correctly pointed out the exact location of this photo:
I was in the middle of Costa Rica. For fear of roaming charges my phone was not on, and I never made a status update. The photo was taken on a D40 at the time, so no location exif data, and frankly the the picture is kind of generic.
My best guess is it used the other photos in the album to gain contextual information. For example this photo was in the same album:
Simply not making a status update isn't enough. You'd have to never visit a page with a Facebook like button from a computer which you'd used ever to log in to Facebook with.
I'm yet to be convinced that this feature requires actually analysing image content at all. GeoIP and time stamps alone can provide a huge amount of context before you even start on things like mobile clients providing location data.
My photo locations seemed to be mainly based on the album title - I was impressed that it got both locations from "St Catharines and Niagara" correct.
I haven't checked to see if the individual photos are correct. However, at the time I didn't have a smart phone and I'm pretty sure my camera wouldn't provide much useful info.
Eventually someone evil will do something truly monstrous on Facebook and it will make this WTF look like a drop in the ocean. I fear for my friends who are still Facehooked.
If they can make guesses this accurate about your photos, imagine they guesses they're making behind the scenes about your life, your personality and your innermost thoughts.
If there was a page on fb where it showed all the inferences they had made about you, sexuality, income, religion, philosophical viewpoints, mental health, etc. then people would run screaming. Of course, a lot of people have already told fb that info voluntarily, and that's why its possible to guess it for everyone else.
Also potentially terrifying: a facebook fortune telling engine. I bet they can predict your future with frightening accuracy, or they will be able to after another decade or so of data anyway.
That is true. Mere statistical analysis on the word count on articles you prefer to read will give a very accurate insight into your interests and mindset. Facebook has access to this data through the "Like" button trojans that exist on many websites. If anything, they haven't been very impressive in putting this data to work yet (or I would be seeing a hell of a lot better targeted ads than currently).
I read Cryptonomicon a month or two ago and really enjoyed it. Seemingly serendipitously, a post about encrypting email turned up on Slashdot the other day:(http://yro.slashdot.org/story/11/12/20/0158227/do-slashdotte...) and I realized that with everyone basically using webmail services now, the chance for something like Ordo to take hold at-large is even less (given that fewer and fewer people actually use email clients anymore).
I'd like to fantasize that there could be a way to use a social network like Facebook with a tool like Ordo -- that is, most the data (photos, in this case) you upload to FB is encrypted, and only people who you've approved can decrypt it. Using steganography this is probably already possible with FB, probably not practical, though.
Indeed, with all the Facebook sharing buttons around the web they have more than your bookmarks too, they have the equivalent of uploading your browser history each day to them. With current machine learning technology, let alone improvements over the coming years, they'd know more or less what someone would know if they stood behind every time you were browsing for the last few years.
Even if you were to cancel your account you'd turn up in photos of parties on other people's accounts and they'd follow pretty well you after you'd left too. With their dodgy history, Zuckerberg's contempt for his own users as evidenced by his quotes, the idea that this won't be abused is laughable. Given national security letters (NSL) it's almost certain the intelligence community has access. Hell you could hit a mid-level admin at one of their data-centers with a NSL and the top level management and legal wouldn't even be aware that there are outside entities with a connection to their database. That's to say nothing of the extra-legal capacities of these people.
While people like to think that it's more or less anonymized clusters and demographics that are being passed to advertisers to target ads better - not a terribly scary thought, just more relevant ads - I think we need to be far more cautious.
While people like to think that it's more or less anonymized clusters and demographics that are being passed to advertisers to target ads better - not a terribly scary thought, just more relevant ads - I think we need to be far more cautious.
How do you go about convincing the "I have nothing to hide" majority of the population that giving away such vast quantities of information is not in their best interests? I'm thinking of friends, family members, etc. who honestly don't care whether the TSA sees them naked at the airport, or how much the other agencies know about them. Where does such extreme deference to authority come from in the first place?
> How do you go about convincing the "I have nothing to hide" majority of the population that giving away such vast quantities of information is not in their best interests?
Continuing and possibly gruesome examples where such data mining ruins lives.
I spoke with a recruiter who prided himself on their HR department's willingness to deny entry based on FB profile (unsure if was public or obtained through some "secret friends" policy, or through a backdoor).
Furthermore, it's becoming almost a certainty that some foolish judge will set precedent somewhere that essentially makes FB account data public (or forces publicity of formerly private data).
I think this article, "Why Privacy Matters Even if You Have 'Nothing to Hide'" [1], makes a compelling case. There are essentially two reasons why privacy is important. The first is that privacy can be used to 'hide bad things'. This is what the 'nothing to hide' argument addresses: if you don't do bad things, you should not worry about privacy. The problem is that not everyone agrees on which things are bad. For example, do you agree that the OWS protesters terrorists?
But privacy is not just about hiding bad things. A multitude of innocuous facts about many people, collected in a database, are themselves a powerful tool. The data can be mined by governments to look for 'suspicious' behavior, by insurance companies to deny insurance, by criminals to commit fraud, etc.
I think the basic problem is that violations of privacy give power in the hands governments and corporations, and these entities tend to abuse whatever power you give them.
I'm not sure how compelling my argument is... but if I have nothing to hide then why does anybody need to look? For example does the TSA seeing everybody naked actually improve security?
As someone with just a fake-info FB account that interacts with it maybe once or twice a week, I am more concerned by the fact that not having an (active|real) account is also a signal that says (or may say at some point in the near future) a lot about me.
From the link - "Did you login with the facebook app on a smartphone while you were at the hospital? If so, they would have your GPS location at a certain time, when you later upload the photos from your camera, it would have a timestamp, and they could just look check where you had logged in from at that time."
If you are a home owner and this is true, you could possibly be identified -- say for instance posting from home 75% of the time and cross-referencing with certain demographic statistics collected from your browsing history and property records. Would that be legal?
Edit - Sorry, I copied the link from the Google results page, it acutally shows up as adspreferences in the ad, but ads/preferences below. Kind of strange.
They already sell that data to credit agencies who use it to produce more accurate credit scores (and get the answers to those questions they are not legally allowed to ask).
Quite a few people mention the pictures were taken with a camera and uploaded at a later date, so GPS wouldn't have been a factor. With the amount of data Facebook has I wouldn't be surprised that they could do more than just use GPS information to place locations.
I'm sure there are other technologies which can be used for geolocalization and can achieve similar "performances" (I'm
thinking about WIFI networks or IP, in some cases).
Still, I wouldn't expect a digital camera to put those informations in the EXIF (assuming it does have a WIFI and/or an IP address)
Also, to me Facebook doesn't seem to be using any magical machine learning algorithm capable to recognize the location of a PIC from a face, it never did on my PICs, not even when there were landscapes in it.
The guy could instead have uploaded the photo using a smartphone which added the GPS coordinates.
But what do YOU think instead? How does that work? Is it working also for your uploads?
Facebook almost certainly has more photo information than TinEye or Flickr, and of indoor environments probably more than Google (which has reverse image search too). Across any given bar or hospital Facebook would have maybe 5-10 other people with albums tagged with the name/gps/check-in. They'd only need one other album though.
SIFT more or less turns every image into a bag-of-words. Your single photo, even at different angles, is going to have a heavy match with photos they have. If you upload a whole album they are going to have tons of matches and they can be more or less certain of the location. To say nothing of adding even the most basic geoip-to-city lookups that would narrow you down to at least five cities that you and your social network inhabit. But the extra information they have is besides the point, SIFT is enough; hospital rooms look alike to us, to SIFT they don't.
But why that whole technology for just making a Suggestion? There is something else they are working on...something really BIG and a complete game changer I guess.
Well if the accuracy is 95% why not turn all Facebook users into one huge mechanical turk batch process to get to 99.9%? And it's just convenient for people not to have to tag.
Then you have the data sitting around for mapping the insides of all buildings like Google has it for street view. Sure your Asimo-style robot butler/"something really BIG" will be that more efficient with internal mappings of most public spaces when you release it in 2030, but the convenience is sufficient. Remember that Facebook beat Myspace more or less on interface (combined with a few other factors like exclusivity), they aren't going to let Google or some other competitor get ahead of them by having an interface convenience edge of 5-10% which might cause a Myspace-to-Facebook style exodus. Kings who have committed regicide on their predecessor are all too aware of how they got into power.
Also consider this, use SIFT on video stills and collect all the handheld video of a particular concert. Use machine learning to combine all the audio tracks and clean them up into high quality audio. Stitch the video together, using textures from the higher resolution stills people take, to allow people to relive the concert with a massive panoramic video (or 3D) with high quality sound. Use it to launch a music competitor to Google or destroy Ticketmaster (well overdue..) as bands and venues won't have to hire video production companies to record concerts if they sell tickets through FB. All that technology is in current research papers and prototypes at the moment, it would probably only take two years to put it together at worst if they aren't working on it already.
If I were to take a picture of a baby at a hospital, wouldn't the majority of the features be of the baby, not the hospital? I suppose if there's at least one picture in the album that's of the actual hospital, that's probably all you would need to infer the rest of the set was taken at the same hospital.
There's plenty of ways to infer location information from a single picture focusing on a baby. The hospital could be identified from something as small as one piece of paper in the background with letterhead or another identifier. Or the face of a nurse in the background that was previously known to be at this hospital. Quite possibly the room layout or particular pieces of equipment in certain arrangements. Landmarks outside a window.
Pictures leak a ton of side channel information outside of their subject matter.
It's not that sophisticated. It's a picture of a baby being taken at a hospital.
There's probably hundreds of photos of babies being taken with GPS from that hosptial.
FB image recognition probably thinks all these babies are the same "person".
Then a separate system comes along and takes photos that do not have location info and tries to match them up with ones that do. It finds a match with the "hospital baby" and then asks for verification. Person says YES and it adds that baby to the "hospital baby" pool as well.
> hospital rooms look alike to us, to SIFT they don't
Well, to SIFT they'll all look different. Except for the hospital room in Portland, OR and the hospital room in Portland, ME that each happen to have a sign with "Portland Hospital" visibile in the picture. While I'm alarmed about the potential of computer vision to compromise my privacy, I have yet to see anything in actual use even be competent, let alone alarming.
SIFT is not enough for this task. There are many computer vision researchers working on this application (automatically inferring location of an image), and accuracies are pretty low right now, except for popular landmarks.
The problem is that single SIFT features are not very distinctive, and so you need many of them in common between two images to get reliable matches. So you essentially need images taken from very similar locations. You can ameliorate this requirement slightly by using fancy tricks in how you aggregate SIFT features together, but the fundamental constraint remains.
This precondition is satisfied at popular landmarks, since there is a good chance that someone's taken a photo from the same location you're standing at, but in general, this condition is not that easy to meet, and hence the poor accuracy of current approaches.
Finally, SIFT is great only for roughly-2d, distinctively textured areas. However, a lot (maybe most?) of the world does NOT fall into this category -- many things are either not textured, not distinctive, or are 3d. Something other than local descriptors (of which SIFT is the best known example) will be needed to understand these kind of scenes.
If you're curious about this line of research, a good place to start is the IM2GPS paper, which was the first major work (that I'm aware of) to look at this problem:
http://graphics.cs.cmu.edu/projects/im2gps/
Wow! A serious competitor to SIRI in place. My guess is they are using combination of techniques - Semantics of your Status as phpnode mentioned with example(heard they have plans to get into semantic search to beat google), EXIF info, IP address and also your friends replies - when is the due? which hospital or Gynic? If hospital name is not mentioned then Gynics details and her hospital location. Do you think all this is used just to make a SUGGESTION?? Its a Billion$?
If image info that users wish to enter can be guessed well this reduces friction substantially. Having this info makes FB a more valuable marketing tool. Why not make such suggestions.
I just tried to upload some photos that facebook hasn't seen yet with or without EXIF info (obviously no or invalid geotagging) and tried geo-tagging some photos I already have uploaded and Facebook hasn't made any suggestion whatsoever. Perhaps I found a way to switch it off but I've just checked the settings and none of the options seems to be concerned with suggestions for geotagging
I took some pics on a regular digital camera (no GPS) in Indore. I uploaded them from New Delhi a few weeks later. And now FB is asking me "Were these pics taken in Indore?". Crazy shit.
Update - I dug through my FB updates. Just before leaving for airport, I updated my status to "Off to Indore" and after coming back to New Delhi, I had some status updates about my office and a local park. Facebook is probably using the the timestamps from image and relating it to locations using some heuristics like status updates, IP addresses, image recognition etc.
I think you are actually getting to the bottom of it. A combination of timestamps, and status updates, and probably that of friends tagged in the same album. Could be quite effective, as we can see.
yeah this is by far most likely. The guy's wife or whatever had her phone turned on while in labour, facebook knows her GPS coords, facebook knows she's his wife, and so on
doesnt take much to throw out a guess based on a single GPS location + timestamp. Even if the guess is only right 5% of the time it is still a profitable guess to make.
It's not perfect yet -- Facebook knows that I went to university in Cambridge (UK) but keeps asking me if photos from my undergraduate years were taken in Cambridge (Massachusetts).
There's anecdotal evidence, a little bit further down, of this actually happening: someone mentions Facebook thinking that their pictures of a trip to London were taken in England, Arkansas!
A test that might fool this. You will need:
1. A facebook status update/checkin/access from a particular location.
2. A photo taken without EXIF data at around the same time as the update/checkin/access above, but from a completely different location.
Upload the photo using your facebook account. Check whether they get the location right. If yes, the mystery continues. If no, but the location is somewhere other than where you updated your status/checked in from, the mystery deepens. Else, they're pairing the times together.
I think I'll set the date on my camera to 1911. A human seeing the timestamp on the picture will just assume its a Y2K bug and will guess the correct data. Since that's extremely unlikely IRL, the Facebook algorithm probably won't know to make this correction.
I've seen both extremes on my photos. One set it suggested the nearby town, which was creepily accurate; another it asked "were these taken in England, Arkansas" which was a hilariously poor guess. Weirdly, they managed the difficult part (figuring out the photos were taken in "England" without any obvious clues) but then failed to sensibly geolocate it.
Highly unlikely that there is anything sophisticated here. It would be too many computing cycles thrown at something with very small RoI. Most likely some straightforward text, date, and location matching.
Many of the previous commenters are assuming you need sophisticated pattern sifting to get any good insights. Not true. Large numbers of facebook users are including this data voluntarily or with their unexpurgated photos. Going after the sliver who don't is just not a worthy investment -- yet.
I have Facebook suggested a few albums today (2 hours ago). Some of them are from 2+ years ago, others are recent ones. The suggestion is quite accurate (89%, 8 out of 9 albums). With the incorrect album, Facebook suggested another city with the same name but from another country!
So Facebook probably just combine as much data as possible and when the matching rate is larger than some threshold, it will temporary tag the album and confirm with users. Data may include:
1. Album info (yeah, most of my album includes the place in it's name or description)
2. Comments group. 3 of my albums have a large number of comments, they are all from my highschool friends (I grouped them all in the same group) so it's quite accurate I guess. Also, if Facebook use the information in their smart lists, that makes sense too.
3. EXIF data. Some people says Facebook can read the data and cross-check the date with its database of check-in. This doesn't happen to me but I think it's possible. And of course, if the EXIF has geo-tagged, that info can be picked up.
Guessing from that data might still be using machine learning (e.g Collaborative filtering). I guess you meant it is not computer vision related. At this point I don't think it matters too much, they are using a combination of techniques (including the famous social graph) and that is giving good results.
I second the idea that this doesn't sound like machine learning. I wouldn't be surprised if there were no machine learning or data mining algorithms involved in Facebook mapping EXIF-embedded geographic coordinates in digital photographs and suggesting locations for an album based on points-of-interest near those coordinates.
Maybe some neat work with GIS enabled this, but I don't see anything that strongly indicates machine learning.
I've tried it. i don't use fb for pics. i don't use wifi on my smartphone and i had the fb app only briefly installed. i also weren't on any trips outside my city.
I've uploaded 4 screenshot-ed landscape (including one building) pics from flicker with absolutely no metadata. at least 2 of them should be very recognizable. the only thing that fb suggested was the single check-in that i had, minimum of 500km off.
i'm pretty sure that there is only minimal (if any) intelligence on image recognition and that they guess locations by information that "leak" from you and your hardware.
Since Facebook knows your social connections, isn't it also possible they check the location of the people to whom you're connected, and use their location as a guide as to where your photos may have been taken? If a cousin checked Facebook on a phone near a hospital while you posted a hospital photo, they know there's a chance the cousin was visiting you and that's where your photo was taken. They use your social grid for everything ... why wouldn't they use it for this too?
I think this is a highly likely scenario. The reddit user isn't taking into account that much of what Facebook knows about us comes from our connections.
LinkedIn appears to build "social graphs" based on profile views. If person A visits the profile of person B and C it is quite likely that some form of relationship exists (perhaps an inverse relationship too). It doesn't just have to be profile views -- LinkedIn most likely keeps track of search terms too. If a random unknown visitor comes along and searches for both "Person A" and "Person C" then it is likely that a connection exists between these two persons.
It would be easy to train this system by displaying a "guess" (a friendship recommendation for person A) to persons B and C. If person B or C show interest in the recommendation then perhaps a relationship exists. Guesses could also be formed by comparing keywords/metadata found on profiles in two different social circles.
The reason I mentioned LinkedIn is that I think they do a better job of recommending/guessing who your acquaintances are than Facebook.
I was about to post about LinkedIn too. I am convinced it uses, among others, reverse view lookups. I have had suggestions for people I know whose LinkedIn profile I had never viewed or searched for myself; the only explanation is they had looked for me. I don't know if it's because I interact with it more frequently and using real personal info but its "people you may know" predictions are eerily more accurate than FB, or anything else for that matter.
167 comments
[ 4.2 ms ] story [ 196 ms ] threadAccording to a redditor,
"
- It's not IP address. Facebook successfully identified a number of specific locations (bars, theaters, etc) even though I had uploaded the photos from my home
- It's not geo-tagging. All of my photos were taken with a camera that does not geo-tag (Nikon d700).
- It's not contextual tagging. There were no people tagged in the photos, no comments in a lot of them, no words or phrases or names in the captions that could have given clues
- It's not image recognition. One set of photos was taken at Cafe du Nord in SF, CA and every single shot was of the performer onstage, with no identifying characteristics or clues to be had.
"
I would really like to know as this is very interesting and none of the reddit comments (as of now, 12 hours after submission) really answer this question. What technology or methods are they using to suggest (accurate?) locations where pictures have been taken from?
Even more strangely, I have never used my mobile phone with Facebook, but when I uploaded a photo just now of a place from my childhood to which I haven't ever been since using Facebook, Facebook correctly suggested the location.
What the heck?!
From an image recognition standpoint, if anyone else was at the event and took similar sets of photos which they then tagged, that could also be used.
It's a picture of a baby at a hospital.
There are probably hundreds of baby photos being taken per day at that hospital being posted to FB. Some have GPS and some don't.
The images that do have GPS, get recognized and lumped together as the same "entity" since they all look the same.
FB then looks up where she lives, then looks for any photos that don't have a location (like her baby photo), then tries to match it with any photos within a 50 mi proximity to her hometown. Bingo finds 90% match on the baby pic in the hospital, and asks user for verification. She says YES, then this pic gets lumped together with all the other entities. FB also asks more often since it's "right".
If so, we're in trouble...
Redditor has facebook app installed on their smartphone (or just uses the website), sets status to "OMG wife is going into labour, at the hospital now". Facebook now knows roughly where redditor was at the specified time based on the ip, they can narrow this down further by looking at keywords in the status message and check it against a list of addresses in the local area and select the best match.
When the redditor comes to upload their photos days or weeks later, facebook just checks the photo timestamp against the user's location +/- X hours and makes a guess at where the photos were taken.
That's the interesting question here.
If they can, then they must be doing image recognition in some way. Perhaps someone who has more time / energy than me could try this by creating a new account and uploading some photos of recognisable objects (but without exif geo data).
No real trick to that; if the user signs in from an IP leased to a Christchurch, NZ ISP on March 5th, 2007, then it's a safe bet that any photos taken that day were taken near Christchurch (and you get date taken from the EXIF data of virtually every camera on the planet).
When you uploaded the photos: [hh:mm dd/mm/yyyy, photo] + above data => [photo, location]
Or, you used facebook at some point while in that city, facebook recognised that your photo was of a baby just born, by comparing it with the millions of other photos tagged with "OMG MY BABY IS BEING BORN!," it knows that babies are born in hospitals, and it knows that there is only one hospital in that city, so it guesses that the photo was taken in that hospital.
Anyway; in 2005 I went on a skiing trip to Chamonix. Pics were uploaded about a month later, no Facebook activity at the time (or while I was there). The other day FB asked me if they had been taken in Chamonix. So... something else used there at the very least.
Regarding your Chamonix images, perhaps your friends uploaded photos taken at roughly the same time, with geo-tags? Perhaps you are tagged in some of them? That would make for pretty simple logic:
- Facebook knows that you were at a certain place a certain time through the geo-tagging of those photos, as your user is tagged in one of them
- You upload photos taken at the same time as the photos you were tagged in
Conclusion: You must have taken those photos where the photos you were tagged in were taken.
Nope. Family holiday. Of those there I am the only one with photos on Facebook - and in fact at the time was the only family member with a Facebook account :)
I am intrigued because if it is image recognition there isn't much for them to have gone on. But I am stumped for what else they could have used.
My creepy moment came when It correctly pointed out the exact location of this photo:
http://www.flickr.com/photos/40127665@N03/4788700749/in/set-...
I was in the middle of Costa Rica. For fear of roaming charges my phone was not on, and I never made a status update. The photo was taken on a D40 at the time, so no location exif data, and frankly the the picture is kind of generic.
My best guess is it used the other photos in the album to gain contextual information. For example this photo was in the same album:
http://www.flickr.com/photos/40127665@N03/4789329722/in/set-...
this, to me, would be extremely easy to recognize.
I'm yet to be convinced that this feature requires actually analysing image content at all. GeoIP and time stamps alone can provide a huge amount of context before you even start on things like mobile clients providing location data.
Kidding! If it was that good, I really would be shocked.
I haven't checked to see if the individual photos are correct. However, at the time I didn't have a smart phone and I'm pretty sure my camera wouldn't provide much useful info.
UPDATE: I take it back: fb thinks I was at a football camp called St Catharines and Niagara! http://www.facebook.com/pages/St-Catharines-and-Niagara/1821...
I think that shows that the title is the most important indicator to fb.
I have a Nokia phone with no GPS. Many programs uses the cell tower info added by the phone to determine where the photo was taken.
If they can make guesses this accurate about your photos, imagine they guesses they're making behind the scenes about your life, your personality and your innermost thoughts.
If there was a page on fb where it showed all the inferences they had made about you, sexuality, income, religion, philosophical viewpoints, mental health, etc. then people would run screaming. Of course, a lot of people have already told fb that info voluntarily, and that's why its possible to guess it for everyone else.
Also potentially terrifying: a facebook fortune telling engine. I bet they can predict your future with frightening accuracy, or they will be able to after another decade or so of data anyway.
I'd like to fantasize that there could be a way to use a social network like Facebook with a tool like Ordo -- that is, most the data (photos, in this case) you upload to FB is encrypted, and only people who you've approved can decrypt it. Using steganography this is probably already possible with FB, probably not practical, though.
Even if you were to cancel your account you'd turn up in photos of parties on other people's accounts and they'd follow pretty well you after you'd left too. With their dodgy history, Zuckerberg's contempt for his own users as evidenced by his quotes, the idea that this won't be abused is laughable. Given national security letters (NSL) it's almost certain the intelligence community has access. Hell you could hit a mid-level admin at one of their data-centers with a NSL and the top level management and legal wouldn't even be aware that there are outside entities with a connection to their database. That's to say nothing of the extra-legal capacities of these people.
While people like to think that it's more or less anonymized clusters and demographics that are being passed to advertisers to target ads better - not a terribly scary thought, just more relevant ads - I think we need to be far more cautious.
How do you go about convincing the "I have nothing to hide" majority of the population that giving away such vast quantities of information is not in their best interests? I'm thinking of friends, family members, etc. who honestly don't care whether the TSA sees them naked at the airport, or how much the other agencies know about them. Where does such extreme deference to authority come from in the first place?
Gilles Deleuze said it comes from a desire to be led.
Continuing and possibly gruesome examples where such data mining ruins lives.
I spoke with a recruiter who prided himself on their HR department's willingness to deny entry based on FB profile (unsure if was public or obtained through some "secret friends" policy, or through a backdoor).
Furthermore, it's becoming almost a certainty that some foolish judge will set precedent somewhere that essentially makes FB account data public (or forces publicity of formerly private data).
But privacy is not just about hiding bad things. A multitude of innocuous facts about many people, collected in a database, are themselves a powerful tool. The data can be mined by governments to look for 'suspicious' behavior, by insurance companies to deny insurance, by criminals to commit fraud, etc.
I think the basic problem is that violations of privacy give power in the hands governments and corporations, and these entities tend to abuse whatever power you give them.
[1] http://chronicle.com/article/Why-Privacy-Matters-Even-if/127...
If you are a home owner and this is true, you could possibly be identified -- say for instance posting from home 75% of the time and cross-referencing with certain demographic statistics collected from your browsing history and property records. Would that be legal?
(The problem, of course, is that a "distant future" today is 10-20 years... Probably even in my lifetime.)
Also they are a fortune telling machine. That's a good pun actually. They are hoping to make a fortune by telling you what ads you'll click.
Some of them where right, but more than half of them was wrong (I am not, for the record, interested in Paleontology).
See "Your categories"
Edit - Sorry, I copied the link from the Google results page, it acutally shows up as adspreferences in the ad, but ads/preferences below. Kind of strange.
Still, I wouldn't expect a digital camera to put those informations in the EXIF (assuming it does have a WIFI and/or an IP address)
Also, to me Facebook doesn't seem to be using any magical machine learning algorithm capable to recognize the location of a PIC from a face, it never did on my PICs, not even when there were landscapes in it.
The guy could instead have uploaded the photo using a smartphone which added the GPS coordinates.
But what do YOU think instead? How does that work? Is it working also for your uploads?
Facebook almost certainly has more photo information than TinEye or Flickr, and of indoor environments probably more than Google (which has reverse image search too). Across any given bar or hospital Facebook would have maybe 5-10 other people with albums tagged with the name/gps/check-in. They'd only need one other album though.
SIFT more or less turns every image into a bag-of-words. Your single photo, even at different angles, is going to have a heavy match with photos they have. If you upload a whole album they are going to have tons of matches and they can be more or less certain of the location. To say nothing of adding even the most basic geoip-to-city lookups that would narrow you down to at least five cities that you and your social network inhabit. But the extra information they have is besides the point, SIFT is enough; hospital rooms look alike to us, to SIFT they don't.
Then you have the data sitting around for mapping the insides of all buildings like Google has it for street view. Sure your Asimo-style robot butler/"something really BIG" will be that more efficient with internal mappings of most public spaces when you release it in 2030, but the convenience is sufficient. Remember that Facebook beat Myspace more or less on interface (combined with a few other factors like exclusivity), they aren't going to let Google or some other competitor get ahead of them by having an interface convenience edge of 5-10% which might cause a Myspace-to-Facebook style exodus. Kings who have committed regicide on their predecessor are all too aware of how they got into power.
Pictures leak a ton of side channel information outside of their subject matter.
There's probably hundreds of photos of babies being taken with GPS from that hosptial.
FB image recognition probably thinks all these babies are the same "person".
Then a separate system comes along and takes photos that do not have location info and tries to match them up with ones that do. It finds a match with the "hospital baby" and then asks for verification. Person says YES and it adds that baby to the "hospital baby" pool as well.
"I just had a baby" - updated from hospital (Upload photo with timestamp near that status time) Match the two
Well, to SIFT they'll all look different. Except for the hospital room in Portland, OR and the hospital room in Portland, ME that each happen to have a sign with "Portland Hospital" visibile in the picture. While I'm alarmed about the potential of computer vision to compromise my privacy, I have yet to see anything in actual use even be competent, let alone alarming.
The problem is that single SIFT features are not very distinctive, and so you need many of them in common between two images to get reliable matches. So you essentially need images taken from very similar locations. You can ameliorate this requirement slightly by using fancy tricks in how you aggregate SIFT features together, but the fundamental constraint remains.
This precondition is satisfied at popular landmarks, since there is a good chance that someone's taken a photo from the same location you're standing at, but in general, this condition is not that easy to meet, and hence the poor accuracy of current approaches.
Finally, SIFT is great only for roughly-2d, distinctively textured areas. However, a lot (maybe most?) of the world does NOT fall into this category -- many things are either not textured, not distinctive, or are 3d. Something other than local descriptors (of which SIFT is the best known example) will be needed to understand these kind of scenes.
If you're curious about this line of research, a good place to start is the IM2GPS paper, which was the first major work (that I'm aware of) to look at this problem: http://graphics.cs.cmu.edu/projects/im2gps/
I took some pics on a regular digital camera (no GPS) in Indore. I uploaded them from New Delhi a few weeks later. And now FB is asking me "Were these pics taken in Indore?". Crazy shit.
Update - I dug through my FB updates. Just before leaving for airport, I updated my status to "Off to Indore" and after coming back to New Delhi, I had some status updates about my office and a local park. Facebook is probably using the the timestamps from image and relating it to locations using some heuristics like status updates, IP addresses, image recognition etc.
doesnt take much to throw out a guess based on a single GPS location + timestamp. Even if the guess is only right 5% of the time it is still a profitable guess to make.
"You know what this means:
Time to rewrite your EXIF info and location bomb the hell out of popular attractions. Eiffel Tower in Paris? Nope, it's in Iowa ..." [1]
Possible? Google bomb with a twist?
1: http://www.reddit.com/r/WTF/comments/nkktm/facebook_is_reall...
http://www.reddit.com/r/WTF/comments/nkktm/facebook_is_reall...
Upload the photo using your facebook account. Check whether they get the location right. If yes, the mystery continues. If no, but the location is somewhere other than where you updated your status/checked in from, the mystery deepens. Else, they're pairing the times together.
I have Facebook suggested a few albums today (2 hours ago). Some of them are from 2+ years ago, others are recent ones. The suggestion is quite accurate (89%, 8 out of 9 albums). With the incorrect album, Facebook suggested another city with the same name but from another country!
So Facebook probably just combine as much data as possible and when the matching rate is larger than some threshold, it will temporary tag the album and confirm with users. Data may include:
1. Album info (yeah, most of my album includes the place in it's name or description)
2. Comments group. 3 of my albums have a large number of comments, they are all from my highschool friends (I grouped them all in the same group) so it's quite accurate I guess. Also, if Facebook use the information in their smart lists, that makes sense too.
3. EXIF data. Some people says Facebook can read the data and cross-check the date with its database of check-in. This doesn't happen to me but I think it's possible. And of course, if the EXIF has geo-tagged, that info can be picked up.
Thought?
Maybe some neat work with GIS enabled this, but I don't see anything that strongly indicates machine learning.
http://www.reddit.com/r/WTF/comments/nkktm/facebook_is_reall...
I've uploaded 4 screenshot-ed landscape (including one building) pics from flicker with absolutely no metadata. at least 2 of them should be very recognizable. the only thing that fb suggested was the single check-in that i had, minimum of 500km off.
i'm pretty sure that there is only minimal (if any) intelligence on image recognition and that they guess locations by information that "leak" from you and your hardware.
It would be easy to train this system by displaying a "guess" (a friendship recommendation for person A) to persons B and C. If person B or C show interest in the recommendation then perhaps a relationship exists. Guesses could also be formed by comparing keywords/metadata found on profiles in two different social circles.
The reason I mentioned LinkedIn is that I think they do a better job of recommending/guessing who your acquaintances are than Facebook.
Facebook usually suggests people I have met but do not want to be "friends" with.