Good attempt, but some of these results are wonky.
Right off the top of the list, the result for Allanburg is actually perfectly right - administratively it's part of Thorold hence why GeoNames came back with that. Dain City is maybe 500 m off the centre rather than outright wrong as suggested in the list. Ayr comes back as "8 George St, Kitchener ON N0B 1E0, Canada <+43.29177700,-80.45398100>" - the coordinates and the postal code are correct, the street and city name are not - I don't know how to rank the correctness of such a result. Same deal with nearby St. Jacobs. Port Robinson is also described incorrectly but has the correct coordinates.
The answer for Burlington has first coordinates that are "close" (well within the city but not at centre/downtown/city hall), and the 'region' one is about 8 km off with a silly large error radius of 15 km. Ditto Kingston, Kitchener, and Markham. Similar deal with Clarington, Clarkson, Falconbridge, "Sudbury", and probably a bunch of others, though there's the added difficulty of Ontario municipal organization thrown in for those ones. Bizarrely, Ottawa's point result is perfect whereas the 'region' result is more than 20 km away with a specified radius of 55 km, but I'm not sure if that warrants a 'wrong' rating.
Echo Bay points to the body of water rather than the settlement around it, and I don't think it's fair to expect the search to know you meant the settlement specifically. Jordan Station is marked "close" but is in fact 500 km off. Wilno is marked "close" but is 10 km of forest away. The results for Mississauga are unreasonably far away from the centre of the city but within the limits.
Then there's search context - if you ask the service for "Baltimore Ontario" with no further context it might get it wrong, if you ask it from a device that knows it's at least in Ontario (or even New York or Michigan) it might work better.
It would also be useful to somehow quantify the results by search likelihood, popularity, community size, etc. A few names I recognize from the missing or wrong list are fairly small settlements. Scrolling through the list (currently on W), the largest results I saw that were flat out wrong so far are Belleville, which is about 10 km off, North Bay about 8 km off, Whitchurch-Stouffville about 6 km off, and Burlington/Kingston/Kitchener/Markham/Mississauga 'region' issues as mentioned above.
Yep. Many of the place names in his "official" list do not have Wikipedia entries, which cast doubt in my mind on whether the places actually exist. I googled some and it seemed like Google doesn't think they exist either.
Also, it looks like he counted a server error as a bad map result. Maybe the server was throttling him or the server is otherwise overloaded?
A better attempt would be to use Wikipedia as a reference for places which (a) exist and (b) are non-trivial. Also he should cross reference against Google since that is the current standard in mapping.
The server code 8 error appears to come up only for less significant placenames (I don't recognize vast majority of code 8 results), lending some credence to a theory that this error is returned for unrecognized queries.
Edit: It has occurred to me the 'region' results might be trying to circumscribe the whole area of the municipality in a circle, which would certainly give odd results for a lot of Ontario municipalities.
Trivia: ebay.com was named after Echo Bay, a Canadian consulting firm, which in turn was named after the township. Echobay.com was taken, the rest is history.
Falconbridge was always a bit fuzzy around the edges -- it was always hard to tell when you'd strayed into Garson or New Sudbury (it was easier to tell when you'd left Gatchell behind and hit the West End) and there weren't too many people upset when Nickel Centre happened in the First Great Amalgamation. (The people of Copper Cliff--which was swallowed by the City--and most of what became Walden were somewhat less happy.)
I haven't been back there for many years, but as I understand it, "Sudbury" can now be better defined by naming the few Northern Ontario municipalities it doesn't yet include. If the X marking the spot is not in Timmins, Espanola/Nairn, Mattawa or the Sound, but somewhere in between those points, it's probably "in Sudbury", on paper at least.
(For you foreigners, that territory includes a moderately-sized urban/commercial/industrial centre, several small hard-rock mining towns, some of the best farmland in Canada, forestry, quarrying, a small inland fishery ... you know, the sort of mixture that's obviously best administered by calling the whole thing one big happy municipality.)
I'm not sure what the symbols mean, but there's a red circle next to Kingston, ON. The Kingston info is correct. It's in the right location and Portsmouth is part of Kingston.
The Belleville information looks mostly correct, though it places Belleville city center a few miles north. Actual street addresses work.
Unlikely, the list of communities has a lot of unincorporated areas that are formally part of a larger municipality and the results for some of them are correct. Clarkson and Dain City off the top of my head.
If this person is applying for a QA job with Apple they have the right mindset for it. Its always great to come up with interesting ways to measure the quality of something, I see a few problems with the methodology but hook it up to an automated bug submission process and it could do wonders for quality over all.
I don't think it would. This particular method is likely to be unactionable, due to comparing a base map data source to another.
The problem is that either Apple have rights to this data source, in which case they should just be fusing it together with their other data. Or they don't have rights, in which case they can't use the results of this comparison to fix data without also risking tainting said data. I'm not a lawyer, so I don't know where the line would be drawn. But automatically or manually filing data bug reports based on results like these seems very dodgy.
Second, automatically generating and filing hundreds or thousands of bugs would just render the submission channels useless. Just triaging geocoding bugs requires ridiculous amounts of skilled human effort by people whose time would most likely be better spent in fixing the geocoder. It might be different for things known to be guaranteed data bugs, but then your methodology certainly shouldn't involve a geocoder. And you'd run into the rights issues of the previous paragraph.
There are good ways of doing automatic quality evaluation or semi-automated quality improvement. But they're really going to be more sophisticated. And unless Apple's expensively bought maps people are totally incompetent, they are already doing such things.
(Edit: I don't mean to belittle the efforts of the original poster. It's good that somebody tried to quantify the quality of Apple maps rather than relying purely on anecdotal evidence. I just don't think that it's a method that'd have much relevance to the way Apple works.)
I was thinking more along the lines that this person was passionate enough about improving the quality that they were willing to code up a prototype way of visualizing relative quality. Perhaps I've known too many engineers where I've asked "How do you know that is ready for prime time?" and they have sort of glazed over and said, "Uh, I don't it seems to work?"
After just today reading news about the ios6 map app failing, and then this article, I really am amazed at the deterioration of the company. Nothing too surprising as all goods things come to an end, but still, this is fast.
I think "most profitable" and "deteriorating" are perfectly capable of being orthogonal. If nothing else, the profit or its first derivative w.r.t. time could be declining while still being the highest in the world. No opinion on Apple.
In August 2008, after being very publicly pilloried by customers and none other than Walt Mossberg, then one of the biggest Apple fans in the press, Apple also apologized to customers. That time? Mobile Me had completely fucked the dog, leaving users locked out of their email and other services – for days.
Arguably a much more catastrophic failure, though limited in scope by a smaller user base.
Apple went on to fix those issues in Mobile Me, rebrand it iCloud, and everything has been gravy ever since. Apple's stock price went from $175 in August that year down to $97 that October. Today it stands at $667.
So if you could, given this context, explain what makes iOS 6 maps point to deterioration in the company in a way that Mobile Me did not. You will receive bonus points if you manage to do so without invoking a red herring in form of the death of Steve Jobs.
I can't type much about mobile me - I tried using it some time ago both personally and at work, and it was a pain. My explanation is based on imagination. I imagine the QA for such an iOS release. I imagine Mr. Jobs using his iPad while traveling (he traveled quite a bit?) and using the map app... he alone would have quickly picked up on issues and never signed off on release until fixed. It's really hard for me to imagine a company with the position and history of Apple to knowingly release and market the app while so flawed. It simply amazes me. Nothing solidly negative, just amazing. Hmmm, re-reading your post, I see that bonus points are out of the question :(
edit. I see that Mr. Cook apologized. No one was harmed, so all seems well to me. Still waiting on the Nokia 920 to hit the streets though ;) Cheers
Apple is using the cartographic data provided by TomTom (who bought TeleAtlas a few years ago), notoriously less accurate in Canada that Navteq's (Garmin's provider). This may be the reason (or Apple's own mapping algorithms - couldn't tell, as I'm not using iOS).
LOL.
Well for what it's worth, I looked up an address in Montréal and Maps had it completely screwed up.
I don't mind the beta quality of the product (Apple has to gather the raw data somehow) but I do mind its hubris and the fanboys' initial rave reviews of iOS 6.
19 comments
[ 3.8 ms ] story [ 71.1 ms ] threadRight off the top of the list, the result for Allanburg is actually perfectly right - administratively it's part of Thorold hence why GeoNames came back with that. Dain City is maybe 500 m off the centre rather than outright wrong as suggested in the list. Ayr comes back as "8 George St, Kitchener ON N0B 1E0, Canada <+43.29177700,-80.45398100>" - the coordinates and the postal code are correct, the street and city name are not - I don't know how to rank the correctness of such a result. Same deal with nearby St. Jacobs. Port Robinson is also described incorrectly but has the correct coordinates.
The answer for Burlington has first coordinates that are "close" (well within the city but not at centre/downtown/city hall), and the 'region' one is about 8 km off with a silly large error radius of 15 km. Ditto Kingston, Kitchener, and Markham. Similar deal with Clarington, Clarkson, Falconbridge, "Sudbury", and probably a bunch of others, though there's the added difficulty of Ontario municipal organization thrown in for those ones. Bizarrely, Ottawa's point result is perfect whereas the 'region' result is more than 20 km away with a specified radius of 55 km, but I'm not sure if that warrants a 'wrong' rating.
Echo Bay points to the body of water rather than the settlement around it, and I don't think it's fair to expect the search to know you meant the settlement specifically. Jordan Station is marked "close" but is in fact 500 km off. Wilno is marked "close" but is 10 km of forest away. The results for Mississauga are unreasonably far away from the centre of the city but within the limits.
Then there's search context - if you ask the service for "Baltimore Ontario" with no further context it might get it wrong, if you ask it from a device that knows it's at least in Ontario (or even New York or Michigan) it might work better.
It would also be useful to somehow quantify the results by search likelihood, popularity, community size, etc. A few names I recognize from the missing or wrong list are fairly small settlements. Scrolling through the list (currently on W), the largest results I saw that were flat out wrong so far are Belleville, which is about 10 km off, North Bay about 8 km off, Whitchurch-Stouffville about 6 km off, and Burlington/Kingston/Kitchener/Markham/Mississauga 'region' issues as mentioned above.
Also, it looks like he counted a server error as a bad map result. Maybe the server was throttling him or the server is otherwise overloaded?
A better attempt would be to use Wikipedia as a reference for places which (a) exist and (b) are non-trivial. Also he should cross reference against Google since that is the current standard in mapping.
Edit: It has occurred to me the 'region' results might be trying to circumscribe the whole area of the municipality in a circle, which would certainly give odd results for a lot of Ontario municipalities.
I haven't been back there for many years, but as I understand it, "Sudbury" can now be better defined by naming the few Northern Ontario municipalities it doesn't yet include. If the X marking the spot is not in Timmins, Espanola/Nairn, Mattawa or the Sound, but somewhere in between those points, it's probably "in Sudbury", on paper at least.
(For you foreigners, that territory includes a moderately-sized urban/commercial/industrial centre, several small hard-rock mining towns, some of the best farmland in Canada, forestry, quarrying, a small inland fishery ... you know, the sort of mixture that's obviously best administered by calling the whole thing one big happy municipality.)
The Belleville information looks mostly correct, though it places Belleville city center a few miles north. Actual street addresses work.
And it place Picton, ON in Australia...
The problem is that either Apple have rights to this data source, in which case they should just be fusing it together with their other data. Or they don't have rights, in which case they can't use the results of this comparison to fix data without also risking tainting said data. I'm not a lawyer, so I don't know where the line would be drawn. But automatically or manually filing data bug reports based on results like these seems very dodgy.
Second, automatically generating and filing hundreds or thousands of bugs would just render the submission channels useless. Just triaging geocoding bugs requires ridiculous amounts of skilled human effort by people whose time would most likely be better spent in fixing the geocoder. It might be different for things known to be guaranteed data bugs, but then your methodology certainly shouldn't involve a geocoder. And you'd run into the rights issues of the previous paragraph.
There are good ways of doing automatic quality evaluation or semi-automated quality improvement. But they're really going to be more sophisticated. And unless Apple's expensively bought maps people are totally incompetent, they are already doing such things.
(Edit: I don't mean to belittle the efforts of the original poster. It's good that somebody tried to quantify the quality of Apple maps rather than relying purely on anecdotal evidence. I just don't think that it's a method that'd have much relevance to the way Apple works.)
I agree, this is fast, but not in the way you mean.
http://money.cnn.com/2012/08/20/technology/apple-most-valuab...
Arguably a much more catastrophic failure, though limited in scope by a smaller user base.
Apple went on to fix those issues in Mobile Me, rebrand it iCloud, and everything has been gravy ever since. Apple's stock price went from $175 in August that year down to $97 that October. Today it stands at $667.
So if you could, given this context, explain what makes iOS 6 maps point to deterioration in the company in a way that Mobile Me did not. You will receive bonus points if you manage to do so without invoking a red herring in form of the death of Steve Jobs.
edit. I see that Mr. Cook apologized. No one was harmed, so all seems well to me. Still waiting on the Nokia 920 to hit the streets though ;) Cheers