This is really amazing. I love data, but the real value comes when I'm able to translate that data into a story. Programs like this are going to become increasingly important as we become deluged with data, but don't know what it means.
All that being said, I want to start hacking around on something like this in a different domain. Any tips on how to get started on the data-to-language transformation? Code or academic articles welcome.
The future. It has arrived. I remember reading a few years ago that sports writing is very formulaic and should be possible to automate. At the time, nobody had really done it...but here we are. It's fantastic to see all this "weak AI" out there, making people money and building a reputation for being better than most humans at a task previously thought to be human-only.
Are we slowly walking directly towards strong AI, with all our little tiny steps of weak AI?
Surely strong AI is by definition such a paradigm shift that it cannot be achieved with lots of little steps of weak AI? I thought that was kind of the point of the strong/weak AI dichotomy.
Sure, and that's why I phrased that suggestion as a question. Maybe we're not walking towards strong AI with weak AI steps. But...with AI writing sports articles now, it seems logical to wonder if perhaps small progressions in the capabilities of weak AI systems will result in real, full AGI.
Maybe we're not walking towards strong AI with weak AI steps.
Just a random thought, but even if all of our weak AI systems don't directly result in us being able to build a strong AI, maybe they will do so indirectly. We might be able to use all of these weak AI systems to train a more general-purpose strong AI system.
A definition has no power to change the engineering feasibility of a task. If it is possible to build a strong AI from a series of weak AI steps, which is certainly not out of the question (one can argue that's how human brains work), no amount of definition will change that.
This takes us back to the problem of emergence (http://en.wikipedia.org/wiki/Emergence), something that AI researchers, entomologists, economists, et al. have been struggling with for some time.
However if we accept the definition, then it is surely meaningless (pointless?) to discuss AI progress in those terms.
EDIT: what i really mean is that I reject the strong/weak AI dichotomy. I really need to resist the urge to play pointless semantic games. Guess thats an occupational hazard of programming all day long.
Playing chess better than humans was also once considered as would-be proof of "Real AI".
As soon as we get an AI doing something that was previously considered intelligent that task is retroactively relabeled as not really requiring intelligence, and consequently "Real AI" remains out of reach.
Personally, I believe there's a tipping point where our society will no longer be able to live in denial that we've built AI. We're not there yet since there are still a huge number of things people can do that computers can't. For example, no AI has really passed the Turing Test yet. So pretty obviously, we don't yet have "Real AI", and I understand the AI Effect, even though it's unfortunate.
But surely, there's a point where it'll be blindingly obvious that our machines are intelligent?
The point where it is blindingly obvious that we have created intelligent machines will require passing the point where it is blindingly obvious what "intelligence" is. I would go further to describe these as a circular dependency.
No, it's not a valid definition, because human capabilities vary way too much.
Name something that "humans can do that computers can't", and I bet I'll find you a human person that can't do it either.
For example, at a previous job we wanted to introduce cognitive captchas to avoid accessibility issues with visual ones. We were told we couldn't do that because some of our target audience had learning difficulties and would not be able to pass them. (Coincidentally, there was HN post the other day about how Wolfram Alpha can pass a good proportion of captchas similar to what we had planned to use)
I fear that society at large will not really get past this denial you speak of until we have a humanoid type of AI, something that the standard, non-science oriented person can look at and say, "Wow this is uncanny." It will need to be shoved right in their face in the most blatant way imaginable.
Advances in natural language processing, metaheuristics and machine learning, as examples, are clearly steps toward the "holy grail" but to the average person it is nigh meaningless.
This heralds the relatively-recent arrival of advanced statistics to the college basketball world - rebounding percentages, scoring percentages, runs -- these are pieces of information anyone could glean by simply looking at a box score, or, at worst, looking at a site with advanced metrics like www.kenpomeroy.com.
All of the game recaps read basically the same - a sure sign of a homogenous data set combined with some fairly crafty name-replacement.
Perhaps this could replace the AP game recaps, but anything beyond that is still far beyond our AI reach.
The next step is pretty obvious: Use voice recognition software to analyze the remarks made by the announcers over the course of the game and blend that into the reporting. If you can properly analyze inflection and special events from the speech of the live game you can add game highlights to the article.
At a hospital that my dad programs for, the radiologists were using medical transcription software to translate "normal foot, normal shoulder" into doctor-speak, and the pre-canned phrases were too repetitive to be pleasant, so they ended up with 50 ways of expanding "normal", and that was acceptable.
This looks like a pretty cool way to interpret stats; I wonder if they could automate Tufte-quality graphics of how the team performed, annotating graphs with news excerpts.
Yep. They do natural language parsing of wire news stories to try and synthesize market intelligence. Which is one day going to lead to a civilization-ending feedback loop due to the sort of thing we're discussing here (in my paranoid hyperbolic opinion :)
The first 12 paragraphs and the last paragraph are completely boilerplate. There are two paragraphs in the middle that were probably human-written. What I don't know is whether this is just a prosthetic "code-generation" tool for their writers, or whether it's just this one writer.
This Google search shows that it is definitely boilerplate, and that some articles appear to be 100% generated:
I saw that done back in 1999 with animated avatars, text-to-speech, market news, stock updates, etc. I can't find the details offhand but I believe it was a Bloomberg thing.
Article says: "Many games, though not all, get detailed write ups. Computer programs turn data from box scores into full sentences that put the reader in the game. "
I have yet to find a game that has a write-up, even for top 25 games. Am I missing the link or something?
Because just automated stats doesn't seem that impressive or very AI.
Oh, I didn't realize that the per team domains have different stuff than what is on StatSheet.com. I just went to the site, clicked on College Basketball in the header, and then picked my team.
I never understood what was interesting about sports or sports journalism until I moved back to Nebraska.
I've been watching Nebrasketball, the mandatory Huskers games in the evenings, and roller derby, all with a group of very sports-aware people and one aspiring sports journalist. His commentary on the football/basketball games converted me from a bored observer to actually enjoying the sports.
I'm impressed by what this site is generating, but I'm not too worried about it displacing actual writers. It's the difference between a local news affiliate press release, and an article in the New Yorker or the Atlantic, or BBC reporting. There's still something that a passionate human can bring to the equation that will be almost impossible to automate.
I agree that we aren't replacing anything. There will never be one source for a particular topic (although our content is significantly less expensive to produce).
But I wouldn't underestimate the potential for automated content to be even better than what the best journalists can produce. We are at the infant stage of automated content. We have a long way to go and it will only get better.
That's a bit of a joke, but the real value automation will provide is not the translation of statistics into narrative, it is providing the right statistics. I can honestly say that your generated stories are better than most AP articles in that regard.
Have you looked at computing any advanced basketball statistics? Highlighting those and turning them into narrative could go a long way toward raising awareness about them in the general (sports) public.
Yes, we compute a bunch of advanced stats for statsheet.com. I've intentionally shied away from doing too much "advanced" stuff on the individual sites because i want them to be consumable by the average fan. Some feedback we get about statsheet.com is that it can be overwhelming.
Bravo! This type of technology will free up sports journalists to concentrate on the areas where they can really add value, such as interviews, analysis, and "color" that the algorithms cannot generate.
A word of caution, though: The potential for bad data to enter the system and get published and spread without a human editor to correct it could cause real problems for teams and athletes. It will require tightening up on the sources that are used, redundancy of sources, and checking mechanisms.
The lesson for journalists here is: if you get into the habit of writing formulaic garbage, you WILL be replaced by a small shell script.
We're a long way from top writers like Hunter S Thompson (who was a sports journalist, among other things) being replaced by programs.
Even far less outstanding writers still provide something valuable, by, for example, extracting knowledge and insight out of the data, rather than just regurgitating data in a human-readable format. If your job is to publish hard facts without insight or analysis, though, your job is toast.
Expectations are high that this year’s <% TEAM_NAME %> team is an improvement on last year’s. They’ll be bringing back a group that played <% TOTAL_MINS %> of last season’s minutes and adding the efforts of <% NUM_TOP100_PLAYERS %> Top 100 recruits that will *###>>>#> RISE UP AND KILL ALL LIFEFORMS ###>> ... losing in the Championship game to <% FINAL_OPPONENT %>
Not quite that easy. Notice that the following sentence is often different. What I suspect they do is have a set of characteristics they look for, like winning team, number of starters coming back, all starters coming back, won conference, made NCAA, made final four, won it all, etc...
And then they have a set of sentence structures they can plug in for these. This allows you to actually generate a different story for the exact same team -- saying really the exact same thing.
How interesting, I was just reading some articles [1] [2] about learning to think like a programmer and program. Both mention how beneficial it is for journalists to code and think like a coder.
While they mainly talk about saving time with repetitive tasks or compiling/understanding complex data, I suppose now its good for job security too...
"Stanford has already started living up to monumental expectations with a good first game": I'm not sure where the "monumental expectations" comes from; Stanford finished below .500 last year and was picked 9th in the Pac-10 preseason media poll.
"Stanford defeated San Diego with domination in offensive rebounding and an additional beat in possession management.": Not sure what "an additional beat in" means here.
"Stanford has incredible expectations for this season, and this victory over Cardinal was a good start.": Should be "over the Toreros" or "for the Cardinal".
Two of those three have been fixed now. The first (around expectations) requires a larger tweak which we are making.
The point to keep in mind is that we are at the infant stage of automated content. Unlike most writers, our content will get significantly better over time.
Similarly, "Duke beat back Princeton with a strong game in free throw shooting and also a beat in three pointers." That "beat" phrasing just reads badly.
There's really two problems they are solving here: identifying the stats that matter, and converting them into English. Seem to do a bit better at the first than the second.
As someone who has long had an interest (my wife would call it an obsession) with transforming the basic stats of sports into something more useful and meaningful, this is fascinating stuff. I think anyone that reads the game previews and recaps on NBA.com, NFL.com or any other major league site can see where things could be automated. That writing IS largely formulaic and forgettable. It would be very interesting if instead of just doing the same thing through automation, it produced innovative stats and interpreted them in ways that gave the reader insight into the outcome of the game.
My current pet project is taking NBA data and transforming it into something beyond points per game, etc. I think that's where the future of sports related AI resides.
I don't know where they get their data but I just checked my teams(FSU) football page and they have a bug in the record (6-4 instead of 7-3). Not too impressed there.
60 comments
[ 3.1 ms ] story [ 124 ms ] threadAll that being said, I want to start hacking around on something like this in a different domain. Any tips on how to get started on the data-to-language transformation? Code or academic articles welcome.
Are we slowly walking directly towards strong AI, with all our little tiny steps of weak AI?
I can't wait.
Just a random thought, but even if all of our weak AI systems don't directly result in us being able to build a strong AI, maybe they will do so indirectly. We might be able to use all of these weak AI systems to train a more general-purpose strong AI system.
EDIT: what i really mean is that I reject the strong/weak AI dichotomy. I really need to resist the urge to play pointless semantic games. Guess thats an occupational hazard of programming all day long.
As soon as we get an AI doing something that was previously considered intelligent that task is retroactively relabeled as not really requiring intelligence, and consequently "Real AI" remains out of reach.
Personally, I believe there's a tipping point where our society will no longer be able to live in denial that we've built AI. We're not there yet since there are still a huge number of things people can do that computers can't. For example, no AI has really passed the Turing Test yet. So pretty obviously, we don't yet have "Real AI", and I understand the AI Effect, even though it's unfortunate.
But surely, there's a point where it'll be blindingly obvious that our machines are intelligent?
Name something that "humans can do that computers can't", and I bet I'll find you a human person that can't do it either.
For example, at a previous job we wanted to introduce cognitive captchas to avoid accessibility issues with visual ones. We were told we couldn't do that because some of our target audience had learning difficulties and would not be able to pass them. (Coincidentally, there was HN post the other day about how Wolfram Alpha can pass a good proportion of captchas similar to what we had planned to use)
Lay people think AI means something like artificial intuition. Which is very hard and can only be done poorly within narrow domains.
Advances in natural language processing, metaheuristics and machine learning, as examples, are clearly steps toward the "holy grail" but to the average person it is nigh meaningless.
This heralds the relatively-recent arrival of advanced statistics to the college basketball world - rebounding percentages, scoring percentages, runs -- these are pieces of information anyone could glean by simply looking at a box score, or, at worst, looking at a site with advanced metrics like www.kenpomeroy.com.
All of the game recaps read basically the same - a sure sign of a homogenous data set combined with some fairly crafty name-replacement.
Perhaps this could replace the AP game recaps, but anything beyond that is still far beyond our AI reach.
The next step is pretty obvious: Use voice recognition software to analyze the remarks made by the announcers over the course of the game and blend that into the reporting. If you can properly analyze inflection and special events from the speech of the live game you can add game highlights to the article.
This looks like a pretty cool way to interpret stats; I wonder if they could automate Tufte-quality graphics of how the team performed, annotating graphs with news excerpts.
http://www.tradershuddle.com/20101117119131/Stocks/pre-marke...
http://collegestock.com/blog/885-metal-stocks-slumped-alcoa-...
There's literally no useful info, just data written as words.
http://www.fool.com/investing/general/2010/11/02/is-activisi...
The first 12 paragraphs and the last paragraph are completely boilerplate. There are two paragraphs in the middle that were probably human-written. What I don't know is whether this is just a prosthetic "code-generation" tool for their writers, or whether it's just this one writer.
This Google search shows that it is definitely boilerplate, and that some articles appear to be 100% generated:
http://www.google.com/search?hl=en&biw=1438&bih=697&...
Definitely a bot to "fool" the financial news aggregators.
I have yet to find a game that has a write-up, even for top 25 games. Am I missing the link or something?
Because just automated stats doesn't seem that impressive or very AI.
Here is Duke: http://bluedevildaily.com
Basically, I was expecting to see a generated recap on this page: http://statsheet.com/mcb/games/2010/11/12/rutgers-73-princet...
I've been watching Nebrasketball, the mandatory Huskers games in the evenings, and roller derby, all with a group of very sports-aware people and one aspiring sports journalist. His commentary on the football/basketball games converted me from a bored observer to actually enjoying the sports.
I'm impressed by what this site is generating, but I'm not too worried about it displacing actual writers. It's the difference between a local news affiliate press release, and an article in the New Yorker or the Atlantic, or BBC reporting. There's still something that a passionate human can bring to the equation that will be almost impossible to automate.
That said, StatSheet is really damn impressive.
But I wouldn't underestimate the potential for automated content to be even better than what the best journalists can produce. We are at the infant stage of automated content. We have a long way to go and it will only get better.
http://www.truthaboutit.net/2010/11/wizards-claw-raptors-109...
That's a bit of a joke, but the real value automation will provide is not the translation of statistics into narrative, it is providing the right statistics. I can honestly say that your generated stories are better than most AP articles in that regard.
Have you looked at computing any advanced basketball statistics? Highlighting those and turning them into narrative could go a long way toward raising awareness about them in the general (sports) public.
That said, we'll be adding more unique stuff over time. We started doing a "Fan Satisfaction" chart: http://carolinaupdate.com/north-carolina-basketball/fan_sati... that attempts to objectively measure the subjective notion of "fan satisfaction".
OMG, statsheet has robot journalists and a robot CEO!
A word of caution, though: The potential for bad data to enter the system and get published and spread without a human editor to correct it could cause real problems for teams and athletes. It will require tightening up on the sources that are used, redundancy of sources, and checking mechanisms.
We're a long way from top writers like Hunter S Thompson (who was a sports journalist, among other things) being replaced by programs.
Even far less outstanding writers still provide something valuable, by, for example, extracting knowledge and insight out of the data, rather than just regurgitating data in a human-readable format. If your job is to publish hard facts without insight or analysis, though, your job is toast.
And then they have a set of sentence structures they can plug in for these. This allows you to actually generate a different story for the exact same team -- saying really the exact same thing.
While they mainly talk about saving time with repetitive tasks or compiling/understanding complex data, I suppose now its good for job security too...
[1] http://infovore.org/archives/2009/01/22/learning-to-think-li...
[2] http://www.charlesarthur.com/blog/?p=1098
"Stanford has already started living up to monumental expectations with a good first game": I'm not sure where the "monumental expectations" comes from; Stanford finished below .500 last year and was picked 9th in the Pac-10 preseason media poll.
"Stanford defeated San Diego with domination in offensive rebounding and an additional beat in possession management.": Not sure what "an additional beat in" means here.
"Stanford has incredible expectations for this season, and this victory over Cardinal was a good start.": Should be "over the Toreros" or "for the Cardinal".
Overall, though, it's pretty slick.
The point to keep in mind is that we are at the infant stage of automated content. Unlike most writers, our content will get significantly better over time.
There's really two problems they are solving here: identifying the stats that matter, and converting them into English. Seem to do a bit better at the first than the second.
My current pet project is taking NBA data and transforming it into something beyond points per game, etc. I think that's where the future of sports related AI resides.
I'm a semi hard-core NBA fan.