What I can say is (i) I don't see anybody succeeding at this an academic environment (TREC took two years to get better than chance, but this nothing to be proud of) and (ii) I don't see anybody succeeding at it commercially.
It's very connected to the problem of scoring in search. In almost all IR systems, the relevance score is meaningless except in that if you sort by it descending you will see results that get worse as you go down the list.
What you really want is to get a probability score as a relevance value and you can convert a conventional score into that with logistic regression but if your system is honest you wind up with 70% or so precision at your maximum score and that is not so good, particularly in the filtering scenario.
I'd imagine a system based on today's machine learning techniques might do pretty good if you supplied a training set of 1000 or so articles (think of it as developing a classifier) and you could probably use dimensional reduction to do better-than-chance with 50 articles, but I think most people would expect to be able to learn from 10 or so examples and would judge the system "unintelligent" if it can't do that.
Notifications, as an active research area, just hasn't developed yet. It's mostly because the concept is very new. But expect much more research and much smarter commercial systems in the next few years. I won't be surprised if notifications become our primary mode of interaction with the digital world.
Also, the training data easily goes into thousands or even millions of samples for even modestly used applications. Unless you mean train a different model per user, in which case, yeah, it won't work.
4 comments
[ 1.7 ms ] story [ 18.2 ms ] threadIt's very connected to the problem of scoring in search. In almost all IR systems, the relevance score is meaningless except in that if you sort by it descending you will see results that get worse as you go down the list.
What you really want is to get a probability score as a relevance value and you can convert a conventional score into that with logistic regression but if your system is honest you wind up with 70% or so precision at your maximum score and that is not so good, particularly in the filtering scenario.
I'd imagine a system based on today's machine learning techniques might do pretty good if you supplied a training set of 1000 or so articles (think of it as developing a classifier) and you could probably use dimensional reduction to do better-than-chance with 50 articles, but I think most people would expect to be able to learn from 10 or so examples and would judge the system "unintelligent" if it can't do that.
Also, the training data easily goes into thousands or even millions of samples for even modestly used applications. Unless you mean train a different model per user, in which case, yeah, it won't work.