Offer HN: Advice from PhD on NLP, machine learning, and data monetization
In particular, I'll field questions on:
* recommender systems
* profit optimization in ecommerce sites
* autotagging
* creative ways that NLP + ML can add value to your product and give you a competitive advantage
* whatever else you like.
If you have a specific technical question about ML or NLP, please post it on the MetaOptimize Q+A site (http://metaoptimize.com/qa/) and post a link here as a comment. I'll give you the detailed answer over there, since the Q+A there is designed to be archival and is more searchable. Other sorts of questions (like "How do I turn this data into money?") can go in this thread directly. If it seems like a longer discussion, email me at joseph at metaoptimize dot com and we can talk about setting up a Skype chat.
p.s. I'm also hiring people for remote project work, if you are kick-ass at ML or NLP, or you simply can ship correct code really fast. Email me at joseph at metaoptimize dot com.
Who am I?
My name is Joseph Turian, and I head MetaOptimize LLC. I consult on NLP, ML, and data monetization. I also run the MetaOptimize Q&A site (http://metaoptimize.com/qa/), where ML and NLP experts share their knowledge. I recently demo'ed autotagging of hacker news to make it automagically browsable (http://metaoptimize.com/projects/autotag/hackernews/).
* I am a data expert, holding a Ph.D. in natural language processing and machine learning. I have a decade of experience in these topics. I specialize in large data sets.
* I’m business-minded, so I focus on business goals and the most direct path of execution to achieve these goals.
* I am also a technology generalist who has been hacking since age 10 and has programmed competitively at a world-class level.
65 comments
[ 2.5 ms ] story [ 120 ms ] thread+ http://metaoptimize.com/qa/
+ http://metaoptimize.com/projects/autotag/hackernews/
I've got a pretty basic grasp on machine learning and collaborative filtering, so I understand some things, but I'm very confused on others, such as:
1)List-wise vs Pair-wise approaches to machine learning. Can you explain in simple terms what is the main differences between them, and in what cases it would be better to use one over the other? I've read a few sources about the differences but it goes over my head a lot of times.
2)When you don't have many users using your site, from my (basic) knowledge, you can't really use KNN algorithms to help with recommendations, because you only have a few people to compare your (lets just say movie preferences) to. What is the best way, then, to get the best recommendations both when your userbase is large and small?
Those are just a few off the top of my head, but I'll be sure to add more later on.
2) You are talking about the cold start problem. Basically, the problem is how do you deliver good recommendations to new users as quickly as possible. I am actually currently talking to a big company about this problem.
Google has talked about how they do this for Google News, especially because they have high item churn (news is not news in a day). http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.80.... Essentially, they do clustering of users and news items.
Another approach is to give the user a hunch-like quiz when they join. Basically, you use a decision tree to cluster the users, getting as much information from them with as few questions as possible.
Another approach that requires almost NO user interaction is to use some ambient information about the user. Maybe you have a cookie for the user and other sites will share the user's behavior with you. Maybe you ask them to log in through facebook or twitter, and then can look at their profile and activity history to build a user profile. So getting creative about possible sources of ambient information is another approach.
Could you provide easy examples where you would use a pair-wise approach, and a list-wise approach? Like, let's say I wanted to build a recommendation system for college football pick-ems (Ie, will Iowa beat Michigan State? Etc.). Which method would be preferred, and why?
Another question:
Let's say (again) that I want to build a recommendation engine for college football picks. How do I know how much (more) data will help for training purposes? IE, will only Win/loss records suffice [probably not], or should I add in the opponent they beat (or were beaten by) as well? What about whether the game was away/at home? What about factoring in the individual players on each team, and their stats? What about the temperature that day? How you do end up knowing when you can stop looking at smaller and smaller variables with your training data, and have something that gives the best results?
If your model is linear, you will probably see diminishing returns as you implement more, different features. It also helps to look at model errors to see which features are pulling things the wrong way, and then you can add other features to compensate.
(feature engineering + linear models feels a lot like writing an old-school AI heuristic program, except you have a computer program give a numeric weight the rules you're writing, so you're free to write a lot (millions) of rules and still get a manageable model)
I had written a longer reply detailing what are listwise methods and how they differ from pairwise methods, but apparently it was eaten by the web hyenas.
Turian's explanation doesn't actually cover what listwise methods are. I've seen them presented in the learning to rank context (learning to rank is when you want to build a machine learning system that given a set of documents, say search results, ranks them from more to less relevant to a query), so I'll follow this context here.
There are two very obvious approach to design a learning system that outputs sorted data: the first is to learn to assign a real number to each "document" (and then you sort according to these numbers) and the second is to learn a classifier that predicts given a pair of "documents" if a <= b (and then you can use this classifier as a comparison function in a sorting algorithm). Both these approaches have a common flaw, however, which is that they are very easily myopic, and will make decisions only looking at a very small window. This is clearly suboptimal in the learning to rank context because, for example, mistakes in the top elements of the ranked list are a lot more important than mistakes further down the list, and sometimes you want to maximize diversity or something like that, and it's hard to do that in an elementwise or pairwise approach.
So they invented listwise approaches, and there are actually two sorts of these: you can either learn a classifier that scores entire sorted sequences of documents (with features like document trigrams, features connecting similar documents, etc) or you can learn one of the above models with a loss function that depends on the entire sorted list of documents.
So essentially a listwise approach is better if you can do it, as listwise approaches come closer to minimizing error measures you actually care about (like precision among the top 10 documents) instead of bogus measures (like number of document pairs misclassified). On the other hand, precisely because listwise approaches allow you to be more specific, they are less generic, and it might be cumbersome to adapt one of them to your recommendation system.
Also, it fundamentally depends on how the results of your recommendation system are used. If you present something like top k recommendations then listwise approaches can be better, but in some other scenarios you will actually care about all the individual decisions, so a pairwise approach can do just as well.
Much as I love them, I've been getting so frustrated with the declining quality of their news page that I'm not sure if this is a good example. I know I'm not the only one.
My main proposal is to post a "How do I build X" question on MetaOptimize. (I don't know any place on the web where experienced NLP people chat, besides MetaOptimize, but I would be interested to hear other options.) NLP isn't really very hard once you know what you're doing, it's just that as an inexperienced person you will spend a lot of time going down dead ends and blind alleys, and there are a lot of pitfalls to avoid. So getting the advice of experienced people, right from the beginning, is the best way to do this.
See also:
* What are the best resources to use when starting machine learning for an experienced programmer? http://metaoptimize.com/qa/questions/334/
* How do I get started understanding statistics & data analysis? http://metaoptimize.com/qa/questions/154/
* New to data mining - where to start? http://metaoptimize.com/qa/questions/362/
The nltk tools them-selves might not always be state of the art (e.g. compare with the java libs from the NLP dept in Stanford) but at least you will get to know all the major concepts.
Other interesting book, although not completely finished yet: http://www.manning.com/ingersoll/ Taming Text by Grant Ingersol, Thomas Morton and Drew Farris involved in the Apache Lucene / Mahout and OpenNLP communities.
I think have a solid ML understanding can serve you well if you are doing NLP.
* Good Freely Available Textbooks on Machine Learning? http://metaoptimize.com/qa/questions/186/
Besides that, I guess it depends specifically on what NLP tasks you are looking at. Can you tell me more?
it's simple enough to get running, fast, and outperforms the state of the art of a couple of years ago.
http://metaoptimize.com/qa/questions/3084/what-components-do...
http://metaoptimize.com/qa/questions/3085/which-techniques-i...
thanks!
(ps please don't upvote me, I like my 1337 karma)
+ http://xlvector.net/blog/?tag=serendipity
I have no easy answer on how to quantify that myself but I guess the papers quoted by xlvector must be a good place to start.
For MSc in Computer Science I wrote my dissertation on sentiment analysis using NLTK for my NLP needs. Although my work was nothing earth shattering, I really enjoyed it and have continued it as a side project investigating different techniques and algorithms. I'm now considering ways that I can turn this interest into my full time career (I'm a .Net developer currently). I don't particularly wish to send out a lot of unsolicited resumes and so was wondering if anyone had some advice on who best to approach? Are there any recruiters and the like that specialize in NLP?
Thanks
Adam p.s. I am an US permanent resident and an Australian Citizen, the website that has my details and some example work is www.emptyorfull.com
I still get contacted about a ML library I haven't touched for 5 years.
I'm particularly frustrated with brainless Java / .net development jobs.
I also think that people don't like having ML or NLP tools through a SaaS API, because you want a tool that you manage on your own hardware.
I believe that we'll see something I call the "machine learning business model". Essentially, you release your code under open source, and you give away a basic model, but then you charge for a premium model. The model is essentially like compiled code. No one knows how you trained it up, or why it works. So you get people using your tools by open-sourcing them, and then you upsell them on increased accuracy.
I looked through the feature lists of many e-commerce analytics packages. It struck me that while they offer impressive-looking graphs and gather huge amounts of data, the "analytics" part seems to be very simple. They mostly present you the data that has been measured.
I am in a situation where I will want to provide e-commerce analytics as part of a more complete solution. But I would like to approach this from a different angle: provide digested data, with insight into patterns that could lead to increased profits for online stores.
What do you think could be extracted from e-commerce data that would be of value to online stores? I'm looking for information that is difficult to extract (e.g. requires a tech advantage), but can be extracted automatically or with minimum human curation.
An example of an interesting direction: e-commerce analytics companies offer "customer segmentation" views. But they are all just simple data views: you can segment based on location, time of visit, etc. I think doing a factor analysis on customer buying patterns would be much more interesting and would provide real value to the store by automatically extracting "customer types" from data. One could then market directly to these customer groups.
So, to rephrase, I'm looking for ideas/pointers on what kinds of analyses would be difficult to create, but very useful for e-commerce.
We've also talked about several other ways of turning historical sales data into actionable business intelligence.
One thing we haven't looked at is doing market analysis based upon customer profiles. This is an interesting idea, and I have some thoughts on how to do it. Essentially, finding different ways to cluster users, and then "explaining" who these customer segments. The explainability is based upon figuring out the most important features they have in common, and exposing these features in plain English.
Thanks!
http://l2r.cs.uiuc.edu/~danr/Papers/RatinovRo09.pdf
It is indeed a good "tricks of the trade" paper. I real enjoy such papers.
A recent such paper I'm excited about is this one on coreference resolution from EMNLP 2010: http://www.aclweb.org/anthology/D/D10/D10-1048.pdf , by the Stanford group. It's a rule-based coreference resolution system you could implement in a couple of days that gets state-of-the-art results. Best of all it looks like you could get it to run about as fast as the parser that's backing it. I'm planning to write a version for the C&C parser.
Having such a runtime available as a standard lib packaged in all linux distribs would make it interesting for lowlevel vector math libs such blas / lapack / atlas and convex optimization solvers to have an implementation based on OpenCL kernels that would work (almost) as fast as the currently manually optimized C + simd intrinsics code but could also 10 to 100 as fast whenever a GPU is available on the machine without having to re-compile anything.
Some advanced and very promising machine learning algorithms (e.g. algorithms of the deep learning family) can really benefit from the computing power of vector processors such as the GPUs.
Right now everybody who wants to be able to gain the perf boost of GPUs uses CUDA but: - it's only useful on NVidia GPU machines (e.g. not on Amazon EC2 machines for instance) - it's not opensource hence has to be manually installed on every single machine you want to use it (no apt-get install from the official repos). This which makes it a heavy dependency for your pet machine learning library: be prepared to support CUDA installation problems on your project mailing list.
[1] http://en.wikipedia.org/wiki/OpenCL
http://deeplearning.net/software/theano/
Theano is a Python library that allows you to define, optimize, and evaluate mathematical expressions involving multi-dimensional arrays efficiently. Theano features:
- tight integration with numpy
- transparent use of a GPU
- symbolic differentiation
- speed and stability optimizations
- dynamic C code generation
- extensive unit-testing and self-verification
(Hi Joseph --- we met briefly at ACL. I was one of James Curran's students. My name's Matthew Honnibal.)
I know this is a rather general question, but any advice that you might have would be useful to me and students in similar situations. Thank you!
I'm currently working on an MSc in IT. My actual project does not involve NLP directly, but all the 'related research' does (ie. all previous systems which achieve the same goal as mine)so I had to read up on it. This was a struggle... YMMV. I find a few good resources along the way though - recommend reading this first http://www.staff.ncl.ac.uk/hermann.moisl/ell236/manual1.htm then on iTunesU/YouTube you'll find a good course on computational linguistics from MIT, watch the first half dozen lectures at least (don't worry if you don't fully understand them all). A lot of the theory can be traced back to Chomsky's work on grammars, so I suggest reading some of his papers (his style is quite clear). I found I needed to do all this just so I could properly follow the recent papers on NLP (specifically NL interfaces to DBs) that I must review. I really had to start from the beginning, light wikipedia'ing wasn't cutting it. I'm currently reading The Oxford Handbook of Computational Linguistics to consolidate my knowledge, it's pretty good.
So anyway... tough subject, try not to get bogged down. If all the reading's too dry, then 'Godel Escher Bach' is a fun alternative that touches on similar theory.
If I were to choose a pure NLP project myself, I'd maybe do something like 'How NLP can add value to existing systems/software.' ie. look for imaginative practical applications, rather than theorizing yet another parser with some new trick/quirk
Good books or important papers is an involved question. If you ask on MetaOptimize Q+A, we can discuss it there.
http://metaoptimize.com/qa/questions/3096/how-to-choose-a-su...
http://metaoptimize.com/qa/questions/3103/morphology-parsing...
I would avoid a ad or data-selling business upfront. I would focus on a simple transactional business (i.e. they give you money and you give them a product or a service for a month), and learning about your market and why they care about what you're offering.
How do you approach social websites without APIs about scraping their data?
What criteria do you use to determine whether an algorithm you are developing qualifies for legal copyright protection?
NLTK? Weka? Other?
p.s. I don't actually use NLTK or Weka that often.
I think you're better off taking the modules you need from open source libraries, favouring fast C and C++ implementations, and then writing the glue yourself. That way, you won't pay for abstractions you don't need. The glue shouldn't take very long to write.
thx.