I found it got off to a bit of a slow start, but was a fun, rewarding, and very real-world read.
In particular, I appreciated admissions like this:
> Writing our own genetic algorithm in C# was a bad idea. It took us weeks to implement, test, and optimize. Not to mention all the time spent waiting for results. There was a better solution available all along (the optim function in R). Because we didn’t do proper research, we overlooked it and lost time. Sigh.
... which far too few eng blogs overlook / fail to mention.
Wow, just... wow. I wonder why they didn't utilize (multiple) GPUs instead? I would guess it would be far more efficient in all aspects. Especially now that there is TensorFlow & co.
Why $150k? Ordinary GPU (<$1k) would probably be equivalent to 56 CPU cores, if not faster.
I imagine the reason for using CPU was that they would throw it out anyway, so there was no need to make an ideal solution - and they already had those 56 CPU cores.
Yes, of course. I seem to remember that their data is still in their own datacenter though, so they might not be comfortable with using Amazon for this.
I had no idea they have this, but it would probably be an overkill for SO. An ordinary (nVidia) GPU which starts at $150 is roughly >10 times faster at NN problems than 4 core CPU.
That's precisely why, indeed. This 56-core machine is what was at hand at that moment. Genetic algorithms is something we tried, but we didn't have enough confidence in it to invest too much money and effort. Which was a good call, since we soon realized there was a much better approach.
I'm not sure if we're planning to use GPUs for future iterations of algorithm testing; the GPUs are installed in hardware serving production traffic so it's unlikely to be used for experimental stuff like this. That said, a bunch of us just got Dell XPS-15s with the NVIDIA GTX 1050 so we can probably have a go on our local machines to get a feel for performance characteristics... watch this space, I guess!
R is a very well supported language on Windows (since MS bought Revolution Analytics), very well integrated with SQL Server and Azure, and R Server solves a pain point that a lot of real-world users have. And of course it's open source.
Question to spark discussion and for me to fill potential gaps in my knowledge, not to criticize the article as I very much appreciate the transparency and build-up from the simple naive initial approach to the final approach used in production:
Is anyone else bothered by the claim that there "is a 100% chance that the new version is better than the current one" shown by using bootstrap? Maybe I've just never come across such a use of bootstrap through my encounters with statistics. I know it as a tool for resampling from a population to build up properties of your estimator (mean, variance, what have you) when all you have is a dataset and no clue about the actual distribution. When I saw bootstrap with that probabilistic claim, I thought the author would calculate a bootstrapped (100-x)% confidence interval for both the current and the new weights: and if the intervals didn't overlap with one another then you can claim with (100-x)% certainty that one is better than the other. But the author creates a new statistic that is a function of both datasets; Z_i = 1 if new is better than current on iteration i (on a random subset of data) else 0, and for all N=10000 iterations Z_i = 1. The chance/probabilistic claim made of new being better than current is based on the fact that no variation was seen on Z_i (I'm also kind of skeptical that out of so many iterations with random subsets that each time the new weights were better than the current). I think at most you can say that you simulated subsets of the data and 100% of the time new > current; the current claim leads me to believe there's inference that isn't there.
Maybe I should just ask one of my past stats profs. Open to someone enlightening me.
A quantile-based confidence interval from bootstrapping can yield a 100% confidence interval that does not contain 0, i.e., with 100% of cases positive/negative. But that does not (necessarily) mean that there is a 100% chance that the new version is better than the old one. Confidence intervals are not Bayesian credible intervals and cannot be treated as such. (That said, making some certain assumptions about the underlying model can in some times allow one to treat nonparametric bootstraps in such a way.)
Right. The author finds 100% of the time for his current dataset but makes a statement that implies some certainty or inference on future cases. Like taking 100 men, 100 women and finding that 100 randomly matched pairs had the man taller than the woman 100 times, and making the claim that there is a 100% chance that men are taller than women.
The more I type the more I realize how pedantic this is, but we're emphasized in stats to pay extra attention to the conclusions we draw from the data we analyze.
Great read. I loved the honest balance between engineering features and how much they ultimately ended up mattering to users. Oftentimes, the features we select can be quite arbitrary -- it's good to do gut checks by running real-time validation of results as often as possible. Fortunately, at Stack, you've got the userbase to do just that :)
18 comments
[ 2.8 ms ] story [ 65.2 ms ] threadIn particular, I appreciated admissions like this:
> Writing our own genetic algorithm in C# was a bad idea. It took us weeks to implement, test, and optimize. Not to mention all the time spent waiting for results. There was a better solution available all along (the optim function in R). Because we didn’t do proper research, we overlooked it and lost time. Sigh.
... which far too few eng blogs overlook / fail to mention.
Wow, just... wow. I wonder why they didn't utilize (multiple) GPUs instead? I would guess it would be far more efficient in all aspects. Especially now that there is TensorFlow & co.
I imagine the reason for using CPU was that they would throw it out anyway, so there was no need to make an ideal solution - and they already had those 56 CPU cores.
I'm not sure if we're planning to use GPUs for future iterations of algorithm testing; the GPUs are installed in hardware serving production traffic so it's unlikely to be used for experimental stuff like this. That said, a bunch of us just got Dell XPS-15s with the NVIDIA GTX 1050 so we can probably have a go on our local machines to get a feel for performance characteristics... watch this space, I guess!
It's not a neural network, so everything would be hand coded TF operations. It's unclear how that would be quicker than gradient decent.
I'd be more interested to know why they didn't try the learning-to-rank implementations in XGB or LightGBM?
Is anyone else bothered by the claim that there "is a 100% chance that the new version is better than the current one" shown by using bootstrap? Maybe I've just never come across such a use of bootstrap through my encounters with statistics. I know it as a tool for resampling from a population to build up properties of your estimator (mean, variance, what have you) when all you have is a dataset and no clue about the actual distribution. When I saw bootstrap with that probabilistic claim, I thought the author would calculate a bootstrapped (100-x)% confidence interval for both the current and the new weights: and if the intervals didn't overlap with one another then you can claim with (100-x)% certainty that one is better than the other. But the author creates a new statistic that is a function of both datasets; Z_i = 1 if new is better than current on iteration i (on a random subset of data) else 0, and for all N=10000 iterations Z_i = 1. The chance/probabilistic claim made of new being better than current is based on the fact that no variation was seen on Z_i (I'm also kind of skeptical that out of so many iterations with random subsets that each time the new weights were better than the current). I think at most you can say that you simulated subsets of the data and 100% of the time new > current; the current claim leads me to believe there's inference that isn't there.
Maybe I should just ask one of my past stats profs. Open to someone enlightening me.
The more I type the more I realize how pedantic this is, but we're emphasized in stats to pay extra attention to the conclusions we draw from the data we analyze.