Ask HN: APL family instead of R for data analysis?
I've read a lot about the APL languages, but never used them because I felt they were too specialized. Recently, I've been roped into some projects using R. I fully understand why R is so popular in statistics departments, but programming in R has been the most migraine inducing experience I've ever had with a language.
Since R is an array language, I though maybe an APL family language would work as a replacement. It wouldn't have the multitude of libraries that R has, but would have a clean design. Note that I am not thinking of this as an alternative for the typical R user, who is a non-programmer and for whom R is their first and possibly last language. I'm thinking of a moderatly experienced programmer who is faced with a data analysis task.
Some APL derivatives, like J and Dyalog APL, have had their capabilities extended well beyond simple array juggling. How are they, compared to R, as data analysis environments?
54 comments
[ 3.2 ms ] story [ 118 ms ] threadhttp://statistics.ucla.edu/system/resources/BAhbBlsHOgZmSSIp...
With R, you can do data preparation, but I find it easier to do it with Perl or awk than import the cleaned data to R. Is a standard workflow with J pretty much the same? I thought that with PCRE and other libs available in J, it might be suitable for more than just juggling cleaned data.
I should admit I was asking myself the question of this topic.
If you're familiar with RC, you know that it encouraged participation with unusual languages, as it's generally hard to find lots of examples of code for them. RC is a code chrestomathy site, but for Java and likes it's easy to find many good examples elsewhere. So rare languages comparatively shine on RC.
That said, in F# (which is a functional-first language with some solid programming language design background), one can call R functions in a fairly nice way using a type provider: http://bluemountaincapital.github.io/FSharpRProvider/.
This probably cannot replace R for typical statisticians, but it is a nice option for programmers...
And R is not so bad as a language. It's OO is really bad, but it has functions as first-class members and great support for NAs. Combined with Hadleyverse and pipe operator (magrittr) is makes a very readable code.
I just finished reading _Data Analysis_ by Peter Huber. One chapter is about programming languages for statistics. In the language that he developed (ISP), he included only a small core of statistical facilities because in his experience that's all people really need. David Hand has also written observations along the same line.
Sophistic algorithms for missing data, sparse solutions, latent modelling. You can find the many industrial grade algorithms in Python, Spark, Julia, Weka etc. but for stats-heavy data science/machine learning R is unmatched. Sans writing your own implementations based on pseudocode, but this is really not effective use of your time for prototyping.
EDIT: And even for simple stuff, the strength of R packages is easy to prove since they are either directly ported to other languages (ggplot) or heavily influence the implementation (pandas).
For everything else, one can call arbitrary R packages with Rpy2, albeit with clunkier syntax.
For agent based modeling, there is no comparison. Python with its better and faster class system and Numba is in another league.
Finally python has distributed and out of core lists, arrays and dataframes: http://dask.pydata.org/en/latest/
One, "clunkiness" of class creation (which I guess is subjective); I don't think it's very readable, but maybe that's bias from other languages.
Second the S3/S4/R4 mess.
As you probably know, Q/K are in the lineage of APL languages, with strong inherent vector-oriented capabilities, but Q uses an SQL-like dialect of words to replace APL's symbols.
I've always found the idea of APL exciting, but it seemed to be a very isolated platform. "How do I use this on my site?"
Q speaks SQL natively (using s.k) so it's easy to work into my MySQL-based flow, though not exactly the same as MySQL's syntax.
It's got great built in facilities for bulk data loading and storing, which saves you a lot of time with the boring "getting my data set up" step when banging out little helper scripts.
I've begun using Q as a cache in place of memcache/redis because I much prefer being able to ask flexible questions with a dynamic query language. Example: A MySQL query on a 1m row table that was often showing up in my slow query log at 1sec+ took less than 50ms in Q.
Obviously Q and MySQL are apples to oranges, but it is still a handy adjunct when you need serious speed without full loss of flexibility.
And writing Q directly is a really interesting exercise. My PHP and Node code has gotten better as a result.
There's also a port of re2 available: http://code.kx.com/wiki/Cookbook/regex
The C API seems pretty simple once you get over the insane function (and macro!) names, so I think a pcre port could be done.
In practice though I think Q gods prefer a split-based vector oriented approach when it comes to string chomping. Q's built in verbs and adverbs function much like regex operators do, but at an abstraction level tied fundamentally to your data.
Getting into J: I like J a lot. I like how professional the software is, their excellent docs, and their amazing community on IRC. But for some darn reason, the syntax just doesn't make sense to me. I know that sounds silly when we're talking about Q or K which are pretty unreadable, but I kinda-sorta grok the latter a lot more easily (the difficult part is adverbs, etc). I get why they chose the ascii chars they chose; with J, I feel totally lost. "NB."?! Cmonnnn.. :)
Kona is based on an earlier version of K, the language that underlies Q, and though I am sure you could learn a ton from it, I need something that's been vetted a bit more because I intend to put it into production (crazy as that sounds) and Kona is still developing much of their HTTP and database support. I also wanted to learn the latest iteration, Q, which Kona hasn't implemented a dialect of just yet. Still a cool project that I follow actively on Github.
I'm struggling with the 32bit vs 64bit thing now. I know I couldn't ever justify the 64bit version with my pathetic budget. The great thing about the 32bit version is that you can easily spawn multiple copies of the interpreter, and IPC is incredibly fast (1mil sync messages to localhost in 52sec on my SurfacePro3, 607msec async). So theoretically you could easily build "microservices" that each live in 4gb and pass messages back in forth[1].
At least in theory. I'll be documenting my experiments soon if all goes as planned. Already seeing big benefits.
[1] That sounds hacky as hell coming from most platforms, but IPC in K/Q is baked in as a fundamental, powerful primitive, a bit like how Erlang's messages underlie the entire system. For instance, their Tick system (http://code.kx.com/wiki/Startingkdbplus/tick) used for insane volume is an ingenious way to feed out tons of data to tons of clients (firehose) and let them pick their own views of it.
J seems quite reasonable - intuitively - after reading very good "J for C programmers" (http://jsoftware.com/help/jforc/contents.htm) . And symbols in J chosen for most operations also seem good - at least after some small use. "NB." is immediately "nota bene" (I guess knowing some Latin doesn't hurt - it's "note well", but I always understand it as "note that") - don't some other languages use this acronym for comments as well?
What is IMO particularly good in K is the small size of the interpreter. J is "for the whole math", while K is "for practical 99% of cases" - at least it feels like that. Having small interpreter which fits in CPU cache allows potentially to run faster than with assembler, because K code is packed densely - so less time to wait for slow memory.
In a Previous Age, there were “rumors that there would be interest in opening up the language if there would be enough community response around it”—wonder if they amounted to anything [1]. Interest in k is only growing it appears.
[1] http://www.kuro5hin.org/story/2002/11/14/22741/791
There's a GPL implementation of K3 called Kona at https://github.com/kevinlawler/kona ; kevin lawler also has a language called kerf (not open source) that is based on the same principles with a different syntax (JSON+SQL+more).
John Earnest has a Javascript implementation of K5 called oK at https://github.com/JohnEarnest/oK
Andrey Zholos has a JITing implementation of a K-like language at https://github.com/zholos/kuc
Thanks also for adding a linux binary today, looking forward to playing with it. Can't say I'm as excited about the 1 month timer though.
After some recent forays into Clojure, Haskell, and R, I find myself getting re-interested in languages in the APL family.
I had read about Q, K and Kdb+ a while ago. Guess I'm going to have to kick the tires.
I also suggest reading some of his papers like "Tidy Data".
R is weird in that you NEED a large set of CRAN libraries to do work, whereas in Python you can do a fair amount with the standard library, and PyPI is relatively weak in comparison.
I'm also a programmer who learned R for data analysis. It does take quite awhile and some pain. Part of the difficulty is that R is weird and has warts, but an even bigger difficulty is that you are learning new concepts and programming abstractions (fundamental difficulty vs accidental difficulty). R passes the test of a language which changes the way you think.
R has its warts but is likely the most practical choice. It has the full set of things you need -- data preparation and cleaning, exploratory analysis, model building, and visualization.
Disclaimer: I'm a civilian, not a programmer
Best solution is to do most of your work in Q and call into R when you want to use a package. The R integration works well, see http://code.kx.com/wiki/Cookbook/IntegratingWithR.
The reality is that if you are using (and paying for) Q/K, you are likely doing so because you're dealing with billion+ point datasets. At least. R just melts and falls apart at that kind of size (without using native code, which isn't really R then is it?). So I often end up just translating R functions into K equivalent, and just using R for the nice-to-have features like latex/brew/ggplot/etc.
The two big things I'm trying to get here are a comparison of how APL-family compares to R in that role and in reaching beyond that role to handle the stuff before and after.
This is pretty unique, and works better than spark for out of core on one machine...(and easier to set up.)
For stats not in the statsmodels and scikitlearn packages, you can easily whip up the bayesian generalization in pymc3.
Then if there is another R package you need, you can use Rpy2 to call it.
Not sure if this would be relevant to your usecase.