Hopefully this can pave the way for deep learning and data analysis for Scala/Java. It's really frustrating having to use a subpar and inexpressive language like Python because of the huge data science network effect.
Or if Julia could get more libraries (it's maybe 25% there, which isn't bad), that would be pretty great as well.
Crippled lambdas, no pattern matching, no static typing (also no HKT), no macros, no threads, no actors, no currying, no practical FP, the list goes on and on..
Moreover, the performance is awful, everything is boxed, and there’s no support for tail call eliminating.
Compared to Scala, OCaml, Common Lisp, or Racket, Python doesn’t even come close in terms of power and speed as the aforementioned languages.
I use python everyday and I find it very hard to work with. I do quant/ml/data science work with it and python has the best libraries of any language in those fields. But there’s no real reason why that should be the case besides maybe that python is so lacking in features that it takes 30 minutes to learn.
> It's really frustrating having to use a subpar and inexpressive language like Python because of the huge data science network effect.
I'm afraid that the "network effect" is gonna continue for the years to come. NDArray is not the only thing you need for data analysis. DataFrame, statistical functions, and data visualization packages like matplotlib and ggplot are all indispensable components in a data analysis workflow. It has taken thousands of contributors a decade to build a solid ecosystem for data science in Python or in R. It's just unlikely for Java to become a mainstream data science language overnight.
I'd say if you are looking for a statically typed language for deep learning, C++ is probably a better bet than Java for now.
PyTorch and Tensorflow both have C++ frontends now, which lowers the barrier of use for me.
Being able to test quickly in Python without compiling is really nice; while I prefer C++ for final products, prototyping PyTorch in Python feels like a better use of my time.
There’s a fuller ecosystem; I needed a PCA implementation. Instead of just using scipy, I had to choose an algorithm and use LAPACK myself.
I don’t really want Java as a data science language, but Scala would be a perfect fit.
There’s already a nice data frame library that I helped develop at my old company called Saddle. Now we just need some quant finance libraries (like the excellent zip line, pyfolio, and alpha lens), some industrial NNs, optimization libs (like cvxpy), and something like scipy.
Is this ever going to happen? Probably not, but a man can dream..
I had to work on some "data pipelines" built with Python & Spark. Terrible experience. All the different data structures for tabular data without a common interface. Oodles of ETL code for transmuting data structures to apply different algorithms. Foraging the docs to figure out even the inconsistent smallest tidbits.
Madness.
I am still aghast. How does anyone get any work done using those tools?
Spark is written in Scala, not Python. I've never used the Python interface, but I've extensively used Spark from Scala and while it's a little annoying to work with sometimes, for "real" Big Data (data set > 1TB maybe), its probably the best tool around.
I'm eagerly awaiting their sparse matrix support. It's unbelievable that the entire JVM doesn't have a single comprehensive, production quality sparse matrix library [0]. This is one of the big things keeping my machine learning in Python.
Yeah, Neanderthal is great (I'm a Clojure a user). It's got support for structured sparse matrices (like Toeplitz) the last I checked, but not general CSC/CSR matrices.
William Kahan, the father of IEEE floattig point arithmetic and one of the main references on numerical errors, has some bad thing to say about doing floatting point computations on the JVM : http://people.eecs.berkeley.edu/~wkahan/JAVAhurt.pdf
An nd array library could be great for machine learning but it is probably not the place where you want to do numerical simulations.
Everything except operator overloading also applies to the JVM... But equally well to python. Basically, the complaint is you don't get any of the hardware fp stuff like traps and signals and all of the math is slightly unportable because it gives different answers on different machines, but these are well-known issues.
There is also the fact that intermediate operation between floats are done in float precision and not double precision (as would be the case with C). It might be the behaviour one would expect but it can lead to serious degradations of a numerical computation.
27 comments
[ 3.1 ms ] story [ 44.4 ms ] threadOr if Julia could get more libraries (it's maybe 25% there, which isn't bad), that would be pretty great as well.
Could you elaborate on this please?
Moreover, the performance is awful, everything is boxed, and there’s no support for tail call eliminating.
Compared to Scala, OCaml, Common Lisp, or Racket, Python doesn’t even come close in terms of power and speed as the aforementioned languages.
I use python everyday and I find it very hard to work with. I do quant/ml/data science work with it and python has the best libraries of any language in those fields. But there’s no real reason why that should be the case besides maybe that python is so lacking in features that it takes 30 minutes to learn.
I'm afraid that the "network effect" is gonna continue for the years to come. NDArray is not the only thing you need for data analysis. DataFrame, statistical functions, and data visualization packages like matplotlib and ggplot are all indispensable components in a data analysis workflow. It has taken thousands of contributors a decade to build a solid ecosystem for data science in Python or in R. It's just unlikely for Java to become a mainstream data science language overnight.
I'd say if you are looking for a statically typed language for deep learning, C++ is probably a better bet than Java for now.
Being able to test quickly in Python without compiling is really nice; while I prefer C++ for final products, prototyping PyTorch in Python feels like a better use of my time.
There’s a fuller ecosystem; I needed a PCA implementation. Instead of just using scipy, I had to choose an algorithm and use LAPACK myself.
There’s already a nice data frame library that I helped develop at my old company called Saddle. Now we just need some quant finance libraries (like the excellent zip line, pyfolio, and alpha lens), some industrial NNs, optimization libs (like cvxpy), and something like scipy.
Is this ever going to happen? Probably not, but a man can dream..
Madness.
I am still aghast. How does anyone get any work done using those tools?
https://www.datacamp.com/community/tutorials/apache-spark-py...
Why didn't they implement RowSet interface?
Or create new tabular data holding interface?
Any way. Whenever someone enthuses about Python, Spark, big data, whatever, I now assume they're insane.
--
The whole notebook (Jupyter) idea is awesome. Are the fans conflating different tools, equating Spark with notebooks?
[0] https://github.com/thrill
[1] https://markus-beuckelmann.de/blog/boosting-numpy-blas.html
[0]: https://neanderthal.uncomplicate.org [1]: https://dragan.rocks/articles/18/Neanderthal-vs-ND4J-vol1
An nd array library could be great for machine learning but it is probably not the place where you want to do numerical simulations.
I believe the main pain points still apply.