I wonder why companies even bother open sourcing their software with all the critics looking for angle's to their motivation. Better to keep it private and not put up with the arm chair open source critics.
The cynic in me thinks it's often a case of a product line they no longer want to support but for which an established user base already exists (Flex comes to mind). The optimist in me thinks this isn't necessarily a bad thing though, it's certainly better than just killing it altogether. The FOSS idealist in me thinks it's great to be getting all this stuff, under an Apache licence!
I can assure you that is not the case. :) We have a very active & growing team focused on the project, and we're all quite excited about it. If you're interested in contributing, please jump in or reach out!
So what's the benefit to using SystemML on top of Spark (or another scheduler) vs. using Spark's own MLlib? From the links below, it seems MLlib supports a superset of SystemML's algorithms (with the small exception of survival analysis). Are there any plans for the projects to use insights from each other, or to merge in some way? Because it seems quite silly for Apache to sponsor two competing machine learning libraries with exactly the same goals.
> Because it seems quite silly for Apache to sponsor two competing machine learning libraries with exactly the same goals.
Apache is willing to provide a home for any community who abides by the Foundation's governance rules and traditions. It is acceptable to have competing projects, and there are many examples in the Foundation's history.
Additionally, Apache doesn't "sponsor" projects per se -- no developers get paid through Apache. The Foundation spends what it takes to support projects, but the annual budget is tiny -- a little over a million each year, when there are ~200 projects and ~5000 committers.
Great question. I'm part of the committer team for the project at IBM, so I'll leave a few comments representing our thoughts. As a quick overview, SystemML provides an R-like DSL, called DML, consisting of linear algebra primitives (vectors, matrices), built-in functions for common functions (such as sums, means, matrix construction, etc.), UDFs, etc., as well as a compiler/optimizer engine that can generate optimized runtime plans from the same DML script for a single node (laptop), Spark, or Hadoop MapReduce. We definitely have algorithms already available as production-ready examples, but the goal of the project is to allow for declarative ML using customizable scripts written at the mathematical DSL level, rather than to provide a fixed library of algorithms at the base language level (Scala, Python, etc.). MLlib (including the newer ML API) is awesome, and provides a great set of algorithms that fit in quite well with Scala, Python (& Java). SystemML is great in that it provides the ability to run customizable, linear algebra-based ML scripts (that can be automatically optimized within the engine) on Spark. Together, it's a great combo. We also have an API for Scala that lets one embed DML into a Scala program similar in manner to how an SQL script can be embedded [http://sparktc.github.io/systemml/mlcontext-programming-guid...].
Very cool - so the algorithms shared with MLlib are just an example of what can be achieved with such a DSL. This could actually be very useful for inference on domain specific generative models that need custom-derived update steps. And probabilistic programming could be built on top of this. I stand very much corrected - this is definitely a project worth attention from the community separate from Spark itself!
This looks interesting and something I will definitely watch, but at this point I think I will still stick with http://h2o.ai/ (another JVM based ML open source project that integrates well with 'Hadoop'). I have been really impressed with the quality of the product and even more so with the quality of the people behind the it.
16 comments
[ 2.8 ms ] story [ 46.9 ms ] threadTheano is from 2007. Torch is from 2002.
[1] http://www.renjin.org/
https://sparktc.github.io/systemml/algorithms-reference.html
https://spark.apache.org/docs/latest/mllib-guide.html
Apache is willing to provide a home for any community who abides by the Foundation's governance rules and traditions. It is acceptable to have competing projects, and there are many examples in the Foundation's history.
Additionally, Apache doesn't "sponsor" projects per se -- no developers get paid through Apache. The Foundation spends what it takes to support projects, but the annual budget is tiny -- a little over a million each year, when there are ~200 projects and ~5000 committers.
Here are our new Apache links:
https://systemml.apache.org
https://github.com/apache/incubator-systemml