Usually when I hear about Apache Sparc, there is Scala being mentioned too. Does Scala have the best support for it and what's the situation on .NET side? (F#?)
All these languages are equal, but Scala tends to be more equal than others in some areas of the API. I also believe R is mostly restricted to the DataFrame API.
Third-party language support:
* Clojure [0]
To develop on Spark in a new, non-JVM language, you'd need a bridge to Java. That's how PySpark works [1], and I believe R follows a similar pattern.
I am thinking that R is the reason why this is happening. Now that Microsoft has supported and invested in R (Smart thing on their part) R will get more Spark features. Having better R connections to Spark's API would be amazing.
Interesting how the whole R thing happened, about three years ago I wouldn't have any idea what R is about and now see it everywhere, even at customer sites.
You're right about Python and R having to pass data back and forth to the JVM for certain operations, but also keep in mind that native code still runs in the native interpreter. That means you have access to the full ecosystem of the native language.
For example, if I want to convert an RDD of JSON strings into Python dictionaries:
Same goes for any external Python libraries I install on the cluster and want to use in my Spark job. You can even run your Python code on PyPy [4]!
For me, working in Python generally feels like a first class experience on Spark. There are areas -- like GraphX [0], certain niche features [1] -- where Scala is definitely easier to work with, but with time that is becoming less [2] and less [3] true thanks to the DataFrame API.
Spark's syntax is basically identical to Scala's, so you can use it as a library with standard Scala functions. In contrast PySpark etc. will still let you get all the same features, but you have to write code that doesn't look like Python, by either using Scala-like code with the RDD API or SQL-like code with the SQL API.
Spark's written in Scala, so Scala will presumably always be the best-supported language. On the inside Spark's also pretty intimately tied to some Java implementation details (e.g., it uses Java serialization to push code around), so any non-JVM language is always going to have to deal with a bit of friction.
That said, from my experience the Python integration is still first-class. I think it might lag the Scala APIs a bit, but that isn't a huge deal unless you really need to be on the bleeding edge at all times. The only really big downside in my book is that there's a small performance hit you take by choosing to use the Python API instead of the Scala one, and that might compound into something of practical significance if you're using it for really big data tasks.
I assume you'd have similar problems using it with R or a .NET language.
There are some people doing ML and data analysis in F#, mostly financial institutions, but Spark is written in Scala so it is all about the JVM eco-system.
Easy to see their motive. This is good for their cloud business, they already know their monopoly days are over, they're making money with Azure and growing that front (127% [1]). Whatever your platform is, you can host it on Azure. And big data will be more lucrative each day. Spark is a cluster computing framework and clusters lead to cloud.
"But there's no way to tease out how much of that $5.1 billion is Azure versus the older products. We also don't know if Azure is profitable, althoug the overall business segment earned $2.58 billion in profits — down 1% from a year ago."
The parent is probably having flashbacks to a time now long gone, in which Microsoft was widely regarded in tech circles as the evil empire. They would attempt to 'embrace and extend' potentially threatening (to them) technology that they didn't control, as a means to hijack or kill it. Many people - including internally at Microsoft - often called it: embrace, extend, extinguish:
I don't know if Microsoft's management still considers E-E-E to be a viable strategy or not.
However, even if they don't, I haven't seen any indication that they've repented of their past scummy tendencies. The Windows 10 rollout is perhaps the most obvious case in point.
I'm curious if you've seen something that makes you more optimistic than I am.
Whether or not you consider their actions evil, they were demonstrably an extremely destructive force, provenly using very untoward methods, much more so than most of their peers.
It's mind boggling to consider how many years their shenanigans have probably set back computing as a field.
As for trusting them now, I don't believe an organization like that can truly change in such a short time.
I'm not sure they can change fully in this short a time, but they've definitely demonstrated some healthy changes under Nadella, including "embracing" open source, VS code, proper handling of the Mojang acquisition, and others.
Funny story from my time at Microsoft (this wasn't representative of the entire population, just a few folks). This was circa 2013:
We were in a meeting talking about infrastructure and testing. I said that we used EC2 for some of our infrastructure, and we were thinking of moving some of our local dev infra to AWS. Someone in the room asked me what that was. I responded with "Amazon web services" and they countered "you mean the company that sells books and stuff?" -- they had never heard of AWS, nor EC2
This comes from someone who had been with the organization for some time. He was very well respected by a lot of folks. He had more awards in his office than I could count. I still think he's a great guy.
If I were to guess, to avoid having to go through the meetings and internal bureaucracy for it. It'd literally be cheaper (after counting in person-hours of time) to just pay for an external vendor.
There's no extra meetings or internal bureaucracy for using Azure internally. It's painless to get an internal subscription for Azure that can be used by your team.
Cheap shot, but this is the company that created TFS in 2005(?). Which as near as I can tell is an answer to the question "How do we reimplement SVN in an MS environment?" Which seems like something no one should have been asking for by that time.
TFS goes rather a bit further than just being a version control system.
That said, I'm not entirely certain why they decided to implement TFVC (TFS's default - and until recently, only - version control system) instead of just basing it on SVN. Aside from that this was at the height of the period when Microsoft had a compulsive need to sink resources into building their own proprietary clone of every damn thing.
Microsoft's dev tools were straight up amazing. As someone who didn't get to take advantage of many of them, I was always envious of engineers using Visual Studio and being able to debug code like it was magic.
Although TFS may seem terrible, it works for super large organizations, and big code bases in a way that SVN and Git just didn't.
I wouldn't say I worked with a super large codebase at the customer who used it, but the only compelling feature seemed to be "it played nice with MS infrastructure."
Teams that had issues grasping the fundamentals of VCS in general still had issues. And in return you got the lesser compatibility and greater number of code warts that came with proprietary enterprise software over open source.
But I'm curious, what worked specifically for you in TFS that wasn't in the SVN ecosystem? It's probably the client was unaware of a lot of features.
I never got to use TFS, but I'd sum up the benefits with /integration/. The MS dev ecosystem is amazing. I write software now for Linux on Mac OS X. I use IntelliJ, Netbeans, virtual box, and a whole bunch of bash to tie it all together.
If I want to do a deploy of my lambda software, I edit some Python in IntelliJ, I then run a build and upload it to lambda using some bash scripts, and test it using the GUI in Chrome.
If I'm editing C code, I edit some code in Netbeans, and edit it over Netbeans SSH integration in VB. I then go compile it using make on the remote machine, and load it. I have a bunch of bash scripts to do testing on that remote machine that are executed via SSH.
-----
I have several more duct taped integrations. With VS, this is all in one system.
I don't think there's a lack of desire, I think there's a lack of ability due to size. I also don't feel it's fair to say that their attitude is "Let's copy AWS! Genius!", Azure has become very mature and a delight to work with.
What part of it seems disorganised? Not trying to be a dick, their roadmap seems clear and detailed and well received by their target audience (primarily MS developers).
I've seen HN light up with joy at news of all the open sourcing and enhancements MS are undertaking in recent years. I don't see the disorganised mess but perhaps I'm missing something?
Users can contribute idle CPU cycles to a variety of projects http://www.hyper.net/dc-howto.html . AFAIK, lot of folks have been doing this for decades.
48 comments
[ 3.1 ms ] story [ 110 ms ] threadThird-party language support:
To develop on Spark in a new, non-JVM language, you'd need a bridge to Java. That's how PySpark works [1], and I believe R follows a similar pattern.[0] https://github.com/yieldbot/flambo
[1] https://cwiki.apache.org/confluence/display/SPARK/PySpark+In...
https://github.com/henridf/apache-spark-node https://github.com/EclairJS/eclairjs-node
Scala -- native API
Java -- java wrappers for Scala API, slightly clunky but same implementation
Python+R -- janky process involving forking a process and feeding strings back and forth between JVM and python/R via pipes
For example, if I want to convert an RDD of JSON strings into Python dictionaries:
Same goes for any external Python libraries I install on the cluster and want to use in my Spark job. You can even run your Python code on PyPy [4]!For me, working in Python generally feels like a first class experience on Spark. There are areas -- like GraphX [0], certain niche features [1] -- where Scala is definitely easier to work with, but with time that is becoming less [2] and less [3] true thanks to the DataFrame API.
[0] https://spark.apache.org/graphx/
[1] http://stackoverflow.com/q/23995040/877069
[2] https://github.com/graphframes/graphframes
[3] http://stackoverflow.com/a/37150604/877069
[4] https://github.com/apache/spark/pull/2144
That said, from my experience the Python integration is still first-class. I think it might lag the Scala APIs a bit, but that isn't a huge deal unless you really need to be on the bleeding edge at all times. The only really big downside in my book is that there's a small performance hit you take by choosing to use the Python API instead of the Scala one, and that might compound into something of practical significance if you're using it for really big data tasks.
I assume you'd have similar problems using it with R or a .NET language.
[1] http://www.businessinsider.com/microsoft-azure-vs-aws-revenu...
Servers and tools did $4.5B of revenue in FY12, Q3 (https://www.microsoft.com/Investor/EarningsAndFinancials/Ear...). This was before Azure really had any footprint.
You do the math.
https://en.wikipedia.org/wiki/Embrace,_extend_and_extinguish
However, even if they don't, I haven't seen any indication that they've repented of their past scummy tendencies. The Windows 10 rollout is perhaps the most obvious case in point.
I'm curious if you've seen something that makes you more optimistic than I am.
It's mind boggling to consider how many years their shenanigans have probably set back computing as a field.
As for trusting them now, I don't believe an organization like that can truly change in such a short time.
Hence, the total apparent lack of desire to do anything new and innovative ("Let's copy AWS! Genius!").
We were in a meeting talking about infrastructure and testing. I said that we used EC2 for some of our infrastructure, and we were thinking of moving some of our local dev infra to AWS. Someone in the room asked me what that was. I responded with "Amazon web services" and they countered "you mean the company that sells books and stuff?" -- they had never heard of AWS, nor EC2
This comes from someone who had been with the organization for some time. He was very well respected by a lot of folks. He had more awards in his office than I could count. I still think he's a great guy.
The Microsoft monoculture hurt quite a bit.
That said, I'm not entirely certain why they decided to implement TFVC (TFS's default - and until recently, only - version control system) instead of just basing it on SVN. Aside from that this was at the height of the period when Microsoft had a compulsive need to sink resources into building their own proprietary clone of every damn thing.
Although TFS may seem terrible, it works for super large organizations, and big code bases in a way that SVN and Git just didn't.
Teams that had issues grasping the fundamentals of VCS in general still had issues. And in return you got the lesser compatibility and greater number of code warts that came with proprietary enterprise software over open source.
But I'm curious, what worked specifically for you in TFS that wasn't in the SVN ecosystem? It's probably the client was unaware of a lot of features.
If I want to do a deploy of my lambda software, I edit some Python in IntelliJ, I then run a build and upload it to lambda using some bash scripts, and test it using the GUI in Chrome.
If I'm editing C code, I edit some code in Netbeans, and edit it over Netbeans SSH integration in VB. I then go compile it using make on the remote machine, and load it. I have a bunch of bash scripts to do testing on that remote machine that are executed via SSH.
----- I have several more duct taped integrations. With VS, this is all in one system.
I've seen HN light up with joy at news of all the open sourcing and enhancements MS are undertaking in recent years. I don't see the disorganised mess but perhaps I'm missing something?