I feel the supply of managed pimped out jupyter lab instances is plenty right now. What is really not clear is the path from there on to pipelining and workflow building with other components in a distributed/cloud architecture. There are also a lot of open source options like KubeFlow, prefect, ludwig etc. all of which have their problems and quirks, as well as the managed offerings like SageMaker.
It's the enterprise version. They can afford it. Besides it's not like everyone in the org will be having a seat. Only the people that are doing data science.
I work in a very large enterprise that could definitely benefit from such a product, but at that cost I'm not even going to try mentioning it, it's never going to pass.
We have engineers in the thousands so that would be a budget in the millions per year, for a tool for which it's hard to demonstrate the productivity benefit over alternatives. Not going to happen.
It's not that the company can't afford it, but enterprises are strict when it comes to spending money. There are lots of processes, checks, and people that sign off.
On-Prem (even in private cloud) solutions like this seem to always be pricier, includes dedicated support too. They have a cheaper $19/m and free tier on https://datalore.jetbrains.com/
Average dev salary at my org is north of 100k, but I do work in the US so lets say we have a developer earning half that. At $125 a month, it works out to be around 3% of monthly compensation (not including taxes and benefits). This only has to improve productivity by a tiny amount to be worth it.
That is not how it works, having a water cooler in the office increase the productivity 100x vs not having water, that does not mean you should pay millions for one. That is a myth invented by SAAS vendors and consultants to justify their sky-high price. The value offered of course factors in the price but many other factors too (scarcity of materials and resource to produce the good,cost of production, maintenance cost, cost of the products of your competitors, risk of vendor lock-in, etc)
Plenty of places pay X for tools that add more than X in productivity value.
In fact, nearly every tool I have ever gotten at a company worked like this. Most of them are also willing to test pricey tools to see if they would pay off, and when they do, the company starts buying such tools.
If you don't work at such a place, look for a place that values developer time.
I worked for a Fortune 20 company so you can stop the patronizing tone. A paper and a pencil also increases productivity by a lot ( perhaps more than any tool) that does not mean you need to pay 5% of your developer salary by month for them.
But, you likely would pay 5% of salary (or more) for a paper and pencil (to continue with your analogy) if you had no other choice and there was no alternative tool that could substitute. So I'm not sure what point you are trying to make.
Create a landing page. I will write a couple of posts saying that Sam Altman and Paul Graham are the smartest guys this century and we will be soon launching a a show HN, new YC company.
I think it'd be really hard to prove any single tool provides you a specific productivity boost. Most engineers probably have a tool-set. Which means all those tools work together nicely. Taking one of them out, as well as adding one, might break the whole setup. Usually established engineers are not working in a vacuum, they already have their setups in place, so justifying a 3% extra cost might be very hard to justify for very unclear benefits, if any. I'm not trying to make a definitive argument, just some food for though.
Is it just me or does this look like flowered up jupyter? Idk, I'm a bit sceptical-JetBrains have a few really solid products but all the ones that come with a web interface are... I can't think of a word that can truly describe them. You get the idea.
The two pain points of using a Jupyter Notebook are,
cell dependencies, not being able to partially re-compute only what's needed when you change something like Excel does, and missing a concept of library/immutable codes.
JetBrain is specialized in listening devs, code mining and smooth integration. It seems like a good match, at least in my head :)
I greatly enjoy using pycharm, but the jupyter integration is and extremely mediocre way of using notebooks... which themselves have issues for serious development work.
I'm not really getting excited over this, sadly.
It's an enterprise-branded product, but nothing in this product stops people from writing the same old spaghetti. Then it's up to the engineers to figure out a way to get the spaghetti to behave.
I'd rather see some dev platform that forces an analyst or data scientist to think about integrating with other infrastructure in a sane way. Your code, conformed and embedded into some CI/CD pipeline. easy to set up data pipelines. unit/integration/data integrity test boilerplate generators geared towards data science.
(I know I'm probably describing something that's impossible... one can dream)
:shakes-fists: why doesn’t everyone understand software engineering as well as the software engineers. Wait, let’s force it on them, what could go wrong?
Might even be closer to rstudio server[1]. License software but run it on your own hardware. IMHO no one has yet come close to rstudio in terms of providing a complete-ish solution: multiple account support, full-featured ide, hosting env taken care of etc.
> Is it just me or does this look like flowered up Jupyter?
Must be JetBrains’ headline writers too, since the headline is “Announcing Datalore Enterprise – The Smart and Secure Jupyter Environment for Data Science Teams” making it sound rather like flowered up Jupyter.
On a tangent, I'm building an open-source data IDE focused on developers/engineering managers rather than data scientists. The goal is to be able to easily make SQL queries, HTTP requests, load files, script in Python, and visualize results all in a single place.
I think compared to Jupyter Notebooks this can more directly solve a problem for anyone who wants to do analysis on customer data, historic log data, JIRA tickets, incidents, etc. The long-term goal for a commercial version would be to have high level connections to all the APIs developers use; to make cross-datasource analysis easier.
Make the REPL ergonomic enough and it will crush any notebook interface. I think that's one reason why R users barely use Jupyter at all - RStudio's REPL is better than most of the out-of-the-box Python REPLs so R users don't have to resort to Jupyter (and there's rmarkdown for the rare cases when notebooks actually make sense).
I've used the cloud version of Datalore a little bit, and it's been excellent. Very nice jupyter environment, and it had very reasonable resource constraints on the free version. Version control works very well, which is a nice upgrade on some similar solutions I've tried to use.
Everything I've ever tried to use that plugged into Hub has been less excellent. I don't think I have the problems that's meant to solve. Our on-premise TeamCity server works just fine without it, and we never laid down the entire suite of other tools that would use Hub.
To be fair, having it for individual use comes at a fair price[0]. The real money[1] is when the grad students go into their jobs and demand jetbrains IDEs since, as you said, they don't want to learn a new IDE.
I’ve brought my jetbrains license with me to three jobs. I see an IDE as no different than a mechanic’s tools, and relatively speaking a personal JB license is less money than a mechanic spends on tools in a year.
Technically, if it’s a personal license, that’s violating the license terms since jetbrains only grants it to you for non-commercial use, but I’m not aware of them enforcing this to any degree.
This is enterprise edition.
There is a community edition[1], and it is 3 yeas old[2]. I wasn't aware of that:
Other cloud offerings (besides Coalb) are more like a vanilla Jupyter. This one seems more customized. Not sure it's a good thing, but it's understandable as a JetBrains product.
I think Colab is indeed more comparable here. Overall Colab seems a bit better on available resources, but Datalore UI seems more powerful. Also, it seems to have R support. That can be a killer for would-be Colab users who aren't opted to Python.
42 comments
[ 4.1 ms ] story [ 88.9 ms ] threadWe have engineers in the thousands so that would be a budget in the millions per year, for a tool for which it's hard to demonstrate the productivity benefit over alternatives. Not going to happen.
It's not that the company can't afford it, but enterprises are strict when it comes to spending money. There are lots of processes, checks, and people that sign off.
Plenty of places pay X for tools that add more than X in productivity value.
In fact, nearly every tool I have ever gotten at a company worked like this. Most of them are also willing to test pricey tools to see if they would pay off, and when they do, the company starts buying such tools.
If you don't work at such a place, look for a place that values developer time.
Oh and noise cancelling headphones.
What makes Jetbrain's iteration far better than any of the other competitors? So far nothing?
JetBrain is specialized in listening devs, code mining and smooth integration. It seems like a good match, at least in my head :)
I'm not really getting excited over this, sadly.
It's an enterprise-branded product, but nothing in this product stops people from writing the same old spaghetti. Then it's up to the engineers to figure out a way to get the spaghetti to behave.
I'd rather see some dev platform that forces an analyst or data scientist to think about integrating with other infrastructure in a sane way. Your code, conformed and embedded into some CI/CD pipeline. easy to set up data pipelines. unit/integration/data integrity test boilerplate generators geared towards data science.
(I know I'm probably describing something that's impossible... one can dream)
Curently in private beta (just sign up for it, you'll get access).
https://cloud.google.com/notebooks
https://notebooks.azure.com/
https://aws.amazon.com/emr/features/notebooks
https://colab.research.google.com/
https://www.kaggle.com/code
https://gradient.paperspace.com/
[1] https://www.rstudio.com/products/rstudio/
edit: clarify
Must be JetBrains’ headline writers too, since the headline is “Announcing Datalore Enterprise – The Smart and Secure Jupyter Environment for Data Science Teams” making it sound rather like flowered up Jupyter.
I think compared to Jupyter Notebooks this can more directly solve a problem for anyone who wants to do analysis on customer data, historic log data, JIRA tickets, incidents, etc. The long-term goal for a commercial version would be to have high level connections to all the APIs developers use; to make cross-datasource analysis easier.
https://datastation.multiprocess.io/
Everything I've ever tried to use that plugged into Hub has been less excellent. I don't think I have the problems that's meant to solve. Our on-premise TeamCity server works just fine without it, and we never laid down the entire suite of other tools that would use Hub.
https://starboard.gg/
They infiltrated my college and got all of our students to use phpstorm for free. Upon graduation they smack you with a huge cost to continue.
I imagine quite a few grads didn't want to learn a new IDE.
0: https://www.jetbrains.com/phpstorm/buy/#personal?billing=yea...
1: https://www.jetbrains.com/phpstorm/buy/#commercial?billing=y...
Other cloud offerings (besides Coalb) are more like a vanilla Jupyter. This one seems more customized. Not sure it's a good thing, but it's understandable as a JetBrains product.
I think Colab is indeed more comparable here. Overall Colab seems a bit better on available resources, but Datalore UI seems more powerful. Also, it seems to have R support. That can be a killer for would-be Colab users who aren't opted to Python.
- [1] https://datalore.jetbrains.com/
- [2] https://blog.jetbrains.com/blog/2018/10/17/datalore-1-0-inte...