Very impressive library! Is it still in the developmental stage, or do you know of anyone using it for real yet?
Also, I love livebook! I keep a tab with it open and it has replaced my use of iex. Can you talk a little bit about how you decided to embark on building that, too? An ML library and framework is a big enough job already, how did you decide to add Livebook to the list?
This release means we are getting closer to prod usage but Nx requires one of the existing compilers/backends to shine. So I am hoping we will see production uses once we officially release Torchx/EXLA packages too.
---
Curiously, I had two ideas for the name Livebook:
1. An interactive book to teach LiveView. LiveView is a new approach for developing web apps and I thought it could be worth exploring different non-traditional ways of teaching it too!
2. Livebook as code notebooks (what we got!)
It just feels a great fit for the platform, considering Phoenix, LiveView, Distributed Erlang, etc. So why not? :) And I am curious to see if and how we can contribute new ideas to the already extensive ecosystem of code notebooks!
In general notes, it is an effort to bring more use cases to Elixir, the Erlang VM, and the ecosystem. But it is an area I have been interested on (back in 2009 my master thesis was actually within Machine Learning). When I met Sean (through the Genetic Algorithms in Elixir book), our interests aligned and we started working on it!
Off-topic: hi Jose, what's your expectation on how much more raw performant Erlang can become in the next 1-2 years?
PHP ~3x in performance from 5.x to 7.x. LuaJIT is even faster over vanilla Lua.
There were multiple attempts at creating an Erlang JIT for well over a decade now and last year BeamAsm was released but in everyday use it seems to net about a 10-15% boast.
I ask because it'd be great if we didn't have to create so many Rust NIFs in order to address Erlang raw slowness.
Curious what your take is on speeding up raw Erlang (please note, I'm not talking about concurrency & I do truly love Exilir/Erlang/OTP)
I am not informed enough to give actual estimates. :)
Although your estimates for JIT improvements seem low on my experience. Compilation times are consistently half of what they were before and test suites range between 33% to 50% faster. Even Whatsapp reported server efficiency increased by 25% and many other factors are likely at play there (source: https://twitter.com/wcathcart/status/1385253969522413568).
I _assume_ we will continue seeing improvements on this front, especially if at some point we start doing cross module optimizations!
I just want to thank you for your incredible work. I'm a fellow countryman migrating a Python program that fetches financial data from a large amount of sources, that's become very hard to maintain, to Elixir. The BEAM fits my use case perfectly, and Elixir has been an absolute joy to work with and debug.
I was particularly delighted to see how you built `defn` on top of existing metaprogramming functionality. This kind of thing has lead me to think of Elixir as a Lisp in Ruby's clothing running on the BEAM. :) I hope that's not too inaccurate. Were there any delightful "aha!" moments you got while building this?
Are there any specific advantage that Elixir being on BEAM VM brings to Machine Learning and Data Science? I am assuming BEAM VM's actor model fits well with the GPU / Tensor usage ? ( I have zero knowledge on ML and Data Science )
Personally I think this is a great move from Elixir community and especially coming directly from the creator. Some people may be dismissive ( as with at least half of HN ) But I see this with Ruby where there is a chicken and egg problem. The community stuck with Web or Rails Development only. And no one has incentive to kickstart the languages ecosystem in different directions.
So instead of Elixir being another language for Web / Phoenix, it could now ( or in the future ) be used as ML with some level of official support.
You are correct. Currently we are waiting for the issue to be tackled upstream (both on LibTorch and Google's XLA). Someone could also implement a specific Nx-backend for Apple. It should actually be fun but not high priority at the moment.
I’ll plug some to Jose Valim’s Advent of Code solutions that included NX approaches alongside “standard” Elixir ones for some of the days. It was good to see and get some ideas of what’s possible.
I was learning Elixir through advent of code, and this was an invaluable resource for getting my head around an more functional (or Elixir-y?) style of problem-solving.
I love Jose Valim's work (even back to the inherited resources days!), but with a very popular Nx already out there (nx.dev a dominant player in the monorepo space) this naming just adds to confusion. Nx is on the thoughtworks radar so not exactly unknown:
Well there isn't an Nx.dev plugin for Elixir, is there? Nx for Elixir is a nice shorthand for Numerical Elixir. Nx.dev just looks like a consulting company jumping on a short domain when the .dev TLD became available and then using it for a project later. The nx.dev site was only launched in 2019, and then as an Angular dev tool, and as an Elixir dev I hadn't heard of it until after Nx elixir existed, which seems to have started some time in mid to late 2020.
it's totally not the worst thing that nx was also the name Valim chose (I have no investment in to tool or company myself), but it is bad for both projects.
My point was that it wouldn't have been on the radar for anyone in Elixir land, and having a name clash with what looks to be a JS focused tool isn't that big a deal when they're in really different domains.
What's the goal here? It seems like they want to make elixir the new AI language, but why should I go from python to Elixir? I know it seems a dumb question but I can't find in the link any reason why. I really like python but at the same time I would like to have something faster but also with pandas, numpy, statsmodel and so on
This a parallel and independent ecosystem for numerical computing with equivalent libraries:
* pandas - explorer
* numpy - nx
* statsmodel - Not sure this exists yet
The article answers this pretty well.
However, our goals are also to:
* make Elixir a suitable platform for new Machine Learning developments
* fully leverage the power provided by the platform Elixir runs on, the Erlang VM
* provide consistency and stability, especially when working on a domain that is still actively evolving
For those reasons, we chose to invest on Nx as its own foundation, agnostic to any particular framework. The road is definitely longer but we also believe the pay-off will be higher too!
I think right now, this is for people who already know and need the advantages of Erlang/Elixir and want to be able to do numerical computing without having to resort to a NIF or ports. Numerical computing has been one of the big weaknesses with the BEAM platform, even though it is among the first things people think about when they find out about how BEAM does concurrency. I can tell you that many Elixir developers are quite excited about this. It is definitely scratching an itch.
I don't know how this is emerging. If I take a stab at it, BEAM/OTP/Erlang/Elixir is very good at coordinating among many concurrent processes as well as handling failures that come with it. There really isn't any other language platform that does that as well, including Python. (Although there is interest in recreating those advantages in Rust). The distributed computing is built on top of those concurrency primitives. So when one thinks about adding numerical computing into the mix, I think of:
- Broadway (an Elixir library) that handles workflows from unreliable data sources. It is already in production use to handle massive, distributed web scraping operations, so embedding a data transform in there with numeric computing will have good synergies
- IoT applications, both at the edge with Nerves, and with the command/control for IoT. Each of the IoT devices are unreliable (power and network), and OTP already has great facility for that. Being able to do numeric computing at the edge is a thing.
Tried to learn Python a few years ago. I generally like almost every language I try. I'll even sing you Perl's praises.
After learning other languages though, I cannot see the point behind using Python aside from broad adoption. Couldn't imagine choosing it for any problem I needed to solve.
I was actually really looking forward to it after hearing for years about how Python would have "one way to do everything" but there's a gazillion different ways just to run Python on your machine, different versions of it to choose from.
Should I be running the local system version, PyPy, Stackless, Anaconda, Miniconda? Is PyEnv the weapon of choice for multiple Python codebases or is it one of the many others? The Python 2 vs 3 migration stuff I expected to be a little wonky but all of the stuff on top of it was just weird for a community that had so long been advertised as having agreed on solutions.
I did finally get why a lot of experienced developers call it "The Okayest" language.
Hi @joelbluminator, may I ask you why you feel the need to frequently jump on Elixir threads here and on Reddit and be negative about other people's work?
It's completely unrealistic Elixir (or anything else - Julia, whatever) will replace Python as the main AI/ML language, that's what I wanted to convey. I agree I was being a bit too snarky but what I said is true and was an answer to what OP asked.
You can't be that sensitive while reading internet comments, you'll have a rough time.
As for my past comments on Elixir - I tend to remain factual. I used to have a hard problem with the way the Elixir community was crapping all over Ruby to poach mind share but I moved on. Ruby also did that to Java when it was starting so that's life.
Yes, and I have replied we are not trying to make Elixir the new AI language either.
In any case, I was not speaking about this comment exclusively. It is a recurring pattern. But given you have somehow managed to make this my fault, I will drop the conversation. Have a good one!
I'm saying for anyone's own mental health, we shouldn't expect constant praise and positivity from comments. You're gonna read stuff you won't like on the internet.
It probably won't and I don't think that's the goal. However, Elixir is making pretty significant roads for distributed data processing pipelines thanks to the Broadway framework (https://elixir-broadway.org/).
AI/ML applications were going to be a natural next step after that. It's exciting for Elixir because number crunching was the biggest computational weakness for the language for the last few years and many thought it was a limitation of BEAM languages in general. Jose's work has shown that it's not.
We are not trying to make Elixir the new AI language, but one possible language for AI. @hosh covered some of the use cases in his reply and I can add two more insights:
* Lately we have seen functional ideas brought into Python through projects like JAX and Thinc.ai and I think exploring those concepts within a functional language is both interesting and exciting (and we are not the only ones doing so!)
* Besides IoT and data pipelines, there is potential in mixing some of the current ML trends, such as Distributed and Federated Learning, with the capabilities of the platform
We likely aren't much attractive to Python developers right now, unless you are interested in contributing/designing ML tooling (for learning, hobby, or professional reasons) or you have a use case that suits one of the mentioned strengths of the platform.
As a language are there any parallels between "Wolfram Mathematica" and "Elixir", since both talk about pattern match, search and replace in the context of evaluation a lot.
45 comments
[ 3.9 ms ] story [ 106 ms ] threadAlso, I love livebook! I keep a tab with it open and it has replaced my use of iex. Can you talk a little bit about how you decided to embark on building that, too? An ML library and framework is a big enough job already, how did you decide to add Livebook to the list?
---
Curiously, I had two ideas for the name Livebook:
1. An interactive book to teach LiveView. LiveView is a new approach for developing web apps and I thought it could be worth exploring different non-traditional ways of teaching it too!
2. Livebook as code notebooks (what we got!)
It just feels a great fit for the platform, considering Phoenix, LiveView, Distributed Erlang, etc. So why not? :) And I am curious to see if and how we can contribute new ideas to the already extensive ecosystem of code notebooks!
PHP ~3x in performance from 5.x to 7.x. LuaJIT is even faster over vanilla Lua.
There were multiple attempts at creating an Erlang JIT for well over a decade now and last year BeamAsm was released but in everyday use it seems to net about a 10-15% boast.
I ask because it'd be great if we didn't have to create so many Rust NIFs in order to address Erlang raw slowness.
Curious what your take is on speeding up raw Erlang (please note, I'm not talking about concurrency & I do truly love Exilir/Erlang/OTP)
Although your estimates for JIT improvements seem low on my experience. Compilation times are consistently half of what they were before and test suites range between 33% to 50% faster. Even Whatsapp reported server efficiency increased by 25% and many other factors are likely at play there (source: https://twitter.com/wcathcart/status/1385253969522413568).
I _assume_ we will continue seeing improvements on this front, especially if at some point we start doing cross module optimizations!
I was particularly delighted to see how you built `defn` on top of existing metaprogramming functionality. This kind of thing has lead me to think of Elixir as a Lisp in Ruby's clothing running on the BEAM. :) I hope that's not too inaccurate. Were there any delightful "aha!" moments you got while building this?
https://lfe.io/
Personally I think this is a great move from Elixir community and especially coming directly from the creator. Some people may be dismissive ( as with at least half of HN ) But I see this with Ruby where there is a chicken and egg problem. The community stuck with Web or Rails Development only. And no one has incentive to kickstart the languages ecosystem in different directions.
So instead of Elixir being another language for Web / Phoenix, it could now ( or in the future ) be used as ML with some level of official support.
any idea on M1 gpu support? (know it's mostly an upstream issue)
excited about "real"/intensive ML workflows locally..
https://github.com/josevalim/aoc
I was learning Elixir through advent of code, and this was an invaluable resource for getting my head around an more functional (or Elixir-y?) style of problem-solving.
https://www.thoughtworks.com/radar/tools/nx
I know for a fact it has already confused elixir developers who were looking for an elixir plugin for Nx (the monorepo tool).
[1]https://web.archive.org/web/20190501153325/https://nx.dev/
the nx.dev site is just what I'm pointing to, dunno what url it was before that.
no idea if there was an elixir plugin for nx (easy enough to integrate https://www.youtube.com/watch?v=IRIXPTIKTmA&t=6s) but I'd have no hope of finding it now .
it's totally not the worst thing that nx was also the name Valim chose (I have no investment in to tool or company myself), but it is bad for both projects.
* pandas - explorer
* numpy - nx
* statsmodel - Not sure this exists yet
The article answers this pretty well.
However, our goals are also to:
* make Elixir a suitable platform for new Machine Learning developments
* fully leverage the power provided by the platform Elixir runs on, the Erlang VM
* provide consistency and stability, especially when working on a domain that is still actively evolving
For those reasons, we chose to invest on Nx as its own foundation, agnostic to any particular framework. The road is definitely longer but we also believe the pay-off will be higher too!
I don't know how this is emerging. If I take a stab at it, BEAM/OTP/Erlang/Elixir is very good at coordinating among many concurrent processes as well as handling failures that come with it. There really isn't any other language platform that does that as well, including Python. (Although there is interest in recreating those advantages in Rust). The distributed computing is built on top of those concurrency primitives. So when one thinks about adding numerical computing into the mix, I think of:
- Broadway (an Elixir library) that handles workflows from unreliable data sources. It is already in production use to handle massive, distributed web scraping operations, so embedding a data transform in there with numeric computing will have good synergies
- IoT applications, both at the edge with Nerves, and with the command/control for IoT. Each of the IoT devices are unreliable (power and network), and OTP already has great facility for that. Being able to do numeric computing at the edge is a thing.
Give elixir a shot. I used to use python (though not as much as I used ruby) and I will never go back to python.
After learning other languages though, I cannot see the point behind using Python aside from broad adoption. Couldn't imagine choosing it for any problem I needed to solve.
I was actually really looking forward to it after hearing for years about how Python would have "one way to do everything" but there's a gazillion different ways just to run Python on your machine, different versions of it to choose from.
Should I be running the local system version, PyPy, Stackless, Anaconda, Miniconda? Is PyEnv the weapon of choice for multiple Python codebases or is it one of the many others? The Python 2 vs 3 migration stuff I expected to be a little wonky but all of the stuff on top of it was just weird for a community that had so long been advertised as having agreed on solutions.
I did finally get why a lot of experienced developers call it "The Okayest" language.
In any case, I was not speaking about this comment exclusively. It is a recurring pattern. But given you have somehow managed to make this my fault, I will drop the conversation. Have a good one!
AI/ML applications were going to be a natural next step after that. It's exciting for Elixir because number crunching was the biggest computational weakness for the language for the last few years and many thought it was a limitation of BEAM languages in general. Jose's work has shown that it's not.
We are not trying to make Elixir the new AI language, but one possible language for AI. @hosh covered some of the use cases in his reply and I can add two more insights:
* Lately we have seen functional ideas brought into Python through projects like JAX and Thinc.ai and I think exploring those concepts within a functional language is both interesting and exciting (and we are not the only ones doing so!)
* Besides IoT and data pipelines, there is potential in mixing some of the current ML trends, such as Distributed and Federated Learning, with the capabilities of the platform
We likely aren't much attractive to Python developers right now, unless you are interested in contributing/designing ML tooling (for learning, hobby, or professional reasons) or you have a use case that suits one of the mentioned strengths of the platform.