I may be wrong. It happens a lot! But I think Arrow's vision encompasses compute, not just a data format and data interface. https://www.slideshare.net/wesm/pycon-colombia-2020-python-f... Slide 43: The "Arrow C++…
Fair. I agree Arrow is still more of a vision than anything else. > it didn't even provide a shape attribute I suspect this has to do with the project's focus. I think they aspire to be a back-end to DataFrame…
What is your point? (I honestly do not understand.) Blackbear's comment was that writing libraries in C allows those libraries to be deployed broadly in many compute environments. Jakob's reply (as I understood it) was…
What are your favorite open source Python packages written in Rust?
> there is no libnumpy or libpandas I think this is changing with the rise of Apache Arrow. See eg here, slide 38 and on: https://www.slideshare.net/wesm/pycon-colombia-2020-python-f...
> for library writers, how many people might use it can be a deciding factor on investing the time Yes. Especially given that a high percentage of library development these days is being done by industry, not just…
Makes sense. I don't use TF Eager, but I do use Jax, and Jax lets you arbitrarily compose JITed and non-JITed code, which made me think that might be a viable pattern. I guess I wondered if there might be something like…
If the next machine learning killer-app model requires autodiff'ed dynamic control flow, do you think Google/Facebook will build that capability into XLA/TorchScript? Seems like if NLP SOTA requires dynamic control…
Always happy to help. Especially the last day or so - I've been waiting on some long training loops so it's been a pleasant diversion. To be 100% honest with you, there's pretty much 0% chance of me adopting Julia in…
> since `fit` would've likely come from the same source No. As described in the great-great-great-great-great-great-great-great-great-parent comment, the problem I am describing is that of trying to combine models from…
Just to be clear, I like Julia, and think it has advantages over Python. I'm writing all this as someone who is cheering for Julia to break out of its HPC niche. Thanks.
@cbkeller I know in Julia I can just precede every function call and object instantiation with "modulename." and solve the namespace problem that way. What I want to do instead, what I do in python, is bind one…
Never made it that far. This was a feature I use all the time in Python ML development (both consuming open source packages via an OOP interface and also writing in-house model classes) that I consider essential for my…
I very frequently run into namespace collisions like that. I think they are quite common in large codebases. I am aware of the ability to do eg "import TensorFlow; model = TensorFlow.model; TensorFlow.fit(model,data)"…
I know nothing about Lisp, so at the risk of talking past eachother.... I never said macros were required. I said implementing this type of code without OOP required more boilerplate, and MLJ uses macros to reduce that…
I agree you can achieve same benefits with Macros. Indeed, I see that MLJ, Julia's attempt at a SciKit type project, makes extensive use of Macros. But I personally think macros are an antipattern. In large projects,…
I'll give you my two cents, recognizing that I very well might just be ignorant about Julia and multiple dispatch, and if so please continue to educate me. Consider if we want to run many different types of models.…
I second most of this. A lot of problems are fixable with time and money. Maybe the Series A will help! But some problems might be related to Julia's design choices. One thing I really missed in Julia is Object Oriented…
Congrats to the Julia team. I am a python developer who has dabbled with Julia but it never stuck for me. I think Julia was built by academics for other academics running innovative high performance computing tasks. It…
Did PyTorch increase or decrease demand for ML developers? Did Ruby on Rails increase or decrease demand for web developers? Did C increase or decrease demand for systems programmers? Automation and abstraction is a…
Learned a bunch from the article and from this thread. Thanks all.
I pip install foo, and foo depends on bar. I pip freeze > lock.txt. My lock file has foo v1 and bar v1, right? Later bar upgrades to v2. I then try to rebuild the container image from lock.txt. My pip freeze lock file…
Mirrors my experience. Walled garden “we handle everything for you” tools are good for simple use cases. One you need to go outside the garden, even a little bit, it becomes more trouble than it’s worth. For non-trivial…
Why is pip freeze not 100%?
Doesn’t pip freeze > requirements.txt capture exact versions of dependencies of dependencies?
I may be wrong. It happens a lot! But I think Arrow's vision encompasses compute, not just a data format and data interface. https://www.slideshare.net/wesm/pycon-colombia-2020-python-f... Slide 43: The "Arrow C++…
Fair. I agree Arrow is still more of a vision than anything else. > it didn't even provide a shape attribute I suspect this has to do with the project's focus. I think they aspire to be a back-end to DataFrame…
What is your point? (I honestly do not understand.) Blackbear's comment was that writing libraries in C allows those libraries to be deployed broadly in many compute environments. Jakob's reply (as I understood it) was…
What are your favorite open source Python packages written in Rust?
> there is no libnumpy or libpandas I think this is changing with the rise of Apache Arrow. See eg here, slide 38 and on: https://www.slideshare.net/wesm/pycon-colombia-2020-python-f...
> for library writers, how many people might use it can be a deciding factor on investing the time Yes. Especially given that a high percentage of library development these days is being done by industry, not just…
Makes sense. I don't use TF Eager, but I do use Jax, and Jax lets you arbitrarily compose JITed and non-JITed code, which made me think that might be a viable pattern. I guess I wondered if there might be something like…
If the next machine learning killer-app model requires autodiff'ed dynamic control flow, do you think Google/Facebook will build that capability into XLA/TorchScript? Seems like if NLP SOTA requires dynamic control…
Always happy to help. Especially the last day or so - I've been waiting on some long training loops so it's been a pleasant diversion. To be 100% honest with you, there's pretty much 0% chance of me adopting Julia in…
> since `fit` would've likely come from the same source No. As described in the great-great-great-great-great-great-great-great-great-parent comment, the problem I am describing is that of trying to combine models from…
Just to be clear, I like Julia, and think it has advantages over Python. I'm writing all this as someone who is cheering for Julia to break out of its HPC niche. Thanks.
@cbkeller I know in Julia I can just precede every function call and object instantiation with "modulename." and solve the namespace problem that way. What I want to do instead, what I do in python, is bind one…
Never made it that far. This was a feature I use all the time in Python ML development (both consuming open source packages via an OOP interface and also writing in-house model classes) that I consider essential for my…
I very frequently run into namespace collisions like that. I think they are quite common in large codebases. I am aware of the ability to do eg "import TensorFlow; model = TensorFlow.model; TensorFlow.fit(model,data)"…
I know nothing about Lisp, so at the risk of talking past eachother.... I never said macros were required. I said implementing this type of code without OOP required more boilerplate, and MLJ uses macros to reduce that…
I agree you can achieve same benefits with Macros. Indeed, I see that MLJ, Julia's attempt at a SciKit type project, makes extensive use of Macros. But I personally think macros are an antipattern. In large projects,…
I'll give you my two cents, recognizing that I very well might just be ignorant about Julia and multiple dispatch, and if so please continue to educate me. Consider if we want to run many different types of models.…
I second most of this. A lot of problems are fixable with time and money. Maybe the Series A will help! But some problems might be related to Julia's design choices. One thing I really missed in Julia is Object Oriented…
Congrats to the Julia team. I am a python developer who has dabbled with Julia but it never stuck for me. I think Julia was built by academics for other academics running innovative high performance computing tasks. It…
Did PyTorch increase or decrease demand for ML developers? Did Ruby on Rails increase or decrease demand for web developers? Did C increase or decrease demand for systems programmers? Automation and abstraction is a…
Learned a bunch from the article and from this thread. Thanks all.
I pip install foo, and foo depends on bar. I pip freeze > lock.txt. My lock file has foo v1 and bar v1, right? Later bar upgrades to v2. I then try to rebuild the container image from lock.txt. My pip freeze lock file…
Mirrors my experience. Walled garden “we handle everything for you” tools are good for simple use cases. One you need to go outside the garden, even a little bit, it becomes more trouble than it’s worth. For non-trivial…
Why is pip freeze not 100%?
Doesn’t pip freeze > requirements.txt capture exact versions of dependencies of dependencies?