Haha, this really nailed several painpoints of TF. Laughed and nodded at the "there are no experts users, instead there's experts of 2017 TF and experts of 2019 TF".
Tensorflow is a proof that some people in Google don't understand software engineering: They don't know and can't decide even what interface they want to expose/hide.
There is no reason why two things can’t be merged together for better UX and functionality, but two humans have egos, want to lead projects, and be in charge, so things remain separate.
Software Engineering isn't just about put stuff out there, it also involves careful planning and roadmap around abstractions.
As high profile as a project as TF, if they don't have a clear way to tell having many parallel implementations for essentially the same functionality is confusing and hurt developer experience and ultimately harms the project itself, then that only explains the disaster , as a result of mismanagement, but doesn't undo the damage.
Oh god, those last three comments are so cringey. They both want to merge their projects together, but can't agree on which namespace to use? Somebody needs to go Solomon on these kids and announce that a complete rewrite from scratch will take place unless they can agree on a namespace, then hand over control of the merge to whichever team cedes the point first.
I totally sympathize with this. As the creator of PyTorch Lightning I focused on the user experience FIRST. Too many times, it seems, engineers forget that NOT everyone using these frameworks are engineers. It really is a fine balance between being opinionated but no more. But as mentioned here, even though Lightning has seen dramatic adoption, there were a few times when the result was judged on the simplicity of it without considering the significant amount of time that went into making the API super simple and infinitely flexible.
At least in the PyTorch world, the team's done a phenomenal job at allowing an ecosystem of related projects to contribute to one another and co-exist!
Conda works really well for this too, all in one. Aside from that, I've set up a requirements.txt that I just use in every project, copy it to my working directory, build, stay in virtual environment.
If PyTorch had something like TensorFlow Lite, TF would probably be dead already. I certainly wouldn't need it. At the moment, though, TFLite is the most sane option for inference on phones and other ARM devices.
PyTorch Mobile is a start and is available for iOS and Android. Given folks like PFN and Microsoft are (or will be heavy contributors) i would expect support for more devices to broaden. Have you tried it out yet? No need for a separate set of op semantics or framework.. :) https://pytorch.org/mobile/home/
Anything that can't use mobile GPU (or DSP/TPU for quantized inference) is pretty useless IMO, because it's just not energy efficient enough to be practical in a battery powered device, even if it's fast enough.
Once pytorch is updated to use XNNPACK (being worked on right now) I think it should be fine to use. That plus QNNPACK makes inference quite low on power usage in my (admittedly limited, just integrated XNNPACK) experience.
As a rule, CPU burns at least 5x the energy per FLOP. So no, CPU is not a viable option on mobile if you need to do inference constantly. For "every now and then" cases, sure.
JAX looks really interesting but it seems that it’s not as straightforward to just JIT things. Generally I think you need to use their flow control mechanisms rather than standard Python ones. There are of course technical reasons for doing so, but this makes me wonder if languages like Julia with packages like Zygote.jl would be nicer for some situations in the long run.
With JAX you _can_ just use python control flow with gradients, so long as you don't try to jit-compile the control flow. This ends up being just like PyTorch / eager execution. However, you have the option of compiling the largest possible blocks of code that are convenient.
When writing code in JAX, I often start by write code eagerly and then try to jit increasingly large pieces to get more performance.
Sure, but OP's point is that TensorFlow's built-in libraries suck and you're forced to use them. They are confusing, change all the time, etc etc.
Since JAX's API is lower-level, you can choose a neural network API that doesn't have the problems of TF's API. Or worst case, you make it yourself; you can see that the higher-level APIs provided by JAX are all very simple.
> "Actually, you know what it reminds me of, in some ways? With the profusion of backwards-incompatible wheel-reinventing features, and the hard-won platform-specific knowledge you just know will be out of date in two years?"
I created this for educational purposes, but it is quite robust, simple, well-tested, and well-documented. It also includes neural network style operations like N-dimensional convolutions and pooling.
Hello. I'm the person that was linked to in that GitHub issue!
I sympathize with the post's frustration. The TF tutorials on the official website are well-written. But they mostly cover basic features, and as a recent Reddit thread described (https://old.reddit.com/r/MachineLearning/comments/e4pxqp/d_i...), the support ecosystem is lacking as StackOverflow and blog posts are out-of-date due to all the software churning. I'm not a TF engineer, but as someone with experience designing libraries on top of TF, even I find myself sifting through Stack Overflow/blog post code to find the new best practices..
Regarding Bayesian layers, it's actually a NeurIPS paper this year (https://papers.nips.cc/paper/9607-bayesian-layers-a-module-f...). I worked on an early prototype in TensorFlow Probability but ended up abandoning the design as I found it inflexible in practice. The solution is the NeurIPS paper, and it's experimental: there are no promises of stability (in fact, we even moved the code from Tensor2Tensor to another repository (https://github.com/google/edward2/), of which has yet to have an official package release!).
Software for uncertainty models is more on the research fringe, and this should be made clearer in official TensorFlow solutions building on these designs.
47 comments
[ 2.8 ms ] story [ 109 ms ] threadPyTorch?
It is a disaster.
This type of exchange is pretty much what happens all the time: https://github.com/williamFalcon/pytorch-lightning/issues/35
There is no reason why two things can’t be merged together for better UX and functionality, but two humans have egos, want to lead projects, and be in charge, so things remain separate.
Software Engineering isn't just about put stuff out there, it also involves careful planning and roadmap around abstractions.
As high profile as a project as TF, if they don't have a clear way to tell having many parallel implementations for essentially the same functionality is confusing and hurt developer experience and ultimately harms the project itself, then that only explains the disaster , as a result of mismanagement, but doesn't undo the damage.
At least in the PyTorch world, the team's done a phenomenal job at allowing an ecosystem of related projects to contribute to one another and co-exist!
The goal of TensorFlow is to put itself in as many ML deployments as possible, paving the way for auto ml, which is where the real money lies.
That what I like with Pytorch, they offer prebuilt packages for different versions of CUDA.
https://github.com/tensorflow/tensorflow/issues/25329
Also, everything you think or do is deprecated. This comment has been deprecated.
Now that the invariants are known, better abstractions have been designed, such as Keras.
First TensorFlow version was released the 9 November 2015.
Both had to follow the evolution of DL, but Keras was developer-friendly from the beginning.
The API is numpy plus like four functions. That's the beginning and end. It does JIT compilation under the hood, so can run quite fast.
When writing code in JAX, I often start by write code eagerly and then try to jit increasingly large pieces to get more performance.
You might also find the experimental control flow interesting since it tries to be more pythonic while also being possible to jit-compile. (https://github.com/google/jax/blob/master/jax/experimental/l...)
Since JAX's API is lower-level, you can choose a neural network API that doesn't have the problems of TF's API. Or worst case, you make it yourself; you can see that the higher-level APIs provided by JAX are all very simple.
Every JavaScript web UI framework ever?
I created this for educational purposes, but it is quite robust, simple, well-tested, and well-documented. It also includes neural network style operations like N-dimensional convolutions and pooling.
Plus backprop through all variations of einsum :)
https://github.com/tensorflow/tensor2tensor
> It is now in maintenance mode — we keep it running and welcome bug-fixes, but encourage users to use the successor library Trax.
I sympathize with the post's frustration. The TF tutorials on the official website are well-written. But they mostly cover basic features, and as a recent Reddit thread described (https://old.reddit.com/r/MachineLearning/comments/e4pxqp/d_i...), the support ecosystem is lacking as StackOverflow and blog posts are out-of-date due to all the software churning. I'm not a TF engineer, but as someone with experience designing libraries on top of TF, even I find myself sifting through Stack Overflow/blog post code to find the new best practices..
Regarding Bayesian layers, it's actually a NeurIPS paper this year (https://papers.nips.cc/paper/9607-bayesian-layers-a-module-f...). I worked on an early prototype in TensorFlow Probability but ended up abandoning the design as I found it inflexible in practice. The solution is the NeurIPS paper, and it's experimental: there are no promises of stability (in fact, we even moved the code from Tensor2Tensor to another repository (https://github.com/google/edward2/), of which has yet to have an official package release!).
Software for uncertainty models is more on the research fringe, and this should be made clearer in official TensorFlow solutions building on these designs.