PyTorch has become dominant in research because of its API (both its stability + having eager mode).
TF has become dominant in industry because A. it came out several years before PyTorch and industry is slow to move, B. It supported a lot of production use cases (mobile, serving, removing Python overhead) that PyTorch didn't for a long time.
PyTorch wraps THNN, not Torch. Moreover, even if this was true, it wouldn't matter at all. Practically 0% of the overhead is related to Python in the first place, all of the time is dominated by the underlying C implementation.
If only articles were as concise as your summary is I would enjoy reading them but as long as they are many pages long I have no time to read beyond the titles, abstracts, conclusions and comments.
BTW I've also read (here on HN) PyTorch learns much faster than TensorFlow does.
My former employer pulled in a few AWS data scientists to consult with us on a few projects and based on my interactions it seemed like they were under some directive to strongly discourage anything that wasn't a built-in AWS plug-and-play sagemaker algorithm. It was not a positive experience because of course most of them are half baked.
MXNet is actually pretty good. It got to the "mixing eager and graph mode" semantics before either PyTorch or TensorFlow did. On top of that, it's also blazing fast (usually the fastest of the frameworks).
Admittedly, I've never used MXNet so it might have more issues that I'm not aware of. Judging from the benchmarks I've seen, however, MXNet got a lot of things right.
Unluckily, I just don't think it added enough on top of PyTorch or TensorFlow for people to consider switching. People switched from TensorFlow to PyTorch because eager mode was just so much easier to use.
MxNet is fantastic though. It's usually faster than tensorflow, has a pytorch like "eager" API that doesn't suck, and can still use symbolic graphs. Amazing documentation too (for the Gluon API).
Anyone has any opinions on TF2.0? They've released it recently, and it seems like it should be much closer to PyTorch now, but I don't know enough to evaluate it properly.
TF2.0 (and in particular their recommended tf.keras) is simply a clone of the Pytorch API in most respects. There is no reason to use it vs just using Pytorch, especially as Pytorch now support easy model exporting for running in production.
tf.keras contains a deprecated API that was the original keras as well as a new API that basically is a clone of Pytorch. You can tell the difference because only the new API works nicely with eager mode.
My very biased opinion: you start with PyTorch because it's easy to develop and debug, and there's no point in having the fastest tools for a model that you can't train properly.
Once your model is running, and if/when you start hitting performance bottlenecks, then you consider migrating your model to TensorFlow.
TF eager mode has been a stable/supported thing for 10 days, since the release of 2.0. Before then it was available as opt-in behavior that once enabled meant all bets were off for if things would work or explode. So I think it's too early to answer your question. Maybe 2.0 bridges the gap to PyTorch in development speed. But maybe the momentum has already shifted to PyTorch.
I think tensorflow dominates industry purely because of its capability of exporting the model into a coreml Android model or easy of moving it to production in a GCP environment or in whatever form. Pytorch might have to build a good production pipeline around it to catch up in this game.
With fastai module that's built on Pytorch learning and developing Deep Learning solutions have become a lot easier. So there's a real game on now
And simply TensorFlow was there earlier, so people implemented stuff in it. I think there's more inertia in industry, whereas researchers may more easily switch frameworks between two papers.
This sounds like in theory and makes you curse in practice. Trained TF model -> deployment using TF Lite is the most robust pipeline currently as far as I know.
There are lots of improvements going into pytorch for mobile at the moment, but for the moment I'll wait and see how it turns out - I didn't have much fun with caffe2 when "train in pytorch and deploy with caffe2" was the storyline FB pushed (e.g. problems with binary size and slow depthwise convolutions) so not too eager to migrate back at the moment.
I've done this with OpenVino and it works well enough. There are some gotchas with Onnx, some layers break or have weird bugs (upsample in particular used to be a problem).
It's difficult because the graph is defined running Python code, which can basically do anything. As I understand it you have to actually run the model and sort of introspect it to work out what the ONNX graph should be.
Whereas the Tensorflow API actually creates a static graph that can be easily converted to ONNX.
Edit: They talk about this problem in the article:
> Although straightforward, tracing has its downsides. For example, it can’t capture control flow that didn’t execute. For example, it can’t capture the false block of a conditional if it executed the true block.
> Script mode takes a function/class, reinterprets the Python code and directly outputs the TorchScript IR. This allows it to support arbitrary code, however it essentially needs to reinterpret Python.
researchers are like Francois Chollet & many more even jeff dean and it is the greatness of the community that it's pretty open to optimization "Why do researchers love PyTorch?
Simplicity. It’s similar to numpy, very pythonic, and integrates easily with the rest of the Python ecosystem. For example, you can simply throw in a pdb breakpoint anywhere into your PyTorch model and it’ll work. In TensorFlow, debugging the model requires an active session and ends up being much trickier.
Great API. Most researchers prefer PyTorch’s API to TensorFlow’s API. This is partially because PyTorch is better designed and partially because TensorFlow has handicapped itself by switching APIs so many times (e.g. ‘layers’ -> ‘slim’ -> ‘estimators’ -> ‘tf.keras’).
Performance. Despite the fact that PyTorch’s dynamic graphs give strictly less opportunity for optimization, there have been many anecdotal reports that PyTorch is as fast if not faster than TensorFlow. It's not clear if this is really true, but at the very least, TensorFlow hasn't gained a decisive advantage in this area."
common people do want to appreciate & adopt the things that seems fit to the knowledge sphere at present from the researchers. tensorflow approaches are better and respected each and everyone of the community as well in exchange enlightened us new ways of understanding of ml solutions. it have turned into a family “If you want to go fast, go alone. If you want to go far, go together.” and given the assets alphabet have a common man can turn into researchers! e.g. >> https://learn.grasshopper.app take this for example "Learn to code anywhere.
Grasshopper is available on iOS, Android, and all web browsers. Your progress syncs seamlessly between devices." << this is the status quo ! it's a gift of a lifetime for generations !
In my experience in computer vision research it doesn’t matter what you use, yes, immediate mode is slightly more convenient, but research time is influenced much more by your computing power, dataset acqusition/labeling/relabeling power and, last but not the least, by code quality and easy and efficient collaboration - thats why you need tools like DVC/Argoproj. We did get amazing results using Caffe v1 back in the day.
I agree. I haven’t encountered a strong preference in academic computer vision or machine learning. Keras and PyTorch, dominate, of course, but I wouldn’t be shocked if everyone started using something new in the future.
Going from doing ML research to data ingestion and analysis to web frameworks to API design, blogging and static site generation is very powerful. The trend seems to be that python will dominate all of these .
It's only a matter of time until PyTorch will also dominate industry.
It's always like this.
Think how Ubuntu took over the server market because amateurs were preferring it instead of Redhat/CentOS. And when they became professionals or were in a position to decide, they also put Ubuntu on the server because this is what they knew best.
I'm not sure that's a great example, given that AWS mostly runs on RHEL-based OSs and Debian is still preferred for Docker. Ubuntu did not "take over the server market".
RHEL is popular for solutions like running a datacenter mostly because it has a nice enterprise support story. It's what the E in that acronym is for, after all. Ubuntu, meanwhile, is quite popular among us mere mortals who have to fix our own boxen.
Debian is popular for Docker images exactly because many of the people trying Docker were already familiar with Ubuntu. Those users quickly ended up wanting smaller images, making Debian an obvious thing to try out since Ubuntu is basically Debian with bells on.
Ubuntu fought a sea of distros and came out as what's very nearly an industry standard, if not an official one. The 90s were a fricking mess by comparison. Slackware on floppies.
(And now I need "Slackware on floppies" dubbed over the "Jesus wept" scene from Hellraiser.)
> Ubuntu fought a sea of distros and came out as what's very nearly an industry standard, if not an official one.
I think you may be living in a bubble. I've been running devops for various shops for half a decade and I've only once used Ubuntu, because it was already being used by an acquisition.
I won't deny that Ubuntu is popular. It's certainly got the lions share of the desktop market. But there is no such consensus in the server market.
Can you provide any evidence to support that claim?
Here's a report that suggests the exact opposite:
"Don't let the revenue numbers lead you to thinking Red Hat Enterprise Linux (RHEL) is more popular than Ubuntu. By The Cloud Market's Jan. 8, 2019 count of Amazon Web Services (AWS) instances, Ubuntu is used in 314,492 instances, more than any other operating system, while RHEL is used in 22,072 instances."
Disclosure: I work for Canonical, but as an engineer; I'm not in marketing or anything and that's not my job. But I do get the impression that Ubuntu is way ahead in general use in the cloud, and is also the generally used base for Docker images (I don't immediately see how to get that statistic out of Docker Hub). I didn't think this statement was controversial.
A more apt comparison would be Ubuntu vs CentOS (and amazon linux and all the other repackaged RHELs). Ubuntu definitely has more market share, but the OP implied that it was "just the way things are done now" which is not true.
The last data seems to be about 3 years old, and Ubuntu was about 1.5-2x the CentOS/amazon linux share. I suspect that's changing with the release of amazon linux 2, but there's no data to back that up.
Amazon itself primarily uses a RHEL based distro, which is what I meant originally.
We use Pytorch extensively in our startup. We tackle a lot of new research problems as consultants/partners to help develop products or devise new algorithms/models to solve tasks for our customers. We have never regretted our choice to pick Pytorch. I found the article pretty spot on when comparing Tensorflow and Pytorch. The things that have appealed to me about Pytorch are:
1. Extremely easy to debug and work with. Being able to debug effortlessly in PyCharm makes life very easy.
2. The API is quite clean and nice and fits in really well with Python and nothing feels hacky. I've developed my own Keras-like framework for experimentation, training and evaluating models quickly and easily and the entire experience has been really enjoyable.
3. The nicest thing though is that as the article points out, a huge percentage of researchers have moved to Pytorch and this allows us to more easily look at other researcher's code and experiment with things easily and incorporate ideas and cutting-edge research into our own work. Even for things that are released in TensorFlow, if it is an important publication that gains attention and traction in the community, you will likely have implementations in Pytorch pop up soon enough.
I do think that TensorFlow still has an edge on the deployment at scale/mobile side of things as pointed out by the article. But Pytorch is a lot younger and they are making a lot of progress with every release in that space.
Last year I was tasked with looking into a NAS (Neural Architectures Search) paper and analyzing the algorithm. The paper came with a TensorFLow implementation. Trying to read that TF code was quite difficult. I searched around and found a PyTorch implementation - much easier to read and understand, and it ran about 50% faster as well (the latter was a bit surprising). I tend to think that TensorFlow lends itself to the creation of code that's difficult to reason about. That may be different now with the various flavors of TF (like TF Eager).
I'll add that it was much easier to install PyTorch with GPU support than it was to install TensorFlow with GPU support - at least that's how it was around November of last year. The PyTorch install was painless, whereas we ended up having to build TF from source to work with out setup. Could be different now as I haven't looked at TF since then.
> That may be different now with the various flavors of TF (like TF Eager)
Unfortunately, if anything I think it's the opposite. The constant creation and deprecation of TF flavors (tf-eager, tf-slim, tf-learn, keras, tf-estimator, tf.contrib [RIP]) has made reading tensorflow code online somewhat disastrous. Everybody, including the TF team, is using a different API and it's difficult to keep all of them straight. It seems that you're doomed to end up using some combination of many of the above in a way that makes sense to you and your team, adding another confusing model to the pile.
Agree overall, but tf.eager doesn't have much to do with the rest of the list.
tf.contrib is just a module where user-contributed code was stored, which included both low-level constructs and higher level APIs.
tf.estimator is an abstraction that is mostly used for productionizing models.
tf.slim/tf.learn were indeed redundant with keras (a library developped externally), but were necessary steps before keras became part of tensorflow.
There will be bad model code in both PyTorch and TensorFlow. The difference is that, bad PyTorch code reads like bad Python code, and I've accumulated a lot of experience reasoning through bad Python code. Bad TensorFlow code can come from any one of of the history of paradigms that TF has gone through, and I don't even know if it's bad or just some funky new TF functionality I'm unfamiliar with.
What made me fall in love with PyTorch was also that the "neural network training process" is defined almost as it is in theory, in code in PyTorch
- loop through epochs
- loop through each batch
- run a forward pass for the batch ( model(batch) )
- calculate the loss for the batch ( criteria(y, yprim)
- compute the gradients/backprop ( loss.backward() )
- update the weights (optimizer.step())
This really enforced everything I learned and I think breaks down the problem. All this of course in addition to everything else already mentioned, and super convenient module/network building and definition.
I remember using early Torch (in Lua! As someone who knew only Matlab!) in 2015-ish; and then using Keras (which is supposed to be an abstraction layer over NN frameworks) and finding it much more verbose and complicated to use without recurring to code snippets.
Perhaps it’s the nature of the game that changed with many new kinds of architectures and so on. But maybe Keras is already overengineered for someone who just wants to make thumbnail sized GAN stuff at home.
Anecdotally, I've dumped TensorFlow in favor of PyTorch for almost all new work I'm doing at my organization (industry focused). Biggest gripes with TensorFlow are overly complex APIs, instability from release to release, constantly broken code in Google's repos, and poor documentation. Maybe TF 2.0 will be better, but for me, the PyTorch ship has already sailed, and I am sailing on it.
We have been using mostly Dlib[0]. There was the need to develop solutions that can be statically compiled and produce dependency-free dlls and dlib delivered remarkbly on that aspect.
I haven't had success doing so using frameworks such as Torch and TF, even if their toolkit is better to develop new solutions.
Also we get to write code in C++, which can be a big positive when developing machine learning SDKs. I personally still do most of the prototyping in Python though.
I'll be checking the link on the post that mentions that pytorch allows models to be converted to c++, looks promising actually.
Researcher's codes are historically not very clean and re-usable. It may work fine if you want to hack together something to get data for a paper, but if you want to run a real production service, and don't want to drown in tech debt in a year; and that often means more code, as you imply above. I don't think it's unnecessarily verbose, it's just that it's more structured and scalable.
The discussion was about pytorch vs tensorflow but your argument seems to be about productionizing research code (which is framework agnostic).
Yes there is a lot of code to be written/optimizations to be done to make things production worthy. However I know a lot of tensorflow research projects that handle data batching in such a terrible way that would take weeks to re-write for production.
As for verboseness of pytorch vs tensorflow, I think either could get more verbose under different circumstances. However for simpler tasks, I think tensorflow is more verbose in general (not accounting for the new release which seems to mimic pytorch/keras a little more). For larger production tasks, its a toss-up depending on whether you need to add new components.
If by some miracle a comp sci researcher does publish something relevant to industry most people would rather wait for some other poor soul to deconstruct their work and make usable code or a library out of it.
Well, the worst engineers don't push any code so maybe it looks like they don't think writing more code is productive.
However its been my experience that the average engineer believes this. They often aim to push some amount of code (meaningful or not) every few days or so.
The best engineers I know write a lot of code overall but are more interested in ensuring they build the right thing and are ok with not pushing code for a while if they need more time.
PyTorch is simpler, easer to use, consumes less memory and allows for dynamic dynamic computational graphs (dynamic operations during the forward pass).
Unfortunately, you're way off the mark here.
Its actually more of the opposite.
Perhaps we have a different set of experiences, but your assumption here is not the cause of our difference in opinions.
I don't even believe the thesis about one dominating the other in a specific domain. I dont think top mentions in conferences in a good measure of usage.
Ok, admittedly, there are a couple reasons. The fact that most papers don't mention the framework they use is a big one. So if users of one framework disproportionately mentioned that framework in their paper, it would be overrepresented.
Basically, some conferences have encouraged researchers to submit code. Instead of checking the papers, I checked their code instead. The results are pretty much the same. So I think that mentions in top conferences probably correlates well with uses in code.
I think its matter of time. New things gets adopted first in research. I think pytorch will take over tensorflow. I was also a tensorflow user and when i switched to pytorch i never looked back. I was also participating in a kaggle competition and top 20 models are all implemented in pytorch.
We are considering to move to PyTorch, we really dislike how the Tensorflow 1.x -> 2.0 transition is handled. For years a lot of stuff has been added to tf.contrib, some things were only in tf.contrib and now that it's dropped in TF a lot of project (including ours) have to do quite large rewrites. Since the last few 1.x iterations, Tensorflow has been complaining that the older RNN layers are deprecated and that we have to move to Keras RNN layers, which they claim to be equivalent. However, when we tried a couple of months back, it made RNN-based training 45% slower. It is all fixable, but it takes time and a lot of testing of all the model variants to see if there are no regressions. It feels quite a bit worse than Python 2 -> 3.
I am a bit saddened by all of this, because I really liked how easy it is to define a graph in Tensorflow in Python, serialize it, and then use its minimalistic C API to use the graph in Go, Rust, or wherever you need it.
How is your experience with PyTorch and backwards API compatibility (I know that they only reached 1.0 fairly recently)?
It's been pretty good with PyTorch. The API has been fairly stable and I've adopted code developed from 0.4.0 to 1.0.0+ with barely the need for any tweaks. Granted, it's a younger project so for now things are stable but maybe 3 years from now they may have some giant API refresh. But I find their API quite nice for the most part so I don't see them needing to switch everything up periodically.
There has basically only ever been one major API shift, which was the shift away from Variables. And, granted, that happened from 0.3->0.4, about a year after the initial release.
Other than that, I've had next to no issues, and the API has only gotten better over time, with more convenient ways to do things.
PyTorch has a much smaller footprint, and is happy to delegate code to separate libraries (e.g. torchvision), so you run into "all-or-nothing" dilemmas less frequently.
For the majority of production use cases (which tend to get all the AI/ML hype), TensorFlow/Keras is more than powerful enough and accessible enough. If you need to dive down to custom layers/optimizers, PyTorch has value there, but for people looking to get their start in AI/ML, the meme that "TensorFlow sucks" is highly misleading.
One thing the AI/data scene gets the best of is data on their own industry. Reminds me of how Ruby used to have the best designed websites for their various tools.
200 comments
[ 4.7 ms ] story [ 226 ms ] threadindustry: tensorRT.
As a summary, though:
PyTorch has become dominant in research because of its API (both its stability + having eager mode).
TF has become dominant in industry because A. it came out several years before PyTorch and industry is slow to move, B. It supported a lot of production use cases (mobile, serving, removing Python overhead) that PyTorch didn't for a long time.
BTW I've also read (here on HN) PyTorch learns much faster than TensorFlow does.
The Keras interface for tensorflow makes it easy & fast to make "good enough" models. That is often a driving factor
Admittedly, I've never used MXNet so it might have more issues that I'm not aware of. Judging from the benchmarks I've seen, however, MXNet got a lot of things right.
Unluckily, I just don't think it added enough on top of PyTorch or TensorFlow for people to consider switching. People switched from TensorFlow to PyTorch because eager mode was just so much easier to use.
https://www.youtube.com/watch?v=EqWsPO8DVXk
[0] https://www.fast.ai/about/ [1] https://www.youtube.com/watch?v=J6XcP4JOHmk&t=4152s
Once your model is running, and if/when you start hitting performance bottlenecks, then you consider migrating your model to TensorFlow.
With fastai module that's built on Pytorch learning and developing Deep Learning solutions have become a lot easier. So there's a real game on now
There are lots of improvements going into pytorch for mobile at the moment, but for the moment I'll wait and see how it turns out - I didn't have much fun with caffe2 when "train in pytorch and deploy with caffe2" was the storyline FB pushed (e.g. problems with binary size and slow depthwise convolutions) so not too eager to migrate back at the moment.
Whereas the Tensorflow API actually creates a static graph that can be easily converted to ONNX.
Edit: They talk about this problem in the article:
> Although straightforward, tracing has its downsides. For example, it can’t capture control flow that didn’t execute. For example, it can’t capture the false block of a conditional if it executed the true block.
> Script mode takes a function/class, reinterprets the Python code and directly outputs the TorchScript IR. This allows it to support arbitrary code, however it essentially needs to reinterpret Python.
common people do want to appreciate & adopt the things that seems fit to the knowledge sphere at present from the researchers. tensorflow approaches are better and respected each and everyone of the community as well in exchange enlightened us new ways of understanding of ml solutions. it have turned into a family “If you want to go fast, go alone. If you want to go far, go together.” and given the assets alphabet have a common man can turn into researchers! e.g. >> https://learn.grasshopper.app take this for example "Learn to code anywhere. Grasshopper is available on iOS, Android, and all web browsers. Your progress syncs seamlessly between devices." << this is the status quo ! it's a gift of a lifetime for generations !
It's always like this.
Think how Ubuntu took over the server market because amateurs were preferring it instead of Redhat/CentOS. And when they became professionals or were in a position to decide, they also put Ubuntu on the server because this is what they knew best.
Debian is popular for Docker images exactly because many of the people trying Docker were already familiar with Ubuntu. Those users quickly ended up wanting smaller images, making Debian an obvious thing to try out since Ubuntu is basically Debian with bells on.
Ubuntu fought a sea of distros and came out as what's very nearly an industry standard, if not an official one. The 90s were a fricking mess by comparison. Slackware on floppies.
(And now I need "Slackware on floppies" dubbed over the "Jesus wept" scene from Hellraiser.)
I think you may be living in a bubble. I've been running devops for various shops for half a decade and I've only once used Ubuntu, because it was already being used by an acquisition.
I won't deny that Ubuntu is popular. It's certainly got the lions share of the desktop market. But there is no such consensus in the server market.
Here's some date I could dig up by a couple of minutes googling:
https://w3techs.com/technologies/details/os-linux/all/all https://w3techs.com/technologies/history_details/os-linux
Other sibling comments link to more.
make menuconfig
Here's a report that suggests the exact opposite:
"Don't let the revenue numbers lead you to thinking Red Hat Enterprise Linux (RHEL) is more popular than Ubuntu. By The Cloud Market's Jan. 8, 2019 count of Amazon Web Services (AWS) instances, Ubuntu is used in 314,492 instances, more than any other operating system, while RHEL is used in 22,072 instances."
https://www.zdnet.com/article/inside-ubuntus-financials/
Disclosure: I work for Canonical, but as an engineer; I'm not in marketing or anything and that's not my job. But I do get the impression that Ubuntu is way ahead in general use in the cloud, and is also the generally used base for Docker images (I don't immediately see how to get that statistic out of Docker Hub). I didn't think this statement was controversial.
The last data seems to be about 3 years old, and Ubuntu was about 1.5-2x the CentOS/amazon linux share. I suspect that's changing with the release of amazon linux 2, but there's no data to back that up.
Amazon itself primarily uses a RHEL based distro, which is what I meant originally.
https://www.phoronix.com/scan.php?page=news_item&px=Canonica...
https://www.flickr.com/photos/42204979@N00/5266983786/sizes/... https://thecloudmarket.com/stats
1. Extremely easy to debug and work with. Being able to debug effortlessly in PyCharm makes life very easy.
2. The API is quite clean and nice and fits in really well with Python and nothing feels hacky. I've developed my own Keras-like framework for experimentation, training and evaluating models quickly and easily and the entire experience has been really enjoyable.
3. The nicest thing though is that as the article points out, a huge percentage of researchers have moved to Pytorch and this allows us to more easily look at other researcher's code and experiment with things easily and incorporate ideas and cutting-edge research into our own work. Even for things that are released in TensorFlow, if it is an important publication that gains attention and traction in the community, you will likely have implementations in Pytorch pop up soon enough.
I do think that TensorFlow still has an edge on the deployment at scale/mobile side of things as pointed out by the article. But Pytorch is a lot younger and they are making a lot of progress with every release in that space.
I'll add that it was much easier to install PyTorch with GPU support than it was to install TensorFlow with GPU support - at least that's how it was around November of last year. The PyTorch install was painless, whereas we ended up having to build TF from source to work with out setup. Could be different now as I haven't looked at TF since then.
Unfortunately, if anything I think it's the opposite. The constant creation and deprecation of TF flavors (tf-eager, tf-slim, tf-learn, keras, tf-estimator, tf.contrib [RIP]) has made reading tensorflow code online somewhat disastrous. Everybody, including the TF team, is using a different API and it's difficult to keep all of them straight. It seems that you're doomed to end up using some combination of many of the above in a way that makes sense to you and your team, adding another confusing model to the pile.
tf.contrib is just a module where user-contributed code was stored, which included both low-level constructs and higher level APIs. tf.estimator is an abstraction that is mostly used for productionizing models. tf.slim/tf.learn were indeed redundant with keras (a library developped externally), but were necessary steps before keras became part of tensorflow.
There will be bad model code in both PyTorch and TensorFlow. The difference is that, bad PyTorch code reads like bad Python code, and I've accumulated a lot of experience reasoning through bad Python code. Bad TensorFlow code can come from any one of of the history of paradigms that TF has gone through, and I don't even know if it's bad or just some funky new TF functionality I'm unfamiliar with.
Yes, Javascript features and frameworks have evolved quite fas... oh wait
- loop through epochs
- loop through each batch
- run a forward pass for the batch ( model(batch) )
- calculate the loss for the batch ( criteria(y, yprim)
- compute the gradients/backprop ( loss.backward() )
- update the weights (optimizer.step())
This really enforced everything I learned and I think breaks down the problem. All this of course in addition to everything else already mentioned, and super convenient module/network building and definition.
Code: https://github.com/Chillee/pytorch-vs-tensorflow
Ablation of claims: https://thegradient.pub/p/cef6dd26-f952-4265-a2bc-f8bfb9eb1e...
JS interactive charts: https://chillee.github.io/pytorch-vs-tensorflow/
Perhaps it’s the nature of the game that changed with many new kinds of architectures and so on. But maybe Keras is already overengineered for someone who just wants to make thumbnail sized GAN stuff at home.
pytorch bug tracker: https://github.com/pytorch/pytorch/issues/755
I haven't had success doing so using frameworks such as Torch and TF, even if their toolkit is better to develop new solutions.
Also we get to write code in C++, which can be a big positive when developing machine learning SDKs. I personally still do most of the prototyping in Python though.
I'll be checking the link on the post that mentions that pytorch allows models to be converted to c++, looks promising actually.
[0] http://dlib.net/
- More code to check-in (Looks more productive)
- More infrastructure, e.g. checkpoints, exporters etc. (Looks like they're doing more work)
- Fancy visualizations (Allows them to look impressive while presenting loss plots)
- Easier to reuse things others have implemented and still get credit for it (TF model zoo, research repo etc.)
Why researchers like pytorch:
- Way easier to hack together their novel idea
- Looks scrappier (which somehow makes the individual look like a better researcher instead of an ordinary programmer)
- Lots of other researchers release code in pytorch so if you're working off of their idea, you use pytorch to avoid re-producing their results.
Open to debate on these ideas, let me know if you have a counterpoint or any other reasons to add
Yes there is a lot of code to be written/optimizations to be done to make things production worthy. However I know a lot of tensorflow research projects that handle data batching in such a terrible way that would take weeks to re-write for production.
As for verboseness of pytorch vs tensorflow, I think either could get more verbose under different circumstances. However for simpler tasks, I think tensorflow is more verbose in general (not accounting for the new release which seems to mimic pytorch/keras a little more). For larger production tasks, its a toss-up depending on whether you need to add new components.
Why I use tensorflow:
- keras
- I used tensorflow yesterday
Not even bad engineers try to pretend like this is true.
However its been my experience that the average engineer believes this. They often aim to push some amount of code (meaningful or not) every few days or so.
The best engineers I know write a lot of code overall but are more interested in ensuring they build the right thing and are ok with not pushing code for a while if they need more time.
With those bullet points, looks like you didn't talk to actual engineers, but rather middle-layer management people.
Ok, admittedly, there are a couple reasons. The fact that most papers don't mention the framework they use is a big one. So if users of one framework disproportionately mentioned that framework in their paper, it would be overrepresented.
I did cover this concern though, in the Appendix. Check out the "Biased Sample" section.(https://thegradient.pub/p/cef6dd26-f952-4265-a2bc-f8bfb9eb1e...)
Basically, some conferences have encouraged researchers to submit code. Instead of checking the papers, I checked their code instead. The results are pretty much the same. So I think that mentions in top conferences probably correlates well with uses in code.
I am a bit saddened by all of this, because I really liked how easy it is to define a graph in Tensorflow in Python, serialize it, and then use its minimalistic C API to use the graph in Go, Rust, or wherever you need it.
How is your experience with PyTorch and backwards API compatibility (I know that they only reached 1.0 fairly recently)?
For it’s successor we chose Pytorch instead of TF 2 and have been very happy with this decision
Other than that, I've had next to no issues, and the API has only gotten better over time, with more convenient ways to do things.
PyTorch has a much smaller footprint, and is happy to delegate code to separate libraries (e.g. torchvision), so you run into "all-or-nothing" dilemmas less frequently.