I have to confess to some confusion about how all these Tensorflow for Swift changes work. Is this being built into a tensorflow-ey fork of Swift? Is it on the roadmap for being incorporated into the core language? How does one tell from these Github manifestos?
Swift for Tensorflow adds language-level support for Tensorflow to Swift. It does start in the fork, but the ultimate goal is for all changes to be merged upstream. Some aspects, like dynamicLookup[0], which allows dot syntax for methods resolved at runtime, are already in the language. Others, like dynamicCallable[1] are accepted proposals.
> It does start in the fork, but the ultimate goal is for all changes to be merged upstream. Some aspects, like dynamicLookup[0], which allows dot syntax for methods resolved at runtime, are already in the language.
Personally, I feel like it would be unwise to merge in all of TensorFlow's requirements into Swift. Many features, such as the ones you mentioned, are actually useful to outside of the context of machine learning, so it does actually make sense to make them generally available. But compiler support for something as specific as @differentiable seems like an anti-goal for Swift.
Ok, but to use terms from that article, Swift is still very much "Software 1.0". It's a general-purpose programming language meant to be written by people. Sure, it's good that people are trying to extend it to fit their specific needs, but I don't think that Swift (or any commonly used language today) is anywhere near being "good" for these types of tasks.
> Neural networks [...] represent the beginning of a fundamental shift in how we write software.
That's an...interesting statement, considering that neural networks are nearly as old as software itself. The original Pitts-McCulloch paper was published in 1943. Active use of them has certainly been ongoing longer than that blog author has been alive.
And yet it's true, since they haven't been used in the way they are now. There was no DeepMind in 1943. Computer science stuff usually takes a few decades to mature.
Thank you. It's weird---Swift is a beautiful language as a language, but all these ecosystem things make it really really hard to use. (The worst IMO is package management---like, some libraries require cocoapods, and some work with swift package manager, but swift package manager for swift 3 and 4 are totally different, and how does Xcode vs command line fit into all of this and libraries vs frameworks and COME ON IT SHOULDN'T BE THIS HARD!)
As someone who wants to see Swift succeed as a main-stream language I could not agree with you more.
I can forgive the API differences between SPM for swift 3 and 4, after all the ecosystem is evolving and I'm a fan of iterating interfaces early rather than having to keep early mistakes in the spec _forever_ for the sake of backward compatibility. But the Package Manager is still pretty far from being nice to use. In several of my projects which are using it, I also have a custom CLI implemented to streamline some of the worst pain points.
As far as dependency management, I think a huge issue is the lack of proper support for Swift Package Manager for iOS projects. Apple's 1st class solution for including software modules in iOS projects is "Just drag the framework into your XCode project", which is of course an absurd solution for any serious workflow.
Cocoapods (and to a lesser extent Carthage) have stepped in to fill the gap as 3rd party solutions, but there's still no real way to use Swift Package Manager to manage the dependancies for an iOS project. As a result a lot of libraries don't even bother with SPM support since the largest portion of their target audience will never use it. And thus adoption is slow, so there's not a large user-base demanding improvements.
I desperately want to write my ML code in a statically typed language. I hope one of these Swift-based projects takes off. But I am worried that they will not treat Linux as a first-class platform.
I guess Apple's ML efforts are focused on Core ML and friends and will probably treat Linux as second class or won't support it at all. But Google works on Tensorflow, Lattner wants to do it with Swift and I'd assume that for them Linux absolutely is the primary target.
Julia relies heavily on type inference, similar to all variables to be marked auto or auto& by default in C++.
Functions are templated on the argument types by default, in C++ terms. So if you call a function f(x) first with an integer and then with a floating point number, Julia will generate separate machine code optimized for each case. Annotating function arguments with types only serves as a filter for whether the function applies to certain types, and can be used for method overloading (function template specialization in C++). The fact that functions are generic by default lends itself well to forward-mode automatic differentiation using method overloading.
Julia is still a dynamic language though. This means that if a variable is bound to an integer in one branch of an if-statement while a string is assigned to it in another branch depending on the value (not type) of a function input, then the variable's type cannot be inferred and needs to be 'boxed', i.e. the runtime type needs to be stored along with the data. In such cases, function dispatch can no longer be done at compile time, and needs to be done dynamically instead. While this is useful for rapid prototyping, it does result in reduced performance, so it should be avoided in code with high performance requirements. The type elision part of this is somewhat similar to boost::any.
Statically typed? As in Go? I invite you to work on Gorgonia (https://github.com/gorgonia/gorgonia). It's written in Go, and neural networks written with Gorgonia has a decent type system (https://github.com/chewxy/hm) . It uses dual numbers for differentiation.
On top of that, the meta language, Go is also statically typed.
We need help on various things: compilation optimizations, more operators, etc.
Well typed, as in Haskell? I invite you to work on Grenade.
Type errors can happen in small programs. In ML applications, which are computationally very expensive and take a lot of time to run, an avoidable type error can result in a considerable waste of time.
Type annotations can be useful in helping a reader get a rough outline at a glance, of what some unfamiliar piece of code is doing. Especially under heavy use of broadcasting.
The second is compactness is true only if you're chaining straightforward library calls. If you're writing your own functions or building non-trivial functionality on top, types will be of aid too.
> I desperately want to write my ML code in a statically typed language.
As annoying as it is to see the ML name being hijacked, that has to be one of the funniest inadvertent comments HN has ever produced.
I prefer MG, for Machine Guessing. This not only conveys "The machine has guessed a mapping from x to y", but also captures the research side of "I don't know what the machine is really doing anymore so here are my guesses".
The story for Swift on Ubuntu is actually quite good now, and is getting better all the time. I was an early adopter of cross-platform Swift and it's night and day from where it was even 18 months ago.
Also it's been announced that LSP support for swift will be a priority, so tooling should be improving substantially in the near future.
Just use C++. Modern C++ is really quite nice, and definitely has by far the best numerical libraries of any statically typed lang. Plenty of ML packages (most of the big names are written in C++, after all). Eigen3 is also the best linear algebra package I've used outside of Julia.
For whom it may concern, would be nice to consider autobatching [0] of DyNet fame when designing AD systems. Makes the world of difference for NLP applications.
Dual numbers are magic. They are somewhat obscure but very powerful e.g for reasoning about dynamically evolving systems. They are like complex numbers but instead of i^2=-1 you have epsilon such that epsilon^=0. Shit gets real from then on.
Dual number functions give you the value at the function as well as the derivative at that point. Very cool.
This is really cool, and I'm excited to see how this goes. When people first hear about automatic differentiation, their first reaction is "that's cool, why isn't it everywhere?", and the answer has been "it's really hard to use in practice".
Until recently, AD systems only worked on restricted subsets of languages, or required extremely cumbersome template usage throughout the code. ML frameworks like TensorFlow pushed AD into the mainstream: they are essentially AD-compatible runtimes, but they are cumbersome to use (you have to use their functions instead of the language-provided ones) and they miss potential optimisations.
So it's become increasingly clear that the next step was to actually embed AD as part of a language. I work on Julia, so I hope Zygote.jl (https://github.com/FluxML/Zygote.jl) gets there first, but it's always good to see others moving in the same direction.
I come from business programming in Swift (the usual CRUD API based stuff) and for me I completely get lost reading this once the formulas kick in. I have this problem as well when reading about more theoretical algorithm articles. Usually when I see the resulting code I do get it but I feel I miss out on a lot of theory.
Can anybody help me grok the more theoretical stuff in this article?
44 comments
[ 3.6 ms ] story [ 51.6 ms ] thread[0] https://github.com/apple/swift-evolution/blob/master/proposa... [2] https://github.com/apple/swift-evolution/blob/master/proposa...
Personally, I feel like it would be unwise to merge in all of TensorFlow's requirements into Swift. Many features, such as the ones you mentioned, are actually useful to outside of the context of machine learning, so it does actually make sense to make them generally available. But compiler support for something as specific as @differentiable seems like an anti-goal for Swift.
That's an...interesting statement, considering that neural networks are nearly as old as software itself. The original Pitts-McCulloch paper was published in 1943. Active use of them has certainly been ongoing longer than that blog author has been alive.
I can forgive the API differences between SPM for swift 3 and 4, after all the ecosystem is evolving and I'm a fan of iterating interfaces early rather than having to keep early mistakes in the spec _forever_ for the sake of backward compatibility. But the Package Manager is still pretty far from being nice to use. In several of my projects which are using it, I also have a custom CLI implemented to streamline some of the worst pain points.
As far as dependency management, I think a huge issue is the lack of proper support for Swift Package Manager for iOS projects. Apple's 1st class solution for including software modules in iOS projects is "Just drag the framework into your XCode project", which is of course an absurd solution for any serious workflow.
Cocoapods (and to a lesser extent Carthage) have stepped in to fill the gap as 3rd party solutions, but there's still no real way to use Swift Package Manager to manage the dependancies for an iOS project. As a result a lot of libraries don't even bother with SPM support since the largest portion of their target audience will never use it. And thus adoption is slow, so there's not a large user-base demanding improvements.
Why? What kind of ML code do you write?
It definitely doesnt treat Linux as second or third class
Is the main difference between static typing and Julia (from a user point of view) that type declaration/checking is only optional in Julia?
Or put in another way, what does Julia lose compared to a statically typed language?
Julia relies heavily on type inference, similar to all variables to be marked auto or auto& by default in C++.
Functions are templated on the argument types by default, in C++ terms. So if you call a function f(x) first with an integer and then with a floating point number, Julia will generate separate machine code optimized for each case. Annotating function arguments with types only serves as a filter for whether the function applies to certain types, and can be used for method overloading (function template specialization in C++). The fact that functions are generic by default lends itself well to forward-mode automatic differentiation using method overloading.
Julia is still a dynamic language though. This means that if a variable is bound to an integer in one branch of an if-statement while a string is assigned to it in another branch depending on the value (not type) of a function input, then the variable's type cannot be inferred and needs to be 'boxed', i.e. the runtime type needs to be stored along with the data. In such cases, function dispatch can no longer be done at compile time, and needs to be done dynamically instead. While this is useful for rapid prototyping, it does result in reduced performance, so it should be avoided in code with high performance requirements. The type elision part of this is somewhat similar to boost::any.
On top of that, the meta language, Go is also statically typed.
We need help on various things: compilation optimizations, more operators, etc.
Well typed, as in Haskell? I invite you to work on Grenade.
https://github.com/owlbarn/owl/blob/master/README.md
Why? ML programs are typically sufficiently compact to not require a type system.
The second is compactness is true only if you're chaining straightforward library calls. If you're writing your own functions or building non-trivial functionality on top, types will be of aid too.
The confusion might arise since ML is also a statically typed programming language, usually seen in the wild in the form of Standard ML and OCaml:
https://en.wikipedia.org/wiki/ML_(programming_language)
As annoying as it is to see the ML name being hijacked, that has to be one of the funniest inadvertent comments HN has ever produced.
I prefer MG, for Machine Guessing. This not only conveys "The machine has guessed a mapping from x to y", but also captures the research side of "I don't know what the machine is really doing anymore so here are my guesses".
"I'd just like to interject for a moment. What you’re referring to as ML, is in fact, GNU/ML, or as I’ve recently taken to calling it, GNU plus ML..."
Also it's been announced that LSP support for swift will be a priority, so tooling should be improving substantially in the near future.
[0] https://arxiv.org/pdf/1705.07860.pdf
Dual number functions give you the value at the function as well as the derivative at that point. Very cool.
I guess one might consider tagged duals hyperduals.
Until recently, AD systems only worked on restricted subsets of languages, or required extremely cumbersome template usage throughout the code. ML frameworks like TensorFlow pushed AD into the mainstream: they are essentially AD-compatible runtimes, but they are cumbersome to use (you have to use their functions instead of the language-provided ones) and they miss potential optimisations.
So it's become increasingly clear that the next step was to actually embed AD as part of a language. I work on Julia, so I hope Zygote.jl (https://github.com/FluxML/Zygote.jl) gets there first, but it's always good to see others moving in the same direction.
Can anybody help me grok the more theoretical stuff in this article?