People often ask me for examples of non-scientific Julia codebases. My favourite one is Franklin.jl (https://franklinjl.org), a static site generator. This blog discusses quite a bit more, and it doesn't even talk about all the improvements in integrating with databases and such.
Another example is that all the infrastructure for serving Julia packages to Julia users in the new 1.5 release is implemented in Julia. This is a content distribution network served by highly concurrent servers over HTTPS. The systems interacts with git servers, GitHub APIs and AWS S3 among many other moving pieces. Code can be found here: https://github.com/JuliaPackaging/PkgServer.jl.
Writing servers in Julia is really pleasant thanks to the clean coroutine-based task model. Under the hood, Julia uses libuv to get efficient high-concurrency I/O with an event loop. But from the user's perspective, you just write simple blocking code and use `@sync` and `@async` to spawn concurrent tasks. If you want to use multiple threads, just use `@spawn` instead of `@async`. This is very similar to Go's highly successful I/O and concurrency model.
That's a great example. Also, JuliaBox.com (discontinued service) ran on Julia in production for 3 years, and has now been replaced by JuliaHub.com, which is also largely written in Julia.
Several Julia Computing customers also run long running Julia jobs in production, and some are discussed in our Case Studies (https://juliacomputing.com/case-studies/).
Having followed it since the 0.4 era I'm excited that Julia is production-ready (though I have professionally moved to elixir). Since I'm working in ml infra, I'm interested in orchestrating Julia jobs (via Elixir), what I'm less clear on is if there are good resources on how to deploy Julia, keep dependencies sane, pin memory usage. Or does one just go straight to containers?
For services we deploy at Julia Computing, we use docker containers since that's just the way devops is done these days, but we also use manifest files to pin all our dependencies which almost makes dockerizing things redundant. It doesn't hurt to have a couple of layers ensuring reproducibility.
Thanks! I'm also very excited by pluto.jl, looks fantastic. I'll probably be offering public machine learning vms with Pluto builtin by the end of the year.
I'm having a lot of fun with Pluto.jl and PlutoUI.jl this past week, it's so easy to use and add interactivity. There are still some rough edges, but that's what PRs are made for!
Yep, to build on Stefan's sibling comment, we deploy a few services through docker that follow the flow of <merge on github> -> <build on dockerhub> -> <deploy through watchtower> that has been working for us really well.
Docker has some nice infrastructure that we like, such as the ability to set memory limits, automatically restart processes on crash/if a healthcheck goes sour/on reboot, easily spin up different versions of the same service (based on which image you tag to a name), etc... Containerization has done a lot for our ability to bring servers up onto heterogenous hardware easily and quickly.
As an example, this repository [0] is deployed on 10 machines around the world, providing the pkg servers that users connect to in geographically-disparate regions. The README on that repository is a small rant that I wrote a while back that walks through some of the decisions made in that repository and why I feel they're good ones for deployment.
To answer your specific questions:
* How to deploy Julia: we use docker containers and watchtower [1] to automatically deploy new versions of Julia.
* Keep dependencies sane: it's all in the Pkg Manifests. We never do `Pkg.update()` on our deployments, we only ever `Pkg.instantiate()`.
* Pin memory usage: We use docker to apply cgroup limits to kill processes with runaway memory usage [2]. This is only triggered by bugs (normal program execution does not expect to ever hit this) but bugs happen, so it's good to not have to restart your VM because you can't start a new ssh session without your shell triggering the OOM killer. ;)
As someone working in power systems (electrical engineer) and who programs in Julia for fun a bit. I'm curious about y'all's business model. I've asked before and only gotten something about load forecasting which I can't see supporting 30 people. Can you shed any light on the business model? Contract work? Do y'all have an actual software product? Please note that I'm not being critical, I am legitimately curious as the work y'all do seems to be very academic which I tend to see more coming out of universities. 400k loc is a lot btw, so it seems like y'all are pretty busy!
The short answer is that you are underestimating the scale.
Invenia operates over most of grids in the continental US.
Think less "helping one powerplant how much to generate, and thus saving them money on wasted fuel if not needed."
And more "helping thousands...", we're more linked to the system operators than the individual plants.
And the systems move huge amount of power each day, so small percentage efficiencies is still a big number dollar wise
No, I know what a system operator is which is why I'm asking the question. It was a fair assumption though to assume I was talking about the utility level of granularity though.
As far as I know, all ISO's in the USA use either ABB, Siemens, or GE for their market optimization software (huge systems) or basically those same vendors plus a few more for their EMS systems. There are also plenty of other amazing products for transmission planning, short term studies, and transient stability studies. More specifically, how are you linked to the system operators? When you say you optimize the grid, that could mean any one of a thousand different things. Are you providing various inputs (forecasts, risk factors, etc) that are used by some markets? Please note that I'm not questioning that y'all work with some of these entities in some fashion, I'm just trying to get a better handle on the specifics of where y'all are plugged in.
I'd be super interested in having you (or someone at Invenia) on as a guest on a podcast that talks about tech stacks. We'd cover things like what development and deployment is like, which tools you use and why you chose them, etc..
At the moment there's no Julia episodes yet. If you want to come on, head over to https://runninginproduction.com/ and click the "become a guest" button near the top right to get the ball rolling.
Oh, that would be awesome! I would really love to listen to a bit more detail about the choice of Julia for a company's stack. (I am an academic, so choosing Julia was very easy for me :) but I can imagine this choice could likely be harder outside of basic research.)
Julia is next gen. The whole experience is so much more pleasant than anything in the space. The interop is also crazy. How is it possible that you can call Matlab, Python, cpp, fortran, Rust from one language?
I legit think that even if you are using say pytorch. You are better off writing your code in julia and using the python interop.
In my experience Flux.jl has a ways to go before it reaches Pytorch performance. Apparently this has something to do with some very efficient GPU kernels that Pytorch uses - I wonder if it's possible for Flux to "borrow" those GPU kernels? It's possible to write GPU kernels in Julia so hopefully this will help close the gap.
The short answer is yes, it's possible and easy. The longer answer is, that's a bit of an apples to oranges comparison. Pytorch is calling optimized c++ kernels, whereas Flux is completely generic and can do GPU and CPU codegen (including AMD). Flux can do things Pytorch just doesn't have CUDA code for, It's more generic and composable and it's getting there with a much smaller team/budget. That means it has more potential, with the drawback that it isn't quite as fast, for now, in some of the common use cases the pytorch has been hard coded/ hyperoptimized for.
In other more complex cases beyond big fat matmuls and such, it's much faster, even comparing to torchscript (sciml, neural ODEs etc).
Still, the codegen is getting there with the funded work on the GPU compiler and memory allocation and there eventually won't be any sort of tradeoff.
In the meantime, Yes, there is a package at allows calling pytorch kernels without modifying your code. This is also only possible with such finesse due to Julia's power and composability. See here for the package: https://fluxml.ai/2020/06/29/acclerating-flux-torch.html
I feel I should point out that if you need performance and you're mostly only interested in classic PyTorch or TensorFlow style machine learning (i.e. you don't need Flux.jl's characteristic flexibility), then you should really check out KNet.jl [1]. KNet is generally around the same flexibility and performance as something like PyTorch. It doesn't get nearly as much attention as Flux because it's less novel, but it really should be looked at more.
It really depends on where you're looking. We tested neural ODE compatible solver benchmarks from 3 ODEs and about 1 million ODEs and saw an advantage for Julia of >30x across the board [1]. Then showed the spiral neural ODE from the original paper train 100x faster in Julia [2]. So it should at least be qualified as, PyTorch has some good kernels, and it has some things it doesn't optimize close to as much as Julia.
Yeah, the quality of the language is pretty incredible. Especially compared to Python, which has been a disaster for as long as I can remember.
I think the ecosystem could be a bit better, but that's more of a personal opinion rather than a serious concern as, like you alluded to, you the interop story is insanely good.
Right now, the DataFrame package doesn't support row indices like Pandas does. Row indices are especially useful time series data, and there's another package called TimeSeries that support a dataframe-like type called a TimeArray that supports dates as row indices.
TimeArray is like a DataFrame, but with a lot less features. You can't really do that much with them, and they only have limited support in an auxiliary package for resampling.
It would be nice if a new DataFrame library could be created that was inspired by Pandas.
In general, I think Julia suffers from the problem of too many small packages. For example, there's 3 libraries I have to use for time series: TimeSeries.jl, TimeSeries Resample.jl,and TimeSeriesIO.jl. None of the libraries are particularly large and should all be merged into one package.
Part of the reason for the dirth of small packages is because, unlike Python, Julia's strong support for function polymorphism allows for seamless extension of existing functionality with new packages. Unfortunately, I think this has created a cultural problem in which new packages are created that only slightly modify existing functionality.
I think Python is less a disaster than it is a tool being used for something it was not designed to do.
It's a pretty good scripting language, and it's a "good enough" interface that implementers of C packages can expose powerful features, and that can only go so far. User's can't easily compose the powerful features that library designers didn't intent.
Julia, on the other hand, was designed for the explicit purpose of high-performance scientific computing, with a syntax informed by the tools people already knew (python, MATLAB). It's no wonder it's so much easier to use.
Julia follows semantic versioning [1] and is now on version 1.5, which means there been five non-breaking releases introducing new features (and dozens of bugfix only patch releases). Code that worked on 1.0 also works on 1.1, 1.2, 1.3, 1.4 and 1.5 and will keep working in 1.6, etc. until such a time as a breaking 2.0 release is made. Even then breaking changes will be kept minimal and only made to enable worthwhile language features.
DataFrames.jl provides very similar functionality to the R DataFrame, plus some functions similar to those from the R reshape package. Other packages build upon this to provide e.g. LINQ-like operators [2].
It's fine, but I think row indices would add a lot. Right now there's also a TimeSeries package that could be obviate if Julia had a Pandas-like implementation. And the TimeSeries functionality is very primitive, not even supporting resampling and forward or backfilling data
Row indices are super powerful and things are annoying without them. I've actually been thinking about programming a new and improved data frame library inspired by Pandas (but not so ugly and complicated), but the amount of work to write and then maintain it would be prohibitive.
I've been thinking about jumping on the Julia ship for a while. I like the LISP-like nature of R and miss its flexibility very much while working in Python (a language I also enjoy, but not particularly for data science). Julia, specifically its macro story and speed, fascinate me.
One of the things that still keeps me at bay is the JIT and the pre-compilation in general. It seems like it's still not very easy to actually compile a Julia library or an executable. There is PackageCompiler.jl, but my understanding is that it necessitates pulling in sys.so, which is some 130 MB in size, making it prohibitive for many projects.
It's more generic than that. For basic general purpose computing with some calculation and where performance is of some importance, I can design and ship the whole thing via something like Lazarus in 5-10 MB, or -- from my understanding -- 130+ MB in Julia. I understand that this might be just asking Julia to break out a bit too far out of its intended use case, but it would be nice.
I know for me the documentation and general process of using PackageCompiler.jl is a little daunting for me in a way that other parts of the ecosystem and docs haven’t been.
I was just trying to do a quick test app the other day to see if I could build a Hello World app, and (this may have been my fault) but I hit the threshold of cognitive overhead before deciding to revisit later. This isn’t a criticism so much as a recognition that compiling binaries may take a bit more care / attention / learning in Julia than something like Go, Rust, or even Racket.
As for binary sizes, for some serverless deployments smaller binaries can open up a little easier path.
I’ll try and take another run at PackageCompiler and give it a more fair shake. If in that process I find anything constructive to report or submit documentation PRs, I will do so.
Thank you for your work on Julia. Julia was on my radar a few years ago and then dropped off again. I’m excited to see where it has come in that time.
In a few use cases, such as big-data processing and things like that, a common pattern is to haul a tiny data-processing kernel to several machines to do an operation on some data they hold, rather than shuffle the data around.
This pattern can be seen in several places and frameworks like Spark & company, and it's also pretty much the foundational idea behind processing data with Joyent's Manta[0]. It always pretty much boils down to "serialize a function, and send it somewhere, to run on the data locally".
Some of these frameworks do it only through native functions (Spark runs serialized JVM procedures, and for Python functions I believe it shells out to an actualy Python interpretes), others are completely agnostic to this and work with binaries (Manta).
It'd be pretty hard to do, and have a lot of overhead, if all of those "functions" that I'm shuffling around are, at the lower bound, 130MB in size.
I understand that a lot of this can be done directly at the Julia layer for Julia things (much like Spark does with JVM things), but it kinda limits the applications of this nonetheless.
Julia can already do this in its `remotecall` API (actually has done this since v0.1), which serializes a closure and sends it to a remote node for computing.
However, this is not related to the system image (sys.so) that is used to start the Julia process, which has a compiled version of base, stdlibs and whatever else you choose to build in.
I don't know about him, but I come from a somewhat weird angle, I'd want to use Julia to replace Python for command line, Opsy/DevOps type tools. Go has quite a few things I'm not a big fan of, at the moment, so Julia seems the perfect candidate. But right now its deployment situation is too complicated, I just want a static binary or a self contained zip.
I think for that, you might want to look into https://github.com/dmolina/DaemonMode.jl. This is essentially a just a clever utility for keeping a background julia session running that you feed scripts to, rather than spinning up a new session every time you launch a new script.
This is a nice alternative to the AOT compilation offered by the likes of PackageCompiler.jl
Sorry, I think I missed your final sentence when I originally read your comment. Yeah, DaemonMode.jl won't help that part. Perhaps combining Docker with with DaemonMode or PackageCompiler would solve some of these woes, but that might still not feel fully satisfactory.
Hopefully we'll have more / better AOT compilation and deployment in the near future. There's a lot of rumblings of exciting things starting in Julia AOT compilation, and I think it'd make a killer combination with our package manager's binary artifact deployment story.
From the Package.jl description: "Creating custom sysimages for reduced latency when working locally with packages that has a high startup time." That sounds really far from statically compiling everything with a compiler that will strip out all the code you don't code, inline aggressively, etc... Static languages at least let us ~pretend that even if our dependencies are bloated, our binaries will be elegant tightly optimized things :)
Julia does at least inline fairly aggressively, and it respects the `@inline` macro so long as the call is type stable.
In practice, it may often inline more aggressively as there aren't any boundaries that will prevent inlining.
If you write function `foo` calling `buz` from package `Buz.jl`, which calls `bar` from package `Bar.jl`, `bar` can be inlined into `foo`.
You can think of it as always have `-lto`, because the source of the Julia functions you're calling will always be available to the compiler and thus subject to (potentially aggressive) optimization.
Do you guys know if it's possible to generate self-standing C++ code from Julia code? (i.e., code that you can then compile into another project, and will not require any interpreter to run or anything like that)? And if it's possible do you know how good the generated code is (in terms of speed)?
We did have this once upon a time, going from Julia->LLVM->C using the llvm-cbe project (https://github.com/JuliaComputing/llvm-cbe). It is not supported any more, since there are easier ways to achieve what you are asking for. If your code does eval, then there is no way out.
If you can ensure your code and dependencies does not have dynamic things that need compilation at runtime, it is fairly easy to remove the dependency on bundling LLVM. Speed is the same, since you are then just running cached native code.
I can currently see Julia only as a replacement for Python, with the biggest advantage, that fast modules can be written in-language, and you do not default back to C. So in a sense it could solve the two language problem.
But then for my applicationsspace there are problems:
- compilation seems possible now, but one has to be careful of GPL (like in the fft)
- no way to turn off GC (so byebye real-time possibilities).
- what about GUI design? Heard some mixed messages about it
> - no way to turn off GC (so byebye real-time possibilities).
While Julia does not offer semantic guarantees that you can avoid the GC, it;s actually quite possible and easy to write code where you're manually managing all your memory and the GC is never invoked.
> - what about GUI design? Heard some mixed messages about it
There's a lot of promising things happening with Makie.jl, Stipple.jl, Dash.jl, Pluto.jl and others. This space is still a little immature in Julia, but it's progressing fast.
I know there are workarounds, but when you do not have guarantees the phrase 'production ready'is not the first that comes to my mind. (Of course in the problem spaces of matlab and python, and probably R, Julia is probably on par)
What I meant: production ready depends on the feature. So for substituting Python (and beeing able to write fast in-language extensions), the language is most likely production ready; so in a sense it is a replacement for Python + C; but not a replacement for C. And this is a bit sad, because it could have been (building linked executables should be possible, since the language is compiled).
But I like your question, lets call it 'which languages are all purpose production ready'; probably only C, C++, Java (there is a real-time version), LabView.
I don't think the blog post above here is about using julia for ultra-low-latency systems to be pair. 'Production ready' is a pretty vague statement and certainly doesn't apply to every usecase.
I'd say that if people have other reasons to use julia in a realtime low-latency production system -- like needing it's scientific stack -- then you can certainly make it work, you just need to do some testing to make sure you're not hitting GC.
I'm also interested in the packaging/deployment story. Can you easily make a zip with everything built-in and 0 OS dependencies? Can you ship a single, statically linked binary?
Now, 8 years later, I'd say essentially everything there has been achieved. The most important difference between Julia now and Julia then is the gigantic, vibrant and interacting ecosystem that's sprung up (though share are other important technical differences too!)
_______________________________________________
Here's the pitch I'd give:
Julia's value proposition is mostly geared towards people trying to do things that are just too difficult, awkward or slow in other languages. It's not just about speed, dynamism and friendly syntax. It's also about solving the expression problem and providing unprecedented composability.
So while it may not necessarily have great appeal to 'end users' (who are people just calling numpy/scipy/pandas/whatever functions the way they were meant to be called by the library writers), it does appeal to people who make the sort of things end users want so the community is currently swelling with very talented people making state of the art research libraries. The language is designed for 'composability', so that it's very easy to combine packages together in ways the authors didn't necessarily intend or forsee.
This approach is really starting to bear fruit. Our package ecosystem has some real, state of the art stuff not easily available in other languages, and it's often built in less time and with less resources. For 'end users', whether or not Julia makes sense for them really just depends on if the things they need are already available.
Another huge advantage is that even if you're not some super researcher developer, Julia substantially lowers the bar for mere mortals such as myself to do package development. The composability I mentioned above means that I can stand on the shoulders of giants as I implement my own stuff, plus the fact that the majority of libraries are written in pure julia means that I can easily "peek under the hood" to figure out how they work and learn from them. This is really special and can't be replicated in python, because 'peeking under the hood' invariably means reading very specialized C code, rather than Python code.
I think Julia will someday eat Python for scientific computing, but it may be a while before that becomes obvious to Python users. In the meantime, we have an excellent package for calling and running Python code from withing Julia, called PyCall.jl [1], so that can make the burden of switching even easier.
Thank you for the detailed answer. I guess I should just give it a proper try by converting an old project into Julia.
So here is an example and my worries ... My project uses GDAL, PIL, SentinelSat libraries to collect Sentinel 1 imagery and process it, pandas, numpy, scipy extensively. Would I find a bunch of those modules missing? I assume so ...
I guess then I might be better of to trying a new project in Julia. Something not too easy but not too hard.
I don't know much about geospatial stuff, but I do know there are people working on that in the community. I'd be rather surprised if you couldn't find Julia packages that fill those needs for you.
Worst case scenario, you might need to do the GDAL, PIL and SentinelSat stuff with PyCall and then the Pandas, Numpy and Scipy stuff should all be quite well supported from the Julia side. I can ask around to see if there's anyone who can answer this better though.
This is a good pitch and I think composability is the major selling point for programming in the large. Time will tell though if the ecosystem doesn’t fragment as different approches or competing packages appear, for political if not technical reasons. The Torch vs TensorFlow choice could have been in Julia.
> I think Julia will someday eat Python for scientific computing, but it may be a while before that becomes obvious to Python users.
I can’t say I understand this rather pervasive attitude. If you have well tested code in a language which you can easily interface with, why bother rewriting it except for reasons of performance or compositionality? Or perhaps you mean that we’ll all be writing Julia even if some algorithms still sit in Python?
Turnover. More and more students will be taught starting with julia instead of python, since it has a foothold in academia. Not all academics, but some academics will be forced to switch over due to performance characteristics, or because they have to reproduce some existing code that's already in julia, and the ergonomic delta will be enough that those people (mostly grad students and postdocs I imagine) will want to their next project in julia-not-python, too. Eventually you won't be able to turn down a job candidate because they learned in julia. The hangers on in python will hang on of course (it's very hard to convince people it's time to move on due to sunk cost emotional attachment).
What's the application? Would love to read more if there is a paper. The Julia Lab folks at MIT are collecting big parallel applications for pushing parallelism in Julia further.
There was no paper, sorry, we talk about some of the research here https://www.youtube.com/watch?v=aP0Y1uAA-2Y
big julia demo 40 minutes in. But the video doesn't feature the results I talk about (that happened a few months after the video) the specific results are somewhere in a DARPA presentation (but I don't have those slides, and the data are like 5 laptops back for me). The parallelism was, well, Gustafson parallelization, I pushed a randomly-generated 10k x 10k matrix solve to each of the nodes with a custom soft floating point to measure the relative numerical accuracy. Each node did all of one run per core, it was done very fast, but it would have taken months on a pc. As it was a gustafson scaling at its finest, there was zero cross-node communication during the computation.
The containerization I used was singularity. I don't really love them much anymore, for various non-technical reasons, but it got the job done.
> I can’t say I understand this rather pervasive attitude. If you have well tested code in a language which you can easily interface with, why bother rewriting it except for reasons of performance or compositionality? Or perhaps you mean that we’ll all be writing Julia even if some algorithms still sit in Python?
I mostly mean that I think Julia will someday have more mindshare in scientific computing than Python.
However, I'd say that the reason you see the "rewrite it in pure julia" attitude everywhere is that we enjoy a lot of benefits from having more and more things in pure julia, namely that our very powerful metaprogramming and code introspection tools can more easily 'get inside' Julia code, and that the compiler is able to perform interproceedural optimizations (IPO). IPO are not always important, but when you need them and don't have them, you really miss them, so at least for numerical kernels and such, it's really nice to have as much pure julia infrastructure as possible.
In the meantime though, as I daydream about a world where everything is written in Julia, there's nothing stopping me from being pragmatic and wrapping / bridging to and from other languages.
I completely agree Julia is fantastic for computational kernel construction. I would like to never again write GPU kernels by hand (though Numba has been quite sufficient for my needs). I just don’t see this as everything.
I'd like to throw my opinion in as well to add to this.
Julia's syntax and language design is brilliant, but the biggest reason Julia has been my favorite language since I first learned to program is how astonishingly easy it is to use tools other people have made, and also to create tools for others to use.
The packaging system is just great for development, both as a user, and as a maintainer. I've been surprised by how easy it is to take stuff I've made and get it to work with a myriad of packages for the purposes I need. Multiple dispatch does wonders here too.
In the recent survey they said that 30% of respondents had contributed to a package. 30%! It's a super active ecosystem and the barrier of entry is low enough where I don't feel "imposter syndrome" as strongly as I have elsewhere--that I need to pass some threshold in order to properly contribute.
As far as I'm aware, it has not been a problem yet, though it's likely only a matter of time as the language community grows. I see conversations happening regularly on how to detect and combat this though.
It's a hard problem to combat, because fundamentally, package code must be able to do arbitrary things. One advantage julia has though is that packages tend to be written in pure julia, which makes the duty of checking and detecting malicious code much easier.
>appeal to people who make the sort of things end users want
> lowers the bar for mere mortals such as myself to do package development
That's a powerful combination. I've been close to contributing to Python packages a few times, but the overhead is pretty high. At this point, I also feel like it would be a waste of effort not to write stuff in Julia instead.
> Julia will someday eat Python for scientific computing
It's well on the way. Julia users have a reputation for being evangelists. Heck, I made a convert even before I actually started using Julia in earnest: my labmate had a problem he couldn't solve in his favorite language, and I encouraged him to try Julia -- it turned out great!
The biggest value prop for me is when you need something done fast that’s not already in scipy etc. In python you will need to write a C extension or fumble with numba / cython / whatever, in Julia you just write it.
I also think writing out math is much more concise in Julia, no more np.whatever(np.sin(x.detach()) + cos(np.sqrt(y)).mean().
It seems about the right time to give it a go. How is the package ecosystem in your experience? Can I expect fairly decent support for numerical optimisation stuff , graphing? Munging xml/Json/etc? Will I find that I have to write my own oauth2 client library to call API? Etc
I’m also looking at this now after more than a decade in Python. I don’t think “move” is the right word but rather “coexist”. You can call back and forth between Python and Julia, so why pretend you have to choose? For example, matplotlib is one of the more feature complete plotting backends for Plots.jl.
The question I am currently researching is not what Julia is good at but what are the pitfalls and drawbacks of particular technical choices. I know them by heart now for Python, and my team relies on knowing what doesn’t work well. For Julia it’s an unknown unknown. Understandably all the blogs are about why Julia is great so it makes seems to go slowly and coexist instead of moving.
I got fed up with Python when I found out that xarray couldn't support unitful quantities. In Julia, it just works. In Python, it turns into a massive project due to duck-typing & single-dispatch. It's been 5 years and xarray still doesn't have units, although it's getting close.
Is the debugger in Vs code or anywhere else better now? I remember trying Julia just a few months ago and the debugger frustrated me enough that I went back to print statement debugging.
I’m not using Julia actively, but I’ve been following the news and I there was a lot of improvements made to the VS Code extension, and better debugging experience as one of them. You should try it out again.
Thanks for sharing this. Are there instructor resources available to teach Julia in a data science program? I use R mostly but I am porting one course on ML applications to Python. The communities in both R and Python are vibrant for instructors to create structured learning experiences for data science. How about Julia?
While being a big fan of Julia in general and recognizing a very significant progress in both language and ecosystem development [kudos to Julia's core team as well as numerous contributors to the language and its growing ecosystem!], I respectfully disagree with the post's author conclusion in its entirety.
For scientific computing and advanced machine learning domains, Julia is certainly a breath of fresh air and a no-brainer decision. However, as a general-purpose language as well as technology stack, I think Julia has still a long way to go. This is mostly due to relative immaturity of that part of the ecosystem, spotty - and sometimes even non-existent! - documentation (of course, except for the language itself and quite a limited number of core and popular packages) as well as some other issues, including tooling, development/compilation performance and limited pool of skilled developers.
So, from a startup founder perspective [who has to select the optimal, risk-minimizing, platform stack], despite Julia's many attractive features (including powerful meta-programming facilities - in my case, for potential DSL development), I'm now leaning toward Python and .NET ecosystems. Both offer very mature and large package ecosystems along with comprehensive tooling support and incomparably larger pool of skilled developers. Additionally, .NET offers stability of consistent development improvements, backing of a major corporation [no acquisition risk] along with support for modern enterprise-focused architectural practices and patterns, including DDD and multi-tenancy.
Not trolling. I don't really know why people like to shoehorn a purpose-oriented language into all domains. Eg: Julia for scientific computing, Rust for low level programming. They would be great if they focused on those "strong zones".
I read this article not as a "Julia is ready to replace the all-purpose languages used in business", but as "Julia is ready to deploy it's scientific computing into production environments" (as opposed to just local/hobby tasks and academic environments which were the early focus). No one is recommending people to do their small e-commerce in Julia (or perhaps Rust) unless you're in for the fun of the languages (or your e-commerce happens to be a special case that will play with either language's strengths). But having a solid web deployment story is important to both regardless, be it in Julia to have live dashboards or integrating your scientific models in your microservice architecture and to serve them directly to clients or Rust for embedded web servers for IoT and other devices. For a language to be great in one domain, it has to be at least good in everything around it for when you need to connect that domain to the real world.
And Julia creators do focus on it's strong zone, with all the works on TPU, HPC and parallel computing, and so does the community with the stuff presented in juliacon like interactive reproducible notebooks (Pluto.jl) or data dashboards (Stipple.jl), both using the web domain to improve the scientific computing domain.
I'm not disagreeing with most of your comment above, except for how the post reads. To be accurate, the post's author set the context to general-purpose computing, not the scientific one. Here are two arguments for why I think so.
1. Intro section (italics emphasis mine).
"... delivering complex enterprise projects:
Julia is fast, and has a very nice syntax, but its ecosystem is not mature enough for use in serious production projects.
For many years I would agree with it, but after JuliaCon 2020 I believe we can confidently announce that
Julia is production ready!"
2. Section "Building microservices and applications". Even more so, practically all sections in the post, except just one ("Managing ML workflows"), describe various general-purpose and enterprise computing aspects.
Therefore, I don't see how one could view the post and author's conclusion purely from scientific computing perspective.
Fair enough, it might be my own view on what I'd use Julia in a production environment clouding my interpretation of the scope of the text, especially as the author defines this as his area at the start and says how it's what Julia shines at. Julia being an acceptable ("production-ready") language means that whenever my problem hit the areas that Julia shines, it becomes a valid candidate to evaluate considering all pros and cons.
And while right now I wouldn't recommend Julia for that web domain unless you also need it's number crunching features (which is more than just scientific computing, I work with large finance systems and there is definitely a lot of that stuff), I do think it's more of a library/community problem than a language problem. The language is well equipped to handle it (easy to use from a scripting perspective, easy parallelism, fast after warm-up, allows for clean abstractions to create sophisticated web frameworks, the mentioned easy FFI), what it lacks is the coverage and maturity of the tools and support (so I don't feel like I'm at a risk at any point that I need to do some integration, such as integrating with Kafka or any other systems, without having to write my own solution or integrating multiple languages at every step).
That's different from systems programming, embedded, game development and even GUI development (at least until smaller binaries with less warm-up) right now. Even with good library it will not be competitive with the languages that already claim those domains. Julia is a general purpose language already and more than a matlab substitute, it does not need to be good at everything (and it shouldn't try), but I don't think it should restrict the scope too much either.
I appreciate your thoughtful comment. Clearly, we are on the same page on the topic at large. As mentioned in my original comment above, I see no problems with Julia language per se (quite the opposite - I find it elegant, flexible and powerful), but rather emphasized that present issues exist around relevant ecosystem (documentation, packages, tools and support).
I doubt this has anything to do with 'feeling a need to identify' or anything like that. Many people who code aren't programmers, they have learnt a language to solve problems they have. They know just one language and do not have the time or capacity to learn another 'low-level' fast (but difficult) language for fixing bottlenecks.
Something ModelingToolkit.jl, JuSDL.jl, NetworkDynamics.jl, Modia.jl, etc. all are extensions of Simulink in some form. Supporting stochastic equations, delays, automatic parallelization, acausal modeling, etc.
The built-in array (which is multidimensional) is unnamed, but Julia was made in a way that anyone can create as a library their own types that are as high performance as the native ones (most of which are written in Julia) so you can just get a named array package like [1].
123 comments
[ 5.7 ms ] story [ 242 ms ] threadWriting servers in Julia is really pleasant thanks to the clean coroutine-based task model. Under the hood, Julia uses libuv to get efficient high-concurrency I/O with an event loop. But from the user's perspective, you just write simple blocking code and use `@sync` and `@async` to spawn concurrent tasks. If you want to use multiple threads, just use `@spawn` instead of `@async`. This is very similar to Go's highly successful I/O and concurrency model.
Several Julia Computing customers also run long running Julia jobs in production, and some are discussed in our Case Studies (https://juliacomputing.com/case-studies/).
[0] https://genieframework.com/
Docker has some nice infrastructure that we like, such as the ability to set memory limits, automatically restart processes on crash/if a healthcheck goes sour/on reboot, easily spin up different versions of the same service (based on which image you tag to a name), etc... Containerization has done a lot for our ability to bring servers up onto heterogenous hardware easily and quickly.
As an example, this repository [0] is deployed on 10 machines around the world, providing the pkg servers that users connect to in geographically-disparate regions. The README on that repository is a small rant that I wrote a while back that walks through some of the decisions made in that repository and why I feel they're good ones for deployment.
To answer your specific questions:
* How to deploy Julia: we use docker containers and watchtower [1] to automatically deploy new versions of Julia.
* Keep dependencies sane: it's all in the Pkg Manifests. We never do `Pkg.update()` on our deployments, we only ever `Pkg.instantiate()`.
* Pin memory usage: We use docker to apply cgroup limits to kill processes with runaway memory usage [2]. This is only triggered by bugs (normal program execution does not expect to ever hit this) but bugs happen, so it's good to not have to restart your VM because you can't start a new ssh session without your shell triggering the OOM killer. ;)
[0] https://github.com/JuliaPackaging/PkgServerS3Mirror [1] https://github.com/containrrr/watchtower [2] https://github.com/JuliaPackaging/PkgServerS3Mirror/blob/c6a...
The size of our Julia codebase is about 400,000 lines of code, spread over numerous internal and open source packages. You can cite the "we're hiring" slide from my Juliacon talk for that https://raw.githack.com/oxinabox/ChainRulesJuliaCon2020/main...
As far as I know, all ISO's in the USA use either ABB, Siemens, or GE for their market optimization software (huge systems) or basically those same vendors plus a few more for their EMS systems. There are also plenty of other amazing products for transmission planning, short term studies, and transient stability studies. More specifically, how are you linked to the system operators? When you say you optimize the grid, that could mean any one of a thousand different things. Are you providing various inputs (forecasts, risk factors, etc) that are used by some markets? Please note that I'm not questioning that y'all work with some of these entities in some fashion, I'm just trying to get a better handle on the specifics of where y'all are plugged in.
https://juliacomputing.com/case-studies/invenia.html
At the moment there's no Julia episodes yet. If you want to come on, head over to https://runninginproduction.com/ and click the "become a guest" button near the top right to get the ball rolling.
I legit think that even if you are using say pytorch. You are better off writing your code in julia and using the python interop.
Though I much prefer Flux.jl
In other more complex cases beyond big fat matmuls and such, it's much faster, even comparing to torchscript (sciml, neural ODEs etc).
Still, the codegen is getting there with the funded work on the GPU compiler and memory allocation and there eventually won't be any sort of tradeoff.
In the meantime, Yes, there is a package at allows calling pytorch kernels without modifying your code. This is also only possible with such finesse due to Julia's power and composability. See here for the package: https://fluxml.ai/2020/06/29/acclerating-flux-torch.html
https://github.com/FluxML/Torch.jl https://fluxml.ai/2020/06/29/acclerating-flux-torch.html
Also the gap between native GPU codegen in Julia and the hand-optimized kernels in PyTorch is lesser than you'd imagine.
In fact, see this talk at JuliaCon, which compares Julia on a 1000 GPUs against a CUDA/C implementation, and finds them to be extremely close.
https://www.youtube.com/watch?v=vPsfZUqI4_0 https://github.com/eth-cscs/ImplicitGlobalGrid.jl
[1] https://github.com/denizyuret/Knet.jl
However, we are integrating a lot of the underlying infrastructure to the point that both Knet and Flux will only differ in API approach.
[1] https://gist.github.com/ChrisRackauckas/cc6ac746e2dfd285c28e... [2] https://gist.github.com/ChrisRackauckas/4a4d526c15cc4170ce37...
I think the ecosystem could be a bit better, but that's more of a personal opinion rather than a serious concern as, like you alluded to, you the interop story is insanely good.
TimeArray is like a DataFrame, but with a lot less features. You can't really do that much with them, and they only have limited support in an auxiliary package for resampling.
It would be nice if a new DataFrame library could be created that was inspired by Pandas.
In general, I think Julia suffers from the problem of too many small packages. For example, there's 3 libraries I have to use for time series: TimeSeries.jl, TimeSeries Resample.jl,and TimeSeriesIO.jl. None of the libraries are particularly large and should all be merged into one package.
Part of the reason for the dirth of small packages is because, unlike Python, Julia's strong support for function polymorphism allows for seamless extension of existing functionality with new packages. Unfortunately, I think this has created a cultural problem in which new packages are created that only slightly modify existing functionality.
It's a pretty good scripting language, and it's a "good enough" interface that implementers of C packages can expose powerful features, and that can only go so far. User's can't easily compose the powerful features that library designers didn't intent.
Julia, on the other hand, was designed for the explicit purpose of high-performance scientific computing, with a syntax informed by the tools people already knew (python, MATLAB). It's no wonder it's so much easier to use.
Please somebody enlighten me
[1] https://semver.org
http://juliadata.github.io/DataFrames.jl/stable/
Edit: it’s a bit more into the weeds than I thought. Honestly the docs give a better picture of DataFrames.jl
[1] https://juliadata.github.io/DataFrames.jl
[2] https://github.com/queryverse/Query.jl
Row indices are super powerful and things are annoying without them. I've actually been thinking about programming a new and improved data frame library inspired by Pandas (but not so ugly and complicated), but the amount of work to write and then maintain it would be prohibitive.
One of the things that still keeps me at bay is the JIT and the pre-compilation in general. It seems like it's still not very easy to actually compile a Julia library or an executable. There is PackageCompiler.jl, but my understanding is that it necessitates pulling in sys.so, which is some 130 MB in size, making it prohibitive for many projects.
I'm curious to understand why the 130MB is prohibitive. Are you looking at embedded or resource constrained environments?
I was just trying to do a quick test app the other day to see if I could build a Hello World app, and (this may have been my fault) but I hit the threshold of cognitive overhead before deciding to revisit later. This isn’t a criticism so much as a recognition that compiling binaries may take a bit more care / attention / learning in Julia than something like Go, Rust, or even Racket.
As for binary sizes, for some serverless deployments smaller binaries can open up a little easier path.
I’ll try and take another run at PackageCompiler and give it a more fair shake. If in that process I find anything constructive to report or submit documentation PRs, I will do so.
Thank you for your work on Julia. Julia was on my radar a few years ago and then dropped off again. I’m excited to see where it has come in that time.
This pattern can be seen in several places and frameworks like Spark & company, and it's also pretty much the foundational idea behind processing data with Joyent's Manta[0]. It always pretty much boils down to "serialize a function, and send it somewhere, to run on the data locally".
Some of these frameworks do it only through native functions (Spark runs serialized JVM procedures, and for Python functions I believe it shells out to an actualy Python interpretes), others are completely agnostic to this and work with binaries (Manta).
It'd be pretty hard to do, and have a lot of overhead, if all of those "functions" that I'm shuffling around are, at the lower bound, 130MB in size.
I understand that a lot of this can be done directly at the Julia layer for Julia things (much like Spark does with JVM things), but it kinda limits the applications of this nonetheless.
[0] https://github.com/joyent/manta
https://docs.julialang.org/en/v1/manual/distributed-computin...
However, this is not related to the system image (sys.so) that is used to start the Julia process, which has a compiled version of base, stdlibs and whatever else you choose to build in.
This is a nice alternative to the AOT compilation offered by the likes of PackageCompiler.jl
Hopefully we'll have more / better AOT compilation and deployment in the near future. There's a lot of rumblings of exciting things starting in Julia AOT compilation, and I think it'd make a killer combination with our package manager's binary artifact deployment story.
In practice, it may often inline more aggressively as there aren't any boundaries that will prevent inlining.
If you write function `foo` calling `buz` from package `Buz.jl`, which calls `bar` from package `Bar.jl`, `bar` can be inlined into `foo`.
You can think of it as always have `-lto`, because the source of the Julia functions you're calling will always be available to the compiler and thus subject to (potentially aggressive) optimization.
https://julialang.github.io/PackageCompiler.jl/dev/devdocs/b...
https://docs.julialang.org/en/v1/manual/embedding/
If you can ensure your code and dependencies does not have dynamic things that need compilation at runtime, it is fairly easy to remove the dependency on bundling LLVM. Speed is the same, since you are then just running cached native code.
Everything is available on their YouTube channel:
https://www.youtube.com/playlist?list=PLP8iPy9hna6Tl2UHTrm4j...
But then for my applicationsspace there are problems:
- compilation seems possible now, but one has to be careful of GPL (like in the fft)
- no way to turn off GC (so byebye real-time possibilities).
- what about GUI design? Heard some mixed messages about it
While Julia does not offer semantic guarantees that you can avoid the GC, it;s actually quite possible and easy to write code where you're manually managing all your memory and the GC is never invoked.
> - what about GUI design? Heard some mixed messages about it
There's a lot of promising things happening with Makie.jl, Stipple.jl, Dash.jl, Pluto.jl and others. This space is still a little immature in Julia, but it's progressing fast.
But I like your question, lets call it 'which languages are all purpose production ready'; probably only C, C++, Java (there is a real-time version), LabView.
I'd say that if people have other reasons to use julia in a realtime low-latency production system -- like needing it's scientific stack -- then you can certainly make it work, you just need to do some testing to make sure you're not hitting GC.
Now, 8 years later, I'd say essentially everything there has been achieved. The most important difference between Julia now and Julia then is the gigantic, vibrant and interacting ecosystem that's sprung up (though share are other important technical differences too!)
_______________________________________________
Here's the pitch I'd give:
Julia's value proposition is mostly geared towards people trying to do things that are just too difficult, awkward or slow in other languages. It's not just about speed, dynamism and friendly syntax. It's also about solving the expression problem and providing unprecedented composability.
So while it may not necessarily have great appeal to 'end users' (who are people just calling numpy/scipy/pandas/whatever functions the way they were meant to be called by the library writers), it does appeal to people who make the sort of things end users want so the community is currently swelling with very talented people making state of the art research libraries. The language is designed for 'composability', so that it's very easy to combine packages together in ways the authors didn't necessarily intend or forsee.
This approach is really starting to bear fruit. Our package ecosystem has some real, state of the art stuff not easily available in other languages, and it's often built in less time and with less resources. For 'end users', whether or not Julia makes sense for them really just depends on if the things they need are already available.
Another huge advantage is that even if you're not some super researcher developer, Julia substantially lowers the bar for mere mortals such as myself to do package development. The composability I mentioned above means that I can stand on the shoulders of giants as I implement my own stuff, plus the fact that the majority of libraries are written in pure julia means that I can easily "peek under the hood" to figure out how they work and learn from them. This is really special and can't be replicated in python, because 'peeking under the hood' invariably means reading very specialized C code, rather than Python code.
I think Julia will someday eat Python for scientific computing, but it may be a while before that becomes obvious to Python users. In the meantime, we have an excellent package for calling and running Python code from withing Julia, called PyCall.jl [1], so that can make the burden of switching even easier.
[1] https://github.com/JuliaPy/PyCall.jl
So here is an example and my worries ... My project uses GDAL, PIL, SentinelSat libraries to collect Sentinel 1 imagery and process it, pandas, numpy, scipy extensively. Would I find a bunch of those modules missing? I assume so ...
I guess then I might be better of to trying a new project in Julia. Something not too easy but not too hard.
Worst case scenario, you might need to do the GDAL, PIL and SentinelSat stuff with PyCall and then the Pandas, Numpy and Scipy stuff should all be quite well supported from the Julia side. I can ask around to see if there's anyone who can answer this better though.
Julia more than covers all the functionality of those packages.
>PIL
https://github.com/JuliaImages/Images.jl
> GDAL
https://github.com/yeesian/ArchGDAL.jl
https://github.com/JuliaGeo
>SentinelSat
I'd use PyCall.jl for that. It's very easy and does automatic conversion of types etc
> I think Julia will someday eat Python for scientific computing, but it may be a while before that becomes obvious to Python users.
I can’t say I understand this rather pervasive attitude. If you have well tested code in a language which you can easily interface with, why bother rewriting it except for reasons of performance or compositionality? Or perhaps you mean that we’ll all be writing Julia even if some algorithms still sit in Python?
On the other hand Julia probably has a very strong long game, given that the Federal reserve bank uses it.
The containerization I used was singularity. I don't really love them much anymore, for various non-technical reasons, but it got the job done.
I mostly mean that I think Julia will someday have more mindshare in scientific computing than Python.
However, I'd say that the reason you see the "rewrite it in pure julia" attitude everywhere is that we enjoy a lot of benefits from having more and more things in pure julia, namely that our very powerful metaprogramming and code introspection tools can more easily 'get inside' Julia code, and that the compiler is able to perform interproceedural optimizations (IPO). IPO are not always important, but when you need them and don't have them, you really miss them, so at least for numerical kernels and such, it's really nice to have as much pure julia infrastructure as possible.
In the meantime though, as I daydream about a world where everything is written in Julia, there's nothing stopping me from being pragmatic and wrapping / bridging to and from other languages.
Julia's syntax and language design is brilliant, but the biggest reason Julia has been my favorite language since I first learned to program is how astonishingly easy it is to use tools other people have made, and also to create tools for others to use.
The packaging system is just great for development, both as a user, and as a maintainer. I've been surprised by how easy it is to take stuff I've made and get it to work with a myriad of packages for the purposes I need. Multiple dispatch does wonders here too.
In the recent survey they said that 30% of respondents had contributed to a package. 30%! It's a super active ecosystem and the barrier of entry is low enough where I don't feel "imposter syndrome" as strongly as I have elsewhere--that I need to pass some threshold in order to properly contribute.
It's a hard problem to combat, because fundamentally, package code must be able to do arbitrary things. One advantage julia has though is that packages tend to be written in pure julia, which makes the duty of checking and detecting malicious code much easier.
> lowers the bar for mere mortals such as myself to do package development
That's a powerful combination. I've been close to contributing to Python packages a few times, but the overhead is pretty high. At this point, I also feel like it would be a waste of effort not to write stuff in Julia instead.
> Julia will someday eat Python for scientific computing
It's well on the way. Julia users have a reputation for being evangelists. Heck, I made a convert even before I actually started using Julia in earnest: my labmate had a problem he couldn't solve in his favorite language, and I encouraged him to try Julia -- it turned out great!
I also think writing out math is much more concise in Julia, no more np.whatever(np.sin(x.detach()) + cos(np.sqrt(y)).mean().
Yes, far exceeding python actually. - https://github.com/jump-dev/JuMP.jl - https://github.com/JuliaNLSolvers
Are just some.
> graphing
Excellent support.
- https://github.com/JuliaPlots - https://github.com/JuliaPlots/Makie.jl
>Munging xml/Json etc
- https://github.com/quinnj/JSON3.jl - https://github.com/JuliaData - https://github.com/Algocircle/Cascadia.jl - https://github.com/queryverse/Query.jl
>oauth2 etc
- https://github.com/JuliaCloud - https://github.com/JuliaWeb/HTTP.jl
for anything not available yet, you can call it from python using PyCall.jjl
The question I am currently researching is not what Julia is good at but what are the pitfalls and drawbacks of particular technical choices. I know them by heart now for Python, and my team relies on knowing what doesn’t work well. For Julia it’s an unknown unknown. Understandably all the blogs are about why Julia is great so it makes seems to go slowly and coexist instead of moving.
Mind elaborating?
Julia's equivalent to xarray: https://github.com/JuliaArrays/AxisArrays.jl
Original issue explaining the problem: https://github.com/pydata/xarray/issues/525
Latest issue detailing the work: https://github.com/pydata/xarray/issues/3594
Edit: This looks like a fantastic resource! Full link: https://juliaacademy.com
For scientific computing and advanced machine learning domains, Julia is certainly a breath of fresh air and a no-brainer decision. However, as a general-purpose language as well as technology stack, I think Julia has still a long way to go. This is mostly due to relative immaturity of that part of the ecosystem, spotty - and sometimes even non-existent! - documentation (of course, except for the language itself and quite a limited number of core and popular packages) as well as some other issues, including tooling, development/compilation performance and limited pool of skilled developers.
So, from a startup founder perspective [who has to select the optimal, risk-minimizing, platform stack], despite Julia's many attractive features (including powerful meta-programming facilities - in my case, for potential DSL development), I'm now leaning toward Python and .NET ecosystems. Both offer very mature and large package ecosystems along with comprehensive tooling support and incomparably larger pool of skilled developers. Additionally, .NET offers stability of consistent development improvements, backing of a major corporation [no acquisition risk] along with support for modern enterprise-focused architectural practices and patterns, including DDD and multi-tenancy.
Not trolling. I don't really know why people like to shoehorn a purpose-oriented language into all domains. Eg: Julia for scientific computing, Rust for low level programming. They would be great if they focused on those "strong zones".
And Julia creators do focus on it's strong zone, with all the works on TPU, HPC and parallel computing, and so does the community with the stuff presented in juliacon like interactive reproducible notebooks (Pluto.jl) or data dashboards (Stipple.jl), both using the web domain to improve the scientific computing domain.
1. Intro section (italics emphasis mine).
"... delivering complex enterprise projects:
For many years I would agree with it, but after JuliaCon 2020 I believe we can confidently announce that 2. Section "Building microservices and applications". Even more so, practically all sections in the post, except just one ("Managing ML workflows"), describe various general-purpose and enterprise computing aspects.Therefore, I don't see how one could view the post and author's conclusion purely from scientific computing perspective.
And while right now I wouldn't recommend Julia for that web domain unless you also need it's number crunching features (which is more than just scientific computing, I work with large finance systems and there is definitely a lot of that stuff), I do think it's more of a library/community problem than a language problem. The language is well equipped to handle it (easy to use from a scripting perspective, easy parallelism, fast after warm-up, allows for clean abstractions to create sophisticated web frameworks, the mentioned easy FFI), what it lacks is the coverage and maturity of the tools and support (so I don't feel like I'm at a risk at any point that I need to do some integration, such as integrating with Kafka or any other systems, without having to write my own solution or integrating multiple languages at every step).
That's different from systems programming, embedded, game development and even GUI development (at least until smaller binaries with less warm-up) right now. Even with good library it will not be competitive with the languages that already claim those domains. Julia is a general purpose language already and more than a matlab substitute, it does not need to be good at everything (and it shouldn't try), but I don't think it should restrict the scope too much either.
People find the need to identify themselves as gopher or rustacean or something like that. Apparently Programmer is not hip anymore.
[1] https://github.com/davidavdav/NamedArrays.jl