I'm curious to hear from people who are using Julia on a regular basis. What's the UX like? I only looked at it briefly and was turned off by some things (1-based indexing comes to mind). But I really liked other things (multiple dispatch).
1-based indexing is strange to a programmer, but it's commonly used in mathematical computing software like Matlab. It's actually more convenient in a lot of cases, since mathematics usually uses this form of indexing.
As a primarily R and Python programmer, I've actually found that thinking in terms of 1-based indexing comes a lot more naturally to me than 0-based indexing, despite having worked exclusively in 0-based index languages for years before picking up R.
The only confusing edge case I've encountered is in biology, when manipulating a genomic region of zero length. Such a region is used, for example, when representing the location of an insertion of additional sequence. Before the insertion, the region has zero length. So, if the insertion occurs after the 500th element, then the zero-length range, in 1-based indexing, starts at 501 and ends at 500. However, it's rarely necessary to deal with this case directly, as such a range can be constructed as "starting at 501 with length 0".
1-based for statistics is a STRONG reason to use it. 0 based is ridiculous for a statistics program and the reason why I left Python. I understand for looping and the arras to be internally 0 but when doing statistics I should have to use two different index.
Mathematical notation is almost always 1 based. Let $v = 1 \dots n$, that sort of thing. So I find translating mathematics into code (which I seem to do quite frequently) much easier with 1 based indexing. (One exception being if you have a lot of modulo arithmetic.)
I've used both 0 based (python, C++), and 1 based (R, matlab, mathematica), but I really belive 1 based is the right choice for Julia. It isn't necessary, but it feels better.
My flippant answer: "No one is ever assigned Patient ID 0"
It's mostly about expectations. Users of statistical software, who may or may not be programmers by inclination, use 1-based indexing. The program they came from uses 1-based indexing (R, SAS, Stata, etc.). When you talk about data, you rarely talk about the 0th observation. It's just a recipe for errors to have to code switch between 0 and 1-based indexing.
can you give me an example? I did all sorts of numerical indexing programming, from fortran, matlab, python/numpy and R, and I am definitely more comfortable with 0 based indexing.
My quick philosophy/logic background can you make a strong arguement for why Julia/R (statistical programming) should use 0 based?
I am totally fine with zero base in Python and Lisp and such. BUT when I am doing statistics and the math is 1 based I think the ability to make a mistake is to big. Also for subset in R df[1, 1] would be the 2nd row and 2nd column would just throw most users of R.
As others have mentioned, numerical packages like Matlab use 1-based indexing. Most of us who have a numerics background are used to switching back and forth, so I'm fairly oblivious to it.
The 1-based indexing is a smaller deal than people make it out to be. If you're doing scientific computing, it's very unlikely you'll be doing too many fidgety things with indexes anyway. I switch between Julia and numpy with ease.
The REPL is on par with ipython (perhaps even better). Also are some IDEs for it but they aren't quite there yet.
TL;DR; This is a pretty big update. There are potentially breaking changes (e.g. errors when calling ambiguous methods). And there are significant performance improvements when dealing with functional-style programming (lambdas, closures, etc).
Errors when calling ambiguous methods are unlikely to be a big hazard – at least we haven't found them to be. Most ambiguous calls are very unlikely to actually occur, which is why that change makes a lot of sense.
The big breaking change is the array indexing one – that will very much require adjusting multidimensional array code.
Exciting to hear -- I have used Julia for prototyping, and have found it to be excellent for that:
In my experience there are still some rough edges as compared to the Python ecosystem (of course!), which together with the 0.x status make it impractical for many production situations. However it is fantastic for prototyping numerical code, the type system is a pleasure, and the JuMP mathematical optimization library is a gem. Being able to have fast code be "first class," as opposed to the impedance mismatch of dealing with numba/Cython, feels great and is a real boon for trying new things. Then, it is fairly straightforward to port the final solution to whatever production language you use (e.g., Python with a sprinkle of numba).
- Needing to interact with an existing codebase, and an existing developer base. If everyone knows and uses Python and only a few use Julia, it is too early to put Julia in production. If there are proprietary libraries, now may not be the best time to commit to porting them to Julia.
- Language and ecosystem stability. I started something on 0.4, and with 0.5 there were a raft of deprecations. If the code will live several years, that's a support commitment with unclear value.
- Library maturity. If I need to build a web app, read an Excel, read a CSV with dates quickly, consume a SOAP endpoint, etc etc in Python -- no problem. With Julia I will mostly be fine, but am likely to run into some cases that are not yet 100% there.
- Most code does not need the extra performance, so once you have a fast prototype as a performance target it is often not that hard to hit similar performance with Python + numba/Cython.
Note for that last point: there is a lot of value to not worrying about this in the exploratory stage, and getting a performance target (for later optimization) as a nice byproduct.
5 years ago I thought for sure I would be using Julia today. (I mostly use R) I have found that news tools have come into play that I haven't really felt the need for a faster language then R for my work.
I'm interested in people's everyday use of Julia and how it has impacted your workflow. I don't work with "Big Data" most of my data sets are bellow 100k in size. Anyone using Julia for medium and small data sets?
I thought the same thing as well. Unfortunately, there is still a bit of work to be done on the Julia side. The tidyverse ecosystem in R has relentlessly focused on ease of use and package compatibility, such that everything "fits" together in a logical way. I can't emphasize enough how much of a difference this makes in being productive with a language. Many of the things I do on a regular basis in R are actually rather mundane, and are possible to do in Julia. But it's just much more of a pleasure to do them in R.
Fortunately the DataFrames ecosystem and related packages (like Query.jl and StructuredQueries.jl) seem to be headed in the right direction.
As the maintainer of StructuredQueries.jl, I appreciate the recognition :) The JuliaStats community is indeed aiming for a more integrated ecosystem. It will take time, but I agree with you that our general direction is heartening.
Hadley verse in general. dplyr/tidyr and piping %>% has made things so seemless that I would hate to lose them for the ability to run faster. I swear I have forgotten 75% of what I learned in base R because these libraries (Outside of data.table) just work so well with one another.
RStudio has also been a God send for me. RMarkdown and now R Notebooks just amaze me. Then I work in a very MS Office environment that ReportRs (http://davidgohel.github.io/ReporteRs/) is the most under valued library right now.
I didn't have these things 5 years ago. (RStudio was initially released Feb 2011 but didn't start with it till 2013.
Not to be snarky, but we announced Julia in Feb 2012, which is less than 5 years ago. Unless you happened to be in the MIT class that used Julia in the Fall of 2011, you're probably overestimating how long you've been following the project.
If you're comfortable using R and the tooling and performance it offers, you should probably keep using it. Julia isn't just about big data, but it does tend to appeal to people who are struggling with problems that are sufficiently hard that their old tools left them in a world of pain. That pain may comes from size, complexity, CPU-intensiveness, or need for more language expressiveness (a hard thing to define). In particular, Julia offers a unique combination of productivity and speed for numerical work that can't be found anywhere else.
Okay 4.5 years :) I really was wowed by the promise of Julia and the road map you presented for it, great job by the way. I just am surprised I haven't picked it up yet personally. I have a feeling that my R ecosystem wouldn't be as strong as it is right now if it wasn't for the friendly Julia competition you have provided.
I guess I am seeking the one "pet project" that would get me to jump in and give Julia a test drive.
Thanks for all your work even though I don't directly benefit from your work.
I've used Julia for prototyping Monte Carlo and data assimilation algorithms, and also solving simple (1d) nonlinear PDEs (Fourier-spectral code with exponential RK timestepping -- not very complicated as these things go). It's also more or less completely replaced Matlab for me in terms of plotting and visualization (most of what I need is very simple). So far it has been very pleasant, so long as one remembers that breaking changes can and do occur between versions.
I generally find Julia to be a more expressive and fun language to program in than R or Python. It may be my background -- I've done a fair bit of work in Scheme, and in many ways Julia has lots of what I liked about Lisp in it. I don't need all the numerics and stats libraries that R or Python have; what little I need is easy to cobble together very quickly in Julia or has already been implemented. And the FFI is easy to use; calling C is pretty straightforward.
I taught myself Racket and that was the single biggest changer in terms of my programming in R and enjoyment of the language. After digging around I learned that scheme was a major influence in R's creation. Your comment now makes me excited to give Julia a spin.
I should add that one of the things I really like about Julia is having closures and higher-order functions, and also lambda. Unlike other Schemers switching to Julia, I don't miss not having tail recursion (if I was writing the kind of code that's more naturally recursive, I likely would have used Scheme / Lisp to start with). About the only thing I have a hard time wrapping my head around is writing macros in Julia -- CL-style macros are somehow a lot easier to think about. But this is likely to change with more experience. Also, I haven't needed Julia macros very oten.
Anyone using Julia for medium and small data sets?
I have used Julia with a 50GB dataset for feature extraction while I was waiting for R to process the same dataset (10 minutes to an hour depending on the function). I actually learned some Julia while waiting for R. For what I was doing, Julia felt roughly 100 times faster (most delays under a minute).
However, I did not know about dplyr at the time and I will certainly try it in the next project. Although, I am skeptical that using fast libraries is a silver bullet since, at some point, the data will pass through some slow custom R code that I wrote myself.
I prefer to use Julia - the API feels cleaner and more manageable, I don't suffer from subtle type errors, and there is also the possibility of running things in parallel from the same repl. I also had no memory leaks -- I suffered a lot from the bloat caused by my use of R closures.
That said, the power and output quality of ggplot means that I will probably stay with R for plotting. There will also be times when CRAN has the only library that does something we need.
dplyr is faster, BUT not that much faster. dplyr advantage is the way it works for programming. Really a lot easier and more consistent way to think about working with your data.
data.table is fast for your project I would imagine.
I'll be honest, giant factor speedups are awesome, and I always look forward to writeups about the discovery process and such, but I'm of the opinion that the presence of those optimizations in recent history means the software is often many years from actual maturity and suitability for production usage.
Either way, looking forward to reading about where the optimizations themselves came from.
WRT julia's type system: I still don't get it. Is Julia dynamically or statically typed? Are type errors detected at compile time (or since it's jitted when loading a file) or at runtime (i.e. when the code is executed)? I personally don't see much use in just another dynamically typed language that gives me similar errors as R does.
Can anybody explain the remark about LLVM towards the end of the post? What were the difficulties of updating it, especially with respect the intermediate versions?
I'm guessing that they use LLVM's C++ API, which is not stable between releases. The C++ API changes dramatically from one version to the next, so upgrading LLVM versions typically a big effort for downstream projects. The LLVM C API, however, is meant to be stable. For C++ users, it may make some sense to write a stable C++ API wrapping the LLVM C API.
Disclaimer: I am not affiliated with the LLVM project.
Hmm. It's hard to remember all the issues, but this was the period where LLVM moved from their "old JIT" backend to MCJIT [1]. IIRC, 3.4 broke the old JIT but MCJIT still wasn't developed enough for Julia to use. At some point the old JIT was removed entirely but MCJIT was still not usable (for us) and that didn't resolve itself entirely until 3.7.
If you're really curious, you can follow some of the breadcrumbs through the "Activate LLVM 3.7" GitHub issue (which was originally started for version 3.5).
LLVM can bet a very good optimized compiler toolchain, but is complex, hard to use and break easily (your code).
I have tried to use it several times for toy compilers, and each time is a pain for do simple things. I can't imagine how much pain is for depend your project on it.
Certainly worth the effort, but as expected of a C++ tech: You need a army of specialist to make it work.
The counter reports all heap-allocated memory, not just memory that persists after the function call, and the description in the post doesn't look quite right. The 208 bytes come almost entirely from calling the function from the global scope. Inside a function, there would only be 16 bytes of memory allocated on the heap, for the generator itself (which we desperately want to get rid of, but that is still WIP) and the return value would live in registers or on the stack.
Would that be covered by https://github.com/JuliaLang/julia/pull/18632 or is it a separate issue? I have some really ugly code that ought to be built out of nested generators, but I can't afford the millions of heap allocations.
That depends on what "like Julia" means. It's not a functional language, so the short answer is "no". Given that it lacks a true static type system and supports macros it's philosophically LISP-like; notably the Julia compiler includes a sizeable chunk of LISP code. But it's also clearly not a LISP.
I realize that your use of the phrase "functional language" seems to entail having a static type system and purity, but most people seem to consider Lisp to be the original functional language and it doesn't have either property. If one considers Lisp to be functional then it seems like Julia should be as well. Do you not consider Lisp to be functional? Real question, not a rhetorical one – I'm trying to assess where people fall on this terminology.
"Functional language" is ill-defined, hence my specific question about static typing and purity. I would really like to have a language for scientific computing that had good scalable performance as well as static types and referential transparency.
As far as I know, no such system exists. Strict static typing and numerical computing are, I suspect, more at odds with each other than they casually appear to be. The seemingly simple `+` operator is the most polymorphic and overloaded function-like entity in many static languages (and often can't be defined as a normal function for that very reason). And `+` doesn't hold a candle to the polymorphism of array indexing and slicing, which is pretty crucial for productive numerical work.
Julia is very strongly typed, much more so than most dynamic languages. Because the type inference can remove type checks in performance-sensitive code, the language and standard library are much more willing to throw type checks around all over the place, which means that errors get caught much closer to where they were caused compared to most dynamic languages.
julia> xs = ["foo", "bar"]
2-element Array{String,1}:
"foo"
"bar"
julia> push!(xs, 42)
ERROR: MethodError: Cannot `convert` an object of type Int64 to an object of type String
This may have arisen from a call to the constructor String(...),
since type constructors fall back to convert methods.
in push!(::Array{String,1}, ::Int64) at ./array.jl:479
julia> xs = Any["foo", "bar"]
2-element Array{Any,1}:
"foo"
"bar"
julia> push!(xs, 42)
3-element Array{Any,1}:
"foo"
"bar"
42
If what you are looking for is a statically typed language, then Rust has a very similar approach to data, types and polymorphism but with much more emphasis on catching errors at compile-time, at the cost of poor interactive development.
Nim is statically typed, very similar to python syntax-wise, fast and with trivial c interop. This makes it a good target for scientific programming, although unfortunately there are not many libraries yet http://rnduja.github.io/2015/10/21/scientific-nim/
As for being functional, strictly speaking it is not, but still has some features like closures and higher order functions, immutable variables, the usual suspects (map, filter, ...) can be inlined with zero overhead, side effect tracking and so on (in addition to a very good macro system)
Congratulations! As a Rust user, I'm always happy to see new programming languages break into the field. :) However I am confused by this paragraph:
> Julia’s LLVM version was upgraded from 3.3 to 3.7.1.
> [...] we’re very happy to be back to using current
> versions of our favorite compiler framework.
LLVM's current version is 3.9, is this a typo or are there problems that prevent Julia from being used with any release newer than 3.7.1?
LLVM 3.7.1 is a little less than 10 months old. Our RC period lasts for about 3 months, the start of which is the last point we'd have possibly upgraded LLVM. Upgrading LLVM is a little bit of a scary prospect for us since we find new bugs in every version (and fix them!), so though we considered moving to 3.8, 3.7 had been stable for a few months, so we didn't want to risk it. There's currently a PR to bump LLVM to 3.9 for 0.6, so I'd expect 0.6 to be released with LLVM 3.9.1.
Cool, mostly I was curious if the show-stopping issues you were seeing were actually fixed for good or whether 3.7.1 just happened to "accidentally" work. :)
63 comments
[ 3.0 ms ] story [ 120 ms ] threadThe only confusing edge case I've encountered is in biology, when manipulating a genomic region of zero length. Such a region is used, for example, when representing the location of an insertion of additional sequence. Before the insertion, the region has zero length. So, if the insertion occurs after the 500th element, then the zero-length range, in 1-based indexing, starts at 501 and ends at 500. However, it's rarely necessary to deal with this case directly, as such a range can be constructed as "starting at 501 with length 0".
Could you elaborate on that? What about statistics do you feel makes it necessary to have 1-based indexing?
I've used both 0 based (python, C++), and 1 based (R, matlab, mathematica), but I really belive 1 based is the right choice for Julia. It isn't necessary, but it feels better.
It's mostly about expectations. Users of statistical software, who may or may not be programmers by inclination, use 1-based indexing. The program they came from uses 1-based indexing (R, SAS, Stata, etc.). When you talk about data, you rarely talk about the 0th observation. It's just a recipe for errors to have to code switch between 0 and 1-based indexing.
I am totally fine with zero base in Python and Lisp and such. BUT when I am doing statistics and the math is 1 based I think the ability to make a mistake is to big. Also for subset in R df[1, 1] would be the 2nd row and 2nd column would just throw most users of R.
The REPL is on par with ipython (perhaps even better). Also are some IDEs for it but they aren't quite there yet.
The big breaking change is the array indexing one – that will very much require adjusting multidimensional array code.
In my experience there are still some rough edges as compared to the Python ecosystem (of course!), which together with the 0.x status make it impractical for many production situations. However it is fantastic for prototyping numerical code, the type system is a pleasure, and the JuMP mathematical optimization library is a gem. Being able to have fast code be "first class," as opposed to the impedance mismatch of dealing with numba/Cython, feels great and is a real boon for trying new things. Then, it is fairly straightforward to port the final solution to whatever production language you use (e.g., Python with a sprinkle of numba).
- Needing to interact with an existing codebase, and an existing developer base. If everyone knows and uses Python and only a few use Julia, it is too early to put Julia in production. If there are proprietary libraries, now may not be the best time to commit to porting them to Julia.
- Language and ecosystem stability. I started something on 0.4, and with 0.5 there were a raft of deprecations. If the code will live several years, that's a support commitment with unclear value.
- Library maturity. If I need to build a web app, read an Excel, read a CSV with dates quickly, consume a SOAP endpoint, etc etc in Python -- no problem. With Julia I will mostly be fine, but am likely to run into some cases that are not yet 100% there.
- Most code does not need the extra performance, so once you have a fast prototype as a performance target it is often not that hard to hit similar performance with Python + numba/Cython.
Note for that last point: there is a lot of value to not worrying about this in the exploratory stage, and getting a performance target (for later optimization) as a nice byproduct.
I'm interested in people's everyday use of Julia and how it has impacted your workflow. I don't work with "Big Data" most of my data sets are bellow 100k in size. Anyone using Julia for medium and small data sets?
Fortunately the DataFrames ecosystem and related packages (like Query.jl and StructuredQueries.jl) seem to be headed in the right direction.
Could you elaborate? Which new tools?
RStudio has also been a God send for me. RMarkdown and now R Notebooks just amaze me. Then I work in a very MS Office environment that ReportRs (http://davidgohel.github.io/ReporteRs/) is the most under valued library right now.
I didn't have these things 5 years ago. (RStudio was initially released Feb 2011 but didn't start with it till 2013.
If you're comfortable using R and the tooling and performance it offers, you should probably keep using it. Julia isn't just about big data, but it does tend to appeal to people who are struggling with problems that are sufficiently hard that their old tools left them in a world of pain. That pain may comes from size, complexity, CPU-intensiveness, or need for more language expressiveness (a hard thing to define). In particular, Julia offers a unique combination of productivity and speed for numerical work that can't be found anywhere else.
I guess I am seeking the one "pet project" that would get me to jump in and give Julia a test drive.
Thanks for all your work even though I don't directly benefit from your work.
I generally find Julia to be a more expressive and fun language to program in than R or Python. It may be my background -- I've done a fair bit of work in Scheme, and in many ways Julia has lots of what I liked about Lisp in it. I don't need all the numerics and stats libraries that R or Python have; what little I need is easy to cobble together very quickly in Julia or has already been implemented. And the FFI is easy to use; calling C is pretty straightforward.
I have used Julia with a 50GB dataset for feature extraction while I was waiting for R to process the same dataset (10 minutes to an hour depending on the function). I actually learned some Julia while waiting for R. For what I was doing, Julia felt roughly 100 times faster (most delays under a minute).
However, I did not know about dplyr at the time and I will certainly try it in the next project. Although, I am skeptical that using fast libraries is a silver bullet since, at some point, the data will pass through some slow custom R code that I wrote myself.
I prefer to use Julia - the API feels cleaner and more manageable, I don't suffer from subtle type errors, and there is also the possibility of running things in parallel from the same repl. I also had no memory leaks -- I suffered a lot from the bloat caused by my use of R closures.
That said, the power and output quality of ggplot means that I will probably stay with R for plotting. There will also be times when CRAN has the only library that does something we need.
data.table is fast for your project I would imagine.
Either way, looking forward to reading about where the optimizations themselves came from.
Julia uses its type system for:
- self-documentation
- reduction of boilerplate manual type checking that litters libraries in dynamic languages
- all those times you need to express the type of something, which happens especially frequently in numerical code
- performance
In the future, some "type linting" could be built into the standard library since we can infer types for so much code, but it isn't the top priority.
Disclaimer: I am not affiliated with the LLVM project.
[1] http://llvm.org/docs/MCJITDesignAndImplementation.html
https://github.com/JuliaLang/julia/issues/9336
I have tried to use it several times for toy compilers, and each time is a pain for do simple things. I can't imagine how much pain is for depend your project on it.
Certainly worth the effort, but as expected of a C++ tech: You need a army of specialist to make it work.
Would that be covered by https://github.com/JuliaLang/julia/pull/18632 or is it a separate issue? I have some really ugly code that ought to be built out of nested generators, but I can't afford the millions of heap allocations.
addendum: read 'static' for 'strong', please
Someday, Haskell will get a good matrix library, and then I'll be happy.
Although I'm not sure if it's dead or just pining for the fjords
As for being functional, strictly speaking it is not, but still has some features like closures and higher order functions, immutable variables, the usual suspects (map, filter, ...) can be inlined with zero overhead, side effect tracking and so on (in addition to a very good macro system)
if I have a function:
``` function addone{x <: Real}(x::T) x + 1 end ```
Wouldn't it be natural that `addone.` is dispatched when I call `addone([2 3])`?