I really like how simply Julia unifies many computational concepts. For example, MPI is a classic solution to some of the same problems, but it is only for coarse parallelism. OS-specific threading libraries are available for fine-grained parallelism, but the ligature between the two is typically messy.
I have heard about Go's parallelism being awesome, does anyone know of a comparison between Go's and Julia's models?
According to this[1], Julias threading/task model is in part inspired by Go. I don't know about any detailed analysis on similarities and/or differences though.
I’ve been playing with GPGPU on Julia lately also, and it really seems like things have come a long way in the last few years. Check out the Juliacon 2021 talk on GPU compute if you’re interested.
TLDR: The entire GPUArrays.jl test suite now passes with AMDGPU.jl. There are still some missing features and it is not as mature as the nvidia version, but this space is progressing rapidly, and benefited from the generic GPU compilation pipeline that was initially built for CUDA.jl
The gfx10 line (6800XT et al) probably work out of the box on a recent release. I think some are even officially supported. I test on a 5700XT which I don't think is officially supported. The change to 32 wide wavefronts took a while to resolve.
Rocm gets releases every few months or so. The llvm project part is mirrored to GitHub in real time.
Definitely not dead; Vega is well supported, and with some tweaks, Polaris probably works too (although it definitely was broken in HIP around ROCm 4.0.0 or so).
I think AMD has some work to do on non-C++/Python ecosystem engagement for sure, but they've built a foundation that's quite easy to build upon and get excellent performance and functionality; AMDGPU.jl is a testament to that.
> I have heard about Go's parallelism being awesome, does anyone know of a comparison between Go's and Julia's models?
There's nothing fundamentally special to Go's "parallelism".
It uses near-preempted (and now actually preempted I think, since a few versions back? before that they'd have implicit yield points at function calls but without function calls you could lock out the scheduler) userland threads with small stacks.
The parallelism comes from the m:n scheduling. Go has no constructs for massive parallelisation, where you hand it a loop and it efficiently runs that on a hundred cores, at least not built in.
Having lightweight userland threads fundamentally changes the way that you write concurrent programs. There is no need to worry about creating too many threads or of thread creation being too heavy. Old school patterns like process/thread/goroutine per connection are very natural to write in Go and result in dead simple code that performs well under load.
Reasoning about threads & basic blocking calls is a simpler mental model compared to async/await style concurrency in my opinion.
Go is fundamentally designed for developing backend applications, not high perf mathematical/scientific computing. Features like unrolling for-loops across hundreds of cores or offloading to GPU's would be out of place in the language.
> Having lightweight userland threads fundamentally changes the way that you write concurrent programs. There is no need to worry about creating too many threads or of thread creation being too heavy. Old school patterns like process/thread/goroutine per connection are very natural to write in Go and result in dead simple code that performs well under load.
Well... there's still a problem in the tens-of-millions of Goroutines.
But the issue is that OS-threads generally had issues at the ~100,000 of OS-threads. Making something "more lightweight" than pthreads makes sense, because 10,000,000 coroutines is a fundamentally different program design than 100,000 pthreads.
Sure, you can rewrite a program using lightweight threads to an async/await model and vice versa. I find the difference is in how easy it is to reason about concurrency in the program.
With threads I find I can more easily mentally organise which code in my application is executing in parallel.
In async/await land, you end up with 2 classes of functions async & non async functions, it's up to the callee to determine wether it is a blocking call or not. With threads you just block by default, and let the caller determine whether it's appropriate to block or execute the function on another thread. See https://journal.stuffwithstuff.com/2015/02/01/what-color-is-... for someone mere eloquent than myself.
This article uses very inefficient mathematical computations to illustrate concurrency. That’s fine, they are just examples. But you might also be interested in learning how it’s really done, for example by reading the Primes package. [1]
Yeah, as I wrote there, “As with all of the examples in the article, isprime() leaves many potential optimizations on the table in the interests of simplicity.”
The source you link to is longer than my entire article. But it is good to see an example of a “real” program.
For a job interview I did the same parallelization of a`isPrime` in Rust, and it the basically as simple as the one for Julia, replacing a call of `into_iter()` with `into_par_iter()`. I also show the strong scaling relationship, which i wish all things talking about concurrency did. Link for the curious: https://github.com/pcrumley/parallel_primes_rs
The Folds.jl package [1] mentioned in the article is very nicely written, IMHO.
For another alternative to Julia's built-in `Threads.@threads` macro, folks may also be interested in checking out `@batch` from Polyester.jl [2] (formerly CheapThreads.jl), which features particularly low-overhead threading.
Every time Julia pops up I can't help but get disappointed all over again. I was so intrigued by multiple aspects of the language, and even the type system seemed promising at first. However the inability to subtype concrete types is a total dealbreaker for me.
Everything else - even much of the type system - is extremely interesting. However the inability to subtype concrete types means you are essentially stuck with the unfortunately common inability to trivially specify how the heck your program is modeling your problem.
I just want the ability to specify an Apple_Count is not an Orange_Count without having to define my own damn operators. Why do so many languages make this so hard?
And Julia was so close, until one little sentence moved it so far away.
I don’t get it. Aren’t apple counts and orange counts both integers? Can you give an example of why you would want to subtype concrete types, and how this deficiency would prevent you from writing the kind of program that you want to write?
The approach might work but I don't think in general it actually solves the same problem. While I used integer-like things in the example, it wasn't actually important.
It could be strings (think Apple_Description and Orange_Description) or even more complex type from a library (maybe something like Apple_Throw_Plot and Orange_Throw_Plot).
Yes, I intended both to be integers - but in general they could be anything and the overall idea applies.
The overall idea is to be able to tell the language that an Apple_Count type represents an integer value and some extra meaning that is unique to the Apple_Count type. Additionally we specify that Apple_Counts interact with other Apple_Counts in the same way integers interact with integers, and that the 'unique meaning' of the type always passes through the operation unchanged (I'm sure there's some cool math term for this?).
Meanwhile we don't say how to mix together the 'extra unique meanings' that come with the _Count types, so any attempts to do so lets the compiler know to yell at us, because that's just nonsense according to the rules we provided.
This kind of perspective applies to every single variable in our programs - they all mean something* beyond their basest value to us, even if it's a string for a joke you haven't cleaned up yet. So my desire is to specify all the general kinds of meanings in my programs, and then be able to describe how all of those types of things can and cannot meaningfully interact when transforming the input into the output.
I've been awake and writing this for far too long and am just going off on wild and grandiose tangents, so I'll try to actually be brief. I've been totally convinced by this sort of approach. The problems uncovered are all very 'real' problems. All of them result from some basic incompatibility between the fundamental 'what' and 'how' of your program.
If you want more detail at how this all can (not must by any means, but can) work, I encourage you to look at how Ada handles type derivations (aka derived types). Do be aware Ada also uses the term 'subtype' but in a very different way. An Ada subtype, more or less, specifies a subset of values from a base type.
---
As far as Julia goes, it's been awhile since I looked into it all that much aside from a bit of a refresher this evening. The I believe the overall point is roughly correct, but I don't stand by the details, and neither should you.
Julia has at least a few issues that mean the kind of type safety I really desire is not trivially easy as the language currently exists - at least as far as I know. Unfortunately the ease-of-use of this kind of thing seems to be rather binary - it's either easy or it's hard. Partly I think that's because of how pervasive the concept is. Even a small amount of boiler plate balloons quite severely even for very simple programs.
However some of the problematic aspects of Julia seem like they might have existing solutions. For example the multiple dispatch mechanism is happy to mix and match any and all types, but type parameters provide a high degree of control over what methods are available to match in the first place. These features seem - to me anyway - to be powerful enough to have some serious potential* for making Julia one of the nicest languages to try to bolt all this typing onto after the fact. (Or maybe there's a little too much complexity to allow very nice solutions, and then the sheer number of type interactions drags Julia down to be one of the worst. Who knows!)
But subtypes of concrete types - or rather their lack - seems different. However I'm increasingly reluctant to say much more about this in any kind of useful detail. (Yes, the irony burns.) What was supposed to be a short look to confirm a few things has led me down a massive hole. The size of this hole seems to be increasing the deeper I go, and I think the expansion is accelerating. I've also passed by a few other, possibly equally sized holes along the way... I don't think I'm even that deep, and it's already pretty scary. Although I'm well past the point of firing on all cylinders, I don't think it matters down here, in the dark.
But anyway. My basic thoughts, some or all of wh...
One thing you might want to check out is https://juliapackages.com/p/unitful which is a package that allows you to add a unit system to numbers that will error if the units don't match in a way that makes sense.
with semantics borrowed from Smalltalk. I'm a bit rusty with Julia, please forgive any syntax errors.
I wonder how hard it would be to hack the compiler, catch method not found exceptions, and do that? It wouldn't be fast, but it would work as a proof of concept.
Edit: the semantics I have in mind go like this. If you call f(x, y), where x and y are Apple_Count, but there is no method defined for f(::Apple_Count, ::Apple_Count), then the compiler tries not_understood(f, x, y).
Speaking as someone who started in Java and moved to Julia, my experience has been that a lack of concrete subtyping has generally not made an impact on difficulty with respect to problem modeling (at least, not relative to Java or other languages with concrete subtyping). Moreover, some of the emergent properties of Julia’s type system (the “Tim Holy Trait Trick,” etc) enable problem modeling in different — and perhaps more “Julian” — ways, modeling by behavior rather than structure.
I spent far too many hours of my life writing a far too long response to another comment. Some... portion of it touches on some of this. I can't in good faith suggest reading it, but you could.
But the more specific and quicker version - I don't find Java is typed the way I want either. It seems to somehow find a way to be too much and not enough. I also don't do a lot of Java, so YMMV. Specifically I'm thinking of Ada when I start comparing type systems, if that helps.
I'd need more brainpower and probably Julia chops to give those a real shot. However I started to get a 'build your own type system' vibe from some of what I was seeing while nearing the end of that gargantuan post. This feels like some initial level of confirmation, so yay?
I'll give it a look later. I'm curious how far you can integrate it with external code, what sort of guarantees you get in practice, and how much it requires doing 'the right thing.'
Maybe in practice the way dispatch works ends up taking care of more problems than I'd expect, but also seeing the fairly widespread use of implicit conversions makes me a bit concerned. Again, maybe a thing sidestepped by other factors.
Do not be fooled, julia is not a oo language, despite some similarities. If you try to replicate oo patterns in Julia it's not going to work well, but in idiomatic julia I've never found the need for concrete subtyping: generally julia under emphasizes class hierarchies in favor of types as dispatch vessels and data holders. It's very different from OOP but when you get used to it it's quite nice. The things I miss from oop is the consistency (when there's just one way to model the world you don't have to think quite as much about design) and things like tab completion for method discovery.
I haven't seen it done yet, but in principle someone could probably write an editor plugin that gives some sort of autocomplete for method discovery in Julia based on `methodswith` or similar, which would be a nice thing to have!
I don't actually do much OO. I typically stick with Ada. It does have OO, but I rarely reach for it if I don't need it. Most of my typing interests are actually from the non-OO side.
Too much detail is around in a far too large response to another commenter, though I can't actually suggest it... But at least a bit of the start should provide a little more detail on what I'm looking for with types.
44 comments
[ 2.7 ms ] story [ 87.0 ms ] threadI have heard about Go's parallelism being awesome, does anyone know of a comparison between Go's and Julia's models?
[1] https://julialang.org/blog/2019/07/multithreading/
https://live.juliacon.org/talk/P8KJSW
TLDR: The entire GPUArrays.jl test suite now passes with AMDGPU.jl. There are still some missing features and it is not as mature as the nvidia version, but this space is progressing rapidly, and benefited from the generic GPU compilation pipeline that was initially built for CUDA.jl
The problem with AMD GPGPU is not software, it is that AMD literally does not care.
Rocm gets releases every few months or so. The llvm project part is mirrored to GitHub in real time.
I think AMD has some work to do on non-C++/Python ecosystem engagement for sure, but they've built a foundation that's quite easy to build upon and get excellent performance and functionality; AMDGPU.jl is a testament to that.
There's nothing fundamentally special to Go's "parallelism".
It uses near-preempted (and now actually preempted I think, since a few versions back? before that they'd have implicit yield points at function calls but without function calls you could lock out the scheduler) userland threads with small stacks.
The parallelism comes from the m:n scheduling. Go has no constructs for massive parallelisation, where you hand it a loop and it efficiently runs that on a hundred cores, at least not built in.
Reasoning about threads & basic blocking calls is a simpler mental model compared to async/await style concurrency in my opinion.
Go is fundamentally designed for developing backend applications, not high perf mathematical/scientific computing. Features like unrolling for-loops across hundreds of cores or offloading to GPU's would be out of place in the language.
Well... there's still a problem in the tens-of-millions of Goroutines.
But the issue is that OS-threads generally had issues at the ~100,000 of OS-threads. Making something "more lightweight" than pthreads makes sense, because 10,000,000 coroutines is a fundamentally different program design than 100,000 pthreads.
With threads I find I can more easily mentally organise which code in my application is executing in parallel.
In async/await land, you end up with 2 classes of functions async & non async functions, it's up to the callee to determine wether it is a blocking call or not. With threads you just block by default, and let the caller determine whether it's appropriate to block or execute the function on another thread. See https://journal.stuffwithstuff.com/2015/02/01/what-color-is-... for someone mere eloquent than myself.
[1] https://github.com/JuliaMath/Primes.jl/blob/master/src/Prime...
The source you link to is longer than my entire article. But it is good to see an example of a “real” program.
For another alternative to Julia's built-in `Threads.@threads` macro, folks may also be interested in checking out `@batch` from Polyester.jl [2] (formerly CheapThreads.jl), which features particularly low-overhead threading.
[1] https://github.com/JuliaFolds/Folds.jl
[2] https://github.com/JuliaSIMD/Polyester.jl
Everything else - even much of the type system - is extremely interesting. However the inability to subtype concrete types means you are essentially stuck with the unfortunately common inability to trivially specify how the heck your program is modeling your problem.
I just want the ability to specify an Apple_Count is not an Orange_Count without having to define my own damn operators. Why do so many languages make this so hard?
And Julia was so close, until one little sentence moved it so far away.
But Julia really does that already. Check out Unitful.jl
It could be strings (think Apple_Description and Orange_Description) or even more complex type from a library (maybe something like Apple_Throw_Plot and Orange_Throw_Plot).
The overall idea is to be able to tell the language that an Apple_Count type represents an integer value and some extra meaning that is unique to the Apple_Count type. Additionally we specify that Apple_Counts interact with other Apple_Counts in the same way integers interact with integers, and that the 'unique meaning' of the type always passes through the operation unchanged (I'm sure there's some cool math term for this?).
Meanwhile we don't say how to mix together the 'extra unique meanings' that come with the _Count types, so any attempts to do so lets the compiler know to yell at us, because that's just nonsense according to the rules we provided.
This kind of perspective applies to every single variable in our programs - they all mean something* beyond their basest value to us, even if it's a string for a joke you haven't cleaned up yet. So my desire is to specify all the general kinds of meanings in my programs, and then be able to describe how all of those types of things can and cannot meaningfully interact when transforming the input into the output.
I've been awake and writing this for far too long and am just going off on wild and grandiose tangents, so I'll try to actually be brief. I've been totally convinced by this sort of approach. The problems uncovered are all very 'real' problems. All of them result from some basic incompatibility between the fundamental 'what' and 'how' of your program.
If you want more detail at how this all can (not must by any means, but can) work, I encourage you to look at how Ada handles type derivations (aka derived types). Do be aware Ada also uses the term 'subtype' but in a very different way. An Ada subtype, more or less, specifies a subset of values from a base type.
---
As far as Julia goes, it's been awhile since I looked into it all that much aside from a bit of a refresher this evening. The I believe the overall point is roughly correct, but I don't stand by the details, and neither should you.
Julia has at least a few issues that mean the kind of type safety I really desire is not trivially easy as the language currently exists - at least as far as I know. Unfortunately the ease-of-use of this kind of thing seems to be rather binary - it's either easy or it's hard. Partly I think that's because of how pervasive the concept is. Even a small amount of boiler plate balloons quite severely even for very simple programs.
However some of the problematic aspects of Julia seem like they might have existing solutions. For example the multiple dispatch mechanism is happy to mix and match any and all types, but type parameters provide a high degree of control over what methods are available to match in the first place. These features seem - to me anyway - to be powerful enough to have some serious potential* for making Julia one of the nicest languages to try to bolt all this typing onto after the fact. (Or maybe there's a little too much complexity to allow very nice solutions, and then the sheer number of type interactions drags Julia down to be one of the worst. Who knows!)
But subtypes of concrete types - or rather their lack - seems different. However I'm increasingly reluctant to say much more about this in any kind of useful detail. (Yes, the irony burns.) What was supposed to be a short look to confirm a few things has led me down a massive hole. The size of this hole seems to be increasing the deeper I go, and I think the expansion is accelerating. I've also passed by a few other, possibly equally sized holes along the way... I don't think I'm even that deep, and it's already pretty scary. Although I'm well past the point of firing on all cylinders, I don't think it matters down here, in the dark.
But anyway. My basic thoughts, some or all of wh...
It includes an implementation of concrete subtyping.
I wonder how hard it would be to hack the compiler, catch method not found exceptions, and do that? It wouldn't be fast, but it would work as a proof of concept.
Edit: the semantics I have in mind go like this. If you call f(x, y), where x and y are Apple_Count, but there is no method defined for f(::Apple_Count, ::Apple_Count), then the compiler tries not_understood(f, x, y).
That said, if you absolutely need concrete subtyping in Julia, you can emulate it for structure with the tricks described in this (old but gold) Chris Rackauckas blog post: https://www.stochasticlifestyle.com/type-dispatch-design-pos...
But the more specific and quicker version - I don't find Java is typed the way I want either. It seems to somehow find a way to be too much and not enough. I also don't do a lot of Java, so YMMV. Specifically I'm thinking of Ada when I start comparing type systems, if that helps.
I'd need more brainpower and probably Julia chops to give those a real shot. However I started to get a 'build your own type system' vibe from some of what I was seeing while nearing the end of that gargantuan post. This feels like some initial level of confirmation, so yay?
I'll give it a look later. I'm curious how far you can integrate it with external code, what sort of guarantees you get in practice, and how much it requires doing 'the right thing.'
Maybe in practice the way dispatch works ends up taking care of more problems than I'd expect, but also seeing the fairly widespread use of implicit conversions makes me a bit concerned. Again, maybe a thing sidestepped by other factors.
Too much detail is around in a far too large response to another commenter, though I can't actually suggest it... But at least a bit of the start should provide a little more detail on what I'm looking for with types.
I don’t know anything about it myself, but I’ve heard that some people have had success setting up private package servers. I see there’s some discussion at https://discourse.julialang.org/t/pkg-private-registries-and...