Some GCs need to reach a safepoint before doing their work; and when a classic Thread is chugging along in a tight loop, that thread is blocking the whole system for a pending GC.
One hacky fix is to provide an opportunity for a safepoint (System.out.print every 10k iterations). Other tools like JVM command line options allow for observing safepoint triggering.
The same goes for virtual threads: inserting a Thread.sleep(1L) every nth iteration of a tight loop does it.
Also there was talk specifying a custom scheduler for the virtual threads. That would not help in inserting switching opportunities, but it would give a way to define what kind of fairness you want.
Seconded. Although I think Thread::yield should do what it says on the tin in the case of lightweight threads and it's hard to understand why they chose to not make it so, I don't care much. If the only problem I have to anticipate to get the full benefit of a well behaved lightweight threat scheduling is strategically sprinkling yield points inside those few tight loops that might actually threaten to hog a core then it's a net win.
It's neither. The frequency of non-blocking CPU bound sections of most applications is low and so the need to introduce scheduling hints is also infrequent. Aside from an endless, non-blocking loop that never yields omitting possibly beneficial yield points isn't an error; scheduling will not be ideal but the process will ultimately produce the same result.
Thinking about this further, a pragmatic enhancement to Thread::yield might look like this:
enum Hint {
ALWAYS,
FREQUENT,
SOMETIMES,
INFREQUENT,
NEVER
}
Thread::yield(Hint);
The method could be intrinsic to the compiler+runtime to provide runtime optimization (loop unrolling, tuning, etc.) of the 2nd - 4th cases. This would neatly avoid the need to:
long n = 0;
while (true) {
// stuff
if (++n > SOME_LIMIT) {
n = 0;
// yield here
}
}
Seems like safepoints are an ideal way to do this. It sounds similar in ways to how the BEAM runtime for Elixir/Erlang will count reductions and preempts a lightweight process. I guess the interesting question is whether the same safepoint testing mechanism can be reused directly or whether some more complex logic is required on the critical path
I only observe Java from a distance but everything I've read about project loom is amazing.
Speaking as someone who has worked intricately on an async runtime in another language, I've more and more started to question the premises around the explicitly async paradigm. Lots of things we were "promised" with async turned out to be just as complicated in the threaded (i.e. blocking) programming models, and now we have two incompatible programming models. I'm now of the opinion that we should improve (instead of replace) our threaded models and do under-the-hood optimizations to sort out the performance issues.
If I had a dime for everytime someone made a blocking call from an async function.. And who can blame them? Most blocking functions aren't even documented as such, you just have to know.
Yes, I think Loom is an amazing step forward and it's brilliant to just change the blocking methods and thus existing code might already benefit with minor changes.
Yeah, I've used a bunch of non-blocking paradigms over the years and all become painful quickly once the system becomes non-trivial, except for green threads. I was very disappointed that Rust dropped this feature in favour of async.
It makes sense for Rust. Rust with a significant runtime wouldn't provide much value over Go / Kotlin / Java / C#, whereas Rust without a heavy runtime is the first legitimate competitor to the dominance of C / C++.
At least Java and C# have a problem of being 32-bit. Most of the standard collections and interfaces use i32 for sizes, offsets, etc. So large parts of the existing codebases can't be directly reused to manipulate data in sizes, that are not so "big" anymore.
I'd like to see how those platforms will be able to get past that legacy.
I think you're both right. Rust with green threads would have been incompatible with their goals for embedded. Otoh, if the runtime was pluggable at compile time, you could retain that choice while catering to the majority user base (i.e. the ones currently using Tokio).
In order to do that, you'd have to have an abstraction over the syscalls. I'm not entirely sure this is feasible with all the ins and outs of Rust, but seeing the success of Loom, I'm at least optimistic.
Rust makes it very easy to use plain old OS threads, which work very well for many, dare I say _most_ use cases. OS threads are a lot lighter and cheaper than many realize, and async has it's own hidden costs. In the cases that OS threads don't cut it, a general purpose async runtime isn't likely to either. Green threading libraries are also possible without language integration [0], and though they require custom I/O types they have a synchronous API, which makes generic abstractions easier. Now of course if you want the async coding model with all it's benefits (selection, cancellation), then there is really no escaping it, but it's certainly not the only option.
It definitely can work, but Rust's model is incredibly powerful, and can make it easy to model complex control flow. Certain patterns, such as cancellation, can get pretty hairy in Go, although this is a place where Rust has some work to do as well :).
Yeah, but those complex control flows are not free. Drop-to-cancel is a rather powerful foot gun in many cases. Basically you throw out invariants that most people takes for granted, e.g. if a function is started it will complete before exiting the caller's scope.
Regarding cancelation, I think Go got it near perfect with contexts, modulo the explicitness of it. What don't you like about it?
The fact that explicit async turned out to be more complicated than expected is precisely why I'm skeptical of loom. When the magic works perfectly it'll be great. When it doesn't...
> If I had a dime for everytime someone made a blocking call from an async function.. And who can blame them? Most blocking functions aren't even documented as such, you just have to know.
Agreed. I'd like to see blocking represented in the type system the same way we do for async, so that you can't mix and match the two unless you deliberately choose to.
Function colouring has always been a good idea, as long as you have a language where you can be polymorphic over it where appropriate (i.e. one with higher-kinded types).
Thank you, I stand corrected. Will have to check the linked paper in depth later, but is IO async only or is CPU calculations made automatically concurrent? Last time I checked I don’t remember it being the case for the latter without explicitly requesting it.
CPU bound calculations are not parallelized automatically. There are some operators (a.k.a strategies) to do that declaratively to pure code and there are monadic libraries to parallelize Haskell code.
I'm not 100% sure, actually, but IIRC there is an automatic insertion of 'yield points' into ordinary chunks of Haskell code, so, yes, it will yield automatically even if CPU-bound. Regardless, it's never been a problem in practice.
Doing this on the JVM is not possible without actually modifying the running bytecode, so that's why they have that caveat for Loom.
That’s the point of loom. You doesn’t need to know whether this something blocks or not, the JVM does know it so it can do the correct thing. Given that the JVM ecosystem is very very pure and barely uses any native calls (where it can’t know about blocking), this will pretty much work automagically everywhere.
My understanding of Loom is, that it is loosely inspired/similar to GHC's runtime. In both every blocking call into a system call is replaced with an equivalent asynchronous system call and some internal thread will manage all those things to wait for. While the operating system is processing the asynchronous request the lightweight thread which made a seamingly blocking call is put aside and some other lightweight thread can execute on the same OS/level thread.
Since the JVM comes with all the blocking call in its own standard libraries, it can rewrite them all internally. Every third-party library that uses the normal Java standard library will benefit from this under the hood replacement of blocking systems call with non-blocking system calls. Only when you use other native libraries you have to be careful to use mostly non-blocking functions.
Therefore Loom allows programmers to think and program with blocking calls, but they get the performance characteristics of non-blocking calls. That is awesome. And there is no need to use function colouring or type-level encoding of blocking/non-blocking code.
From .NET and C++ point of view, async/await, turns out to be a pain because unless the complete call stack is written to be async, we need to create async bubbles to be able to call into those functions.
Microsoft tried to make everyone program the right way in WinRT and it was one of the reasons why many developers hate it (among the other reasons vs Win32), because you get async everything.
> If I had a dime for everytime someone made a blocking call from an async function.. And who can blame them? Most blocking functions aren't even documented as such, you just have to know.
This is apt timing. I wrote a thought recently of chunking loops and yielding after a certain number of chunks.
The Loom scheduler does NOT guarantee resource starvation avoidance for high CPU usage. You need to Thread.yield.
I also thought of creating a mapping from synchronous code and rewriting it to a tree of LMAX disruptors. I ported a wait-free ringbuffer from C++ to Java today - it was written by Alexander Krizhanovsky[1]
In LMAX disruptor you split each IO request in half - you have a disruptor that enqueuing events events to request the IO and to pass the event on to a callback you enqueue the response on a different disruptor.
If a synchronous request handler has 27 lines it represents a tree of 27 disruptor threads independently scaled.
We can pipeline the request with 27 threads all in an event loop that pipeline each task × core count and without blocking the servicing thread or other work. So event loops without blocking! So CPU heavy tasks do not block other heavy CPU tasks and do not block the servicing thread and are not blocked by IO.
The "State of Loom" document from May 2020 discusses this sort of problem. Long-term I believe the solution is to allow preempting at any safe-point and custom schedulers, but neither of those will be delivered in the first preview.
A safepoint in the JVM is a state where all the objects on the heap are consistent and the GC roots are known. These safe points are important for the GC to able to safely work.
A safepoint poll points are points in a threads execution where it's able to go into a safepoint and transfer control to the JVM so it polls to see if it should.
No, safepoints are added to your code by the JVM automatically.
When bytecode is being interpreted, every bytecode instruction has a safepoint after it. In compiled code polls are scattered thoughout the code, for example on method entries and loop back edges (so the comment above about needing to add code in tight loops to help the GC is wrong).
Can anyone comment on the duebugability of Loom (e.g. how do stack traces work)? Can Loom threads jump between native threads? Is it easy to understand what is going on when I run all this under a debugger? Having worked with some user level threading systems in the past as part of some libos work the the fact the tooling wasn't really set up to handle such issues made it really difficult to understand what was going on when we ran into problems.
I can’t answer everything, but good debugging is a very important design goal of Loom, so the usual debug methods should work the exact same way they do with native threads.
Virtual threads are normally run on a fork join pool or other executor service, and can run on any OS thread in that pool. They are not tied to any single OS thread.
Yes, it is easy to understand in a debugger as support has been added to the debugger interface and to JVMTI, along with some
Useful JFR events. We hope that in the future structured concurrency will make it even easier to understand huge numbers of virtual threads in production.
Right but the problem we had was that when you fired up the debugger (GDB) you could only see the stack traces of the currently active user level thread on each host thread. To understand what was going on in other user threads (e.g. who is holding what locks) you had to manually go into the scheduler data structures to find the descheduled threads etc, which was extremely tedious. If that all happens out of the box for Loom then that is great.
51 comments
[ 2.6 ms ] story [ 122 ms ] threadSome GCs need to reach a safepoint before doing their work; and when a classic Thread is chugging along in a tight loop, that thread is blocking the whole system for a pending GC.
One hacky fix is to provide an opportunity for a safepoint (System.out.print every 10k iterations). Other tools like JVM command line options allow for observing safepoint triggering.
The same goes for virtual threads: inserting a Thread.sleep(1L) every nth iteration of a tight loop does it.
Also there was talk specifying a custom scheduler for the virtual threads. That would not help in inserting switching opportunities, but it would give a way to define what kind of fairness you want.
It's neither. The frequency of non-blocking CPU bound sections of most applications is low and so the need to introduce scheduling hints is also infrequent. Aside from an endless, non-blocking loop that never yields omitting possibly beneficial yield points isn't an error; scheduling will not be ideal but the process will ultimately produce the same result.
Speaking as someone who has worked intricately on an async runtime in another language, I've more and more started to question the premises around the explicitly async paradigm. Lots of things we were "promised" with async turned out to be just as complicated in the threaded (i.e. blocking) programming models, and now we have two incompatible programming models. I'm now of the opinion that we should improve (instead of replace) our threaded models and do under-the-hood optimizations to sort out the performance issues.
If I had a dime for everytime someone made a blocking call from an async function.. And who can blame them? Most blocking functions aren't even documented as such, you just have to know.
I'd like to see how those platforms will be able to get past that legacy.
This one has been open for 3 years now (and it is probably not the first one), and the completion is nowhere in sight.
It also only targets runtime, but does not yet solve the library compatibility issue.
In order to do that, you'd have to have an abstraction over the syscalls. I'm not entirely sure this is feasible with all the ins and outs of Rust, but seeing the success of Loom, I'm at least optimistic.
Isn't Go proof that this works just fine, arguably even better, with the blocking model?
Regarding cancelation, I think Go got it near perfect with contexts, modulo the explicitness of it. What don't you like about it?
> If I had a dime for everytime someone made a blocking call from an async function.. And who can blame them? Most blocking functions aren't even documented as such, you just have to know.
Agreed. I'd like to see blocking represented in the type system the same way we do for async, so that you can't mix and match the two unless you deliberately choose to.
Function coloring as a feature?
[1]: Runtime Support for Multicore Haskell, Simon Marlow, Simon Peyton Jones, Satnam Singh (https://www.cs.tufts.edu/~nr/cs257/archive/simon-peyton-jone...)
Doing this on the JVM is not possible without actually modifying the running bytecode, so that's why they have that caveat for Loom.
Since the JVM comes with all the blocking call in its own standard libraries, it can rewrite them all internally. Every third-party library that uses the normal Java standard library will benefit from this under the hood replacement of blocking systems call with non-blocking system calls. Only when you use other native libraries you have to be careful to use mostly non-blocking functions.
Therefore Loom allows programmers to think and program with blocking calls, but they get the performance characteristics of non-blocking calls. That is awesome. And there is no need to use function colouring or type-level encoding of blocking/non-blocking code.
Microsoft tried to make everyone program the right way in WinRT and it was one of the reasons why many developers hate it (among the other reasons vs Win32), because you get async everything.
Isn't that exactly what loom aims to resolve?
The Loom scheduler does NOT guarantee resource starvation avoidance for high CPU usage. You need to Thread.yield.
I also thought of creating a mapping from synchronous code and rewriting it to a tree of LMAX disruptors. I ported a wait-free ringbuffer from C++ to Java today - it was written by Alexander Krizhanovsky[1]
In LMAX disruptor you split each IO request in half - you have a disruptor that enqueuing events events to request the IO and to pass the event on to a callback you enqueue the response on a different disruptor.
If a synchronous request handler has 27 lines it represents a tree of 27 disruptor threads independently scaled.
We can pipeline the request with 27 threads all in an event loop that pipeline each task × core count and without blocking the servicing thread or other work. So event loops without blocking! So CPU heavy tasks do not block other heavy CPU tasks and do not block the servicing thread and are not blocked by IO.
https://github.com/samsquire/ideas4#51-rewrite-synchronous-c...
[1]: https://www.linuxjournal.com/content/lock-free-multi-produce...
Currently you have to Thread.sleep(1) to force Loom to unmount the OS thread in a CPU-bound green thread.
https://cr.openjdk.java.net/~rpressler/loom/loom/sol1_part2....
> We can forcefully preempt virtual threads at any safepoint poll point
What is a "safepoint poll point"? A sleep call?
A safepoint poll points are points in a threads execution where it's able to go into a safepoint and transfer control to the JVM so it polls to see if it should.
https://openjdk.java.net/groups/hotspot/docs/HotSpotGlossary...
https://shipilev.net/jvm/anatomy-quarks/22-safepoint-polls/
When bytecode is being interpreted, every bytecode instruction has a safepoint after it. In compiled code polls are scattered thoughout the code, for example on method entries and loop back edges (so the comment above about needing to add code in tight loops to help the GC is wrong).
This is something where the answer is "it depends". Hotspot will omit the safe point poll in counted loops since it knows those loops will terminate.
Virtual threads are normally run on a fork join pool or other executor service, and can run on any OS thread in that pool. They are not tied to any single OS thread.
Yes, it is easy to understand in a debugger as support has been added to the debugger interface and to JVMTI, along with some Useful JFR events. We hope that in the future structured concurrency will make it even easier to understand huge numbers of virtual threads in production.