You're using Go's race detector wrong if you expect it to actually catch all races. It doesn't, it can't, it's a best effort thing.
The right way to use the go race detector is:
1. Only turn it on in testing. It's too slow to run in prod to be worth it, so only in testing. If your testing does not cover a use-case, tough luck, you won't catch the race until it breaks prod.
2. Have a nightly job that runs unit and integ tests, built with -race, and without caching, and if any races show up there, save the trace and hunt for them. It only works probabilistically for almost all significant real-world code, so you have to keep running it periodically.
3. Accept that you'll have, for any decently sized go project, a chunk of mysterious data-races. The upstream go project has em, most of google's go code has em, you will to. Run your code under a process manager to restart it when it crashes. If your code runs on user's devices, gaslight your users into thinking their ram or processor might be faulty so you don't have to debug races.
4. Rewrite your code in rust, and get something better than the go race detector every time you compile.
The most important of those is 3. If you don't do anything else, do 3 (i.e. run your go code under systemd or k8s with 'restart=always').
I'm so glad to be out of the dark ages of parallelism. Complaining about Go's race detector or exactly which types of logical races Rust can't prevent is such a breath of fresh air compared to all those other single-core languages we're paid to write with that had threading, async, or concurrency bolted-on as an afterthought.
I can only hope Go and Rust continue to improve until the next language generation comes along to surpass them. I honestly can't wait, things improved so much already.
That is typical Go design school, even the channels stuff, we already had that in Java and .NET ecosystem, even if the languages don't have syntax sugar for launching co-routines.
But go-routines!
Well, on .NET land we would be using Task Processing Library, or Dataflow built on top of it, with tasks being switched over the various carrier threads.
Or if feeling fancy, reach out to async workflows on F# with computation expressions, even before async/await came to be.
While on the Java side, we would be using java.util.concurrent, with future computations, having fun with Groovy GPars, or Scala frameworks like Akka.
In both platforms, we could even go the extra mile and write our own scheduling algorithms, how those lightweight threads would be mapped into carrier threads.
Naturally not having some of the boilerplate to handle all of that, or using multiple languages on the same project, makes it easier, hence why now we have all those goodies, with async/await or virtual threads on top.
I always run my Go code with `-race`, but I feel more comfortable writing C++ multithreaded code than Go thanks to the thread sanitizer annotations ( `__attribute__((guarded_by(guard)))` and others in the family).
The annotation also help me discover patterns, like when most of the functions of a class have the same annotations, maybe it means that all the functions of the class should have the same annotations.
I really wish an equivalent to those annotations came to Go.
In all honesty, if you “do work” using channels then all your goroutines are “thread safe” as the channel keeps things in order. Also, mutex is working as intended. As you see in your post, -race sees this, it’s good. Now have one goroutine read from a chan, get rid of the mutex, all other goroutines write to the chan, perfection.
The race detector has always only worked at run time, and is documented to detect concurrent memory accesses. This means the memory has to actually be accessed in order for it to see the race condition. It does not do static analysis.
> Because of its design, the race detector can detect race conditions only when they are actually triggered by running code, which means it’s important to run race-enabled binaries under realistic workloads.
This isn't a mutex blind spot. It is a side effect of goroutine/thread scheduling, which will obviously be based on workload and other factors. There are a bunch of other cases, that are not mutex related, that it won't see unless execution actually triggers them.
7 comments
[ 2.8 ms ] story [ 25.4 ms ] threadThe right way to use the go race detector is:
1. Only turn it on in testing. It's too slow to run in prod to be worth it, so only in testing. If your testing does not cover a use-case, tough luck, you won't catch the race until it breaks prod.
2. Have a nightly job that runs unit and integ tests, built with -race, and without caching, and if any races show up there, save the trace and hunt for them. It only works probabilistically for almost all significant real-world code, so you have to keep running it periodically.
3. Accept that you'll have, for any decently sized go project, a chunk of mysterious data-races. The upstream go project has em, most of google's go code has em, you will to. Run your code under a process manager to restart it when it crashes. If your code runs on user's devices, gaslight your users into thinking their ram or processor might be faulty so you don't have to debug races.
4. Rewrite your code in rust, and get something better than the go race detector every time you compile.
The most important of those is 3. If you don't do anything else, do 3 (i.e. run your go code under systemd or k8s with 'restart=always').
I can only hope Go and Rust continue to improve until the next language generation comes along to surpass them. I honestly can't wait, things improved so much already.
But go-routines!
Well, on .NET land we would be using Task Processing Library, or Dataflow built on top of it, with tasks being switched over the various carrier threads.
Or if feeling fancy, reach out to async workflows on F# with computation expressions, even before async/await came to be.
While on the Java side, we would be using java.util.concurrent, with future computations, having fun with Groovy GPars, or Scala frameworks like Akka.
In both platforms, we could even go the extra mile and write our own scheduling algorithms, how those lightweight threads would be mapped into carrier threads.
Naturally not having some of the boilerplate to handle all of that, or using multiple languages on the same project, makes it easier, hence why now we have all those goodies, with async/await or virtual threads on top.
The annotation also help me discover patterns, like when most of the functions of a class have the same annotations, maybe it means that all the functions of the class should have the same annotations.
I really wish an equivalent to those annotations came to Go.
In all honesty, if you “do work” using channels then all your goroutines are “thread safe” as the channel keeps things in order. Also, mutex is working as intended. As you see in your post, -race sees this, it’s good. Now have one goroutine read from a chan, get rid of the mutex, all other goroutines write to the chan, perfection.
https://go.dev/blog/race-detector
> Because of its design, the race detector can detect race conditions only when they are actually triggered by running code, which means it’s important to run race-enabled binaries under realistic workloads.
This isn't a mutex blind spot. It is a side effect of goroutine/thread scheduling, which will obviously be based on workload and other factors. There are a bunch of other cases, that are not mutex related, that it won't see unless execution actually triggers them.