looks very promising, one of the biggest issue in golang for me is profiling and constant memory leaks/pressure. Not sure if there is an alternative of what people use now
> Instead of adding the Valgrind headers to the tree, and using cgo to
call the various Valgrind client request macros, we just add an assembly
function which emits the necessary instructions to trigger client
requests.
Love that they have taken this route, this is the way bootstraped toolchains should be, minimal building blocks and everything else on the language itself.
Valgrind is a hidden super-power. In much of the software I write, there's 'make check' which runs the test cases, and 'make check-valgrind' that runs the same test cases under valgrind. The latter is only used on developer machines. It often reveals memory leaks or other subtle memory bugs.
That's quite nice. There is a small risk that the client request mechanism might change . The headers don't change much - mostly when a new platform gets added. Go is only targeting amd64 and arm64.
This isn't so much about leaks. The most important thing that this will enable is correct analysis of uninitialised memory. Without annotation memory that gets recycled will not be correctly poisoned. I imagine that it will also be useful for the other tools (except cachegrind and callgrind).
Not even a Go user, and yet this is one of the best things I have read today morning. Valgrind is possibly one of the most powerful tools I have in my belt!!
I'd be interested to know why Valgrind vs the Clang AddressSanitizer and MemorySaniziter. These normally find more types of errors (like use-after-return) and I find it significantly faster than Valgrind.
I'm interested too. I'm using a Go program that call a cpp library with SWING and I was interested in find out if that library had a memory leak, or maybe the SWING wrap I wrote. But this kind of problem can't be detected via pprof, so I tought, what if Go support Valgrind?? and find out this changes.
I'm not sure if this will work though, will it @bracewel?
Author of the linked CL here: we added this mostly so that we could abuse the memory initialization tracking to test the constant-time-ness of crypto code (similar to what BoringSSL does, proposed by agl around fifteen years ago: https://www.imperialviolet.org/2010/04/01/ctgrind.html), which is an annoyingly hard property to test.
We're hoping that there are also a bunch of other interesting side-effects of enabling the usage of Valgrind for Go, in particular seeing how we can use it to track how the runtime handles memory (hopefully correctly!)
edit: also strong disclaimer that this support is still somewhat experimental. I am not 100% confident we are properly instrumenting everything, and it's likely there are still some errant warnings that don't fully make sense.
This is super cool. Hopefully it will flush
out other issues in Go too.
But I wonder why its not trivial to throw a bunch
of different inputs at your cyphering functions and measure
that the execution times are all within an
epsilon tolerance?
I mean, you want to show constant time
of your crypto functions, why
not just directly measure the time under
lots of inputs? (and maybe background Garbage
Collection and OS noise) and see how constant
they are directly?
Also some CPUs have a counter for conditional
branches (that the rr debuger leverages), and you
could sample that before and after and make
sure the number of conditional branches does
not change between decrypts -- as that AGL post
mentions branching being the same is
important for constant time.
Finally, it would also seem trivial to track
the first 10 decrypts, take their maximum
time add a small extra few nanoseconds tolerance,
and pad every following decrypt with
a few nanoseconds (executing noops)
to force constant time when it is varying.
And you could add an assert that anything over that
established upper bound crashes the program since it
is violating the constant time property. I
suppose the real difficulty is if the OS deschedules
your execution and throws off your timing check...
Don't get me wrong, I love Valgrind, and have been using it extensively in my past life as a C developer. Though the fact that Go needs Valgrind feels like a failure of the language or the ecosystem. I've been doing Rust for ~6 years now, and haven't had to reach for Valgrind even once (I think a team member may have use it once).
I realize that's probably because of cgo, and maybe it's in-fact a step forward, but I can help but feel like it is a step backwards.
I'm glad to see rsc still actively involved. And commenting on commit messages.
The older I get the more I value commit messages. It's too easy to just leave a message like "adding valgrind support", which isn't very useful to future readers doing archaeology.
22 comments
[ 3.0 ms ] story [ 52.7 ms ] threadOtherwise the relevant warnings get swamped by a huge amount by irrelevant warnings.
This is why running Valgrind on Python code does not work.
Love that they have taken this route, this is the way bootstraped toolchains should be, minimal building blocks and everything else on the language itself.
But maybe others will find a way to use it. Who knows?
This isn't so much about leaks. The most important thing that this will enable is correct analysis of uninitialised memory. Without annotation memory that gets recycled will not be correctly poisoned. I imagine that it will also be useful for the other tools (except cachegrind and callgrind).
I'd be interested to know why Valgrind vs the Clang AddressSanitizer and MemorySaniziter. These normally find more types of errors (like use-after-return) and I find it significantly faster than Valgrind.
I'm not sure if this will work though, will it @bracewel?
We're hoping that there are also a bunch of other interesting side-effects of enabling the usage of Valgrind for Go, in particular seeing how we can use it to track how the runtime handles memory (hopefully correctly!)
edit: also strong disclaimer that this support is still somewhat experimental. I am not 100% confident we are properly instrumenting everything, and it's likely there are still some errant warnings that don't fully make sense.
But I wonder why its not trivial to throw a bunch of different inputs at your cyphering functions and measure that the execution times are all within an epsilon tolerance?
I mean, you want to show constant time of your crypto functions, why not just directly measure the time under lots of inputs? (and maybe background Garbage Collection and OS noise) and see how constant they are directly?
Also some CPUs have a counter for conditional branches (that the rr debuger leverages), and you could sample that before and after and make sure the number of conditional branches does not change between decrypts -- as that AGL post mentions branching being the same is important for constant time.
Finally, it would also seem trivial to track the first 10 decrypts, take their maximum time add a small extra few nanoseconds tolerance, and pad every following decrypt with a few nanoseconds (executing noops) to force constant time when it is varying.
And you could add an assert that anything over that established upper bound crashes the program since it is violating the constant time property. I suppose the real difficulty is if the OS deschedules your execution and throws off your timing check...
Don't get me wrong, I love Valgrind, and have been using it extensively in my past life as a C developer. Though the fact that Go needs Valgrind feels like a failure of the language or the ecosystem. I've been doing Rust for ~6 years now, and haven't had to reach for Valgrind even once (I think a team member may have use it once).
I realize that's probably because of cgo, and maybe it's in-fact a step forward, but I can help but feel like it is a step backwards.
The older I get the more I value commit messages. It's too easy to just leave a message like "adding valgrind support", which isn't very useful to future readers doing archaeology.