I think they have so much potential but in my experience, they are still so far off from mass market appeal. rr is amazing but it doesn't work well in cloud environments where it needs you to configure settings in the virtual machine (which you can't). So this limits it to desktop Linux machines. Valuable, but not as valuable as some other options. Plus, using GDB is. Ugh.
There are amazing tools like Jive which is absolutely... Wow. It generates UML diagrams representing actual code execution. That's fantastic!
If you can get it to run your code base. I couldn't get anything beyond a hello world. Replay is pretty impressive, but front-end issues don't have as much state as other issues and are typically easier to track. It isn't great with long recordings so you can't just "leave it running".
I hope this field matures. Right now my hope is that vendors will steal good ideas and incorporate them into the IDEs such as IntelliJs stream debugger.
"Time travel debugging" seems to be taking off as the term - "reverse debugger" was around for a long time but doesn't seem to have taken root. It has fixed the conversation of "We make a reversible debugger", "Ah so you put the bugs in", though!
> rr is amazing but it doesn't work well in cloud environments where it needs you to configure settings in the virtual machine (which you can't). So this limits it to desktop Linux machines.
Hopefully we'll see cloud environments find a way to expose more of the hardware performance counter stuff by default. The alternative is to use an instrumentation-based system that doesn't need access to performance counters. That said, as with machine-level virtualisation, it'd be nice to see the CPU vendors provide more explicit hardware support over time to make implementation easier / higher performance than current approaches.
> Valuable, but not as valuable as some other options. Plus, using GDB is. Ugh.
There's also Pernosco https://pernos.co/ which is a cloud service (hosted versions possible) from the original developers of rr. It post-processes uploaded rr traces into a database. This post-processing is compute-intensive but happens on their beefy worker machines. Once it's done, they can retrieve program state across time almost instantly - their website is worth checking out for some examples.
> Replay is pretty impressive, but front-end issues don't have as much state as other issues and are typically easier to track. It isn't great with long recordings so you can't just "leave it running".
Are you referring to replay.io here? (I was not sure whether you were talking about a replay behaviour in Jive - this is the risk of company / product names that are verbs, a problem we also have!)
I've not used their stuff but the debug interface they provide looks really intuitive. They support recording Node so I presume they can do some backend stuff too? (Assuming it's in JS).
Looks like they want to apply their techniques to other languages eventually.
I looked at Pernosco but it's obviously not as useful without the recording.
Both Undo and Pernosco aren't useful on a Mac.
Yes I was talking about replay.io.
I agree that these tools have huge potential. I think they have a chance to make a breakthrough. But to cross that chasm into mass market we need something that's easier to use. I had hopes for video bug too. The interface seems promising initially but the developer experience is just awful.
I agree that instrumentation is probably the way to go. Especially for VM code.
FWIW, Undo works fine in Mac Docker - but that's only useful if you're actually targeting Linux. I presume rr could work there as well, though I've not tried it to see if the performance counters are accessible.
Videobug I had not heard of. Where's the developer experience lacking? I've just had a peek at their video but honestly have little experience of what a Java experience should look like!
The interface in the plugin of video bug looks like it was designed by a blind person. It's pretty problematic. The steps to follow aren't clear and had I not known how to install an agent I would have been bewildered. I can't imagine someone without deep experience in this field using a tool like that.
There are some points in this article that I don't see mentioned often and are really important benefits.
"Story 1: What if we could let the bug happen and drink a coffee (or more)?" - the fact you can just leave the recorder running until your flaky issue has reproduced is really powerful. Once captured, you can debug at leisure, knowing invisible goblins won't make the problem vanish again.
"Story 2: Let’s print, but not build" - the `dprintf` command in GDB is so cool. Being able to tag additional print commands into the code without having to rebuild is powerful. You can do this iteratively with time travel debug, since you know you're running on the same trace every time.
The challenge of reverse debuggers is knowing when it should dispose of the recording; otherwise, if the debugger keeps recording the full sequence of states and actions, it would cause a memory leak and/or unbounded growth in disk usage. Such debugger would not be able to sustain a DoS attack if debugging in production (it would spam the debugger with lots of different states generated by the attacker); in fact, it would likely make it easier for an attacker to DoS a service as it would consume computational resources.
> if the debugger keeps recording the full sequence of states and actions,
It's worth noting that the main time travel debugging technologies don't keep the state of every operation performed, only the non-deterministic ones (i.e. where new information flows into the recorded program). A compute-intensive simulation can run a few days and generate little-to-no state that needs recording.
Your point stands, though, that a malicious attacker could do many non-deterministic operations (e.g. read data in from a socket). This can be solved - relatively easily - with a circular logging approach is workable to address this. i.e. discard chunks of older history to free up space as you go.
(Whether individual time travel debuggers support that is, as far as I'm aware, more a case of balancing demand for the feature against engineering effort.)
> Such debugger would not be able to sustain a DoS attack if debugging in production (it would spam the debugger with lots of different states generated by the attacker); in fact, it would likely make it easier for an attacker to DoS a service as it would consume computational resources.
In-production use could potentially be very useful if that's where some weird flaky issue occurs, so this is worth worrying about. It's definitely true that you've got to have the resources / settings to not overwhelm your machine. The other thing is that adding a complex layer of additional software risks adding additional exploits into the mix.
If your production issue is at a customer site or in a semi-trusted context (corporate network service) - which for some software is the norm - then I think that makes the threat model a bit less scary.
Otherwise, I guess it's a case of sandboxing the software within other measures and maybe spawning a separate instance that's recorded so you've still got "normal" instances running in parallel?
Never use these things in production. They slow your servers very noticeably. It's OK for development. QA and even for unit test runs (which is an amazing use case for flaky tests). But production is a completely different ballgame.
This reminds me of a program I wrote whose entire input comes from a blockchain. Everything that the program does is in response to some complex sequences of events which occurred on a blockchain.
This means that my program is deterministic since I can replay any part of the blockchain history and it will produce exactly the same results/output on each run. If a bug happens, I can replay the bug as many times as I like, exactly as it happened until I understood the cause of the issue and fixed the code - I can even go back and re-process the entire blockchain history to make sure that the new code changes didn't break any previously supported functionality.
That makes for a great debugging experience, I'm almost hoping that my project will hit a bug so that I can have fun debugging it. I only had to debug it once in the last 12 months because of a usability/UX issue.
This approach of writing deterministic code seems to produce much more robust code too. It requires a fair amount of work to weed out non-determinism from the code though (I.e. async logic running in parallel is the main challenge).
Anybody know any good resources on scripting the debugger? It seems like you should be able to use some kind of concolic execution approach + fuzzing to do something like a binary search between forward and backward breakpoints to automatically find likely bug-containing regions. Or maybe union-find to partition the code into some (minimal) quite-buggy/(maximal) less-buggy sets of code regions.
Eg if you could get it to figure out the random values will probably always lie within some interval, you might be able to make it figure out “there is a path from <randint> to <if statement> to <raise segfault>”
Additionally it seems really powerful to be able to introspect a program from outside. I run into situations sometimes where I would like to be able to make a realtime statistical gui of the internal state of some 3rd party code without necessarily butchering its codebase, seems like I could just write a little python to sample the local variables from within gdb then have that serve from an http server.
Or if I want to extract some library internal scratch calculations that are private or get cleaned up at the end of some long computation, I’d much rather write 10 lines of Python to copy and paste the data I want than attempt to refactor a copy of the codebase. Learning enough about it to predict the side-effects of my modification could get quite tedious.
Not sure of the feasibility of this, or how well such an approach will work with 1000x the lines of code in the real world, however :D
Yep, this is why we're building a time-travel debugger for JS at https://replay.io .
The basic idea of Replay: Use our special browser or Node forks to make a recording of your app, load the recording in our debugger, and you can pause at _any_ point in the recording. In fact, you can add print statements to any line of code, and it will show you what it _would_ have printed _every time that line of code ran_!
From there, you can jump to any of those print statement hits, and do typical step debugging and inspection of variables. So, it's the best of both worlds - you can use print statements and step debugging, together, at any point in time in the recording.
We record the OS-level syscalls from the browser, and replay those in the cloud against hundreds of process forks of the exact browser version while you're debugging. When you step backwards, evaluate a print statement that ran many times, or jump to a different pause point, the backend is using those many forked processes to evaluate those results and ship the info to the debugging client.
14 comments
[ 3.2 ms ] story [ 40.1 ms ] threadI think they have so much potential but in my experience, they are still so far off from mass market appeal. rr is amazing but it doesn't work well in cloud environments where it needs you to configure settings in the virtual machine (which you can't). So this limits it to desktop Linux machines. Valuable, but not as valuable as some other options. Plus, using GDB is. Ugh.
There are amazing tools like Jive which is absolutely... Wow. It generates UML diagrams representing actual code execution. That's fantastic!
If you can get it to run your code base. I couldn't get anything beyond a hello world. Replay is pretty impressive, but front-end issues don't have as much state as other issues and are typically easier to track. It isn't great with long recordings so you can't just "leave it running".
I hope this field matures. Right now my hope is that vendors will steal good ideas and incorporate them into the IDEs such as IntelliJs stream debugger.
> rr is amazing but it doesn't work well in cloud environments where it needs you to configure settings in the virtual machine (which you can't). So this limits it to desktop Linux machines.
Hopefully we'll see cloud environments find a way to expose more of the hardware performance counter stuff by default. The alternative is to use an instrumentation-based system that doesn't need access to performance counters. That said, as with machine-level virtualisation, it'd be nice to see the CPU vendors provide more explicit hardware support over time to make implementation easier / higher performance than current approaches.
> Valuable, but not as valuable as some other options. Plus, using GDB is. Ugh.
Looks like there's some support / guidance for using an IDE interface to rr's GDB - https://github.com/rr-debugger/rr/wiki/Using-rr-in-an-IDE
There's also Pernosco https://pernos.co/ which is a cloud service (hosted versions possible) from the original developers of rr. It post-processes uploaded rr traces into a database. This post-processing is compute-intensive but happens on their beefy worker machines. Once it's done, they can retrieve program state across time almost instantly - their website is worth checking out for some examples.
At Undo (disclaimer: where I work) we've got a VS Code plugin that's being very actively developed https://marketplace.visualstudio.com/items?itemName=Undo.udb to make it easier to expose the time travel stuff in our LiveRecorder product.
> Replay is pretty impressive, but front-end issues don't have as much state as other issues and are typically easier to track. It isn't great with long recordings so you can't just "leave it running".
Are you referring to replay.io here? (I was not sure whether you were talking about a replay behaviour in Jive - this is the risk of company / product names that are verbs, a problem we also have!)
I've not used their stuff but the debug interface they provide looks really intuitive. They support recording Node so I presume they can do some backend stuff too? (Assuming it's in JS).
Looks like they want to apply their techniques to other languages eventually.
Yes I was talking about replay.io.
I agree that these tools have huge potential. I think they have a chance to make a breakthrough. But to cross that chasm into mass market we need something that's easier to use. I had hopes for video bug too. The interface seems promising initially but the developer experience is just awful.
I agree that instrumentation is probably the way to go. Especially for VM code.
Videobug I had not heard of. Where's the developer experience lacking? I've just had a peek at their video but honestly have little experience of what a Java experience should look like!
The interface in the plugin of video bug looks like it was designed by a blind person. It's pretty problematic. The steps to follow aren't clear and had I not known how to install an agent I would have been bewildered. I can't imagine someone without deep experience in this field using a tool like that.
"Story 1: What if we could let the bug happen and drink a coffee (or more)?" - the fact you can just leave the recorder running until your flaky issue has reproduced is really powerful. Once captured, you can debug at leisure, knowing invisible goblins won't make the problem vanish again.
"Story 2: Let’s print, but not build" - the `dprintf` command in GDB is so cool. Being able to tag additional print commands into the code without having to rebuild is powerful. You can do this iteratively with time travel debug, since you know you're running on the same trace every time.
It's worth noting that the main time travel debugging technologies don't keep the state of every operation performed, only the non-deterministic ones (i.e. where new information flows into the recorded program). A compute-intensive simulation can run a few days and generate little-to-no state that needs recording.
Your point stands, though, that a malicious attacker could do many non-deterministic operations (e.g. read data in from a socket). This can be solved - relatively easily - with a circular logging approach is workable to address this. i.e. discard chunks of older history to free up space as you go.
(Whether individual time travel debuggers support that is, as far as I'm aware, more a case of balancing demand for the feature against engineering effort.)
> Such debugger would not be able to sustain a DoS attack if debugging in production (it would spam the debugger with lots of different states generated by the attacker); in fact, it would likely make it easier for an attacker to DoS a service as it would consume computational resources.
In-production use could potentially be very useful if that's where some weird flaky issue occurs, so this is worth worrying about. It's definitely true that you've got to have the resources / settings to not overwhelm your machine. The other thing is that adding a complex layer of additional software risks adding additional exploits into the mix.
If your production issue is at a customer site or in a semi-trusted context (corporate network service) - which for some software is the norm - then I think that makes the threat model a bit less scary.
Otherwise, I guess it's a case of sandboxing the software within other measures and maybe spawning a separate instance that's recorded so you've still got "normal" instances running in parallel?
This means that my program is deterministic since I can replay any part of the blockchain history and it will produce exactly the same results/output on each run. If a bug happens, I can replay the bug as many times as I like, exactly as it happened until I understood the cause of the issue and fixed the code - I can even go back and re-process the entire blockchain history to make sure that the new code changes didn't break any previously supported functionality.
That makes for a great debugging experience, I'm almost hoping that my project will hit a bug so that I can have fun debugging it. I only had to debug it once in the last 12 months because of a usability/UX issue.
This approach of writing deterministic code seems to produce much more robust code too. It requires a fair amount of work to weed out non-determinism from the code though (I.e. async logic running in parallel is the main challenge).
Eg if you could get it to figure out the random values will probably always lie within some interval, you might be able to make it figure out “there is a path from <randint> to <if statement> to <raise segfault>”
Additionally it seems really powerful to be able to introspect a program from outside. I run into situations sometimes where I would like to be able to make a realtime statistical gui of the internal state of some 3rd party code without necessarily butchering its codebase, seems like I could just write a little python to sample the local variables from within gdb then have that serve from an http server.
Or if I want to extract some library internal scratch calculations that are private or get cleaned up at the end of some long computation, I’d much rather write 10 lines of Python to copy and paste the data I want than attempt to refactor a copy of the codebase. Learning enough about it to predict the side-effects of my modification could get quite tedious.
Not sure of the feasibility of this, or how well such an approach will work with 1000x the lines of code in the real world, however :D
That should get you somewhere useful for time travel systems based on GDB. You can also use Guile if you'd prefer a Lisp - https://sourceware.org/gdb/onlinedocs/gdb/Guile-API.html
Depending on your debugger of choice there are other options - e.g. WinDbg (which has time travel debug) can, I think, be scripted with JS.
The basic idea of Replay: Use our special browser or Node forks to make a recording of your app, load the recording in our debugger, and you can pause at _any_ point in the recording. In fact, you can add print statements to any line of code, and it will show you what it _would_ have printed _every time that line of code ran_!
From there, you can jump to any of those print statement hits, and do typical step debugging and inspection of variables. So, it's the best of both worlds - you can use print statements and step debugging, together, at any point in time in the recording.
We record the OS-level syscalls from the browser, and replay those in the cloud against hundreds of process forks of the exact browser version while you're debugging. When you step backwards, evaluate a print statement that ran many times, or jump to a different pause point, the backend is using those many forked processes to evaluate those results and ship the info to the debugging client.
I can say that we personally use Replay to debug our own dev work on Replay on a daily basis, and it makes a _huge_ difference. I recently used Replay to debug an obscure E2E test failure ( https://twitter.com/acemarke/status/1547682026181996554 ), and a bug with React-Redux v8's `connect` and React's experimental `<Offscreen>` ( https://twitter.com/acemarke/status/1559643125865390083 ).
See https://replay.io/record-bugs for the getting started steps to use Replay, and we've got some neat articles about some of the technology at https://docs.replay.io/docs/how-replay-works-2adcccaf8f7743f... .
If you've got any questions, please come by our Discord and ask! https://replay.io/discord