Strange, the impression I got from MS devs on Twitter is that this is something they have been working on for a while that they just finished and premiered at a CPP conference.
The core technology was born out of a group in MS research long ago. Moving from research code to production is highly non-trivial. I first heard how time travel debugging was implemented before i knew about Facebook. I suspect SSDs are part of what made this more practical as it used to simply saturate disk IO, even on specialized systems. With a pci express SSD this becomes much more pragmatic then when the initial idea was implemented.
If the overhead doesn’t prevent capturing a trace reproducing the symptom, there is zero reason you cannot find the root cause. But that actually pales in comparison with treating the execution traces as data.
Yes I believe TTT, or iDNA as it was known, was first released internally around 15 years ago. Or maybe that was just the Nirvana process instrumentation framework it is based on. Either way it's a long time.
A modified version of the tracer is used sometimes to help with support cases, but this is the first public release of the WinDbg replay bit as well.
I can't speak for Microsoft itself, but wouldn't the effort to implement a production-ready time-traveling debugging system (as explained here by SteveJS) be at least somewhat influenced by the growing popularity of time-travel debugging in other languages?
TTD is efficient and works to add as little as possible overhead as it captures code execution in trace files.
...and then in the table below the screenshot...
Large overhead at record time. May collect more data that is needed. Data files can become large.
Based on past experiences with debugging using tracepoints etc., I'm more inclined to believe the latter.
Also, as a side-note, the "more modern visuals" of "WinDbg Preview" look horrible. It's a debugger, not a toy for the barely-computer-literate. Those who want the "friendly experience" will use Visual Studio instead, which amusingly enough continues to have menus instead of the disgusting ribbons as of the latest 2017 version. The screenshot also shows two lines of slightly-misaligned "Command" "Memory" "Source". Yuck.
The overhead varies depending on how cpu/IO bound the application is. IO isn't really affected, so IO bound applications tend to not see a big slowdown. In theory, you could see a very large slowdown in the worst case, but in the average case for a "medium sized" process the slowdown would be noticeable but not affect the usability. This technology isn't based on tracepoints, it's based on in-process cpu emulation. The emulation overhead is on the order of 10-20x in many cases, whereas tracepoint overhead would be on the order of 1000x I believe (maybe worse).
If there is something specific you dislike about the visuals of WinDbg Preview, let us know through the feedback hub or emailing windbgfb@microsoft.com. We realize that folks that have been using WinDbg for 20 years are likely to not be interested in a new UI, but we face 20 years of legacy code every time we want to add a new feature to the UI. As an example, the new WinDbg UI has a javascript window for writing scripts that extend and automate the debugger. It took us approximately 8x less time to implement in WinDbg Preview than what we estimated it would cost in the legacy UI (and a much more junior dev was able to do it as well). We want to innovate without disrupting folks that have effective workflows in WinDbg, so we really want to hear feedback on the new UI. If there are specific things that we can change to make you more efficient in the new UI, please let us know.
We realize that folks that have been using WinDbg for 20 years are likely to not be interested in a new UI, but we face 20 years of legacy code every time we want to add a new feature to the UI.
You can rewrite the UI code so it's easier for you to work on internally, but to the user it looks and acts like it was before --- but proceed cautiously[1]. I'm not against adding features and TTD sounds extremely useful, but having in the overview page some slight contradictions and a screenshot of a dumbed-down UI with obvious WTFs like the two duplicate lines really soured the first impression for me. (Looking at it again, I now see the path in the titlebar has been cut off, despite plenty of empty space after it...)
The debugger isn't written from scratch, just the UI. All of the underlying functionality is essentially the same, just in a more usable shell, and if you collapse the ribbon and retheme the UI to look like the 90s, you could almost squint and think it was the old WinDbg. The change is clearly very polarizing, but we're nearly at parity with what you could do in the old WinDbg UI, and we've already been able to give people features that we could have never dreamed of supporting in the old UI (not for lack of trying). The JavaScript support is just one example.
(Also, the cut-off title bar was an issue in the Fluent.Ribbon component we use, and I think it's fixed in an updated version that we're taking soon)
I'd guess that userbinator is referring to the ribbon items "Command", "Memory", and "Source" where the group name on the ribbon has been given the same name as the single menu below it (in contrast to something like Word where the contextual ribbon items for a table are Table, with Design and Layout as children).
At LogRocket (https://logrocket.com) we're building a TTD of sorts for recording production users in your web app. While we can't replay exact code, we can get you most of the way there with DOM recording/logs/network waterfall.
I believe all of those projects are based on single-core debugging. Restricting the trace to a single core is one way of introducing determinism in the process as a way of allowing replay later. Microsoft time travel debugging is a multi-core technology, which allows you to capture interactions between threads in multi-threaded processes.
Edit: Since I'm still "posting too fast", let me respond here to clarify. Both rr and undodbg support multiple threads, but do not record them simultaneously. They restrict execution to a single core. That's what I mean by recording interactions between threads. For instance, if you have shared memory between processes, that won't work with rr/undodb generally (although I know at least rr can work around this by recording both processes).
I'm not saying one is intrinsically better than the other since there are tradeoffs with both approaches. There is a high constant overhead for recording all cores, but it does have the advantage of scaling well to large numbers of cores.
"rr doesn't record shared-memory multithreading, so it forces your application's threads to execute serially. Your application can see an additional slowdown if it takes advantage of multicore parallelism."
Yes, there is definitely a significant tradeoff here. If you're recording something like Exchange Server having true multi-core execution is a big deal. If you're recording a web browser it's not as important, because most things are still gated on the UI thread. Hopefully y'all can get the constant factor overhead for recording down further over time.
With rr and Firefox we found that with sufficiently pathological scheduling we can eventually tease out many races and other bad thread interactions. This became rr's "chaos mode". I think the biggest drawback of not having true multi-core support in rr today is simply the performance hit that a parallel program will take during recording.
We at VMware had this working and released in 2008. We supported windows and Linux. We even supported things like network applications that had lots of outside state. It was called replay debugging and besides Mozilla, few used it. Sadly it was killed when workstation 8.0 came out.
That really was a cool feature of VMware (and I used to work on the vmware hypervisor briefly), although it used very different techniques and was limited to single core record/replay. Microsoft TTD is based on a different technology, and captures traces of individual processes. More useful for usermode folks, less useful for kernel mode folks compared to the vmware technology.
I'm an escalation engineer at Microsoft, and we have been using this for many, many years to solve complex customer cases. Having it go public is a pretty big deal for us.
I'm the dev lead for WinDbg and the windows debugging platform, and I'm a former dev working on time travel debugging (I wrote the cpu emulation for the project). Happy to answer any questions that anyone has.
Since I'm "posting too fast", let me put my answers here for now:
qod: Right now you can attach TTD to a process after initialization (or any point in time prior to the repro), but you can't trace starting from a breakpoint. We'd love to add that functionality though, so I'm sure you'll see it in a future version of the tool.
Glad you like the new WinDbg! It's been very polarizing (which you can see just reading the comments on this post!), so it's good to hear the positive feedback sometimes :)
We've got a ton of plans to make debugging even faster and more effective in WinDbg, so hopefully we'll win more people over as we make our tools easier to use and more powerful.
First I would just like to say how impressed I am with the new WinDbg and TTD, this is fantastic work and a huge improvement on the existing debugging tools.
My question is: if I want to start taking a TTD trace after hitting a particular breakpoint in WinDbg, can this be done with the current tooling? Use case would be if there is a large amount of initialization code that doesn't need to be traced because the bug or crash is known to occur at a later stage in the program.
Right now you can attach TTD to a process after initialization (or any point in time prior to the repro), but you can't trace starting from a breakpoint. We'd love to add that functionality though, so I'm sure you'll see it in a future version of the tool.
Glad you like the new WinDbg! It's been very polarizing (which you can see just reading the comments on this post!), so it's good to hear the positive feedback sometimes :)
We've got a ton of plans to make debugging even faster and more effective in WinDbg, so hopefully we'll win more people over as we make our tools easier to use and more powerful.
For more than two decades now, I've been using the limited availability of reversible debugging as an example of just how dysfunctional software engineering is as a field. The shoemaker's family's tattered feet.
Reversible debugging been around for six decades (eg EXDAMS on Multics), been generally practical for two or three, been available in gdb for one, and now, finally, here we are. It's taken two bloody human generations to get this close to widely deploying TTD.
So yay. Maybe I can now retire TTD as an example of glacial progress. There's no shortage of others. Sigh.
How many keyboard keys can you hold down without interference effects? Some keyboards offer 10-ish. Or keycaps which are touch pads. Or change color. Or are little active displays. And touch pads which are pressure sensitive. Or detect proximity. Or have a nice stylus. But not mine. It's not space-cadet, or old thinkpad, but chicklets.
One nifty aspect of VR is that a lot of things get unstuck. Even if you're legally blind, VR brings you hand tracking, and vibrational haptics, and spatial audio. You might stroke home-row keys, and make little 3D gestures with fingers. But current Leap Motion hand-tracking software doesn't do linux, so you have to run it in a VM, and point a camera at the keyboard, and...
Here we are on HN. USENET had kill files - people you didn't want to see again. Recommendation systems track preferences and predict what might be of interest. Filters and hints - at lunch I only want to see high value content, but after work, I don't might conversational humor, but that varies by thread topic. But here we are on HN.
I was recently playing python 2 vs 3 games again. So that's an easy "from __past__ import python2", right? Not without scoped method dispatch. Like Ruby's refinements.
Here we are using browsers. Self language prototypical programming now available to the masses? Smalltalk-like live coding? Well, sort of. But try inheriting from a built-in "object" - "error: magical method X only works on magical pseudo-object type Y". Browser implementations have sometimes written the fast path, skipped implementing the general case, and considered themselves done. So does the dearth of prototypical-style code mean it's not a useful style, or is inaccessible, or still hasn't been widely tried? Shrug.
We increasingly attempt to decompose our big math, algorithm, and data structure libraries along orthogonal and algebraic lines. Sketching lattices of theories, with types and operators and laws. "1 is not an object; + is not a message" (1980's Smalltalk critique - does anyone remember where to find this? a technical memo?). If an algorithm only depends on + and a partial order, it says that. But it's not yet common. How often do you use units, instead of a primary school "what's the resistance? 3! Three what? 3!... volts!". And how many languages let you be explicit about laws? Or help you avoid "well, in wetware I model the problem domain, and in wetware I design the solution, and in wetware I compile the solution to an intermediate form... which is at long last understood by the idiot software compiler of narrowly circumscribed helpfulness". And how many such lattices support pushouts? Add a law, add an operator, and you get the same thing as if done in the other order. Haskell spent what, a half decade?, with lots of commitee and community discussion, and broken code, just to insert Applicative in its type hierarchy.
ARPA folks were once considering adding lots of little "lubricant" grants, analogous to the WWII troubleshooters which scurried around industry asking "what are you bottlenecked on? what might speed your work?" A new adminstration, and ARPA had more existential worries. But one strawman was something like "much of our knowledge about how to write good VM's is in person X's head; they've been meaning to write a book, but aren't getting to it; instead of waiting years for that knowledge to slowly diffuse, it would be inexpensive to pay someone to help, sitting outside their office, and stand on their desk". Instead we did the wait for years thing.
As far as I can tell EXDAMS was never implemented. The RAND EXDAMS memo is more of a thought experiment than anything else.
Getting reverse-execution debugging to work adequately at scale is difficult. gdb's approach doesn't scale at all. I dispute that reversible debugging has been "generally practical for two or three decades"; as far as I know the first practical products were UndoDB and VMWare.
> "generally practical for two or three decades" [...] the first practical products
I was thinking of "we now know how to do this" (snapshot and replay (and input log)), rather than "and it's a product".
Which seems sometimes valid, and sometimes not. We know how to make a 4K/eye VR HMD with existing parts, and it's simply a market failure that one's not on my desk. But for other things, "know how to do it, sort of, in a lab" and "know how to do it, practically, for real" can be greatly separated in time and effort.
> first practical products were UndoDB and VMWare
My very fuzzy recollection is DEC alpha had a reversible debugger in the late 1990's? - but I'm not finding it.
Oh my god they redesigned WinDbg with a Microsoft Office UI lookalike starter kit including ribbons. And at least one of the screenshots showed JavaScript.
So many fingers crossed hoping they don't ruin this indispensable basic tool. Every platform needs to have a no-nonsense debugger that doesn't fall over immediately when it has no symbols and source. WinDbg (was?) that.
51 comments
[ 3.3 ms ] story [ 81.2 ms ] threadThis has approximately nothing to do with C++ or even "language communities".
If the overhead doesn’t prevent capturing a trace reproducing the symptom, there is zero reason you cannot find the root cause. But that actually pales in comparison with treating the execution traces as data.
A modified version of the tracer is used sometimes to help with support cases, but this is the first public release of the WinDbg replay bit as well.
...and then in the table below the screenshot...
Large overhead at record time. May collect more data that is needed. Data files can become large.
Based on past experiences with debugging using tracepoints etc., I'm more inclined to believe the latter.
Also, as a side-note, the "more modern visuals" of "WinDbg Preview" look horrible. It's a debugger, not a toy for the barely-computer-literate. Those who want the "friendly experience" will use Visual Studio instead, which amusingly enough continues to have menus instead of the disgusting ribbons as of the latest 2017 version. The screenshot also shows two lines of slightly-misaligned "Command" "Memory" "Source". Yuck.
(For those who don't understand, this what the original WinDbg looks like --- simple and functional: http://sandsprite.com/blogs/images/main_ui.png )
If there is something specific you dislike about the visuals of WinDbg Preview, let us know through the feedback hub or emailing windbgfb@microsoft.com. We realize that folks that have been using WinDbg for 20 years are likely to not be interested in a new UI, but we face 20 years of legacy code every time we want to add a new feature to the UI. As an example, the new WinDbg UI has a javascript window for writing scripts that extend and automate the debugger. It took us approximately 8x less time to implement in WinDbg Preview than what we estimated it would cost in the legacy UI (and a much more junior dev was able to do it as well). We want to innovate without disrupting folks that have effective workflows in WinDbg, so we really want to hear feedback on the new UI. If there are specific things that we can change to make you more efficient in the new UI, please let us know.
You can rewrite the UI code so it's easier for you to work on internally, but to the user it looks and acts like it was before --- but proceed cautiously[1]. I'm not against adding features and TTD sounds extremely useful, but having in the overview page some slight contradictions and a screenshot of a dumbed-down UI with obvious WTFs like the two duplicate lines really soured the first impression for me. (Looking at it again, I now see the path in the titlebar has been cut off, despite plenty of empty space after it...)
[1] https://www.joelonsoftware.com/2000/04/06/things-you-should-...
The debugger isn't written from scratch, just the UI. All of the underlying functionality is essentially the same, just in a more usable shell, and if you collapse the ribbon and retheme the UI to look like the 90s, you could almost squint and think it was the old WinDbg. The change is clearly very polarizing, but we're nearly at parity with what you could do in the old WinDbg UI, and we've already been able to give people features that we could have never dreamed of supporting in the old UI (not for lack of trying). The JavaScript support is just one example.
(Also, the cut-off title bar was an issue in the Fluent.Ribbon component we use, and I think it's fixed in an updated version that we're taking soon)
I'd guess that userbinator is referring to the ribbon items "Command", "Memory", and "Source" where the group name on the ribbon has been given the same name as the single menu below it (in contrast to something like Word where the contextual ribbon items for a table are Table, with Design and Layout as children).
At LogRocket (https://logrocket.com) we're building a TTD of sorts for recording production users in your web app. While we can't replay exact code, we can get you most of the way there with DOM recording/logs/network waterfall.
I also recently discovered WDD[3] (which is for Windows) as well.
[0]: https://undo.io/
[1]: https://www.gnu.org/software/gdb/news/reversible.html
[2]: http://rr-project.org/
[3]: https://github.com/ipkn/wdd
https://software.intel.com/en-us/articles/program-recordrepl... https://software.intel.com/en-us/articles/pintool-drdebug
Edit: Since I'm still "posting too fast", let me respond here to clarify. Both rr and undodbg support multiple threads, but do not record them simultaneously. They restrict execution to a single core. That's what I mean by recording interactions between threads. For instance, if you have shared memory between processes, that won't work with rr/undodb generally (although I know at least rr can work around this by recording both processes).
I'm not saying one is intrinsically better than the other since there are tradeoffs with both approaches. There is a high constant overhead for recording all cores, but it does have the advantage of scaling well to large numbers of cores.
"rr doesn't record shared-memory multithreading, so it forces your application's threads to execute serially. Your application can see an additional slowdown if it takes advantage of multicore parallelism."
[0]: https://github.com/mozilla/rr/wiki/Usage#recording-an-execut...
"Handles concurrency
Record and replay multi-threaded programs, and those that use shared memory and asynchronous I/O."
[0] - https://undo.io/technology/features/
With rr and Firefox we found that with sufficiently pathological scheduling we can eventually tease out many races and other bad thread interactions. This became rr's "chaos mode". I think the biggest drawback of not having true multi-core support in rr today is simply the performance hit that a parallel program will take during recording.
What ever happened to the intellctual property for that? Any chance it could be open sourced?
I'm an escalation engineer at Microsoft, and we have been using this for many, many years to solve complex customer cases. Having it go public is a pretty big deal for us.
Since I'm "posting too fast", let me put my answers here for now:
qod: Right now you can attach TTD to a process after initialization (or any point in time prior to the repro), but you can't trace starting from a breakpoint. We'd love to add that functionality though, so I'm sure you'll see it in a future version of the tool.
Glad you like the new WinDbg! It's been very polarizing (which you can see just reading the comments on this post!), so it's good to hear the positive feedback sometimes :)
We've got a ton of plans to make debugging even faster and more effective in WinDbg, so hopefully we'll win more people over as we make our tools easier to use and more powerful.
My question is: if I want to start taking a TTD trace after hitting a particular breakpoint in WinDbg, can this be done with the current tooling? Use case would be if there is a large amount of initialization code that doesn't need to be traced because the bug or crash is known to occur at a later stage in the program.
Glad you like the new WinDbg! It's been very polarizing (which you can see just reading the comments on this post!), so it's good to hear the positive feedback sometimes :)
We've got a ton of plans to make debugging even faster and more effective in WinDbg, so hopefully we'll win more people over as we make our tools easier to use and more powerful.
Reversible debugging been around for six decades (eg EXDAMS on Multics), been generally practical for two or three, been available in gdb for one, and now, finally, here we are. It's taken two bloody human generations to get this close to widely deploying TTD.
So yay. Maybe I can now retire TTD as an example of glacial progress. There's no shortage of others. Sigh.
> I'm curious to hear some.
How many keyboard keys can you hold down without interference effects? Some keyboards offer 10-ish. Or keycaps which are touch pads. Or change color. Or are little active displays. And touch pads which are pressure sensitive. Or detect proximity. Or have a nice stylus. But not mine. It's not space-cadet, or old thinkpad, but chicklets.
One nifty aspect of VR is that a lot of things get unstuck. Even if you're legally blind, VR brings you hand tracking, and vibrational haptics, and spatial audio. You might stroke home-row keys, and make little 3D gestures with fingers. But current Leap Motion hand-tracking software doesn't do linux, so you have to run it in a VM, and point a camera at the keyboard, and...
Here we are on HN. USENET had kill files - people you didn't want to see again. Recommendation systems track preferences and predict what might be of interest. Filters and hints - at lunch I only want to see high value content, but after work, I don't might conversational humor, but that varies by thread topic. But here we are on HN.
I was recently playing python 2 vs 3 games again. So that's an easy "from __past__ import python2", right? Not without scoped method dispatch. Like Ruby's refinements.
Here we are using browsers. Self language prototypical programming now available to the masses? Smalltalk-like live coding? Well, sort of. But try inheriting from a built-in "object" - "error: magical method X only works on magical pseudo-object type Y". Browser implementations have sometimes written the fast path, skipped implementing the general case, and considered themselves done. So does the dearth of prototypical-style code mean it's not a useful style, or is inaccessible, or still hasn't been widely tried? Shrug.
We increasingly attempt to decompose our big math, algorithm, and data structure libraries along orthogonal and algebraic lines. Sketching lattices of theories, with types and operators and laws. "1 is not an object; + is not a message" (1980's Smalltalk critique - does anyone remember where to find this? a technical memo?). If an algorithm only depends on + and a partial order, it says that. But it's not yet common. How often do you use units, instead of a primary school "what's the resistance? 3! Three what? 3!... volts!". And how many languages let you be explicit about laws? Or help you avoid "well, in wetware I model the problem domain, and in wetware I design the solution, and in wetware I compile the solution to an intermediate form... which is at long last understood by the idiot software compiler of narrowly circumscribed helpfulness". And how many such lattices support pushouts? Add a law, add an operator, and you get the same thing as if done in the other order. Haskell spent what, a half decade?, with lots of commitee and community discussion, and broken code, just to insert Applicative in its type hierarchy.
ARPA folks were once considering adding lots of little "lubricant" grants, analogous to the WWII troubleshooters which scurried around industry asking "what are you bottlenecked on? what might speed your work?" A new adminstration, and ARPA had more existential worries. But one strawman was something like "much of our knowledge about how to write good VM's is in person X's head; they've been meaning to write a book, but aren't getting to it; instead of waiting years for that knowledge to slowly diffuse, it would be inexpensive to pay someone to help, sitting outside their office, and stand on their desk". Instead we did the wait for years thing.
Getting reverse-execution debugging to work adequately at scale is difficult. gdb's approach doesn't scale at all. I dispute that reversible debugging has been "generally practical for two or three decades"; as far as I know the first practical products were UndoDB and VMWare.
Ah, good point.
> "generally practical for two or three decades" [...] the first practical products
I was thinking of "we now know how to do this" (snapshot and replay (and input log)), rather than "and it's a product".
Which seems sometimes valid, and sometimes not. We know how to make a 4K/eye VR HMD with existing parts, and it's simply a market failure that one's not on my desk. But for other things, "know how to do it, sort of, in a lab" and "know how to do it, practically, for real" can be greatly separated in time and effort.
> first practical products were UndoDB and VMWare
My very fuzzy recollection is DEC alpha had a reversible debugger in the late 1990's? - but I'm not finding it.
The memo[1] mentions
> Since EXDAMS is currently being debugged and is not operational, no performance statistics are available. [p12]
Suggesting implementation was attempted, though perhaps not successfully?
[1] "EXDAMS: extendable debugging and monitoring system" Memo RM-5772-ARPA APRIL 1969 http://www.dtic.mil/get-tr-doc/pdf?AD=AD0686373
https://github.com/mozilla/rr
So many fingers crossed hoping they don't ruin this indispensable basic tool. Every platform needs to have a no-nonsense debugger that doesn't fall over immediately when it has no symbols and source. WinDbg (was?) that.