Came across a reference to this in my bookmarks. The original was gone and before I go reliable snapshots of my bookmarks but I found at least a couple of interesting open source projects, like the linked one about and also https://pleiad.cl/tod/ which also has videos:
Generally speaking we do not see much mention of them because the technology involved is reasonably tricky to develop and software developers and companies are as a rule penny wise, but pound foolish with respect to tooling. So, there is very little money in actually expending the development effort to make a polished, accessible tool and any such tool will see relatively limited mass market uptake unless it is free as in beer. This largely precludes the existing relatively polished commercial offerings from achieving ubiquity and thus why it stays relatively niche and underused. That is also why despite there being so many examples over time none of them have been able to achieve escape velocity.
Literally every person who uses it for debugging a hard problem absolutely raves about it. Debugging something like a stack corruption or a heap corruption is trivial. And every other class of bugs are also incredibly easy, as long as it doesn't rely on very precise timing where the tracing changes the behavior. So why don't more people use it? Why haven't more people heard about it?
I'm not entirely sure, but I do have a guess. Most bugs are shallow. We tend to think about the really hard, really deep bugs, but the vast majority of devs are working on bugs where it's easier to just add some logging to figure out a logic error.
If an app is set up to output all variables, doesn't debugview or debugview++ effectively give you the same ability to step back in time?
Of course, it means adding code to the app to output everything to debugview where as the debugger can use the debug symbols and then log everything which is alot less work, but some languages dont come with a debugger.
The answer, as always with engineering is a "yes but"
It's pretty appealing. I also believe MS Visual Studio had something that effectively worked like that albeit without changing the coffee manually.
But - if you've got bugs due to memory corruption, race conditions, etc then the logging doesn't necessarily tell you what you need as you don't know when the change actually happened.
The other issue is that once you've started logging everywhere the string formatting is likely going to be a significant slowdown - at which point you could have used a time travel debugger and probably get better performance.
Time-travel debugging and omniscient debugging are different. For example, https://rr-project.org is a time-travel debugger --- it can simulate executing backwards through time --- but it is not an omnisicient debugger because it doesn't provide fast access to all program states. https://pernos.co/about is an omniscient debugger built on top of rr: it provides fast access to all program states and visualizations built on top of that.
GDB supports reverse execution out of the box (not on all targets / remote), but it can be useful in some cases [0]. The tricky part is I/O and all the other side effects, which can not be undone as easily as memory / register use. It can still be useful for complex algorithms and other side-effect free code. For more practical debugging, record/replay based systems such as radare2 [1] can be more useful.
I've worked on a lot of sophisticated data pipelines, in robotics, built customized wysiwyg editors, distributed systems, append-only logs with replicated data types, in around a dozen languages. While I have found occasionally remote profiling useful for optimizing hot code paths, I have always found that trace-level logs and structured logging with aggregation and indexing is superior to debugging in most applications.
Essentially - println debugging is superior to interactive debugging, especially for intermittent bugs, data races, deadlocks, resource leakage, modeling and serialization bugs, binary or source code incompatibility bugs, and bugs that are catastrophic and rare or pathological in nature.
The problem with interactive debugging is that you typically only have a view into not only a single process of a system, but also only a single concurrent execution thread of the single process.
When you have structured logging and aggregation and indexed log search, you have a much broader view of the systems under investigation and can see the conditions causing the bug with a much higher frequency than with interactive debugging.
For any non-trivial bug in any sufficiently large input or program you have to have insight into the root cause via intuition to cause it to occur quickly enough to inspect the program via the debugger. This obviously doesn't scale well. It's much better to have a detailed, structured log mode built into your systems that is capable of being enabled on demand - via environment variable, remote procedure call, request header, data packet, or command flag during deployment. This allows you to observe the system behavior and incur the cost of the detailed logging on demand, much like you would when connecting via remote debugging, but on a system of systems scale, increasing the probability of capturing the bug's root cause over stepping forward and backward through interactive execution.
Additionally, care should be taken to program in a value-centric way when you can get away with it performance or space-wise. Essentially - use FP principles of immutability and programs as values and composition to build your programs where you can. Values have a unique property in that they are serialization state that can be inspected or substituted in place of references to the value without changing the behavior of a program at runtime. This often simplifies debugging even in complex side-effecting concurrent algorithms.
When you can't, comprehensive argument and shared state change log events that can be shared amongst many members of the development, stakeholder, and operations teams asynchronously helps to narrow the surface area scan of the system you are debugging.
Formal methods during modeling, design, and development can help to increase constraints and enable more comprehensive test suites, but property-based testing with randomized generated valid inputs also serve as a sort of automated preemptive debugger. Things like chance in the js world, quickcheck/hedgehog in haskell and their ports or siblings in other languages, like hypothesis in Python, fall into this category of testing tools.
Interactive debugging is the tool of last resort, and still serves a useful purpose when all other mitigation and system state capture methods have failed, but I personally find myself believing that if a bug's root cause hasn't been discovered by this stage I have made some set of mistakes in the design of the system, either in the application of its logical rules or in the basic assumptions fom interpretation of business requirements to the choices of abstractions/tradeoffs made in development. It's usually a bad day when I have to resort to interactive debugging.
I think a "Omnicient"/"Time-travel" debugger with strong tracing support would be interesting.
I'd like to be able to take a recorded run and add tracepoints after the fact, and be able to toggle them on and off, or add or modify conditions on the output, watching the trace change as I do it.
When I looked at the Pernosco demo it looked like it had the capabilities to do this sort of thing, but only a rather primitive user interface.
Mentioned it up thread, but this is _exactly_ what our Replay time-traveling debugger for JS lets you do!
We specifically let you add "print statements" to any line of code from the recorded app, let you toggle them or change conditions, and recalculate the logged output every time you make a change:
Longer-term, we have plans to implement functionality for "persistent object IDs" and tracing individual values over time. No ETA on that, but from what our runtime team has said it's feasible. (We actually have a partial hardcoded implementation of that that we're using to track anything that looks like a React internal "Fiber" data structure, and I used that to build some of our support for the React DevTools: https://blog.replay.io/how-we-rebuilt-react-devtools-with-re... )
Reply to Grandparent and this: Very cool and definitely useful to record and replay the session and modify breakpoint conditions for replay. I wonder if session recording like this could be safely brought to backend systems. You'd almost have to stub every remote call to allow for replayability in isolation of external services, and the potential for data leakage from a downloaded session is also a concern. I'm glad someone else is working on problems and tools like this.
I should note I'm not disparaging the use of debuggers in my comments, but lamenting the nature of the process of debugging the types of systems we deploy today, and the largely non-interactive nature tradeoffs we've made for scalability, reliability, availability, and operability. Sometimes I long for the simplicity of a fully self-contained local system, but I know we just don't live in that world anymore, and haven't for a long time now.
Yeah, data leakage is a concern in the sense that _everything_ about the process gets recorded. (At Replay, all the recordings are encrypted and access-controlled: https://www.replay.io/security-and-privacy )
The way Replay specifically works is that we fork and modify the Firefox and Chrome browser and Node runtime to add the necessary support for capturing and stubbing all OS-level syscalls, as well as making other changes to ensure near-deterministic playback. So, when the recording gets replayed, the process gets all the same external inputs it did the first time:
One of our long-term goals is to bring down recording perf overhead enough that we can do "always-on" recording into a buffer, so that the browser or Node just _always_ runs in recording mode, and if something fails the last minute or so of the buffer gets uploaded for analysis.
That makes a lot of sense - having a deterministic recording makes a big difference to learning how the code actually works.
I also sometimes think that, even when I want a debugger interface to look at the details, I'd also like to see a "big picture" view showing how key values evolve over time.
(something like replay.io does for visualising logging statements is probably in about the right place for both of us, though I'm not a web dev)
20 comments
[ 4.2 ms ] story [ 61.2 ms ] threadhttps://pleiad.cl/tod/screenshots.html
Does anyone on HN have experience with any of these or other similar tools? Why don't we hear about this anymore?
Edit: Probably bookmarked it based on this old HN thread: https://news.ycombinator.com/item?id=4801710
http://jakob.engbloms.se/archives/1564
Generally speaking we do not see much mention of them because the technology involved is reasonably tricky to develop and software developers and companies are as a rule penny wise, but pound foolish with respect to tooling. So, there is very little money in actually expending the development effort to make a polished, accessible tool and any such tool will see relatively limited mass market uptake unless it is free as in beer. This largely precludes the existing relatively polished commercial offerings from achieving ubiquity and thus why it stays relatively niche and underused. That is also why despite there being so many examples over time none of them have been able to achieve escape velocity.
Literally every person who uses it for debugging a hard problem absolutely raves about it. Debugging something like a stack corruption or a heap corruption is trivial. And every other class of bugs are also incredibly easy, as long as it doesn't rely on very precise timing where the tracing changes the behavior. So why don't more people use it? Why haven't more people heard about it?
I'm not entirely sure, but I do have a guess. Most bugs are shallow. We tend to think about the really hard, really deep bugs, but the vast majority of devs are working on bugs where it's easier to just add some logging to figure out a logic error.
Of course, it means adding code to the app to output everything to debugview where as the debugger can use the debug symbols and then log everything which is alot less work, but some languages dont come with a debugger.
Stepping back in time is useful though!
It's a big deal, for example when the "app" is a third party library that cannot be altered.
It's pretty appealing. I also believe MS Visual Studio had something that effectively worked like that albeit without changing the coffee manually.
But - if you've got bugs due to memory corruption, race conditions, etc then the logging doesn't necessarily tell you what you need as you don't know when the change actually happened.
The other issue is that once you've started logging everywhere the string formatting is likely going to be a significant slowdown - at which point you could have used a time travel debugger and probably get better performance.
(disclaimer: I work on a time travel debugger)
[0] https://sourceware.org/gdb/onlinedocs/gdb/Reverse-Execution.... [1] https://rada.re/n/
Also saw someone post an indie gaming company's TTD development environment yesterday:
https://www.reddit.com/r/programming/comments/11a2meo/tomorr...
"there are no non-deterministic problems"
Edit: (disclaimer: I work on one such system)
Essentially - println debugging is superior to interactive debugging, especially for intermittent bugs, data races, deadlocks, resource leakage, modeling and serialization bugs, binary or source code incompatibility bugs, and bugs that are catastrophic and rare or pathological in nature.
The problem with interactive debugging is that you typically only have a view into not only a single process of a system, but also only a single concurrent execution thread of the single process.
When you have structured logging and aggregation and indexed log search, you have a much broader view of the systems under investigation and can see the conditions causing the bug with a much higher frequency than with interactive debugging.
For any non-trivial bug in any sufficiently large input or program you have to have insight into the root cause via intuition to cause it to occur quickly enough to inspect the program via the debugger. This obviously doesn't scale well. It's much better to have a detailed, structured log mode built into your systems that is capable of being enabled on demand - via environment variable, remote procedure call, request header, data packet, or command flag during deployment. This allows you to observe the system behavior and incur the cost of the detailed logging on demand, much like you would when connecting via remote debugging, but on a system of systems scale, increasing the probability of capturing the bug's root cause over stepping forward and backward through interactive execution.
Additionally, care should be taken to program in a value-centric way when you can get away with it performance or space-wise. Essentially - use FP principles of immutability and programs as values and composition to build your programs where you can. Values have a unique property in that they are serialization state that can be inspected or substituted in place of references to the value without changing the behavior of a program at runtime. This often simplifies debugging even in complex side-effecting concurrent algorithms.
When you can't, comprehensive argument and shared state change log events that can be shared amongst many members of the development, stakeholder, and operations teams asynchronously helps to narrow the surface area scan of the system you are debugging.
Formal methods during modeling, design, and development can help to increase constraints and enable more comprehensive test suites, but property-based testing with randomized generated valid inputs also serve as a sort of automated preemptive debugger. Things like chance in the js world, quickcheck/hedgehog in haskell and their ports or siblings in other languages, like hypothesis in Python, fall into this category of testing tools.
Interactive debugging is the tool of last resort, and still serves a useful purpose when all other mitigation and system state capture methods have failed, but I personally find myself believing that if a bug's root cause hasn't been discovered by this stage I have made some set of mistakes in the design of the system, either in the application of its logical rules or in the basic assumptions fom interpretation of business requirements to the choices of abstractions/tradeoffs made in development. It's usually a bad day when I have to resort to interactive debugging.
I'd like to be able to take a recorded run and add tracepoints after the fact, and be able to toggle them on and off, or add or modify conditions on the output, watching the trace change as I do it.
When I looked at the Pernosco demo it looked like it had the capabilities to do this sort of thing, but only a rather primitive user interface.
We specifically let you add "print statements" to any line of code from the recorded app, let you toggle them or change conditions, and recalculate the logged output every time you make a change:
https://docs.replay.io/reference-guide/print-statements
Longer-term, we have plans to implement functionality for "persistent object IDs" and tracing individual values over time. No ETA on that, but from what our runtime team has said it's feasible. (We actually have a partial hardcoded implementation of that that we're using to track anything that looks like a React internal "Fiber" data structure, and I used that to build some of our support for the React DevTools: https://blog.replay.io/how-we-rebuilt-react-devtools-with-re... )
I should note I'm not disparaging the use of debuggers in my comments, but lamenting the nature of the process of debugging the types of systems we deploy today, and the largely non-interactive nature tradeoffs we've made for scalability, reliability, availability, and operability. Sometimes I long for the simplicity of a fully self-contained local system, but I know we just don't live in that world anymore, and haven't for a long time now.
The way Replay specifically works is that we fork and modify the Firefox and Chrome browser and Node runtime to add the necessary support for capturing and stubbing all OS-level syscalls, as well as making other changes to ensure near-deterministic playback. So, when the recording gets replayed, the process gets all the same external inputs it did the first time:
- https://docs.replay.io/contribute/how-replay-works
One of our long-term goals is to bring down recording perf overhead enough that we can do "always-on" recording into a buffer, so that the browser or Node just _always_ runs in recording mode, and if something fails the last minute or so of the buffer gets uploaded for analysis.
So you can combine the best of debugging and logging.
At Undo we've done some investigation of how to get this out into the world for time travel debug, so it's interesting to see your idea lines up.
What sort of problem do you think this would be useful for?
I also sometimes think that, even when I want a debugger interface to look at the details, I'd also like to see a "big picture" view showing how key values evolve over time.
(something like replay.io does for visualising logging statements is probably in about the right place for both of us, though I'm not a web dev)