To be fair, people also think "rest and relaxation" is amazing, and we probably need to capture some of the value of this technology via a for-profit company as well.
I've never heard about rr http://rr-project.org/), his tool to record program execution and debug it afterwards. The alleged 1.2x slowdown in execution is negligible. The ability to reverse in time is almost a nirvana. I'm sad I didn't have it when I used to program C++. Hope he succeeds in his new endeavor. We'd all win big.
Does anyone knows of similar tools for Python and in-browser Javascript?
I understand you never claimed they are similar. But to be fair (and with due credit to redux for being a great framework), all redux does is save method call arguments in an array. rr is on a different scale altogether.
the python case is interesting ... after all, the interpreter is just another binary executable. perhaps there's a way to write a layer above rr to "step" in Python code rather than the underlying C code. same with JS (Node.js) or any other language run-time.
Python and in-browser Javascript are also programs, in theory rr covers the entirety? But I don't know what the ergonomics are like (e.g., how much will you see program activity in your platform that you don't care about, and how well can you see the activity in the code you are debugging?)
Yes, rr can record python and JS execution just fine, but the debugging experience is hopeless, since right now rr just gives you gdb. With an astounding tolerance for pain you can make progress by calling JS engine functions from gdb, but nobody wants that.
TimeMachine needs a specialized hardware probe connected via JTAG, if I'm not mistaken. The probe is powerful enough to record everything (registers, operations, etc) in its memory without affecting the embedded device. Then you can replay the info however you want, in as much detail as you like.
The feature here is more like software emulation, and it will really allow you to take the instruction pointer in reverse direction. Unlike TimeMachine which records in real time and let's you browse the recorded data as you wish, but after execution has occurred.
My thoughts were very similar to yours. In 1992, when I started out in embedded on a Philips 8051 and debugging it was via an ICE and had direct visibility and control of the 8051.
The ICE would be connected to a bonded out version of the 8051 and I was almost able to capture every single clock cycle of 8051's execution. When I hit a failure, I would just replay the trace and see what went wrong.
It looks like the folks on the big machines have caught up :-)
I had access to a Huntsville Microsystems ICe that used a bog-standard - non-bonded-out - 68008 in harness. I had scripting tools that would capture and replay all the registers to back the machine up.
That HMI was a brilliant machine.
It's really the last debug tool I ever used besides AVR, PIC and one ARM JTAG for doing BSP work.
If I'm understanding this correctly, then Java has had it for ages now via the "Chronon" debugger. It's the only reason why I pay for an IntelliJ Ultimate Edition license, since it comes bundled.
I'm building something like this for in-browser Javascript, but in a larger context. We're not ready for public consumption yet, but: https://fabric.fm
I was really impressed when finding out about Microsoft's Chakra JS engine Time travelling debugger[1], I think that's what you're looking for, haven't tried it myself though.
It's an interesting different approach because it allows you to rewind the entire machine, so you can interact with interactions between different processes, etc (happy to answer questions, I am a dev on it)
I wonder how it works. Assuming it must capture state at t=n, and somehow calculate the diff to get to t=n+1. Could theoretically pop diffs to get to any point in the past.
I've used it, and I've been using rr. gdb's reverse debugging was useful to me a few times, in extremely limited situations.
I often do my initial runs under rr, just in case I might want to debug, because it's useful most of the time and it's not enough overhead to worry about.
gdb's reverse debugging works after you narrow the range of interest down to a very small region, because otherwise its buffer will repeatedly overflow and it'll take ages.
Yes, per the blog post he plans to spend more time working on rr than he thinks it's reasonable for Mozilla to have to support:
And now there's rr. People think it's amazing, and it is, but we're only scratching the surface of what could be done. I'm bursting with ideas, and realizing them will require a lot of resources, resources that Mozilla cannot and should not provide --- we probably need to capture some of the value of this technology via a for-profit company. We can change the way people debug software, and in its own way that may be as important as my Web platform work, and it's work I desperately want to do.
From reddit to answer the question some might pose ("who?"):
> One of Mozilla's most senior engineers who has worked on Mozilla projects as a contributor and employee for well over a decade particularly working in the graphics code and helping make native audio and video in the browser a reality.
I suppose it's nice to know that he is a big deal but as someone who doesn't follow mozilla religiously a little more background is always appreciated.
Why is he leaving? That's covered in the OP: he wants to devote his full-time attention to working on rr, a reversible debugger for native code that imposes impressively low overhead, and possibly spin it into a startup.
I think roc would give me a sly smile for pointing out that this doesn't really answer the "why". It's what he's going to do after he leaves. Doesn't quite answer why he's leaving.
My, cynical much? Seems to me that 10+ years at a non-profit, and choosing to live in a place outside the traditional startup locations (New Zealand) with fewer networking opportunities, might indicate fairly little interest in that exact trajectory, for those exact reasons.
But what do I know, I'm just another ten-year Mozillian myself. And if or when I leave, it'll also surely be to get filthy rich at a startup, right?
Well, it'd be foolish to assume that someone doesn't have multiple reasons for making a dramatic change in their life.
It'd also be foolish to assume that most of those reasons are any of our business.
Robert is an amazing guy, and we're all lucky to have benefited from his work over the years. Now, he's set off on a new adventure, for reasons of his own... And has hinted that he'll continue sharing some of the fruits of his labor.
There were about ten NZ Mozilla employees when I worked there a year ago. Not all at the Auckland office but there were plenty there to keep the office utilized at the time.
I worked with Robert at Mozilla for many years. He might be the most talented software engineer I've ever met, and he's a strong candidate for nicest person as well. An absolute pleasure to work with. Mozilla will miss him dearly but some other project (apparently rr) just got really lucky.
"Christian" ... Since when has it become fashionable to define oneself by religion? I mean especially in professional circles like software development. I don't expect to see "Jew" or "Moozlim" (sic) or "Amoral Atheist" used to define people. Lots of respect for his work but I it does turn me off because of what it might imply As far as personal biases.
Christians of all denominations actively proselytise - it's a core part of being a Christian, spreading the 'good news'. This distinguishes the Christian ethos from most other mainstream religions which have a weak or non-existent missionary instinct.
So no, it's not unusual at all for a Christian to highlight the fact. From the point of view of a Christian in an increasingly secular world it probably feels more necessary than ever.
Most strains of Christianity will have some form of proselytizing, though it might be muted. Plus there also tends to be ebbs and flows in the history and culture of such things. It's interesting to note that proselytizing arguably has a very significant (positive) effect on formation of liberal democracies [1].
It's the bio in his personal blog. He can put whatever he wants in it. And I don't think he includes it because it is "fashionable", but rather a reality that a true faith defines who you are and how you operate.
Is it different from calling oneself an abstract impressionist? Or a functional programmer? Christian teachings can absolutely be applied in software. I mean, look no further than the Temple operating system. Christianity is not a "skin" on that project, it's intrinsic to its design.
I always found it odd, especially because he lives in NZ which is a very secular country. For instance no serious politicians will ever admit to their religiosity or lack thereof. Therefore it's a very unfashionable thing to announce in the context of where he lives.
Although an Atheist I kind of have to admire the fact that he believes so much as to put it out there and have it define him, especially when it is so unfashionable here.
I also find it interesting when clearly rational and intelligent people have faith. It's kind of something that you would expect from people who lack in education, as it is a crutch. But I guess that's why it's called faith, it defies logic at least by my way of thinking.
Not meaning to offend anyone with the above, so sorry if I did.
I'm not entirely sure what you think isn't offensive about your comment.
1. You are responding to the comment "Since when has it become fashionable to define oneself by religion?" (aside: the answer is since forever) in the affirmative
2. You find it "interesting" that "clearly rational and intelligent people have faith", by which you say that most people who have faith (and by association, Christians) are irrational and stupid.
3. You say that you believe that you expect those who have faith must be seriously uneducated.
4. You say that people who have faith use it as a crutch.
I am offended by you saying that people who have faith aren't irrational. That goes against my worldview and is therefore offensive to me.
Irrationality is cognition, thinking, talking or acting without inclusion of rationality. It is more specifically described as an action or opinion given through inadequate use of reason, emotional distress, or cognitive deficiency.
I'm not familiar with Robert or his work, but I quite liked this post. The brevity and honesty was refreshing. I couldn't help but feel excited at the end!
> we probably need to capture some of the value of this technology via a for-profit company
I hope this doesn't mean making it proprietary. I mean obviously a person gets to decide how to best compensate themselves for the fruits of their labor. But I've seen too many compelling technologies fail to achieve the impact they could have because they were proprietary.
I mean, DTrace languishes despite being open-source, because it wasn't open-source enough
It wouldn't be distributed as a kernel module (DTrace probes are far more invasive than that). But yes, I posted this before I realised that Oracle Linux has sublicensed it under the GPL in their port to Linux (for their customers), so it's fine for us to do the same as long as we "base it on their port".
Let me put it this way. In the last say five years, I can't think of a single article I've come across where someone describes DTrace making their life better (except one I wrote in 2011: http://blog.reverberate.org/2011/05/using-dtrace-on-os-x-to-...), or a single time anyone I know or work with has even mentioned DTrace.
This is a technology that received an innovation award from the Wall Street Journal in 2006. Whatever impact DTrace is having today, it should have been much larger.
Dtrace isn't completely dead; it's used to great effect as the underlying technology behind Instruments (performance analyzer/visualizer) on OSX.
It never has really managed to catch on as a standalone tool by itself, though. Part of that's the licensing situation on Linux, but I'd hypothesize that part of it is also that it's a pretty complicated tool to use, interpret, and act on-- you don't hear a lot about SystemTap (same concept, GPLed for Linux-friendliness) these days, either.
I'd push back on the 'complicated' part. The dtrace language is heavily based on awk, which every modern *NIX ships with, and that every software engineer and sysadmin have at least a passing familiarity with.
The main difference between the two is whereas awk is meant to operate over a stream of semi-structured lines, dtrace operates over a stream of probe firings. The high-level structures of the scripts, though, are the same:
/pattern/ { statements }
In a awk, when a line matches a the patter, the corresponding statements fire. In dtrace, when a probe matches a pattern, the corresponding statements fire. Integers, strings, arrays, and all those other sorts of features of the language are the same (though there are some special functions introduced by dtrace to help you collect and visualize aggregations).
So from a language perspective, if you know awk, you mostly know dtrace. All that's left to do is learn the pattern language. If you want to do that effectively, you have to know not just the syntax of the patterns, but useful probes to listen for, and useful ways to use the information that's gleaned.
This involves knowing things about operating systems. Most software engineers don't know much about operating systems so they think the tool is stupid.
I use DTrace fairly often to look at the boundaries between whatever interpreted language my app's written in and the kernel.
For example, at $job recently found that some of our Python code was doing 25k open64 calls but only 5k close calls during a scheduled job - this led us to find/fix a bug that would have been a disaster later.
I think you've accidentally set up a straw man because mentions of low level tools don't correlate to use. In fact "trending" popularity is how a lot of bad technical decisions are made, cue crappy database flavor of the month. The hammer is a pretty useful tool but builders don't blog du jour about how elegant combining the strike face and nail puller on one implement are. None the less, millions are used every day.
DTrace is used extensively in FreeBSD development these days.
It's being used to bring up new generations of Systems Programmers: http://teachbsd.org/
My team uses DTrace to develop TCP, which ensures open standards and a copyfree implementation that ends up in Apple products, Juniper, NetApp, and who knows where else. Of course, also Netflix and the large CDN I work for which run at least 1/3 of North American Internet volume.
We also use DTrace to fix the reference copyfree Intel drivers, which may in a small way pay back Illumos for the awesome tech they share. I think those end up in some popular embedded microkernels too.
I grin thinking about people that emphasize "GNU/Linux", but I will empathize with FSF for a moment to show that low level software is not particularly exciting to most. GCC and GDB are hugely influential tools for the whole open source ecosystem, especially the high growth during the 80s and 90s that led to "Enterprise" acceptance. It's hard to imagine where everyone else would be, including the BSDs and many chip companies, were it not for the GNU toolchain. Nonetheless, most people just begrudgingly use these and hope they don't give them any trouble because of course they are usually trying to do something else.
I use dtrace often, but am too busy to write articles about it, sorry.
A couple of times, dtrace has even been THE tool that helped me understand exactly why our big production systems were dying seemingly inexplicable and horrible deaths. This has directly resulted in us still being in business instead of fading into irrelevance. That's plenty impact for me.
> Let me put it this way. In the last say five years, I can't think of a single article I've come across where someone describes DTrace making their life better (except one I wrote in 2011: http://blog.reverberate.org/2011/05/using-dtrace-on-os-x-to-...), or a single time anyone I know or work with has even mentioned DTrace.
Like others who replied to you, I use DTrace every day. Very often, it solves in seconds what would have taken hours without it. Frequently, it's helped solve problems I'd never have been able to understand without it. My colleagues use it all the time as well. Sometimes, I write about the general techniques (I wrote the "runtime log snooping" blog post above), and most of our bug reports in recent years are public, but we don't write about individual bugs all that often. (Seems like you wouldn't have run into them if we had, though.)
I don't blog, and therefore have never written about the multiple times DTrace helped me and saved me a lot of time.
It also made my job fun again as a sysadmin.
Here are couple of examples I remember, from my days at a large financial institution.
We're planning a samba upgrade on a server with ~700 clients.
Non regression tests show the performance of copying large files has degraded by 30%
Dtrace, aggregating on smbd stacks, helped us figure out the root cause in half an hour. This was reported and eventually properly patched upstream.
We started to dtrace most large batch jobs or apps, and found lots of easy performance fixes. In house apps launching thousands of 'grep", "cut", etc.. with a LD_LIBRARY_PATH set to search over 50 directories OVER NFS. haha, libc's not there, not there.. ! ohhh there it is.
Identifying stupid behavior in vendor apps for which we had no source. (but could complain about and sometimes get a fix)
NIS groups (yeah Im old) slowing down NFS a lot, etc well, I guess I used it with profit at least once a week.
You could probably do some of what DTrace does with other tools, but: it was safe in production, it had little overhead, and it was fun :-)
Ah! I also used it to teach our interns a little about the OS internals. fbt probes with flowindent, watch them go "ahhhhhh that's what happens!".
My current job is all about linux and I wish everyday for something half as good.
So yeah, "it made my life better" is an understatement.
There's a decent market for selling consulting consulting services on top of open source technology. Being the expert on applying an open source technology to solve specific problems comes with a decent rate card.
You're still selling your labour rather than a product, though. It's possible to be well off doing that, but basically impossible to become ludicrously rich.
He didn't say his goal is to become ludicrously rich. He said he has ideas that Mozilla can't and shouldn't provide resources to realize, so some for profit company would work better for it.
There are multiple ways to fail at spreading technology. One way is to choose a business model that makes it difficult for potential users to actually use the technology. Another way is to choose a business model that doesn't generate enough revenue to bring the technology to its full potential.
Collabora does for-profit work on LibreOffice, but integrates their work into LibreOffice's master. Mostly they even do this fairly openly and don't batch up commits then push them but make their changes small(ish) then push them.
You can definitely be a for-profit company AND work in the open.
Perhaps they don't want to end up having to beg [0, 1] for donations so that they can maintain a high quality application while also being able to afford trivial niceties like "food" and "housing."
Aha. Probably some co-worker noticed his self-description on his blog. Mozilla is staffed by open-minded Californians who regrettably feel compelled to burn wrongthinkers at the stake.
Reverse debugging has been around for several years, but hasn't seemed to be widely adopted. Which seems kind of unfortunate, as the concept of it looked really useful.
Does rr make reverse debugging much easier than it's previously been in gdb or something? :)
rr (http://rr-project.org/) is a piece of magic. I've heard lots of people talk about using it in wild and awesome ways.
The second part of a magic trick is to connect it to your audience. Making time-travel style debugging is... not easy, but possible. Making it relevant to people, across languages and other conceptual domains, is hard.
If your next work is tackling this, good luck and godspeed :)
115 comments
[ 4.2 ms ] story [ 212 ms ] threadI wish him the best of luck. rr is really awesome and deserves even more attention.
(http://rr-project.org/)
So no, he is not leaving for "rest and relaxation"
Does anyone knows of similar tools for Python and in-browser Javascript?
This is fixable, however, and on my agenda.
The feature here is more like software emulation, and it will really allow you to take the instruction pointer in reverse direction. Unlike TimeMachine which records in real time and let's you browse the recorded data as you wish, but after execution has occurred.
The ICE would be connected to a bonded out version of the 8051 and I was almost able to capture every single clock cycle of 8051's execution. When I hit a failure, I would just replay the trace and see what went wrong.
It looks like the folks on the big machines have caught up :-)
That HMI was a brilliant machine.
It's really the last debug tool I ever used besides AVR, PIC and one ARM JTAG for doing BSP work.
https://developer.mozilla.org/en-US/docs/Mozilla/Projects/We...
https://bugzilla.mozilla.org/show_bug.cgi?id=1207696
It looks like Microsoft is implementing a similar time-travel feature in Chakra:
https://github.com/Microsoft/ChakraCore/commits/TimeTravelDe...
Basically it's an extension of pdb that allows you to jump back and forth in time.
[1] https://youtu.be/1bfDB3YPHFI?t=304
It's an interesting different approach because it allows you to rewind the entire machine, so you can interact with interactions between different processes, etc (happy to answer questions, I am a dev on it)
I often do my initial runs under rr, just in case I might want to debug, because it's useful most of the time and it's not enough overhead to worry about.
gdb's reverse debugging works after you narrow the range of interest down to a very small region, because otherwise its buffer will repeatedly overflow and it'll take ages.
[1] http://caml.inria.fr/pub/docs/manual-ocaml/debugger.html
And now there's rr. People think it's amazing, and it is, but we're only scratching the surface of what could be done. I'm bursting with ideas, and realizing them will require a lot of resources, resources that Mozilla cannot and should not provide --- we probably need to capture some of the value of this technology via a for-profit company. We can change the way people debug software, and in its own way that may be as important as my Web platform work, and it's work I desperately want to do.
> One of Mozilla's most senior engineers who has worked on Mozilla projects as a contributor and employee for well over a decade particularly working in the graphics code and helping make native audio and video in the browser a reality.
http://www.reddit.com/r/firefox/comments/48ix55/robert_ocall...
"Why?"
I suppose it's nice to know that he is a big deal but as someone who doesn't follow mozilla religiously a little more background is always appreciated.
But what do I know, I'm just another ten-year Mozillian myself. And if or when I leave, it'll also surely be to get filthy rich at a startup, right?
It'd also be foolish to assume that most of those reasons are any of our business.
Robert is an amazing guy, and we're all lucky to have benefited from his work over the years. Now, he's set off on a new adventure, for reasons of his own... And has hinted that he'll continue sharing some of the fruits of his labor.
That's more than enough ;)
So no, it's not unusual at all for a Christian to highlight the fact. From the point of view of a Christian in an increasingly secular world it probably feels more necessary than ever.
P.S. I am a Christian by birth but not vocation.
[1]: http://isites.harvard.edu/fs/docs/icb.topic965299.files/Wood...
Ughh. One would hope you would pick a better example than workings of a confirmed schizophrenic with an anger management issue who "talks to God".
If I were to pick an example it would be something like Larry Wall's Perl treatises.
If you don't know what Hillsong is... well.. Wikipedia. :)
What is it with Mozilla people and this sort of stuff?
Although an Atheist I kind of have to admire the fact that he believes so much as to put it out there and have it define him, especially when it is so unfashionable here.
I also find it interesting when clearly rational and intelligent people have faith. It's kind of something that you would expect from people who lack in education, as it is a crutch. But I guess that's why it's called faith, it defies logic at least by my way of thinking.
Not meaning to offend anyone with the above, so sorry if I did.
1. You are responding to the comment "Since when has it become fashionable to define oneself by religion?" (aside: the answer is since forever) in the affirmative
2. You find it "interesting" that "clearly rational and intelligent people have faith", by which you say that most people who have faith (and by association, Christians) are irrational and stupid.
3. You say that you believe that you expect those who have faith must be seriously uneducated.
4. You say that people who have faith use it as a crutch.
Hope this helps.
I just wanted to share my observations but not push an agenda. Obviously topics are sensitive which makes rational discussions impossible.
Good Karma all around.
I am offended by you saying that people who have faith aren't irrational. That goes against my worldview and is therefore offensive to me.
Irrationality is cognition, thinking, talking or acting without inclusion of rationality. It is more specifically described as an action or opinion given through inadequate use of reason, emotional distress, or cognitive deficiency.
And I'm curious, where does it "define" him? he posted a Hillsong song.
No respect for your comment I'm afraid, as it shows your own personal biases very clearly.
> we probably need to capture some of the value of this technology via a for-profit company
I hope this doesn't mean making it proprietary. I mean obviously a person gets to decide how to best compensate themselves for the fruits of their labor. But I've seen too many compelling technologies fail to achieve the impact they could have because they were proprietary.
I mean, DTrace languishes despite being open-source, because it wasn't open-source enough
This is a technology that received an innovation award from the Wall Street Journal in 2006. Whatever impact DTrace is having today, it should have been much larger.
In case others are wondering: https://en.wikipedia.org/wiki/Bryan_Cantrill
It never has really managed to catch on as a standalone tool by itself, though. Part of that's the licensing situation on Linux, but I'd hypothesize that part of it is also that it's a pretty complicated tool to use, interpret, and act on-- you don't hear a lot about SystemTap (same concept, GPLed for Linux-friendliness) these days, either.
So from a language perspective, if you know awk, you mostly know dtrace. All that's left to do is learn the pattern language. If you want to do that effectively, you have to know not just the syntax of the patterns, but useful probes to listen for, and useful ways to use the information that's gleaned.
This involves knowing things about operating systems. Most software engineers don't know much about operating systems so they think the tool is stupid.
For example, at $job recently found that some of our Python code was doing 25k open64 calls but only 5k close calls during a scheduled job - this led us to find/fix a bug that would have been a disaster later.
DTrace is used extensively in FreeBSD development these days.
It's being used to bring up new generations of Systems Programmers: http://teachbsd.org/
My team uses DTrace to develop TCP, which ensures open standards and a copyfree implementation that ends up in Apple products, Juniper, NetApp, and who knows where else. Of course, also Netflix and the large CDN I work for which run at least 1/3 of North American Internet volume.
We also use DTrace to fix the reference copyfree Intel drivers, which may in a small way pay back Illumos for the awesome tech they share. I think those end up in some popular embedded microkernels too.
I grin thinking about people that emphasize "GNU/Linux", but I will empathize with FSF for a moment to show that low level software is not particularly exciting to most. GCC and GDB are hugely influential tools for the whole open source ecosystem, especially the high growth during the 80s and 90s that led to "Enterprise" acceptance. It's hard to imagine where everyone else would be, including the BSDs and many chip companies, were it not for the GNU toolchain. Nonetheless, most people just begrudgingly use these and hope they don't give them any trouble because of course they are usually trying to do something else.
Then you haven't been looking:
http://dtrace.org/blogs/brendan/2012/08/09/10-performance-wi...
http://dtrace.org/blogs/brendan/2014/02/11/another-10-perfor...
https://www.joyent.com/blog/node-js-in-production-runtime-lo...
https://twitter.com/b0rk/status/681536144362848257
https://speakerdeck.com/sartak/dtrace-war-stories
https://www.bignerdranch.com/blog/hooked-on-dtrace-part-1/
https://medium.com/@jlouis666/erlang-dirty-scheduler-overhea...
Seriously, it wasn't that hard to find this stuff. I found the last one by searching HN: https://hn.algolia.com/?q=dtrace
Like others who replied to you, I use DTrace every day. Very often, it solves in seconds what would have taken hours without it. Frequently, it's helped solve problems I'd never have been able to understand without it. My colleagues use it all the time as well. Sometimes, I write about the general techniques (I wrote the "runtime log snooping" blog post above), and most of our bug reports in recent years are public, but we don't write about individual bugs all that often. (Seems like you wouldn't have run into them if we had, though.)
It also made my job fun again as a sysadmin.
Here are couple of examples I remember, from my days at a large financial institution.
We're planning a samba upgrade on a server with ~700 clients. Non regression tests show the performance of copying large files has degraded by 30% Dtrace, aggregating on smbd stacks, helped us figure out the root cause in half an hour. This was reported and eventually properly patched upstream.
We started to dtrace most large batch jobs or apps, and found lots of easy performance fixes. In house apps launching thousands of 'grep", "cut", etc.. with a LD_LIBRARY_PATH set to search over 50 directories OVER NFS. haha, libc's not there, not there.. ! ohhh there it is.
Identifying stupid behavior in vendor apps for which we had no source. (but could complain about and sometimes get a fix)
NIS groups (yeah Im old) slowing down NFS a lot, etc well, I guess I used it with profit at least once a week.
You could probably do some of what DTrace does with other tools, but: it was safe in production, it had little overhead, and it was fun :-)
Ah! I also used it to teach our interns a little about the OS internals. fbt probes with flowindent, watch them go "ahhhhhh that's what happens!".
My current job is all about linux and I wish everyday for something half as good. So yeah, "it made my life better" is an understatement.
You can definitely be a for-profit company AND work in the open.
Why?
[0] https://www.propublica.org/article/the-worlds-email-encrypti...
[1] http://arstechnica.com/information-technology/2014/04/tech-g...
Does rr make reverse debugging much easier than it's previously been in gdb or something? :)
The second part of a magic trick is to connect it to your audience. Making time-travel style debugging is... not easy, but possible. Making it relevant to people, across languages and other conceptual domains, is hard.
If your next work is tackling this, good luck and godspeed :)