This feels like good advice not just for writing a paper but for reading one as well. Explicitly thinking about the structure of a paper certainly helps me get through it. And if something is missing--say, good examples--I know to look elsewhere for them before continuing.
Sometimes, finding a slide deck the author of the paper used is more useful than the paper itself just because they have more room for examples! I've found this particularly true for very dense theory papers where I only half-understand the relevant mathematics.
Perhaps no one is going to say it, so I'll say it: The nonchalant font makes these slides easy and fun to read.
Also, for who are not in the know, Simon Peyton Jones is a well respected computer scientist, mostly known for his work on developing the functional programming language Haskell (together with others ofcourse), Haskell research is funded for a large part by Microsoft I think.
It's also great advice. I especially liked the part at the end about using simple and direct language and the active voice. This alone makes a paper much easier to digest even if the subject matter is hard.
Precisely because he's that respected, I was surprised at the font choice, which I disliked (it just seems to mean extra mental effort for me over something I'm more used to).
He has said elsewhere, in response to questions about his choice of fonts:
This is a very funny question, why I use Comic Sans. So. All my talks use
Comic Sans, and I frequently see little remarks, “Simon Peyton Jones, great
talk about Haskell, but why did he use Comic Sans?” But nobody’s ever been
able to tell me what’s wrong with it. I think it’s a nice, legible font, I
like it. So until someone explains to me — I understand that it’s meant to be
naff, but I don't care about naff stuff, it’s meant to be able to read it. So
if you’ve got some rational reasons why I should not, then I’ll listen to them.
But just being unfashionable, I don’t care.
Yep, he once asked if there was anything he should change about this talk - I said get rid of the Comic Sans, but keep the casual feel. This was in 2010. You know that a is talk finely tuned when the only thing someone can think to change is the font!
Ha, I remember seeing him talk at a conference in 1991 and it was hand-written transparencies on an overhead projector. Comic Sans is a relief. :-)
Also, to add what others have said, he is one of the best speakers I've ever been lucky enough to see in person: SPJ, Philip Wadler, Guy Steele... good company.
This works well for giving talks as well. I have often signed up to give talks on subjects I wanted to learn more about. In fact I believe the best time to give a talk is when you're still learning a topic, as you're more enthusiastic and have any easier time empathizing with those new to the subject.
Is the suggestion of putting "related work" at the end non-standard? As an occasional reader of CS papers I generally remember this part appearing after the introduction, and feeling quite sapped once I'd got through it and the references.
I've always seen "related work" appearing immediately following the introduction. The layout I see most often is:
* Abstract: This is why you should read this paper.
* Introduction: What the problem is that we're trying to solve and a claim to having solved it better.
* Related work: How other people have solved the same problem (and why we don't like their solutions).
* One or more sections go here here describing how we solved the problem.
* Experimental results: Look at the awesome results our solution provides. (Not always applicable depending on the paper.)
* Future work: Here's some ideas we haven't worked out fully -- they may appear in future papers but we're putting them here so we can claim to have published them first if someone else writes a paper about them before we do.
* Conclusions: Everything in the Introduction, except written in the past tense.
* References: Everything your potential referees have written which might conceivably be relevant.
In PL and systems, related work almost always comes at the end. HCI is the only field I know where related work is front loaded. Pandering to the PC in related work is also bad form.
That said, a quick glance over the PC/ERC to ensure you cite-and-compare is a good idea. It's always easier to answer questions in your paper than the rebuttal. Or next year's submission.
That is an incredibly biased way of handling related work. Doing it right means being blind wrt the PC, if the conference isn't run by dysfunctional egoists and you cover related work in good faith, the reviewer won't dock too many points for missing something and they'll fill in your missing information in the review.
In AI, related work is usually at the beginning, but sometimes at the end. A common form is to set up the problem in the intro (section 1), then use the related-work section (section 2) to survey existing solutions to the problem and explain 1) whether you build on any of them, and 2) what's still left to do. It's sort of like an extended motivation: the intro motivates the problem, and the related-work section motivates your choice of jumping-off point.
We have to mention previous work in the intro and body of the paper to establish context, but the related work at the end is more of comprehensive since the paper has already been fully stated. If your paper is low on context, the reviewer might flip to related work after the intro to fill in the gaps, but you are in negative points territory at that point.
I have seen both ways (in computer science/systems). I buy the reasons SPJ says in the talk, but if the problem you are working on is too crowded that the first thing that comes to a reader's mind is -- "How is this different from the N other proposals I've read in the past?" -- the related work helps set the context.
It's a little non-standard, and some would argue that it deviates from the philosophy of "standing on the shoulders of giants", but I think it's acceptable.
Note that you should still "cite relevant work in passing".
It's probably non-standard, but I totally buy SPJ's suggestion. Often when I read the related work section at the beginning I'm confused: I don't understand the paper I'm reading yet, let alone any other related work.
Breaking with this sort of practice is exactly why agile development and it's ilk were born.
While writing a formal specification can be useful, it is rarely as useful as defining the problem and placing your solution in the context of other possible solutions.
I've been lucky enough to attend one of Peyton Jone's talks in person. He is not suggesting to "write a formal spec" before you do research. In contrast, his attempt is to be agile: keep going back and forth in the paper writing as you do research. Only then you will see what's missing.
Essentially, what he is suggesting is actually "agile academic research", compared to the "waterfall academic research" (do stuff, then write-up a paper with what you've got...)
A manual isn't particularly useful on its own, since it's full of implementation details for one particular widget. SPJ is emphasising that details and descriptions of implementations should be avoided in favour of ideas and intuitions.
A purely theoretical paper can still be useful. It's even more useful if it's followed up by an experimental paper demonstrating or refuting the idea.
For example, it's silly to write a manual "How to use Google", then make Google. It's very sensible to write a paper "A proposal for improving Web search with mutual-popularity measures", then try to make Google. If Google succeeds, we can write "Mutual-popularity measures and Web search: large-scale experimental results". If it fails, we can write "Scaling issues on the Web: impacts on mutual-popularity algorithms".
Skilled programmers use different techniques to this end: some write a first version and throw it away, some write extensive manual pages or design documents, others fill out a code template where every requirement is identified and assigned to a specific function or comment. For example, in Berkeley DB, we created a complete set of Unix-style manual pages for the access methods and underlying components before writing any code.
However, I have to issue a statement of warning that when an idea doesn't pan out like written out, people will go to great lengths to fulfill their agenda (e.g., put undue pressure on their students to come up with the "correct" result). For computer science and mathematics, it may be more difficult to fudge results. But in experimental work or data analysis, I've seen students suddenly come up with a solution that fits their advisor's expectation.
There has to be some mechanism to guard against this; the most common answer is the moral character and integrity of the scientist, but psychologists and economists will quickly point out that there are gray areas where people are willing to cheat just enough to get their reward while preserving their conscience.
This link came together at a good time because I usually get bogged down in the research stage. I always over research and then I lose hope and confidence in myself and mankind.
I actually apply this to my hacking these days. The statement you often realize that problems are more interesting once you sit down and analyse them I have found to be particularly true.
When I'm faced with a design problem I write a "white paper" setting out the problem, framing the desired properties of any solution then as systematically as possible setting out the alternatives and justifying why a particular solution has been chosen.
I usually find that in going through this process I am forced to change my initial design decisions, often ones that seemed obviously, even trivially, correct to begin with.
He actually addresses this question. His advice is: start with any idea, even a bad one, and write about it. You're ideas will develop (and new ones will emerge) as you go. (Thoughtful) writing is the discovery mechanism.
I have to say that this sounds like a good idea to me.
A few months ago I found a way to reduce a particular AI algorithm's memory usage from O(n) to O(1), but I've been bogged-down in irrelevant details when trying to implement this improved version.
I think I'll take SPJ's advice and write up a theoretical/motivational paper describing the improvement, and leave the actual implementation for afterwards.
I've been holding off writing up since I'm an amateur, and AI has a reputation for attracting cranks with hand-wavey conjectures :S
I'm not the best at writing the papers, but this is precisely how I approach giving talks. I take my idea, submit the talk, then as I build the talk I understand my idea better and better and finally, when I am on stage, explaining my idea using the structure I developed while preparing the talk, I have moments of inspiration that I never get elsewhere.
I have, mid-sentence, come to understand truths about my subject that had evaded detection for months prior. My talks thrive, but now I need to transition to papers. I'm going to try to put this method into practice for myself.
I seem to get the active/passive voice criticism a lot. It comes from both sides of the table. Do you have references on what to prefer? They can be opinionated, but preferably I would like to see something backed up by a reading comprehension study. Thanks in advance.
Pretty cool. Hopefully some of the "lol acadmics" folks on HN will see how good research practices are very similar to startups (and yes that would be the biggest asset of any good academic you hire) :)
yep. getting stuff done in academia today is quite similar to startups, s/investors/government-agencies, s/grants/investment-rounds, s/papers/pitches, s/prototypes/MVPs, s/hiring-students-and-postdocs/hiring-employees, etc.
I want to point out slide 21: No “rest of this paper is...”
This practice drives me crazy, because it's essentially just an outline, which is useless. It's a computer science research paper, I already know the outline! His advice to not have that paragraph, but instead put in forward references to the relevant sections in the introduction is spot-on.
Now if you'll excuse me, I have to get back to reviewing papers which did not take this advice...
I wish he also recommended that every research paper needs to have a date on it. It seems to be the golden rule never to specify any date on research papers, which makes it really hard to evaluate the paper based on when it was published.
I agree, all papers should specify the date. Unfortunately, when submitting to journals or conferences, the author often doesn't have a choice in the matter, as papers have to follow a given template.
Another way to look at this, which is how I learned it in grad school, is that your goal is to tell a Story that others Believe. The Machievellian corollary is: whether the story itself represents reality is less important than whether you have people who believe in your story. When people believe in your story, you have power. You use that power to get funding, which gives you runway to work on real problems.
I think you are seeing a dichotomy where there is none. Simon Peyton Jones is advocating a process for how to go about doing research and communicating your ideas. Bret Victor is advocating a method for better communicating ideas. One could easy do both.
63 comments
[ 3.4 ms ] story [ 130 ms ] threadSometimes, finding a slide deck the author of the paper used is more useful than the paper itself just because they have more room for examples! I've found this particularly true for very dense theory papers where I only half-understand the relevant mathematics.
Highly recommended. Simon is a very good speaker.
Also, for who are not in the know, Simon Peyton Jones is a well respected computer scientist, mostly known for his work on developing the functional programming language Haskell (together with others ofcourse), Haskell research is funded for a large part by Microsoft I think.
(I probably shouldn't be making any handwriting/typography comments.)
Also, to add what others have said, he is one of the best speakers I've ever been lucky enough to see in person: SPJ, Philip Wadler, Guy Steele... good company.
* Abstract: This is why you should read this paper.
* Introduction: What the problem is that we're trying to solve and a claim to having solved it better.
* Related work: How other people have solved the same problem (and why we don't like their solutions).
* One or more sections go here here describing how we solved the problem.
* Experimental results: Look at the awesome results our solution provides. (Not always applicable depending on the paper.)
* Future work: Here's some ideas we haven't worked out fully -- they may appear in future papers but we're putting them here so we can claim to have published them first if someone else writes a paper about them before we do.
* Conclusions: Everything in the Introduction, except written in the past tense.
* References: Everything your potential referees have written which might conceivably be relevant.
Note that you should still "cite relevant work in passing".
Basically, if you put a huge discussion of related work at the beginning you'll either make your reader feel tired or stupid or both.
Breaking with this sort of practice is exactly why agile development and it's ilk were born.
While writing a formal specification can be useful, it is rarely as useful as defining the problem and placing your solution in the context of other possible solutions.
Essentially, what he is suggesting is actually "agile academic research", compared to the "waterfall academic research" (do stuff, then write-up a paper with what you've got...)
A purely theoretical paper can still be useful. It's even more useful if it's followed up by an experimental paper demonstrating or refuting the idea.
For example, it's silly to write a manual "How to use Google", then make Google. It's very sensible to write a paper "A proposal for improving Web search with mutual-popularity measures", then try to make Google. If Google succeeds, we can write "Mutual-popularity measures and Web search: large-scale experimental results". If it fails, we can write "Scaling issues on the Web: impacts on mutual-popularity algorithms".
Google’s search engine, however, is, and it would be perfectly sensible to first write a manual for a search engine before implementing it.
Skilled programmers use different techniques to this end: some write a first version and throw it away, some write extensive manual pages or design documents, others fill out a code template where every requirement is identified and assigned to a specific function or comment. For example, in Berkeley DB, we created a complete set of Unix-style manual pages for the access methods and underlying components before writing any code.
However, I have to issue a statement of warning that when an idea doesn't pan out like written out, people will go to great lengths to fulfill their agenda (e.g., put undue pressure on their students to come up with the "correct" result). For computer science and mathematics, it may be more difficult to fudge results. But in experimental work or data analysis, I've seen students suddenly come up with a solution that fits their advisor's expectation.
There has to be some mechanism to guard against this; the most common answer is the moral character and integrity of the scientist, but psychologists and economists will quickly point out that there are gray areas where people are willing to cheat just enough to get their reward while preserving their conscience.
When I'm faced with a design problem I write a "white paper" setting out the problem, framing the desired properties of any solution then as systematically as possible setting out the alternatives and justifying why a particular solution has been chosen.
I usually find that in going through this process I am forced to change my initial design decisions, often ones that seemed obviously, even trivially, correct to begin with.
I believe the updated link would be http://http://www.cs.cmu.edu/~mleone/how-to.html
A few months ago I found a way to reduce a particular AI algorithm's memory usage from O(n) to O(1), but I've been bogged-down in irrelevant details when trying to implement this improved version.
I think I'll take SPJ's advice and write up a theoretical/motivational paper describing the improvement, and leave the actual implementation for afterwards.
I've been holding off writing up since I'm an amateur, and AI has a reputation for attracting cranks with hand-wavey conjectures :S
I have, mid-sentence, come to understand truths about my subject that had evaded detection for months prior. My talks thrive, but now I need to transition to papers. I'm going to try to put this method into practice for myself.
See also http://research.microsoft.com/en-us/um/people/simonpj/papers...
This practice drives me crazy, because it's essentially just an outline, which is useless. It's a computer science research paper, I already know the outline! His advice to not have that paragraph, but instead put in forward references to the relevant sections in the introduction is spot-on.
Now if you'll excuse me, I have to get back to reviewing papers which did not take this advice...
On the other hand, a paper that is 80% typewriter font (or 80% mathematics) is a paper for putting down and leaving alone.
o_O
http://worrydream.com/#!/ScientificCommunicationAsSequential...