We were frustrated by the experience of reading machine learning papers on screens (particularly phones/tablets). There are lots of good tools for authoring HTML papers (Distill, Authorea, etc) but nothing that deals with the vast number of PDF papers that already exist.
So, we built Arxiv Vanity: a site that renders Arxiv papers as web pages. It’s still pretty janky, but for the papers that do render correctly, the experience is so much better than reading a PDF. For example:
I was looking for a way to turn my (soon-to-be-defended) PhD thesis into an epub, and investigated the various LaTeX2Html converters.
I was pretty disappointed when I realized that all of them are terrible and have no hope of handling my manuscript. My current solution is to create a rendering of my thesis in a5 format. :/
This look quite a bit better, so here is the question: what do you not support at the moment?
A lot of things. LaTeX and its packages have so much surface area. Our approach so far is to just make the papers that we read readable. That probably covers the 20% of LaTeX features that 80% of people use.
One final thing, and wildly off topic, is that when you do your defense, remember that you probably know more about the specifics of the subject than anyone else in the room. Many folks stress over it, but you're almost certainly going to be the actual expert in the room. Good luck!
bioRxiv doesn't expose LateX files, they explicitly only use PDFs to make things easier. Which means you're going to need to reflow PDFs (a la https://docushow.com/), and I would guess there are a lot more edge cases there
Thanks for linking to https://docushow.com
Also a work in progress, but PDF reflow is a hard problem so you never ship if you want to solve all cases :)
Your solution using the LaTex source generates really nice HTML, congrats!
In all three cases I find the original PDFs more pleasant to read. HTML typography is not up to snuff. I read them on a laptop, however, and I can see that this would be useful if one is forced to read on a phone.
(One thing that is very ugly in the PDFs, and most scholarly papers, is the use of different-colored boxes for hyperlinks. Authors, please consider putting
For me the PDF was fuzzier https://imgur.com/a/WJ5y3 and the HTML version was more convenient to read in a single column. The two-column format is nice if I'm skimming to see if a paper is going to be interesting, but when I sit down to read it the HTML version definitely wins.
One of the things that I came across when writing my own janky pdf/latex->html converter for lecture notes[0] is that Pandoc doesn't handle references and subfigures correctly, even with pandoc-crossref and pandoc-citeproc enabled. I had to write a little python module[1] that used regex to extract those and then handle them on my own separately... This is definitely something you should look at.
I actually made one quickly, published here: https://chrome.google.com/webstore/detail/arxiv-vanity-plugi... . It injects the arxiv vanity link on abstract pages and if you click the button when viewing an online arxiv pdf it opens the respective arxiv vanity link.
I'm not sure I understand. Well, I understand what you're doing but I'm not sure why you'd dislike PDF.
PDF has the great benefit of rendering the same on every system. With very few exceptions, PDF will look exactly the same on every system and will print the same on every system.
HTML doesn't really have that same benefit.
Don't get me wrong, I think your service is a great idea for those who would like HTML formatted results, but I'm not understanding the complaint about PDF.
I think part of the issue is that PDF isn't responsive to the size of the device. A PDF is not much more than an image from the perspective of layout. I'd love to be able to reflow text from a PDF such that a single column fills my screen edge-to-edge and scrolling allows me to advance through the paper, as opposed to requiring me to reposition the viewport every time I reach the end of a column. I know this isn't the purpose of PDFs, and I love them in different contexts where layout (including typography) does matter to me. But I also really want to be able to easily consume papers in a way that isn't constrained by the PDF layout.
Yup. I agree. I even find the experience fantastic on the original iPad, as well as a brand new one. I find it just fine on my phone, which isn't nearly as large a screen.
I am guessing it is an individual taste thing. That makes some sense.
Yeah fair point, I use an iPad mini. But I have heard similar complaints from older folks (40+) who have full-size iPads. I think much of it stems from dual-column printing, which is just kind of antiquated/annoying on digital.
Fwiw, this conversation greatly varies depending on who is doing the reading -- the rather banal fact is that the average 25-year-old student has much different ability to screen-read than the average 60-year-old professor (or a 60-year-old student, for that matter) :)
so no need to search tablet specs for the culprit. PEBCAK :)
The screen of a tablet is large enough to display a PDF. But PDFs are split into pages. That's perfect with paper, where we flip pages. It's very unnatural on screens, especially touchscreens, where we use vertical scrolling to move around.
Then there are minor issues of margins, possibly zooming to make text readable, etc.
That's why PDFs are so bad on mobile. The ideal format is one column text, figures and tables between paragraphs of that column, no page breaks, bidirectional links to notes. That's HTML, I guess.
Not really, look at the myriad rendering issues between the most popular browsers. PDF should result in pixel-for-pixel reproducibility, browsers don't do that in practice.
That's why we still test pages in different browsers and end up using browser specific code to ensure proper rendering - which often only reaches the 'close enough' format.
IMO, Reading two-column papers on an iPhone (through PDF) is a real pain -- IMO the format relies on you using your eyes to jump from bottom left to top right, rather than having to scroll from the very bottom to the very top (diagonally). Same problem even exists for single-column styles -- you need to zoom in so much that you have to scroll horizontally as well as vertically.
The need to scroll doesn't exist on a large screen or on a piece of A4, but on smaller devices like mobile phones or even tablets, it's annoying. Having a responsive page means you can scroll vertically as you read, rather than having to make a big jumps (or constant horizontal scrolls) that can really break the flow.
I wonder if that's a personal thing? Over the past year, I've been trying to join the mobile revolution - sort of. The majority of my browsing is now done on a tablet.
I read quite a few PDFs and don't actually have any complaints. I am not personally seeing any readability issues and don't mind consuming PDFs at all.
That said, I think I now understand your complaint. Thanks! I just don't personally have any trouble with it. I use multiple tablets, of varied sizes, and I've had good experiences with all of the devices. While some PDFs are horribly formatted, I find that the device choice doesn't help that and it's a design choice from the author.
> The majority of my browsing is now done on a tablet.
Reading PDFs on a tablet isn't too bad because of large screen real estate.
Reading PDFs on a small mobile phone requires me to zoom in to make the font big enough for me to read, and then I have to scroll right to read, and left and down to move to a new section of the column.
Try reading a PDF on a smaller device than a tablet. I'm sure you'll be able to see what we mean.
Pdf pages are usually based on A4 size which is 210mm wide. Even at full size the writing is often tiny. Once rendered on a 10cm wide screen (landscape) it's pretty darn hard to read.
Also in general the mobile pdf reading experience sucks.
For example you have to download a file (rather than browse to) on Android and the hunt it down to open it.
The pdf readers I've used easily accidentally scroll you to a random page if you make a mistake in where you touch the screen. Kindles probably the best but then you have to email yourself the pdf which is a hassle.
I consume almost all my media on phone. The problem with pdf is precisely that it renders the same on every screen - this makes most PDFs virtually unusable on the phone as you have to scroll down one column in a page, then up for the second column etc.
I understand the problem with a phone, but PDFs on an ipad/tablet are beautiful and a joy to read. Much better to read the text as originally typeset than to put it through a process such as this which risks corrupting minor but important details in the mathematical content.
On my phone I put it in landscape mode and that allows me to read a PDF OK, but I don't really get why one would read academic papers on a phone, why not use a tablet?
However I'm very interested in engrafo. It sounds like it will allow me to automatically publish blog style content from my LaTeX sources without having to fork the LaTeX content into a markdown / HTML version.
I just don't understand why you don't like reading academic papers as PDFs on tablets!
PDF is the only format that will preserve the typographical details that are important in many technical papers; it also avoids the relatively bad rendering created by the browsers.
PDF is usually bad, of course, on small screens, unless the publisher makes special versions.
I think this is a dramatic difference of opinion between coders and everyone else in science, if you excuse the generalization. I say that because of the large number of tools to try to one-up the PDF for academic scholarship (e.g. Readcube). Publishers looking to invent ways to define their own value have been slowly trying to force these things onto readers, but almost everyone in my field hates them and wishes they would die. Formatting and typography are critical in many fields, and a PDF is the canonical way to maintain these aspects of a paper.
This is awesome ! Going on my home screen now. Love the design. Maybe you could ask Arxiv to have a button on their site that would direct it so that it opens on your site .
Only issue I've run into so far is that cross-references to theorem numbers don't seem to always work correctly, e.g. you'll see a lot of "Theorem ?" in https://www.arxiv-vanity.com/papers/1607.06711/.
That said, on cursory look, this is pretty impressive. latex->web converters have existed for a long time, and this appears to have navigated some aspects quite well!
Looks like it's failing to process some standard tex commands (e.g. \textup) as well as some user defined macros. See the many display errors in https://www.arxiv-vanity.com/papers/1710.07406/
Of course, it goes without saying that I want this.
I have to admit I am not impressed. My first paper, which I tried to render, does not work properly; references are removed and rendered poorly, figures are misplaced and tables incomplete. Given that not all arXiv papers are under a permissive license and you do not have permission to do this, I would much prefer if you at least made sure that arxiv-vanity rendered papers do not show up in search results, e.g. by offering a suitable robots.txt and with a bigger link to the author-endorsed version of the paper.
Edit to clarify: If people want to use or develop a broken sort-of-PDF viewer, that’s fine. However, if someone searches for a paper of mine, I would like them to only find the version where I at least had a chance to see that it renders correctly and is complete. In particular, I do not want to be "responsible" for broken rendering on random third-party websites. This website actually operating illegally does not make me more inclined to support it.
I believe in most countries only a court can decide if a site is illegal or not. Not you. And as far as I know this is true in both France [0] and Germany[1].
Each arXiv paper has a well-defined license linked-to from the upper right hand corner "(license)" link below the PDF and source downloads. If they only re-hosted and re-compiled papers for which they have a license to do so, I wouldn’t complain at all, but re-hosting and modifying content without a valid license is clearly illegal, no?
> re-hosting and modifying content without a valid license is clearly illegal, no?
No.
Longer version: it's illegal only if a license is required, which is a matter of the copyright law of the jurisdiction relevant to the act. In the US, that question may turn on things like fair use analysis, which can be tricky.
Fair use applies to citations (not of the whole work), parodies and similar creative processes. Similar requirements in Germany include some creative input by the person claiming fair use which usually should exceed the creative content taken from the original work. Simply re-compiling the LaTeX source is certainly not creative work sufficient for a fair use exception. Checking the other limits of copyright law in e.g. Germany ( https://de.wikipedia.org/wiki/Schranken_des_Urheberrechts ) nothing remotely applies to this site.
Could you clarify why you think that this site does not require a valid license to re-host and re-compile papers?
> Fair use applies to citations (not of the whole work)
Time-shifting is one of many examples of where copying a whole work was found to be fair use; the idea that fair use applies only to citations is very, very wrong.
Fair use is extremely precedent dependent (and very hard to predict without clear applicable precedent) because the statute law gives only factors to weigh in the analysis.
> Could you clarify why you think that this site does not require a valid license to re-host and re-compile papers?
I didn't state an opinion on that; I said that, because it skips the question of whether license is required, the blanket statement that rehosting without a valid license is “clearly illegal” is inaccurate and overbroad.
"Some papers do not render correctly, for example figures and tables in mine [1][2]. Beside that, some articles may not be posted under a permissive license, so you want to double check that you're not running into copyright violation troubles by modifying or publishing them.
This is fair. If somebody stumbles across this thinking it is how you intended it to be displayed, I can understand you'd be unhappy. We should make it clearer that we're just a conversion tool, not a source.
If you want us to remove your paper and just point at the PDF, we're happy to do so. My email's in my profile if you don't want to post the broken render here!
Thank you for your reply. Ideally I’d prefer for you to respect the license associated to each paper and only re-compile and re-host if the license actually allows you to do that (i.e. CC0, CC-BY, CC-BY-SA and maybe CC-BY-NC-SA, depending on whether you think you act commercially).
I also don’t want to keep tabs on every arXiv rehoster and inform them manually by e-mail every time a new paper goes up.
May I ask why this was not done together with the arXiv itself? I.e. have the infrastructure run there, let authors check the HTML render at the same time as the PDF render and then, if the author thinks they look ok, have them show directly on the abstracts page? This would even avoid all your license problems, as the arXiv already has the corresponding license!
A big challenge is to get references working correctly.
LaTeXML is quite good at converting latex documents to html [1], including references such as Theorem 2.1, equation (8.1) etc.
For instance, the paper [2] appears to be quite readable on mobile, and clicking/tapping on a reference such as (8.1) leads you to equation (8.1) as you would expect.
The auto-generation of Arxiv-Vanity is really nice, maybe it would be easy to add the LatexML output too?
Definitely useful in certain situations. Can't comment on how well the conversion works (yet) but I can see how this might be useful to a lot of people.
Me? I still mostly prefer reading physical academic papers because of needing to flip back and forth for re-reading (clarification) and adding personal notes/graphs/calculations.
Hallejuhah, we need to force academics and people who go around touting "whitepapers" to be ripped out of the proprietary PDF era of terrible UX/readability and into modern readable web documents. This is definitely the way to do it, this is brilliant!
Those problems are normally Pandoc parsing errors. Considering it's open source, perhaps we should print the error message so people can actually help fix it...
The MathJax failures are either things that MathJax doesn't support, or use of \DeclareMathOperator which we haven't added support for yet.
This is amazing. I hope you'll keep working on it. There's always a long tail of details that need taking care of when trying to cover a large corpus, and ploughing through successive 80%'s is (as you are no doubt acutely aware) serious grunt work. But you've made a fabulous start, so I hope you find the stamina to do it!
Yeah, even building upon Pandoc's LaTeX parsing, 3 months of grunt work got us this 20% working. Over the next 12 months we'll get the other 80% working. :)
137 comments
[ 5.1 ms ] story [ 202 ms ] threadSo, we built Arxiv Vanity: a site that renders Arxiv papers as web pages. It’s still pretty janky, but for the papers that do render correctly, the experience is so much better than reading a PDF. For example:
https://www.arxiv-vanity.com/papers/1705.04085v3/
https://www.arxiv-vanity.com/papers/1708.00884/
https://www.arxiv-vanity.com/papers/1705.06031v2/
The source for the LaTeX to HTML renderer is on GitHub[0]. It’s built on Pandoc[1] and Distill.pub’s template[2].
[0] https://github.com/arxiv-vanity/engrafo
[1] https://pandoc.org
[2] https://github.com/distillpub/template
This look quite a bit better, so here is the question: what do you not support at the moment?
Here is the broken stuff we are keeping track of: https://github.com/arxiv-vanity/engrafo/issues (feel free to add to it!)
I would think that panflute would allow for more readable code, which helps whendealing with all the corner cases and rough edges of latex.
[1] https://github.com/sergiocorreia/panflute
https://github.com/arxiv-vanity/engrafo/issues/160
https://pandoc.org/epub.html
One final thing, and wildly off topic, is that when you do your defense, remember that you probably know more about the specifics of the subject than anyone else in the room. Many folks stress over it, but you're almost certainly going to be the actual expert in the room. Good luck!
Your solution using the LaTex source generates really nice HTML, congrats!
For now, it's hard to read: https://docushow.com/viewdoc?url=https%3A%2F%2Farxiv.org%2Fp...
(One thing that is very ugly in the PDFs, and most scholarly papers, is the use of different-colored boxes for hyperlinks. Authors, please consider putting
\usepackage[colorlinks]{hyperref}
in your LaTeX preambles.)
[0] https://dmaitre.phyip3.dur.ac.uk/NPP/notes/ [1] https://github.com/JBorrow/latex-pandoc-preprocessor
PDF has the great benefit of rendering the same on every system. With very few exceptions, PDF will look exactly the same on every system and will print the same on every system.
HTML doesn't really have that same benefit.
Don't get me wrong, I think your service is a great idea for those who would like HTML formatted results, but I'm not understanding the complaint about PDF.
Could you expand on why you don't like PDF?
Yes, I want my cake and a pony. Cakepony.
I am guessing it is an individual taste thing. That makes some sense.
so no need to search tablet specs for the culprit. PEBCAK :)
Then there are minor issues of margins, possibly zooming to make text readable, etc.
That's why PDFs are so bad on mobile. The ideal format is one column text, figures and tables between paragraphs of that column, no page breaks, bidirectional links to notes. That's HTML, I guess.
Yes, it does.
That's why we still test pages in different browsers and end up using browser specific code to ensure proper rendering - which often only reaches the 'close enough' format.
The need to scroll doesn't exist on a large screen or on a piece of A4, but on smaller devices like mobile phones or even tablets, it's annoying. Having a responsive page means you can scroll vertically as you read, rather than having to make a big jumps (or constant horizontal scrolls) that can really break the flow.
I read quite a few PDFs and don't actually have any complaints. I am not personally seeing any readability issues and don't mind consuming PDFs at all.
That said, I think I now understand your complaint. Thanks! I just don't personally have any trouble with it. I use multiple tablets, of varied sizes, and I've had good experiences with all of the devices. While some PDFs are horribly formatted, I find that the device choice doesn't help that and it's a design choice from the author.
But, again, thanks for helping me understand.
Reading PDFs on a tablet isn't too bad because of large screen real estate.
Reading PDFs on a small mobile phone requires me to zoom in to make the font big enough for me to read, and then I have to scroll right to read, and left and down to move to a new section of the column.
Try reading a PDF on a smaller device than a tablet. I'm sure you'll be able to see what we mean.
Also in general the mobile pdf reading experience sucks.
For example you have to download a file (rather than browse to) on Android and the hunt it down to open it.
The pdf readers I've used easily accidentally scroll you to a random page if you make a mistake in where you touch the screen. Kindles probably the best but then you have to email yourself the pdf which is a hassle.
I understand the problem with a phone, but PDFs on an ipad/tablet are beautiful and a joy to read. Much better to read the text as originally typeset than to put it through a process such as this which risks corrupting minor but important details in the mathematical content.
On my phone I put it in landscape mode and that allows me to read a PDF OK, but I don't really get why one would read academic papers on a phone, why not use a tablet?
However I'm very interested in engrafo. It sounds like it will allow me to automatically publish blog style content from my LaTeX sources without having to fork the LaTeX content into a markdown / HTML version.
I just don't understand why you don't like reading academic papers as PDFs on tablets!
PDF is usually bad, of course, on small screens, unless the publisher makes special versions.
Only issue I've run into so far is that cross-references to theorem numbers don't seem to always work correctly, e.g. you'll see a lot of "Theorem ?" in https://www.arxiv-vanity.com/papers/1607.06711/.
Thanks!
That said, on cursory look, this is pretty impressive. latex->web converters have existed for a long time, and this appears to have navigated some aspects quite well!
Personally, I prefer the PDF versions, but this could be very useful on a phone.
Of course, it goes without saying that I want this.
I would love to see a bookmarklet that lets me hop from an arxiv page straight to Arxiv Vanity.
Also, the manicure emoji for the favicon was a great choice!
I tried it on this one: https://www.arxiv-vanity.com/papers/1702.03277/
Some commands don't work (\textsl, \rotatebox, ...) and the thank you footnote is incorporated into the title, but otherwise very readable!
Edit to clarify: If people want to use or develop a broken sort-of-PDF viewer, that’s fine. However, if someone searches for a paper of mine, I would like them to only find the version where I at least had a chance to see that it renders correctly and is complete. In particular, I do not want to be "responsible" for broken rendering on random third-party websites. This website actually operating illegally does not make me more inclined to support it.
IANAL.
[0] I got the impression it was a French site
[1] just guessing where you live
No.
Longer version: it's illegal only if a license is required, which is a matter of the copyright law of the jurisdiction relevant to the act. In the US, that question may turn on things like fair use analysis, which can be tricky.
Could you clarify why you think that this site does not require a valid license to re-host and re-compile papers?
Time-shifting is one of many examples of where copying a whole work was found to be fair use; the idea that fair use applies only to citations is very, very wrong.
Fair use is extremely precedent dependent (and very hard to predict without clear applicable precedent) because the statute law gives only factors to weigh in the analysis.
> Could you clarify why you think that this site does not require a valid license to re-host and re-compile papers?
I didn't state an opinion on that; I said that, because it skips the question of whether license is required, the blanket statement that rehosting without a valid license is “clearly illegal” is inaccurate and overbroad.
[1] https://... [2] https://..."
Sounds a lot more positive and might get better results than being as adversarial and negative (imo) as the original comment?
If you want us to remove your paper and just point at the PDF, we're happy to do so. My email's in my profile if you don't want to post the broken render here!
I also don’t want to keep tabs on every arXiv rehoster and inform them manually by e-mail every time a new paper goes up.
May I ask why this was not done together with the arXiv itself? I.e. have the infrastructure run there, let authors check the HTML render at the same time as the PDF render and then, if the author thinks they look ok, have them show directly on the abstracts page? This would even avoid all your license problems, as the arXiv already has the corresponding license!
For instance, the paper [2] appears to be quite readable on mobile, and clicking/tapping on a reference such as (8.1) leads you to equation (8.1) as you would expect.
The auto-generation of Arxiv-Vanity is really nice, maybe it would be easy to add the LatexML output too?
[1]: http://www.albany.edu/~hammond/demos/Html5/arXiv/lxmlexample...
[2]: http://www.albany.edu/~hammond/demos/Html5/arXiv/LaTeXML/110...
Me? I still mostly prefer reading physical academic papers because of needing to flip back and forth for re-reading (clarification) and adding personal notes/graphs/calculations.
Good job guys.
https://en.wikipedia.org/wiki/Portable_Document_Format
https://www.arxiv-vanity.com/papers/1608.04012/
https://www.arxiv-vanity.com/papers/0903.3065/
Also a lot of MathJax failures (maybe Latex variables names?) https://www.arxiv-vanity.com/papers/1709.09439/
The MathJax failures are either things that MathJax doesn't support, or use of \DeclareMathOperator which we haven't added support for yet.
Edit: Added a more useful error message. :) https://www.arxiv-vanity.com/papers/1608.04012/