171 comments

[ 2.6 ms ] story [ 196 ms ] thread
Always glad to see pandoc get some attention. This tool is probably in my top 5 overall, I barely make it through a day without it.
Huh. What do you use it for on a daily basis?
I'm not the OP, but for me it's converting statistical analyses done in Rmarkdown to PDF or HTML.
I'm in college, and my profs send a lot of .docx files. In general I prefer not to start up libreoffice, so I just use a script and mailcap file to view it automatically with pandoc and zathura. I also use it to write for both assignments and personal stuff, though for anything long or with weird formatting I prefer Latex.
I've been using pandoc for a while now, but did not know that it could handle docx (I should probably read through the manual again!).

Does it handle embedded pictures well?

It handles them pretty ok when converting to pdf, obviously not converting them to markdown.
Thank you! This has been useful to know, and I now have some amount of fiddling around with pandoc and docx files on my TODO list :)
If you do I highly recommend looking into using a reference doc. I struggled to make the markdown -> docx conversion until I set a few reference docs up to keep consistent style.
Pandoc works great as a high-level wrapper around latex, where you can write the content in highly-readable markdown, while adding embedded latex for more complex stuff. Being able to use BibTex instead of MSWord's god-awful reference system for footnotes was an eye-opener, as was the ability to keep your manuscripts in text-based .md and .tex formats instead of docx, so you can track your revisions with git.
I write my lectures and labs in .md and convert to pdf with pandoc. I like the results tex produces but I don't love the language, so pandoc is ideal.
Why not use LyX as your front end into latex/Tex?
I've self-published a couple of paperback novels that I create using LaTeX, then I run them through pandoc to get a perfectly formatted .epub that I use to sell the e-book versions.

Flawless!

Pandoc is great at bridging the gap between science-oriented data control needs, and management-oriented reporting needs.

I was on a modeling project that used scripts to generate hundreds of input parameters, embed them in models, run the models, and produce reports. The inputs and outputs shifted a lot over the course of the project, as we came to understand the domain and implications of the work better. At every update, the changes had to be transferred to a Microsoft Word document that went to the project sponsors.

Pandoc made this easy -- we just added scripts to write out the model inputs as Markdown tables, then embed those tables in a larger writeup, also written in Markdown. Pandoc turned it all into a Word document. Thus, the same toolchain that did the actual work, also drove the final report. I really don't think we could have had confidence all the tabular data was right, had it not been automated through Pandoc.

Pandoc ubuntu apt installation is horrible.

I have installed the latest texlive in home directory.

When I invoke 'sudo apt install pandoc' it requires me to install a massive texlive setup at the system level as part of it.

This is not specific to pandoc but many other packages. I have anaconda3 installed in my home, but image-magick requires a massive numpy/scipy system-level install (ignoring for the moment my bewilderment at why would image-magick require numpy/scipy).

I refuse to put up with this kind of bloated bs.

What are you complaining about exactly? That your package manager doesn't automatically know you've installed something manually?
You're asking the system to install a package. System packages are available to all users. If the package is going to work for all users, its dependencies also need to be available to all users. This naturally leads to what you're seeing: the system will not consider software installed only for your user, so it'll end up installing the same dependency system-wide that you had installed in your home directory. While I understand your frustration, I can't immediately think of a better way to handle this.
I think these packages contains pdf as well which makes the whole texlive installation over 1gb. Even without pdf, texlive is pretty big. I don't think there is a way around it. You can use a docker image to isolate pandoc from the system.
Considering what you get for 1G it is worth it for most users. I would guess that you aren't the target audience for it if you're that concerned over space. 1G of space these days is nothing unless you're using an older system. It's just sitting on your disk and that takes nothing away from you if you aren't loading it. It handles 10s of file types and that requires a lot of libraries
In Ubuntu and Debian the dependency from pandoc to texlive is of the "suggests" type, not "required". So you do not have to install texlive to use pandoc. You may use an interactive front-end like aptitude and simply deselect all the suggested dependencies you don't care about (or configure aptitude not to install suggested deps by default).
pandoc is one of the few packages (among with tetex) i black listed on my distribution for automatic updates because it seems to pull in hundreds of other packages which are not used by anything else.

I don't know how they did it, but somehow they put dependency hell on a completely new level.

Yes i'm sure it's a great tool, but there's a limit how much bloat I can tolerate for a single program.

This has little do with pandoc and everything with how awfully Haskell packages are packaged for some distros. Imagine if installing a program that runs on node would pull in every single npm dependency as its own package.
That's the point though: you should only need one package manager.
> That's the point though: you should only need one package manager.

That's orthogonal to the issue. Even with just one package manager, how packages are created and maintained is a separate task.

So, if the pandoc package has dependencies vendored, dependency hell is avoided regardless of which package manager is used to install it.

If, however, the pandoc package has all dependencies listed as separate packages, dependency hell is created, again regardless of which package manager is used to install it.

So this is a matter of policy, not tooling.

I think the difference is in the visibility of dependency hell, not the existence of it.
Can I wager a guess that you are on Arch?

The Arch (and some other linux maintainers) have made the decision to package all Haskell libraries as separate OS packages and install those as dependencies when you install, say, pandoc. This model of distribution doesn't really make much sense for distributing Haskell binaries, though.

There's a few reasons for this: 1) since most people don't have many Haskell binaries and the few that people use don't share many libraries, 2) Haskell packages are normally statically linked when building executables.

If linux maintainers would simply build/ship pandoc as a single static executable all these issues disappear.

Nice!

I'm on Arch, but I was under the impression that Debian did the same thing for Node/Haskell modules. Or am I mistaken?

That's your distro's problem.

    $ zypper info --requires pandoc

    libm.so.6()(64bit)
    libpthread.so.0()(64bit)
    libm.so.6(GLIBC_2.2.5)(64bit)
    libpthread.so.0(GLIBC_2.2.5)(64bit)
    libm.so.6(GLIBC_2.29)(64bit)
    libdl.so.2()(64bit)
    libdl.so.2(GLIBC_2.2.5)(64bit)
    libz.so.1()(64bit)
    libc.so.6(GLIBC_2.17)(64bit)
    ld-linux-x86-64.so.2()(64bit)
    ld-linux-x86-64.so.2(GLIBC_2.3)(64bit)
    libgmp.so.10()(64bit)
    libpthread.so.0(GLIBC_2.3.2)(64bit)
    libm.so.6(GLIBC_2.27)(64bit)
    librt.so.1()(64bit)
    libutil.so.1()(64bit)
    libpthread.so.0(GLIBC_2.12)(64bit)
    libnuma.so.1()(64bit)
    libnuma.so.1(libnuma_1.1)(64bit)
    libnuma.so.1(libnuma_1.2)(64bit)
    libffi.so.8()(64bit)
    libffi.so.8(LIBFFI_BASE_8.0)(64bit)
    libffi.so.8(LIBFFI_CLOSURE_8.0)(64bit)

    $ rpm -ql pandoc | grep -v '^/usr/share'

    /usr/bin/pandoc

    $ ll -h /usr/bin/pandoc

    -rwxr-xr-x 1 root root 162M Sep 30 13:33 /usr/bin/pandoc
That would seem that your distro is statically linking all the Haskell libraries. On distros that use dynamic linking for everything, it's also going to pull in (directly or indirectly) ~130 Haskell libraries.
That's probably because it's compiled dynamically in your distro's package manager. If you look for a statically compiled option, it might be more to your taste.
And there are statically-compiled versions available for multiple platforms on Pandoc's download page. (I tend to use those for the Mac, rather than installing through Homebrew.)
If you’re concerned, run it in docker and dispose the container after you’ve got your output docs
Does that actually matter? To me dependency hell is when you have lots of conflicts where some software requires one version but some other software needs a different version. So you can't upgrade one version without breaking something else.

With pandoc and all the haskell dependencies, the only downside is the length of the list of packages when you upgrade. If it was all bundled up as haskell-all I doubt I'd even notice.

Arch, I presume? That's mostly due to a man-power problem on the side of the Arch Haskell maintainers. Try our pandoc Docker images or use pandoc-bin from AUR for a bloat-less version. https://hub.docker.com/u/pandoc
Considering what pandoc does and how it is used, docker is a massive overkill imho. What pandoc should actually do, is come as a tar ball and be buildable the traditional configure make make install way like all unix tools of a similar fashion do. Haskell, atm, is no language for this.
Hahaha, that's actually some quality and funny trolling. Not bad :D

For everybody interested in alternative installation methods: all pandoc releases are available as statically compiled binaries for Linux, and via installers on macOS and Windows. Any major package managers ship a more-or-less recent version of pandoc. Compiling is as simple as getting the "stack" tool and running `stack install`.

Pandoc is awesome! One of my favorite usecases is for Orger [0], which I'm using to automatically convert data from different services into org-mode for easier local-first/offline search, navigation etc. Often API would give you markdown (e.g. Github), and while I could embed a markdown source block in org-mode, with Pandoc I can just convert it and display in native Org syntax.

[0] https://github.com/karlicoss/orger#readme

Pandoc is a tool used daily by those of us who write code notebooks (rmd or jupyter) or are into using markdown for their notes and occasionally need to print said notes. It is hard to overstate how useful Pandoc is for me.

I would bet many people who use Pandoc have no idea they rely on it. I don't think Jupyter or RStudio make a big fuss about it even though they both use it.

I’m a big fan of keeping md documents in source control, then publishing them wherever they need to go in the CI/CD pipeline, and I’ve used pandoc a lot for that.

I always ponder whether it’s the most practically useful Haskell tool ever written.

Either pandoc or shellcheck, for sure. Both of them are sensible choices to use Haskell for
Yes, RStudio uses it and I find it lives up to the title "The Swiss Army Knife of document conversion."
This is great to know. I use markdown for journaling, note taking, and documentation. I don't need to print anything but if I did then I'd probably go the way of mardown to html with custom css - now I will give pandoc a try first.
One thing I love about pandoc that I don't see mentioned here is the ability to apply filters to transform documents mid-conversion.

I'm using Pandoc to write my PhD thesis at the moment, from Markdown source, using certain filters to "augment" what Markdown can do. Examples:

https://github.com/LaurentRDC/pandoc-plot

https://github.com/lierdakil/pandoc-crossref

More info here: https://pandoc.org/filters.html

It must be pretty slow if you have time to change the settings while it is converting?
None of this has anything to do with how much time is taken. The whole thing could run in a millisecond.
> One thing I love about pandoc that I don't see mentioned here is the ability to apply filters to transform documents mid-conversion.

I read that as "mid-conversion" meaning that he can apply filters while the document is being converted

Not sure if this is a joke but if you read the linked docs [0] you'll see that the concept of filters is that you the user can write programs (essentially plugin-style) that modify the AST Pandoc generates in order to perform the conversion. But this explanation is ultimately worse than the actual doc page, so I'd recommend just reading that.

https://pandoc.org/filters.html

What a coincidence! I am also writing my PhD thesis with pandoc and filters, but I use panflute for the latter: http://scorreia.com/software/panflute/
I'm the author of panflute, and--in fact--wrote my PhD thesis in pandoc+panflute!
I'll be starting a PhD soon and would love to use a pandoc-based workflow (with MD or another format) for the gross of my writing.

How did you all structure the commenting on your writing? I find converting to odt/doc before sending, managing all the exported versions with comments etc. becomes quite tedious. But I'm a bit reluctant to force my supervisors to use eg. git+criticmarkup[1]. I would love to hear you experiences!

[1]: http://criticmarkup.com

A few things:

- In my case, my supervisors mostly had handwritten notes which rendered that point a bit moot. However, when I send the almost-complete draft to a professional copy editor, it was indeed a pain to add the comments. Either handwritten+scanned, as acrobat comments, or word comments, they had to be manually input into the markdown file.

- Everything else worked relatively better. It was a bit tedious to type loooong pandoc commands "pandoc --filter=... etc etc" so I recently coded pandocmk [1] to make my life easier. It's not super well documented (but it's a quite short script, so readable), but the idea is that you type the command line options as metadata at the top.

[1] https://github.com/sergiocorreia/pandocmk

I've used that to process tables, essentially using markdown as a commented CSV format. The only nuisance is that a table can't yet have attributes — https://github.com/jgm/pandoc/issues/6317 — the workaround being a pre-filter to copy them from a surrounding div.

I've also toyed with using it to process code blocks, as a dead-simple literate programming tool.

my uni requires MS Word...
That is the point of pandoc.

You can write in markdown and then convert it to word for your uni.

For MS Word Pandoc is a blessing: I configured a master file and now pandoc does all the tedious formatting!
I wrote my MSc in markdown with Pandoc. Loved it.
This is going on a decade old, but I wrote my bachelor's thesis in pandoc. It made the otherwise painful very straightforward.
Yeah, filters are great. Writing filters is easy: Pandoc basically converts the input document into a universal AST (json), and a filter is just any program that takes this json as an input and outputs a modified json AST.

I wrote a filter that automatically converts URL citstions in markdown to "real" citations in any style you want - very useful for writing papers without fighting with bibtex and managing bibliographies manually: https://github.com/phiresky/pandoc-url2cite

I had an interesting conversation with John MacFarlane, the maintainer and author of Pandoc (lovely human being and excellent maintainer), and the subject of day jobs came up. He's a professor of logical philosophy at UC Berkeley which I thought was fascinating. It certainly makes sense given the number of document formats and such that academia deals with.
And a great fiddle player!
What is it with amazing professors and musical prowess? My Cryptology professor is also a fiddle player! Ivan Damgård, of the Merkele-Damgård construction.
I've noticed many people in STEM are into music
Almost every upper class parent invest in musical education. Learning to play instrument is both enjoyable and discipline building activity.
They also provide an excellent break from the kind of thinking required in things like programming IME. I can be exhausted from a day of coding and happily sit down and practice with the piano in a way I couldn't with other intellectual topics like maths.
That observation doesn't seem right. There are a lot of people in STEM. Most do not come from upper class parents. Many of the children of upper class parents go into non-STEM fields, including law. (Some go into music, which is not STEM.)

My decidedly working class parents insisted we learn to play piano. Most of my relatives had a piano in the house. I think it was a holdover of the days when home entertainment was self-made.

A friend lived in both the Los Angeles and New Orleans areas. He compared the two as: in LA, the parties of rich people have live music. In New Orleans, the parties of poor people have live music.

And Damgård, mentioned earlier, was born in 1957 Denmark, and plays Danish and Nordic folk music. Postwar Denmark was poor. Perhaps this interview (in Danish) explains why he started? https://www.youtube.com/watch?v=AUF_EkN4Z-g

So, 1) is there a significantly high proportion of people in STEM who are into music than non-STEM? (and not simply some sort of observational bias), and 2) is the major contributing factor to the high proportion because the parents of those people were upper class? (and not some other factor like STEM fields paying enough so people have free time for hobbies.)

I would add lots of kids in lower income homes are exposed to music being played by family members, peers, school programs, or church groups (examples). It's true that these kids might not be playing Mozart but there is nothing wrong with bluegrass, gospel, or whatever, to instill a love of playing music.

Maybe listen to "Juke Box Hero" or "Coat Of Many Colors" for inspiration on how people from modest backgrounds can have the same fulfilling experiences as wealthy people. (sorry - personal soapbox)

Actually, Music is STEM, in the true and classical sense - it's only our perverted modern view that has severed music from its moorings in mathematics.

It was even part of the quadrivium of medieval education: Arithmetic, Geometry, Astronomy, and MUSIC! A classical education not only taught these subjects, but how they were all inextricably interrelated (or intertwingled, as Ted Nelson famously says...)

STEM is not a classical term. Don't go thinking that because we don't follow ancient Greek philosophy or medieval education philosophy that are somehow perverted.

FWIW, I strongly dislike "STEM" as a term because it makes no sense to me in an educational or philosophical sense. I see it more as an attempt to lower the cost of hiring engineers and scientists by increasing the supply. For example, compare the funding going into getting more programmers and EEs, vs. marine biologists and paleontologists, even though all of them are STEM.

To clarify "no sense to me", I despise Pirsig's "Zen and the Art of Motorcycle Maintenance" because of its insistence on a clean division between romantic and classical views. I view "STEM"'s treatment of the rest of the liberal arts as being similarly incorrect in its dichotomous classification. Eg, mathematics is important for the humanities too.

But it's clear what _def is talking about by "STEM", and there's no need to suggest we or modern culture are following along with a perversion because the conversation isn't aligned with your personal views.

(comment deleted)
My pet theory is that they both require perseverence to get to a fun level and once there doing it provides intrinsic reward.
Great thing about Pandoc - it has a clear, descriptive and yet unique name that aptly describes what it does.

That aside, I find the markdown + additional features (e.g. latex math, inline code eval), mainly as implemented in Rstudio and Rmarkdown, to be the sweet spot of power and convenience of typing and legibility in plain text form. Thanks pandoc!

Pandoc is on the the programs that always surprises me with how good it is. Everything I throw at it works perfectly. I write my assignments for class as Markdown or plain text and it easily makes them a good looking Word or LaTeX document seamlessly.

It's also fantastic for converting my class notes from Markdown with LaTeX equations into beautiful PDFs.

I'm using pandoc for generating pdf/epub ebooks from GitHub style markdown. The default output is good enough and there are various themes that can be selected. But I wanted to customize a lot of things like chapter breaks, background color for inline code, bullet styles, blockquote style, etc. I didn't know Latex but was able to find snippets from stackexchange sites to suit my needs. I wrote a blog post on this: https://learnbyexample.github.io/customizing-pandoc/
I can't express enough my gratitude on a daily basis for what pandoc enables me to do. I made a simple Emacs script that I use to output files, and I use it constantly for Latex PDFs, HTML output, RevealJS slides, and odt/docx/etc. All with bibliographies fron Zotero in zillions of formats. As a professor and journalist, I need to use a wide range of output formats, but as a human being I like to work in clean, simple text files that will never be obsolete. Pandoc, way more than any tool, gives me the freedom to work in any writing environment I like and keep that distinct from whatever weird formatting preferences a journal, magazine, or publisher might have. I've written two books with Markdown and a huge variety of articles. I am so thankful for the care with which it has been built and maintained. Thank you.
Pandoc is great but I think it falls a bit short of being a Swiss army knife; there are a lot of conversions it cannot do, like PDF-to-anything. Thankfully Calibre's 'ebook-convert' tool covers many of pandoc's blindspots.
But real Swiss army knife does not include any magic either - even simply extracting text from PDF (ignoring all formatting) is completely non-trivial. Do not know any (non-magical) specialized tool that can convert PDF formatting.
Calibre (`ebook-convert`) makes a decent attempt at converting PDFs to other formats. This of course is very far from perfect, but it takes a good stab at it and I've sometimes found the results to be usable (often with some manual cleanup.)

Another example where Caliber compliments Pandoc well is when generating ebooks for sideloading onto kindles. Pandoc can create epubs which Calibre can in turn convert to mobi.

Exactly, Pandoc chooses robustness over buggy half baked conversion. Swiss Army Knife is no good when you need to a debone a Tuna. Every tool on a Swiss Army Knife is sub-optimal. It's a terrible popular analogy in general.
I don't know much about tuna, but I once cleaned a bass with my swiss army knife.

Anyway, I don't really expect Pandoc to do everything, but when you have both Calibre and Pandoc in your toolbox, it sure feels like you could manage close to anything.

(comment deleted)
Probably overkill, but I use Pandoc to generate tailored resumes for roles and jobs I’m interested in.

I keep a list of all my skills, experience and education in a YAML file and have a LaTeX template that I clone when creating a new resume. Then it’s just a matter of replacing the template fields with YAML metadata and running Pandoc.

I have the same set up to generate both my resume and my website using an HTML template. Makes it easy to update one YAML file and update both my CV and my personal website

https://mehalter.com

The man page is a very nice touch! Do you have source in GH or elsewhere about this harness? I am using Restructured text and rst2pdf but this looks so much nicer!
Disregard, I followed your Keybase to you personal git server. Very nice and inspired, I will check it out!
I also use pandoc to generate CVs, happy to know I'm not alone :) I don't do anything as sophisticated as you, but my main resume is in markdown so I use it to create a .pdf or word doc and to apply .css styling where appropriate.
Pandoc is a true work of art. Everything about it embodies the Unix philosophy of "Do One Thing and Do It Well".

I've been using Pandoc (and make) daily for over 6 years for all sorts of document writing (letter, report, thesis, design doc, performance review, you name it) and solve the occasional "interesting" format conversion problem. Its robust, reliable, fast, and a pleasure to use (and script).

Hadn't heard of pandoc before. Momentarily thought it converted from PDF to anything, and my heart leapt. Alas, it only converts to PDF. My hopes dashed...
That's not really a reasonable expectation, as PDF is and output format not an input format. If you want to make a PDF that others can read, the best solution is to generate a PDF that embeds the original input. LibreOffice can do this.
Poster child for Haskell
Yes, it is the only program coded in Haskell I have ever used for anything practical, to my knowledge.

I have heard of others, like git-annex, but not used them myself. I wonder if there are any I just didn't know were.

I also wonder if anything about Haskell makes it particularly suited as the implementation language for Pandoc. It must have a lot of parsers in it, and Haskell is supposed to be good for coding parsers.

There are parser generation libraries and meta-libraries for certain other languages, notably C++. I wonder what Pandoc in C++ would look like. Probably a pretty good parser meta-library could be spun out of such a project.

Xmonad is another famous one
I have never understood the interest in window managers. All I have seen that try to automate to expectations fail to approach mine.

I.e. a window manager does not seem to me like a useful practical application, once there is a minimal one already. Fvwm was fine.

If this list rounds up the most-used Haskell programs, I can safely conclude that I don't use any Haskell program besides Pandoc.

Apparently I use a few Go programs--Docker, maybe others?--but no Java programs at all, because I delete all the JVMs from my machines without noticeable effect. Likewise, no C# programs, because I have no Mono runtime. Probably no Lisp, Smalltalk, Julia, or OCaml. Some things I run almost certainly are or use Lua, and of course Python, Perl, and even Tcl. I don't know of any in Rust, but it would be hard to tell because of static linking.

You can write filters in Python and several other languages. These let you perform arbitrary computation triggered by tags in your source document, and let you extend Pandoc’s Markdown to include your own custom tags to do anything you can imagine.

Here is an article where I show how to use Panflute, a library that lets you write filters in Python, and how I wrote a set of filters to automate the tedious parts of writing a complex technical manual:

https://lee-phillips.org/panflute-gnuplot/

This is probably a silly question, but the last (and first) time I used pandoc, my conversion of org files to markdown resulted in a lot of whitespace within the document itself. I followed the instructions on the website, but is there a flag that I should have used to get rid of excess whitespace?
This is great! Anyone know what the format for Google Docs is and whether Pandoc or another tool is good for importing GGocs into other formats?
Google Docs exports pretty well to docx, pandoc can handle it.
I absolutely love Pandoc, I use it in my Makefile based static site generator. Pandoc is probably one of the most valuable pieces of open source tooling next to ffmpeg and imagemagick.
Pandoc for text, ffmpeg for audio/video and imagemagick for images?

I've used pandoc for pdf generation and ffmpeg for some audio recording/encoding/playback. I can't imagine what I would use imagemagick by itself for though (that I wouldn't use some common image processing application for). What do you use imagemagick to do?

> What do you use imagemagick to do?

Automate various transformations:

- resize - change orientation or ratio - adjust colors - convert format - do all of the above to generate thumbnails of large photos, in one command