Cap'n Proto v0.2: Compiler rewritten from Haskell to C++11 (kentonv.github.io)

I'm kind of shocked there hasn't been a horrified response from HN Haskell fans.

Looking at the code for the 0.1.0 release, it looks like you were using parsec as opposed to attoparsec (which typically has better performance). About virtual tables, I'm not exactly sure of what you were trying to do, but the usual solution to having different kind of stuff you want to put in a single collection is to just use a sum type - though it assumes you know in advance what kind of things you want in your collection, or at worst, use existential types. But the FFI question is a good point.

So basically he's concluding, having written some Haskell that he's not confident enough to maintain/improve a large project in the language?

Seems fair enough to me really?

From the article: "For instance, to actually encode a Cap’n Proto message, I couldn’t just allocate a buffer of zeros and then go through each field and set its value. Instead, I had to compute all the field values first, sort them by position, then concatenate the results."

Sounds reasonable to me, but I think it's just personal preference. Without actually seeing the code (which I'm sure makes a huge difference), the imperative algorithm actually doesn't read as easily as the functional one does to me. We've become accustomed to imperative since it's what we all learned on, but there is a massive amount of book-keeping (although I'm sure the execution speed is much faster). At any rate, here is Scala with imaginary API's detailing how I would envision the functional algorithm...

    def encodeFields(fields: List[Field]): String =
      fields.map { f => f.position -> f.computeValue }.sortBy { _._1 }.map { _._2 }.mkString

    destroyer:alphaHeavy$ find . -name "*.hs" | xargs wc -l | tail -1
      132272 total

    destroyer:alphaHeavy$ grep "\!" -rI . --include "*.hs" | egrep -v "\\$\!|\.\!|\!\?|\!\!" | wc -l
        1344

    destroyer:alphaHeavy$ grep "\\$\!" -rI . --include "*.hs" | wc -l
         762

    destroyer:alphaHeavy$ grep UNPACK -rI . --include "*.hs" | wc -l
         298

  import qualified Data.Vector.Unboxed.Mutable as V

  encode len fields = do
    buffer <- V.replicate len 0
    forM_ fields $ \(fieldPos, fieldData) ->
      V.copy (V.slice fieldPos (length fieldData) buffer) fieldData
    return buffer

> But with the schema parser written in C++, things become much simpler. [see article for more context]

I think this is debatable part.

First, calling Haskell from Python is quite easy. I've wanted that once and things went surprisingly well, see yourself: https://github.com/drdaeman/haskell-library-ffi-example

Second, not everyone has C++ compiler installed and readily available (although, it's true that it's far more likely that there's already a C++ compiler than Haskell one). But in the end, I don't think it really matters whenever one does `$package-install-command g++` or `$package-install-command ghc`, as, I believe, neither comes on out of box installs on most popular non source-based GNU distros, and certainly not on Windows. I don't know how things are in *BSD land.

The name Cap'n Proto is bit strange. The name makes me thing of CAP'N CRUNCH CEREAL and prototyping, but then I read about the connection to protocol buffers. Might want to think of a different name, e.g. Zero Data X "zero cost data interchange" or something.

I am using JSON to save structured data for an application, and this is interesting, although my app doesn't spend alot of time encoding/deconding the JSON since the data structures are fairly simple.

If I wanted to use this for Java, wouldn't there still be decode step to convert the fields into java objects?

Can someone explain what protobuf is in the real world? I've read this page twice now and have a fair amount of exp but still cannot figure out when i would need a protobuf.

I attended a talk at CUFP (Commercial Users of Functional Programming) by a compiler team at Intel on why they were using SML (which -- like caml -- not lazy by default and allows I/O and mutability without having to isolate them ala IO and St Monads) and the comments were quite interesting: many graph algorithms are best expressed in a mutable fashion and he saw no fundamental advantage of simulating mutable state via recursion/CPS (for algorithms that primarily used mutable state) and actual mutability; it wasn't a performance based argument -- the compiler still outputs extremely efficient (obviously) imperative assembly code, but the additional complexity.

Obviously the functional parts of the language still applied with primarily functional algorithms, like parse-tree traversal and transformation, which is an excellent example of monads being useful and handy outside of pure and lazy language.

Note that the author also notes two significant problems that have nothing to do with functional programming or purity per-se:

1) Calling language-x from Python/Ruby/whatever is difficult unless language-x is C/C++. I fully agree here. At one employer we've built custom bindings from Perl to OCaml (for a DSL we've built) and it was troublesome (e.g., it was broken during 64-bit conversion and our patch fixing it was not initially accepted, although the Nria team later came up with the same approach -- which involved "burning" a register -- later anyway).

This is less of a problem for large scale distributed applications that talk via RPC, but this create a problem of building and re-building first-class RPC libraries in each language -- which (as in this very example!) is again best done by building the core RPC protocol implementation in C/C++ and linking it into the applications (with perhaps a native JVM implementation as JNI is a pile of manure? [Personal curiosity: with Google having a huge Java presence, how does Google handle maintaining up-to-date Java clients for BigTable/Spanner et al?]).

2) Lack of OO.

OCaml has OO but it's never used. OCaml has a marvelous module system that is far better than Haskell's. Unfortunately it lacks type classes (hence -- very annoyingly -- no generic "print"/"show" function), but first class modules and campl4 (equivalent of template Haskell) can help there. I am guessing that it doesn't solve the author's problem (which I honestly can't claim I have, but that is like my opinion, man) -- lack of dynamic dispatch.

Problem is inheritance based dynamic-dispatch polymorphism and generic polymorphism are hard to reconcile. I am very impressed by the work Odersky has done with Scala as well as C++11 (and boost/tr1 before it), but these are by no means simple languages (by simple I don't mean simple like Visual Basic, I mean simple as in "implementing a compiler for it is feasible in an advance undergraduate or beginner graduate class").

When I write golang, Haskell, Erlang, or OCaml I honestly don't miss OOP as modules, interfaces/signatures, type classes, and the like provide what I want out of OOP. However, I've never maintained (as opposed to wrote) commercial code in these languages.

tl;dr That's all folks, I'm now switching to an emacs window to write more C++ code -- while (as a big FP enthusiast) hoping practices for programming-in-the-large would emerge for non-OO statically typed functional languages (from places like Basho, Galois, and Jane Street who have built impressive multi-year projects in those languages), OCaml gets multi-core support, and Scala succeeds in its mission.

This seems like a reasonable conclusion. Trying to learn the ins and outs of a new language/library/framework while trying to get something done is not a great way to do either. My primary languages are Haskell and C++, and I write a lot of language tools. It seems to me that both languages are good choices in this department, but for very different reasons. Haskell is difficult to beat in terms of sheer data-munging expressiveness, and I spend 20% of my time in C++ doing what “deriving” does for me in Haskell, but if you know C++ well, you can use it to write very efficient code that’s still fairly expressive. That ain’t nothing. Maybe next time choose OCaml though. ;)

> For instance, to actually encode a Cap’n Proto message, I couldn’t just allocate a buffer of zeros and then go through each field and set its value.

Hah, dealing with native-encoded binary data in Haskell is rather cumbersome. I asked on r/haskell about decoding a native representation of float/double to corresponding Float/Double Haskell value, and this is the thread it generated:

http://www.reddit.com/r/haskell/comments/1k56i1/how_to_read_...

People have generally come with helpful suggestions, and rwbarton's excellent answer [check out the whole thread] has fully explained to me how to essentially reinterpret_cast a byte buffer.

I have been successfully playing with Haskell to implement some nontrivial _algorithms_ and liked. Like the OP, I'm now rethinking whether I want to invest more time into Haskell after seeing how cumbersome it can be to coerce the compiler into "DO THIS" where THIS is not a pure computation.

Now I'll take a pause from Haskell and start reading about Scala as it seems to provide a mix of FP and imperative (OO) that is more palatable to me.

If you do decide to use Haskell for another project come by one of our monthly meetups at the Hacker Dojo: http://www.meetup.com/haskellhackersathackerdojo/. Having more experienced Haskellers around to bounce ideas off of or pair program with can be rather enlightening.