HN Explorer

MessagePack - Extremely efficient object serialization library (msgpack.sourceforge.net)

28 points by kzk_mover ↗ HN
MessagePack is a binary-based efficient object serialization library. It enables to exchange structured objects between many languages like JSON. But unlike JSON, it is very fast and small.<p>Typical small integer (like flags or error code) is saved only in 1 byte, and typical short string only needs 1 byte except the length of the string itself

20 comments

[ 2.7 ms ] story [ 66.1 ms ] thread
jrockway
I bet gzipped JSON compares favorably for larger objects, though, and has the advantage of not being a new protocol. The fact that it supports Haskell and Perl with proper hackagedb / cpan modules is nice, however. Very rare for web 2.0 stuff.
houseabsolute
I don't understand why JSON would get faster for larger types. Anyway a 524 byte type is already fairly large.
jrockway
No, the compression algorithms begin to behave as well as hand-optimized binary protocols as the message size increases, because the constant overhead for mapping away the redundancy becomes a small percentage of the overall message.
houseabsolute
But if just building the JSON representation already takes more time than the binary protocol, I can't see how adding gzip to it is going to make it faster.
KirkWylie
gzipping JSON or XML compares relatively favorably against a pure-binary protocol for suitably large messages, but not for pretty small messages.

In addition, the speed of the compression stream really becomes an issue if you're sending messages within a cluster. Over the internet your bandwidth and latency will start to become more important, but within a data centre all pure-binary protocols are much faster.

slpsys
Very interesting, though I'll take more of a look later on. The speed test does seem a little contrived, as it's for one case, and it looks [briefly] like some of the gains are due to greater specificity of primitive types.
prodigal_erik
I recommend ASN.1 PER over any of the myriad byte-aligned no-schema formats that waste bits on ranges of values and entire types you aren't sending. But this is better than some other formats I've seen.
andrewcooke
isn't ASN.1 proprietary? i wanted to implement it once and found i had to pay someone for specs.

is that wrong? if not, i think many people would prefer something more open (more implementations; less likely to die from lack of support)

[edit: sorry, i am talking rubbish here. they are all linked from here, for example: http://en.wikipedia.org/wiki/Abstract_Syntax_Notation_One

i have no idea why i thought they were closed. has the ITU policy changed recently?]

alavrik
The specs were definitely closed last time when I checked (about 6 months ago).

Still don't think this old telecom technology is superior compared to modern serialization tools like Protocol Buffers.

viraptor
old and modern do not mean anything in programming world. C is old - doesn't mean it's better or worse than modern languages. Could you rephrase it in terms of features we can actually compare? I prefer to rely on "old telecom technology" many times, since it's meant to be as close as possible to 100% reliability (telecom being the keyword, not old).
alavrik
I agree that old and modern are not very appropriate terms when comparing technologies. However I didn't mean to express an objective statement in my comment.
prodigal_erik
If you send a thousand key/value pairs, protocol buffers waste sixteen thousand bits redundantly expressing the type and tag of each key and value, when the recipient already knows they're all the same. (They revved their format to special-case this but only for a sequence of numbers, not any other type you can construct.)

PER is the only format I've seen that uses the schema to decide how much information the recipient actually needs to decode the message and just sends that, rather than adding pure overhead just in case a recipient has no idea what's going on yet is somehow supposed to do something useful with the message.

alavrik
It is true that Protocol Buffers are not optimized for low-mem systems. On the other hand, for preserving bandwidth and storage one can use lightweight compression algorithms on top of general-purpose serialization library. (This is the approach that Google ended up using, AFAIK.)

I think such approach is more flexible than making explicit assumptions on how much data is actually carried in integers.

houseabsolute
Why does their graph show the JSON bar as twice as tall as protocol buffers when it's not even 50% slower.
viraptor
PB takes 0.47s, JSON takes 1.2s. JSON is 2.55 times slower, so it's over 2 times higher... Not sure where the problem is.
houseabsolute
The label for the JSON column on http://msgpack.sourceforge.net/ is "0.68 sec" and the label on the PB column is "0.47sec". So, yes, if the times were what you claimed, the graph would be fine, but according to the graph they were not.
viraptor
I think we're hitting some browser cache issue. Could you try force-reloading everything? My graph shows the numbers I mentioned above. Maybe the old version showed something different (if I wget the image without any caching, I still get MP: 0.11s, PB: 0.47s, JSON: 1.2s)
houseabsolute
Browser cache it is. After a reload the number shows as 1.2.
[deleted]
(comment deleted)