Fast JSON parser for iOS (blog.nextive.com)

Anyone benched this against a binary protocol like thrift or protocol buffers?

Any company that wants to show benchmarks should, at a minimum show:

    1. The hardware on which the tests were run
    2. The software (OS) on which the tests were run
    3. The design of their benchmarking utility
    4. How much memory/CPU was available for each test (i.e. how much crap is running in the background)
    5. The version of each thing (parser) they are testing
    6. How many trials they ran for each test (no reason for a test like this to not have several hundred or thousand)
    7. The average and standard deviation for each thing (parser) they are comparing
    8. The data ran for the tests. (Some data suites may favour certain parsers over others). Best case: running different kinds of data suites in separate tests, to see if some things (parsers) handle certain kinds of data better than others. (i.e. there might not be a global optimum parser)

Additionally, they should also run the parser on a few passes prior to starting benchmarking to "warm it up". This reduces variance in the runs, especially via things like initialization/memory allocation in the first run.

Without most (or all!) of that kind of information, one can't make a reasonable judgement.

Another one?

Not quite. See these tweets from the author of TouchJSON (Spoiler: JSONKit is still faster, TouchJSON still has a better interface):

http://twitter.com/#!/schwa/status/94835100075819008 http://twitter.com/#!/schwa/status/94840915075674112

Somewhat off-topic, but looking at the readme of JSONKit [1] I found: "At this time, there are no plans to support ARC in JSONKit. Although tenative [sic], it is extremely unlikely that ARC will ever be supported, for many of the same reasons that Mac OS X Garbage Collection is not supported."

Which seems kind of weird, seeing as ARC is going to be the default pretty soon.

[1] https://github.com/johnezang/JSONKit

Disclaimer: I'm the author of TouchJSON - the library the authors are claiming 100x speed up over.

As pointed out by other commenters the 100x speed up claim is incorrect. I added NXJSON to Sam Soffe's iOS benchmark test app and definitely did not see a 100x speed up.

I'm _guessing_ that the JSON deserialization actually failed when testing and the 100x speed up was due to the tests bailing out early. Sam'e JSON benchmark tool while handy isn't 100% fool proof - there's no validating the output of the code to make sure it matches the expect results.

While NXJSON is faster than a lot of it's competitors it really doesn't seem to do anything interesting. It is Obj-C based (you can definitely get better performance with straight to the metal C) and gets some speed improvements via macros rather than Obj-C message dispatches.

Unfortunately the 100x claims are floating around on twitter and are repeated here on hacker news. If it's too good to be true - it probably is.

It's not even plausible. Native binary plist reading is typically about 2x as fast as JSONKit (in my informal benchmarks).

So they claim to be about an order of magnitude faster than binary plists, which would be astonishing if it were true.

haberman's rule for publishing benchmarks: you don't get to claim that you're >2x faster than your already-highly-optimized competitor without explaining why. If you don't know why, you need to figure out why. Without explaining why, I will assume that it's not an apples-to-apples comparison, or that you made a mistake in your benchmarking.

If JSON parsing is a bottleneck in your application, you're doing it wrong.