Ask HN: How do you view large JSON files?

If not, I think I would just split data into multiple smaller files. Something like the python code below should work, assuming the file can fit in memory. If not, I assume you can find some json lib that can work in streaming mode and then do the same thing.

    import json, os

    json_input = '''{
    "foo": 1,
    "bar": 2,
    "baz": 3,
    "bug": 4,
    "thing": 5
    }'''

    entries_per_group = 2

    if not os.path.exists('sub_files'):
      os.mkdir('sub_files')

    main_d = json.loads(json_input)
    iter = main_d.iteritems()
    for group_count in range(10 ** 6):
      sub_d = {}
      try:
        for _ in range(entries_per_group):
          k, v = iter.next()
          sub_d[k] = v
      except StopIteration:
        break
      finally:
        json_output = json.dumps(sub_d, indent=2)
        with open(os.path.join('sub_files', '{}.json'.format(group_count)), 'w') as f:
          f.write(json_output)

  cat unformatted.json | python -mjson.tool | vim -

;)

[1] https://github.com/atom/atom/issues/8864 [2] https://github.com/atom/atom/issues/979

For the most common cases, I just copy and paste the string into the chrome console if I need to inspect some random JSON file really quick.

If it is too big to copy and paste, pipe it into a html file as a global variable and view the object in the console.

You can also just open a node repl and load the JSON into memory. node lets you auto complete property names by pressing tab, this is great for inspecting some unknown object.

Here is how it works - keep the network tab ready. When you see the JSON data request, click on the request and hit the "Preview" tab. It gives you data in a collapse / expand format.

Advantages: 1. There is one less plugin that scans all your browsing activity, 2. Slightly extra battery life when just browsing and not developing stuff.

Disadvantage: You need to keep the network tab ready, otherwise you will have to reload the entire page with the network tab open.

EDIT:

My apologies if it wasn't clear. I was talking about the "Developer Tools" option in Chrome, in which there is a "Network" tab. It is available in "Chrome Menu" > "More Tools" > "Developer Tools". Alternatively you can hit Command + Option + I in mac, or some equivalent in Linux / Windows to get there.

That's a pretty large bug. Considering what JSON is supposed to be used for, passing data between two programs/processes, files larger than 1 MB should have been accounted for.

Does it freeze then crash, freeze until the OS takes care of it, or freeze temporarily then resume?

Imagine you have a json structure like this:

{"records" : [ {"name":"joe", ...lots of other fields in each record}, {"name":"fred", ...lots of other fields}, ... thousands of records ]}

Now a lot of people in this thread have mentioned tools for collapsing the json, but the trouble I have with this is that you can't browse the record names without opening each one and looking at all the fields. It would be nice if there was a tool that took a few key identifiers (name, id) and "bubbled them up" so that in a collapsed view you would see something like:

{"records":[ {...click to expand...}, // name=joe {...click to expand...}, // name=fred ]}

I have the same problem when working with XML as well (though in XML sometimes the ID's are attributes of the parent which mitigates the problem, but other times the ID would be nested in a child element of the structure more like you would have in json). I even found it to be enough of a problem that I wrote my own XML editor to solve this issue.

Of course one issue with this is that there's so clear standard as to which fields of an object represent "ID" information which would be important/useful to "bubble up" to the next level when collapsing. It would have to be something user-configurable (though having some sensible defaults like looking for "name" and "id" keys would work in a lot of cases). In the XML world, there's probably something to do with Schemas that would help with this problem, and fancy editors which understand your data using a schema, though some of the editors I looked at which went into that level of detail seemed like way overkill for what i wanted to do.

So essentially my question here is whether this concept of "collapse child, but keep important identifying information of the collapsed child visible" exists in any json tools? is this a thing that has a name/buzzword associated with it that I don't know about? or is it purely an issue that's my own personal quirk which nobody else cares about?

Instead, use a tool like Schema Guru (https://github.com/snowplow/schema-guru ; disclaimer: we wrote this at Snowplow) to programmatically extract the JSON Schema (http://json-schema.org/) which represents all JSON instances in the file.