A few months ago, a college friend reached out because his consulting company had just received a 60 Gb spreadsheet, and they didn't know what to do with it! They actually tried opening it in Excel.
I'm excited to Show HN CSV Explorer - a simple web tool for opening really big CSVs! Try it out, and let me know how it goes!
Not sure if you have this feature, but I think it would be really cool if you could open APIs to your own datasets. This could be really useful for enterprise applications that do a lot of flat file imports/exports to push/pull data.
Some consultant friends asked for help when they got a 60Gb CSV file from their client! Since then, I've also found that lots of tech companies share CSV internally. I've also worked with journalists attempting to look at large public government datasets.
I usually use csvkit (https://csvkit.readthedocs.io/en/1.0.1/index.html). There are commands to list the columns, filter, browse the data in a somewhat formatted way with less. Typing the commands is a huge pain though, and I would be very interested in a tool that could instantly pop open and let me peruse. Excel can take minutes to load and eagerly does a lot of unhelpful formatting on things like dates and decimals.
That being said I would never want to (and often legally cannot) etl my data to some third party. That would be terribly slow. But I would happily pay for a nice desktop tool to do it for me; command line or GUI.
Csvkit is great, and I use it a lot. The target users for CSV Explorer though are mostly non-engineers ie. people who don't use the command line or code.
Kibana has quite a few rough edges but it takes non-technical users very far in terms of data exploration. Local setup also alleviates privacy concerns.
The bummer is import/export: 'Upload CSV' feature has been almost introduced in 5.0 release but removed in the end [1], and CSV export has been asked for many times [2] but is still not there. So a user-friendly fork might be a worthy business idea.
At a former job, our embedded device logs decoded to csv. Some of them were too large for Excel.
Pandas handled them without a burp. Pandas in Jupyter (Ipython Notebook) was a godsend.
There's a minimal amount of variable setup, but once you've done that once or twice it's easy.
Of course, any analysis or manipulation takes a bit of python code, but I see that as a feature, not the least because you can read it right there in the open instead of having to hunt for formulas in cells.
Same here. I load the data using pandas or parsing the rows by hand in go. It's interesting to watch your RAM getting filled up while the data is loaded.
Looks like this tool is for non-programmers, it's interesting to see that there seems to be a market here.
I've been prototyping simple desktop GUI tools on top of dask/pandas and PyQt that let you lazily load large CSVs (and other types supported by pandas) and interactively filter based on smart histograms (the per column histograms are fully interactive and provide crossfiltering across the attributes):
The idea is to map a lot of the basic functionality of dataframes onto simple GUI interactions (for example, changing column types, stacking and unstacking columns, pivoting) and couple that with an ipython console for more complicated data manipulation. And then maybe even adding adding Tableau like charting functionality:
For quick throwaway exploration/analysis. It can easily handle about a million rows just using generic pandas and a bit of memory. There's lots of cool database techniques that can also be used on small local data (for example, compressed bitmaps using EWAHBool for interactive filtering).
Do you plan to release something soon? Even just a way to visualize the rows by loading them lazily would be a huge improvement. I personally use pandas but some of my colleagues are not familiar with it, and it pains me when they try to inspect a large dataset by opening it on Excel on our small university-provided desktops instead of spending a couple of minutes writing a few python lines to extract what they need.
I regularly test applications that generate big CSV reports. As I don't always have influence on the input data (we want to test on real datasets pulled from production servers of our partners), I fall back to defining constraints that must be satisfied. I ensure that they remain satisfied by analysing the csv files with powershell (of all the tools).
I just find the magic of "import-csv bla.csv | where some condition | select some existing or calculated value | group | format-list | out-file output.txt" to be extremely helpful, it's like SQL for the csv files.
And the ability to question live services and parse JSONs, and cross-check with other reports, or merge multiple reports into one... It's indispensable.
Think not of a desktop app, but of doing everything client-side in JS. That way, it's still a web app, but you're not schlepping [sensitive|large] data between front and back-ends. Also, by offloading the work onto clients, it scales much better - you could host the app on a CDN and have no real back-end.
I originally wrote it in entirely client side JS, but it didn't scale nicely past a few hundred thousand rows. For the really big datasets, CSV Explorer loads them into Redshift - queries takes a few seconds!
Perhaps - I tried a client-side implementation, but I had issues scaling past a few hundred thousand rows. I understand hesitations of uploading data to the cloud, but I now have users looking at hundreds of millions of rows in seconds thanks to Postgres!
I suggest exposing a config variable in the js client to set the hostname of the Postgres server. This way you can provide a self hosted version of the backend, and customers can self host it either locally on their machine, or within an internal network. Same benefits of the browser, just need to change config.
This tool would be great were it not for me (rightfully) getting fired and possibly sued by any of my clients if I uploaded even a single file. In fact, what kind of business with millions-of-rows kind of files would entrust said datasets to a service that to me seems strangely vague [1] on how the data is secured, or where it is actually going to be physically stored.
34 comments
[ 2.7 ms ] story [ 98.4 ms ] threadA few months ago, a college friend reached out because his consulting company had just received a 60 Gb spreadsheet, and they didn't know what to do with it! They actually tried opening it in Excel.
I'm excited to Show HN CSV Explorer - a simple web tool for opening really big CSVs! Try it out, and let me know how it goes!
Not sure if you have this feature, but I think it would be really cool if you could open APIs to your own datasets. This could be really useful for enterprise applications that do a lot of flat file imports/exports to push/pull data.
That being said I would never want to (and often legally cannot) etl my data to some third party. That would be terribly slow. But I would happily pay for a nice desktop tool to do it for me; command line or GUI.
[1]: https://github.com/elastic/kibana/pull/8497 [2]: https://github.com/elastic/kibana/issues/1992
That should give you a 10k-line random sample to play with that should open quite quickly in Excel.
Pandas handled them without a burp. Pandas in Jupyter (Ipython Notebook) was a godsend.
There's a minimal amount of variable setup, but once you've done that once or twice it's easy.
Of course, any analysis or manipulation takes a bit of python code, but I see that as a feature, not the least because you can read it right there in the open instead of having to hunt for formulas in cells.
Looks like this tool is for non-programmers, it's interesting to see that there seems to be a market here.
http://imgur.com/a/vfAmV
The idea is to map a lot of the basic functionality of dataframes onto simple GUI interactions (for example, changing column types, stacking and unstacking columns, pivoting) and couple that with an ipython console for more complicated data manipulation. And then maybe even adding adding Tableau like charting functionality:
http://imgur.com/a/z8d1w
For quick throwaway exploration/analysis. It can easily handle about a million rows just using generic pandas and a bit of memory. There's lots of cool database techniques that can also be used on small local data (for example, compressed bitmaps using EWAHBool for interactive filtering).
I just find the magic of "import-csv bla.csv | where some condition | select some existing or calculated value | group | format-list | out-file output.txt" to be extremely helpful, it's like SQL for the csv files.
And the ability to question live services and parse JSONs, and cross-check with other reports, or merge multiple reports into one... It's indispensable.
What's the value-add here?
[1] https://www.csvexplorer.com/legal/privacy/