Ask HN: What should I learn if i want to manage and parse lots of data?
I have some large csv files that I'd like to parse for information, sort, and just figure interesting things out. I don't really know how to code, but I'd like to be able to create a searchable web interface for this data. What are the best programming languages or frameworks I should learn to accomplish this?
12 comments
[ 3.2 ms ] story [ 46.5 ms ] threadEdit: without knowing the specifics, you might like to look at Elasticsearch (http://www.elasticsearch.org/). That'll give you an HTTP/JSON API into your data, so you might be able to do all the UI work you need client side, e.g. with Backbone.js (http://backbonejs.org/) or similar.
cat input_file.csv | sort | uniq > output_file.csv
If you are not a programmer, you might take a look at baseX. It would allow you to import your csv then use xquery to explore your data as xml
If you have the data in CSV files, it's probably not that large. Consider using Google Fusion Tables ( https://developers.google.com/fusiontables/docs/v1/sql-refer... ) You can set up simple queries by using SQL syntax (it's easy to learn).
After that, you can set up a simple interface by connecting with Google Apps Scripts, which is easy to learn and to create a simple interactive web site with.
Insert the results of the parsing into any popular database.
Google "<my_chosen_database> delete duplicates"
You need to talk about the number of entries you have though, to give us an idea of the sort of tools you need.
* DBD::CSV - Treats your CSV file like it was a RDBMS so can use SQL to inspect/change it - https://metacpan.org/module/DBD::CSV
* Parse::CSV - For parsing large CSV files - https://metacpan.org/module/Parse::CSV
* Text::CSV_XS - Fast CSV parser - https://metacpan.org/module/Text::CSV_XS