HN Explorer

Ask HN: Recommended language to learn data manipulation

10 points by digdugdirk ↗ HN
I have a few very large datasets (in csv format) that I need to work with that completely overwhelm Excel. I'm looking for recommendations on a single language (or minimal number of languages) that I could learn in order to take these datasets in and start running various calculations on.

Simplicity and future usefulness of the programming environment I'll be learning are my highest ranked criteria.

I'm not necessarily looking for a how-to book (though I wouldn't turn it down) I'm really just looking for language advice. There's a lot out there on the Internet, and much of the advice is conflicting.

9 comments

[ 3.5 ms ] story [ 36.1 ms ] thread
dozzie
SQL. PostgreSQL has decent CSV support.
apohn
100% agree. I've loaded multi-gigabyte datasets into Postgres on my laptop and SQL is a great language for manipulating and aggregating data. Plus, when you use a database and SQL you can access that data from a wide range of applications and languages via drivers (JDBC and ODBC).

You can even access Postgres from Excel!

Jugurtha
>Simplicity and future usefulness of the programming environment I'll be learning are my highest ranked criteria.

Given this criteria, I think SQL would be a good choice: simple to get started with, useful in the future.

SQLite doesn't get in your way.. https://sqlite.org/

throw_away_777
What do you mean by large? Probably as your data is in csv format it is small enough that python or R are both good choices. Personally I'd recommend python as it has a large community using it so googling questions about python libraries such as pandas usually provides good answers.
ptype
Will it fit in ram? Python + pandas. Pandas is really amazing.

Otherwise, try postgresql or SQLite.

mindcrash
Any language is suitable for data processing, but if you want something which is written especially for data processing and analysis, and especially data processing and analysis for large scala data sets, I personally would go for R: https://www.r-project.org/.