HN Explorer

Ask HN: What are the best tools for analyzing large bodies of text?

83 points by CoreSet ↗ HN
I'm a researcher in the social sciences, working on a project that requires me to scrape a large amount of text and then use NLP to determine things like sentiment analysis, LSM compatibility, and other linguistic metrics for different subsections of that content.

The issue: After weeks of work, I've scraped all this information (a few GB's worth) and begun to analyze it using a mixture of Node, Python and bash scripts. In order to generate all of the necessary permutations of this data (looking at Groups A, B, and C together, A & C, A & B, etc), I've generated an unwieldy number of text files (the script generated > 50 GB before filling up my pitiful MBP hard drive), which I understand is no longer sustainable.

The easiest way forward is loading this all into a database I can query to analyze different permutations of populations. I don't have much experience with SQL, but it seems to fit here.

So how do I put all these .txt files into a SQL or NoSQL database? Are there any tools I could use to visualize this data (IntelliJ, my editor, keeps breaking). And where should I do all this work? I'm thinking now either an external hard drive, or on a VPS I can just tunnel into.

Thanks in advance for your advice HN!

55 comments

[ 4.1 ms ] story [ 115 ms ] thread
drallison
It seems to me that you are driving from the wrong direction. Given that you have a large body of text, what is it you want to learn about/from the text. Collecting and applying random tools and making measurements without some inkling about what you want or expect to discover makes no sense. Tell us more about the provenance of your corpus of text and what sort of information you want to derive from the data.
danso
If I could upvote your comment 10 times, I would. It doesn't seem that the OP knows what exactly they want, and so the question of whether they should be doing things in Node/Python/Bash/SQL/NoSQL is putting the cart way before the horse. It's quite possible grep -- and some regex, and maybe some scripting logic for tokenizing -- is all that's needed, if the hypothesis is clear enough. But right now, it just sounds like OP has a ton of text and is hoping that some tool can somehow produce an unanticipated, useful insight...in my experience, that rarely (OK, never, actually) happens.
ganeshkrishnan
I was wondering what exactly is OP looking for. Most probably he forgot to put his problem in.

If he just wants to parse the text his bare minimum ubuntu install is enough with grep, cat etc.

contingencies
Yep.

Success really depends on the conception of the problem, the design of the system, not in the details of how it's coded. - Leslie Lamport

You're not going to come up with a simple design through any kind of coding techniques or any kind of programming language concepts. Simplicity has to be achieved above the code level before you get to the point which you worry about how you actually implement this thing in code. - Leslie Lamport

You're not going to find the best algorithm in terms of computational complexity by coding. - Leslie Lamport

Sometimes the problem is to discover what the problem is. - Gordon Glegg, 'The Design of Design' (1969)

The besetting mistake of expert designers is not designing the thing wrong, but designing the wrong thing. - Frederick P. Brooks, 'The Design of Design: Essays from a Computer Scientist' (2010)

... then again ...

In practice, designing seems to proceed by oscillating between sub-solution and sub-problem areas, as well as by decomposing the problem and combining sub-solutions. - Nigel Cross

... but did you want re-use? ...

A general-purpose product is harder to design well than a special-purpose one. - Frederick P. Brooks, 'The Design of Design: Essays from a Computer Scientist' (2010)

Design up front for reuse is, in essence, premature optimization. - AnimalMuppet

... lines of thought from my fortune clone @ https://github.com/globalcitizen/taoup

abannin
I'd like to understand a bit more about your data structures. Raw text should live in a file system, but if you're creating highly structured data using a SQL database could be a good fit. I'd also suggest taking a look at Spark; you can run it locally on your machine (does not need a Hadoop cluster) and can interact with flat files. If you're finding that your machine is lacking in horsepower or space, AWS could be your friend.
quizotic
Man, this comment is simultaneously both SO right and SO wrong. I guess the tools we have today are set up for knowing what you want to learn. Named Entity Retrieval? Topic classification? Sentiment Extraction?

OTOH, why should OP have to know what's interesting a-priori? The mantra of big data is "listen to the data, let it tell you, leave your preconceived notions at the door." The fault is with our current tool chain. We need something that tells us what about the text is interesting before we dive in for a closer look.

I'm imagining a tool that told me: "this text seems to have a lot of opinions and sentiment," "it's about a product that was returned," "a number of people's names are mentioned," "this text was loaded along with some structured data that appears to reference a price, a location, and a date."

Why is it such a stretch to combine the tools we already have to generate and push summarizations? Maybe it's just the cost of computation? If you know you're looking for topic and don't care about sentiment, then you can avoid paying for it?

skadamat
The first immediate thing I would recommend is moving all of your files into AWS S3: http://aws.amazon.com/s3/

Storage is super cheap, and you can get rid of the clutter on your laptop. I wouldn't recommend moving to a database yet, especially if you don't have any experience working with them before. S3 has great connector libraries and good integrations with things like Spark and Hadoop and other 'big data' analysis tools. I would start to go down that path and see which tools might be best for analyzing text files from S3!

nhaehnle
AWS-stuff is also a new tool that first has to be learned, and frankly it sounds like overkill for this kind of problem (not to mention that it probably comes with its own kind of overhead).

lsiebert's comment about not generating such large text files in the first place is a good one: If you already have a tool that generates the text file, and you already have the tool that analyzes it, it should be possible to just run them simultaneously via some form of pipe, so that the intermediate result never has to be stored.

Finally, if this is not possible for some unclear reason and depending on the kind of space and throughput that really is required, simply getting an external disk may be the better way to go.

lsiebert
What social science?

You shouldn't be generating the text in advance and then processing it. You should be dynamically generating the text in memory, so you basically only have to worry about the memory for one text file at a time.

As for visualizations, R and ggplot2 may work (R can handle text and data munging, as well as sentiment analysis etc.) It may be worth using it as a social scientist.

ggplot2 has a python port.

That said, you are probably using nltk, right? There are some tools in nltk.draw. There is probably also a user's mailing list for whatever package or tool you are using, consider asking this there.

fole
I worked for an NLP research think tank for a while and we always created text files as intermediate steps to each part of our system. It was basically a cache of each step, and you could restart the system at whatever step did work.

Hard drive space is cheap. Use as much as you want.

lsiebert
Making it clear that he doesn't need to buy equipment is a good thing. I agree with you that logging results as you go is worthwhile, but for data munging, I think it's better to keep your data in it's original source, and document how you get your data into the system in code, and not require somebody reproducing your results to have a huge HD or buy something.

As an aside, I was also a social scientist originally. My first degree was in Psychology. The first time I felt like a programmer was taking supplied R code that would have taken 8+ days to finish (2400 Rausch scores at 5 minutes each), and got the whole thing to run in less than a minute by moving from sequential search of every possibility to a probing strategy to find the score that best fit the curve. Learning how to be more efficient in your code, to use less space, or time through a better algorithm to handle your data, is both useful in it's own right, and intellectually rewarding.

wodenokoto
> You shouldn't be generating the text in advance and then processing it.

That depend on how long the preprocessing take and how often you need to manipulate the processed text.

abannin
R is going to give you some headaches as it relies heavily on the local machine's memory. Using RStudio on a beefed up AWS instance might help make calculation time a bit more palatable.
pvaldes
I totally agree, but If you want to use R consider also this:

install.packages("LaF")

Package for fast access to Large ASCII Files. Can be used with big files that don't fit in the available memory.

koopuluri
What tools exactly are you using in Node and Python? Python has a nice data analysis tool Pandas(http://pandas.pydata.org/) which would help with your task of generating multiple permutations of your data. Check out plot.ly to visualize the data (it integrates well with a pandas pipeline from experience); It would also help if you mentioned exactly what kind of visualizations you're looking to create from the data.

With regards to your issue of scale, this might help: http://stackoverflow.com/questions/14262433/large-data-work-...

I had similar issues when doing research in computer science, and I feel a lot of researchers working with data have this headache of organizing, visualizing and scaling their infrastructure along with versioning data and coupling their data with code. Also adding more collaborators to this workflow would be very time consuming...

cmarciniak
More information would be helpful. In terms of having a data store that you can easily query text I would recommend Elasticsearch. Kibana is a dashboard built on Elasticsearch for performing analytics and visualization on your texts. Elasticsearch also has a RESTful api which would play nicely with your Python scripts or any scripting language for that matter. I would also recommend the Python package gensim for your NLP.
ashley
Ah, somehow I skipped your comment when reading through them; otherwise I would've just upvoted instead of posted :)
nutate
Right now the fastest alternative to nltk is spaCy https://honnibal.github.io/spaCy/ definitely worth a look. I don't know what you're trying to do with the permutations part, but it seems like you can generate those on the fly through some reproducible algorithm (such that some integer seed describes the ordering in a reproducible way) then just keep track of the seeds, not the permuted data.
[dead] nutate
Right now the fastest alternative to nltk is spaCy https://honnibal.github.io/spaCy/ definitely worth a look. I don't know what you're trying to do with the permutations part, but it seems like you can generate those on the fly through some reproducible algorithm (such that some integer seed describes the ordering in a reproducible way) then just keep track of the seeds, not the permuted data.
mark_l_watson
One approach is to put text files in Amazon S3 and write map reduce jobs that you can run with Elastic MapReduce. I did this a number of years ago for a customer project and it was inexpensive and a nice platform to work with. Microsoft, Google, and Amazon all have data warehousing products you can try if you don't want to write MapReduce jobs.

That said, if you are only processing 2 GB of text, you can often do that in memory on your laptop. This is especially true if you are doing NLP on individual sentences, or paragraphs.

chubot
This sounds like an algorithmic issue. How many permutations are you generating? Are you sure you can scale it with different software tools or hardware, or is there an inherent exponential blowup?

Are you familiar with big-O / computational complexity (I ask since you say your background is in social sciences.)

A few GB's of input data is generally easy to work with on a single machine, using Python and bash. If you need big intermediate data, you can brute force it with distributed systems, hardware, C++, etc. but that can be time consuming, depending on the application.

jaz46
I'd have to know a little more about your setup to be sure, but Pachyderm (pachyderm.io) might be a viable option. Full disclosure, I'm one of the founders. The biggest advantage you'd get from our system is that you can continue using all of those python and bash scripts to analyze your data in a distributed fashion instead of having to learn/use SQL. If it looks like Pachyderm might be a good fit, feel free to email me joey@pachyderm.io
gt565k
apache solr or elasticsearch
codeonfire
If you want high performance and simple why not use flat files, bash, grep (maybe parallel), cut, awk, wc, uniq, etc. You can get very far with these and if you have a fast local disk you cat get huge read rates. A few GB can be scanned in a matter of seconds. Awk can be used to write your queries. I don't understand what you are trying to do, but if it can be done with a SQL database and doesn't involve lots of joins then it can be done with a delimited text file. If you don't have a lot of disk space you can also work with gzipped files, zcat, zgrep, etc. I would not even consider distributed solutions or nosql until I had at least 100GB of data (more like 1TB of data). I would not consider any sort of SQL database unless I had a lot of complex joins.
cactusface
This is definitely the most straightforward way to go. It is almost certainly not the way he will go. In particular Awk is just great for this job and very easy to learn.
tedchs
Have you considered Google Bigquery? It's a managed data warehouse with a SQL-like query language. Easy to load in your data, run queries, then drop the database when you're done with it.
developer1
Does the NSA allow its employees to look for help from the general public like this? Seems odd for such a secretive organization to post publicly asking for help on how to parse our conversations.
huskyr
Where do you see NSA in the question of the OP?
machinelearning
We're building a database specifically to solve this problem. We're almost production ready and will be going OpenSource in a couple of months. Send us an email at textedb@gmail.com if you'd like to try it out. http://textedb.com/
ninebrows
Is it a standalone tool or a web based offering?
machinelearning
standalone tool, which can be deployed on a remote host identical to any other database
halayli
it's not clear what your objective is. I can have 1kb text file and end up with a 1TB file after "analyzing" if I don't have a goal in mind.
[deleted]
(comment deleted)
ashley
>>I'm thinking now either an external hard drive, or on a VPS I can just tunnel into. Consider setting up an ElasticSearch cluster somewhere, like with AWS, which takes plugins for ElasticSearch. Once you've indexed your data with ES, then queries are pretty easy (JSON-based). This would also solve your other problem with data visualization. ElasticSearch has an analytics tool called Kibana. Pretty useful and doesn't require too much effort to set up or use. I'm using this setup for a sentiment analysis project myself.

You didn't mention the libraries in your NLP pipeline (guessing NLTK bc of the Python?), but if you're doing LSM compatibility, I'm guessing you might be interested in clustering or topic-modelling algorithms and such...Mahout integrates easily with ElasticSearch.

hodwik
Are there libraries you would suggest for someone using nltk & ElasticSearch to get started doing sentiment analysis?
[dead] ashley
>>I'm thinking now either an external hard drive, or on a VPS I can just tunnel into. Consider setting up an ElasticSearch cluster somewhere, like with AWS, which takes plugins for ElasticSearch. Once you've indexed your data with ES, then queries are pretty easy (JSON-based). This would also solve your other problem with data visualization. ElasticSearch has an analytics tool called Kibana. Pretty useful and doesn't require too much effort to set up or use. I'm using this setup for a sentiment analysis project myself.

You didn't mention the libraries in your NLP pipeline (guessing NLTK bc of the Python?), but if you're doing LSM compatibility, I'm guessing you might be interested in clustering or topic-modelling algorithms and such...Mahout integrates easily with ElasticSearch.