Ask HN: Transcript parsing and NLP?

2 points by staticautomatic ↗ HN
I'm considering embarking on a project to parse and analyze semi-structured pure text transcripts. I will need to identify and be able to quickly retrieve things like "all things a given person has said" and "all things said about [insert word or phrase]." I don't know the exact number of documents I'll be dealing with but let's say it's about 10K pages of text. I have a few questions and am hoping someone may be able to offer some guidance. I'll be doing the project in Python.

1. Are there any decent open source packages for transcript analysis? I can't seem to find much except a few little projects on github that deal mainly in specific kinds of transcripts (e.g. movies, debate transcripts, etc.). I had hoped to find something in the legal tech world relating to deposition or trial transcripts but I haven't seen anything and most of those tools are proprietary.

2. Are there canonical approaches to parsing, tagging, and structuring data extracted from transcripts? I'm particularly interested in something like a DOM structure where I can easily traverse nodes in the document and query things like sibling elements (e.g. What was the question that this answer was in response to?).

3. What databases or document formats should be considered for the task? I will occasionally add new transcripts to the DB but it will otherwise be essentially read-only once constructed. I figure one option would be a document DB like ElasticSearch, but I haven't used it before. A second option I figure would be some other DB like Mongo with each transcript stored as JSON, but I'm not sure how I'd go about storing the meta-data in JSON and whether it could go into the same doc or should be in a separate one. A third option would be using a SQL database along with something like NLTK where I have columns for meta-data and word tokens.

0 comments

[ 1.1 ms ] story [ 12.9 ms ] thread

No comments yet.