HN Explorer

51 comments

[ 2.3 ms ] story [ 108 ms ] thread
adw
We've got quite a lot of public economic data: http://timetric.com/.

If you're up to something in the economic data space we'd love to talk. Happy to take this to email (andrew@timetric.com) if anyone's interested.

hessenwolf
I looked at the site, and I see some data but I didn't find what I would have hoped for. I couldn't find yield curves, and historical exchange rates <i> up to <i/> today (available on the ecb site in xml format). Certainly I would have thought yield curves were a front page item.

Things that would be very cool would be 1. financial statements in a database format. I know you can scrape this but I don't know if they are available legitimately? 2. Historial Implied volatilities and historical observed volatilities.

zipdog
The wikipedia dump is great, but I've started using http://wiki.dbpedia.org/ which has an API to query the dumps.

Thanks for these, iisbum. I wish more public data was available in db, xml or similar structures - too often I find myself scraping government sites or pdfs to get the tables I need

sosuke
Heh, a day after he leaves HN he makes the first page. He will still be here whether he visits the site or not.
dmpayton
Kinda surprised no one has mentioned Factual. I'm using some of their diabetes data for my side-startup.

http://www.factual.com/

casperc
Their write that most the data is available for download. I can't find it anywhere though, only the various APIs. Have they remove the possibility of downloading the data?
svag
There is also the IMDB database in various format provided by IMDB itself here: http://www.imdb.com/interfaces

Edit: Although the use of this database is not free, I believe for personal use is just fine to download and experiment...

wladimir
Wow, useful stuff. This thread goes into my bookmarks.
[deleted]
(comment deleted)
jmtame
i've had trouble finding geographical boundaries on neighborhoods in U.S. cities (e.g. downtown areas and residential neighborhoods). anyone know where i can find this?
_topher
Thank you all for posting links and links to links to datasets, I have an unrelenting interest in data aggregation and machine learning, and didn't even know where to start. So helpful, and I am no longer stuck. :)
fedd
do all of them have some uniformed api? that would be great, ideally. query and cache all of them on demand from your own app without additional programming.

bookmarked and shared this thread.

kaffeinecoma
This is a real treasure to come across. I hope we'll keep seeing jacquesm's blog postings here.

Anyone know of any publicly available song lyric databases?

bravura
get.theinfo is the best way to find data sets. They are a bunch of data hoarders who can help you: http://groups.google.com/group/get-theinfo/?pli=1

I always ask there if I can't find what I'm looking for.

Here are more and more data sets. These are general data sets. Email me if you have a specific data set in mind (e.g. web-as-corpus, spam, images, social, reviews, etc.). I have a big file of information.

    http://theinfo.org/
    http://infochimps.org/datasets
    http://ckan.org [Comprehensive Knowledge Archive Network]
    http://www.datawrangling.com/some-datasets-available-on-the-web.html
    http://del.icio.us/pskomoroch/dataset
    http://www.reddit.com/r/datasets/
    http://news.ycombinator.com/item?id=1242029
    http://www.reddit.com/r/opendata
    http://www.trustlet.org/wiki/Repositories_of_datasets
    http://www.daniel-lemire.com/blog/data-for-data-mining/
    http://www.quantlet.org/mdbase/
    http://datamob.org/
    http://freebase.com/
    http://infochimp.info/ics/data/ripd/www-personal.umich.edu/~mejn/netdata/
    http://www.archive-it.org/public/all_collections

    Large:
        http://www.ckan.net/tag/read/size-large
        http://www.diggingintodata.org/Repositories/tabid/167/Default.aspx
Web as corpus:

    Good instructions:
        http://corpus.leeds.ac.uk/internet.html#description
    http://sslmit.unibo.it/~baroni/bootcat.html

    http://www.drni.de/wac-tk/index.php/Documentation
etc. Email me if you need more http://cleaneval.sigwac.org.uk/ http://liste.sslmit.unibo.it/pipermail/sigwac/2007-November/... http://wacky.sslmit.unibo.it/doku.php?id= http://clic.cimec.unitn.it/marco/research.html