HN Explorer

Ask HN: Review My Doohicky

19 points by ajkirwin ↗ HN
I got bored this weekend and decided I wanted to data mine twitter and over the course of a day and a half, I made this:

http://malicelabs.com/linkalyzer/

and I would like suggestions from people as to how to improve it and what they think of it. :)

Edit: Intermittently broke things and blew away some data intentionally. Add(ing|ed) in tinyurl/is.gd/etc through-finding of real URLs and page title grabbing.

13 comments

[ 4.5 ms ] story [ 29.0 ms ] thread
gsmaverick
Neat analyzer! I think there is so much data that can be gathered from Twitter that we have no idea exists.
riklomas
Maybe try and reverse engineer the TinyURLs so that users can see where they link to?
iamdave
This. Or maybe find a way to grab the page <title> from the target URL and create a separate column with the appropriate information.

Otherwise, nifty little app.

nirmal
Would be good to expand the tinyurls or at least provide the domain they are pointing too. This may even help you consolidate tinyurls if multiple services are used for the same original url. In Python I sometimes do the following to get the finalurl (not fool-proof):

  finalUrl = urllib.urlopen("http://www.tinyurl.com/5w5wod").geturl()
ajkirwin
I added this. A bunch of people demanded it. :)

As it stands, it has a few things that twitturly doesn't have, but also they have things I don't.

But, it's a beta product and hopefully soon, I can get a slicehost/linode server and get it off of dreamhost and I can spruce it up and add more features.

braindead_in
view more information gives an warning.

Warning: mysql_fetch_array(): supplied argument is not a valid MySQL result resource in /home/.dodi/jamesgarrett/malicelabs.com/linkalyzer/get_urlstats3.php on line 7

more warnings follow

joshsharp
Also getting various MySQL warnings.
ajkirwin
That was my bad! I added some fields to a table and forgot that'd crap all over my UNION.
wesley
Always always always turn off error display_errors on a production server :)
nirmal
Also, every image in your TwitPic section appears once as a png and again as a jpg. Some of these are broken but they still link back to the same TwitPic page.
physcab
Nice project. It would be interesting to see maybe a "popularity tree". Like for example, it would be fun to see if Scoble or Kawasaki put out the link first and see how it was distributed. Or if someone unknown sent out the link and someone popular picked it up. Not sure if its possible to trace though.

Also, I just started a topic on a data mining related note. If you have any thoughts, I'd love to read them. http://news.ycombinator.com/item?id=406705

jackowayed
Twitpic Hot 30 gives is missing about half of the images. There's spaces for them, but they just show the alt text.