HN Explorer

96 comments

[ 2.4 ms ] story [ 167 ms ] thread
spontaneus
I had this same idea a while back, but never had the time to build it. It could be a great way to generate some affiliate money. Good luck!
clobber
Tested this on a website written in ColdFusion and it's reporting back as ASP.NET
arkitaip
I tried ColdFusion powered metafilter and it reported nothing.
tectonic
Please add a matcher for ColdFusion - I don't know it, but this is a wiki, so anyone can add it.
arkitaip
I just understood the concept of tech matcher, which could be an incredibly powerful concept. I think you should highlight this more.
rmason
I had the same experience with several sites that I own. No asp on the server, some others report just apache or IIS and completely miss ColdFusion.

Even Adobe.com, a ColdFusion site, doesn't list it.

ashwinurao
A Chrome/FF extension would be much more useful.
tectonic
There is a bookmarklet. Browser extensions are forthcoming.
Nican
I would like to see the list of websites using IIS.
[deleted]
(comment deleted)
tectonic
Hey guys, I made this site and just gave a talk about it at SHDH. Someone must have submitted it. Thanks for all your feedback, I really appreciate it!
mmaunder
Awesome! Suggestions:

Do a reverse lookup on the site's IP and add matchers for the hostnames that show up.

Mention if they're round robining their DNS.

Add nmap OS fingerprinting.

Do a traceroute and log the IP of the closest router (final hop) to the site and add matching for that.

Add a wiki interface and build a crunchbase like app.

Add archiving of data and monitoring over time (as netcraft did in their original app).

JoshTriplett
nmap OS fingerprinting seems a bit aggressive, but reverse-DNS checking seems like a great way to check which hosting platform a site uses.
[deleted]
(comment deleted)
16s
nmap is too aggressive. It's a prelude to actual hacking attempts and labeled by IDS systems as such. Don't use it for this or you may end up in legal trouble.
tectonic
Agreed. UnderTheSite makes a specific point of only considering information that would be returned when a user's browser hits a website. It doesn't scan or probe ports / urls.
tectonic
Thanks for the great feedback!
suyash
I posted it buddy, I liked your demo a lot!
jf
Awww, I wish I didn't have to leave SHDH before the lightning talks started!

I wrote a Ruby program to do something similar to what you're doing: https://github.com/jpf/domain-profiler - If you ever start profiling sites using information from places other than what the server returns, perhaps what I've done can help inspire you?

adrianwaj
Is there a working example of domain profiler please?
tectonic
Follow @underthesite for updates as I implement all of the great suggestions that you guys have made here.
[dead] seanp2k
I was there! Great presentation, but Wapalyzer and WhatWeb already do the same thing as this, as far as I can tell.
dorkitude
Cool site! I'm having a lot of fun. However, I was confused by this flow:

First: http://drktd.com/8Jgn

After clicking the 'Add It!' link: http://drktd.com/8Iaf

The wording on the previous page had me expecting a page for manually adding technologies to my site's stack a la Bagcheck.com.

tectonic
Ah! This is a feature that I plan to add. Do you want to select technologies from a list, or write free-form text about your technology stack?
[deleted]
(comment deleted)
JoshTriplett
I'd like to suggest avoiding manual additions of technology for as long as possible. Focus on adding more ways to match specific technologies. After all, a site could always advertise more technologies in server headers or meta generator tags.
tectonic
What if we propose an extension to humans.txt called technologies.txt where a site can self-describe their stack?
JoshTriplett
A mechanism for sites themselves to advertise their stack seems like a great idea (though I'd prefer it not occur via fixed URLs like humans.txt or robots.txt, but via headers or meta tags). I'd just suggest not allowing arbitrary additions to a site's stack without any way to verify them.
thomasfl
Would love to see if it could detect more scripting languages and frameworks than PHP.
araneae
Nice tool! Usually when I see a site I like I view source to see what it's made in, but it's not always that obvious.
shuri
my 2c. Nice design (graphics). I would make the first page seem less busy. BuiltWith is going to be a tough competitor. You need to match them (precision/recall) and add stuff that they don't have (trending techs? Add info: who is the host provider? where in the world is it hosted? Response times? Bad link stats?...?)
[deleted]
(comment deleted)
JoshTriplett
I love this. I particularly like the broad definition of "technologies", and the variety of ways to write matchers.

More importantly, I like that users can easily add their own technologies and matchers.

kingkilr
Interesting, but seems a little primitive compared to the cleverness in https://github.com/mitsuhiko/probe
tectonic
Does probe try and fetch signal urls? (Like admin pages.). UnderTheSite.com makes a point of only looking at data that would normally be fetched by your browser. Probing a site for urls could be considered offensive.
kingkilr
Yes, it does probe at site URLs, you can look at libprobe.py to see all the indicators it takes into account.
aphexairlines
This doesn't really tell you anything about a site. Most of the interesting stuff happens on the server side.
JoshTriplett
What triggers the error message "The pattern that you entered appears to be too general - can you make it more specific for this technology?"? I tried to add a technology for the use of rel="nofollow", using an XPath expression //a/@rel[contains(., "nofollow")] , and got that error message. What do I need to do to make it more specific?
tectonic
Good question, I think rel=nofollow must match too many websites. I'll look into it.
JoshTriplett
Please consider allowing a small icon for a technology, to make lists of technologies more immediately recognizable.
[deleted]
(comment deleted)
[deleted]
(comment deleted)
JoshTriplett
Looks like you've already started to get spam in the technologies list.
nbertram
Nice site guys; reminds me of www.quarkbase.com in some ways. But I like the focus on what's under the hood - keep up the good work.
aorshan
Noticed that not a single website is using django. Am I missing something?
tectonic
I'm not sure that there is a django matcher yet. Django is hard to reliably detect. Do you know of a good signature (like a header or form token) that it always uses?
lloeki
You could implement "maybe" matchers, that look for stuff which has a good chance of telling you what's underneath but not with 100% certainty.

Now, the following is a hack, but CSRF stuff[0] gives a good indication:

If there's a form, there's a good chance you'll find a construct like:

    <form action="." method="post"><input type='hidden' name='csrf_token' value="thetokenvalue" />
Also, there may be a 'csrftoken' value in the cookie.

You will also most probably find the jQuery function that sets X-CSRFToken on XHRs (see the doc[0] at #ajax). For prototype it'll look like this [1]

[0] https://docs.djangoproject.com/en/dev/ref/contrib/csrf/

[1] http://stackoverflow.com/questions/5551914/protecting-protot...

nfm
Bang on for my app's site! Very cool.

In terms of features, I'd love to see more emphasis on the aggregate/comparison data. For example, most popular server side framework, most popular JS libraries, most popular hosting platforms and so on.

I like it a lot :)