HN Explorer

8 comments

[ 3.1 ms ] story [ 19.5 ms ] thread
JonnieCache
I've traditionally managed this with just 1 LoC:

    wget --mirror --page-requisites --convert-links localhost:9292
But this is probably more robust.

EDIT: for bonus points, cd into your wget-created mirror, fire up an http server with

    python -m SimpleHTTPServer
Then check for any broken links with

    wget --spider --page-requisites locahost:8000
wget is cool.

Ultimately though, correct use of cache headers and varnish or similar gives you the same effect automatically, if you're just looking for speed.

jamesbritt
I don't know if this is clever or not, but it was practical for me: I wanted to have some basic sanity checks on all public (i.e. non-log-in) pages of a site, so I used wget to mirror it. What I was really after was the wget progress output so I could get a list of all the page URLs. I then used that to drive Watir to hit every page and a) check for any 404 or such errors and b) take a screen shot and save it off.

I could then fire up an image browser and quickly see if any pages looked wonky ( missing or misaligned images images, crappy text layout, etc.).

wget is very cool. :)

alperakgun
if your web site hits high traffic, will be quite useful to create a static version; a nice "enterprise feature" in sinatra.
DanielRibeiro
You do realize, of course, that you can do it in one line right?

    require 'sinatra'
Just put everything else in the public subfolder of the script, and run it.

If you want the public folder to be the root, you can make it in two lines:

    require 'sinatra'
    set :public_folder, '.'
paulasmuth
the point is, that the pages are pre-rendered to static *.html files and can be served by a lighttpd or apache much faster and without any dependencies (ruby for example)
tptacek
Besides speed, static site generators solve a security vs. convenience issue; most dynamic CMS systems are more pleasant to use (especially for casual users) than Jekyll or Nanoc, so the ability to run one internally and pickle it up to static HTML files makes it easier for people to front-end their companies with static files instead of error-prone dynamic web apps.

If you can get away with static content for your public facing website, or at least most of it, I think it's generally worth it for you to do that.