HN Explorer

Ask HN: Best way to visualize / analyze website stats in realtime?

12 points by joevandyk ↗ HN
Say I'm running an e-commerce website.

I want to be able to (data updated once a minute):

- See how many orders we're getting. I want to be able to answer questions like "are we taking more orders this evening than normal?" and "what hour do we make the most profit each day?"

- How many people people did X? (where X could be commenting, playing games, registering, tweeted about us, etc)

There's a lot of analytics software up there, but not many let you break data down into minute or hour chunks of time. Also, we'd want to be able to customize what data is being monitored.

We already use Chartbeat and Google Analytics. Chartbeat doesn't let you customize data (so we couldn't see how much profit we make per hour, for example), and Analytics isn't realtime at all.

15 comments

[ 4.0 ms ] story [ 43.8 ms ] thread
gilbax
Is Chartbeat the only real time analytics engine on the market ? ( I'm asking this because I assume you invested more than me, and I'm interested in knowing the current state of this market )
elliottcarlson
You could look in to customizing Hummingbird Stats (http://hummingbirdstats.com/) - the real time factor is already there; you would just have to add some additional things to track against.
joevandyk
Hummingbird doesn't look it supports going back in time well?

i.e. we'd want to ask "how many t-shirts did we sell yesterday".

joevandyk
Plus I think we want something that doesn't require javascript.
elliottcarlson
I guess I misunderstood your requirement of real time - I am guessing you meant that you want to see up to date information when visiting a page?
joevandyk
Yes, pretty much. It's fine to hit refresh and see the data as of a couple minutes ago.

Needing to wait a day (as in Google Analytics) would be too much.

elliottcarlson
That's where the customizing comes in. The actual data is stored and can be referenced using other tools/extensions/customizations.
pmjoyce
Excuse the blatant plug, but my startup http://geckoboard.com might be a reasonable fit for the visualise side of your requirement. We pull data from a bunch of different sources (e.g. Google Analytics, Chartbeat, Zendesk, Freshbooks etc out of the box, but we also allow custom data uploads) and display them on a status board.

You can use a script on your own server or a YQL query to create the file for custom data and we pull it at regular intervals into your status board. Earlier this week we added support for custom chart widgets.

We're in an invite freeze period for our private beta right now, but if you mail me (details in my profile) I'll send yours out.

Just to re-iterate; we don't help you gather stats; we display stats you've already got and want to visualise in near real time.

joevandyk
Is it possible to view stats in the past?
pmjoyce
That depends on what you mean, for the custom data uploads you can feed it anything. For Google Analytics you can view the stats (e.g. pageviews) for today (partial), yesterday, past 7 days & past 30 days and compare them to the previous period e.g. page views in the past 30 days are up 12.5% on the previous 30 days.

However, we don't store data (besides temporarily caching it) so if you want to view history it has to be present in the existing feed.

joevandyk
When you query the feed, can you pass in date/time ranges? So, we could query last month?
pmjoyce
There are 2 types of feed we accept. The first is a standard connection to one of a variety of web services such as Google Analytics where you authenticate (in this case via OAuth) then choose what you want to display. The second type is custom data uploads where you simply point widgets on your dashboard to an XML/JSON source with an explicitly defined structure and we pull it on a regular basis.

The first type use logic which we have defined up front to extract the appropriate information for display. We apply little or no logic on the custom data feeds as that is assumed to have been carried out by the provider, usually when the XML/JSON file is generated.

If you have an obscure source of data, or it's sourced from a service we don't yet extract from, then I'd suggest writing a script to extract and massage the data into a format we accept and point your dashboard at it. We have definitions and example scripts in our support forum and I'm more than happy to give you a hand setting it up if that would be useful for you.

fudge
A good way to visualize what happens on your site right now is glTail at http://www.fudgie.org/ - you can do a lot of the real-time queries you want with a custom parser.