I wanted to try Volument but I noticed that their website wasn't redirecting to https, they had set the HSTS header and forgot about http since they only ever saw https on their browsers. This put me off because I thought, if they got this wrong, who knows what else they got wrong?
I emailed them and they replied very soon and fixed the problem within a day, though, so at least response time was excellent.
Cheers for shipping a product! I recall the post, and as a heavy Google Analytics user the landing page made some very salient points. This post was honest and challenges the fantasies of many hopeful hackers. "Catastrophic Success" is no joke.
I can sympathize with the challenges of running a web scale analytics tool, after helping build and run an analytics stack for an ad network. Even with our relatively small group of customers and lots of planning, the data was abundant, the queries became quite slow, and the pipeline was never "real-time" enough for some.
The web is a noisy place! URLs parameters are tricky to canonicalize into pages when dealing with many properties. There is a constant stream of nonsensical referrers and even referrer spam. Traffic can be exceptionally spiky, and cascading failure in the pipeline is inevitable without careful planning. Data loss is irrecoverable, and any incident needs to be carefully explained to users of the product so they don't deceive themselves into making decisions based on missing data.
One specific point I'm concerned about from the post is that while comparing HackerNews traffic to organic search, their analysis fails to account for relative traffic volume. Its always easy for a small volume traffic source to show outstanding metrics, but miss the broader point: quality traffic sources usually don't scale. A useful analysis should assign weight to overall volume. ORDER BY MAX(CTR) will tell me that some keyword/referrer was the top performer at 80%, but without knowing that was for 3 visitors I could easily get the wrong ideas. I think analytics tools aspire to make it difficult to come to incorrect conclusions, acknowledging that most of their users will have no meaningful background in statistics.
All of those challenges are surmountable, but it definitely gives some credence to the talented engineers who have make analytics stacks look easy. I wish Volument luck in their journey and will keep an eye on their progress, because their pitch remains quite appealing.
Volument shows the size of each segment and doesn't generate any report unless the size is "sufficient". However, it doesn't reward bigger segments when calculating the score for traction.
I was expecting this to be the usual "Hacker News commenters were awful and made me feel horrible", but was pleasantly surprised that Volument found value from launching here.
Volument reveals us, for example, how the people that come from Google search generate much more traction than the other market segments. A super valuable piece of information that we will use on our marketing.
Does it show that people coming from HN want to glance at your startup and see what it does, whereas people coming from Google have no idea what to expect and take longer to work out who you are and what you do?
Does it show that neither Google users nor HN users know what you do, but HN users are more experienced with websites and spend less time mentally categorising them?
Does it show that HN users are less interested in marketing / conversion than people who Googled for a related term enough to find your site?
What's the super-valuable insight and how will you use it?
It indeed shows which market segments just glances the startup (3, 7, 15 second stay rates) and which segments find the content more interesting. It doesn't tell you the exact emotional reason why someone leaves early.
The super-valuable insight is to know how the various segments proceed on the AIDA funnel (attention, interest, desire action) so you understand how your visitors convert.
It also reveals your best target markets and landing pages that generate the most traction. This data is not available elsewhere.
This article comes across as being really fake. You can clearly tell that the goal is to to drive additional traffic to their v1.0001 instead of actually diving into what they did wrong. Were they really investigating an entire new DB and user interface in the few days after launch? If your initial design is so far off that it requires a complete rewrite on day 1 of launch, then you have serious problems. That is not “avoiding pre-mature optimization”. That is an architectural failure.
Let's not jump the guns too soon. Analytic products are notoriously hard from scaling perspective. UI, APIs, Infra that works crisply for 1x data, starts to become sluggish at 10x and really breaks down at 100x.
Startups solve the 100x scale problem when they get there (sensibly so). In this case, they got there way too quickly thanks to HN traffic.
IMO, they probably failed way before 100x expected usage. It was probably before 10x expected usage. This isn’t crazy. This is a problem that happens for a lot of teams dealing with scale. You engineer for 1x (checking off requirements) and anything over that tears everything to shreds. The difference between developing a solution on your local and hitting your PERF environment is massive. e.g. I can’t run a TB DB on my local, so... if I don’t see queries running in less that 10ms-100ms, I basically know that the queries are never going to complete in PROD. Scale is hard.
It's a strange type of depression where you come right back for more. "Learned helplessness"? Masochism?
I'm slightly irked by this use of "depression" anyway. It somewhat trivialises one of the top 10 deadliest diseases, whose sufferers are already facing the constant pain of their disease being considered "fake" or a lack of character or laziness etc.
Sorry for being the "language police". It's just a minor point, and I know the word is commonly used in this way. But even ignoring the issue above, I think there are simply better words for the specific situation: "depression" evokes the spectre of inactivity, of helpless, and of despair. But you did tackle the problems that came up, so words such as "challenging" (yeah, I know), "frustrating", "stress", "maelstrom", or "adventure" may not only improve the text's language, but also project a more positive image for potential customer.
I always ask my clients wanting to sell their manufactured products online : are you ready if the website is successful fast ? Can you handle twice your actual sales ? Ten times ?
I looked at your problem page and it's not true that Google Analytics bounce rate is not useful for single-page applications.
You can trigger events with JavaScript and choose whether they affect bounce rate, so you can manually trigger an event when the user scrolls down, clicks something, spends a few minutes on the site, etc. So your problems with the bounce rate metrics are just not true.
Also I saw you mention it doesn't track returning users, that's not true either. I can actually see insights into how many users return month after month, and it'll tell me what my activity is per channel and if one channel outperformed others, so it's pretty powerful in this regard for a free platform.
I skimmed it but from just reading those it sounds like you should try to get better familiarity with Google Analytics and talk to users to see what their main pain points with existing platforms are, as well as establish more concrete value propositions. Google Analytics is free after all, I'm not convinced yet to move to a paid platform that tries to address problems I'm not experiencing.
Google Analytics bounce rate isn't useful for single-page applications unless you monitor user activity with JavaScript and GA event tracking. Not everyone is aware of this and/or can do this. We should update the docs with regards to events.
GA can do retention reports or "cohort analysis" and show how visitors return in a monthly basis, but there are two critical limitations on it:
1) It doesn't show you which landing pages, target markets, or their combinations are retaining the most.
2) You can only study one metric by the time and not the total retention + conversions each segment generate, which should be the deciding factor
There is no initial interest, nor predictions either. So when doing a new campaign you cannot take any insights from cohort analysis until you wait for enough return visitor data.
We certainly agree that Google Analytics is an awesome tool and it has served our sites really well in the past. It gives tons of general insights for a free service, but isn't designed specifically for conversion optimization. Hence Volument.
21 comments
[ 2.9 ms ] story [ 55.2 ms ] threadI emailed them and they replied very soon and fixed the problem within a day, though, so at least response time was excellent.
I can sympathize with the challenges of running a web scale analytics tool, after helping build and run an analytics stack for an ad network. Even with our relatively small group of customers and lots of planning, the data was abundant, the queries became quite slow, and the pipeline was never "real-time" enough for some.
The web is a noisy place! URLs parameters are tricky to canonicalize into pages when dealing with many properties. There is a constant stream of nonsensical referrers and even referrer spam. Traffic can be exceptionally spiky, and cascading failure in the pipeline is inevitable without careful planning. Data loss is irrecoverable, and any incident needs to be carefully explained to users of the product so they don't deceive themselves into making decisions based on missing data.
One specific point I'm concerned about from the post is that while comparing HackerNews traffic to organic search, their analysis fails to account for relative traffic volume. Its always easy for a small volume traffic source to show outstanding metrics, but miss the broader point: quality traffic sources usually don't scale. A useful analysis should assign weight to overall volume. ORDER BY MAX(CTR) will tell me that some keyword/referrer was the top performer at 80%, but without knowing that was for 3 visitors I could easily get the wrong ideas. I think analytics tools aspire to make it difficult to come to incorrect conclusions, acknowledging that most of their users will have no meaningful background in statistics.
All of those challenges are surmountable, but it definitely gives some credence to the talented engineers who have make analytics stacks look easy. I wish Volument luck in their journey and will keep an eye on their progress, because their pitch remains quite appealing.
Thank you for the solid feedback!
Does it show that people coming from HN want to glance at your startup and see what it does, whereas people coming from Google have no idea what to expect and take longer to work out who you are and what you do?
Does it show that neither Google users nor HN users know what you do, but HN users are more experienced with websites and spend less time mentally categorising them?
Does it show that HN users are less interested in marketing / conversion than people who Googled for a related term enough to find your site?
What's the super-valuable insight and how will you use it?
The super-valuable insight is to know how the various segments proceed on the AIDA funnel (attention, interest, desire action) so you understand how your visitors convert.
It also reveals your best target markets and landing pages that generate the most traction. This data is not available elsewhere.
Let's not jump the guns too soon. Analytic products are notoriously hard from scaling perspective. UI, APIs, Infra that works crisply for 1x data, starts to become sluggish at 10x and really breaks down at 100x.
Startups solve the 100x scale problem when they get there (sensibly so). In this case, they got there way too quickly thanks to HN traffic.
I'm slightly irked by this use of "depression" anyway. It somewhat trivialises one of the top 10 deadliest diseases, whose sufferers are already facing the constant pain of their disease being considered "fake" or a lack of character or laziness etc.
Sorry for being the "language police". It's just a minor point, and I know the word is commonly used in this way. But even ignoring the issue above, I think there are simply better words for the specific situation: "depression" evokes the spectre of inactivity, of helpless, and of despair. But you did tackle the problems that came up, so words such as "challenging" (yeah, I know), "frustrating", "stress", "maelstrom", or "adventure" may not only improve the text's language, but also project a more positive image for potential customer.
You never know. Are you ready for success ?
This story looks like that.
You can trigger events with JavaScript and choose whether they affect bounce rate, so you can manually trigger an event when the user scrolls down, clicks something, spends a few minutes on the site, etc. So your problems with the bounce rate metrics are just not true.
Also I saw you mention it doesn't track returning users, that's not true either. I can actually see insights into how many users return month after month, and it'll tell me what my activity is per channel and if one channel outperformed others, so it's pretty powerful in this regard for a free platform.
I skimmed it but from just reading those it sounds like you should try to get better familiarity with Google Analytics and talk to users to see what their main pain points with existing platforms are, as well as establish more concrete value propositions. Google Analytics is free after all, I'm not convinced yet to move to a paid platform that tries to address problems I'm not experiencing.
GA can do retention reports or "cohort analysis" and show how visitors return in a monthly basis, but there are two critical limitations on it:
1) It doesn't show you which landing pages, target markets, or their combinations are retaining the most.
2) You can only study one metric by the time and not the total retention + conversions each segment generate, which should be the deciding factor
There is no initial interest, nor predictions either. So when doing a new campaign you cannot take any insights from cohort analysis until you wait for enough return visitor data.
We certainly agree that Google Analytics is an awesome tool and it has served our sites really well in the past. It gives tons of general insights for a free service, but isn't designed specifically for conversion optimization. Hence Volument.