HN Explorer

Ask HN: Review my new A/B testing and analytics startup

35 points by oni0n56 ↗ HN
Hey everyone, I've spent the last year designing and developing a tool for running A/B tests and keeping tabs on site traffic. It's just launched and live at http://getcharmed.com/

I’d love to get your feedback and help improving it and making it as awesome as possible.

Some of the things I’m most excited about:

• Sharing. I’m hoping it does for analytics/testing what github did with code. Here is what the permissions look like for one of my sites http://cl.ly/0I1N3P0f3g451b3v0W2L Let me know if you want me to add you to it as well!

• Goals. It has the basic “conversions” like visiting a page, clicking link. but also magnitude-based measures like time, pages viewed, scroll depth, which can all be tested at the same time to get a real accurate picture of how changes to the site affect user behavior.

• The traffic panel. Instead of having every possible measurement under the sun, my goal here was to have a very easily digestible representation of what is happening on the site. This means things like using the favicon and page thumbnails for easy recognition, parsing the URLs easy readability, and having a visual representation of the weight of all the numbers. Here is what it looks like one of my sites http://cl.ly/1D1f0J3N2T0A2X0l111V

This is just the tip of the iceberg, there are a lot of features planned or even just commented out for launch, but I really want to nail the basics first and make sure they are as useful and practical as possible.

36 comments

[ 0.24 ms ] story [ 113 ms ] thread
klochner
You're wasting space on the homepage in explaining the benefits of a/b testing, you would be better off devoting more content to how your product works.

I want to know what's involved in setting up and running a test, but came away from your site without really knowing what the user experience is like.

bravura
I assume he's actually A/B testing whether explaining about A/B testing helps his conversion. If not, the irony is killing me.
[deleted]
(comment deleted)
jeffmould
Looks great! Congrats on your launch. A few things that stick out for me:

1. I like the free option, and would most likely start there to get a feel for everything. But...if I wanted to start with a premium plan instead, there is not enough information on the website to tell me how this product is different (besides look/feel) from products such as Google Optimizer, Google Analytics, or Optimizely. Sell me on the benefits of using your product over these other products (in particular Google since it is free for all levels).

2. This is a little pet peeve on my part so you'll have to excuse me, but fix the transparency on the signup button background. Right now it stands out and looks thrown together compared to the rest of the site.

Overall though, the layout of your site looks nice and the screenshots you provided look great as well. Looks very user friendly. Can I suggest maybe creating a video for the website that walks people through the system or even adding more screen shots?

Good job and look forward to hearing/seeing more.

oni0n56
Thanks! I think I just fixed #2, I am guessing it was a Chrome/Win bug with rounded corners. Though I can't see it myself.. is the problem still there?
jeffmould
Yep, all fixed now. Looks good I'll give it a try when I get my site launched and let you know.
gsiener
This looks great. Just signed up, I'm very interested in trying it out. Shoot me an email, I'm happy to talk through your roadmap and give feedback.
animeshjain
why three domains - getcharmed.com, charminc.com and usecharm.com. seems a little unusual and might even be confusing.

edit: looks pretty neat on the UI

javery
The first thing I thought of when I got to the site is that this looks like a bought template. I don't know if it is or not, but that's not what is really important. It feels that way because it looks like a case of the "lorem ipsum" curse - when you have to fill out an already designed page and just had to keep adding stuff. The Clarke quote adds to that impression, if you don't have a quote about your product does that mean you don't have any beta users? Or were none of them willing to be quoted? Then on your signup page you have Voldemort and Dumbedore providing quotes.

Also, your twitter feed isn't very re-assuring on the infrastructure side.

Overall it looks like an interesting product though and something I might consider using.

iampims
The site looks great. When I tried to add a site, the link for the js to be included was missing the scheme (http|https):

    <script src="//usecharm.com/xx.js" type="text/javascript"></script>
If we could have a restricted live demo, or a video to get a feel of how things work before needing to get some fair amount of data, that’d be even better.

Great job and good luck with it.

edit: apparently // is the way to go, see comments below.

aonic
Not including the http|https tells the browser to use the current page to figure out if the file should be loaded over http or https. Loading http content if you're on a https page will trigger non-secure content warnings with browsers
iampims
Thanks for the tip. I assumed we’d have to manually add the scheme to the url.
bretthopper
// is a protocol relative URL so it will work for both http and https
ry0ohki
wow how long has this worked? I learn something new here every day.
aonic
Must have been around for a while as it works in IE6 as well
oni0n56
Yeah, the reason for that is that 1) if/when we add HTTPS support it will run seamlessly. For now it just returns a blank file if included over HTTPS, and so the page will degrade gracefully (no tracking) without any security errors that would've come up from loading an http resource. 2) it's shorter.

Though you're right it does look a bit weird, I will add a note to the instructions explaining what it is and that its not a bug or anything. Replacing it with http:// would've worked perfectly fine as well.

We use the same scheme to link to the media on our CDN so the app can switch seamlessly between http and https secured sections (account,billing) without any different <head> code

mmelin
I like how your support options progress from "Online" to "CEO's iPhone" :)

I'd like to read more about how it works, maybe see an integration guide, but it seems you only have the one (beautiful) page right now?

oni0n56
Thanks. Yes it is only one page right now. I had originally planned a whole site with different sections and pages but it seemed a bit wasteful since the current integration guide is: • have jquery on your site • add a line of code linking to your JS file

But behind the scenes how it works is that JS file (ie. usecharm.com/mmelin.js or something) returns different code depending on the cookie of the visitor and the configuration of the project. So if you launch a test, the goals and changes will come out in that file and it is also based on what page it is requested from, so the change could be different when requested from the homepage of your site vs a content page. By default it just has basic analytics stuff like referrer-detection and measuring demographics.

Do you think I should add a "behind the scenes" or similar section that explains all the technical stuff going on?

dflock
Yes. Currently it looks great, but nowhere does it actually explain what it does - in specific technical terms.
thenduks
Looks very nice, design-wise -- unfortunately I can't actually try it out because I'm not willing to put your javascript directly on my production site without experimenting first. I get that your app needs access to do it's thing, so I'm not sure how you'd fix this... but it's a real stopper for me.
oni0n56
Well you could install it on a development site or something first. You can't currently add in a domain that's not live (since it verifies its existence by pinging)— but contact me and I can add it in manually and you could try running it locally or on a private site.
thenduks
Yea the ping-ing thing is what stopped me. I need to test it as un-live as it gets, on my local development instance, before I start pitching this stuff to my boss.

I'll shoot you an email. Thanks. edit: Or, I would, but you don't seem to have it listed in your profile.

oni0n56
Try support@charminc.com — Will update my profile now.
tlack
I think this is a great business to be in right now.

As others have said, I'd include screenshots and details about the optimization process works, not the monitoring bits and referral tracking - after all, most people already have a very involved process they use to do that.

Design wise it looks good, but the signup button transparency is distracting.

aonic
Love the user interface screenshots. What did you use for the graphs? Custom library?
oni0n56
It's pretty much all CSS-based at this point. Javascript is used for tiny things like the centering of the confidence intervals.

We were looking into some of the canvas libraries out there but that really only becomes necessary when doing pie-charts and other non-rectangular shapes.

SupremumLimit
I like the landing page and the screenshots look nice, but I would appreciate an explanation of how A/B testing works from the technical standpoint. How do I set up a test? What if multiple pages are involved? Will returning users get the same page consistently?

It's definitely an interesting service though, and I'll be happy to pay for it if it saves me time compared to setting up my own A/B testing framework.

petewailes
I'll deploy next week on a site and let you know my thoughts
parasctr
jQuery requirement doesn't suit me because I am currently using YUI. Are you planning on making YUI version available? If not it would be nice to have a version with no framework dependencies.
kqueue
off topic, I like the design. Can I get in touch with the designer?
dflock
Okay, signed up and went a bit deeper into the site:

* You've got the same rounded-corners-not-transparent issue at the bottom of every page

* Only when you get to the 'copy this bit of JS into your site' stage, do you find out that it _also_ requires jQuery. I know a lot of people will already be using jQuery (including me) bit the site I wanted to test with isn't using jQuery, so this has moved me from 'Go', to 'Stop and think'. I wish I'd known this earlier on.