Ask HN: How to price a service that saves hundreds of engineering hours?
The only catch is that individual tests have to be under 30 seconds.
The idea came out of frustration running Cypress in my previous company, where it was taking 10 minutes+ to run all Cypress tests while parallelizing across 30 VMs and costing us in excess of USD 2k/month.
In order to achieve this, I have effectively built a new Cypress test runner from the ground up. It understands Cypress syntax, but otherwise have nothing in common with how Cypress works. The way it achieves this performance is by splitting each spec into individual tests and starting all of the tests at once, i.e. if you have 10 specs with 5 tests each, this program will start 50 VMs to run your tests.
I have two companies trialing this at the moment and the feedback has been incredibly positive, saying that it is saving _hundreds_ of engineering hours. I am trying to establish how to price this. The challenge is that this model is profitable at scale, but losing money if there is not high density of clients. This is because it costs me USD ~0.15 to run a VM for 1 hour and I need to spin up enough VMs to complete all tests, and I am charged in increments of an hour.
My thinking is to charge 2 cents per a test-minute, i.e. Using previous example of 10 specs with 5 tests each, it would cost USD 1 to run all tests once. If you run integration tests 100 times per day, that's USD 100/day.
This may sound much. However, if prior to using this you were waiting 15 minutes to run all tests, that is 14 minutes saved. If avg. engineer in your company is earning USD 60/hour, you are saving USD 14 by having engineers get immediate results rather than waiting for them. If positioned that way, it doesn't sound expensive.
I am currently targeting companies with 30-50 engineers (existing customers are series A and series B companies). At this size, they don't have crazy amount of tests, so I can deliver on the 1 minute promise, and they care a lot about moving fast.
What sounds reasonable?
88 comments
[ 5.7 ms ] story [ 209 ms ] thread100/day is not reasonable, at least not for small/medium teams. Trying to do arithmetic on run times and hourly rates to convince people it is reasonable is not only overly simplistic, but actively distracts from the conversation you really want to have - how does your solution solve their problem? People buy when they believe you can solve a problem better than they could without you. So keep the conversation in that space and don't set up pricing that invites nitpicky arguments about hourly costs, etc.
Instead, I'd just price based on your own needs - figure out your actual operating cost per test-minute, tack on a reasonable profit margin and operating costs, and bundle up some packages that have a quota of test-minutes per month in sizes ranging from 20/month to 100/month. Make sure that you are profitable at every package, and see if you get takers at that price point. Then let prices increase naturally when they run out of minutes and need to ask for more.
If I'm understanding OP correctly then at that level this tool would be saving the customer's engineers 14 minutes of disruptive downtime 100 times per day. If that's correct then $100/day may indeed be unreasonable, but because it's way too low.
I doubt the useful target market for a tool like this is in the $20/month or even $100/month category. What does a series A/B company that runs 100 slow rounds of integration testing per day spend on its cloud hosting every month? That's obviously going to be a much bigger number than they'd pay for a service like this alone but it will give you some idea of the magnitude of numbers that are reasonable to discuss. How much does running those tests slowly cost them not only in direct developer salary but also in lost productivity because of the slow feedback, reduced developer satisfaction leading to people being demotivated or even leaving if it gets too frustrating, etc? What is the extra risk to them of not having new projects ready fast enough to add serious value before the next funding round?
Running 100x test suites would be $0.90 to $4, in server costs.
Charging 25-100x the server costs is optimistic, at best.
That being said, I am constantly experimenting how to reduce costs. I think the final result is going to be a fairly complicated orchestration of pre-booted machines that are rented by month and those that we spin up ad-hoc to handle demand fluctuation.
That would be $0.05 per core for an hour… or $0.05 for 60 copies for a minute. (Price for m5.metal on demand; a1.metal is half the price; m6a.metal is around $0.04)
I think the idea is cool — and wish you the best of luck.
I’m just trying to share why as a senior SDE your product doesn’t make sense to me: 100 integration tests a day is a fairly substantial team… and you’re charging a lot for it, at $2-3k per month.
If that's so, you can take sever for around 60$ month there and put some load there - I doubt it gonna dip your pocket for such test.
In the case of the two companies that are helping me to test this service, they both pay a fixed fee of USD 2000 for an agreed maximum parallelization of 50 browsers. This means that up to 50 concurrent tests, I can still deliver the promise of 1 minute tests.
For context, BrowserStack, which they used before, is charging USD 999 for 5 parallel browsers. I believe BrowserStack has this pricing model because they assume long running tests. It is a lot harder to share VMs if tests take 10-15 minutes than if they take 1 minute.
I could have pricing as USD 999 for 50 browser parallelization, and then charge extra if either of their tests take longer than 1 minute to execute. This model would ensure that we don't have one client hogging all of the resources.
Indeed. There is a qualitative difference between tests completing fast enough that a dev is OK with just waiting a few seconds for the results and tests becoming a "go do something else for a while" background activity. There is also a qualitative difference between tests running fast enough that if something does go wrong the code can often be backed out before it interferes with anyone else and tests that run slowly enough that the CI system can't keep up with merges from the team and changes start piling up that then need sorting out somehow every time a set of tests does fail.
If all integration tests could complete in pre-push commit hook (which is reasonable if they take less than a minute), then we would dramatically reduce the context switching involved in day to day engineering.
If I were in your position I would absolutely be emphasizing these kinds of qualitative benefits in the marketing even if your pricing model is actually very quantitative in the end.
Anyone who runs the kind of engineering group you're currently aiming at won't have to think very hard to get an idea of the quantitative scale of the benefits they might enjoy as a result. Ideally you want authorising a few (or more than a few) thousand bucks a month to be a quick and obvious decision for them. "We provide a simple tool that enables radically better development processes with likely ROI between 10x and 100x" is a decent elevator pitch to people like VP Engineering and CFO who are going to end up making a purchasing decision.
If you know your market, you know your price. It’s very easy to A/B SaaS pricing. Does the proposed pricing sounds too high? To me, yes, but go out there and test it. Worse thing that can happen is nobody buys it.
So I actually do think this would benefit that, as these tests are run "independantly", however that's just going to hide the problem of your bad/flaky tests, and you'll never solve the underlying problem.
The thing that I built takes a different approach. It parses Cypress tests, interprets the commands that Cypress wants to run, and then uses Chrome DevTool protocol (CDP) to evaluate commands such as navigation, click, etc. If you are familiar with the JavaScript ecosystem, my tool effectively transforms tests to Playwright compatible tests and then uses custom orchestration logic to run them.
That’s why many recommend unit tests or pseudo E2E tests that save a copy of an API response. Unfortunately this advice is rarely followed.
You really have to work out your ideal customer, though. And find a way to scale it down - you can potentially use the capacity of more VMs spread across many smaller companies.
Maybe your business model is having a few clients pay you a couple grand a month, maybe it is having a couple hundred paying you 20-50 or so a month.
You do seem to have solved a problem, which is a great first step.
That said, pricing correctly is hard and requires a lot of experimentation. Growth and adoption might be more important at this stage.
Is this a test suite I run during my development cycle? If so, I get distracted if it takes more than half a minute, open HN, and lose a bunch of time.
Is this a test suite that runs once I open a pull request? It can run an hour or two, I don't mind. I'll move on to the next thing and check tomorrow morning if it's passed. If it failed I start my day by investigating why. I care very little about speed here.
I think having something like this might have actually kept me at that job, at least a bit longer, though we had the same problem with our unit tests, and I don't even recall if Cypress was what we used (probably not, now that I think about it).
maybe give them knobs on the amount of maximum concurrent tasks?
I do have some very strong advice on how not to charge: (1) do not charge based on your costs (customers won't care), and (2) do not charge based on fine-grained usage. The latter is important because if a VPE signs a contract for $3k/mo, he/she doesn't want to sit there worrying that devs run a couple extra tests or add a few tests, etc, and boom: all of a sudden the VPE is going to the cfo to explain $10k in overage charges.
So: price in broad tiers, and price based on value. If you save hundreds of eng hours/mo, charge for the first 20 hours saved. At $150 per, that's $3k/mo. Multiply by 5 to have a functional company (you can't do sales for under $10k; every successful sale -- which will be 20% if you're very good and lucky -- must pay for all failed sales, all marketing, all dev, etc. Oh, and you're doing sales.) So $15k/mo. That's within the approval authority for a vpe with 50 reports.
And accept that bigger companies don't worry about $2k-$3k/mo. So maybe your target should also be smaller shops that can't accept the infra overhead to build this out, but will buy a cheap solution that saves their 1-4 devs a ton of time. Dunno.
edit: and you need a strong cross-tenant security story.
Counterpoint: I have personally seen companies of the size OP is talking about run into unexpected big jumps in spending on key infrastructure and tooling because they made some architectural change and then suddenly realised it was going to run the tests 10x as often or something. Without exception the response has been twofold. First they did take the immediate financial hit because they couldn't afford to be without a critical system. Second they diverted significant resources to getting the usage levels back under control very quickly so they didn't have to rework that critical infrastructure as well.
If anything, since bringing the costs of some important infrastructure down is exactly the problem OP's tool is solving and its value is directly demonstrable in terms of time and therefore money saved, your general argument should be a positive sign for OP that they probably have a viable business model.
I could buy an argument that tiered pricing might work better for OP than metered if someone showed up with data supporting the theory. But metered pricing seems to be working out pretty well for a lot of popular cloud-hosted dev tools and of course is almost universal among cloud infrastructure providers. People complain a lot about AWS having unpredictable pricing but I heard AWS has still had some success breaking into the cloud hosting market and probably it's making enough money that Amazon might keep it running. To be fair that's partly because although AWS lacks safeguards that would certainly be welcomed by many it does also have a reputation for being very decent about support in the occasional "Oh s@£t" scenarios.
Flat rate or bucketed pricing is common across many dev tools. eg ci/cd: see bamboo, gitlab ci/cd (purchased in blocks), circleci (credits w/ account, more credits sold in blocks). apms like honeycomb.io. etc.
"Sorry I can't price it below X". See how they respond.
If I simply transferred current costs on to the end user, it would be too much. It would be USD 0.20 x number of tests x number of hours, e.g. a company that works 9 to 5 and has 100 tests, that is 0.201008=USD 160 per day.
However, changes the more clients you have. If I have 10 clients paying for 100 concurrent test capacity, I wouldn't need 1000 VMs, as they are unlikely to be all running tests at the same time, all the time.
Aren’t you able to (fairy quickly) autoscale up/down based on the size of some queue of test requests?
As I have only two clients at the moment who are helping to test this, keeping a hot pool allows me to focus on other feature development while sacrificing some of the costs.
Keeping a hot pool makes it a lot easier to immediately start test runs. Run tests, restart browser, clear sessions, return to the pool.
You could even have tiered services. For example, some users may be willing to pay more for more consecutive tests.
Charge fairly. If you get more customers and improve efficiency again, be honest, "thanks to some improvements in scalability I'm able to reduce your costs to X".
You'll also build a long-term relationship by being fair and taking advantage of your customers.
At the high end, some enterprise companies won’t blink at the prices you’re suggesting. However, they will expect a level of customer support and service agreements that are going to make you work hard to deliver what they ask. You’re also going to need to be heavily involved in sales cycles and support, which will start costing you many, many hours just to get the sale.
Are you prepared to handle this and stay responsive? If not, you will have mixed results pushing high priced offerings, especially if they have deal-breaker limitations (like your 30 second test limit) that could saddle the customer with even more work to integrate and use your tool. You have to acknowledge that your service isn’t purely savings for the customer. It’s work to implement and maintain on their end, as well as handle billing and so on. You cannot make sweeping calculations about savings without ignoring the costs it brings to a company to add your tool.
On the other end of the spectrum, you could target the self-serve market with lower priced services. These offerings can get away with a “take it or leave it” type of service where customers are mostly on their own to evaluate, pay for, and self-support the software. If the tool is good, engineers will try to make it work. However, I suspect you’re massively overestimating the way that test times are a blocker for most engineer workflows. Even with long text suites, we don’t just sit and stare at the monitor while we wait. That’s time for responding to communications, checking other things, taking a short break and so on. You could have a hard time selling engineers on the idea that they can do more work and take fewer breaks while paying for the privilege. Choose your marketing angle wisely.
Speaking from experience, my productivity is down by like 80% if I know that there is a test I am waiting to complete. I can do other things, but I will constantly switch back and forth to check on the progress. I know from talking with other engineers that they feel similar, esp. when flakky tests are involved.
However, I get what you are staying. I will think how to position this in a more compelling way that cannot be easily challenged.
What is your customer base and target market? Both pricing models and numbers will come out very differently for the same underlying technical work depending on if you're aiming for fewer how-much-could-a-banana-cost-biggies (these need service agreements and someone who they can talk to on a first-name basis) or a turnkey system for every startup and corporate using Cypress out there (they just register a credit card and e-mail and start integrating).
While you are considering your margins, you're going by value-based pricing here - you focus less on how much it costs you to build and run and your bottom line and more on how valuable it is to the potential customer. Some told you hundreds of hours were saved.
It sounds like you have the pricing model for a mass-market SaaS with the pricing for the high-end.
One risk with such high value-based pricing is that assuming initial success, some happy customers might get restless after a few weeks/months/years and realize that for the price they're paying you they could actually save money by building and rolling something similar in-house. The customers who are ready to pay those prices are precisely the ones likely to do it, I think. In any case, unless your market is super niche or you have some crazy edge, you will get competitors if you're successful enough. Maybe even straight-up copycats. This is nothing bad at all but it can be good to keep in mind that you may lose customers if they're able to slaughter you on cost-performance.
Also consider that these higher-end clients likely care a lot about availability and stability. If you don't have brand-recognition or a personal reference, it can be hard to convince them to take a shot. Free trials will never be enough for them to be confident either. Initial underpricing (not too much obv) is a way to compensate for that initially.
But again, I think we need more context before we can make a fair assessment of if the price is reasonable or not.
Market Pricing: what are your competitors charging.
Cost plus: What is your investment to date and what do you see ongoing marginal costs as, and what type of pricing do you need to make this worthwhile for you financially?
nearest substitute pricing: what’s the financial savings of your product versus the competitor? Charge a price that captures some of this value (what you’re proposing)
Also be mindful of Cypress possible taking steps to improve their own run times. What will you do to stay ahead and keep your marketshare? include this in your considerations as ongoing r&d.
(A month. Bill quarterly or yearly at their option; raise prices annually to capture increase in use.)
Remember that their alternative is not “spin up VPSes” it is “staff an engineering rotation to maintain this service” which will cost them minimally a million annually.
We already agreed to discuss the idea of doubling the capacity.
But the default alternative is "continue making limited use of these integration tests and continue accepting the delays of running them slowly" which has a cost of $0 compared to status quo.
What proportion of series A companies that are trying to scale up and show results quickly so they are still around to raise a series B is going to spend $1M+ on a dedicated team to run a service for this? I'm guessing 0.000%. That means the comparison isn't a very compelling argument for the leaders in that organisation because really you're just arguing that your solution is better than something they obviously weren't going to do anyway.
> My thinking is to charge 2 cents per a test-minute
Doesn't that reduce to 2c/run?
That aside, my gut reaction was don't charge for time if your selling point is how much time you save.
Charge for #tests, or something even better targetted at what makes it slow usually (I'm not familiar) - you charge some (<100) percentage of the reason tests are expensive and make that go away, and it seems like a sound choice to use your service doesn't it?
However, based on everything I've read in this thread, I am leaning towards max parallelization based model, e.g. maximum 50, maximum 100, etc, and charge extra for tests that take longer than 1 minute.
This allows straightforward sales while providing most of the value to an avg. client.
(But also now there are 'tests that take longer than 1 minute'? It's gone from wow to not really clear what you're offering beyond managed test-running (that's otherwise equivalent to self-hosted) to be honest.)
Edit: do you mean they're parallelised, so they all complete in however long the longest one takes (which in OP is <1m but in comments might be more) and you'll charge for #concurrent jobs?
Using Cypress you are going to have specs and specs will have tests.
You can parallelize Cypress per spec. Meaning that if you had 50 specs and 50 VMs, then your entire test suite will take at most what your longest spec takes.
Most companies are not going to have 50 VMs ready to run tests. However, even if you did, you still have the problem that your tests will run for as long as it takes to run the longest spec, which can be a lot.
I've seen companies try to diminish the latter issue by splitting specs into multiple files. However, that will get you only so far.
So what I do is... I read all your specs, extract individual tests, and spin up VM for each test.
If all your individual tests complete in under a minute, then you will wait at most 1 minute.
However, if there are individual tests that take longer, than yes – you will wait for the longest test to complete.
What are you replacing? What else could they pay for to achieve an equivalent speed up?
What are your enabling? How much time are you freeing up that can be used for other purposes?
What is the impact on time to market / release inter-arrival time?
What is the shadow backlog of tests that are not being run but now could be?
What is the impact on quality: for example time to fix (MTTR) or production problems prevented because of increased testing?
You are welcome to schedule a no cost office hours session to walk around your situation: https://www.skmurphy.com/blog/2016/08/01/skmurphy-office-hou...
You are thinking too much. Just tell them $5K/month. If they balk re-think your product.
> existing customers are series A and series B companies
With 2 VC funded customers, try getting a referral for cloud credits. If you cant get $10K credits try to find a business-side co-founder.
Finally, would you be interested in some sort of experimental "guaranteed minimum activity" swap contract? No cost to you, but you may need to provide analytics. Email my puppet email which i dont check often enough
This is going to be a lot of trial and error and no amount of analyzing is going to give you a good first answer. So start with something you find reasonable (the pricing you mentioned) and see how it goes with your current clients.
If they are totally ok with that then you’re priced too low and on the next sales cycle change it up. If they all balk and threaten to walk away then adjust your price down, until they begrudgingly pay.
Since this is essentially a cost-saving thing, I don't think it's realistic to try to charge the cost saved. If I as a customer can save $100/day by using your software, but your software costs $100/day, why bother?
You are not only saving engineer time. By delivering more in a shorter amount of time, you have compounding effects across the entire organization.
Eg, If I sell an animal feed supplement that increases cow weight by 1%, it means customers can increase output by 1%. If the same supplement can make racehorses 1% faster, that's something entirely different.