Launch HN: Dashdive (YC W23) – Track your cloud costs precisely
We talked to dozens of software engineers and kept hearing about three problems caused by poor cloud cost observability:
(1) Cost anomalies are slow to detect and hard to diagnose. For example, a computer vision company noticed their AWS costs spiking one month. Costs accrued until they identified the culprit: one of their customers had put up a life-size cutout of John Wayne, and they were running non-stop facial recognition on it.
(2) No cost accountability in big orgs. For example, a public tech company’s top priority last year was to increase gross margin. But they had no way to identify the highest cost managers/products or measure improvement despite tagging efforts.
(3) Uncertain and variable per-customer gross margins. For example, a SaaS startup had one customer generating >50% of its revenue. That customer’s usage of certain features had recently 1,000x’ed, and they weren’t sure the contract was still profitable.
(If you’ve had an experience like this, we’d love to hear about it in the comments.)
We built Dashdive because none of the existing cloud cost dashboard products solves all three of these problems, which often requires sub-resource cost attribution.
Existing tools combine AWS, GCP, Datadog, Snowflake, etc. cost data in a single dashboard with additional features like alerting and cost cutting recommendations. This is sufficient in many cases, but it falls short when a company (a) wants per-customer, per-team or per-feature cost visibility and (b) has a multitenant architecture.
By contrast, Dashdive uses observability tools to collect granular cloud usage data at the level of individual user actions (e.g. each API call or database transaction). We attribute this activity to the corresponding feature, the responsible customer and team and estimate its cost based on the applicable rate. The result is more detailed cost and usage data than can be obtained with tagging. This information can be used to detect anomalies in real-time and identify costly teams, features and customers. One of our customers is even using Dashdive to charge customers for their cloud usage.
We use Kafka to ingest large volumes (>100m/day) of product usage events, and our web dashboard supports real-time querying thanks to ClickHouse. This makes it fast and easy to answer questions like: “Over the past 14 days, how much vCPU time did customer X use on Kubernetes cluster A, and how much did that cost me?” You can answer such questions even when the same container or pod is shared by multiple customers, features and/or teams.
You can test drive the product with example data here: https://demo.dashdive.com/. Given the high per-customer cost of our infrastructure and the manual steps required for setup on our part, we don’t offer self-serve onboarding or a public “free tier” to monitor your own cloud usage, but this demo gives a basic view of our product.
Right now, Dashdive supports S3 and S3-compatible object storage providers. We’re working to add support for other providers and services, particularly compute services (EC2, GCP VMs, ECS, EKS, GKE, etc.).
If there’s any service in particular you want to see supported, please tell us in the comments. We’re eager to see your comments, questions, concerns, etc.
55 comments
[ 3.0 ms ] story [ 125 ms ] threadFeature request: I have really struggled with turning the thing costing me money off in AWS.
If, with the right master credentials, I could consistently and easily do that somehow, that'd be a 10x feature. If you made that use-case free, you'd get tons of installations from people who desperately need this in the top of your sales funnel.
edit: This used to say "in your app" and that wasn't quite what I want, so I changed that language, but jedberg's objections in the comments below were, and are, valid concerns with what I was stating and any implementation.
But a good stopgap would be a feature to spit out an API command that someone could run (or a CloudFormation or TF file) where you can put your own credentials in and run it yourself.
The best way to do it would be to run the delete with no access, see what permission errors you get, and then only give those permissions until you've successfully deleted the object.
The safest way (but obviously more work) to do one off work like this is start permissionless and slowly open up. There are tools that can help with this, extracting the permission errors and generating the files to update the permissions.
1 - https://github.com/turbot/flowpipe 2 - https://github.com/turbot/flowpipe-mod-aws 3 - https://github.com/turbot/steampipe 4 - https://github.com/turbot/steampipe-mod-aws-thrifty
https://resoto.com/defrag https://github.com/someengineering/resoto
The magic behind the clean up is Resoto's inventory graph - the graph captures the clean up steps for each individual AWS resource.
One of Resoto's users, D2iQ (now part of Nutanix), reduced their monthly cloud bill by ~78%, decreased from $561K to $122K per month. There's a step-by-step tutorial on our blog how they did it.
I don't mean to hijack Dashdive's thunder here though, congrats on the launch!
For example, let's say you've figured out that a particular EC2 instance or database is too costly. What is the sticking point for you in turning it off? Is it that the resource has other essential functions unrelated to the cost spike? Or is it the identification of the exact resource that's the problem?
I used Vantage, which helped me see the problem, but then taking action on it was traumatic.
The barriers are:
- who owns it?
- what service is it in (if it is logs, for example)?
- where is the screen it is on?
- how do I get the permissions to kill it?
I agree that this would be a great feature. To be honest, our product isn't currently focused on this sort of automatic management of resource lifecycle; we're much better at data collection. But thank you for flagging this! We'll definitely keep it in mind as we add support for compute services (right now we only support S3 and there's nothing to "spin down").
Edit: The part less related to permissions (how can I kill it) and more related to discoverability (which resource is it) is more adjacent to what we've already built and is something we can take a look at soon. Perhaps we can take a crack at the permissions aspect afterwards.
Managing cost without impacting any infrastructure is the right way to do it in my opinion.
The nice part here is that when you see an instance burning $3000 a month, you can click on it and see cost break down based on traffic, compute, etc...
Also what about utilisation rates ?
The same goes for on prem. We don't have any on prem customers currently, but it would be easy to add a feature where you input the total capacity and/or monthly cost of your on prem infrastructure, and use the collected usage data to calculate utilization rate and "effective cost" incurred by each feature, customer, etc. Thanks for the questions!
Vantage is in the category mentioned above: it combines AWS, GCP, Datadog, Snowflake, etc. cost data in a single dashboard and supports tagging. For example, if I have a single tenant architecture where every customer has their own Postgres RDS instance, I can tag each RDS node with `customerId:XXX`. Then I can get cost broken down by customer ID in Vantage.
However, if my entire app (including every customer) uses the same large RDS instance, or if I'm using a DBaaS like Supabase, tools like Vantage, which rely on tagging at the resource level, cannot show a breakdown of usage or cost per customer. By contrast, we record each query to your monolith or "serverless" DB (SELECT/INSERT/UPDATE) along with the vCPU and memory consumed, tag the query with `customerId` and other relevant attributes, and calculate the cost incurred based on the billing rules of RDS or Supabase.
When we were originally researching to see if anyone had done something like Dashdive already (i.e., specifically applying observability tools / high volume event ingestion to cloud costs), I did manage to find a Yotascale video in which they mentioned defining custom usage events in Kubernetes for a specific customer. It seemed more like a custom feature than a generally available part of their product, but I could be mistaken.
There are quite a few cloud cost tools out there which use AWS's cost and usage reports (or GCP/Azure equivalents) as their sources of truth, and as a consequence their data is largely based on tagging cloud resources with attributes of interest (e.g. EC2 instance XXX has tags customerId:ABC, teamId:DEF, featureId:GHI). These include Ternary, Vantage, Yotascale, and others (CloudChipr, CloudZero, Archera, Cloudthread, Finout, Spot by NetApp, DoiT, Tailwarden, CAST AI, Densify, GorillaStack, Economize Cloud). Some of these offer AI-based automatic tagging as well.
But even so, if I - for example - have a lots of k8s replica Pods which serve most of my application traffic, I can't use any of these products to figure out which customers, API endpoints, code paths, features, etc. are costing me the most. At best I could tag the entire Deployment, or maybe even each Pod. But the problem is that every Pod is serving lots of endpoints, customers, and features. However, Dashdive can give you this info.
From a technical implementation standpoint, Dashdive is much closer to application performance monitoring (APM) products or usage based billing products than it is to most cloud cost dashboard products.
One more thing is kinesis can have duplicates while kafka is exactly once delivery.
To answer your question, though, no: in the Kafka connector, the frequency of inserts into ClickHouse is configurable relatively independent of the batch size, so you don't need massive scale for real-time CH inserts. To save you a couple hours, here's an example config for the connector:
You also might need to increase `message.max.bytes` on the broker/cluster side.If you're still deciding, I'd recommend Kafka over Kinesis because (1) it's open source so more options, e.g. self host or Confluent or AWS MSK and (2) it has a much bigger community, meaning better support, more StackOverflow answers, a plug-and-play CH Kafka connector, etc.
Would be very interested to see this working with GCP Cloud Run.
However, as a developer, the pricing seems a bit steep for me to give it a try. Any plans for more affordable options or perhaps a developer-friendly tier in the future?
It's on our roadmap in the next 1-2 months to eliminate these manual steps and make these last parts of our infra multitenant. At that point, we plan to release a cheaper tier for individual devs.
Are you thinking of expanding in that direction, or integrating with such products? I would look favorably at a product that combined pricing (them) and observability (you). You're solving basically the same problem; don't make me buy two products.
However, it's not always possible to infer one's own costs from UBP events. In UBP products, the user defines what constitutes a "usage event" (e.g. "customer generates a PDF") and so these events can be quite disconnected from underlying cloud usage. In other words, there's nothing that prevents some "someone generated a PDF" events from incurring large amounts of EC2 usage while other "someone generated a PDF" events incur very little EC2 usage, depending on the input parameters to the workload. And in most UBP scenarios, this difference in underlying cloud usage from PDF generation to PDF generation is not taken into account; often all UBP events of a given type are billed at the same rate. In fact, we've seen this exact issue in the wild: namely, a company implementing UBP but still being unsure about profit margin because certain UBP event types had high variance in cloud usage per-event.
One company is planning to use Dashdive's S3 storage data to charge their customers based on usage, so in some cases the data we collect can serve as a substitute for UBP.
I agree that it would be more convenient if we also offered user-defined UBP events. This way, we could be a single vendor for the folks that want both usage monitoring and usage-based billing, where the UBP events don't necessarily align super well with underlying cloud usage.
If that sort of understanding can be better communicated throughout a company, particularly to sales, then that's great news.
An advisor suggested we offer a sort of "pricing heuristics" deliverable to sales as part of our product (e.g. "if a customer needs feature foo, add $bar to the price"), and your story makes me think he's onto something.
For example, if I have a database costing $100/m, and 4 equal sized customers (by data volume, compute, requests), you could say that each is costing $25/m. However if I'm over provisioned by 50%, then an incremental user is only actually $12.50. To understand this difference requires understanding resource utilisation, which particularly for compute can be really hard. There's also a difference between desired headroom (for redundancy) and undesired headroom (due to inefficiencies), and these will be nuanced per company or service deployment.
This could also be taken in reverse – rather than the cost for an incremental user, the savings from removing a user. Given 2x $100/m databases with 2 customers, one accounting for 60% and one for 40%, losing the larger one would allow a 50% reduction on cost, but losing the smaller one would not.
What's your thinking here? Are you trying to communicate the incremental cost of a user? Are you trying to communicate the cost of a user against the current infrastructure, whatever that infrastructure might be?
At a minimum, every customer is assigned the cost resulting from the usage that is directly attributable to them. This "directly attributable" figure is obtained by "integrating" vCPU-seconds and RAM-seconds on a per-DB-query basis, per-API-invocation basis, or similar. For example, if Customer X used 5 vCPU seconds and 7 RAM-GiB-seconds due to queries over a period of 10 seconds on our RDS cluster with total capacity 1 vCPU and 2 GiB RAM, then they directly utilized 50% of vCPU capacity and [7 GiB-sec / (2 GiB * 10 sec)] = 35% of RAM capacity over that period.
The question remains of how to distribute the cost of the un-utilized capacity over that period amongst the customers, perhaps distributing some portion to special "desired headroom" and "undesired headroom" values. As you mentioned, the answer is subjective and can vary between Dashdive users (or even over time for the same Dashdive user, e.g. a user decides they can reduce their desired headroom from 30% to 20%). The only sensible approach in our opinion is to make this configurable for the user, with sane and explicit defaults.
Let's go through both your examples to illustrate. In example 1, each of the 4 equally sized customers would be assigned only 12.5% of the total cost of the cluster. The dashboard would show that, by default, 30% headroom is desired, so out of the remaining 50% capacity, 30% would be marked as desired headroom, and 20% would be marked as undesired headroom. The user can override the desired headroom percentage. Although in our opinion it is most correct for all headroom to be treated as a fixed cost in multitenant scenarios, we would also provide the option to distribute both/either headroom types amongst all customers, either proportionally or equally.
For example 2, our model is not sophisticated enough to capture the nuance that losing only the larger customer would allow cost reduction. Assuming both customers used both databases (let's say they're replicas or shards), and 0% headroom, we would simply assign 40% of costs to the smaller customer and 60% of costs to the larger one. This is subtle, but the missed nuance is only important if $100/m is the finest-grained resource you can get. Otherwise, if you lose the 40% customer, you can switch to a 2x $60/m DB, for example.
This is a very astute callout! It has come up a couple times as a point of concern from prospective customers. Would be keen to hear if this diverges from your expectations at all.
In a sentence, these tools display the same cost data available within AWS - where max granularity is per-database or per-EC2 instance - whereas, Dashdive can accurately attribute portions of usage on the same DB or instance to different features/customers.
Your product is something that most managers don’t even realize they are missing.
I’m kind of obsessed with activity-based costing as applied to software firms. Hit me up at my username at gmail if you want to nerd out about managerial accounting and/or if you might need some marketing material! I could write for days on the subject!