Launch HN: Opstrace (YC S19) – open-source Datadog
Seb here, with my co-founder Mat. We are building an open-source observability platform aimed at the end user. We assemble what we consider the best open source APIs and interfaces such as Prometheus and Grafana, but make them as easy to use and featureful as Datadog, with for example TLS and authentication by default. It's scalable (horizontally and vertically) and upgradable without a team of experts. Check it out here: http://opstrace.com/ & https://github.com/opstrace/opstrace
About us: I co-founded dotCloud which became Docker, and was also an early employee at Cloudflare where I built their monitoring system back when there was no Prometheus (I had to use OpenTSDB :-). I have since been told it's all been replaced with modern stuff—thankfully! Mat and I met at Mesosphere where, after building DC/OS, we led the teams that would eventually transition the company to Kubernetes.
In 2019, I was at RedHat and Mat was still at Mesosphere. A few months after IBM announced purchasing RedHat, Mat and I started brainstorming problems that we could solve in the infrastructure space. We started interviewing a lot of companies, always asking them the same questions: "How do you build and test your code? How do you deploy? What technologies do you use? How do you monitor your system? Logs? Outages?" A clear set of common problems emerged.
Companies that used external vendors—such as CloudWatch, Datadog, SignalFX—grew to a certain size where cost became unpredictable and wildly excessive. As a result (one of many downsides we would come to uncover) they monitored less (i.e. just error logs, no real metrics/logs in staging/dev and turning metrics off in prod to reduce cost).
Companies going the opposite route—choosing to build in-house with open source software—had different problems. Building their stack took time away from their product development, and resulted in poorly maintained, complicated messes. Those companies are usually tempted to go to SaaS but at their scale, the cost is often prohibitive.
It seemed crazy to us that we are still stuck in this world where we have to choose between these two paths. As infrastructure engineers, we take pride in building good software for other engineers. So we started Opstrace to fix it.
Opstrace started with a few core principles: (1) The customer should always own their data; Opstrace runs entirely in your cloud account and your data never leaves your network. (2) We don’t want to be a storage vendor—that is, we won’t bill customers by data volume because this creates the wrong incentives for us. (AWS and GCP are already pretty good at storage.) (3) Transparency and predictability of costs—you pay your cloud provider for the storage/network/compute for running Opstrace and can take advantage of any credits/discounts you negotiate with them. We are incentivized to help you understand exactly where you are spending money because you pay us for the value you get from our product with per-user pricing. (For more about costs, see our recent blog post here: https://opstrace.com/blog/pulling-cost-curtain-back). (4) It should be REAL Open Source with the Apache License, Version 2.0.
To get started, you install Opstrace into your AWS or GCP account with one command: `opstrace create`. This installs Opstrace in your account, creates a domain name and sets up authentication for you for free. Once logged in you can create tenants that each contain APIs for Prometheus, Fluentd/Loki and more. Each tenant has a Grafana instance you can use. A tenant can be used to logically separate domains, for example, things like prod, test, staging or teams. Whatever you prefer.
At the heart of Opstrace r...
119 comments
[ 2.8 ms ] story [ 184 ms ] threadWe will also have a managed version where we deploy and maintain it for the customer in a cloud account they provide us.
I like supporting open source projects, and while SSO is pretty useless to me, I always like custom domains.
Only point was of there was going to be a smaller plan for homelabs or the like that doesn’t have the raw amount of Traffic or features as the enterprise plans do.
Haha, the "ouch, what have I done again?" moment... :-).
One of the most interesting lessons from making mistakes is to appreciate the fact that we make mistakes even when we try so hard not to.
I (generally) like taking myself offline. Thats how I learned everything I know (by fucking up). But thats also why I like to have my mail/monitoring outside my own selfhosted infrastructure. Cause those two things are something that should be running 100% of the time. And if I dont feel like fixing whatever I broke right this moment, I like to feel like I dont have to.
Sometimes I just go to bed whenever I broke something and fix it tomorrow. If I have to flick my light switches by hand until then, so be it, I can live with that. But mail and monitoring cant wait that long :P
2. When opstrace is setup in AWS/GCP, what is the typical fixed cost?
(1) As it stands today, you can already use https://vector.dev/docs/reference/sinks/prometheus_remote_wr... to write metrics directly to our Prometheus API. You can also use https://vector.dev/docs/reference/sinks/loki/ to send your logs to our Loki API. Vector is very cool in our opinion and we’d love to see if there is more we can do with it. What are your thoughts?
(2) As for cost, our super early experiments (https://opstrace.com/blog/pulling-cost-curtain-back) indicate that ingesting 1M active series with 18-month retention is less than $30 per day. It is a very important topic and we've already spent quite a bit of time on exploring this. Our goal is to be super transparent (something you don’t get with SaaS vendors like Datadog) by adding a system cost tab in the UI. Clearly, the cost depends on the specific configuration and use case, i.e. on parameters such as load profile, redundancy, and retention. A credible general answer would come in the shape of some kind of formula, involving some of these parameters -- and empirically derived from real-world observations (testing, testing, testing!). For now, it's fair to say that we're in the observation phase -- from here, we'll certainly do many optimizations specifically towards reducing cost, and we'll also focus on providing good recommendations (because as we all know cost is just one dimension in a trade-off space). We're definitely excited about the idea of providing users useful, direct insight into the cost (say, daily cost) of their specific, current Opstrace setup (observation is key!). We've talked a lot about "total cost of ownership" (TCO) in the team.
Also, please don't forget about people (like me) who don't run on $MAJOR_CLOUD_PROVIDER. I'd be curious to try this e.g. on self-operated Docker w/ Minio.
Will keep an eye out, to see if optrace might be a fit for us.
https://victoriametrics.github.io/
But we're still implementing it - so we might still run into surprises.
My two points - right now docs are clearly targeting users familiar with the competition but for someone like me who does not know similar products, a 'how it works' section with examples would be awesome.
Fingers crossed!
If you can disable them at the agent level and avoid the data out that would be even better.
At a previous employer the defaults were quite literally half of our log volume, that we were paying for. I was doing a sanity check before renewing our datadog contract and was very not-pleased to discover that.
They really don't give one whit about log discipline or allowing the user to influence the agent's log levels
We just moved away from Datadog because their log storage pricing is too high for us. We moved to BigQuery instead. But the interface kind of sucks.
Would love to get this up and running. A couple of questions:
1. Is it possible to setup outside of AWS/GCP? I would like to set this up on a dedicated server.
2. If not - then do you have a pricing comparison page where you give some example figures? e.g. to ingest 1 billion log lines from Apache per month it will cost you roughly $X in AWS hosting fees and $Y per seat to use Opstrace
We've done a deep dive into the cost model for metrics and posted more about it here: https://opstrace.com/blog/pulling-cost-curtain-back. We are still working on a full cost analysis for logs - I'd be happy to send it to you once we have it (feel free to email me mat@opstrace.com to chat about your use case). Our goal is to be super transparent (see https://news.ycombinator.com/item?id=25992081) with cost and to have a page on our website that helps someone determine what to expect (probably some sort of calculator with live data). Our UI will also show you exactly what your system is currently costing you with some breakdown for teams or services so you know who/what is driving your monitoring cost. We're doing user testing on our to-be-released UI now and would love to have people like yourself give us early feedback (since you mentioned the BigQuery interface).
Thanks for the feedback, we appreciate it!
It's from this page: https://opstrace.com/docs/quickstart
Opstrace does ship with a "system" tenant designed for monitoring the Opstrace system itself. This tenant has built-in dashboards that we've designed to show you the health of the Opstrace system.
Incidentally, having sharable "dashboards" across people/teams/organizations is something we are also working on, so people don't have to re-invent dashboards all the time.
We also have some guidelines for you to ingest metrics from Kubernetes clusters (https://opstrace.com/docs/guides/user/instrumenting-a-k8s-cl...) and are building native cloud metrics collection. Feel free to follow along in GitHub: https://github.com/opstrace/opstrace/issues/310.
One thing comes to mind: we don't bill by data volume. Wavefront is charging you for the volume of data your applications produce. This can lead to negative outcomes, such as surprise bills from a newly deployed service and a subsequent scramble to find and limit the offenders.
We think this pricing model forms the wrong incentives. Charging by volume means a company is more incentivized to have their customers (you) send you more data, and less incentivized to help them get more value from that data. This is a fundamental change we want to bring to the market--we want our incentives to align with yours, we want to be paid for the value we bring to your company. We charge on a per-user basis. You should monitor your applications and infrastructure the right way, not afraid to send data because it might blow the budget.
I know it can scale to massive volumes without interaction from us.
I know it'll be available when our infrastructure isn't. By being a third party we can be confident that any action on our part (such as rolling an SCP out to an AWS org, despite unit tests) won't impact the observability we rely on to tell us we've screwed that up.
I can plug 100s of AWS accounts and 10s of payers into it and I don't have to think about that in terms of making self-hosted infrastructure available via PrivateLinks or some other such complication.
I pay mid six-figure sums annually for these things to "just work". If you folks believe I can achieve this functionality on a per-seat basis I'd be interested in saving those six figures.
As mentioned in the original post here, at the core of Opstrace is Cortex (https://cortexproject.io). We know that Cortex scales well to hundreds of millions of unique active metrics, so depending on the exact characteristics of your workload, the fundamentals should be there.
However, Cortex is a serious service to run and if you were to DIY it would require operations work that you currently don’t have with Wavefront. This is the problem we’re trying to solve—making these great OSS solutions easier to use for people like you.
Opstrace is made to be exposed on the internet (which is optional of course), so you can easily run it in an isolated account to keep it safe from all other operations. And in fact, this is the configuration we recommend for production use.
Regarding “100s of AWS accounts and 10s of payers”... does that include any form of multi-tenant isolation? We support multi-tenancy out of the box to enable controlling rate limits and authorization limits for different groups. We’d need to talk in more detail about that. If you’d like to do that privately, please shoot me at chris@opstrace.com. We’re of course happy to continue the discussion here with you as well.
Thanks for sharing this perspective, stressing the relative value of predictability.
Of course, when things go pear-shaped the last thing you want to discover is that your monitoring pipeline doesn't work as expected. We feel you.
Your skepticism is justified and I'm super happy to see that here. We know that our future users are (and should be) quite demanding with respect to robustness of the platform.
We're not naively assuming that it's easy to build a platform that is highly available, auto-scaling, and generally worry-free.
In fact, based on our experience, we really know that we'll have to invest an incredible amount of engineering effort in order to make things super reliable and predictable. On the other hand, by making some smart decisions we can get far with little effort. We have super strong building blocks that we can rely on (such as using a cloud-provided database for storing critical configuration state).
> If you folks believe I can achieve this functionality on a per-seat basis I'd be interested in saving those six figures.
The bet is on, but of course we need a bit of time :)
I was the engineer who was heavily involved with monitoring at my last job and a lot of what this is doing aligns with what I would have done myself. At my new job, I work on different stuff but I can see we're going to run into monitoring issues soon too. I'm so, so, so glad this is an option because I do not want to rebuild that stuff all over again. Getting monitoring scalable and robust is HARD!
As mentioned earlier (https://news.ycombinator.com/item?id=25993825), our goal is to be super transparent; we want you to fully understand what you’re spending on infrastructure. We feel good that there’s an incentive to help you work through the problems that you’ve mentioned.
Attributing collection and querying is made easier with authentication enabled by default. You can make your tenants as fine- or coarse-grained as you want, handing out authentication tokens to the producers writing to those tenants. This makes it easier to trace back to sources of bloat. You can also place rate limits on individual tenants to prevent bloat in the first place.
Additionally, we think users might reconsider the premise of the problem. Because the cost of running Opstrace follows cloud economics (because it runs in your own cloud account), it's basically as cheap as it can possibly be. So you might consider that you do not have as much pressure to curate what is stored as you think. (I didn’t say "no" pressure, but "less" might be a huge improvement. :-) )
On top of bad UX, I do think the storage layer is where customers are really getting hit by these companies. The big players are using very unoptimized ingestion and querying layers and pretending like tiered storage never happened. Developers share some of the blame too by not being at all pragmatic about how long and how much to keep. It's a tough nut to crack.
What's the plan for commercial? They run it themselves and pay per user? If so, that's refreshing.
We will also have a blog post about bad UX in a couple weeks… stay tuned. What are some of your biggest gripes about UX?
Wish you all the best. and Congratulations!
DataDog has a UI. Does Opstrace? Or is it just a CLI/API based tool?
If you actually have a UI element to your product you’re doing a huge disservice to yourself by not actually showing this anywhere...
EDIT: I don’t mean to sound negative, I’m wondering if positioning this against Datadog is going to create immediate, potentially incorrect, expectations in people’s minds as to what this product might provide.
From first impressions I’d say this is much closer to Prometheus (which does have a UI but it’s so basic it may as well not - but then the UI is not the point of Prometheus).
Not to say you won’t get there, but why not shout about your current strengths right now (if you’re going to talk about the product) and not necessarily where you hope to be.
I only make this point because the “open source DataDog” headline just spins me off in the complete wrong direction as a newcomer and I end up confused and somewhat underwhelmed. With better positioning this wouldn’t have been the case!
That said, the project looks v interesting so keep up the good work and I hope you take my constructive criticism in the way it’s intended!
I’m surprised the mods haven’t edited this title.
Source: I’m an engineer that’s used, operated and hacked on a medium-sized prom+grafana; and used Datadog at a large, multi-region, global scale.
What we want to emphasize is that it is possible to build this and have the advantages of a Datadog without the drawbacks.
I just felt let down because the promise of the title (which I was excited by) doesn’t match reality in that it’s quite impossible to deliver Datadog’s feature set for metrics, traces, and logs — and tie all three together — on top of prometheus+grafana because the underlying TSDB doesn’t even support the notion of user-customizable indexing. Prometheus indexes all labels, which is why most Prometheus users even have to worry about high cardinality and time series explosion. This is one point highlighting how OpsTrace’s current architecture can not satisfy their positioning; there are many more. I’m sure they’re aware of them.
As an aspiration, the title makes sense. As someone in their target market, I was a bit turned off by the embellishment.
That’s my point: your software architecture must evolve to deliver on your promises (open-source Datadog) because it’s impossible to satisfy your promises with the architecture as it exists today.
Nonetheless, I appreciate all the responses. Good luck to OpsTrace!
Seems to me that these are at odds. If you're open source, why does anyone have to pay for these things?
If you're open core, I think it's mighty misleading to say things like "We will always be open source" because then not only is it untrue on its face, but also if someone contributes useful features to the open source project that compete with or supplant your paid proprietary bits, you are incentivized to refuse to merge their work - extremely not in the spirit of open source.
My perspective, which you asked for, is that open core is dishonest, and that you should be honest with yourselves about being a proprietary software vendor if that's indeed your plan, and stop with the open source posturing.
If I've misunderstood you, then I apologize.
If they used "free software" language, then we might have a case for posturing.
Our intention is to be really transparent with how we build and price software, which is why our commercial features will also be public in our repo, but commercially licensed. Transparency is critical in our opinion.
This is the model we’ve seen work for other highly impactful software projects.
We’ve created a ticket to track our addition of commercial code to our repo: https://github.com/opstrace/opstrace/issues/319
Don't call source-available proprietary software "open source", or say that you'll "always be open source" if you're just going to be source-available for parts.
There’s nothing inherently dishonest when a company emphasizes their open source strategy. Open source community building is as much about shipping code as it is leading people, and that requires you to be transparent about your intentions. I’ve interpreted opstrace’s release as just that.
I think the concern about neutral project governance is an important one. It’s early days, but from what I’ve seen it seems clear what is being sold vs what is open today. The fact that the project is released under the Apache v2 license means that folks are able to reuse, distribute, and sell the project as they wish — even fork it if they dislike the direction. That said, if governance is a priority for your use I’d definitely look to project in neutral software foundations like the Apache Software Foundation and CNCF.
Also, it's great to see the early focus on developer experience - "opstrace create".
In a nutshell, running all these various components (Grafana, etc) is a royal pain in the neck. Even if `opstrace create` spawns them easily, the problem is running/maintaining them. We want someone to run these for us as a SaaS/PaaS and we're happy to pay them.
Re: your principles:
(1) The customer should always own their data --> we agree. However, we are happy for you to be a custodian of that data.
(2) We don’t want to be a storage vendor --> neither do we. We want storage to be someone else's problem. We're happy for you to use a cloud platform like AWS/GCP and charge us a 50% markup.
(3/4) Transparency, predictability of costs, open source --> all excellent.
> is a royal pain in the neck.
It's fun to see how different people put the same unpleasant experience into words in this thread. Thanks for adding your personal touch. Every time we hear something like that, we're re-assured that we're on the right track.
> Even if `opstrace create` spawns them easily, the problem is running/maintaining them
Yes. You're right. While we can be proud of our setup/installation process already, we know that there's so much more to it. We don't underestimate that. Maybe also see https://news.ycombinator.com/item?id=25998587, where I just commented on the robustness topic.
> However, we are happy for you to be a custodian of that data.
Great.
> We want storage to be someone else's problem.
I share that perspective. We, of course, are happy to let S3/GCS do the actual job.
> We're happy for you to use a cloud platform like AWS/GCP and charge us a 50% markup.
That's great to hear, and I hope you can be enthusiastic about the fact that our markup is _not_ going to be relative to storage volume. It's going to be independent of that.
> Transparency, predictability of costs, open source --> all excellent.
Thanks for sharing. That's incredibly motivating.
Keep an eye on us, and we'd love to hear from you!
By the way, we are requiring that any vendor we choose in this area support OpenTelemetry as we've already instrumented our apps with it. Lightstep, Datadog, and others already supporting.
And yeah OpenTelemetry is where lots of the industry is going and so are we. You can already do cool things with us and the collector but we plan on automating a lot and expand on that.
A little bit about that in this commment: https://news.ycombinator.com/item?id=25995424
It might not fit your use case...but it might.
On the cost topic, last week we published a blog post analyzing the cost of running Opstrace on AWS (https://opstrace.com/blog/pulling-cost-curtain-back). (In fact, feel free to do a local repro to confirm our results.) As mentioned elsewhere here on HN, we are incentivized to provide total transparency in terms of what you spend on your cloud infrastructure. We haven’t compared ourselves to everyone, but feel confident that letting our customers pay S3 directly is the best deal possible.
As for scrubbing PII, they are now supporting the OpenTelemetry tracing API which does this as standard. For query endpoints you will see something like "QUERY Users where name=? email=?", i.e. masking with "?" chars as you only care about the keys since those what determine your indexing or lack thereof. (This is handled in the OpenTelemetry application library/plugin level.)
As an aside, PII scrubbing should be done that whether or not you own the cluster, because even if you own the cluster you generally don't want your support staff seeing PII esp. as organization grows larger.