Launch HN: Metaplane (YC W20) – Datadog for Data
Data teams are often the last to know about data-related issues. They commonly find out only when an executive messages them about a broken dashboard. This is comparable to finding out about your servers being down only when your end users report it! In software engineering, this problem is solved with observability tools like Datadog and SignalFx. These monitor your system over time by tracking metrics (like CPU, memory usage or any arbitrary value), and sending alerts when they hit thresholds or are anomalous.
Metaplane solves this problem for data teams. We continuously monitor our users’ data warehouse tables and columns, testing for things like row counts, freshness, cardinality, uniqueness, nullness, and statistical properties like mean/median/min/max, as well as schema changes. After we build up a baseline of data points for each of these tests, we send alerts on anomalies to the user's Slack channel. Each alert includes metadata like upstream/downstream tables and BI dashboards affected by the issue, so that the user can assess how important the issue is and how quickly it should be addressed.
We're particularly careful about alert fatigue and false positives. Since we can't ask users to set manual thresholds (they would be changing all the time), we have to make a reasonable prediction based on past data, which can result in false positives and false negatives. If we under-alert, we miss important issues, but if we over-alert, users become desensitized and start ignoring alerts. Our solution is to include "Mark as anomaly" and "Mark as normal" buttons with each alert, for users to provide feedback to the model.
To give a common example, Metaplane can tell you that a revenue metric in a Snowflake column has spiked from $100 to $10,000 in an unexpected way. The alert includes upstream dependencies in dbt and downstream Looker dashboards that are impacted. Another example is if a table in Redshift that is usually updated every day hasn’t been updated in over 48 hours. A third example is if a table in BigQuery that typically increments 10M rows every day suddenly adds only 1M rows because of an upstream vendor bug. These are all what we think of as “silent data bugs” — all systems are green, but your data is just wrong!
Over the last eight months, we've caught problems like these for data teams at dozens of companies including Imperfect Foods, Drift, Vendr, Reforge, Air Up, Teachable, and Appcues.
Today, we’re excited to launch our self-serve product and free plan with the HN community. Setting up monitoring for your data stack takes less than 10 minutes. Here's a 4 minute demo video to see how it works: https://www.loom.com/share/1aa54eb8b45548e180f6ab3a4a580cc5. We make money by charging for more tests and team/enterprise features. You can use our new free plan or try out all of our features in a 30 day trial, no credit card required.
Our goal is to help data teams of any size be the first to know about data issues. We think observability will become as much of a no-brainer to data teams as it is to software engineers today. Starting on AWS?—get Datadog. Bringing on Snowflake?—get a data observability tool (hopefully ours!). Eventually we want to support more use cases that you’d expect from a Datadog for data, like log centralization and diagnostics, spend monitoring, performance insights, and deep integration with upstream applications. For now, we’re just starting where the pain is highest.
We'd love to hear your ideas, experiences, and feedback, and will be...
70 comments
[ 0.28 ms ] story [ 146 ms ] threadThat said, we’re still experimenting with pricing though, and I can see the argument for a tier that’s more suited for individual paid plans. We also want the free plan to be pretty generous — is there a specific constraint that feels too limiting? Thanks for your feedback!
I incorrectly assumed that once I enter the Growth plan $400/mo, I get access to dbt/lineage. But those are only "checked" when you pay $800/mo version of the Growth plan.
So it really feels like you have 4 plans: Free, Growth, Business?, and Enterprise.
When you sign up, you'll have access to everything immediately, so you can connect and try it out and when we start enforcing the limits, we'll give you ample notice.
I'm curious if you have opinions on the plans and pricing. Do the plans make sense as 1. individual 2. team for warehouse 3. team for whole stack 4. enterprise?
Looking forward to hearing what you think, and please reach out to team@metaplane.dev because we’d love to explore building this integration for you!
We want every data team to have have observability as soon as they have data in a warehouse, and that means: 1) letting you implement the tool within 10 minutes, without talking to us, 2) providing a free plan and paid plans that make sense for modern data teams, and 3) focusing on being as helpful as we can with as little configuration as possible.
Monte Carlo has built a strong team and has done a great job telling the story of data observability for larger companies. Overall we want to support even the smallest data teams, who we feel are being underserved by other companies in this space.
1) Do you have Metabase on your roadmap? Lightdash?
2) I see that you alert on schema changes, which is great. Can you monitor for schema changes of a Postgres database? Reason I ask: Fivetran (and others) will try to buffer some schema changes from you to prevent data loss (drop columns, rename columns, etc). There is some more complex nuance I have in mind here, but it’s a bit too long to type out on my phone, :)
2) Several of our customers use us to alert on schema changes in Postgres, specifically so they can get ahead of application database changes that will end up in the warehouse, so you're definitely not alone! Here's a link on how to connect postgres: https://docs.metaplane.dev/docs/postgres
That's an excellent stack and one we kept front and center when building out Metaplane, so definitely let us know if you have any feedback or suggestions here!
[1]: https://github.com/lightdash/lightdash/issues/632
My plan was to monitor the postgres database in the staging environment, so we can be alerted to schema changes before they are released into production (and hopefully stop the production deploy).
I have a goal of moving this even further upstream into the CI build for the source application itself (Ruby on Rails in this case), so that the application's test suite will fail a developer introduces a breaking schema change. Note: this is a pretty tricky problem to solve without a) the tests being way too brittle OR b) super slow end to end tests. I have some goals of introducing which is a mashup of: Spectacles [1], Pact [2], and dbt models [3].
[1] https://www.spectacles.dev [2] https://pact.io [3] https://docs.getdbt.com/docs/building-a-dbt-project/using-so...
Mitigating the impact with monitoring is where we're at right now, but we're with you that preventing errors can be even more important.
If it's interesting to you, we're happy to open up a shared slack channel to dig into the nuance as well! Just email me (guru@metaplane.dev) with the email you'd like to be added.
When Nick Schrock created dagster, he argued that many "data cleaning" tasks which people attribute to "data engineering" aren't actually "cleaning", but are architecture problems. I believe schema changes also fall into this category. I'm extremely new to data engineering, but when I think about "What are the things which will break this system?" an application engineer thinking "I'm going to rename this column and my tests pass, so this should be fine" will break things all the time. (Similar goes for dropping a column, changing a one-to-many into a many-to-many)
The reason our customers haven’t required this is because we’ve tried to take security seriously from day one and Metaplane doesn’t store any customer data (just the metadata). We received our SOC2 Type II report, support IP whitelisting, SSH/Reverse SSH tunnels and are always exploring other integration options like AWS’ PrivateLink.
That said, we definitely understand the need to keep even metadata on-prem, so we plan on tackling that later next year.
another security approach is to enable your customers to close their inbound firewall ports and link listeners. this helps cloud and on prem models have far stronger security.
example here (disclosure: i am a founder of the company behind this solution) with both open source and OEM/SaaS models:
https://github.com/openziti-incubator/zdbc (code for one implementation - a wrapper around the JDBC drivers)
https://netfoundry.io/zero-trust-database-security/ (blog post with links to developer example video, whitepaper, etc)
How does this differ from a data reliability platform like Datafold? https://www.datafold.com/
And can this replace what https://atlan.com/ does as well?
Datafold is primarily known for their Data Diff regression testing that simulates the result of a PR on your data within a CI/CD workflow. There’s definitely a need for proactively preventing data issues from occurring in the first place, but issues introduced via code are only one subset of potential data quality issues.
Metaplane is focused on catching the symptoms first via continuous monitoring. Regression tests don’t replace the need for observability, and vice-versa.
Atlan is primarily known for their data workspace features that make collaboration easier, like a data dictionary, SQL editor, and governance.
Data collaboration is a huge unsolved problem and data monitoring does play a role there. But Metaplane is focused squarely on the problem of detecting data issues and giving you relevant metadata to prioritize and debug.
https://github.com/capitalone/DataProfiler
Load any document, profile and monitor the profiles for changes that would impact downstream applications.
Very common problem, you all are in a great space! Very interested and will check out!
We definitely want to explore suggesting data tests based on profiling. Don’t be surprised if you see a fork!
e.g. we are just 2 of us - yet need this.
Also, apologies - but I litereally hear in my head "Meatplane" every time I read 'Metaplane' (but I do like the name - Suckit Zuck, metaplane is just way more meta-ier than 'Meta'
Tell me that you'll data me
Watch me like you'll never let me down
'Cause I'm Aler-tin' on a meat-plane
Don't know when I'll be backed-up again
Oh Ops, I hate to go
https://youtu.be/SneCkM0bJq0?t=34
With dbt and Snowflake poised to take over this space, I can see this fitting right in on top of these tools. One idea would be to build in dbt integration into metafold
I'm curious - how did you settle on the pricing? I can see it being a differentiator from Datafold, Supergrain depending on your feature set
Good idea, we actually do have a dbt integration that pulls in lineage and job metadata from your dbt manifests: https://docs.metaplane.dev/docs/dbt. Eventually we want to let you configure Metaplane tests from your dbt YAML.
Pricing is still in flux to be honest. We wanted to start with a price that was approachable for small teams, comparable to other tools in your stack, and could be paid for without going through a whole procurement process. But we’re trying to stay as flexible on pricing as possible!
We've already had those "wow I'm glad we have this tool" moments, just a couple of months in.
I guess that'd be kinda a crossover between data integrity and marketing... in this case, an anomaly IS the data, where an unexpected increase/decrease in pageviews somewhere is something we'd love an alert on, but only after some threshold. As you pointed out, doing that manually just results in a bunch of false positives for those of us who aren't professional BI or statisticians.
I wanted to +1 what you said about “organizations that don’t have a proper data warehouse and dedicated BI staff.” At the end of the day, a huge number of companies (maybe even most?) don’t have dedicated data teams but still want to know be alerted about data anomalies. Heck, we at Metaplane even fall into that camp.
In short, in many of the small businesses I've worked for, it was trivially easy to collect analytics data, yet monumentally difficult to analyze or make use of it in a meaningful way. The signal to noise ratio was incredibly low. One aspect of that was the need to manually discover unusual events, usually by creating manual dashboards and alert thresholds. But lacking a strong data science background, designing proper statistical tests to identify significant events was not something I could easily do.
It's a different approach than Metaplane, which we are actually evaluating for our dwh and backends monitoring. Trackingplan is more about creating trust on the data collection side of the problem.
Drop me a line, josele (at) trackingplan (dot) com if you want to know more!
I guess I just have a hard time seeing why this would help people solve real data quality issues.
Btw congratulations on the launch. The tool looks great.
you can check it out
Of particular interest would be able to correlate upstream events/logs with key product metrics - e.g. DAP/WAP/MAP, click-through-rates, revenue, etc.
Really looking forward to seeing how the product grows!