Hey HN, I wanted to show off my project Marmot! I decided to build Marmot after discovering a lot of data catalogs can be complex and require many external dependencies such as Kafka, Elasticsearch or an external orchestrator like Airflow.
Marmot is a single Go binary backed by Postgres. That's it!
It already supports:
Full-text search across tables, topics, queues, buckets, APIs
Glossary and asset to term associations
Flexible API so it can support almost any data asset!
Terraform/Pulumi/CLI for managing a catalog-as-code
When should you reach for a data catalog via a data warehouse or data lake? If you are choosing a data catalog this is probably obvious to you, if you just happened on this HN post less so.
Also, what key decisions do other data catalogs make via your choices? What led to those decisions and what is the benefit to users?
Hey there,
Great to see Marmot here and I'm a huge fan of your project. Recently, we deployed a catalog but we went with open-metadata https://open-metadata.org/ another amazing project.
What we missed on marmot was existing integrations with Airflow and other plugins like Tableau, PowerBI etc as well as other features such as sso, mcp etc.
We're an enterprise and needed a more mature product. Fingers crossed marmot reaches there soon.
How are you able to see a datasets lineage across storage types. For example how are you able to see that an s3 buckets files are the ancestor of some table in Postgres?
Not to be confused with Marmot, the multi-master distributed SQLite server, which has been around for a couple years longer and just came out of 2 years in hibernation, shed its NATS/Raft fat in favour of a native gossip protocol for replication.
I’ve been burned by metadata platforms twice now and honestly, it’s exhausting.
The demo is always incredible - finally, we’ll know where our data lives! No more asking “hey does anyone know which table has the real customer data?” in Slack at 3pm.
Then reality hits.
Week 1 looks great. Week 8, you search “customer data” and get back 47 tables with brilliant names like `customers_final_v3` and `cust_data_new`. Zero descriptions because nobody has time to write them.
You try enforcing it. Developers are already swamped and now you’re asking them to stop and document every column? They either write useless stuff like “customer table contains customers” or they just… don’t. Can’t really blame them.
Three months in, half the docs are outdated.
I don’t know. Maybe it’s a maturity thing? Or maybe we’re all just pretending we’re organized enough for these tools when we’re really not.
Hey, that's a good question! At the moment, it treats the latest run as the desired state. So any new changes to a schema will simply overwrite the old version. I'd like to version these so people can navigate schema versions in the UI.
If using a plugins, they currently are triggered either via the CLI or a schedule on the UI, so updates will only appear in the catalog after a plugin has run.
I'd also love to have some native integrations beyond Airflow. Once I've matured the existing plugin ecosystem a bit more, it's high on my list (along with column-level lineage).
11 comments
[ 4.4 ms ] story [ 31.3 ms ] threadMarmot is a single Go binary backed by Postgres. That's it!
It already supports: Full-text search across tables, topics, queues, buckets, APIs Glossary and asset to term associations
Flexible API so it can support almost any data asset!
Terraform/Pulumi/CLI for managing a catalog-as-code
10+ Plugins (and growing)
Live demo: https://demo.marmotdata.io
Also, what key decisions do other data catalogs make via your choices? What led to those decisions and what is the benefit to users?
What we missed on marmot was existing integrations with Airflow and other plugins like Tableau, PowerBI etc as well as other features such as sso, mcp etc.
We're an enterprise and needed a more mature product. Fingers crossed marmot reaches there soon.
https://github.com/maxpert/marmot
The demo is always incredible - finally, we’ll know where our data lives! No more asking “hey does anyone know which table has the real customer data?” in Slack at 3pm.
Then reality hits.
Week 1 looks great. Week 8, you search “customer data” and get back 47 tables with brilliant names like `customers_final_v3` and `cust_data_new`. Zero descriptions because nobody has time to write them.
You try enforcing it. Developers are already swamped and now you’re asking them to stop and document every column? They either write useless stuff like “customer table contains customers” or they just… don’t. Can’t really blame them.
Three months in, half the docs are outdated.
I don’t know. Maybe it’s a maturity thing? Or maybe we’re all just pretending we’re organized enough for these tools when we’re really not.
I'd also love to have some native integrations beyond Airflow. Once I've matured the existing plugin ecosystem a bit more, it's high on my list (along with column-level lineage).