The slides [1] helped me grok what problems this tool solves pretty quickly.
Everything big data is moving to blob storage these days, but streaming can lead to small files problem or longer latencies. File fragments stored locally with proxied readers seems like a simple solution to that.
The GitHub README and docs mention that Gazette has been running in production for 5 years, but I don’t see any mention of _where_. I assume this began as an internal project at some company - does anyone know which?
Hi, I’m the primary author of gazette. Interesting seeing it here, happy to answer questions.
We’re continuing to improve it - it’s a core implementation detail of our current project & company Estuary Flow (https://estuary.dev), which aims to further simplify and democratize low latency data products.
8 comments
[ 2.2 ms ] story [ 43.2 ms ] threadEverything big data is moving to blob storage these days, but streaming can lead to small files problem or longer latencies. File fragments stored locally with proxied readers seems like a simple solution to that.
[1] https://gazette.readthedocs.io/en/latest/overview-slides.htm...
I am going to give this project a big no if it has anything to do with with liveramp.
We’re continuing to improve it - it’s a core implementation detail of our current project & company Estuary Flow (https://estuary.dev), which aims to further simplify and democratize low latency data products.
we are looking forward to qualified replication in upcoming pg15.