Launch HN: Jitsu (YC S20) – Open-Source Segment Alternative
I've been doing data engineering for more than ten years (half of that time, I didn't know that it's called "data engineering”). Before Jitsu, I was a co-founder and CTO of GetIntent, an ad-tech startup. Although it was ad-tech (I'm sorry for that!), we also built a quite fascinating technology platform. We processed up to 1 million events per second at peak, and all those events needed to be stored somewhere.
We churned through a few data warehouse platforms along the way. In 2013, we started with Hadoop's HDFS and a bunch of map-reduce jobs on top of it. Then, when we decided to allow our customers to run ad-hoc reports, we switched to BigQuery. BigQuery was great, but expensive—especially with some customers obsessively clicking the refresh button. Finally, in 2017 we migrated to self-hosted ClickHouse which in my opinion is still the best analytics database in the world.
All that time, we spent a fair amount of effort to get data to the database. When you're dealing with millions of events per minute, running an INSERT statement per event won't work. What if the DB is down for maintenance? How can you be sure that all 50+ edge nodes are aware of recent DB schema changes? Also, did you know streaming data to BigQuery is costly while batching data is free?
We tried different approaches: first, we would write local log files, sync them to HDFS, and load data to BQ (or ClickHouse) with map-reduce jobs. To improve data freshness, we ditched HDFS and started to send data in batches to the DB directly from edge servers. We experimented with Kafka, but it felt too complex for that task at the time.
I always dreamed about a straightforward service, to which I'd throw JSON objects, and it would take care of the rest: queueing, retrying, updating database schema, etc.
Then I discovered Segment. I liked it at first. It seemed very developer-friendly with a nice API and excellent documentation. But the pricing model and data delays (the event gets to DB in 12 hours after it has been sent to Segment) killed the whole idea. And it was not open-sourced. In my opinion, being open-source and self-hostable is a must for such a fundamental part of the architecture as data collection.
I left GetIntent and got accepted to YC with a different idea for the Summer 2020 batch. The idea was to build a churn prevention and BI tool for online retailers. It didn't take off, but in the process we made a component to collect customer's app events and put it to DB. We tried to hack a solution on top of the ELK stack, but I was frustrated with ElasticSearch’s lack of SQL support. Here I was back to square one: there's no good open-source event collection service yet, and we needed to build one, once again.
So we decided to focus solely on that problem. We ditched all the previous code, which was in Java, rewrote the data collection server in Go and hacked together what we called EventNative [1]. It was received very well, and we started to get users.
Over the last 11 months, we've been busy building the UI, adding Connectors (to pull data from external APIs), polishing data warehouse support, adding javascript support to transform incoming data, and implementing dozens of other features.
Now we're launching Jitsu, an open-source Segment alternative. With Jitsu, we make it easy to collect data and send it to databases (we support all major players: ClickHouse, Redshift, Snowflake, BigQuery and Postgres). We’re deployed in production, including into a large gaming publisher, eSignature service, and many other great co...
113 comments
[ 3.6 ms ] story [ 190 ms ] threadIt was fun watching you grow Jitsu and love the way you provide support!
I'm a little bit out of the loop in this event processing space. Do you think Jitsu could replace lower-level event processing implementations as Kafka/Kinesis? Or this is thought for more "high level" marketing stuff.
- Kafka as a company wide message bus: dozen's of (micro)services sending data there, and consumers listens to data. Each service doesn't know which other service will consume the data. For that case, we're not looking to replace Kafka — we're going to work along with it. We have a PR about supporting Kafka as destination [1] (Jitsu sends data to Kafka), and we will support Kafka as a source at some point (PRs are always welcome :))
- Kafka is used just as a transport between web-app and DB. In that case Jitsu is a perfect replacement
[1] https://github.com/jitsucom/jitsu/pull/537
P. S. The same applies to Kinesis too
How are your handling data between collection agents and storage? With Kafka, I know what I'm getting when it sits between the two and the advantages it offers.
https://github.com/jitsucom/jitsu/blob/0aaa74b59eb9d8c885c80...
I see that it instantiates an "AsyncLogger" - does the service wait until data is written to the log prior to returning success to the client?
Is the WAL the same source used to feed both database storage destinations and other SaaS destinations?
As someone who also builds an open-core product (though not directly modeled after an existing closed-source product) I really hope this kind of business model will become more accepted.
Look at the alternatives to closed source analytics that they have funded
Founder of RudderStack here.
- We truly believe that our product should be accessible for small teams too. That's why se made Jitsu very easy to deploy. I'm not sure you can deploy Rudder on Heroku, or on any service with a single Docker file.
- Our ETL component is open-source (and based on other great OSS projects - Airbyte & Singer). RudderStack haven't published the Cloud Extract (their ETL) to my knowledge.
- RudderStack aims to replace Segment, we go beyond that. We didn't copy Segment API one-to-one, we just added a Segment compatibility layer. Jitsu can be used for any kind of data. An example: a few companies (including our-selves) using Jitsu to collect open-source telemetry (anonymous usage). I'm not sure Rudder can be used for that use-case
Just to answer some of this:
- rudderstack has deploy ready helm charts, which I'd argue are significantly better than docker compose or docker setups because they set up all the other niggly parts. Would be cool to see that here :)
- rudderstack has gone quite far away from replacing segment. It's true that their core API is compatible and I think your transformation layer is really cool. However it can be used for those use cases because rudderstack doesn't really care about users or user IDs and can be used for any sort of data generally.
There's a piece in the docs talking about the fact that you don't get caught by adblock - whilst this may be true when someone launches it, that's not true of your platform. That's just the fact that a lot of smaller businesses will not get their URLs added to the ad block lists. I think it's a bit misleading to mention that in such a way because technically we're all tracking users and ad block is a way for users to choose not to be tracked, not be tricked into being tracked because someone has masked the tracking script ;) if a huge client (a la Adidas or something) decided to use your scripts I'm sure someone would eventually add it to the ad block lists that get propagated.
One of the things that would be cool would be some sort of opt in configuration. Segment has some awful consent SDK that is really bad, would be cool to see what you do there. GDPR is a big deal and browser fingerprinting is data processing. It's worth looking at your comments on being GDPR compliant btw https://www.eff.org/deeplinks/2018/06/gdpr-and-browser-finge...
-RudderStack team.
Reading through their comment below - "Jitsu can be used for any kind of data while Segment compatibility is just a thin layer on top".
I am guessing they have built a generic event API that can be used to send any JSON payload while RudderStack (like Segment) has a opinionated view of events - e.g. there has to be a userID (or anonymousID), that ID is persisted in a cookie (for web), every event must include that userID. Furthermore, there are certain standard for event tracking for specific verticals which RudderStack supports (e.g eCommerce https://rudderstack.com/docs/rudderstack-api/api-specificati...)
Having this opinionated view helps us map these events to all the 100s of destinations, otherwise, you cannot send any arbitrary JSON to these destinations. It also lets us build more post processing in the warehouse (e.g identity stitching, user sessions etc https://hub.getdbt.com/rudderlabs/, we are going to build more MLish applications like churn-models and release them too).
On the other hand, it becomes hard to send generic events (e.g. application telemetry) via RudderStack which seems possible via Jitsu. With RudderStack, you would have to create hacky userID to tag on every event which doesn't make sense.
In summary, go deep on one use case (customer-data) or wide as a generic event streaming platform.
Beyond this, there are other feature differences (transformation, reverse-ETL etc) but that's not a fundamental difference imo. They are just getting started and are a much smaller team so that's expected. Impressive to see what they have built.
And we have DBT models too https://hub.getdbt.com/jitsucom/jitsu/latest/ !
Noticed a typo on jitsu.com - DHW should probably be DWH.
For those who don't know what "Segment" is (like me) - this Jitsu thing seems to only be relevant to web-based/web-oriented apps.
Segment is a fancy event router / multiplexer. You emit events to it and it sends them to reporting and storage destinations.
It does have more features for web apps but that is not the only use case.
[1] https://jitsu.com/docs/sending-data/mobile-apps/ios-sdk [2] https://jitsu.com/docs/sending-data/api
This line alone is enough to infuriate me. So I am unable to block spying and data collection now?
I don't understand why we are still praising spyware tools?
Yes, jitsu can be deployed at custom domain such as track.yourcompany.com. And while some AdBlockers will block *.segment.com, track.yourcompany.com will remain functional. We don't consider this feature unethical, though. It depends on how data collected by Jitsu is used. If the app owner sells it without telling end-users that's probably bad. However, I believe most of our users using the data to improve product experience. And Jitsu can be configured to respect do-not-track/gdpr settings.
There are dozens of valid uses for this beyond ad-tech. Where we use Segment it barely even touches with marketing. Most of the value for Segment is piping user lifecycle events around to every platform and service you use to help enrich customer experience. Sure, call it sales or marketing or ad-tech, but that's really just an umbrella for trying to maximize revenue-per-customer - and isn't that the point of a SAAS platform?
I think we should be cautious about throwing the terms "spyware" and "malware" around right now; there are lots of very valid cases that should be labeled as such, but if we over-use the word it just makes it harder for us to delineate between powerful tools being used for good/valid purposes or deceptive ones.
[0] https://github.com/uBlockOrigin/uBlock-issues/issues/1357
[1] https://github.com/snowplow/snowplow
Think of us as Snowplow 2.0 )
Snowplow started at the same time as Segment (2012) but has evolved along a separate tech tree. Micro-service architecture, cloud native, using Kinesis or Cloud Pub-Sub as the data transit, enrichment framework plus a Confluent-style schema registry supporting very rich and versioned JSON Schema-based event payloads. We are built by and for data platform teams; our open-source behavioral data engine doesn't have a UI (our commercial Behavioral Data Platform does). Hosted trial here https://try.snowplowanalytics.com/
Definitely room for both product families in the market! I'm sure Jitsu will do great.
Looking at Jitsu as a Snowplow familiar person I tried to do a quick browse of their marketing site and couldn’t find anything about their back end architecture. Was immediately thinking that wasn’t the focus here which is concerning when thinking about enterprise scalable data patterns.
Also appreciate you taking the high road “room for both” while the founder of Jitsu says “we are better”
I’ll stick with the product with a solid schema strategy, thank you…
i am wondering how this compares to it?
Meltano will pull data from Singer connectors and do transformations, but they won't run Airbyte connectors, and you can't push data to Meltano
Jitsu will use Airbyte or Singer to pull data, and you can push the data to Jitsu. But Jitsu won't run DBT transformation. Although we can trigger DBT cloud jobs: https://jitsu.com/blog/dbt-integration
P.S. Meltano has a CLI, and we don't (yet)
HN doesn’t allow lesser users with lesser eyesight to read light grey on beige self text, unfortunately.
Our core is push integrations, that's the most complex part of the system. We see "pull" integrations as an additional feature that helps to enrich the data after events made it to DHW
A little note of advise: I wouldn’t start my company description as “the Y of X” or “the Y alternative to X”.
It’s okay to mention if you are similar to another well known company, but don’t use it to describe your company, specially not in the first line.
We are building Jitsu, (https://github.com/jitsucom/jitsu, https://jitsu.com/) We help companies collect events from their apps, websites, and APIs and send them to databases.
Think of us as an open-source Segment alternative.
However, I think that the product matters the most. You can change the tagline in a few clicks. Can't say the same about the product
First tell me what you do, then tell me what you are similar to.
>> We are building Jitsu, (https://github.com/jitsucom/jitsu, https://jitsu.com/) We help companies collect events from their apps, websites, and APIs and send them to databases. Think of us as an open-source Segment alternative
I hadn't heard of Segment and now I'm reading your competitors website.
Overall, I like this recent trend a lot - more companies are building open-source, lightweight, GDPR compatible analytics, chats (e.g. Papercups). I hope there will be good ways to monetize and sustain this. Wish you all the best, folks!
Thoughts:
1. You've got most major ads sources that I care about, but it seems that there is a higher bar to implementation. Segment lets me just plug in Google & FB ads and dump the entire shebang right into my data warehouse. A lot of marketing teams are going to have less time/resources to deal with implementation so smoothing this out is key.
2. Functions are an underrated and highly powerful feature of Segment. The ability to operate on data in transit, create custom connectors that "just run" (akin to CF Workers) and the like is a big selling point for more technically advanced marketing teams. It doesn't seem present here and that would hold a customer such as myself back on bigger scope projects.
3. I'd love to see a "compare us to your segment usage" where I select my data sources and destinations to see what you cover vs. Segment in a specific use case (and possibly pricing advantages on a self-hosted vs. non). This would make it much easier to sell through procurement and devops for new customers that are switching.
4. There are going to be a lot of people like me that are soon to start fresh in terms of marketing stack, so going after people before they select Segment might also be a play.
Looking forward to seeing where you all take this. Good luck!
1. Thats exactly the reason we have native connectors for Facebook and Google Ads (we didn't use ones from Airbyte and Singer). Jitsu can pull any combination from FB/GAds — it's almost like SQL! Airbyte/Singer just can't do that. Later we're going to vet other connectors too and decide if we need to re-implement them
2. We have functions too! https://jitsu.com/blog/javascript-transform
Downside: you'll need to build all visualization by yourself. Fortunately that's easy with tools such as Looker, Mode, Metabase etc
Upside: you can do with your data whatever you want - built any reports, join with other datasets etc. You not limited by reports MixPanel team build.
In reality, Jitsu and MixPanel could co-exist. Jitsu support s MixPanel as a destination (e.g. you send data to Jitsu ; Jitsu sends it to MixPanel and data warehouse).
This is not really clear from the website. Mixpanel is mentioned in https://jitsu.com/sources but NOT in https://jitsu.com/destinations. Also the docs seem very clear about that.
We are currently looking at Segment, Jitsu and others. While we generally liked Jitsu, this was kind of a big deal for us and made us lean towards Segment.
However, no final decision has been made yet. ;-)
Are you following the same tracking convention spec as Segment?
It looks like it might play well with my current logging system of choice, Seq [0].
Do you support inbound webhooks? I can see webhooks as a destination but not as a source?
[0] https://datalust.co/seq
As a bit of feedback, I highly suggest adding Webhooks as a source on your marketing site.
The first thing I did is navigate to the Sources page and searched for "webhook" which brought up no results.
I then searched your docs which only mention Webhooks in the context of being a destination rather than a source.
I realise now that you have quite a flexible ingestion API, but it took quite a while (and your confirmation above) to understand this!
The product looks awesome though! Good luck with the launch.
This way I can setup my source in Jitsu, get a unique URL, and then paste that URL into the tool generating webhook events (e.g. Shopify). A normalized schema based on the JSON payload doesn't need to be created for this to be useful.