Launch HN: Bracket (YC W22) – Two-Way Sync Between Salesforce and Postgres
We have two main use cases: 1) building apps on top of Postgres instead of Salesforce, and 2) replacing existing Salesforce <> Postgres syncs (either Heroku Connect or in-house builds).
Postgres makes a bunch of things easy for developers: building responsive web apps, handling SSO and user access rules, and scaling large datasets like time-series data. But Sales, Customer Success, and Ops teams usually don’t have direct access to Postgres - instead, they rely on Salesforce as a type of database. These teams need up-to-date data on users, orders, and services, and they need to edit this data. As a result, in a lot of organizations there’s a sort of abyss between Postgres and Salesforce.
For example: say you're a car rental company. People rent cars via your web app built on Postgres, but your CX team uses Salesforce to track/update users, cars, and rentals. One of your users calls in to say that they were in an accident and the car is totaled. Your CX team (on the Salesforce side of the abyss) needs to manually update the status of the car - "Unavailable" - and reassign upcoming reservations to other cars. These edits made in Salesforce must sync to Postgres so that the user sees their updated reservation details.
We first came across this syncing problem when we were deep in pivot hell during YC W22. At the time, we were two weeks away from Demo Day and we had been pivoting for five weeks. We felt like failures every morning, and it seemed inevitable that we’d drop out. Then we talked to a founder who told us how hard it is to simply keep Airtable and MongoDB in sync with each other. He tried Zapier, he tried writing custom scripts, all to no avail. At last, we had 1) a technical problem 2) frustrating a smart founder 3) with a big potential market. We started to build, raised a conservative amount of money at Demo Day, and kept our burn extremely low.
After a year, we had a product keeping Airtable, Notion, and Google Sheet tables in sync with larger databases, but it still felt like a stop-gap: companies were often using us to stand up lightweight BI, avoid creating internal tools from scratch, or build quick admin dashboards. Once they felt the limitations of, say, Google Sheets, they moved off Bracket to a more permanent solution. Not only did this shrink the size of the market, but also left us feeling like we were creating a vitamin, not a painkiller.
Then we talked with companies who wanted to sync CRMs - specifically Salesforce - with Postgres. They either had high-maintenance in-house solutions for syncing, or they were using Heroku Connect, which enables two-way syncs between Salesforce and Heroku-hosted Postgres. They couldn’t get rid of Salesforce and they couldn’t allow the two systems to get out of sync, so they were stuck with Heroku Connect.
There are two major problems with Heroku Connect, though: 1) it's super expensive, and 2) it ties you to Heroku Enterprise as a hosting platform. These companies wanted something as reliable as Heroku Connect, but hosting-agnostic and priced competitively. We sensed an opportunity to build something useful here, so we got to it.
Bracket makes it easy (90 seconds of setup) to get a Salesforce object and Postgres table syncing with each other in near-real-time. Using our app, you connect your Salesforce via oauth, connect your Postgres via connection URI (with options for SSL protocols), and either have Bracket generate a Postgres table from scratch or map fields between Salesforce and an existing Postgres table.
Once connected, Bracket can sync two ways or one way at a cadence decided by the user. We offer two sync methods: polling (default) and streaming. Using the polling sync method, changes sync on average every 30-60 seconds. Using streaming, changes sync ...
81 comments
[ 0.28 ms ] story [ 188 ms ] threadI've always been surprised that fivetran/hightouch don't ever throw in a .reverse().
That’s one reason why people trying to stand up two-way syncs have been stuck with dated tech (Heroku Connect) or have had to build in-house.
We’ve worked with some folks who were cobbling together ELT and rETL to achieve a type of two-way sync, but they faced three problems with this approach: most importantly, handling for infinite loops, where an insert in one data source triggers an insert in the other, which then gets picked up anew, etc. Second, it wasn’t clear how to handle merge conflicts when the same record is edited from both sides (we give priority to the primary source). Finally, it was frustrating to debug problems when having to log into two separate tools.
It's an immature tool, so I don't yet know that I can claim we've spent _less_ than Fivetran on the additional engineering and ops time, but it feels like it has potential to do so once stabilized.
[1] https://docs.usebracket.com/streaming
Right now, we're focusing on use cases that require changes to be synced within a minute. Fivetran and its ilk are generally more focused on large data batches at less frequent intervals (e.g., to unlock 1-minute syncs at Fivetran, you must be on the Enterprise plan, whereas we offer that out of the gate with the Starter plan).
> With the polling method, Bracket stores an encrypted copy of your data as an intermediate source of truth. We mostly do this to prevent infinite event loops, but it also helps with merge conflict resolution.
I see you have an on-prem version, but i am still not convinced why you need to store it? Can it not just be stored in an extra table at the client's side?
At first, it was a matter of doing discovery with people who we thought would find Bracket useful. These weren't sales convos, but pure feedback/discovery. From that process, we got a good sense of which roles face these pain points, and exactly what those pain points are.
Since then we've split our time between 1) writing content/distributing it in places people would find it helpful and 2) reaching out directly to people who would use Bracket. In the latter case, we really don't like generic sales campaigns, so we spend a good bit of time crafting each message.
The polling solution is neat but I am imagining it’ll run into issues with API limits and performance, especially for tenants on shared infrastructure: have you encountered that? Are you limited to working with customers that are paying Salesforce enough to have fast/reliable infrastructure?
(I don’t think it’s a problem for your success as your product is most valuable to those paying Salesforce for reliability, I’m just curious how you are thinking about the problems I gave up on! Maybe things with Salesforce have changed in the last 5 years.)
If I'm understanding correctly, the scheduled jobs refer to the Bulk API (I agree it executes at seemingly random speeds). We only use the bulk api on the initial “seed”, where we write a large amount of data from salesforce to postgres. Otherwise, when it comes to reading/writing data, we stick to the REST API, which we’ve found pretty performant and which Heroku Connect seems to rely on, too: https://devcenter.heroku.com/articles/mapping-configuration-...
> Are you limited to working with customers that are paying Salesforce enough…
Yeah, right now we do require that users are on Salesforce plans that include API access, which are Performance, Developer, Unlimited, and Enterprise (or Professional w/ API add-ons).
500 million? Billion? Billion plus?
I'm curious how big a business like this can grow, by estimation, based on someone else's platform like this.
We think the Salesforce syncing market by itself is 500 million to 1B. If you start including all of the tools we have our eyes on (Hubspot, Monday, ERPs), the market size comfortably gets into the billions.
Going after Heroku Connect makes sense as a starting point, but we've got our sights beyond that.
What's the tech stack?
[1]: https://docs.estuary.dev/reference/Connectors/capture-connec...
We also have a NetSuite to Postgres sync that's currently in Alpha.
For those of you considering an integration to NetSuite, keep in mind that NetSuite has both SOAP and REST connections, but (as of today) neither works with every single object. So anything you build will likely have to leverage both APIs. Pagination can also take 30-60 seconds per page retrieval, so that's an added complication for large data sets.
(Yes, you could possibly leverage NetSuite Restlets, but then you have custom code on each and every customer, hence we avoid that).
If anyone wants help on building an integration with NetSuite, happy to see if I can help. My email is in my about.
If you're curious, you can read more about our approach to it and the actual SQL we use here: https://docs.usebracket.com/streaming#postgres
Message me, might have a use-case with a Fortune 100.
IMO, the reason many YC companies struggle with enterprise sales is because it's generally much harder than PLG, with long sales cycles and delayed feedback loops. It was definitely a learning curve for us!
An underrated challenge with enterprise sales is often that customers often misguidedly believe that 1) they can build it themselves, 2) they are worried about your companies ability to "scale" to their needs, and/or 3) the customer wants to "own" that responsibility for personal/political/career reasons.
Sales is hard for everyone including Azure, AWS, etc, just gotta keep grinding at it.
Rolling up all the YC single-feature SaaS business into a single sales org and providing an a la carte set of solutions to partners would make a ton of sense.
I guess that is the theory behind SaaS.group and its ilk.
If so, then Heroku connect probably does need an alternative.
A business user (non-technical 10x audience) just wants to sleep at night knowing their daily backup was successful and they have recovery capabilities on-demand.
There are unicorn scale companies in this space (such as https://www.owndata.com/). And Salesforce is getting back into the backup game.
So we decided to double down in this smaller, underserved area for now as we try to build something people love, even if it's a bit niche.
That being said, there probably is room in the backup space for challengers against the incumbents - it's a massive market.
A few things I can't immediately see from the docs: do you support subsetting a data source -- only sync records matching criteria? Do you support to/from different instances of the same connector (e.g. Salesforce to Salesforce)? Can you perform any transformations like map over the data, normalize or denormalize tables, etc?
Many clients I can think of this being most useful for would rather host it themselves, is that an option?
One critique: I can't imagine recommending this to a client without SSL support. I'd highly recommend just baking that in to every tier. It would demonstrate that you're serious about keeping your customers' data secure.
> do you support subsetting a data source -- only sync records matching criteria?
In a one-way sync from Postgres to Salesforce, yes, you can apply filters using a SQL statement, but we’re working on adding this to the Salesforce side as well as two-way syncs. From your perspective, how would you want to set these filters? A SOQL query, or something else?
> Do you support to/from different instances of the same connector (e.g. Salesforce to Salesforce)?
Yup, our infra is data-source agnostic! But Salesforce <> Salesforce is not heavily tested, so there may be some funky behavior with oauth tokens if you’re trying to connect objects from two different instances during onboarding. Curious what use case you have in mind?
> Can you perform any transformations like map over the data, normalize or denormalize tables, etc?
Besides the one-way SQL filter I mention above, we try to make field mapping easy between the sources by automatically transforming when necessary (e.g., transforming a Salesforce picklist to a Postgres integer and vice versa). But we’re working on allowing users to create more detailed field-level transformations in the next few months.
Thanks for the feedback! Totally hear you on SSL, we’ll move that to every tier.
For myself, I want to write SOQL there, though I'd guess many of your target customers will want a point-and-click option. Selecting a list view for that object could be an interesting UX hack that might be worth exploring: there's some Salesforce Labs product that does that.
> Salesforce <> Salesforce is not heavily tested... Curious what use case you have in mind?
I've seen a number of configurations for different purposes, the most common one being sandbox data movements.
Management of multiple production orgs gets complex fast, but there are a few places where a tool like this could find a niche: org migrations come to mind. There's often an interim period where you're two way syncing (even though you'd rather not!) before the org being phased out is done.
Good luck!
> Many clients I can think of this being most useful for would rather host it themselves, is that an option?
Right now you can self-host the associated datasets (like the Postgres event log table), but we're still working on allowing you to self-host the entire service. Stay tuned :)
I also liked the submission recently about a git client that uses SQL. I like the idea that more things can be exposed as SQL (either directly or by syncing with a RDBMS). There is a lot of good tooling around it and despite the S not meaning Standard, the dialects are close enough that it isn't a problem.
Sounds similar to the use cases we're seeing, where it's not only easier to process/build on Postgres, but also saves you on the Salesforce API.
(And, how are conflicts resolved? In a huge system with millions of records coming from everywhere it can fast become nightmarish?)
Our polling approach is relatively database-agnostic. We just need to handle each DB's quirks with our transformers (e.g. dealing with MySQL's lack of BOOL field types).
Streaming is currently Postgres-specific. We're planning on rolling out support for MySQL next, after we've finished our Hubspot integration. Do you have a specific DB in mind?
> (And, how are conflicts resolved? In a huge system with millions of records coming from everywhere it can fast become nightmarish?)
The primary source wins any merge conflicts that happen within a sync period. With polling, it's pretty straightforward: at every poll, we see how each side has changed, and for any record pairings for which there were edits on both sides, we prefer the primary source.
With streaming, we employ a hybrid method, where we only poll when events occur in either Salesforce or Postgres. If at that poll, the same record has been edited on both sides since the previous poll, we still prioritize the primary source (Salesforce). You can read the step-by-step flow here: https://docs.usebracket.com/streaming#the-streaming-sync-met...
This is a very fancy way of saying that you just drop conflict and pretend they didn't happen. Syncing databases is very, very tricky. Conflicts are a big part of the trickiness.
1. Do you have any plans to release an API rather than utilizing a webapp for defining the sync? 2. Does your Salesforce integration support syncing metadata, including custom metadata? 3. Do you have data on the impact Bracket has on platform events and other Salesforce limits? 4. Can you share any information on pricing?
Thanks!
Salesforce objects map quite nicely onto Laravel’s Eloquent models (and booleans work fine! ref @ianyanusko in a sibling thread)
On the Salesforce side we use triggers to send a summary of field changes to Laravel to apply to its MySQL database. These are very cheap in terms of Salesforce limits and consumption!
Changes originating from the Laravel side use Salesforce’s REST API. It’s handy taking the round trip through Salesforce when saving changes as it lets flows/processes run and formulas to do their thing before data is persisted in MySQL.
Syncing data from Salesforce (to seed a database for example) is done via REST too. It works OK.
I considered productising it at one point, but ideas are a dime a dozen, it’s a lot harder to execute well and Bracket is doing exactly that, kudos!
This has been a thoroughly interesting post, and I’ll keep my eye on Bracket. You are however out of budget for my client, we had to achieve syncing on a shoestring ;)
> Syncing data from Salesforce (to seed a database for example) is done via REST too. It works OK.
Have you thought about using the Bulk API for seeding? We started relying on that instead of REST, which helped us seed massive DBs much faster / more efficiently.
As you know, the REST API will deliver a maximum of 2000 records per request, so beyond a certain scale it's not really tenable in terms of speed & consumption of API calls.
So yes, Bulk API is probably going to feature soon.
Cheers Ian!
Salesforce's scattered docs don't make best practices super clear, so designing the syncs took us some time.
It's an absolute hell, most community frameworks barely work, there's zero good solutions to error handling, we had our Azure Application Insights constantly red because of timed-out requests that are in fact good, because they just mean there were no events to transmit.
I refuse to believe that multi-billion dollar company that is Salesforce couldn't come up with a RabbitMQ sink or virtually any better solution to that problem, especially that they could gate it behind a subscription most companies would pay for since they spend millions on SF either way...
https://aws.amazon.com/blogs/compute/building-salesforce-int...
I know even less about Azure, but it looks like Azure Data Factory (?) provides some kind of similar functionality?
https://learn.microsoft.com/en-us/azure/data-factory/connect...
Disclosure: I'm one of the HubSpot founders.