Launch HN: Castodia (YC W20) – Populate Google Sheets from databases
Before we started working on this, we were working on an idea around data pipelines. It didn’t take off so we had to pivot mid-batch. We had less than 1 month to do user interviews, build our product, get it approved, and launch it. To start with, we knew we wanted to stay in the data space. We spent the first week talking to over 30 people at different companies. In those conversations, we noticed that sales, marketing, and operations teams constantly need to ask developers to help them export data from the database and often have to submit follow-on data requests because they forgot to add a dimension needed for analysis (they often do this by submitting Jira tickets). This is an inefficient yet surprisingly common thing at companies today. Perhaps even more surprising is that data scientists at big companies like Facebook, who have invested heavily in data infrastructure and analytics tools, often use spreadsheets as a step in their data analysis workflow. While they can pull data for analysis themselves, they too export data into CSV and then open it in a spreadsheet. This means that their data doesn’t update automatically and the process has to be repeated each time.
At first, we were considering building our own BI tool to solve this problem. However, during our conversations we noticed that people feel new-tool fatigue, especially when their companies rotate through different tools that are used for the same purpose. For instance, we’ve heard of companies going from Tableau to Looker and back to Tableau. Or from Kibana to Sisense to Looker. Sometimes it feels like companies are paying a lot of money for tools that people aren’t really using just because it’s the thing everybody “needs to have”. Each one comes with its own structure, data modeling and steep learning curve. It's overwhelming. At some point people just realize that whatever tool they learn will likely change once a new VP gets hired and wants to do things in a new way. In anticipation of that, they default to the one tool they know how to use and likely won’t be replaced soon. Spreadsheets also happen to be an elegant solution for simple calculations, quick pivots, and high level data exploration. They are elegant in a way that no $50k a year enterprise visualization tool can be - not because they can’t do those things, but because people don’t know how to do it quickly with them. It’s also hard to change workflows for people who live in spreadsheets. Spreadsheets are a pretty good for ad-hoc analysis and summaries that are used for presentations and reports.
So instead of another BI tool, we built a Google Sheets add-on that connects to a database and lets them search through tables, filter/order the data, and then load it into Sheets directly. In the future, we’ll let them schedule data refreshes for any saved query, so all their calculations and pivot tables update with the latest data from the database. We’ve built our first integration with Postgres and testing a MySQL, MongoDB, and public dataset connector. We plan to add integrations to more data sources, including data warehouses such as Snowflake and BigQuery.
We implemented it to only get read-access to the database and we don’t store connection credentials or replicate data in our database. We are focused on data analysis at the moment, but a few customers have requested features to also write into database (write access to the database). Not sure yet how to do this safely with spreadsheets or whether we should at all. We’ve heard suggestions such as database write roll-backs or some intermediate data queue that can be approved by table owners.
Other challenges include: (1) building data connectors because many data sources spits ...
52 comments
[ 3.7 ms ] story [ 124 ms ] threadI'm not in the market for something like this but it resonates with me since a few times now we've run into the need to jury-rig some scripts to pull data from a MySQL DB to populate a Google Sheet, which then does some simple calc/re-packaging for display on a dashboard or other simple BI use-case. But to do it you end up maintaining basically a simple "app", even if it's just a cron job hitting a python script hourly or something.
I also like the idea of connecting the world's most powerful / most adopted BI tool (spreadsheets) and the more proper data stores one finds in DBs.
Its certainly more affordable than Tableau and other traditional BI tools.
The userbase is probably not well represented on HN...
I think the Google Sheets export is very valuable, it opens up data to non-developer analysts and general business users. But I also can't see how this product is competitive at this price with only this functionality.
[1] https://redash.io/help/user-guide/integrations-and-api/how-t...
[2] https://redash.io/integrations/
For problem (2), I'm not sure if you're directly returning data from lambda or if you're using lambda to use the sheets API to update the sheet after the data is returned.
If it's the first one, there's no way around it since lambda functions called through API Gateway have the 30 second timeout that you're running into. If it's the second you should be able to use two lambdas one as one that ingests the requests and adds it to a queue, and the second one (which can have up to a 15 minute timeout) that consumes the queue.
Shameless plug : We built a multi-directional sync engine and a company called Syncari around it to solve exactly this problem. And added data pipelines around it. Mostly meant for sales operations/marketing operations folks, self-service style.
Other challenges:
1. Building uniform data connectors suck. GLHF.
2. Long queries: can you return a promise, then callback to an Apps Script endpoint when the data is ready?
3. Row Limits: I don't see aggregation in your screenshots, that would certainly help. Spreadsheet users know about row limts, just give them good UX and control. Sheets gets sluggish pretty quickly on large data sets, so you may want to have a lower default limit so the UX stays snappy and a warning for large queries.
1. Yes, We learned that pretty quickly. Especially when working on postgres and mongoDB. And this is the value we’re trying to deliver. As a user, you don’t have to know technical differences between databases and query them all the same way.
2. Interesting idea, we’ll give it a try, thanks!
3. Yes, we don’t have an aggregate feature just yet, working on it. For large datasets, we're thinking about loading a sampled version of the dataset instead for a quick analysis (like histogram analysis and so on). Lowering default limit is interesting, but we are worried it might compromise the experience (if you can only load 10,000 out of 1,000,000 records, you might not be able to do all the analysis you want). Still exploring solutions
What kind of data governance does your service support? Can you automatically filter/redact PII? This would be a great feature.
Cool product, by the way!
How are you handling MySQL differently than Postgres such that it would warrant a higher price for the consumer?
It's also not super clear what you get with your 100 queries, you can hook up DataGrip or another tool to any of these data sources and get infinite queries for free.
disclaimer: I used to work at Stitch Data
"Other Data While using our Service, we may also collect the following information: sex, age, date of birth, place of birth, passport details, citizenship, registration at place of residence and actual address, telephone number (work, mobile), details of documents on education, qualification, professional training, employment agreements, non-disclosure agreements, information on bonuses and compensation, information on marital status, family members, social security (or other taxpayer identification) number, office location and other data. " https://www.castodia.com/privacy-policy
Edit.
To be specific, the limit on runtime of the container. It is longer on the services I mentioned above.
1. We’re in healthcare, so data privacy / security and HIPAA compliance are a must. One way to limit this problem might be by architecting it so that our data doesn’t touch your servers.
2. Support for ActiveRecord-like abstraction layers would be great. Our underlying schema is fairly gnarly due to multiple rounds of regulation-driven changes to our data model so directly querying the Postgres DB isn’t really feasible for most of the team that does basic analysis.
Pricing is fine/cheap for what it could make possible for business intelligence/analytics.