Looker on top of Redshift. Events streamed in from Segment and ELT transforms managed by Airflow. Looker gives you nice visualizations and charting with RBAC and some some lightweight ETL functionality. The real advantage of Looker is their modeling layer which sits on top of your physical data and is done via a DSL called LookML. Source control is managed via a wrapper around git. The end result is that analysts can move lightning fast building their own models in SQL or do analysis via drag and drop explorations. Looker's customer support is the best I've experienced and hasn't changed since Google acquired Looker. We're likely moving off Redshift to Snowflake in the next 6 months because it is slow to pick up changes and we want to decouple storage and compute for scaling reasons. Airflow is an incredible platform but needs customization and templated workflows to make it great. Data build tool (DBT) is how we plan on managing ETL code via Airflow in the near future. We're also adding Spark, but more for data science.
If you're interested in an alternative to dbt that's a little more analyst friendly, check out Dataform. A lot of teams are using Dataform between the warehouse and looker, to handle transformation logic using pure SQL, but in a more user friendly format than Airflow. Get in touch if you'd like to chat! dan at dataform.co
- Amazon Redshift (data sourced from many different backend DBs; e.g. PostgreSQL, MySQL, other Redshift instances etc.)
- BigQuery (Some teams store the data for their projects here. For reporting, they're selectively transferred to Redshift.)
Reports:
- Tableau (extracts of pre-aggregated data from Redshift)
- Looker (connects live to and executes its own queries on Redshift)
Anything that is based on pre-aggregated (rather small, e.g. n_rows < 5mil) data is visualized on Tableau. If users want to work on a self-service tool they use Looker which can connect to (cleaned) raw data optimized for this use case.
ETLs for raw data run daily on Redshift. Reports are also scheduled on Redshift (with Airflow) but the refresh interval is report-specific.
Airflow -> S3 -> DBT with Spark/EMR or Redshift/Spectrum -> Redshift data marts -> Looker
At least, that’s the way we like our pipelines to work. In practice we have a couple of extractions that land directly in Redshift (we extract Zendesk data, for instance, with Stitch Data). We use Snowplow for click stream analytics. And we’ll likely move from Redshift to Snowflake (or mayybbbeee Bigtable) in Q2 of 2020.
We used to do all of our transforms via DBT within Redshift but have been offloading the heavier-duty pieces (like Snowplow event processing) to Spark jobs because they were too taxing on Redshift.
We’ve gone through 3 “generations” of Looker reporting — gen 1 was just wrapping LookML around our schema and forcing Looker to do the joins and generate SQL for everything we wanted to know. Second generation involved a bunch of common table expressions within Looker itself that worked, but were developed without much thought as to data mart design. Gen 3 is where we are now with super deliberate scoping and implementation of warehouses in DBT. Before any of that we rolled our own tool [0].
Happy to answer any questions. We’ve gone from ~30 to ~1000 employees in the 3 years since we started using Looker and have learned a thing or two along the way.
We are revamping our management of Snowplow event structures presently to make them more semantic and easier to use.
For our core data marts, we rely on naming conventions for the LookML fields, good tool tips, hold monthly training sessions for new employees, weekly office hours with analysts, and do occasional “train the trainer” sessions to try and make sure there is at least one Looker power-user in each business team.
I don't want to derail the thread, but I recently started a company (https://syndetic.co) that's working on this problem. We've been focusing on the external data dictionary use case (how does a data-as-a-service company explain a dataset to their prospective customers) but we've been encountering a number of companies that are evaluating data catalogs and other internal tools for their data science teams.
I would really appreciate getting your perspective - I'm steve (at) syndetic.co
The legacy players are a mess, the new cloud native offerings are overly Engineering focused and immature, and the few startups in this space are recreating the legacy solutions in a SAAS formats
Go forth and claim the significant checks that I and others plan to write for these mediocre offerings.
We’ve been battling Redshift scaling problems so it’s fluctuated a lot over the last 6 months (both up and down). $15k per month to an order of magnitude (not counting Looker licensing).
Just curious, if you could highlight some of the problems you've experienced with each "generation" of looker reporting? Was there any specific issues that encouraged the re-implementation?
1. Complex joins and poor performance.
2. Not pre-computing important attributes made them hard to slice and dice on (because they weren’t true dimensions)
2nd gen:
1. Really unmaintainable. Like....really unmaintainable. A rat’s next of ad-hoc common table expressions, that referred to each other in undocumented ways, used inconsistent CASE statements to transform enums to human-readable values, inconsistent time stamp formats (and time zones), etc, etc, etc. Fields in different explores that were named the same but didn’t quite match and no one could explain why.
2. All of it was SQL in LookML and the persistence strategy (e.g. when were the CTEs running, were they writing to disk or ephemeral, etc) was not manageable.
If we had been smarter, we would have introduced DBT after gen 1 and not gotten into the gen 2 mess.
There’s a ton of other nuance here, but that’s the high level.
> If we had been smarter, we would have introduced DBT after gen 1 and not gotten into the gen 2 mess.
Is this a roundabout way of saying you wouldn't have exposed looker to the underlying staged data?
We have two layers within our data warehouse: The ingestion/staging layer, and the organized/cleansed/segmented layer. We only give Looker projects access to the latter.
We still have a lot of chaos within our Looker projects (views with select star, dimensions for every column bc of select star, hard-coded schemas, etc...). Slowly working it out though.
Hi, do you manage your transformations both in redshift and EMR using dbt? Do you have a single dbt project/repo covering both or separate? How do you coordinate your dbt run against spark with dbt run against the data warehouse?
We use Segment for event tracking, Postgresql for transactional data and a number of spreadsheets and third-party integrations with Stitchdata. Since our data is relatively small, we use PG as a data-warehouse and heavily use DBT for ETL.
The people who are familiar with SQL just use DataGrip, for the UI we use our tool https://rakam.io.
Shameless plug: It's basically similar to Looker but it plays well with the product data and integration with Segment as well.
I believe this is the best combination of cheap/powerful for early-stage startups. My very non-technical cofounder is able to use metabase's simple GUI interface to create graphs/insights (even joining and aggregating across tables!), and for anything complex I can step in a give a helper SQL query. We have around 10M records we aggregate around for daily insights.
Metabase can also be run as a pseudo-desktop/web application to save additional cash (we don't do this though).
it is licensed under AGPL. For white-label embedding it requires a commercial license: https://www.metabase.com/enterprise/pricing.html
Pricing for variant with row-level security is not cheap as you might expect, so if you plan to embed reports in your web app it might be better to consider another alternatives.
Metabase has been serving our startup well. Not as good as Tableau, but the minimalist approach is really good and SQL is better than Tableau's LOD expressions.
Worlds apart, superset provides powerful visualisation capabilities, but is very difficult to use/setup/maintain (although they just launched as a company [0], so perhaps that will change).
Metabase is a very easy to use data exploration, data vis and reporting tool designed to give insights into data in an easy/self service way for mostly non-technical users.
I've POC'd both, and would recommend Metabase or redash unless you really know what you need out of superset
Pretty similar feature wise and intended use, I did a direct comparison and went for Metabase because it allows you to schedule reports delivered to email/slack (only allows you to schedule individual queries, not entire dashboards).
I didn't demo redash so can't comment on the actual use comparison.
Sorry not a question, just to say again that your UI rocks. After I came to this post, I stopped everything I was doing to install metabase for my company (and then used it for a project :) ). I have spotted some issues with the installation instructions where do I post them? Issue at Github?
Agreed. I helped setup a company on this last year and they're very happy. There's a 'Deploy to Heroku' push button install that was incredibly easy to setup. Great job Metabase folks.: https://www.metabase.com/start/heroku
For our team, using an ELT architecture (as opposed to ETL) [1] for managing our data warehouse has greatly reduced the complexity of our data processes. Instead of creating ETLs for every table we want to load into the data warehouse, we create the minimum necessary setup to copy the table into our data warehouse. Then, we write transforms, which are simply SQL statements, to generate wide-column tables that our non-technical users can use to explore data without worrying about joins or having to learn esoteric naming conventions.
Custom EL Scripts -> Redshift -> Transform Statements -> Redshift -> Metabase supports the data needs of all our departments with no dedicated data team members.
+1 for metabase. I’ve deployed it at 2 companies. The latest version allows you to compose “questions” you create in the system which is really powerful for organizing layers of data reporting.
Also, their API conforms 100% to Zapier’s polling feed structure, and I managed to build an integration that syncs our customer base to Pipedrive in a few hours. A bit more time and I have it automatically moving deals along in the pipeline as users progress with using the product. That alone was game changing for our customer success team.
We've also been using Metabase lately. It doesn't have some of the querying power of a Tableau/Data Studio, but in terms of being able to just click through and "discover" trends in your data, it's really nice. Super quick and easy to get started even for non-technical folks, and best of all it's OSS! Really impressive project.
Looks interesting but I would like to know how is it different than Apache Superset. What is the concept of "Question" here? Is it about some "official terminology" for a "SELECT" query? Any video demo?
Have you ever tried to install/setup Superset? The chances of you completing this task fall somewhere between "Next to impossible" and "Not going to happen".
Luigi, AWS S3, DBT, Snowflake and Re:dash (currently analyzing Metabase or Looker to allow queries without SQL)
Luigi runs our scrapers and other workflow management tasks (e.g. DB backups).
All raw data lives in S3. We make an effort to be able to recreate the whole data warehouse from the raw data, so if any cleaning/normalization process fails, we have this safety net. I'm curious to hear if others use a similar pattern, or if there are better options.
DBT handles both loading the data from S3 into Snowflake (by creating the Snowflake Stages), and transforming the data in Snowflake. This isn't how DBT is used usually, but it felt wasteful to add Stitch or another tool to load the data into Snowflake, as snowflake supports it out of the box. I also created a `setup_snowflake` macro operation that creates our users, warehouses, databases, etc., in Snowflake (a kind of "poor man's Terraform")
I don't think Snowflake requires introduction. It's an amazing tool. We used Postgres before, but Snowflake is much much better, even though our DB is pretty small (~200 GB).
Finally, we use Re:dash as a BI, but I'm checking other options that allow usage without SQL (currently Metabase and Looker).
> All raw data lives in S3... I'm curious to hear if others use a similar pattern, or if there are better options.
You still may need to maintain information about what is in S3. If you're already using Snowflake you can probably achieve this via External tables. Otherwise you could run your own hive meta-store or use AWS external tables.
We're also trying to decide whether it's cheaper/easier to store things in S3 or just keep everything internal in snowflake. In some cases the compression is so good on some of our data, it's better keeping data stored in a table than in internal or external staging. Obviously this is bad if we ever have to move away from Snowflake, but we haven't committed to either approach and lots of data is still backed up in S3. Our total data warehouse is about 200TB at the moment and we're projecting significant growth over the next couple of years.
Business Intelligence. Making reports. Usually over a large set of data. A normal query would be too slow so the results are pre calculated. Maybe you will calculate the average and total sales per city then dump the results into a table. From there you can quickly generate more granular reports like sales per state.
Stitch -> Redshift (with DBT for data modeling) -> Looker
For a smaller company, it makes a lot of sense for us to use off-the-shelf tools like these rather than rolling our own pipelines. Every once in a while we run into issues where Stitch isn't quite as flexible as we'd like (can't set dist/sortkeys etc), but that's the tradeoff you make for its simplicity.
DBT is amazing and I can't recommend it highly enough!
Looker works for analytics, but we're starting to do more data-sciency work, and it doesn't have built-in Jupyter notebooks or anything like that. Does anyone have a solution they use and like for hosting and sharing notebooks alongside a stack like this?
how does one end up doing BI | Data Engineering ? currently a frontend | full-stack JS dev. tired of that world and want to switch to something stable ?
Technical skills with databases are a must. It's basically a database engineer + being able to generate accurate and insightful reports/metrics/visualizations.
More value comes from database engineer skills as well as being able to understand the structure of a business. Little actually comes from the BI platform you choose.
I'd highly recommend the Kimball books, e.g. "The Data Warehouse Toolkit".
Unless you're working with a lot of data, the technical parts of a data warehouse are relatively easy to grasp (the dbt team wrote some great posts on how they structure their projects [1]). After that, what really makes a difference is how you structure our data warehouse, how you model your data in a way that allow you to query it efficiently and ask many questions. That's where the Kimball books shine.
If you're already doing engineering work you may be able to just go get a job focusing on data - if you're upfront about wanting to learn. I've found most people have learned the skills on the job. Besides the books mentioned I commonly recommend these resources
DataCamp.com has some great courses
Tableau has some great online education - many people learned BI from them
Many free resources for getting started with data at different levels (note: this is one of my sites)
https://dataschool.com
If you have a data team:
Stitch / Segment -> BigQuery -> Dataform -> BigQuery -> Looker
I work with many companies helping them set up their data stack, and from what I've seen this is pretty much the optimal set up. These tools all require very little maintenance, are relatively cheap (compared to the man power required to set these things up from scratch internally), and scale well as companies grow and complexity increases.
If you don't have a data team:
Segment -> Amplitude/Mixpanel
If you don't have a dedicated data team, you can do a lot with tools like Amplitude and Mixpanel. Get your engineering team set up with Segment and tracking all of the important interactions within your product, and set Segment to pass those events to Amp/Mix. Self serve analytics in these tools is then pretty easy and will serve you well until you start hiring a data team.
Full disclosure: I work for Dataform (used to be a customer, loved it so much I decided to join them)
Good point. I guess I mean, if you put a gun to my head and forced me to pick one, this is what I'd pick (and I think it works for a pretty meaningful percentage of companies that are looking for an answer to the original question)
We've recently changed from : Segment -> BigQuery -> Looker to Segment -> BigQuery -> Dataform -> BigQuery -> Looker.
The addition of Dataform into our stack has completely changed the game, allowing us to maintain one source of objective truth and maximise looker (minimising the joining etc. in LookML, instead pointing it straight at tables in BigQuery).
This in turn means our downstream data users are happier to use a fast and reliable looker instance!
Sorry to be that one avoids-the-question comment, but:
Without design sessions to figure out your data store design (look up Kimball, Immon), and then monitoring/testing to make sure everything is running smoothly, any data stack will be hell.
Badly designed data brings fatigue and rancor and unnecessary work and is a huge hard-to-quantify money suck.
IMO the right solution to this is not spending a bunch of time upfront trying to design your warehouse. You just want to adopt a tool that makes it easy to continuously refactor into more optimal formats as your warehouse scales. Happy to discuss more if you're interested! dan at dataform.co
Everyhting goes through S3 because Snowflake storage is on it.
dbt is amazing, we began using it a month ago and it already transformed the way our data team work. It really is a value multiplier for everyone. Data engineers are happier because they don't need to write and maintain data transformations, analysts are happier because they can maintain their own SQL pipelines & the whole company is happier because we now have a great documentation tool to explore our data.
We also are big fans of Snowflake, make operating a data warehouse a breeze.
Then, we use a mix of Redash & Tableau for reporting.
Redash for static reporting (open to the whole company) & Tableau to create more complex data tools we expose to some internal teams; Marketing, Risk, Finance ...
Custom pipes mostly coordinated by Luigi (heavily customized) -> Redshift -> DBT -> Looker.
Some spark mixed in at the ingestion and transformation phases.
Like someone else said in this thread, we're currently battling Redshift scaling problems and are looking to offload more of the raw data to S3 using Spark to create read views.
No data catalog right now but the Looker setup is extremely well-maintained. Hoping to deploy Lyft's Amundsen to fix this more generally.
The main reason is that only a subset of all our datasets are actually exposed in Looker, and Amundsen would have applicability to other data sources - certain production semi-OLAP DBs, the future hive-on-S3 deployment... that we have no plans to expose in Looker at all.
That said I haven't looked closely at how far we could go with Looker, I'll check it out.
I see. We actually develop a BI tool similar to Looker but also want to implement a data catalog feature embedded into it. I'm familiar with Amundsen but I believe that this kind of tool needs to have tight integration with the BI tool and Amundsen still has a way in terms of integrations. (searching among saved reports, dashboards and collaboration with teams, etc.). Otherwise, using spreadsheets is still easier and maintainable.
I would love to hear the features that you would need in a data catalog system.
181 comments
[ 4.5 ms ] story [ 252 ms ] threadCurrently: Internal Data Warehouse -> RDS -> Internal web app (Django, React)
- Amazon Redshift (data sourced from many different backend DBs; e.g. PostgreSQL, MySQL, other Redshift instances etc.)
- BigQuery (Some teams store the data for their projects here. For reporting, they're selectively transferred to Redshift.)
Reports:
- Tableau (extracts of pre-aggregated data from Redshift)
- Looker (connects live to and executes its own queries on Redshift)
Anything that is based on pre-aggregated (rather small, e.g. n_rows < 5mil) data is visualized on Tableau. If users want to work on a self-service tool they use Looker which can connect to (cleaned) raw data optimized for this use case.
ETLs for raw data run daily on Redshift. Reports are also scheduled on Redshift (with Airflow) but the refresh interval is report-specific.
Airflow -> S3 -> DBT with Spark/EMR or Redshift/Spectrum -> Redshift data marts -> Looker
At least, that’s the way we like our pipelines to work. In practice we have a couple of extractions that land directly in Redshift (we extract Zendesk data, for instance, with Stitch Data). We use Snowplow for click stream analytics. And we’ll likely move from Redshift to Snowflake (or mayybbbeee Bigtable) in Q2 of 2020.
We used to do all of our transforms via DBT within Redshift but have been offloading the heavier-duty pieces (like Snowplow event processing) to Spark jobs because they were too taxing on Redshift.
We’ve gone through 3 “generations” of Looker reporting — gen 1 was just wrapping LookML around our schema and forcing Looker to do the joins and generate SQL for everything we wanted to know. Second generation involved a bunch of common table expressions within Looker itself that worked, but were developed without much thought as to data mart design. Gen 3 is where we are now with super deliberate scoping and implementation of warehouses in DBT. Before any of that we rolled our own tool [0].
Happy to answer any questions. We’ve gone from ~30 to ~1000 employees in the 3 years since we started using Looker and have learned a thing or two along the way.
[0] https://github.com/groveco/django-sql-explorer
We are revamping our management of Snowplow event structures presently to make them more semantic and easier to use.
For our core data marts, we rely on naming conventions for the LookML fields, good tool tips, hold monthly training sessions for new employees, weekly office hours with analysts, and do occasional “train the trainer” sessions to try and make sure there is at least one Looker power-user in each business team.
I would really appreciate getting your perspective - I'm steve (at) syndetic.co
The legacy players are a mess, the new cloud native offerings are overly Engineering focused and immature, and the few startups in this space are recreating the legacy solutions in a SAAS formats
Go forth and claim the significant checks that I and others plan to write for these mediocre offerings.
https://looker.com/platform/blocks/embedded/create-a-data-di...
1. Complex joins and poor performance. 2. Not pre-computing important attributes made them hard to slice and dice on (because they weren’t true dimensions)
2nd gen:
1. Really unmaintainable. Like....really unmaintainable. A rat’s next of ad-hoc common table expressions, that referred to each other in undocumented ways, used inconsistent CASE statements to transform enums to human-readable values, inconsistent time stamp formats (and time zones), etc, etc, etc. Fields in different explores that were named the same but didn’t quite match and no one could explain why. 2. All of it was SQL in LookML and the persistence strategy (e.g. when were the CTEs running, were they writing to disk or ephemeral, etc) was not manageable.
If we had been smarter, we would have introduced DBT after gen 1 and not gotten into the gen 2 mess.
There’s a ton of other nuance here, but that’s the high level.
Is this a roundabout way of saying you wouldn't have exposed looker to the underlying staged data?
We have two layers within our data warehouse: The ingestion/staging layer, and the organized/cleansed/segmented layer. We only give Looker projects access to the latter.
We still have a lot of chaos within our Looker projects (views with select star, dimensions for every column bc of select star, hard-coded schemas, etc...). Slowly working it out though.
Shameless plug: It's basically similar to Looker but it plays well with the product data and integration with Segment as well.
I believe this is the best combination of cheap/powerful for early-stage startups. My very non-technical cofounder is able to use metabase's simple GUI interface to create graphs/insights (even joining and aggregating across tables!), and for anything complex I can step in a give a helper SQL query. We have around 10M records we aggregate around for daily insights.
Metabase can also be run as a pseudo-desktop/web application to save additional cash (we don't do this though).
-Another Metabase team member
Metabase is a very easy to use data exploration, data vis and reporting tool designed to give insights into data in an easy/self service way for mostly non-technical users.
I've POC'd both, and would recommend Metabase or redash unless you really know what you need out of superset
[0] https://preset.io/blog/2019-10-09-series-a-announcement/
I didn't demo redash so can't comment on the actual use comparison.
https://github.com/metabase/metabase/issues/4387 opened this issue on Feb 15, 2017
https://github.com/metabase/metabase/issues/2052 opened this issue on Feb 29, 2016
https://github.com/metabase/metabase/issues/6741 opened this issue on Jan 22, 2018
For our team, using an ELT architecture (as opposed to ETL) [1] for managing our data warehouse has greatly reduced the complexity of our data processes. Instead of creating ETLs for every table we want to load into the data warehouse, we create the minimum necessary setup to copy the table into our data warehouse. Then, we write transforms, which are simply SQL statements, to generate wide-column tables that our non-technical users can use to explore data without worrying about joins or having to learn esoteric naming conventions.
Custom EL Scripts -> Redshift -> Transform Statements -> Redshift -> Metabase supports the data needs of all our departments with no dedicated data team members.
[1] https://www.dataliftoff.com/elt-with-amazon-redshift-an-over...
Also, their API conforms 100% to Zapier’s polling feed structure, and I managed to build an integration that syncs our customer base to Pipedrive in a few hours. A bit more time and I have it automatically moving deals along in the pipeline as users progress with using the product. That alone was game changing for our customer success team.
Thanks, team Metabase!
P.S. I'm another Metabase + Postgres + Bigquery fan. Thanks Team!
Luigi runs our scrapers and other workflow management tasks (e.g. DB backups).
All raw data lives in S3. We make an effort to be able to recreate the whole data warehouse from the raw data, so if any cleaning/normalization process fails, we have this safety net. I'm curious to hear if others use a similar pattern, or if there are better options.
DBT handles both loading the data from S3 into Snowflake (by creating the Snowflake Stages), and transforming the data in Snowflake. This isn't how DBT is used usually, but it felt wasteful to add Stitch or another tool to load the data into Snowflake, as snowflake supports it out of the box. I also created a `setup_snowflake` macro operation that creates our users, warehouses, databases, etc., in Snowflake (a kind of "poor man's Terraform")
I don't think Snowflake requires introduction. It's an amazing tool. We used Postgres before, but Snowflake is much much better, even though our DB is pretty small (~200 GB).
Finally, we use Re:dash as a BI, but I'm checking other options that allow usage without SQL (currently Metabase and Looker).
It's an alternative to Looker but more affordable, and more powerful than Metabase.
You still may need to maintain information about what is in S3. If you're already using Snowflake you can probably achieve this via External tables. Otherwise you could run your own hive meta-store or use AWS external tables.
We're also trying to decide whether it's cheaper/easier to store things in S3 or just keep everything internal in snowflake. In some cases the compression is so good on some of our data, it's better keeping data stored in a table than in internal or external staging. Obviously this is bad if we ever have to move away from Snowflake, but we haven't committed to either approach and lots of data is still backed up in S3. Our total data warehouse is about 200TB at the moment and we're projecting significant growth over the next couple of years.
Most probably, Business Intelligence.
https://en.m.wikipedia.org/wiki/Business_intelligence
"Two words combined that won't make sense" etc.
For a smaller company, it makes a lot of sense for us to use off-the-shelf tools like these rather than rolling our own pipelines. Every once in a while we run into issues where Stitch isn't quite as flexible as we'd like (can't set dist/sortkeys etc), but that's the tradeoff you make for its simplicity.
DBT is amazing and I can't recommend it highly enough!
Looker works for analytics, but we're starting to do more data-sciency work, and it doesn't have built-in Jupyter notebooks or anything like that. Does anyone have a solution they use and like for hosting and sharing notebooks alongside a stack like this?
More value comes from database engineer skills as well as being able to understand the structure of a business. Little actually comes from the BI platform you choose.
Unless you're working with a lot of data, the technical parts of a data warehouse are relatively easy to grasp (the dbt team wrote some great posts on how they structure their projects [1]). After that, what really makes a difference is how you structure our data warehouse, how you model your data in a way that allow you to query it efficiently and ask many questions. That's where the Kimball books shine.
[1] https://community.snowflake.com/s/article/Use-Case-How-We-Co...
DataCamp.com has some great courses
Tableau has some great online education - many people learned BI from them
Many free resources for getting started with data at different levels (note: this is one of my sites) https://dataschool.com
Description of common roles in BI/data: https://chartio.com/learn/data-analytics/distinguishing-data...
Holistics handles both ETL, transformation and self-service visualiation (Looker alternative), all in 1 tool.
I work with many companies helping them set up their data stack, and from what I've seen this is pretty much the optimal set up. These tools all require very little maintenance, are relatively cheap (compared to the man power required to set these things up from scratch internally), and scale well as companies grow and complexity increases.
If you don't have a data team: Segment -> Amplitude/Mixpanel
If you don't have a dedicated data team, you can do a lot with tools like Amplitude and Mixpanel. Get your engineering team set up with Segment and tracking all of the important interactions within your product, and set Segment to pass those events to Amp/Mix. Self serve analytics in these tools is then pretty easy and will serve you well until you start hiring a data team.
Full disclosure: I work for Dataform (used to be a customer, loved it so much I decided to join them)
Warehouse: PostgreSQL
Reporting platform: Looker
Easy, agile, and cheap.
Lots more documentation here https://about.gitlab.com/handbook/business-ops/data-team/
We have many of our KPIs embedded in the handbook (look for KPI index mapping link).
This is also our primary project where you can see all of our dbt code https://gitlab.com/gitlab-data/analytics/
The addition of Dataform into our stack has completely changed the game, allowing us to maintain one source of objective truth and maximise looker (minimising the joining etc. in LookML, instead pointing it straight at tables in BigQuery).
This in turn means our downstream data users are happier to use a fast and reliable looker instance!
Any affiliation with DataForm?
Without design sessions to figure out your data store design (look up Kimball, Immon), and then monitoring/testing to make sure everything is running smoothly, any data stack will be hell.
Badly designed data brings fatigue and rancor and unnecessary work and is a huge hard-to-quantify money suck.
Everyhting goes through S3 because Snowflake storage is on it.
dbt is amazing, we began using it a month ago and it already transformed the way our data team work. It really is a value multiplier for everyone. Data engineers are happier because they don't need to write and maintain data transformations, analysts are happier because they can maintain their own SQL pipelines & the whole company is happier because we now have a great documentation tool to explore our data.
We also are big fans of Snowflake, make operating a data warehouse a breeze.
Then, we use a mix of Redash & Tableau for reporting.
Redash for static reporting (open to the whole company) & Tableau to create more complex data tools we expose to some internal teams; Marketing, Risk, Finance ...
Some spark mixed in at the ingestion and transformation phases.
Like someone else said in this thread, we're currently battling Redshift scaling problems and are looking to offload more of the raw data to S3 using Spark to create read views.
No data catalog right now but the Looker setup is extremely well-maintained. Hoping to deploy Lyft's Amundsen to fix this more generally.
That said I haven't looked closely at how far we could go with Looker, I'll check it out.
I would love to hear the features that you would need in a data catalog system.
Agreed there's more to be done in terms of integrations. Our integrations are pretty focussed on databases and people (https://github.com/lyft/amundsen#supported-connectors). Work on indexing dashboards is being scoped, you can see a rough product spec with a rough mock here: https://docs.google.com/document/d/16cSKgM2sCYvhKq54yfwaHKws...
Feedback welcome!