Looking into setting up a Lakehouse/Datalake over the coming quarters. I'd say the biggest appeal to us is that Iceberg can handle schema drift/evolution, and a little more open.
I definitely think that some kind of semi-structured storage is the future. "Here's a giant heap of files" was always kinda a hack for when you outscaled your RDBMS but didn't have time to build something better.
Generation ->
Ingestion ->
Transform (and possible looping back as derived data is created) ->
Resting place ->
Final Useful Product
Not saying that's a perfect model, just something to hang the terms I use in this post on.
And then consider that in order to get from the beginning of that process to the end, there is a certain amount of "Data Cleanup" to be done, ranging from merely validating that the data is sensible to in the limit literally handing huge blobs of text and unstructured data to humans and making them input something useful into the system out of it.
My assessment of the whole data community right now (and please, by all means react to this with your own opinions, I'm curious about them) is that the entire flush of fads going back and forth right now amounts to an argument about how exactly to distribute the necessary Data Cleanup work across that pipeline. The theoretical ideal is for everything to just be super awesome at the generation phase and nothing else has to worry about it, but it was rapidly discovered that making the generation part so expensive inhibits the data from ever being generated. With clean data, downstream could do all sorts of database-y storage technologies and do all sorts of clever things with the clean data, but the data is never clean.
The natural overreaction is to flip entirely in the other direction and just get it in and push the validation as far down the pipeline as possible. Here you get the "big piles of vaguely organized files". You get more data this way because you lower the costs of generation and to some extent ingestion, but you complicate everything downstream.
It seems to me we're currently in a phase where everyone is just sort of hoping somebody else will do it, and we're flailing around a bit.
Very opinionated: Where we're going to settle in, and where you can already see the shape forming up, is that it'll be a little mix & match at each level. Do what's easiest in each level at that level, and you end up with the cheapest and most effective result across the pipeline considered as a whole, even though no individuals working in any part of it will be 100% happy. There won't be a magic solution, but if, for instance, Ingestion demands that the Generation at least be amenable to some tabular view, even if there are some escape hatches for generic JSON bits, they can start operating with sensible tools (SQL-ish like Clickhouse or something) instead of just having a pile of opaque nothingness, and then the next levels down won't be able to count on data quality or coherence 100% but you can start layering in cleanliness and coherence as you go, etc. There just isn't a magic solution that fits into bullet points cleanly.
(There's this "bronze/silver/gold" thing going on, which I think is silly because there's really not much benefit to trying to force an arbitrarily-deep and complicated pipeline into such classifications, but the idea is there.)
Or, in short, yes I expect to see more tabular data. It just won't be tabular for the same reason that relational DBs use tables. It'll be tables even fairly early just because you need some sort of handle on the data to do any sort of useful manipulation on it. If relational DBs use tables as an emphasis on tables qua tables of data, data lakes will use tables as defined handles on individual pieces of data to be able to manipulate them as opposed to pure unstructured piles of "something".
It reminds me of the 20+ year, still ongoing argument about where in the "Browser -> Server -> Backend Services (including DB)" stack the work needs to be done. There's a certain amount of work that has to be done. You've got a bajillion choices about where to do it, and it's been sloshing back and forth across the entire time the web has existed (&q...
We're currently looking into datalake implementations. Right now, we only have 1 or 2 data sources. Current thinking is reading them on the fly, combine them using pandas dataframe and query that. Anyone have experience with doing something similar?
Depends on the type of data you're processing, your business goals and the existing consumers that you need to support.
At a minimum I'd suggest planning to load the data from the data lake into an RDBMS (OLAP/columnar preferably). Then it's accessible to more than just Python scripts (BI tools, users of other languages, etc).
Depending on how much data there is, should also plan on data summarization strategies. You can either build some common rollups to ensure that consumers are all looking at the same summaries or you can let consumers build their own transform/load pipelines from the raw data lake or you can let consumers build their own transform pipelines from the data in the data warehouse (using something like dbt).
The benefits of a data lake architecture really appear when you have lots of sources, lots of disparate consumers, and lots of data, with some schema evolution & unstructured parts thrown in. If you only have 1 or 2 sources, small enough data to query raw data in Pandas, and consumers are restricted to Python scripts, then you can skip a lot of the architectural headache of building a data lake for now (just make sure to archive your raw data somewhere if you want to be able to pull it into a data lake in the future).
It can seem like overkill, but I can't recommend Snowflake enough. It's so simple to setup and manage, and JSON support makes it easy to just drop JSON line files into a table and query on the fly.
Depending on your data/query volume, it can also be very cheap.
However, this article was about table formats so maybe OP wasn't thinking about JSON. If so, looks like Snowflake only supports read operations atm. Snow Summit this week, so maybe that will change?
You can do this for small datasets and I built an app around exactly this use case [0].
As your data gets bigger (just over time even if individual days don't emit more data) you'll end up building out partitioning schemes yourself or you'll move to a system that does that for you like bigquery/snowflake/etc.
How does the concept of a table here differ from that of a standard relational table (if at all)? Is it that the table is a logical abstraction over a distributed set of files?
The underlying storage is less structured, which means that some common database optimizations (in particular row-level indexes) are unavailable in these kind of systems, but it's conceptually the same as a standard relational table.
These are effectively external tables. They likely exist on blob storage and not in any particular DB's storage system. As a result, you have a myriad of ways to interact with that data. Moreover, compute and storage resources are now decoupled.
This supports lightweight branches, and transactional commits and merges. I haven't used it—and it seems cool—but it also seems a little heavyweight to get cross-table transactions on top of these table formats (which would be my primary use case).
Project Nessie also powers Dremio's Arctic service, so you can get all the branching and benefits of Nessie with an intuitive UI to browse branches, create branches and merge branches. Also, it is a cloud managed service with a free tier.
Check out lakeFS (https://github.com/treeverse/lakeFS).
It doesn't rely on the object store for ordering and is highly influenced by Git itself (but designed to work on object stores at very large scales)
42 comments
[ 3.3 ms ] story [ 159 ms ] threadAnd then consider that in order to get from the beginning of that process to the end, there is a certain amount of "Data Cleanup" to be done, ranging from merely validating that the data is sensible to in the limit literally handing huge blobs of text and unstructured data to humans and making them input something useful into the system out of it.
My assessment of the whole data community right now (and please, by all means react to this with your own opinions, I'm curious about them) is that the entire flush of fads going back and forth right now amounts to an argument about how exactly to distribute the necessary Data Cleanup work across that pipeline. The theoretical ideal is for everything to just be super awesome at the generation phase and nothing else has to worry about it, but it was rapidly discovered that making the generation part so expensive inhibits the data from ever being generated. With clean data, downstream could do all sorts of database-y storage technologies and do all sorts of clever things with the clean data, but the data is never clean.
The natural overreaction is to flip entirely in the other direction and just get it in and push the validation as far down the pipeline as possible. Here you get the "big piles of vaguely organized files". You get more data this way because you lower the costs of generation and to some extent ingestion, but you complicate everything downstream.
It seems to me we're currently in a phase where everyone is just sort of hoping somebody else will do it, and we're flailing around a bit.
Very opinionated: Where we're going to settle in, and where you can already see the shape forming up, is that it'll be a little mix & match at each level. Do what's easiest in each level at that level, and you end up with the cheapest and most effective result across the pipeline considered as a whole, even though no individuals working in any part of it will be 100% happy. There won't be a magic solution, but if, for instance, Ingestion demands that the Generation at least be amenable to some tabular view, even if there are some escape hatches for generic JSON bits, they can start operating with sensible tools (SQL-ish like Clickhouse or something) instead of just having a pile of opaque nothingness, and then the next levels down won't be able to count on data quality or coherence 100% but you can start layering in cleanliness and coherence as you go, etc. There just isn't a magic solution that fits into bullet points cleanly.
(There's this "bronze/silver/gold" thing going on, which I think is silly because there's really not much benefit to trying to force an arbitrarily-deep and complicated pipeline into such classifications, but the idea is there.)
Or, in short, yes I expect to see more tabular data. It just won't be tabular for the same reason that relational DBs use tables. It'll be tables even fairly early just because you need some sort of handle on the data to do any sort of useful manipulation on it. If relational DBs use tables as an emphasis on tables qua tables of data, data lakes will use tables as defined handles on individual pieces of data to be able to manipulate them as opposed to pure unstructured piles of "something".
It reminds me of the 20+ year, still ongoing argument about where in the "Browser -> Server -> Backend Services (including DB)" stack the work needs to be done. There's a certain amount of work that has to be done. You've got a bajillion choices about where to do it, and it's been sloshing back and forth across the entire time the web has existed (&q...
At a minimum I'd suggest planning to load the data from the data lake into an RDBMS (OLAP/columnar preferably). Then it's accessible to more than just Python scripts (BI tools, users of other languages, etc).
Depending on how much data there is, should also plan on data summarization strategies. You can either build some common rollups to ensure that consumers are all looking at the same summaries or you can let consumers build their own transform/load pipelines from the raw data lake or you can let consumers build their own transform pipelines from the data in the data warehouse (using something like dbt).
The benefits of a data lake architecture really appear when you have lots of sources, lots of disparate consumers, and lots of data, with some schema evolution & unstructured parts thrown in. If you only have 1 or 2 sources, small enough data to query raw data in Pandas, and consumers are restricted to Python scripts, then you can skip a lot of the architectural headache of building a data lake for now (just make sure to archive your raw data somewhere if you want to be able to pull it into a data lake in the future).
Depending on your data/query volume, it can also be very cheap.
However, this article was about table formats so maybe OP wasn't thinking about JSON. If so, looks like Snowflake only supports read operations atm. Snow Summit this week, so maybe that will change?
As your data gets bigger (just over time even if individual days don't emit more data) you'll end up building out partitioning schemes yourself or you'll move to a system that does that for you like bigquery/snowflake/etc.
[0] https://github.com/multiprocessio/datastation
It's like inventing SVN for data. Soon enough git will have to be invented as well.
https://projectnessie.org/
This supports lightweight branches, and transactional commits and merges. I haven't used it—and it seems cool—but it also seems a little heavyweight to get cross-table transactions on top of these table formats (which would be my primary use case).