I am saying this as a lifelong supporter and user of open source software: issues like this are why governments and enterprises still run on Oracle and SQL Server.
The author was able to rollback his changes, but in some industries an unplanned enterprise-wide data unavailability event means the end of your career at that firm, if you don’t have a CYA email from the vendor confirming you were good to go. That CYA email, and the throat to choke, is why Oracle does 7 and 8 figure licensing deals with enterprises selling inferior software solutions versus open source options.
It seems that Linux, through Linus’ leadership, has been able to solve this risk issue and fully displace commercial UNIX operating systems. I hope many other projects up and down the stack can have the same success.
> The author was able to rollback his changes, but in some industries an unplanned enterprise-wide data unavailability event means the end of your career at that firm
If a (major) software update cause you an outage, you shouldn’t blame the software, but insufficient testing and validation. Large companies (I worked for many) are slow to adopt new technologies precisely because they are extremely cautious and want to make sure everything was properly tested before they roll it out. That’s also why they still use Oracle and SQL Server (and HP-UX, and IBMi) - these products are working and have been working for generations of employees. The grass needs to be significantly greener for them to consider the move to the other side of their fence.
Yeah I had to wait years to really use Parquet effectively in Python code back in the 2010s because there were two main ones (Pyarrow and Fastparquet), and they were neither compatible with either other nor compatible with Spark. Parquet support is much like Javascript support in browsers. You only get to use the more advanced features when they are supported compatibly on every platform you expect them to be used.
As long as iceberg and delta lake won't support v2, adoption will be really hard. I'm working aot with parquet and wasn't even aware that there is a version 2.0.
I was quite confused when I learned that the spec technically supports metadata about whether the data is already pre-sorted by some column(s); in my eyes seemed like it would allow some non-brainer optimizations. And yet, last I checked, it looked like pretty much nothing actually uses it, and some libraries don't even read this field at all.
> Although this post might seem like a critique of Parquet, that is not my intention. I am simply documenting what I have learned and explaining the challenges maintainers of an open format face when evolving it. All the benefits and utilities that a format like Parquet has far outweigh these inconveniences.
Yes, it is a critique (or at least its user community). It's a critique that's 100% justified too.
Have we all been so conditioned by corporate training that we've lost the ability to say "hey, this sucks" when it _does_ in fact suck?
We all lose when people communicate unclearly. Here, the people holding back evolution of the format do need to be critiqued, and named, and shamed, and the author shouldn't have been so shy about doing it.
The reference implementation for Parquet is a gigantic Java library. I'm unconvinced this is a good idea.
Take the RLE encoding which switches between run-length encoding and bit-packing. The way bit-packing has been implemented is to generate 74,000 lines of Java to read/write every combination of bitwidth, endianness and value-length.
I just can't believe this is optimal except in maybe very specific CPU-only cases (e.g. Parquet-Java running on a giant cluster somewhere).
If it were just bit-packing I could easily offload a whole data page to a GPU and not care about having per-bitwidth optimised implementations, but having to switch encodings at random intervals just makes this a headache.
It would be really nice if actual design documents exist that specify why this is a good idea based on real-world data patterns.
> The reference implementation for Parquet is a gigantic Java library. I'm unconvinced this is a good idea.
I haven't though much about it, but I believe the ideal reference implementation would be a highly optimized "service like" process that you run alongside your engine using arrow to share zero copy buffers between the engine and the parquet service. Parquet predates arrow by quite a few years and java was (unfortunately) the standard for big data stuff back then, so they simply stuck with it.
> The way bit-packing has been implemented is to generate 74,000 lines of Java to read/write every combination of bitwidth, endianness and value-length
I think they did this to avoid the dynamic dispatch nature of java. If using C++ or Rust something very similar would happen, but at the compiler level which is a much saner way of doing this kind of thing.
I'd rather have this file format with an incomplete reference and confusing implementation, than not have this file format at all. Parquet was such a tremendous improvement in quality of life over the prior status quo for anyone that needs to move even moderate amounts of data between systems, or anyone who cares about correctness and bug prevention when working with even the tiniest data sets. Maybe HDF5 and ORC would have filled the niche if Parquet hadn't, but I think realistically we would just be stuck with fragile CSV/TSV.
74 KLOC for a decoder? That's ridiculous. Use invokedynamic. Yes, people more typically associate invokedynamic with interpreter implementations or whatever, but it's actually perfect for this use case. Generate the right code on demand and let the JVM cache it so that subsequent invocations are just as fast as if you'd written it by hand.
Jesus Christ this isn't 2005 anymore and people need to learn to use the real power of the JVM. It's stuff like this that sets it apart
Similarly, Apache Spark and Scala versions. Spark ran on Scala 2.12 for a long time to eventually support 2.13. To this day, no plans to support Scala 3.x. Databricks started supporting 2.13 only in May this year...
TLDR: There are two versions of the Parquet file format, but adoption of Version 2 is slow due to limited compatibility in major engines and tools. While Version 2 offers improvements (smaller file sizes, faster write/read times), these gains are modest, and ecosystem support remains fragmented. If full control over the data pipeline is possible, using Version 2 can be worthwhile; otherwise, compatibility concerns with third-party integrations may outweigh the benefits. Parquet remains dominant, and its utility far surpasses these challenges
21 comments
[ 3.3 ms ] story [ 47.2 ms ] threadThe author was able to rollback his changes, but in some industries an unplanned enterprise-wide data unavailability event means the end of your career at that firm, if you don’t have a CYA email from the vendor confirming you were good to go. That CYA email, and the throat to choke, is why Oracle does 7 and 8 figure licensing deals with enterprises selling inferior software solutions versus open source options.
It seems that Linux, through Linus’ leadership, has been able to solve this risk issue and fully displace commercial UNIX operating systems. I hope many other projects up and down the stack can have the same success.
If a (major) software update cause you an outage, you shouldn’t blame the software, but insufficient testing and validation. Large companies (I worked for many) are slow to adopt new technologies precisely because they are extremely cautious and want to make sure everything was properly tested before they roll it out. That’s also why they still use Oracle and SQL Server (and HP-UX, and IBMi) - these products are working and have been working for generations of employees. The grass needs to be significantly greener for them to consider the move to the other side of their fence.
It is a command line wrapper to generate a Pandas SF and save it as CSV (or the other way around)
First paragraph under that heading as a markdown error
Never seen any v1 in the wild.
"Simple Binary Encoding" v2 has been stuck at release candidate stage for over 5 years and has flaws that mean it'll probably never be worth adopting.
Yes, it is a critique (or at least its user community). It's a critique that's 100% justified too.
Have we all been so conditioned by corporate training that we've lost the ability to say "hey, this sucks" when it _does_ in fact suck?
We all lose when people communicate unclearly. Here, the people holding back evolution of the format do need to be critiqued, and named, and shamed, and the author shouldn't have been so shy about doing it.
Take the RLE encoding which switches between run-length encoding and bit-packing. The way bit-packing has been implemented is to generate 74,000 lines of Java to read/write every combination of bitwidth, endianness and value-length.
I just can't believe this is optimal except in maybe very specific CPU-only cases (e.g. Parquet-Java running on a giant cluster somewhere).
If it were just bit-packing I could easily offload a whole data page to a GPU and not care about having per-bitwidth optimised implementations, but having to switch encodings at random intervals just makes this a headache.
It would be really nice if actual design documents exist that specify why this is a good idea based on real-world data patterns.
I haven't though much about it, but I believe the ideal reference implementation would be a highly optimized "service like" process that you run alongside your engine using arrow to share zero copy buffers between the engine and the parquet service. Parquet predates arrow by quite a few years and java was (unfortunately) the standard for big data stuff back then, so they simply stuck with it.
> The way bit-packing has been implemented is to generate 74,000 lines of Java to read/write every combination of bitwidth, endianness and value-length
I think they did this to avoid the dynamic dispatch nature of java. If using C++ or Rust something very similar would happen, but at the compiler level which is a much saner way of doing this kind of thing.
Jesus Christ this isn't 2005 anymore and people need to learn to use the real power of the JVM. It's stuff like this that sets it apart
(slams gavel)
Parquet Court.
Parquet is amazinggggggg