> Heresy I suppose, but doesn't feel (to me) like it "specifically" has to be cons cells, as long as it's a "list" of some sort Yeah, "cons" is definitely a implementation detail. Maybe the central ideia is just…
> Homoiconicity, as I understand, is that the code is structured data that is easy to programmatically modify I think a more accurate description is that lisp code is just cons cells and cons cells is both how we write…
If safe keeper exposes the changes to the tables somehow, a type2 scd is just a windowed lag over the primary key sorted by the timestamp
[dead]
Even when the layout is friendly to simd, auto vectorization can be finicky. As a programmer, it’s really annoying to be constantly inspecting compiler output to see if the code was properly vectorized. Even if it was,…
Yeah, especially a bloomfilter which has a pretty easy formula for its false positive rate.
Don’t you think that the provider of the LLM is also a dimension on these discussions about responsibility? We often talk about the tech itself (LLM driven development) but how we access it is just as important imo.…
I think this argument only holds if you believe that LLMs are at a point where it can handle any combination of craziness that you throw at it. From my own experience working with agents is that there’s “snowball of…
Can you elaborate on this please? Do you mean that’s basically impossible for rust std to provide a default runtime that makes “everyone” (embedded on one end and web on the other) happy?
As of now I don’t think there’s an alternative. I’m not a Rust expert but the core issue to me is that “async” goes beyond just having a Futures scheduler. Async stuff usually needs network, disk, os interaction, future…
I may have missed something, but how does “sans-io” deal with CPU heavy code? For example, if there’s some heavy decoding/encoding required on the data? Does the event loop only drive the network side and the heavy part…
Agree with the other commenters that the title is a bit too dramatic. The content was well written and got the point across. I still don’t have enough experience to have a strong opinion on Rust async, but some things…
Another strategy to avoid redistribution is simply having a big enough number of partitions and assign ranges instead of single partitions. A bit more complex on the coordination side but works well in other domains…
> The reference implementation for Parquet is a gigantic Java library. I'm unconvinced this is a good idea. I haven't though much about it, but I believe the ideal reference implementation would be a highly optimized…
> Folks I know in the industry are not very happy with databricks Yeah, big companies globing up everything does not lead to a healthy ecosystem. Congrats on the founders for their the acquisition but everyone else…
I recommend anyone who works with databases to write a simple engine. It's a lot simpler than you may think and it's a great exercise. If using python, sqlglot (https://github.com/tobymao/sqlglot) let's you skip all the…
Same thing with columnar/vectorized execution. It has been known for a long time that's the "correct" way to process data for olap workflows, but only became "mainstream" in the last few years(mostly due to arrow). It's…
> Spark is "in-memory" in the sense that it isn't forced to spill results to disk between operations I see your point, but that's only true within a single stage. Any operator that requires partitioning (groupBys and…
just out of curiosity, why do you say that spark is "in-memory"? I see a lot people claiming that, including several that I've interviewed in the past few years but that's not very accurate(at least in the default…
> Most Parquet files are bulk exports from various data analysis pipelines or databases, often appearing as full snapshots rather than incremental updates I'm not really familiar of how datasets are managed by them, but…
From the announcement “As of now, we have mined 1,580 PySpark tests from the Spark codebase, among which 838 (53.0%) are successful on Sail. We have also mined 2,230 Spark SQL statements or expressions, among which…
This is a fair argument that's often brought up but I never see actual raw data backing it up. Housing is fucked in several places around the world, including a bunch of countries in Europe that don't have any tax…
Early Athena (managed prestodb by AWS) had a similar bug when measuring colunar file scans. If it touched the file, it considered the whole file instead of just the column chunks read. If I’m not mistaken, this was a…
Great series. If I'm not mistaken, there's an additional layer to the unreliable narrator part because the book is supposed to be a translation of that biography. So, when certain words are used, the reader knows that…
100% agree. mapReduce hype always seemed strange to me because it's basically the volcano paper from the 90s but with custom user defined operators instead of pre baked ones in a more traditional engine. To make…
> Heresy I suppose, but doesn't feel (to me) like it "specifically" has to be cons cells, as long as it's a "list" of some sort Yeah, "cons" is definitely a implementation detail. Maybe the central ideia is just…
> Homoiconicity, as I understand, is that the code is structured data that is easy to programmatically modify I think a more accurate description is that lisp code is just cons cells and cons cells is both how we write…
If safe keeper exposes the changes to the tables somehow, a type2 scd is just a windowed lag over the primary key sorted by the timestamp
[dead]
Even when the layout is friendly to simd, auto vectorization can be finicky. As a programmer, it’s really annoying to be constantly inspecting compiler output to see if the code was properly vectorized. Even if it was,…
Yeah, especially a bloomfilter which has a pretty easy formula for its false positive rate.
Don’t you think that the provider of the LLM is also a dimension on these discussions about responsibility? We often talk about the tech itself (LLM driven development) but how we access it is just as important imo.…
I think this argument only holds if you believe that LLMs are at a point where it can handle any combination of craziness that you throw at it. From my own experience working with agents is that there’s “snowball of…
Can you elaborate on this please? Do you mean that’s basically impossible for rust std to provide a default runtime that makes “everyone” (embedded on one end and web on the other) happy?
As of now I don’t think there’s an alternative. I’m not a Rust expert but the core issue to me is that “async” goes beyond just having a Futures scheduler. Async stuff usually needs network, disk, os interaction, future…
I may have missed something, but how does “sans-io” deal with CPU heavy code? For example, if there’s some heavy decoding/encoding required on the data? Does the event loop only drive the network side and the heavy part…
Agree with the other commenters that the title is a bit too dramatic. The content was well written and got the point across. I still don’t have enough experience to have a strong opinion on Rust async, but some things…
Another strategy to avoid redistribution is simply having a big enough number of partitions and assign ranges instead of single partitions. A bit more complex on the coordination side but works well in other domains…
> The reference implementation for Parquet is a gigantic Java library. I'm unconvinced this is a good idea. I haven't though much about it, but I believe the ideal reference implementation would be a highly optimized…
> Folks I know in the industry are not very happy with databricks Yeah, big companies globing up everything does not lead to a healthy ecosystem. Congrats on the founders for their the acquisition but everyone else…
I recommend anyone who works with databases to write a simple engine. It's a lot simpler than you may think and it's a great exercise. If using python, sqlglot (https://github.com/tobymao/sqlglot) let's you skip all the…
Same thing with columnar/vectorized execution. It has been known for a long time that's the "correct" way to process data for olap workflows, but only became "mainstream" in the last few years(mostly due to arrow). It's…
> Spark is "in-memory" in the sense that it isn't forced to spill results to disk between operations I see your point, but that's only true within a single stage. Any operator that requires partitioning (groupBys and…
just out of curiosity, why do you say that spark is "in-memory"? I see a lot people claiming that, including several that I've interviewed in the past few years but that's not very accurate(at least in the default…
> Most Parquet files are bulk exports from various data analysis pipelines or databases, often appearing as full snapshots rather than incremental updates I'm not really familiar of how datasets are managed by them, but…
From the announcement “As of now, we have mined 1,580 PySpark tests from the Spark codebase, among which 838 (53.0%) are successful on Sail. We have also mined 2,230 Spark SQL statements or expressions, among which…
This is a fair argument that's often brought up but I never see actual raw data backing it up. Housing is fucked in several places around the world, including a bunch of countries in Europe that don't have any tax…
Early Athena (managed prestodb by AWS) had a similar bug when measuring colunar file scans. If it touched the file, it considered the whole file instead of just the column chunks read. If I’m not mistaken, this was a…
Great series. If I'm not mistaken, there's an additional layer to the unreliable narrator part because the book is supposed to be a translation of that biography. So, when certain words are used, the reader knows that…
100% agree. mapReduce hype always seemed strange to me because it's basically the volcano paper from the 90s but with custom user defined operators instead of pre baked ones in a more traditional engine. To make…