> Communication matters most when you're dealing with cross-org concerns and those that master it are usually the more friendly and pleasant ones. I don't agree with the second one, but agree with the first. Throughout…
> Thanks for that! It is worth noting that taking advantage of the post-rotation distribution I again feel this claim is too strong. Rotations have been used in information theory/wireless communications for decades at…
Wow, yes - you are completely correct (read through the note in detail now). Though, as your paper also notes, the quantizer values themselves aren't fundamentally novel to either paper. Lloyd Max scalar quantizers have…
There are also more papers on similar themes. For example, TurboQuant makes use of QJL (quantized Johnson Lindenstrauss transformations). One of the first papers to characterize the QJL and in fact the rate distortion…
TurboQuant is known across the industry to not be state of the art. There are superior schemes for KV quant at every bitrate. Eg, SpectralQuant: https://github.com/Dynamis-Labs/spectralquant among many, many papers. >…
The bigger challenge is GPU/NPU. Branches for fast vs accurate path get costlier, among other things. On CPU this is less of a cost. Most published libm on GPU/NPU side have a few ULP of error for the perf vs accuracy…
> That's why you need to put your scope The problem is, "scope" is often equated to "how many people worked in my empire" rather than "how much business value did my work X generate". The two things are vastly…
>80%-90% or so of real life vectorization can be achieved in C or C++ just by writing code in a way that it can be autovectorized. Yep. I was pleasantly surprised by the autovectorization quality with recent clang at…
Welcome to the brave new world these days: 1 - Very few people conduct "proper scholarship", and fail to trace ideas back to their original inception and cite them correctly. This happens time and again in deep…
> I don't think there are many (or any) upsides to the well documented downsides. C++ template metaprogramming still remains extremely powerful. Projects like CUTLASS, etc could not be written to give best performance…
As others have pointed out, these phenomena are well known to many folks across companies in the AI infra space. It doesn't really break new ground. This article is a good exposition of the basic strategies though. What…
I guess you have never worked with a slow induction cooktop. Literally we had to spend 15 minutes more for cooking things on induction compared with our previous apartment's gas connection. Maybe they are better now but…
+1 - there are just so many Asian recipes that can not be done anywhere near as easily on induction stovetops (high heat from direct flame for flatbreads, etc). Plus a whole bunch of cookware doesn't work with induction…
There is a vast number of sysctl in xnu that have not really been re-examined in over 15 years. Many tunings date back to the spinning rust drive era (for example). There are plenty of examples like this. Disclaimer: I…
The big problem is a bunch of folks actually take these things seriously and use it as an excuse to freeze the junior hiring pipeline. At the senior levels this is not actually believed by the powers that be, since a…
> Large corporations believe anyone is replaceable. This is definitely true. By design, large corporations are structured so that there is no single point of failure. > Again I am an IC & don’t see/hear any extra work…
Same. The compensation is substantially better at FAANG, but in terms of actual on the ground work being rewarded, almost never the case. Meta-work (lots of "cross functional" documents, alignment meetings, sync ups…
This is much more nuanced now. See Apple "Private Cloud Compute": https://security.apple.com/blog/private-cloud-compute/ ; they run a lot of the larger models on their own servers. Fundamentally it is more efficient to…
Maybe on a particular model/dataset but extremely unlikely in general. Again, like another commenter pointed out: if you truly believe it isn't that hard we would love to hire you at Meta ;)
Our group works on some of this stuff at Meta, and we have a pretty good diversity of backgrounds - high performance computing (the bulk), computer systems, compilers, ML engineers, etc. We are hiring. Feel free to DM…
lmkd (low memory killer daemon) works fairly differently off of a different set of signals and different policy. But yes, conceptually they try to achieve the same goal. I also do not know if Android combines system…
A couple of additional points on how the "low-RAM" works: 1 - https://www.lifewire.com/understanding-compressed-memory-os-... : Apple devices have support for memory compression, see…
Umm, Apple still sells devices with just 1 GB of RAM on them ;)
Same, doing it once out of my own curiosity to see how the corporate machine works. Not doing it again - seeing first hand how it is due to managerial incompetence more than anything else. The "reward" ratio is just not…
> based on "impact" rather than arbitrary metrics Umm, from whatever I have seen in big tech "impact" is also fairly arbitrary. It all is based on how cozy one is with one's manager, skip manager, and so on. More…
> Communication matters most when you're dealing with cross-org concerns and those that master it are usually the more friendly and pleasant ones. I don't agree with the second one, but agree with the first. Throughout…
> Thanks for that! It is worth noting that taking advantage of the post-rotation distribution I again feel this claim is too strong. Rotations have been used in information theory/wireless communications for decades at…
Wow, yes - you are completely correct (read through the note in detail now). Though, as your paper also notes, the quantizer values themselves aren't fundamentally novel to either paper. Lloyd Max scalar quantizers have…
There are also more papers on similar themes. For example, TurboQuant makes use of QJL (quantized Johnson Lindenstrauss transformations). One of the first papers to characterize the QJL and in fact the rate distortion…
TurboQuant is known across the industry to not be state of the art. There are superior schemes for KV quant at every bitrate. Eg, SpectralQuant: https://github.com/Dynamis-Labs/spectralquant among many, many papers. >…
The bigger challenge is GPU/NPU. Branches for fast vs accurate path get costlier, among other things. On CPU this is less of a cost. Most published libm on GPU/NPU side have a few ULP of error for the perf vs accuracy…
> That's why you need to put your scope The problem is, "scope" is often equated to "how many people worked in my empire" rather than "how much business value did my work X generate". The two things are vastly…
>80%-90% or so of real life vectorization can be achieved in C or C++ just by writing code in a way that it can be autovectorized. Yep. I was pleasantly surprised by the autovectorization quality with recent clang at…
Welcome to the brave new world these days: 1 - Very few people conduct "proper scholarship", and fail to trace ideas back to their original inception and cite them correctly. This happens time and again in deep…
> I don't think there are many (or any) upsides to the well documented downsides. C++ template metaprogramming still remains extremely powerful. Projects like CUTLASS, etc could not be written to give best performance…
As others have pointed out, these phenomena are well known to many folks across companies in the AI infra space. It doesn't really break new ground. This article is a good exposition of the basic strategies though. What…
I guess you have never worked with a slow induction cooktop. Literally we had to spend 15 minutes more for cooking things on induction compared with our previous apartment's gas connection. Maybe they are better now but…
+1 - there are just so many Asian recipes that can not be done anywhere near as easily on induction stovetops (high heat from direct flame for flatbreads, etc). Plus a whole bunch of cookware doesn't work with induction…
There is a vast number of sysctl in xnu that have not really been re-examined in over 15 years. Many tunings date back to the spinning rust drive era (for example). There are plenty of examples like this. Disclaimer: I…
The big problem is a bunch of folks actually take these things seriously and use it as an excuse to freeze the junior hiring pipeline. At the senior levels this is not actually believed by the powers that be, since a…
> Large corporations believe anyone is replaceable. This is definitely true. By design, large corporations are structured so that there is no single point of failure. > Again I am an IC & don’t see/hear any extra work…
Same. The compensation is substantially better at FAANG, but in terms of actual on the ground work being rewarded, almost never the case. Meta-work (lots of "cross functional" documents, alignment meetings, sync ups…
This is much more nuanced now. See Apple "Private Cloud Compute": https://security.apple.com/blog/private-cloud-compute/ ; they run a lot of the larger models on their own servers. Fundamentally it is more efficient to…
Maybe on a particular model/dataset but extremely unlikely in general. Again, like another commenter pointed out: if you truly believe it isn't that hard we would love to hire you at Meta ;)
Our group works on some of this stuff at Meta, and we have a pretty good diversity of backgrounds - high performance computing (the bulk), computer systems, compilers, ML engineers, etc. We are hiring. Feel free to DM…
lmkd (low memory killer daemon) works fairly differently off of a different set of signals and different policy. But yes, conceptually they try to achieve the same goal. I also do not know if Android combines system…
A couple of additional points on how the "low-RAM" works: 1 - https://www.lifewire.com/understanding-compressed-memory-os-... : Apple devices have support for memory compression, see…
Umm, Apple still sells devices with just 1 GB of RAM on them ;)
Same, doing it once out of my own curiosity to see how the corporate machine works. Not doing it again - seeing first hand how it is due to managerial incompetence more than anything else. The "reward" ratio is just not…
> based on "impact" rather than arbitrary metrics Umm, from whatever I have seen in big tech "impact" is also fairly arbitrary. It all is based on how cozy one is with one's manager, skip manager, and so on. More…