> The entire essay is about stopping development until it can be made more safe, with specific ideas on how to do that. No, the essay does not talk about "stopping development". That's a very clever sleight of hand made…
> The idea that open-weight models are more efficient seems unfounded. https://artificialanalysis.ai/ intelligence vs cost per task disagrees with this statement.
> You will typically find your staff and higher level engineers rarely are actually writing code themselves. Many so called staff+ engineers are basically glorified Google Docs engineers, or management who want to call…
> Communication matters most when you're dealing with cross-org concerns and those that master it are usually the more friendly and pleasant ones. I don't agree with the second one, but agree with the first. Throughout…
> Thanks for that! It is worth noting that taking advantage of the post-rotation distribution I again feel this claim is too strong. Rotations have been used in information theory/wireless communications for decades at…
Wow, yes - you are completely correct (read through the note in detail now). Though, as your paper also notes, the quantizer values themselves aren't fundamentally novel to either paper. Lloyd Max scalar quantizers have…
There are also more papers on similar themes. For example, TurboQuant makes use of QJL (quantized Johnson Lindenstrauss transformations). One of the first papers to characterize the QJL and in fact the rate distortion…
TurboQuant is known across the industry to not be state of the art. There are superior schemes for KV quant at every bitrate. Eg, SpectralQuant: https://github.com/Dynamis-Labs/spectralquant among many, many papers. >…
The bigger challenge is GPU/NPU. Branches for fast vs accurate path get costlier, among other things. On CPU this is less of a cost. Most published libm on GPU/NPU side have a few ULP of error for the perf vs accuracy…
> That's why you need to put your scope The problem is, "scope" is often equated to "how many people worked in my empire" rather than "how much business value did my work X generate". The two things are vastly…
>80%-90% or so of real life vectorization can be achieved in C or C++ just by writing code in a way that it can be autovectorized. Yep. I was pleasantly surprised by the autovectorization quality with recent clang at…
Welcome to the brave new world these days: 1 - Very few people conduct "proper scholarship", and fail to trace ideas back to their original inception and cite them correctly. This happens time and again in deep…
> I don't think there are many (or any) upsides to the well documented downsides. C++ template metaprogramming still remains extremely powerful. Projects like CUTLASS, etc could not be written to give best performance…
As others have pointed out, these phenomena are well known to many folks across companies in the AI infra space. It doesn't really break new ground. This article is a good exposition of the basic strategies though. What…
I guess you have never worked with a slow induction cooktop. Literally we had to spend 15 minutes more for cooking things on induction compared with our previous apartment's gas connection. Maybe they are better now but…
+1 - there are just so many Asian recipes that can not be done anywhere near as easily on induction stovetops (high heat from direct flame for flatbreads, etc). Plus a whole bunch of cookware doesn't work with induction…
There is a vast number of sysctl in xnu that have not really been re-examined in over 15 years. Many tunings date back to the spinning rust drive era (for example). There are plenty of examples like this. Disclaimer: I…
The big problem is a bunch of folks actually take these things seriously and use it as an excuse to freeze the junior hiring pipeline. At the senior levels this is not actually believed by the powers that be, since a…
Same. The compensation is substantially better at FAANG, but in terms of actual on the ground work being rewarded, almost never the case. Meta-work (lots of "cross functional" documents, alignment meetings, sync ups…
This is much more nuanced now. See Apple "Private Cloud Compute": https://security.apple.com/blog/private-cloud-compute/ ; they run a lot of the larger models on their own servers. Fundamentally it is more efficient to…
Maybe on a particular model/dataset but extremely unlikely in general. Again, like another commenter pointed out: if you truly believe it isn't that hard we would love to hire you at Meta ;)
Our group works on some of this stuff at Meta, and we have a pretty good diversity of backgrounds - high performance computing (the bulk), computer systems, compilers, ML engineers, etc. We are hiring. Feel free to DM…
lmkd (low memory killer daemon) works fairly differently off of a different set of signals and different policy. But yes, conceptually they try to achieve the same goal. I also do not know if Android combines system…
A couple of additional points on how the "low-RAM" works: 1 - https://www.lifewire.com/understanding-compressed-memory-os-... : Apple devices have support for memory compression, see…
Umm, Apple still sells devices with just 1 GB of RAM on them ;)
> The entire essay is about stopping development until it can be made more safe, with specific ideas on how to do that. No, the essay does not talk about "stopping development". That's a very clever sleight of hand made…
> The idea that open-weight models are more efficient seems unfounded. https://artificialanalysis.ai/ intelligence vs cost per task disagrees with this statement.
> You will typically find your staff and higher level engineers rarely are actually writing code themselves. Many so called staff+ engineers are basically glorified Google Docs engineers, or management who want to call…
> Communication matters most when you're dealing with cross-org concerns and those that master it are usually the more friendly and pleasant ones. I don't agree with the second one, but agree with the first. Throughout…
> Thanks for that! It is worth noting that taking advantage of the post-rotation distribution I again feel this claim is too strong. Rotations have been used in information theory/wireless communications for decades at…
Wow, yes - you are completely correct (read through the note in detail now). Though, as your paper also notes, the quantizer values themselves aren't fundamentally novel to either paper. Lloyd Max scalar quantizers have…
There are also more papers on similar themes. For example, TurboQuant makes use of QJL (quantized Johnson Lindenstrauss transformations). One of the first papers to characterize the QJL and in fact the rate distortion…
TurboQuant is known across the industry to not be state of the art. There are superior schemes for KV quant at every bitrate. Eg, SpectralQuant: https://github.com/Dynamis-Labs/spectralquant among many, many papers. >…
The bigger challenge is GPU/NPU. Branches for fast vs accurate path get costlier, among other things. On CPU this is less of a cost. Most published libm on GPU/NPU side have a few ULP of error for the perf vs accuracy…
> That's why you need to put your scope The problem is, "scope" is often equated to "how many people worked in my empire" rather than "how much business value did my work X generate". The two things are vastly…
>80%-90% or so of real life vectorization can be achieved in C or C++ just by writing code in a way that it can be autovectorized. Yep. I was pleasantly surprised by the autovectorization quality with recent clang at…
Welcome to the brave new world these days: 1 - Very few people conduct "proper scholarship", and fail to trace ideas back to their original inception and cite them correctly. This happens time and again in deep…
> I don't think there are many (or any) upsides to the well documented downsides. C++ template metaprogramming still remains extremely powerful. Projects like CUTLASS, etc could not be written to give best performance…
As others have pointed out, these phenomena are well known to many folks across companies in the AI infra space. It doesn't really break new ground. This article is a good exposition of the basic strategies though. What…
I guess you have never worked with a slow induction cooktop. Literally we had to spend 15 minutes more for cooking things on induction compared with our previous apartment's gas connection. Maybe they are better now but…
+1 - there are just so many Asian recipes that can not be done anywhere near as easily on induction stovetops (high heat from direct flame for flatbreads, etc). Plus a whole bunch of cookware doesn't work with induction…
There is a vast number of sysctl in xnu that have not really been re-examined in over 15 years. Many tunings date back to the spinning rust drive era (for example). There are plenty of examples like this. Disclaimer: I…
The big problem is a bunch of folks actually take these things seriously and use it as an excuse to freeze the junior hiring pipeline. At the senior levels this is not actually believed by the powers that be, since a…
Same. The compensation is substantially better at FAANG, but in terms of actual on the ground work being rewarded, almost never the case. Meta-work (lots of "cross functional" documents, alignment meetings, sync ups…
This is much more nuanced now. See Apple "Private Cloud Compute": https://security.apple.com/blog/private-cloud-compute/ ; they run a lot of the larger models on their own servers. Fundamentally it is more efficient to…
Maybe on a particular model/dataset but extremely unlikely in general. Again, like another commenter pointed out: if you truly believe it isn't that hard we would love to hire you at Meta ;)
Our group works on some of this stuff at Meta, and we have a pretty good diversity of backgrounds - high performance computing (the bulk), computer systems, compilers, ML engineers, etc. We are hiring. Feel free to DM…
lmkd (low memory killer daemon) works fairly differently off of a different set of signals and different policy. But yes, conceptually they try to achieve the same goal. I also do not know if Android combines system…
A couple of additional points on how the "low-RAM" works: 1 - https://www.lifewire.com/understanding-compressed-memory-os-... : Apple devices have support for memory compression, see…
Umm, Apple still sells devices with just 1 GB of RAM on them ;)