I've found breathing exercises to be effective for the duration of the exercise, but I'm more interested in the possibility of training myself to adjust my respiration patterns over sustained durations. Would it be…
Kind of a backwards take, both in your interpretation of this announcement and the company in general. The point of this announcement is to draw attention to the fact that the currently hyped topic is what they have…
I can assure you that a fully deterministic and equally effective claude is possible to build. And yes, that would mean identical prompts would yield 100% identical output 100% of the time. It would still make the…
I see what you're getting at, but determinism isn't the right word either. LLMs are fundamentally deterministic -- they are pure functions which output text as a function of the input text and the network parameters[1].…
The guy writing a thumbnail pipeline isn't getting petabytes (exabytes?) of storage to cache all videos from the past week in their entirety. If this quantity of data is being stored, it's being stored deliberately and…
The other explanations here don't explain the long delay between the start of the investigation and the release of the footage. Yes, storing customer data is what we'd expect from Google and yes, the FBI can coerce…
The implication that OpenAI is a YC company in the same sense as the other listed companies is somewhere between misleading and dishonest. Even more distasteful to show founding teams for all the others, then just Sam…
Of all Schmidhuber's credit-attribution grievances, this is the one I am most sympathetic to. I think if he spent less time remarking on how other people didn't actually invent things (e.g. Hinton and backprop, LeCun…
I can only imagine what the Taiwanese can do in Arizona. Truly a synergy for the ages.
My personal experience is that the cost of enduring a negative stimulus is not simply a function of the magnitude of the negative stimulus, but rather the magnitude of the negative stimulus in relation to the magnitude…
For future reference, if you want proper python bindings for ffmpeg* you should use pyav. * To be more precise, these are bindings for the libav* libraries that underlie ffmpeg
And on the seventh day, God ended His work which He had done and began vibe coding the remainder of the human genome.
Key-based attention is not attributable to the Transformer paper. First paper I can find where keys, queries, and values are distinct matrices is https://arxiv.org/abs/1703.03906, described at the end of section 2. The…
The scientific impact of the transformer paper is large, but in my opinion the novelty is vastly overstated. The primary novelty is adapting the (already existing) dot-product attention mechanism to be multi-headed. And…
Tensorflow is a lot like IBM -- it deserves praise not because it's great in its current state, but for its contributions towards advancing the broader technological front to where it is today. Tensorflow walked so JAX…
I've seen and ignored a lot of "pytorch good, tensorflow bad" takes in my time, but this is so egregiously wrong I can't help but chime in. Facilitating graph-level optimizations has been one of the most central tenets…
I think that while researchers would agree that the empirical success of deep learning has been remarkable, they would still agree that the language used here -- "an algorithm that could really, truly learn any…
> humanity discovered an algorithm that could really, truly learn any distribution of data (or really, the underlying “rules” that produce any distribution of data) He's hand-waving around the idea presented in the…
Imagine reading a math or programming textbook where each statement was true with probability 0.95.
Is batched inference for LLMs memory bound? My understanding is that sufficiently large batched matmuls will be compute bound and flash attention has mostly removed the memory bottleneck in the attention computation. If…
Furthermore, each of those 16 channels would typically be mutibyte floats as opposed to single byte RGB channels. (speaking generally, haven't read the paper)
Apparently my innuendo has not been taken well, so I'll clarify in more straightforward terms: @abidlabs is the employer of the individual running the aforementioned twitter account.
Do you by chance have a vested interest in attesting this? A disclaimer might be appropriate if so.
So glad the AK account exists. As a researcher, I've always wanted some guy with an econ degree and a year of ML eng to recommend me papers after glancing at them for maybe 30 seconds. I am genuinely baffled that…
When I open a large pdf on arxiv (100+ MB, not uncommon for ML papers focused on hi-res image generation), there is a significant load time (10+ seconds) before anything is rendered at all other than a loading bar. Does…
I've found breathing exercises to be effective for the duration of the exercise, but I'm more interested in the possibility of training myself to adjust my respiration patterns over sustained durations. Would it be…
Kind of a backwards take, both in your interpretation of this announcement and the company in general. The point of this announcement is to draw attention to the fact that the currently hyped topic is what they have…
I can assure you that a fully deterministic and equally effective claude is possible to build. And yes, that would mean identical prompts would yield 100% identical output 100% of the time. It would still make the…
I see what you're getting at, but determinism isn't the right word either. LLMs are fundamentally deterministic -- they are pure functions which output text as a function of the input text and the network parameters[1].…
The guy writing a thumbnail pipeline isn't getting petabytes (exabytes?) of storage to cache all videos from the past week in their entirety. If this quantity of data is being stored, it's being stored deliberately and…
The other explanations here don't explain the long delay between the start of the investigation and the release of the footage. Yes, storing customer data is what we'd expect from Google and yes, the FBI can coerce…
The implication that OpenAI is a YC company in the same sense as the other listed companies is somewhere between misleading and dishonest. Even more distasteful to show founding teams for all the others, then just Sam…
Of all Schmidhuber's credit-attribution grievances, this is the one I am most sympathetic to. I think if he spent less time remarking on how other people didn't actually invent things (e.g. Hinton and backprop, LeCun…
I can only imagine what the Taiwanese can do in Arizona. Truly a synergy for the ages.
My personal experience is that the cost of enduring a negative stimulus is not simply a function of the magnitude of the negative stimulus, but rather the magnitude of the negative stimulus in relation to the magnitude…
For future reference, if you want proper python bindings for ffmpeg* you should use pyav. * To be more precise, these are bindings for the libav* libraries that underlie ffmpeg
And on the seventh day, God ended His work which He had done and began vibe coding the remainder of the human genome.
Key-based attention is not attributable to the Transformer paper. First paper I can find where keys, queries, and values are distinct matrices is https://arxiv.org/abs/1703.03906, described at the end of section 2. The…
The scientific impact of the transformer paper is large, but in my opinion the novelty is vastly overstated. The primary novelty is adapting the (already existing) dot-product attention mechanism to be multi-headed. And…
Tensorflow is a lot like IBM -- it deserves praise not because it's great in its current state, but for its contributions towards advancing the broader technological front to where it is today. Tensorflow walked so JAX…
I've seen and ignored a lot of "pytorch good, tensorflow bad" takes in my time, but this is so egregiously wrong I can't help but chime in. Facilitating graph-level optimizations has been one of the most central tenets…
I think that while researchers would agree that the empirical success of deep learning has been remarkable, they would still agree that the language used here -- "an algorithm that could really, truly learn any…
> humanity discovered an algorithm that could really, truly learn any distribution of data (or really, the underlying “rules” that produce any distribution of data) He's hand-waving around the idea presented in the…
Imagine reading a math or programming textbook where each statement was true with probability 0.95.
Is batched inference for LLMs memory bound? My understanding is that sufficiently large batched matmuls will be compute bound and flash attention has mostly removed the memory bottleneck in the attention computation. If…
Furthermore, each of those 16 channels would typically be mutibyte floats as opposed to single byte RGB channels. (speaking generally, haven't read the paper)
Apparently my innuendo has not been taken well, so I'll clarify in more straightforward terms: @abidlabs is the employer of the individual running the aforementioned twitter account.
Do you by chance have a vested interest in attesting this? A disclaimer might be appropriate if so.
So glad the AK account exists. As a researcher, I've always wanted some guy with an econ degree and a year of ML eng to recommend me papers after glancing at them for maybe 30 seconds. I am genuinely baffled that…
When I open a large pdf on arxiv (100+ MB, not uncommon for ML papers focused on hi-res image generation), there is a significant load time (10+ seconds) before anything is rendered at all other than a loading bar. Does…