The premise of this post is that tech founders used to be admirable nerds, but have since changed. I wonder if it isn't the other way around. We're the nerds. Us. Here. We used to admire tech founders because sometimes…
Elementwise equality! Given two dataframe columns or ndarrays, users often expect `==` to give out a column or ndarrays of bools (like `+`, ``, `*, `&`, and just about every other binary operator).
Personally I think it's fine to use "diffusion" to refer to the whole family of models
Yah I think E2E-TTT is a lot more like what people in this comments section are picturing. I can't tell that this method updates model weights at all during the "sleep" period, only the usual SSM state updated by any…
The abstract and method sections only mention updating the SSM state during "sleep" (ie the same vectors that change after each token in stock Mamba) not any of the actual weight matrices. AFAICT this is just another…
Under the hood, the StartWRT UI is just another OpenWRT package, and it plays nicely with luci.
I helped a bit to develop this UI myself. Support for vlans was baked into it from day 1. The idea being good admin/guest/iot/hosted/etc separation without extra access points.
My understanding is that humans have very limited genetic diversity compared to most other animals, because of the population bottlenecks we've been through. And further, that diversity is mostly between individuals,…
I think that this is something of a misunderstanding. There isn't a litteral RISC processor inside the x86 processor with a tiny little compiler sitting in the middle. Its more that the out-of-order execution model…
I've been working on a reactivity system for rust over the past couple of years, which uses a lot of these ideas! It also tries to make random concurrent modification less of a pain, with transactional memory and CRDT…
https://ruffle.rs/ is pretty solid
I feel like my papers are better for having gone through peer review, and I'm a better researcher for having had a few rejections. Of course the reviewers can't hover around in your lab watching everything you do. But…
Several people in my family have a MTHFR gene mutation that screws stuff up, including causing problems with anxiety+depression. But a simple B12 shot every couple of weeks does wonders.
Yes. Pretraining and fine-tuning use standard Adam optimizers (usually with weight-decay). Reinforcement learning has been the odd-man out historically, but these days almost all RL algorithms also use backprop and…
I'm skeptical that we'll see a big breakthrough in the architecture itself. As sick as we all are of transformers, they are really good universal approximators. You can get some marginal gains, but how more _universal_…
In a library of all possible strings, this is just text compression (as the other comment observes). But in a finite library it gets even simpler, in a cool way! We can treat each text as a unique symbol and use an…
My old team shipped a web port of our 3D modeling software back in 2017. The entire engine is the same as the desktop app, written in C++, and compiled to wasm. Wasm is not now and will never be a magic "press here to…
The simulatable stuff is almost perfect. It's the stuff that can't be simulated that fails. Take the last flight as an example. The booster experienced what was (probably) a structural failure in the propellant fuel…
And remember, optimization problems can be _incredibly_ high-dimensional. A 7B parameter LLM is a 7-billion-dimensional optimization landscape. A grid-search with a resolution of 10 (ie 10 samples for each dimension)…
This also tracks with my experience. Of course, technical progress never looks smooth through the steep part of the s-curve, more a sequence of jagged stair-steps (each their own little s-curve in miniature). We might…
Except for some niche Janko-layout keyboards like the WholeTone Revolution and Lumatone, isomorphic pianos haven't every really caught on. However, isomorphic layouts are very common on accordions! I have a chromatic…
I did read it, all the way through! It's really good. The part you are quoting is setting up the ELS, which does not memorize entire images due to the inductive biases of a CNN (translation symmetry, limited receptive…
I have mixed feelings about this interpretation: that diffusion models approximately produce moseics from patches of training data. It does a good job helping people understand why diffusion models are able to work. I…
Deepness in the Sky is one of my favorite books of all time! Fire Upon the Deep is a serious let-down by comparison, but the wolves are a cool concept. As I've gotten older I've realized that I have very little in…
I have a crate with a "perfect" derive macro that generates where clauses from the fields instead of putting them on the generic parameters. It is nice when it works, but yah cyclical trait matching is still a real…
The premise of this post is that tech founders used to be admirable nerds, but have since changed. I wonder if it isn't the other way around. We're the nerds. Us. Here. We used to admire tech founders because sometimes…
Elementwise equality! Given two dataframe columns or ndarrays, users often expect `==` to give out a column or ndarrays of bools (like `+`, ``, `*, `&`, and just about every other binary operator).
Personally I think it's fine to use "diffusion" to refer to the whole family of models
Yah I think E2E-TTT is a lot more like what people in this comments section are picturing. I can't tell that this method updates model weights at all during the "sleep" period, only the usual SSM state updated by any…
The abstract and method sections only mention updating the SSM state during "sleep" (ie the same vectors that change after each token in stock Mamba) not any of the actual weight matrices. AFAICT this is just another…
Under the hood, the StartWRT UI is just another OpenWRT package, and it plays nicely with luci.
I helped a bit to develop this UI myself. Support for vlans was baked into it from day 1. The idea being good admin/guest/iot/hosted/etc separation without extra access points.
My understanding is that humans have very limited genetic diversity compared to most other animals, because of the population bottlenecks we've been through. And further, that diversity is mostly between individuals,…
I think that this is something of a misunderstanding. There isn't a litteral RISC processor inside the x86 processor with a tiny little compiler sitting in the middle. Its more that the out-of-order execution model…
I've been working on a reactivity system for rust over the past couple of years, which uses a lot of these ideas! It also tries to make random concurrent modification less of a pain, with transactional memory and CRDT…
https://ruffle.rs/ is pretty solid
I feel like my papers are better for having gone through peer review, and I'm a better researcher for having had a few rejections. Of course the reviewers can't hover around in your lab watching everything you do. But…
Several people in my family have a MTHFR gene mutation that screws stuff up, including causing problems with anxiety+depression. But a simple B12 shot every couple of weeks does wonders.
Yes. Pretraining and fine-tuning use standard Adam optimizers (usually with weight-decay). Reinforcement learning has been the odd-man out historically, but these days almost all RL algorithms also use backprop and…
I'm skeptical that we'll see a big breakthrough in the architecture itself. As sick as we all are of transformers, they are really good universal approximators. You can get some marginal gains, but how more _universal_…
In a library of all possible strings, this is just text compression (as the other comment observes). But in a finite library it gets even simpler, in a cool way! We can treat each text as a unique symbol and use an…
My old team shipped a web port of our 3D modeling software back in 2017. The entire engine is the same as the desktop app, written in C++, and compiled to wasm. Wasm is not now and will never be a magic "press here to…
The simulatable stuff is almost perfect. It's the stuff that can't be simulated that fails. Take the last flight as an example. The booster experienced what was (probably) a structural failure in the propellant fuel…
And remember, optimization problems can be _incredibly_ high-dimensional. A 7B parameter LLM is a 7-billion-dimensional optimization landscape. A grid-search with a resolution of 10 (ie 10 samples for each dimension)…
This also tracks with my experience. Of course, technical progress never looks smooth through the steep part of the s-curve, more a sequence of jagged stair-steps (each their own little s-curve in miniature). We might…
Except for some niche Janko-layout keyboards like the WholeTone Revolution and Lumatone, isomorphic pianos haven't every really caught on. However, isomorphic layouts are very common on accordions! I have a chromatic…
I did read it, all the way through! It's really good. The part you are quoting is setting up the ELS, which does not memorize entire images due to the inductive biases of a CNN (translation symmetry, limited receptive…
I have mixed feelings about this interpretation: that diffusion models approximately produce moseics from patches of training data. It does a good job helping people understand why diffusion models are able to work. I…
Deepness in the Sky is one of my favorite books of all time! Fire Upon the Deep is a serious let-down by comparison, but the wolves are a cool concept. As I've gotten older I've realized that I have very little in…
I have a crate with a "perfect" derive macro that generates where clauses from the fields instead of putting them on the generic parameters. It is nice when it works, but yah cyclical trait matching is still a real…