Well I think in the absence of convincing pieces of evidence to the contrary you might be right. You’re making an empirical statement but we have already answered it today: - we get novel, emergent properties and…
I think you make a fair point, but also remember: new paradigms don't necessarily require confusing / contradictory observations. You could have the simple idea of "what if gravity is an inertial force?" at any period…
Do you mean after? People do this!! But I think it’s a bit different. It won’t be apples to apples because the data volume I think is just so much different. Maybe there are good experiments for something like this.
I think they mean improve our understanding of physics with new theoretical results or paradigms. Like if it’s 1899, would Astra develop General and Special relativity on its own?
My job has gone from coding, plotting, writing to solving the riddle of what Opus 5 is saying and figuring out what is bullshit, what is valuable, what is a total divergence from what I asked it to do. Then eventually…
I don't think it's a stall, two ways I would believe there is a stall: - Does the epoch capability index progress show signs of plateauing? I consider this a good aggregate measure of diverse benchmarks into a single…
I’ve noticed this too but it hasn’t been obvious to me that this style of search is not a learned behavior. Tool calling is very much part of the post training phase, I would expect that these style searches just…
The Opus-human patois is unbearable
Im curious if you find this to be a parody in a bad way or simply a “the state of the art in math research right now is telling a machine to believe in itself”. I am in the latter camp…
Oh my god do not trigger me about the news reporting “points” for the DOW or the S&P, I have seriously considered calling news stations about this. Or not providing inflation adjusted values when referring to historical…
Yes! I am aware of this and they are really great. Problem is nobody will ever read that, but they WILL watch Fox News, CNBC, etc. and those outlets never even point out that there are errorbars/uncertainty. Then I hear…
Oh I’m well aware the BLS knows what they’re doing and publish error bars, they are great. The problem is news articles, no one ever reads BLS reports but they do watch Fox News, CNBC, etc. and those outlets never ever…
As a data scientist it is very frustrating to see a consistent lack of error bars on these numbers or any discussion about the magnitude of uncertainty around them by all news reports every time these numbers get…
I agree and that’s why we need and indeed have an ever evolving landscape of benchmarks > Private, refreshed test sets attack the mechanism itself, and in my view they are the only intervention that does. If the…
I don’t think that’s fair to say, there are two principle sources of data that paint a fairly consistent picture - one is scaling laws, where we found years ago that pretraining validation loss scales in an almost…
I work in AI evaluation, lots of problems and leakage is an issue as is ecological validity, but they definitely do not explain the progress we see. I think Epoch has the best analysis I’ve seen on evaluation trends;…
I find them useful bellweathers of genuinely out of domain performance and capability, regardless of their theoretical importance. What I see is that performance trends are remarkably stable both upstream (miraculous…
I think the question and definition game is interesting only inasmuch as it helps us understand ourselves (what actually explains some of the mysterious properties of our perceived consciousness) or helps guide us…
Yes, I was hoping they would actually offer some kind of partnership with the sports leagues to stick a few of their cameras in stadiums and offer live sports games, but I don't think there's nearly enough users to…
It is absolutely unreal the experience watching the ~10 minute sports games on the Vision Pro. The thing is 4 grand but that’s how much the first 4K TVs were. Hope the price can drop where this becomes a more widespread…
Of course but I think labs is a good term. Like a lot of terms like this it comes from history. The people working on AI at these places come predominantly from academia and predominantly do research. Of course now…
I don't have personal experience with this -- but if I were in the situation where my job was jeopardized by better price efficiencies, I don't know that I personally would have a problem with that. If Ukrainians can do…
Immigration having downward pressure on salaries is a common and demonstrably false misconception. Immigration adds more members to the labor pool while also increasing demand. The two, in most major studies, cancel…
I think the biggest phase transition happened roughly fall of last year with AI adoption. My entire job is now AI orchestration, and then a LOT of my time spent writing the specs/prompts, reviewing and validating the…
In my experience Data Science looks very little like it used to a few years ago, and the priority skill these days is good strong understanding of the basics and very good sense of judgement. To me, statistics is the…
Well I think in the absence of convincing pieces of evidence to the contrary you might be right. You’re making an empirical statement but we have already answered it today: - we get novel, emergent properties and…
I think you make a fair point, but also remember: new paradigms don't necessarily require confusing / contradictory observations. You could have the simple idea of "what if gravity is an inertial force?" at any period…
Do you mean after? People do this!! But I think it’s a bit different. It won’t be apples to apples because the data volume I think is just so much different. Maybe there are good experiments for something like this.
I think they mean improve our understanding of physics with new theoretical results or paradigms. Like if it’s 1899, would Astra develop General and Special relativity on its own?
My job has gone from coding, plotting, writing to solving the riddle of what Opus 5 is saying and figuring out what is bullshit, what is valuable, what is a total divergence from what I asked it to do. Then eventually…
I don't think it's a stall, two ways I would believe there is a stall: - Does the epoch capability index progress show signs of plateauing? I consider this a good aggregate measure of diverse benchmarks into a single…
I’ve noticed this too but it hasn’t been obvious to me that this style of search is not a learned behavior. Tool calling is very much part of the post training phase, I would expect that these style searches just…
The Opus-human patois is unbearable
Im curious if you find this to be a parody in a bad way or simply a “the state of the art in math research right now is telling a machine to believe in itself”. I am in the latter camp…
Oh my god do not trigger me about the news reporting “points” for the DOW or the S&P, I have seriously considered calling news stations about this. Or not providing inflation adjusted values when referring to historical…
Yes! I am aware of this and they are really great. Problem is nobody will ever read that, but they WILL watch Fox News, CNBC, etc. and those outlets never even point out that there are errorbars/uncertainty. Then I hear…
Oh I’m well aware the BLS knows what they’re doing and publish error bars, they are great. The problem is news articles, no one ever reads BLS reports but they do watch Fox News, CNBC, etc. and those outlets never ever…
As a data scientist it is very frustrating to see a consistent lack of error bars on these numbers or any discussion about the magnitude of uncertainty around them by all news reports every time these numbers get…
I agree and that’s why we need and indeed have an ever evolving landscape of benchmarks > Private, refreshed test sets attack the mechanism itself, and in my view they are the only intervention that does. If the…
I don’t think that’s fair to say, there are two principle sources of data that paint a fairly consistent picture - one is scaling laws, where we found years ago that pretraining validation loss scales in an almost…
I work in AI evaluation, lots of problems and leakage is an issue as is ecological validity, but they definitely do not explain the progress we see. I think Epoch has the best analysis I’ve seen on evaluation trends;…
I find them useful bellweathers of genuinely out of domain performance and capability, regardless of their theoretical importance. What I see is that performance trends are remarkably stable both upstream (miraculous…
I think the question and definition game is interesting only inasmuch as it helps us understand ourselves (what actually explains some of the mysterious properties of our perceived consciousness) or helps guide us…
Yes, I was hoping they would actually offer some kind of partnership with the sports leagues to stick a few of their cameras in stadiums and offer live sports games, but I don't think there's nearly enough users to…
It is absolutely unreal the experience watching the ~10 minute sports games on the Vision Pro. The thing is 4 grand but that’s how much the first 4K TVs were. Hope the price can drop where this becomes a more widespread…
Of course but I think labs is a good term. Like a lot of terms like this it comes from history. The people working on AI at these places come predominantly from academia and predominantly do research. Of course now…
I don't have personal experience with this -- but if I were in the situation where my job was jeopardized by better price efficiencies, I don't know that I personally would have a problem with that. If Ukrainians can do…
Immigration having downward pressure on salaries is a common and demonstrably false misconception. Immigration adds more members to the labor pool while also increasing demand. The two, in most major studies, cancel…
I think the biggest phase transition happened roughly fall of last year with AI adoption. My entire job is now AI orchestration, and then a LOT of my time spent writing the specs/prompts, reviewing and validating the…
In my experience Data Science looks very little like it used to a few years ago, and the priority skill these days is good strong understanding of the basics and very good sense of judgement. To me, statistics is the…