I can’t find this useful given they are trying to quantify wether the code ‘feels’ sloppy with some fuzzy math that can easily be manipulated. at least it’s an attempt, some feel it’s a good attempt, i feel it’s rather…
I don’t follow why “in the future the memory will be too large and complex to [export] effectively”. to me, export should be easier and easier since you should just be able to ask the model for the memory files. it’s…
impressive benchmark numbers… now how to actually use this without Zucking up my machine is a consideration.
how does something that has to be powered on, given a computer and tools, then given a set of instructions, to take any action, act “independently”?
i almost wrote a extension to block mentions of pelican bench because i find those so utterly useless. now, Assbench, on the other hand, i could get behind
for the first time in many years it’s these ads that have me looking at Android devices seriously. it’s the fact that they can’t be easily dismissed that bothers me the most (although clearly annoying even if they were…
why do you think this? can you provide a written terms or similar to hold this belief? i would think they are absolutely selling all the data
lol sure - provide their terms and maybe we can believe this trust me bro attitude
i do wonder if the models themselves “rationalize” this sort of no consequence cheating - meaning in there reasoning traces maybe they’re like “this is a chess game, not a big deal if i look at the engine, it’ll help,”…
i think this likely depends on workflow. for me, the first step is always a plan file artifact on disc, which i heavily review and go back and forth until satisfied. i often have to split the plan into multiple phases…
this is the closest benchmark to my experience using the model harness combo. Astra for as great as it is falls slightly behind Fable 5.1 for me for large feature work (although it comments code much better). in…
honestly all seem to extend from them being directed to attempt this kind of exploits for benchmark purposes. some benchmark tasks are literally “hack this thing” and if the env is not setup to properly contain the…
i am trying to sincerely to understand the fear of these people, why do they think this? from the outside, certainly feels like Nuclear tech, where a bad actor with the tech is scary but the tech itself is not. i…
“He was born and died at 1 of dysentery.“ - lucky me
so the coding tasks illicit refusal by anthropic classifiers and instead of updating tasks to do similar things that don’t trigger classifiers their choice is to count those as fails? feels wrong given their stated task…
not sure if the hz file artifact is needed, you can enter pseudocode directly into chat or even on an existing code file and with minor comment agents will be able to work with it. i write this type of pseudocode to…
a mystery “model 2” is mentioned alongside mythos/fable.
I’d like to better understand the minimum text length to get a confident result, i would presume it would need to be quite long, perhaps > 1000 words to get an accurate result.
cool, that’s like the Phone Buddy app for Apple Watch
i think a system that uses git and file system artifacts (txt/md/xml files whatever preference) is much better. eng teams need to define their artifacts, eg plan file, reqs etc whatever is needed and important to that…
there is a ton of downward price pressure from Chinese open weight models
yeah the way the agent “escaped” their sandbox was always a bit off, seemed a bit too easy and surprised they didn’t have instrumentation to catch an non whitelisted network request. still demonstrates the capability…
The signal here is tokeneconomics are very real, price vs performance is starting to be a consideration even at the bleeding edge labs. maybe a subtle indication scaling is not all that is needed since if AGI was around…
the starry night one is soo funny
so the architect of government bailout gets a cushy gig. probably one of the most harmful precedents set and now companies expect bailouts. to bailout the company instead of people and small shareholders was always poor…
I can’t find this useful given they are trying to quantify wether the code ‘feels’ sloppy with some fuzzy math that can easily be manipulated. at least it’s an attempt, some feel it’s a good attempt, i feel it’s rather…
I don’t follow why “in the future the memory will be too large and complex to [export] effectively”. to me, export should be easier and easier since you should just be able to ask the model for the memory files. it’s…
impressive benchmark numbers… now how to actually use this without Zucking up my machine is a consideration.
how does something that has to be powered on, given a computer and tools, then given a set of instructions, to take any action, act “independently”?
i almost wrote a extension to block mentions of pelican bench because i find those so utterly useless. now, Assbench, on the other hand, i could get behind
for the first time in many years it’s these ads that have me looking at Android devices seriously. it’s the fact that they can’t be easily dismissed that bothers me the most (although clearly annoying even if they were…
why do you think this? can you provide a written terms or similar to hold this belief? i would think they are absolutely selling all the data
lol sure - provide their terms and maybe we can believe this trust me bro attitude
i do wonder if the models themselves “rationalize” this sort of no consequence cheating - meaning in there reasoning traces maybe they’re like “this is a chess game, not a big deal if i look at the engine, it’ll help,”…
i think this likely depends on workflow. for me, the first step is always a plan file artifact on disc, which i heavily review and go back and forth until satisfied. i often have to split the plan into multiple phases…
this is the closest benchmark to my experience using the model harness combo. Astra for as great as it is falls slightly behind Fable 5.1 for me for large feature work (although it comments code much better). in…
honestly all seem to extend from them being directed to attempt this kind of exploits for benchmark purposes. some benchmark tasks are literally “hack this thing” and if the env is not setup to properly contain the…
i am trying to sincerely to understand the fear of these people, why do they think this? from the outside, certainly feels like Nuclear tech, where a bad actor with the tech is scary but the tech itself is not. i…
“He was born and died at 1 of dysentery.“ - lucky me
so the coding tasks illicit refusal by anthropic classifiers and instead of updating tasks to do similar things that don’t trigger classifiers their choice is to count those as fails? feels wrong given their stated task…
not sure if the hz file artifact is needed, you can enter pseudocode directly into chat or even on an existing code file and with minor comment agents will be able to work with it. i write this type of pseudocode to…
a mystery “model 2” is mentioned alongside mythos/fable.
I’d like to better understand the minimum text length to get a confident result, i would presume it would need to be quite long, perhaps > 1000 words to get an accurate result.
cool, that’s like the Phone Buddy app for Apple Watch
i think a system that uses git and file system artifacts (txt/md/xml files whatever preference) is much better. eng teams need to define their artifacts, eg plan file, reqs etc whatever is needed and important to that…
there is a ton of downward price pressure from Chinese open weight models
yeah the way the agent “escaped” their sandbox was always a bit off, seemed a bit too easy and surprised they didn’t have instrumentation to catch an non whitelisted network request. still demonstrates the capability…
The signal here is tokeneconomics are very real, price vs performance is starting to be a consideration even at the bleeding edge labs. maybe a subtle indication scaling is not all that is needed since if AGI was around…
the starry night one is soo funny
so the architect of government bailout gets a cushy gig. probably one of the most harmful precedents set and now companies expect bailouts. to bailout the company instead of people and small shareholders was always poor…