I am very happy to provide any information I missed to include -- but exactly what you ask for is written in the paper's acknowledgements (no sponsorship, but some model runs).
Infinitly far away. These questions are about "have you understood and can you apply existing research" not about "create new research". For humans, these two correlate quite strongly. So we ask PhD students to work on…
You see yourself that your own solution is purely of theoretical nature and not at all what you wrote before, right? (And no, I am not commenting on your answer.)
Please indicate which other models you would like to see included. (And I agree that the context window limitations were not reasonable to have.) Finally: running this few prompts would have been $10-20k if I would have…
Well, you are correct that one should distinguish the two. But we give no indication that the questions are hard because of computational tasks and we give many indications that the problems are of theorecical nature…
I cannot keep answering everyone's comments of the type "Why did you consider / not consider?" or "Here are much better ideas". I promise you that we have thought quite a bit about the setup and have discussed it with…
When you write "there's a notable difference between performing a literature search versus solving something de novo", you suggest that the questions we provided can be solved doing a literature search. This is…
I don't understand why you expect that an answer known to the researcher but which has never been published should be in the training data. You possibly missunderstand what these problems look like -- we made them all…
You are 100% correct with your assessment of the situation. But I do not agree with either of your conclusions: 1. These questions cannot and must not be compared as being similar to homework questions. These are…
No, they only provided large-scale model runs for us (this is explained in the ackonowledgements). These runs would have been too expensive to perform myself, so I am happy they offered to provide them.
I think you are underestimating the complexity of such problems. A PhD in the exact field of research would need days to weeks to understand what the problem means and how to solve it. This is far beyond "throwing…
Let me also add: there is zero chance of the problems being included in the training data. The results are quite impressive: leading experts struggled to write questions with well-defined unique answers on existing…
the difference between gpt and gemini concerning the "retry until..." can almost be ignored. I did rerun gpt a few times, but still way below what gemini was not able to answer at all.
But it still remains far away from mathematics research. Solving any of the problems would not result in a new research paper.
I am the leader of the study and the author of the benchmark paper: let me add: the problems are much harder than any exam question in any exam. Think of it as: a PhD student studying exactly this area of mathematics…
I am very happy to provide any information I missed to include -- but exactly what you ask for is written in the paper's acknowledgements (no sponsorship, but some model runs).
Infinitly far away. These questions are about "have you understood and can you apply existing research" not about "create new research". For humans, these two correlate quite strongly. So we ask PhD students to work on…
You see yourself that your own solution is purely of theoretical nature and not at all what you wrote before, right? (And no, I am not commenting on your answer.)
Please indicate which other models you would like to see included. (And I agree that the context window limitations were not reasonable to have.) Finally: running this few prompts would have been $10-20k if I would have…
Well, you are correct that one should distinguish the two. But we give no indication that the questions are hard because of computational tasks and we give many indications that the problems are of theorecical nature…
I cannot keep answering everyone's comments of the type "Why did you consider / not consider?" or "Here are much better ideas". I promise you that we have thought quite a bit about the setup and have discussed it with…
When you write "there's a notable difference between performing a literature search versus solving something de novo", you suggest that the questions we provided can be solved doing a literature search. This is…
I don't understand why you expect that an answer known to the researcher but which has never been published should be in the training data. You possibly missunderstand what these problems look like -- we made them all…
You are 100% correct with your assessment of the situation. But I do not agree with either of your conclusions: 1. These questions cannot and must not be compared as being similar to homework questions. These are…
No, they only provided large-scale model runs for us (this is explained in the ackonowledgements). These runs would have been too expensive to perform myself, so I am happy they offered to provide them.
I think you are underestimating the complexity of such problems. A PhD in the exact field of research would need days to weeks to understand what the problem means and how to solve it. This is far beyond "throwing…
Let me also add: there is zero chance of the problems being included in the training data. The results are quite impressive: leading experts struggled to write questions with well-defined unique answers on existing…
the difference between gpt and gemini concerning the "retry until..." can almost be ignored. I did rerun gpt a few times, but still way below what gemini was not able to answer at all.
But it still remains far away from mathematics research. Solving any of the problems would not result in a new research paper.
I am the leader of the study and the author of the benchmark paper: let me add: the problems are much harder than any exam question in any exam. Think of it as: a PhD student studying exactly this area of mathematics…