It does! See the effect strength plot near the bottom of the blog post, or Figures 2b and 3d in the paper. The effect strength on humans ranges from a few percent deviation of human judgements from chance for subtle…
This comment is incorrect. For experiments 1 through 4, N was 38, 389, 396, and 389. The subjects were not undergrad psych students. The article linked in the parent comment does not correspond to any experiment in the…
I can't reply directly -- is there a maximum thread depth, or a maximum conversation depth? Anyway -- I wanted to apologize for misreading -- I missed the parenthetical "interpolation" in your comment. I think we are…
(the plot shows extreme overfitting with a 10 parameter model, and interpolation with a 10,000 parameter model)
It's true that I don't go into detail about double descent, though I do describe how increasing capacity often reduces overfitting. I believe the figure labeled "Figure 1" illustrates what your are suggesting (despite…
Thanks! I'm also adding this to my reading list now.
Blog post author here. A brief note that I do discuss the deep double descent phenomenon in the blog. See the section starting with "One of the best understood causes of extreme overfitting is that the expressivity of…
See response to alexmolas -- I'm using GitHub Pages + Jekyll + Markdeep. You don't see the source repo because it's private, but I'm happy to share a code snapshot with you if you like -- email me for it.
Yes! In the post I talk about both under- and over-parameterization being mitigations for overfitting, though I don't use the term double descent.
+1. It doesn't require there to be a time axis -- but in practice, we almost always optimize incrementally, so it takes a while for the strong version of Goodhart's law to kick in.
Yup, it's GitHub pages. It's just a private repo, so the world doesn't get to see my embarrassing edits and half written drafts. I would be happy to share a snapshot of the source code with you if you have a specific…
+1 to this comment! The barrier between (thinking we) know what changes should happen, and realizing those changes in the real world, is complex and frustrating and political.
A gentle note that an incomplete piece of a goal (e.g. a loss function computed on a subset of the data) is a proxy for the full goal (e.g. the loss function on the full dataset). Similarly, concept drift can be a…
A note that the datapoints you train on are part of the training objective. If you are using different data at test time than you use at training time, then you are measuring the wrong thing during training, the same as…
(blog post author here) Yes! The post actually talks about that a bit: how overfitting can result from treating "leaders that have the most support in the population" as a proxy for "leaders that act in the best…
Blog post author here. Just for the record, I am very much in favor of metric-driven decision making. I suggest some ways we can make it more robust, and also that we should be aware that our metrics may not be…
Author of blog post here! Overfitting can happen in many ways -- your training objective can be different at train and test time, or as you suggest the datapoints you use can be different at train and test time. For…
They show the source images next to the super-resolution output on the website for the iterative refinement paper: https://iterative-refinement.github.io/
Don't forget Neural Tangents, a high level library for building and running experiments with infinite width neural networks: https://github.com/google/neural-tangents
It does! See the effect strength plot near the bottom of the blog post, or Figures 2b and 3d in the paper. The effect strength on humans ranges from a few percent deviation of human judgements from chance for subtle…
This comment is incorrect. For experiments 1 through 4, N was 38, 389, 396, and 389. The subjects were not undergrad psych students. The article linked in the parent comment does not correspond to any experiment in the…
I can't reply directly -- is there a maximum thread depth, or a maximum conversation depth? Anyway -- I wanted to apologize for misreading -- I missed the parenthetical "interpolation" in your comment. I think we are…
(the plot shows extreme overfitting with a 10 parameter model, and interpolation with a 10,000 parameter model)
It's true that I don't go into detail about double descent, though I do describe how increasing capacity often reduces overfitting. I believe the figure labeled "Figure 1" illustrates what your are suggesting (despite…
Thanks! I'm also adding this to my reading list now.
Blog post author here. A brief note that I do discuss the deep double descent phenomenon in the blog. See the section starting with "One of the best understood causes of extreme overfitting is that the expressivity of…
See response to alexmolas -- I'm using GitHub Pages + Jekyll + Markdeep. You don't see the source repo because it's private, but I'm happy to share a code snapshot with you if you like -- email me for it.
Yes! In the post I talk about both under- and over-parameterization being mitigations for overfitting, though I don't use the term double descent.
+1. It doesn't require there to be a time axis -- but in practice, we almost always optimize incrementally, so it takes a while for the strong version of Goodhart's law to kick in.
Yup, it's GitHub pages. It's just a private repo, so the world doesn't get to see my embarrassing edits and half written drafts. I would be happy to share a snapshot of the source code with you if you have a specific…
+1 to this comment! The barrier between (thinking we) know what changes should happen, and realizing those changes in the real world, is complex and frustrating and political.
A gentle note that an incomplete piece of a goal (e.g. a loss function computed on a subset of the data) is a proxy for the full goal (e.g. the loss function on the full dataset). Similarly, concept drift can be a…
A note that the datapoints you train on are part of the training objective. If you are using different data at test time than you use at training time, then you are measuring the wrong thing during training, the same as…
(blog post author here) Yes! The post actually talks about that a bit: how overfitting can result from treating "leaders that have the most support in the population" as a proxy for "leaders that act in the best…
Blog post author here. Just for the record, I am very much in favor of metric-driven decision making. I suggest some ways we can make it more robust, and also that we should be aware that our metrics may not be…
Author of blog post here! Overfitting can happen in many ways -- your training objective can be different at train and test time, or as you suggest the datapoints you use can be different at train and test time. For…
They show the source images next to the super-resolution output on the website for the iterative refinement paper: https://iterative-refinement.github.io/
Don't forget Neural Tangents, a high level library for building and running experiments with infinite width neural networks: https://github.com/google/neural-tangents