They're just generating observational hypotheses for future investigators to examine further and maybe test in a trial. It should be presented as an observational hypothesis.
Feels like the minimum standard should be sharing the exact query/design choices and being very explicit about what biases the analysis can and cannot address
Exactly. The people at the John Hopkins Camel-Winston Center for Respiratory Research know that Camel and Winston are gonna stop cutting the checks if they don't get something they can trot out to support them when they get hauled in front of congress even if that's only a minority of the work output they funded.
Even if nobody means to do evil the evil will be done just as a result of people factoring that in subconsciously and the pressure that applies systemically.
Where is really gets spicy is when you have dueling funding sources. Where I went to school you had the environmental compliance industry funding the public policy research to say that nobody should be allowed to install a fencepost without paying their way through some hoops while the Kochs (through some indirection) were funding different research in the same department to say no akshually that compliance stuff is making us all poorer.
>Where is really gets spicy is when you have dueling funding sources. Where I went to school you had the environmental compliance industry funding the public policy research to say that nobody should be allowed to install a fencepost without paying their way through some hoops while the Kochs (through some indirection) were funding different research in the same department to say no akshually that compliance stuff is making us all poorer.
Researcher/academics pay/promotoins should be contingent on reviewing,challenging and reproducing papers rather than publishing quantity, because publishing cartels and AI has already degraded most research fields.
As a former scientist it's embarrassing how easily the dismal state of science today could have been predicted decades before by applying Goodhart's Law, or any simple train of thought on incentives and moral hazard. Instead we chose to assume scientists collectively behave on a higher plane. No wonder the general public distrusts "the intelectual elite", we deserved it.
I can explain what's going on here. For context, you need to know how somebody becomes a doctor in the US.
1. You get a 4 year degree in college. You hopefully get a very good GPA. You need to do so-called pre-med classes that really don't have much to do with medical education but are known as "weed out" classes, particularly Organic Chemistry. If you don't do these in your 4 year degree, you can do a program afterwards called a post-BAC;
2. At some point you take the MCAT. You may need to take it multiple times to get a sufficient score;
3. You apply to med school with your transcripts, any relevant experience, your MCAT, a personal statement and letters of recommendation. This is an onerous process. Demand greatly exceeds supply. You will need to do an interview (if you get that far);
4. If you get accepted you will do a 4 year program that's broadly characterized as MD or DO. It's easier to get into a DO school but they have worse match rates into residency, particularly for competitive specialties. There's also the international option, particularly Caribbean schools. They have even worse match rates;
5. Now begins the US Medical License Exam ("USMLE") process to become a doctor. You take Step 1 as an M2 (second year medical student). Typically the first 2 years of med school are academic. The last 2 are mostly clinical where you do rotations in various specialties;
6. As an M4 you have to do these rotations as well as take Step 2 (of the USMLE) and do your residency applications. This is probably the most stressful part because you can end up unmatched and then you've spent $400-800k+ to not become a doctor, at least not immediately and probably not in your preferred specialty;
7. To apply for residency you apply to programs, hopefully get an interview and then submit an application for each program you're interested in. This again includes letters of recommendation (very important), transcripts, your Step 2 results (Step 1 is now pass/fail, more on this below), research, etc. Applicants rank their programs. Programs rank their applicants. A matchin algorithm compares the two and attempts to essentially place each applicant in their most preferred program. Not all specialties do this. You can also attempt to match as a couple (usually used by married people);
8. If you match you're now contractually obligated to do that program. Depending on the specialty it's going to be 3-7 years, more if you do a fellowship afterwards. You basically get paid minimum wage for that entire time. Somewhere in there you need to take Step 3 and at the end do your medical boards to be licensed to operate independently as a medical doctor.
9. If you don't match, it gets real awkward. You either scramble for an open spot (a process called the SOAP), extend medical school for a 5th year (so you don't have the stink of having failed to match, seriously) or do a research year to improve your odds next year. Note that you can match into incomplete programs (eg an intern year only program).
So, let's do the math. In a perfect world you graduate high school at 18, college at 22, get accepted immediately, graduate medical school at 26, match immediately and then complete residency at 33 (for a general surgery residency program). That's a lot of education and training. You likely have $400k=$1M in debt by this point. And only now do you earn a real income.
But it often doesn't go that way. You may fail to get into medical school the first time. You may not have realized you wanted to have been a doctor so you had to do 1-2 years of a post-BAC. So you might be 25-26 before you start medical school. You may fail to match or not try and do a research year. Or you might do an MD-PhD program and take a few extra years to graduate. Combined with a fellowship, that 33 years of age might turn into 40 years old.
So one thing that changed in the last few years is that Step 1 went from a score to pass/fail. This is ostensibly ...
Some of the examples of bad design mentioned in the article are quite shocking, if we are assuming 'medical student' is not some 18 year old rookie but a person who already has several years of university level study behind them.
That aside I am a bit perplexed that almost absolute insistence of medical students to become researchers as well, it seems just become a pure practitioner is not a feasible option.
To make it worse it looks more acceptable that a doctor really provide bad service or lacking communication and empathy to patients than not being a researcher.
My potentially unpopular opinion: Congress should ban residency programs from using Medicare training dollars to pay for research. They should do this with the goal of speeding up the training pipeline for actual practitioners, many of whom are now required to spend a year of their residency doing research of some kind.
If medical residents, or teaching hospitals, want people to do research, they should go get funding from established research funding sources that have standards and practices for funding and monitoring research.
residencies have decided to outsource part of their hiring decisions to journal peer-review processes. so now for some submissions, editors and reviewers are not actually doing scientific peer review, but rather screening job candidates for hospitals.
peer review is built to assume good faith work by people who are all part of a community of scholarship, it can partially hold up to people within the community gaming metrics. if people are just going to appear, game the system to publish some papers, and then disappear into their real careers, there's no hope of this working.
i don't understand why residencies want med students to publish papers anyway. it's very difficult to do good scientific research, it requires training, time, and almost always apprenticeship. none of this is part of the medical school curriculum, which is why we need special MD-PhD programs for people who want to do both. nobody expects that doing a PhD in biology or epidemiology would give you any clinical know-how, why is it reasonable to expect the reverse?
There is something a bit ironic that the evidence to refute the "more research is better" position is selecting (cherry-picking?) a particularly bad study.
Are there better ways to measure whether the population of studies using these databases has declined as a whole, rather than just saying there are more of them and at least some are pretty bad?
Some context to this is recently the big "step 1" board exam that covered the basic science portion of medicine went from being graded to being pass/fail. Since then, all the medical students where I work have started trying to get any kind of research on their CV to stand out during their residency application. New 120 participant "conferences" where med students can present posters are popping up, more med students trying to get into the labs for rotations than ever before. A lot of it is really low quality, but for them, if they don't have any research, and someone else does, they will be ranked lower and get worse jobs. Weird effect of whatever happened to make Step 1 pass/fail
I’m currently facing a severe health condition and I can’t help but ask various LLMs about it. They all will eventually offer solutions or avenues that sound promising or ‘easy’ when I know better, that the path ahead of me is hard, however they will non-chalantly offer a path forward they insist will work. The best part is when it cites commercial websites promotional statements as facts, though it will also misinterpret medical journals if I say to restrict itself to that.
In closing, my Redfin escapism has shifted to LLM medical escapism, I know better but if you don’t or you are in even more dire straits, it provides such an illusion of hope and that’s dangerous.
31 comments
[ 2.9 ms ] story [ 50.3 ms ] threadSuch obvious common sense appears not obvious after all.
For residency, the two most important things are: 1) board scores. 2) research output.
It's not uncommon to see 40-50 publications for competitive residencies.
incentives, incentives, incentives.
Exactly. The people at the John Hopkins Camel-Winston Center for Respiratory Research know that Camel and Winston are gonna stop cutting the checks if they don't get something they can trot out to support them when they get hauled in front of congress even if that's only a minority of the work output they funded.
Even if nobody means to do evil the evil will be done just as a result of people factoring that in subconsciously and the pressure that applies systemically.
Where is really gets spicy is when you have dueling funding sources. Where I went to school you had the environmental compliance industry funding the public policy research to say that nobody should be allowed to install a fencepost without paying their way through some hoops while the Kochs (through some indirection) were funding different research in the same department to say no akshually that compliance stuff is making us all poorer.
Amazing. How was that solved in practice?
1. You get a 4 year degree in college. You hopefully get a very good GPA. You need to do so-called pre-med classes that really don't have much to do with medical education but are known as "weed out" classes, particularly Organic Chemistry. If you don't do these in your 4 year degree, you can do a program afterwards called a post-BAC;
2. At some point you take the MCAT. You may need to take it multiple times to get a sufficient score;
3. You apply to med school with your transcripts, any relevant experience, your MCAT, a personal statement and letters of recommendation. This is an onerous process. Demand greatly exceeds supply. You will need to do an interview (if you get that far);
4. If you get accepted you will do a 4 year program that's broadly characterized as MD or DO. It's easier to get into a DO school but they have worse match rates into residency, particularly for competitive specialties. There's also the international option, particularly Caribbean schools. They have even worse match rates;
5. Now begins the US Medical License Exam ("USMLE") process to become a doctor. You take Step 1 as an M2 (second year medical student). Typically the first 2 years of med school are academic. The last 2 are mostly clinical where you do rotations in various specialties;
6. As an M4 you have to do these rotations as well as take Step 2 (of the USMLE) and do your residency applications. This is probably the most stressful part because you can end up unmatched and then you've spent $400-800k+ to not become a doctor, at least not immediately and probably not in your preferred specialty;
7. To apply for residency you apply to programs, hopefully get an interview and then submit an application for each program you're interested in. This again includes letters of recommendation (very important), transcripts, your Step 2 results (Step 1 is now pass/fail, more on this below), research, etc. Applicants rank their programs. Programs rank their applicants. A matchin algorithm compares the two and attempts to essentially place each applicant in their most preferred program. Not all specialties do this. You can also attempt to match as a couple (usually used by married people);
8. If you match you're now contractually obligated to do that program. Depending on the specialty it's going to be 3-7 years, more if you do a fellowship afterwards. You basically get paid minimum wage for that entire time. Somewhere in there you need to take Step 3 and at the end do your medical boards to be licensed to operate independently as a medical doctor.
9. If you don't match, it gets real awkward. You either scramble for an open spot (a process called the SOAP), extend medical school for a 5th year (so you don't have the stink of having failed to match, seriously) or do a research year to improve your odds next year. Note that you can match into incomplete programs (eg an intern year only program).
So, let's do the math. In a perfect world you graduate high school at 18, college at 22, get accepted immediately, graduate medical school at 26, match immediately and then complete residency at 33 (for a general surgery residency program). That's a lot of education and training. You likely have $400k=$1M in debt by this point. And only now do you earn a real income.
But it often doesn't go that way. You may fail to get into medical school the first time. You may not have realized you wanted to have been a doctor so you had to do 1-2 years of a post-BAC. So you might be 25-26 before you start medical school. You may fail to match or not try and do a research year. Or you might do an MD-PhD program and take a few extra years to graduate. Combined with a fellowship, that 33 years of age might turn into 40 years old.
So one thing that changed in the last few years is that Step 1 went from a score to pass/fail. This is ostensibly ...
This is pretty uniquely a US-Canada specific system, that you first do a different 4-year BSc degree, before entering a 4-year medical school.
In most countries, you go a 6-year medical school right after high school.
Hoping more folks like him defend the guardrails.
ill-incentives have always influenced academia, but I’m hoping we’re able to walk it back a bit
If medical residents, or teaching hospitals, want people to do research, they should go get funding from established research funding sources that have standards and practices for funding and monitoring research.
peer review is built to assume good faith work by people who are all part of a community of scholarship, it can partially hold up to people within the community gaming metrics. if people are just going to appear, game the system to publish some papers, and then disappear into their real careers, there's no hope of this working.
i don't understand why residencies want med students to publish papers anyway. it's very difficult to do good scientific research, it requires training, time, and almost always apprenticeship. none of this is part of the medical school curriculum, which is why we need special MD-PhD programs for people who want to do both. nobody expects that doing a PhD in biology or epidemiology would give you any clinical know-how, why is it reasonable to expect the reverse?
In closing, my Redfin escapism has shifted to LLM medical escapism, I know better but if you don’t or you are in even more dire straits, it provides such an illusion of hope and that’s dangerous.