As a former scientist it's embarrassing how easily the dismal state of science today could have been predicted decades before by applying Goodhart's Law, or any simple train of thought on incentives and moral hazard. Instead we chose to assume scientists collectively behave on a higher plane. No wonder the general public distrusts "the intelectual elite", we deserved it.
This article is about MD doctors, not scientists (although there may be some MD-PhDs in it, and yes MDs can get research positions).
MD doctors poll at extraordinarily high levels of trust, over almost any other professional group in the United States. So it really isn't correct to directly link this article's topic to "distrust". The effect you're talking about may exist in science, but this article is essentially a counter example to the effect you propose: clinicians publishing bullshit, but retain a high level of public trust.
Especially because the article is basically entirely quoting practicing scientists who identified this problem in the first place! More real scientific training or collaborating for clinicians who want to (or have to) do research could potentially improve the situation.
MD doctors (and even to a large extent DO doctors nowadays) are philosophically grounded on science. An MD fundamentally practices science. That used to be one of the key differences between MD and DO physicians, but science has been so successful at advancing the standard of care that DOs cannot ignore it anymore. That's just to say that MDs are expected to be fluent in science and it's not some arbitrary expectation of bureaucracy run amok.
If the solutions are so simple, what are they? Yes, incentives are distorted, but what system - that does not rely on everyone being on a higher plane - would have better incentives?
By your definition, every human endeavor is dismal and always has been - all are corrupted and flawed to some degree. Is there evidence that current science is more dismal than others or than before? You can look at any day in history and see people saying the same things about how it's so dismal and not like the good old days.
> No wonder the general public distrusts "the intelectual elite", we deserved it.
The general public has no idea about scientific publishing, publish or perish, or the distorted incentives it creates. Science has delivered at an incredible level for centuries, arguably more than any other human enterprise. Covid-19 vaccines were available in record time - it wasn't the science that caused it to go somewhat off the rails.
I’m currently facing a severe health condition and I can’t help but ask various LLMs about it. They all will eventually offer solutions or avenues that sound promising or ‘easy’ when I know better, that the path ahead of me is hard, however they will non-chalantly offer a path forward they insist will work. The best part is when it cites commercial websites promotional statements as facts, though it will also misinterpret medical journals if I say to restrict itself to that.
In closing, my Redfin escapism has shifted to LLM medical escapism, I know better but if you don’t or you are in even more dire straits, it provides such an illusion of hope and that’s dangerous.
I was having a conversation with someone about AI taking over medical jobs the other day, and one of the things that came out of my thoughts about it is related:
It obviously has flaws, and we should never stop trying to improve it, but I think AI can be a great way to help connect a bunch of information they have to a bunch of information they don't, or to help spot patterns and potential avenues that they happened to miss. And obviously you want to be careful about becoming over-reliant on it, or being too trusting when it can be wrong. But I think we've been at the point for a long time where a doctor using a search engine to find medical literature should be a very reasonable thing to do, and I think AI can at least be an incremental (but massive) improvement on that workflow.
But I hope the end result of that is that doctors can not only deliver better (and maybe better-informed and more open-minded) treatment, but can spend the time focusing on patient care, managing expectations / risk management around uncertainty, managing the emotions inherent in someone who may be losing their life or the lives of their loved ones. Those are things AI is definitely not well-equipped to do as you point out.
residencies have decided to outsource part of their hiring decisions to journal peer-review processes. so now for some submissions, editors and reviewers are not actually doing scientific peer review, but rather screening job candidates for hospitals.
peer review is built to assume good faith work by people who are all part of a community of scholarship, it can partially hold up to people within the community gaming metrics. if people are just going to appear, game the system to publish some papers, and then disappear into their real careers, there's no hope of this working.
i don't understand why residencies want med students to publish papers anyway. it's very difficult to do good scientific research, it requires training, time, and almost always apprenticeship. none of this is part of the medical school curriculum, which is why we need special MD-PhD programs for people who want to do both. nobody expects that doing a PhD in biology or epidemiology would give you any clinical know-how, why is it reasonable to expect the reverse?
Residency is a guild (aka, anticompetitive trust) designed to limit the medical workforce and drive up doctor salaries. It needs artificial metrics to ratonalize the increasing salaries as being for "increasing talent".
That aside I am a bit perplexed that almost absolute insistence of medical students to become researchers as well, it seems just become a pure practitioner is not a feasible option.
To make it worse it looks more acceptable that a doctor really provide bad service or lacking communication and empathy to patients than not being a researcher.
Guidelines and standards can change quickly. It's important for clinicians to have a foundational understanding of statistical methods and to be able to critique studies.
For certain specialties, the number of residency positions is so limited that medical students have to publish research to be competitive, even if they have no interest in doing so later in their career.
But altogether I sort of agree, the incentives are pretty maligned such that for many it's just easier to become a bad scientist with more publications than a good one with fewer.
Publishing a study does not show "a foundational understanding of statistical methods and to be able to critique studies." It's literally the opposite!
Clinical research is in a weird spot, where you need both clinical experience and research experience. Getting the former already requires long hours and you are swamped with work. The latter is highly age gated and if you want to pursue research you often need to achieve specific milestones before a certain age, e.g. to be considered for tenure etc.
Researcher/academics pay/promotoins should be contingent on reviewing,challenging and reproducing papers rather than publishing quantity, because publishing cartels and AI has already degraded most research fields.
Academia is almost entirely steered by government policy, at least in the Anglosphere. There is a bureaucratic not-very-scientific evaluation process for research proposals. And things like the https://en.wikipedia.org/wiki/Research_Excellence_Framework , which has been criticized for practically requiring people to game the "impact" metric in order to keep their jobs.
Reproducibility in many scientific as areas has been made almost impossible. We have got to the stage where IP matters more than scientific rigour so methodology is purposely left out.
I wouldn't say so, more that lawyers and capitalistic interests goes before scientific advancements, that doesn't preclude independant (or guided) re-discovery and eventual replication of findings.
I disagree - there's already a reproducibility crises in the literature of many fields. And medical adjacent-field like nutrition science are among the worst. I think the trends in our culture of people going against medical professionals and "doing their own research" (poorly, and from poor sources) is a net harm on society, and things like the reproducibility crisis and genuinely misleading studies described in the article only serve to harm and dilute the legitimate work done in the field and feed into that cycle. If even "research" by qualified professionals is actually allowed to be garbage by the responsible institutions, we truly are doomed.. It does no good to do discovery if you can't refine or even validate those discoveries.
I'm not entirely disagreeing, right now however there is so much fraud that when it intersects with things of interest causes millions(billions?) to be spent on chasing the wrong leads (see findings on that 2006 Amyloid Plaque paper regarding Alzheimers research).
I'm mostly saying that being reproducible should become a higher badge of quality, right now reviewers in cartels can boost a researchers credibility by accepting each others articles to papers to let them become "influential" and money is then redirected even more to bullshit research (ie pure waste).
If up to 50% of research grants is spent on bullshit research based on fraud, spending 10% by earmarking it for reproduction to weed out irreproducible fraud is money well spent.
As a clinician-academic who published in The Lancet during medical school, I think this goes a bit far. Unfortunately student doctors are encouraged to publish whether or not they actually have an interest in research… but that shouldn’t discount the work of those who are genuinely engaged.
But certainly we should always approach the literature critically, including the author list, journal of publication and its peer-review practices, and the methods.
> student doctors are encouraged to publish whether or not they actually have an interest in research… but that shouldn’t discount the work of those who are genuinely engaged.
How do you propose the interested public make the distinction between genuine engagement and forced encouragement? Isn't it the task of journals to make that distinction before publishing? I don't think you can fault the public for dismissing everything out of hand when both academia and the journals are actively turning scientific publishing into a market for lemons.
As usual HN posters are hyper aware of other's credentials while ignoring that their BS in CS (if that) doesn't magically qualify them to assess everything in every domain.
"I'm a software engineer, I'm sure if I had the time to study Neuroscience, I'd figure out what all of these researchers failed to realize all these decades! I (alone) have the magic of critical and logical thinking"
Well, that is a statement..! As an MD PhD with over 60 (co-)publications including multiple in top 1% journals I can say for sure that this is untrue. Of course this may be different per topic and country, but there is perfect research being published by non-PhD scientists. In fact, the PI from a top-tier US university I collaborate with for over 10 years doesn't even have a PhD.
Do most medical students publish useless case studies trying to jockey for residency spots and signal hustle/devotion? No doubt!
But there are a good handful of medical students who are still (surprisingly) in it for the medicine and not the money. And that handful is exceedingly capable; no reason they can’t publish valuable work with the right collaborators and resources.
> no reason they can’t publish valuable work with the right collaborators
Despite h-index claiming to balance quantity and quality, it obviously incentives quantity over quality (no single publication can increment h-index as much as churning out a few worthless publications that cite each other); med students overwhelmingly follow those incentives trying to secure better residencies
in that case, it's a question of proportion. we cannot automatically conclude that a (supposed) "good handful" doing good research makes up for "most students" doing bad research.
A med student can absolutely contribute useful work, especially with good supervision. The issue is more that inexperienced authors plus publication pressure plus easy tooling is a bad combination
a friend of a friend who did a stint in biomedical academia told me that the researchers in their field did not hold research coming from the medicine community in high regard
Knew a professor statistics from a world renowned institution. He worked in nephrology for 10-20 years and would tell many stories about the worst practices he's seen and researchers pushed him to allow.
Medicine was among the worst if not the worst according to him. Didn't really want much to do with it anymore. Basically a case of subpar statistical knowledge and bad incentives.
LLVM was a thing that demonstrated its value by working when you used it. And you can't judge a population (all non-PhD theses) by its tippety-top performers, particularly when there are poor incentives involved.
I guess it depends on who the coauthors and PI are - some academic mentors can be overly trusting and ‘hands-off.’ A lone medical student’s self published paper shouldn’t be worth much though…
In the end it is about personal integrity and idealism, no matter what the titles are.
Totally different if someone's self image is that of a researcher for benefit of humankind or if they pick the career because they want to drive a Porsche.
Since we have seen that 50%+ of findings even in medical and other natural sciences are not repruductible it's obvious that even PhD people are mostly incompetent.
Some of the examples of bad design mentioned in the article are quite shocking, if we are assuming 'medical student' is not some 18 year old rookie but a person who already has several years of university level study behind them.
Some context to this is recently the big "step 1" board exam that covered the basic science portion of medicine went from being graded to being pass/fail. Since then, all the medical students where I work have started trying to get any kind of research on their CV to stand out during their residency application. New 120 participant "conferences" where med students can present posters are popping up, more med students trying to get into the labs for rotations than ever before. A lot of it is really low quality, but for them, if they don't have any research, and someone else does, they will be ranked lower and get worse jobs. Weird effect of whatever happened to make Step 1 pass/fail
My potentially unpopular opinion: Congress should ban residency programs from using Medicare training dollars to pay for research. They should do this with the goal of speeding up the training pipeline for actual practitioners, many of whom are now required to spend a year of their residency doing research of some kind.
If medical residents, or teaching hospitals, want people to do research, they should go get funding from established research funding sources that have standards and practices for funding and monitoring research.
It's medical students, not residents, who take research years, and that's only for extremely competitive specialties.
The lack of doctors, as it has always been, is because of the shortage of residency spots.
That cannot be addressed without Congress reversing 50+ years of neoliberalism trends and political failure and refusal to invest in public services and/or a communist revolution. Good luck fixing that.
Feels like the minimum standard should be sharing the exact query/design choices and being very explicit about what biases the analysis can and cannot address
I can explain what's going on here. For context, you need to know how somebody becomes a doctor in the US.
1. You get a 4 year degree in college. You hopefully get a very good GPA. You need to do so-called pre-med classes that really don't have much to do with medical education but are known as "weed out" classes, particularly Organic Chemistry. If you don't do these in your 4 year degree, you can do a program afterwards called a post-BAC;
2. At some point you take the MCAT. You may need to take it multiple times to get a sufficient score;
3. You apply to med school with your transcripts, any relevant experience, your MCAT, a personal statement and letters of recommendation. This is an onerous process. Demand greatly exceeds supply. You will need to do an interview (if you get that far);
4. If you get accepted you will do a 4 year program that's broadly characterized as MD or DO. It's easier to get into a DO school but they have worse match rates into residency, particularly for competitive specialties. There's also the international option, particularly Caribbean schools. They have even worse match rates;
5. Now begins the US Medical License Exam ("USMLE") process to become a doctor. You take Step 1 as an M2 (second year medical student). Typically the first 2 years of med school are academic. The last 2 are mostly clinical where you do rotations in various specialties;
6. As an M4 you have to do these rotations as well as take Step 2 (of the USMLE) and do your residency applications. This is probably the most stressful part because you can end up unmatched and then you've spent $400-800k+ to not become a doctor, at least not immediately and probably not in your preferred specialty;
7. To apply for residency you apply to programs, hopefully get an interview and then submit an application for each program you're interested in. This again includes letters of recommendation (very important), transcripts, your Step 2 results (Step 1 is now pass/fail, more on this below), research, etc. Applicants rank their programs. Programs rank their applicants. A matchin algorithm compares the two and attempts to essentially place each applicant in their most preferred program. Not all specialties do this. You can also attempt to match as a couple (usually used by married people);
8. If you match you're now contractually obligated to do that program. Depending on the specialty it's going to be 3-7 years, more if you do a fellowship afterwards. You basically get paid minimum wage for that entire time. Somewhere in there you need to take Step 3 and at the end do your medical boards to be licensed to operate independently as a medical doctor.
9. If you don't match, it gets real awkward. You either scramble for an open spot (a process called the SOAP), extend medical school for a 5th year (so you don't have the stink of having failed to match, seriously) or do a research year to improve your odds next year. Note that you can match into incomplete programs (eg an intern year only program).
So, let's do the math. In a perfect world you graduate high school at 18, college at 22, get accepted immediately, graduate medical school at 26, match immediately and then complete residency at 33 (for a general surgery residency program). That's a lot of education and training. You likely have $400k=$1M in debt by this point. And only now do you earn a real income.
But it often doesn't go that way. You may fail to get into medical school the first time. You may not have realized you wanted to have been a doctor so you had to do 1-2 years of a post-BAC. So you might be 25-26 before you start medical school. You may fail to match or not try and do a research year. Or you might do an MD-PhD program and take a few extra years to graduate. Combined with a fellowship, that 33 years of age might turn into 40 years old.
So one thing that changed in the last few years is that Step 1 went from a score to pass/fail. This is ostensibly to reduce the stress of having a bad score. Some med schools are also pass/fail rather than having a class ranking. What this means in practice is that school reputation and ranking become more important. These are harder to get into obviously so it has a knock-on effect into undergrad. So if you go to Harvard undergrad, you'll generally have a better chance of going to a T20 med school. But how do you get into Harvard?
But let me bring this long-winded thing back to research. Over the past decade, the number of research items for each matched resident has massively increased, more than doubled in some cases. Some med schools are research-heavy so going to those has become a competitive advantage. It means people who successfully match into a competitive specialty are more likely to take a research year before applying. This is particularly true for neurosurgery.
Income potential and lifestyle massively vary. Primary care (family medicine) and pediatrics have awful earning potential. Any surgical specialty, dermatology (I honestly don't understand this one) and radiology have much higher earning potential. The difference can be 5x or more.
So I guess this is a really long way of saying that churning out low-quality research is resume-padding. Residency programs don't even tend to care about the quality of your research. It's just the number of research items you have. Increased competitiveness of certain programs combined with reduced signal in other areas (particularly Step 1 going pass/fail) may have exacerbated the situation.
So anyone who complains about how much doctors earn should look at the time it takes and the years of exploitation as a resident. Maybe a doctor wouldn't be so expensive if it wasn't so expensive to become a doctor. You will also find a large number of physicians who would take a big pay cut if they didn't have to deal with insurance.
do you know why residencies value the number of research items? Why would having a large number of garbage papers be seen as a positive signal at all?
Attendings and existing residents are consulted in the ranking process, they are picking people they will have to work pretty closely with for 4 years, they have skin in the game. Why does anyone put any weight on such a clearly bogus metric?
don't existing doctors aim to keep supply short? Seems like it's like the homeowner problem (existing homeowners always want their house prices to go up)
It's complicated. The AMA seeks to restrict supply. The number of medical schools has been cut dramatically from historic highs. The AMA relented in WW2 because of the need for combat physicians but then went right back to their old ways after WW2 ended. Doctors don't necessarily share that view.
But there are multiple issues that contribute to shortages. Just like with homes (as you brought up), there can be homes where nobody wants to live. Likewise, some specialties never fill all their places. So earning potential is a factor. Not wanting to live rural is often also a factor, despite efforts to attract people to both using things like PSLF. PSLF itself is on shaky ground under this administration and you will see physicians unwilling to sacrifice career potential for a program that won't trust will be there to forgive their debt.
And then there's burn out. Many doctors leave the profession in their 40s and 50s. And if you didn't really become a doctor until your 30s, that's a relatively short professional life. But why do they burn out? Insane hours, administration, insurance, work-life balance are all up there.
There is something a bit ironic that the evidence to refute the "more research is better" position is selecting (cherry-picking?) a particularly bad study.
Are there better ways to measure whether the population of studies using these databases has declined as a whole, rather than just saying there are more of them and at least some are pretty bad?
Exactly. The people at the John Hopkins Camel-Winston Center for Respiratory Research know that Camel and Winston are gonna stop cutting the checks if they don't get something they can trot out to support them when they get hauled in front of congress even if that's only a minority of the work output they funded.
Even if nobody means to do evil the evil will be done just as a result of people factoring that in subconsciously and the pressure that applies systemically.
Where is really gets spicy is when you have dueling funding sources. Where I went to school you had the environmental compliance industry funding the public policy research to say that nobody should be allowed to install a fencepost without paying their way through some hoops while the Kochs (through some indirection) were funding different research in the same department to say no akshually that compliance stuff is making us all poorer.
They're just generating observational hypotheses for future investigators to examine further and maybe test in a trial. It should be presented as an observational hypothesis.
As a former scientist it's embarrassing how easily the dismal state of science today could have been predicted decades before by applying Goodhart's Law, or any simple train of thought on incentives and moral hazard. Instead we chose to assume scientists collectively behave on a higher plane. No wonder the general public distrusts "the intelectual elite", we deserved it.
This article is about MD doctors, not scientists (although there may be some MD-PhDs in it, and yes MDs can get research positions).
MD doctors poll at extraordinarily high levels of trust, over almost any other professional group in the United States. So it really isn't correct to directly link this article's topic to "distrust". The effect you're talking about may exist in science, but this article is essentially a counter example to the effect you propose: clinicians publishing bullshit, but retain a high level of public trust.
Especially because the article is basically entirely quoting practicing scientists who identified this problem in the first place! More real scientific training or collaborating for clinicians who want to (or have to) do research could potentially improve the situation.
MD doctors (and even to a large extent DO doctors nowadays) are philosophically grounded on science. An MD fundamentally practices science. That used to be one of the key differences between MD and DO physicians, but science has been so successful at advancing the standard of care that DOs cannot ignore it anymore. That's just to say that MDs are expected to be fluent in science and it's not some arbitrary expectation of bureaucracy run amok.
Due to the high trust they get from the public, I have run into many MDs who are used to making unfounded assertions with little pushback.
Why is this? MDs are among the people I trust the least, along with lawyers, financial advisors, and real-estate agents.
Information asymmetry
"fasting is a disaster for the body" says radio MD with complete confidence.. one of my personal favorites
>Instead we chose to assume scientists collectively behave on a higher plane.
People are people.
If the solutions are so simple, what are they? Yes, incentives are distorted, but what system - that does not rely on everyone being on a higher plane - would have better incentives?
By your definition, every human endeavor is dismal and always has been - all are corrupted and flawed to some degree. Is there evidence that current science is more dismal than others or than before? You can look at any day in history and see people saying the same things about how it's so dismal and not like the good old days.
> No wonder the general public distrusts "the intelectual elite", we deserved it.
The general public has no idea about scientific publishing, publish or perish, or the distorted incentives it creates. Science has delivered at an incredible level for centuries, arguably more than any other human enterprise. Covid-19 vaccines were available in record time - it wasn't the science that caused it to go somewhat off the rails.
I’m currently facing a severe health condition and I can’t help but ask various LLMs about it. They all will eventually offer solutions or avenues that sound promising or ‘easy’ when I know better, that the path ahead of me is hard, however they will non-chalantly offer a path forward they insist will work. The best part is when it cites commercial websites promotional statements as facts, though it will also misinterpret medical journals if I say to restrict itself to that.
In closing, my Redfin escapism has shifted to LLM medical escapism, I know better but if you don’t or you are in even more dire straits, it provides such an illusion of hope and that’s dangerous.
I was having a conversation with someone about AI taking over medical jobs the other day, and one of the things that came out of my thoughts about it is related:
It obviously has flaws, and we should never stop trying to improve it, but I think AI can be a great way to help connect a bunch of information they have to a bunch of information they don't, or to help spot patterns and potential avenues that they happened to miss. And obviously you want to be careful about becoming over-reliant on it, or being too trusting when it can be wrong. But I think we've been at the point for a long time where a doctor using a search engine to find medical literature should be a very reasonable thing to do, and I think AI can at least be an incremental (but massive) improvement on that workflow.
But I hope the end result of that is that doctors can not only deliver better (and maybe better-informed and more open-minded) treatment, but can spend the time focusing on patient care, managing expectations / risk management around uncertainty, managing the emotions inherent in someone who may be losing their life or the lives of their loved ones. Those are things AI is definitely not well-equipped to do as you point out.
residencies have decided to outsource part of their hiring decisions to journal peer-review processes. so now for some submissions, editors and reviewers are not actually doing scientific peer review, but rather screening job candidates for hospitals.
peer review is built to assume good faith work by people who are all part of a community of scholarship, it can partially hold up to people within the community gaming metrics. if people are just going to appear, game the system to publish some papers, and then disappear into their real careers, there's no hope of this working.
i don't understand why residencies want med students to publish papers anyway. it's very difficult to do good scientific research, it requires training, time, and almost always apprenticeship. none of this is part of the medical school curriculum, which is why we need special MD-PhD programs for people who want to do both. nobody expects that doing a PhD in biology or epidemiology would give you any clinical know-how, why is it reasonable to expect the reverse?
Residency is a guild (aka, anticompetitive trust) designed to limit the medical workforce and drive up doctor salaries. It needs artificial metrics to ratonalize the increasing salaries as being for "increasing talent".
That aside I am a bit perplexed that almost absolute insistence of medical students to become researchers as well, it seems just become a pure practitioner is not a feasible option. To make it worse it looks more acceptable that a doctor really provide bad service or lacking communication and empathy to patients than not being a researcher.
Most doctors drop the research once they get a residency spot. It's just the selection mechanism for residency.
Guidelines and standards can change quickly. It's important for clinicians to have a foundational understanding of statistical methods and to be able to critique studies.
For certain specialties, the number of residency positions is so limited that medical students have to publish research to be competitive, even if they have no interest in doing so later in their career.
But altogether I sort of agree, the incentives are pretty maligned such that for many it's just easier to become a bad scientist with more publications than a good one with fewer.
Publishing a study does not show "a foundational understanding of statistical methods and to be able to critique studies." It's literally the opposite!
Clinical research is in a weird spot, where you need both clinical experience and research experience. Getting the former already requires long hours and you are swamped with work. The latter is highly age gated and if you want to pursue research you often need to achieve specific milestones before a certain age, e.g. to be considered for tenure etc.
Researcher/academics pay/promotoins should be contingent on reviewing,challenging and reproducing papers rather than publishing quantity, because publishing cartels and AI has already degraded most research fields.
Should be, but you've got to tell the funders that.
I think I was more referring to academia than commercial research even if there is a large intersection.
Academia is almost entirely steered by government policy, at least in the Anglosphere. There is a bureaucratic not-very-scientific evaluation process for research proposals. And things like the https://en.wikipedia.org/wiki/Research_Excellence_Framework , which has been criticized for practically requiring people to game the "impact" metric in order to keep their jobs.
Reproducibility in many scientific as areas has been made almost impossible. We have got to the stage where IP matters more than scientific rigour so methodology is purposely left out.
so... empiricism is over?
I wouldn't say so, more that lawyers and capitalistic interests goes before scientific advancements, that doesn't preclude independant (or guided) re-discovery and eventual replication of findings.
No. You can't spend all your money on rehashing past results. Some, OK, all, not. In many fields, the money is needed for discovery.
I disagree - there's already a reproducibility crises in the literature of many fields. And medical adjacent-field like nutrition science are among the worst. I think the trends in our culture of people going against medical professionals and "doing their own research" (poorly, and from poor sources) is a net harm on society, and things like the reproducibility crisis and genuinely misleading studies described in the article only serve to harm and dilute the legitimate work done in the field and feed into that cycle. If even "research" by qualified professionals is actually allowed to be garbage by the responsible institutions, we truly are doomed.. It does no good to do discovery if you can't refine or even validate those discoveries.
Discovery is quite worthless if the discovery can't be trusted enough to continue building upon.
I'm not entirely disagreeing, right now however there is so much fraud that when it intersects with things of interest causes millions(billions?) to be spent on chasing the wrong leads (see findings on that 2006 Amyloid Plaque paper regarding Alzheimers research).
I'm mostly saying that being reproducible should become a higher badge of quality, right now reviewers in cartels can boost a researchers credibility by accepting each others articles to papers to let them become "influential" and money is then redirected even more to bullshit research (ie pure waste).
If up to 50% of research grants is spent on bullshit research based on fraud, spending 10% by earmarking it for reproduction to weed out irreproducible fraud is money well spent.
My assumption is the credibility of a non-PhD-holding medical student’s research is 0, just like (almost) any other inexperienced researcher.
As a clinician-academic who published in The Lancet during medical school, I think this goes a bit far. Unfortunately student doctors are encouraged to publish whether or not they actually have an interest in research… but that shouldn’t discount the work of those who are genuinely engaged.
But certainly we should always approach the literature critically, including the author list, journal of publication and its peer-review practices, and the methods.
> student doctors are encouraged to publish whether or not they actually have an interest in research… but that shouldn’t discount the work of those who are genuinely engaged.
How do you propose the interested public make the distinction between genuine engagement and forced encouragement? Isn't it the task of journals to make that distinction before publishing? I don't think you can fault the public for dismissing everything out of hand when both academia and the journals are actively turning scientific publishing into a market for lemons.
I was severely disillusioned about the quality of clinical studies.
Would you publish if the head honcho of your double-blind study insists to know what treatment a certain patient is receiving?
You have this discussion about research ethics and subsequent beratement once, and then you either mentally check out or go to another hospital.
I think this is the right distinction
Were you first-author on an unsupervised project and paper, or did someone stake their reputation on your work?
As usual HN posters are hyper aware of other's credentials while ignoring that their BS in CS (if that) doesn't magically qualify them to assess everything in every domain.
"I'm a software engineer, I'm sure if I had the time to study Neuroscience, I'd figure out what all of these researchers failed to realize all these decades! I (alone) have the magic of critical and logical thinking"
A lot of us here have masters, PhDs, have published in academia, worked in the hard sciences or different engineerinf disciplines.
But I agree, when youre on the internet no ones knows you're a dog.
Well, that is a statement..! As an MD PhD with over 60 (co-)publications including multiple in top 1% journals I can say for sure that this is untrue. Of course this may be different per topic and country, but there is perfect research being published by non-PhD scientists. In fact, the PI from a top-tier US university I collaborate with for over 10 years doesn't even have a PhD.
Both your statement and the OPs are an appeal to expertise which is not a substitute for the scientific method.
You can be a PI without having a PhD?
I've seen this a few times- for example, my manager at Berkeley Lab was a PI but only had an undergrad degree. This is very rare, however.
Even Hassabis found time to do a PhD. This is extra strange.
This is really far too broad a brush.
Do most medical students publish useless case studies trying to jockey for residency spots and signal hustle/devotion? No doubt!
But there are a good handful of medical students who are still (surprisingly) in it for the medicine and not the money. And that handful is exceedingly capable; no reason they can’t publish valuable work with the right collaborators and resources.
> no reason they can’t publish valuable work with the right collaborators
Despite h-index claiming to balance quantity and quality, it obviously incentives quantity over quality (no single publication can increment h-index as much as churning out a few worthless publications that cite each other); med students overwhelmingly follow those incentives trying to secure better residencies
in that case, it's a question of proportion. we cannot automatically conclude that a (supposed) "good handful" doing good research makes up for "most students" doing bad research.
A med student can absolutely contribute useful work, especially with good supervision. The issue is more that inexperienced authors plus publication pressure plus easy tooling is a bad combination
a friend of a friend who did a stint in biomedical academia told me that the researchers in their field did not hold research coming from the medicine community in high regard
Knew a professor statistics from a world renowned institution. He worked in nephrology for 10-20 years and would tell many stories about the worst practices he's seen and researchers pushed him to allow.
Medicine was among the worst if not the worst according to him. Didn't really want much to do with it anymore. Basically a case of subpar statistical knowledge and bad incentives.
How can you tell when you are reading a paper?
The author list will show "M.D." instead of (or in addition to) "Ph.D.". The paper terminology will be skewed towards medical terms.
LLVM was a masters thesis project (not medicine related but research by non PhDs should not be disregarded imo)
LLVM was a thing that demonstrated its value by working when you used it. And you can't judge a population (all non-PhD theses) by its tippety-top performers, particularly when there are poor incentives involved.
I guess it depends on who the coauthors and PI are - some academic mentors can be overly trusting and ‘hands-off.’ A lone medical student’s self published paper shouldn’t be worth much though…
In the end it is about personal integrity and idealism, no matter what the titles are.
Totally different if someone's self image is that of a researcher for benefit of humankind or if they pick the career because they want to drive a Porsche.
What has PhD got to do with anything. Research is research regardless of who does it if using proper scientific method.
Such obvious common sense appears not obvious after all.
Since we have seen that 50%+ of findings even in medical and other natural sciences are not repruductible it's obvious that even PhD people are mostly incompetent.
Really grateful for people like Wang making an effort to deter this behavior.
Hoping more folks like him defend the guardrails.
ill-incentives have always influenced academia, but I’m hoping we’re able to walk it back a bit
Some of the examples of bad design mentioned in the article are quite shocking, if we are assuming 'medical student' is not some 18 year old rookie but a person who already has several years of university level study behind them.
Some context to this is recently the big "step 1" board exam that covered the basic science portion of medicine went from being graded to being pass/fail. Since then, all the medical students where I work have started trying to get any kind of research on their CV to stand out during their residency application. New 120 participant "conferences" where med students can present posters are popping up, more med students trying to get into the labs for rotations than ever before. A lot of it is really low quality, but for them, if they don't have any research, and someone else does, they will be ranked lower and get worse jobs. Weird effect of whatever happened to make Step 1 pass/fail
My potentially unpopular opinion: Congress should ban residency programs from using Medicare training dollars to pay for research. They should do this with the goal of speeding up the training pipeline for actual practitioners, many of whom are now required to spend a year of their residency doing research of some kind.
If medical residents, or teaching hospitals, want people to do research, they should go get funding from established research funding sources that have standards and practices for funding and monitoring research.
This is wrong.
It's medical students, not residents, who take research years, and that's only for extremely competitive specialties.
The lack of doctors, as it has always been, is because of the shortage of residency spots.
That cannot be addressed without Congress reversing 50+ years of neoliberalism trends and political failure and refusal to invest in public services and/or a communist revolution. Good luck fixing that.
Congress should end the AMA guild's chokehold on linking "Medicare" to medical training at all. It's a complete grift.
Feels like the minimum standard should be sharing the exact query/design choices and being very explicit about what biases the analysis can and cannot address
I can explain what's going on here. For context, you need to know how somebody becomes a doctor in the US.
1. You get a 4 year degree in college. You hopefully get a very good GPA. You need to do so-called pre-med classes that really don't have much to do with medical education but are known as "weed out" classes, particularly Organic Chemistry. If you don't do these in your 4 year degree, you can do a program afterwards called a post-BAC;
2. At some point you take the MCAT. You may need to take it multiple times to get a sufficient score;
3. You apply to med school with your transcripts, any relevant experience, your MCAT, a personal statement and letters of recommendation. This is an onerous process. Demand greatly exceeds supply. You will need to do an interview (if you get that far);
4. If you get accepted you will do a 4 year program that's broadly characterized as MD or DO. It's easier to get into a DO school but they have worse match rates into residency, particularly for competitive specialties. There's also the international option, particularly Caribbean schools. They have even worse match rates;
5. Now begins the US Medical License Exam ("USMLE") process to become a doctor. You take Step 1 as an M2 (second year medical student). Typically the first 2 years of med school are academic. The last 2 are mostly clinical where you do rotations in various specialties;
6. As an M4 you have to do these rotations as well as take Step 2 (of the USMLE) and do your residency applications. This is probably the most stressful part because you can end up unmatched and then you've spent $400-800k+ to not become a doctor, at least not immediately and probably not in your preferred specialty;
7. To apply for residency you apply to programs, hopefully get an interview and then submit an application for each program you're interested in. This again includes letters of recommendation (very important), transcripts, your Step 2 results (Step 1 is now pass/fail, more on this below), research, etc. Applicants rank their programs. Programs rank their applicants. A matchin algorithm compares the two and attempts to essentially place each applicant in their most preferred program. Not all specialties do this. You can also attempt to match as a couple (usually used by married people);
8. If you match you're now contractually obligated to do that program. Depending on the specialty it's going to be 3-7 years, more if you do a fellowship afterwards. You basically get paid minimum wage for that entire time. Somewhere in there you need to take Step 3 and at the end do your medical boards to be licensed to operate independently as a medical doctor.
9. If you don't match, it gets real awkward. You either scramble for an open spot (a process called the SOAP), extend medical school for a 5th year (so you don't have the stink of having failed to match, seriously) or do a research year to improve your odds next year. Note that you can match into incomplete programs (eg an intern year only program).
So, let's do the math. In a perfect world you graduate high school at 18, college at 22, get accepted immediately, graduate medical school at 26, match immediately and then complete residency at 33 (for a general surgery residency program). That's a lot of education and training. You likely have $400k=$1M in debt by this point. And only now do you earn a real income.
But it often doesn't go that way. You may fail to get into medical school the first time. You may not have realized you wanted to have been a doctor so you had to do 1-2 years of a post-BAC. So you might be 25-26 before you start medical school. You may fail to match or not try and do a research year. Or you might do an MD-PhD program and take a few extra years to graduate. Combined with a fellowship, that 33 years of age might turn into 40 years old.
So one thing that changed in the last few years is that Step 1 went from a score to pass/fail. This is ostensibly to reduce the stress of having a bad score. Some med schools are also pass/fail rather than having a class ranking. What this means in practice is that school reputation and ranking become more important. These are harder to get into obviously so it has a knock-on effect into undergrad. So if you go to Harvard undergrad, you'll generally have a better chance of going to a T20 med school. But how do you get into Harvard?
But let me bring this long-winded thing back to research. Over the past decade, the number of research items for each matched resident has massively increased, more than doubled in some cases. Some med schools are research-heavy so going to those has become a competitive advantage. It means people who successfully match into a competitive specialty are more likely to take a research year before applying. This is particularly true for neurosurgery.
Income potential and lifestyle massively vary. Primary care (family medicine) and pediatrics have awful earning potential. Any surgical specialty, dermatology (I honestly don't understand this one) and radiology have much higher earning potential. The difference can be 5x or more.
So I guess this is a really long way of saying that churning out low-quality research is resume-padding. Residency programs don't even tend to care about the quality of your research. It's just the number of research items you have. Increased competitiveness of certain programs combined with reduced signal in other areas (particularly Step 1 going pass/fail) may have exacerbated the situation.
So anyone who complains about how much doctors earn should look at the time it takes and the years of exploitation as a resident. Maybe a doctor wouldn't be so expensive if it wasn't so expensive to become a doctor. You will also find a large number of physicians who would take a big pay cut if they didn't have to deal with insurance.
do you know why residencies value the number of research items? Why would having a large number of garbage papers be seen as a positive signal at all?
Attendings and existing residents are consulted in the ranking process, they are picking people they will have to work pretty closely with for 4 years, they have skin in the game. Why does anyone put any weight on such a clearly bogus metric?
don't existing doctors aim to keep supply short? Seems like it's like the homeowner problem (existing homeowners always want their house prices to go up)
It's complicated. The AMA seeks to restrict supply. The number of medical schools has been cut dramatically from historic highs. The AMA relented in WW2 because of the need for combat physicians but then went right back to their old ways after WW2 ended. Doctors don't necessarily share that view.
But there are multiple issues that contribute to shortages. Just like with homes (as you brought up), there can be homes where nobody wants to live. Likewise, some specialties never fill all their places. So earning potential is a factor. Not wanting to live rural is often also a factor, despite efforts to attract people to both using things like PSLF. PSLF itself is on shaky ground under this administration and you will see physicians unwilling to sacrifice career potential for a program that won't trust will be there to forgive their debt.
And then there's burn out. Many doctors leave the profession in their 40s and 50s. And if you didn't really become a doctor until your 30s, that's a relatively short professional life. But why do they burn out? Insane hours, administration, insurance, work-life balance are all up there.
> dermatology (I honestly don't understand this one)
Botox and other cosmetic procedures. In any big city you can find swanky dermatology practices offering expensive cosmetic procedures to rich people.
Why can't we cutoff their tool outright? How is this even allowed to happen?
There is something a bit ironic that the evidence to refute the "more research is better" position is selecting (cherry-picking?) a particularly bad study. Are there better ways to measure whether the population of studies using these databases has declined as a whole, rather than just saying there are more of them and at least some are pretty bad?
admissions and residency matching give a lot of weight to "research output", aka publications.
For residency, the two most important things are: 1) board scores. 2) research output.
It's not uncommon to see 40-50 publications for competitive residencies.
incentives, incentives, incentives.
>incentives, incentives, incentives.
Exactly. The people at the John Hopkins Camel-Winston Center for Respiratory Research know that Camel and Winston are gonna stop cutting the checks if they don't get something they can trot out to support them when they get hauled in front of congress even if that's only a minority of the work output they funded.
Even if nobody means to do evil the evil will be done just as a result of people factoring that in subconsciously and the pressure that applies systemically.
Where is really gets spicy is when you have dueling funding sources. Where I went to school you had the environmental compliance industry funding the public policy research to say that nobody should be allowed to install a fencepost without paying their way through some hoops while the Kochs (through some indirection) were funding different research in the same department to say no akshually that compliance stuff is making us all poorer.
AI BS sourced from even more BS
They're just generating observational hypotheses for future investigators to examine further and maybe test in a trial. It should be presented as an observational hypothesis.
90% biomedicine papers are bullshit. These students are just practicing bullshit.
90% of statistics on the internet are made up anyway
"Don't believe random quotes on the internet"
- Albert Einstein
Pretty much extends to 90% of all papers are bullshit. And with all the new bullshit generators we have now, that will probably increase.