One thing not mentioned: people are paid to participate in clinical trials. For example, in a vaccine clinical trial I know about, participants are paid over $1000 (in dribs and drabs over the course of the trial, as a retention mechanism.)
With tens of thousands of participants, this cost adds up.
Patient compensation is intentionally limited to prevent economic incentives from interfering with informed consent; pay people too much and they may participate in research that they would have passed on. Also, speaking from experience, patient compensation is a very small amount (5% or less) of overall cost. The vast majority goes to clinical staff costs, room costs, hospital overhead, labs etc.
That's not a vaccine trial though, right? I'm figuring those need more participants and have more retention problems, as well as less risk of informed consent problems.
> Patient compensation is intentionally limited to prevent economic incentives from interfering with informed consent; pay people too much and they may participate in research that they would have passed on.
I'm not sure I understand the logic here. How does agreement suddenly become less informed or less consensual because the participant gets paid more? Sure, if they get paid more, they might be willing to accept more risk or do something more unpleasant. But that seems fair, not immoral. Why would it be more fair to say "yes, perhaps this trial is risky, but we only want participants who don't demand to be compensated much for the risk they take"? If anything, maybe that approach leads to less informed consent by filtering out the people who actually understand the risk and would have only agreed to participant in exchange for more compensation!
Should we also be morally offended any time an employer raises wages in order to attract more workers?
I think it's still a fair question. The money is intended to be an incentive. If the trial is ethical and has a positive risk benefit ratio, there is no harm from coercion.
Okay, but what if a person sincerely does not want to participate in a medical trial? But then when he sees the dollar figure, he thinks "well, shit, I hate doing this, but I can't afford to turn that much money down."
Same reason we don't allow the purchase of kidneys, and the same reason we think it's unethical for a millionaire with a tophat and a monocle to offer two homeless guys a wad of money to the winner if they fight each other. Sure, it's an adult making a decision of their own free will- but it's obviously unethical.
Professors historically were prohibited from relationships with students because it creates the appearance (and high likelihood) of favoritism. And it probably pisses parents off a lot because the age difference seems creepy.
"Power" is a relatively new argument. These days the student has the power to utterly destroy the professor. The professor has at best the charismatic "power" equivalent to any number of smooth pickup artists occupying college dorms, of getting a temporary good time.
That's true. Good Clinical Practice guidelines say that Institutional Review Boards must evaluate compensation is not too high otherwise it constitutes "undue influence".
...but I think that's a terrible idea. I understand that, to come up with a far-fetched example, high compensation could result in some weird edge cases where, for example, a Phase I trial with healthy volunteers and a non-preventive drug candidate (e.g. an antibiotic rather than a COVID vaccine), but you're confident that drug is going to kill 99% of the volunteers so you offer $10 million to each volunteer. A totally informed volunteer might view this as something akin to a life insurance policy that pays out to the family even upon suicide. I don't personally see an ethical problem with that so long as volunteers are really well-informed, but I could see how others and the law would be uncomfortable with the ethics of such a situation.
But aside from that far-fetched situation, it seems harmful to limit compensation... especially if it's true that patient compensation is a small part of overall cost. Perhaps if every trial doubled compensation to 10% of today's overall costs, absolute overall costs would decrease because less effort has to go into patient recruitment.
I do see one real-world issue with high compensation, but it has nothing to do with ethics. It has to do with statistical interpretation. If you start dangling too much money to participate in a trial, you may change the kind of people who participate in the trial. These individuals might not be representative of the actual users of the drug. The problem is similar to running a drug trial in the US versus Japan. The US has a pretty diverse population with respect to genetics. Japan, in contrast, is almost entirely ethnic Japanese. If you were to somehow succeed in using Japanese trial data to market an acne cream in the US, the same drug that cleared up Japanese faces might result in Venezuelans having their noses fall off!
However, I'm cautiously optimistic that higher compensation could make trials more representative. In my mind, I could see how African-Americans, who tend to participate in trials at a lower rate than their Caucasian fellow humans, would, if only for the shameful history of the Tuskegee Syphilis Study, require greater compensation to take the risk of participating in a trial.
As you said, at the end of the day it's an ethical guideline that must be upheld, more than something based in economics. We do compensate participants as well as we're allowed within these guidelines!
I agree with your points, but compensation must be approved by the independent institutional review board, who is accountable to national ethical review boards and thus acts very conservatively.
I would think the worst of such edge cases should be prohibited by law anyway. A set of more subjects is presumably a superset of a set with less subjects. So any issues can be addressed by screening them better. I suppose there could be a greater problem with fraud as people claim to have issues when they don't.
It just seems unfair to subjects to me. Not to mention paternalistic. Every other actor in the process gets enriched based on (limitless it seems) market rates for their contribution, except for the subjects who take on all the risk.
One point in there - data management. It varies in cost because on the upper end, that includes support for data change requests for the life of the trial. Zillions of people involved, human error, need paper trails and sign offs to fix most things.
This is usually downstream of the CROs, or, at least contracted out. What's most interesting to me about the post is that the author doesn't mention any efficiencies gained by moving to electronic data collection. But, I also understand why it isn't mentioned -- I work for an EDC/eCOA provider, and our biggest competitor for new business isn't our competitors in the space, it is paper-based data collection.
This is incredibly important if you want to understand healthcare. Every little thing is expensive in a way that is hard to imagine before you get into the field.
I worked in clinical development and the rough estimate of phase 3 clinical trial costs (this is a few years back) was $15,000 per patient per year.
Depending on the trial design it can be less (less frequent follow up, less invasive testing) or a lot more (say doing multiple MRIs or other expensive imaging).
It’s kind of mind blowing when you think about the really big trials for medications like cholesterol. Trials are often 30,000 or more patients, in two trials for 2-3 years.
You’re roughly talking about $2B or more just for phase 3 (ignoring all the earlier trials and preclinical work).
Another huge cost is comparator products. Some trials budgets I have seen spent 100-200+ million purchasing the drugs they are trying to outperform. This is for trials with <1000 patients.
I worked at a LVAD company (think artificial heart) that was doing a trial, they had to buy dozens (hundreds?) of their competitors LVADs, at $120k a piece
There's a reason why we ended up putting our trials together essentially by trial and error! You learn a lot by writing your own clinical protocol, interacting with the FDA etc. It's an intimidating process at the outset, but if you can save $200k on a first-in-man study by not contracting a CRO, that's gonna extend your runway by a lot!
I don't completely disagree with your point, but for one's clinical regulatory strategy, perhaps "trial and error" isn't the best branding to appeal to investigators, patients, regulators or investors.
Perhaps I was being too self deprecating! We worked closely with the FDA through multiple QSub meetings to make sure it was up to snuff, and did a lot of legwork on the backend to make sure we understood what they were asking for. 3 or 4 revisions later, we had an approval!
When we started putting the trial together, we were still self funded. In medical device, to get to series A you need human data, so by doing the trial protocol ourselves, we avoid the chicken and egg problem of needing money in order to raise money. Once you get human data, your valuation skyrockets.
Great summary! And yet, for all their problems, CROs are sometimes still the fastest alternative. We did a clinical trial for a software platform a few years ago, and hoped to partner with a prestigious hospital system to run the trial.
The hospital partner was 100% on board—our protocol just had to pass their internal review board (IRB). IRB is a black box, and they could only give us a vague estimate of how long this would take. We had a huge team, and every week we sat around waiting cost about $200k.
Finally the client couldn't take the uncertainty and burn rate anymore, and they reached out to a CRO they had used in the past. The CRO's IRB had a fixed timeline (2 weeks?), and we were onboarding patients a short time later.
Yeah the lesson with getting IRB approval if you want it done in less than 6 months is to always use an external IRB (if possible). Institutional IRBs are sloooooowwww.
We did keep working on the platform, but from the perspective of the client, we already had the MVP for the trial. So to them we were working on “nice to haves” for 200k/wk
Any particular issues? I am most opposed to the rigid 30-day cycles and the need to spend one just to get on the agenda. But in terms of the questions asked and the due diligence required, I have not found our IRBs to be onerous
While there are lots of new biotech tools, most of the illnesses left to treat are caused by aging, when the human system falls apart from the interaction of old parts, therefore super hard to treat.
At this point I gave up waiting for a medical solution for my chronic illness, and have a better hope that I can live 15-20 years with it and rejuvenate my body after that time period.
This is so, so interesting. Thanks so much for posting this!
The author already touched upon a few hypotheses why the industry has arrived at this state. There's one I'd like to highlight: The author, now having gone through this experience, would actually be in a perfect position to found their own CRO and implement all their efficiency learnings there. However, they probably look back upon this experience and think it was super painful and would prefer to do fun stuff like developing software instead.
That, for me, is the main reason. There are smart, structured, pragmatic people who got acquainted with this industry, but they choose to leave it.
And that's sad!
So the solution would be to find a group of smart, structured, pragmatic people with a high tolerance for bureaucratic pain and give them the resources to make this industry more efficient.
And here's my plug: That's exactly what we're doing at OpenRegulatory [1], albeit for medical device software and not for clinical trials. Among other things, we've published all our document templates for free (also on GitHub), host a free Slack community for companies to help each other out, and offer a compliance SaaS with transparent and fair pricing. It's still a long way, but it's a start :)
Your company is very interesting. I briefly looked over the website, but couldn't tell what country(ies?) you operate in? My assumption is the regulatory requirements of each country are dramatically different, so seeing the pricing in not $ makes me think it's unlikely you offer services in the US, for instance.
As jtwaleson already correctly replied (thanks!), we're based in Germany and are mostly focused on EU regulations. The "good" news is that if you gain regulatory approval within the EU you can bring your device to market in any EU country.
There's quite an overlap with US (FDA) regulations but we're less experienced there.
Most startups (at least in my bubble?) focus on one of those two areas.
How many industries could these sentences descrive, stripped of context?
Or more easily, are there any industries where this is not the norm?
> an industry which is severely lacking in human capital and fully captured by bureaucratic tendencies
> going through this process is an extended exercise in sheer absurdity.
> meaningless bloat
> a farcical “data-driven” scoring process
> there was an intermediary clinical organization we attempted to work with which was so obviously attempting to grift money from our coffers, the entire relationship escalated into the level of back-and-forth legal threats
> One cannot help but get the impression that this is an industry which is, at best, severely lacking in anything that could be called “human capital” and, in reality, simply deeply unserious
> despite the very large amount of boilerplate content that every single actor in this ecosystem has spent thousands of hours of manpower developing, there seem to have been very few actual gains made in simplifying common tasks or processes!
... Et cetera.
To me, this sounds a lot like processes in education, teaching, energy, environmental protection, agriculture, research, arms, media, law, IT, etc.
I don't believe we can afford to remain this stupid, however much money the inefficiencies are making for those at the top.
Interesting article, but the lack of mention of 'patent' or 'intellectual property' sort of stands out. There may be many out-of-patent drugs that are effective treatments for various conditions, but typically clinical trials for alternative uses of out-of-patent drugs are not pursued as anyone could then manufacture and sell them.
This is not new - it dates back to the origins of the modern for-profit pharmaceutical industry, such as with Bayer's discovery of the anti-bacterial action of sulfanilimide (patented 1909) in the early 1930s (by which time patent had expired). Bayer made a derivative of sulfanilimide, i.e. Prontosil (patented in 1932) and tried to convince the world that this was the more effective product. In reality, it just broke down to sulfanilimide once ingested, that being the active species.
As far as clinical trials in the USA, there have been numerous scandals regarding lack of consent, failure to halt trials immediately on indication of severe side effects, etc. Major stories about this came out in 2005, 2008, etc. Then there's the world of unregulated third-world clincal trial testing, and actual falsification of clinical trial data by contractors hired to run trials for various Big Pharma outfits, such as this from 2022.
> "Tellus, which has worked with big-name pharma sponsors in the past like Pfizer, Gilead, Takeda, Boehringer Ingelheim and Eli Lilly, initially came under scrutiny for two trials related to opioid dependency, two for irritable bowel syndrome, and one for diabetes."
That's just organized crime 101: hire a contractor to do the dirty work, keep your own hands clean, claim ignorance, act shocked, promise reforms, etc.
Best solution I've heard of is a move to open-source (patent-free) drug discovery with federally financed clinical trials. Since almost all new drugs are discovered at universities with federal funds (see remdecivir, mRNA vaccines, etc.), before being exclusively licensed to private interests, the quickest way to that is to revoke Bayh-Dole and state that universities must non-exclusively license their discoveries to any parties for a flat fee.
I worked at a small-to-mid-size clinical research organization for a hot minute. That company had its own set of - let's say "unique" - problems*, but the author touched on many of the issues I saw in my short experience at a CRO. I was in an odd (for me), pseudo-managerial, HR-type role at the company (a role in which I'll be the first to admit I wasn't good at) so perhaps I can offer some additional perspective.
But first, I want to give some background on an important part of the CRO world: the "clinical research associate" or "CRA". CRAs are are the CRO's meat-and-potatoes, boots-on-the-ground, core, etc. after contracts have been signed. A CRA is the person who actually goes to clinical trial "sites" (e.g. a doctor's office, a hospital, or, even a dedicated facility for clinical trial participants). The CRA goes to each trial site, talks to the doc, makes sure participant recruitment is going well, checks/writes documentation, and follows up as the trial goes on. In other words, the performance of the CRAs strongly & directly affects the performance of the clinical trial.
Anyway, my top issues with clinical trials today are:
(1) Poor patient recruitment - this is a major problem. There is, of course, lots of variability between trials and CROs with respect to recruitment, but it should be telling that "trial rescue" is often done (at least where I worked) as a result of the incumbent CRO's inability to recruit enough patients.
(2) (Related) Limiting pay to study participants: although it's up to the institutional review board's interpretation (IRB, an ostensibly independent entity that's supposed to sign off on the trial's ethics), good clinical practice (GCP) guidelines states that paying participants too much would be "undue influence". I have no idea how this plays out in practice, but I think it's a bad guideline. Most of the other parts of GCP are good things - stuff like informed consent and not coercing subjects - but this one is (again, in my opinion) probably raising trial costs through recruitment delays or total non-recruitment, even accounting for the extra cost of paying participants more.
(3) Clinical Research Associate (CRA) training & hiring: this isn't unique to the CRO industry but it seems like a bigger problem than in other industries. I've seen US median salary ranges from $70,000 to $120,00 from publicly available sources but my time in the industry rarely demonstrated numbers that high. Although CRAs don't treat patients directly, it is a role that requires medical knowledge, attention to detailed paperwork, and good communication skills. There seemed to be a lot of cutting corners on training & hiring CRAs - e.g. hiring highly experienced CRAs from India and putting them into entry-level roles in the US on the cheap. I don't mean to say there aren't great experienced CRAs from India who are also great when dropped into an entry-level CRA role but doing so as a matter of unofficial policy is a bad idea: India's drug industry has a different regulatory environment (although there is significant cooperation between FDA and Indian manufacturing facilities) and, perhaps more importantly, there will be a wide variation in communication skills simply because english is a second language (or, in the case of India in particular, "Indian english" has taken on a life of its own).
(4) Over-promising, under-delivering - true of any contractor even outside the CRA industry. Even when working on a fixed price basis, there are perpetual "change orders" and arguments over change orders.
Side-note: if you want to learn more about the process by which drugs come to market, I strongly recommend New Drugs by Lawrence T. Friedhoff. It's short and the HN crowd won't have any problem understanding. Amazon link: esel2k↗
I would add to your list of problems: Decades of easy-money milk-the-cow style companies (Often CROs) with very outdated systems with folks sittings in cheap countries to do some data change requests and manual tests. Truely horrible and outdated tech, mixed with bad salaries, high turnover and bad management.
I used to work for a company were because of timezone issues a wrong date was entered, CRA requested correction (edit manually, sign, scan) and wrong assumptions etc we had to change 7 times the whole thing and send CRA onsite and recorrect the mess. In short: Inefficient mess.
I left that industry quickly and can say I that others are better. Lab tracking is way better with diagnostic samples with private labs or when mixed with good logistics partners. It is not everywhere as bad as this.
An interesting company I talked to a while back working on (1) in the spaced is deep6.ai. Interesting approach and there is a lot of work that can be done here sourcing patients more effectively. I think they said something like only 10% of studies are able to be carried out due to lack of patient recruitment.
I worked on one in the 90's that was $50,000 a day for three years. The drug company, assuming the trial was successful, had about 4 years to make all its money back before the patent expired. Testing blood, urine, and stool for multiple things is expensive.
Intellectually, I can understand the need for control groups, but I still think it's immoral. When you stare at a spreadsheet and see 70% of the control group is dead because some random number generator sorted them there like hell's own sorting hat, and the 98% of the people getting the drug are alive, you have no business talking about statistics. That graph will haunt me til my dying day.
You didn't kill 70% of the control group, you saved 98% of the experimental group.
Also, with the appropriate trial design, you can stop the control and transfer patients once you see these big differences. Same thing should happen if you see the opposite, e.g. killing 70% of your experimental group.
If that was truly the case, then - as the parent hinted - it was the trial design that was the problem, and not the practice of having control groups.
It is pretty common to stop trials early when it is obvious that the treatment works. Conversely, it is easy to think that "yea of course the new treatment works" despite the evidence not being there. The need for robust analysis must also be respected.
Yes, exactly. Control groups are incredibly important for ensuring good quality of clinical studies. It's a technique that solves multiple "calibration" problems, including:
- being able to draw causal conclusions
- being able to adjust against placebo effects
A lot of clever people have tried to come up with ways of doing away with control groups. But ultimately, the best we can achieve is to stop early, as soon as the trial has a clear outcome. I do perhaps think this has become more common in recent times though, so perhaps the study you were involved in was at a time when early stopping wasn't really "the done thing".
But it still beats "studies" done 100 years ago, when you might give someone a cough mixture, see that they improved (if they died, let's just ignore that), and conclude that it was the cough mixture that did it!
> the best we can achieve is to stop early, as soon as the trial has a clear outcome
That must be really hard: if you wait for 95% confidence, you are selecting for 5% noise. If you repeatedly re-measure for 95% confidence, you strongly select for random noise.
Not many medical advancements provide such an anomalously strong signal (98% survival versus 30% survival).
> If you repeatedly re-measure for 95% confidence, you strongly select for random noise.
You can't use standard methods for early stopping - as you rightly point out, you get gibberish if you naively keep peeking at a growing data set. Instead, you have to use statistical methods that explicitly adjust for the repeated sampling in early stopping trials. This does make early stopping more complicated to analyse than a trial with a pre-determined duration.
No, in fact it is quite common to do "open label" clinical studies in which patients know what drug they are getting and no one is given the current standard of care. This is especially common in cancer, where the standard of care is so poor, and in rare diseases, where the patient population isn't large enough to admit such a comprehensive study.
It is harder to have the same statistical confidence of efficacy and safety in such studies, but clinical researchers try to address those issues by varying e.g. dosage quantities, time in between doses, etc.
Source: place I work is currently doing studies of this nature, and in general such studies seem to be well-understood and accepted by the FDA.
Randomization is independent from blinding. Open label studies can be either randomized or not, and controlled or not.
The FDA will usually push for blinding if possible (sometimes it is not). They will also usually push for a randomization zed control with standard of care. They way the FDA views it (and I agree), is that control patients are not harmed because they are still getting the same care they would outside of the trial.
It is usually unethical to have a treatment free arm. However, this has its own problem, where if you keep using equivalence comparisons, the end of the chain might not actually be any better
For such a large effect it is quite possible to implement an adaptive trial design with unbalanced arms and interim analysis for efficacy. But you have to ask for this, the FDA will not necessarily suggest it directly.
I don't feel guilt, I still feel rage. I didn't determine any of the parameters or rules of the trial so I have no guilt on me. I just observed that the control group is dead because of some belief that they provided value when we already knew exactly what was going to happen to them.
If you can prevent harm to others, then not doing so is just being an a$$. The trolley problem is just counting souls and really doesn't happen much in real life. The true problem is thinking the only choices are us/them versus everyone. I'm not a doctor so I have no idea how their ethics applies to their professional decisions.
Control groups exist because they get us closer to global optimality.
We can't eliminate suffering and dying of untreated people but it's generally considered ethical to eliminate suffering and dying of many while withholding that treatment from a subgroup of the study ("allowing them to die" by not taking action, rather than the "causing them to die" by taking action).
as long as that selection is done randomly and "fairly" I think it's an entirely acceptable risk. There have been trials that were truly badly run, and people died and suffered more than necessary, due to mistakes (often amateur ones) in the protocol. I'm more concerned about those types of deaths.
This does not happen in a vacuum - it is not about this group at all. It is all about the future and if this will put a stop to a lot more people dying in the future. If it is only going to save 20 people a year, it hardly matters, but if 1,000,000 people will be saved every year, that's different story.
I don't know how bit the groups were, but if you have 500 people in the control group and 150 die, and you save 1,000,000 per year, that is a great trade-off.
Plus, when you go into this experiment, you know you it is an experiment and know in advance it is a 50/50 proposition that you will get the drug/placebo, but at least you have a 50% chance. Better than no chance. And personally, which normally I hate personal experiences, but personally, I wouldn't mind giving up my life in a test if I know I might have a 50% chance of getting the drug, but if not, I save 1,000,000 people a year (or whatever it is).
It's like being in the army. If you are the commander, the general, you might have to send 5,000 men charging up a hill that you know is well-defended and 80% will die, but you are using it as a diversionary tactic and take 80,000 men towards your real objective and you win it, then too bad for the 5,000 men. It's just the way things are. And if someone doesn't have the stomach to be the general, then they shouldn't be the general and look for another line of work.
If that was the case, the study actually was unethical.
Some designs do include interim analyses and stopping rules.
However in my view the real scandal is we are still using NHST for clinical trials. We should be continuously updating a sensible prior for the effect size and approving the drug/stopping the trial when we have sufficient evidence one way the other.
As it is, the results of many underpowered trials are essentially thrown away because p > .05, which is stupid. This says nothing about the balance of evidence for the efficacy of an intervention… only the inertia and innumeracy of many clinicians preserves the tradition.
It's possible to stop a trial because the new drug is much more effective than the old drug. It's also possible to stop a trial because the new drug's significantly worse than the old. We've done both.
Also, I hope that no one ever tells you that you have to do a drug schema change because "too many kids are dying."
I used to be in enterprise software sales and consulting and have been involved for RFIs/RFPs across multiple industries. I don't know a single person who thought the process was sensible or efficient. Unless you have an person whose job is dedicated to responding to the questions, it's basically just a game of passing a hot coal around until you have something you can submit.
I work in an engineering consultancy organisation.
A lot of the big tenders are incredibly wasteful, with lots of money being wasted on them.
We have an aggressive sales pipeline process. We prune our sales pipeline hard, and ensure we are spending money where we can win.
Result, we win about 70% of tenders that go to the end - we save money by recognising where we can't win and walking away early.
On a related note, we are great fans of the 'pre-proposal'. You should be able to get the gist of any proposal down in 2 sides. That should be enough to get buy in from stakeholders on all sides. Only when everyone agrees on the pre-proposal do you then write the full proposal - so you save effort (and make the client happy because they don't have huge amounts of boilerplate to read!)
Nicely written piece highlighting the value of in-house expertise and capabilities, providing a lesson that translates very well imo to why government projects should have more in house expertise and not contract everything to the private sector.
The logistical inefficiencies mentioned in this article are mind boggling when compared to what we've come to expect in different industries. We all know healthcare is always a little behind the curve, but with clinical trials, we're talking decades.
For example: most clinical studies have no way to track samples going from sites to labs beyond a hope, prayer, and constellation of unconnected spreadsheets. The status quo is to write a 50 page lab manual and include a wall of text and bullets outlining a list of multiple stakeholders that need to be emailed a tracking number each time a sample is shipped. Not to mention there are days you really shouldn't ship samples. Labs have no idea when samples are going to show up. Seriously, samples just show up, they open the box expecting a piece of paper to identify the samples.
An incredible amount of perfectly good, usable samples get LOST or DESTROYED due to the fact that there's no way to track any of these samples or coordinate stakeholders. We're talking critical samples, like blood from kids with cancer. It's absolutely ridiculous.
Compare this sob story to the fact that I know in realtime where all the dumb shit I bought on Amazon is. The cognitive dissonance here is beyond comparison!
+ + +
About 8 years ago my brother and I were given a window into this problem. Being Amazon FBA sellers ourselves, we thought, "hey now that can't be too hard to fix" and thus began the wild journey of Slope [0]. What we learned working on this problem is that clinical research sites literally get crapped on left and right with ridiculous contracts & protocols written by people who don't understand the clinical setting or logistics in general. With each research site running 30-50 clinical studies and each of those studies having its own catalog of kits, shippers, and other inventory, it's no wonder why clinical research is so expensive! And this is just supply chain folks.
Slope started by making an inventory and sample management software to address the needs of sites and have worked backwards into selling this platform to Sponsors and CROs. While I have your attention, we're hiring a head of engineering now and will have a JD up for a product manager next week [1]. Thanks for reading!
People complain about industries with bureaucracy but nobody wants to take the hit and actually work in those industries. They want magically for their pay/benefits/culture incentives to align.
Exactly - like the OP you replied to - Slope are trying to hire a Head of engineering for a salary that competent mid/senior level engineers should be easily getting. It's no tech industry, and the work is less fun
I'm one of those that switched from tech into an industry that is steeped in bureaucracy, secrecy, red-tape, and whatever other term you can find. It's been about 10 years, but I occasionally lurk on HN to see people in tech are up to.
For several reasons, but basically I was burned out, and wanted a new challenge. Working in biotech manufacturing was also a way to make something that helps people.
Best explanation I've read online. Most attempts at describing why boil down to the circular "clinical trials are very expensive because the process of getting a new drug approved is lengthy, complex and expensive".
I worked as Director of Engineering at a company for 10 years that built a SaaS for clinical trial document management.
There is a weird perversion of incentives in the industry driven under the guise of "quality" and "safety" that from a software engineer's perspective, looked pretty bad to me because of how manual and error prone the process is.
The problem is that the people who have the knowledge aren't incentived to make it more efficient or cost effective; the purpose of many of these folks is to simply extract rent from the system and blame regulations and "patient safety" concerns.
At an MES startup, one of the industry stalwarts (starts with V) was more or less using mafia tactics to force the startup to use their software or they would voice no confidence with pharma companies where big V was already entrenched.
One of my passion projects would be to build an open-core clinical trial document management system with a focus on automation and templatization of common processes and workflows. It's insane how much waste this system has.
Call me cynical, but this could apply in most any industry:
Suppose that one tries to push back on inefficiency in generalーsomeone naive but well-intentioned, like myself, who is tired of the Long March of interminable meetings. In fact, I expressed my concerns about the proliferation of unproductive meetings in the clinical department to the C-suite executives (with whom I remain on good terms!) and, together, we carefully planned a series of reforms to company culture designed to reduce meeting burden. We hoped to implement a soft cap on the number of meeting participants and the length of a meeting, to normalize efficient behavior such as leaving meetings midway if you no longer need to participate in the remainder, starting meetings on time without 15 minutes of small talk, and a mandated review of all recurring meetings with the intention of pruning overall meeting load by >50%.
Ironically, on the morning that we planned on roll out these reforms, we received urgent and vehement protests from a member of middle management who was notorious for obsessively overscheduling meetings with huge participant lists just for the sake of “inclusion” and “engagement.” Because this particular manager was currently playing a crucial role in managing the rollout of several clinical trials at the same time, executive management judged that the risk of alienating him was too high, and to this day I believe these reforms remain unimplemented (although I hear he has since left the company, so perhaps my slides will eventually see the light of day).
The author doesn't really understand the details of the operations of clinical trials, though this is a good overview of the kinds of outcomes that show up. I worked on operations software for clinical trials for about a decade, and there are (mostly necessary) regulatory drivers a large fraction of the complexity he complains about. The first order effects of the regulations are pretty straightforward, but when you get into second, third, and fourth order impacts, you start seeing some of the outcomes he's describing.
I totally understand the frustrations experienced by the author of this post. I am a nurse and my exposure to the Clinical Trial Industry came from a more bottoms up entry point. Its mind boggling to the the total costs CRO's charge for the services compared to the eventual experience for the staff and patients taking part in these studies.
CRO's are in dire need of disruption. I also agree about the incestuous industry hence the very slow progress being observed. It's hard to show any progress when it's the same people moving around the same companies and implementing ideas that are somewhat outdated.
Also, from a cost perspective, i think the future is also coming to a place where Clinical trials get outsourced outside the US.
Sites and PI's outside the US should be empowered to run this clinical trials as they provide a more cost effective solution with regards to operational costs and also access to an ever growing number of patients, reducing clinical trials timelines that lead to increased costs.
I understand the need for CRO's to maintain quality, and they can do this by building out the right relationships and partnerships with overseas contractors, but i believe in the long run, this is the solution.
I believe this so much that i launched my own CRO company. Infiuss Health. The goal of this company is to unlock the potential for studies to the run by CROs and smaller life science companies outside the US.
We are starting with Africa. As soon as we can unlock this population that has traditionally just been taboo to life science companies, imagine the lengths we could go with with access other geographies.
103 comments
[ 2.8 ms ] story [ 180 ms ] threadWith tens of thousands of participants, this cost adds up.
But yes, the largest phase 3 trials ever are generally vaccines. https://www.evaluate.com/vantage/articles/news/snippets/its-...
I'm not sure I understand the logic here. How does agreement suddenly become less informed or less consensual because the participant gets paid more? Sure, if they get paid more, they might be willing to accept more risk or do something more unpleasant. But that seems fair, not immoral. Why would it be more fair to say "yes, perhaps this trial is risky, but we only want participants who don't demand to be compensated much for the risk they take"? If anything, maybe that approach leads to less informed consent by filtering out the people who actually understand the risk and would have only agreed to participant in exchange for more compensation!
Should we also be morally offended any time an employer raises wages in order to attract more workers?
Why shouldn't college professors sleep with their students? How does the relationship become less informed or less consensual? The answer is power.
Money is power, and too large of a power imbalance corrupts a relationship. Money is pressure, and too much pressure is coercive.
Same reason we don't allow the purchase of kidneys, and the same reason we think it's unethical for a millionaire with a tophat and a monocle to offer two homeless guys a wad of money to the winner if they fight each other. Sure, it's an adult making a decision of their own free will- but it's obviously unethical.
Simply working in a job where you have to kneel or lift your arms above your head all day leads to long term injury.
"Power" is a relatively new argument. These days the student has the power to utterly destroy the professor. The professor has at best the charismatic "power" equivalent to any number of smooth pickup artists occupying college dorms, of getting a temporary good time.
...but I think that's a terrible idea. I understand that, to come up with a far-fetched example, high compensation could result in some weird edge cases where, for example, a Phase I trial with healthy volunteers and a non-preventive drug candidate (e.g. an antibiotic rather than a COVID vaccine), but you're confident that drug is going to kill 99% of the volunteers so you offer $10 million to each volunteer. A totally informed volunteer might view this as something akin to a life insurance policy that pays out to the family even upon suicide. I don't personally see an ethical problem with that so long as volunteers are really well-informed, but I could see how others and the law would be uncomfortable with the ethics of such a situation.
But aside from that far-fetched situation, it seems harmful to limit compensation... especially if it's true that patient compensation is a small part of overall cost. Perhaps if every trial doubled compensation to 10% of today's overall costs, absolute overall costs would decrease because less effort has to go into patient recruitment.
I do see one real-world issue with high compensation, but it has nothing to do with ethics. It has to do with statistical interpretation. If you start dangling too much money to participate in a trial, you may change the kind of people who participate in the trial. These individuals might not be representative of the actual users of the drug. The problem is similar to running a drug trial in the US versus Japan. The US has a pretty diverse population with respect to genetics. Japan, in contrast, is almost entirely ethnic Japanese. If you were to somehow succeed in using Japanese trial data to market an acne cream in the US, the same drug that cleared up Japanese faces might result in Venezuelans having their noses fall off!
However, I'm cautiously optimistic that higher compensation could make trials more representative. In my mind, I could see how African-Americans, who tend to participate in trials at a lower rate than their Caucasian fellow humans, would, if only for the shameful history of the Tuskegee Syphilis Study, require greater compensation to take the risk of participating in a trial.
I agree with your points, but compensation must be approved by the independent institutional review board, who is accountable to national ethical review boards and thus acts very conservatively.
It just seems unfair to subjects to me. Not to mention paternalistic. Every other actor in the process gets enriched based on (limitless it seems) market rates for their contribution, except for the subjects who take on all the risk.
Depending on the trial design it can be less (less frequent follow up, less invasive testing) or a lot more (say doing multiple MRIs or other expensive imaging).
It’s kind of mind blowing when you think about the really big trials for medications like cholesterol. Trials are often 30,000 or more patients, in two trials for 2-3 years.
You’re roughly talking about $2B or more just for phase 3 (ignoring all the earlier trials and preclinical work).
The hospital partner was 100% on board—our protocol just had to pass their internal review board (IRB). IRB is a black box, and they could only give us a vague estimate of how long this would take. We had a huge team, and every week we sat around waiting cost about $200k.
Finally the client couldn't take the uncertainty and burn rate anymore, and they reached out to a CRO they had used in the past. The CRO's IRB had a fixed timeline (2 weeks?), and we were onboarding patients a short time later.
At this point I gave up waiting for a medical solution for my chronic illness, and have a better hope that I can live 15-20 years with it and rejuvenate my body after that time period.
The author already touched upon a few hypotheses why the industry has arrived at this state. There's one I'd like to highlight: The author, now having gone through this experience, would actually be in a perfect position to found their own CRO and implement all their efficiency learnings there. However, they probably look back upon this experience and think it was super painful and would prefer to do fun stuff like developing software instead.
That, for me, is the main reason. There are smart, structured, pragmatic people who got acquainted with this industry, but they choose to leave it.
And that's sad!
So the solution would be to find a group of smart, structured, pragmatic people with a high tolerance for bureaucratic pain and give them the resources to make this industry more efficient.
And here's my plug: That's exactly what we're doing at OpenRegulatory [1], albeit for medical device software and not for clinical trials. Among other things, we've published all our document templates for free (also on GitHub), host a free Slack community for companies to help each other out, and offer a compliance SaaS with transparent and fair pricing. It's still a long way, but it's a start :)
[1] https://openregulatory.com
There's quite an overlap with US (FDA) regulations but we're less experienced there.
Most startups (at least in my bubble?) focus on one of those two areas.
How many industries could these sentences descrive, stripped of context?
Or more easily, are there any industries where this is not the norm?
> an industry which is severely lacking in human capital and fully captured by bureaucratic tendencies
> going through this process is an extended exercise in sheer absurdity.
> meaningless bloat
> a farcical “data-driven” scoring process
> there was an intermediary clinical organization we attempted to work with which was so obviously attempting to grift money from our coffers, the entire relationship escalated into the level of back-and-forth legal threats
> One cannot help but get the impression that this is an industry which is, at best, severely lacking in anything that could be called “human capital” and, in reality, simply deeply unserious
> despite the very large amount of boilerplate content that every single actor in this ecosystem has spent thousands of hours of manpower developing, there seem to have been very few actual gains made in simplifying common tasks or processes!
... Et cetera.
To me, this sounds a lot like processes in education, teaching, energy, environmental protection, agriculture, research, arms, media, law, IT, etc.
I don't believe we can afford to remain this stupid, however much money the inefficiencies are making for those at the top.
porn?
Or in general, any business with highly mobile customers/employees and narrow profit margins.
This is not new - it dates back to the origins of the modern for-profit pharmaceutical industry, such as with Bayer's discovery of the anti-bacterial action of sulfanilimide (patented 1909) in the early 1930s (by which time patent had expired). Bayer made a derivative of sulfanilimide, i.e. Prontosil (patented in 1932) and tried to convince the world that this was the more effective product. In reality, it just broke down to sulfanilimide once ingested, that being the active species.
As far as clinical trials in the USA, there have been numerous scandals regarding lack of consent, failure to halt trials immediately on indication of severe side effects, etc. Major stories about this came out in 2005, 2008, etc. Then there's the world of unregulated third-world clincal trial testing, and actual falsification of clinical trial data by contractors hired to run trials for various Big Pharma outfits, such as this from 2022.
https://endpts.com/clinical-trial-coordinators-sentenced-to-...
> "Tellus, which has worked with big-name pharma sponsors in the past like Pfizer, Gilead, Takeda, Boehringer Ingelheim and Eli Lilly, initially came under scrutiny for two trials related to opioid dependency, two for irritable bowel syndrome, and one for diabetes."
That's just organized crime 101: hire a contractor to do the dirty work, keep your own hands clean, claim ignorance, act shocked, promise reforms, etc.
Best solution I've heard of is a move to open-source (patent-free) drug discovery with federally financed clinical trials. Since almost all new drugs are discovered at universities with federal funds (see remdecivir, mRNA vaccines, etc.), before being exclusively licensed to private interests, the quickest way to that is to revoke Bayh-Dole and state that universities must non-exclusively license their discoveries to any parties for a flat fee.
https://www.fda.gov/drugs/development-approval-process-drugs...
Orphan drug exclusivity is 7 years. The FDA will not approve a generic version of that drug for the same indication for 7 years after your approval.
Of course the challenge is that doctors are free to prescribe whatever they want, including compounded versions of your drugs.
There are ways around it (formulation), but it's not like there is no option at all if you don't have a patent.
Have you tried stellate gangelion block for your long covid ?
But first, I want to give some background on an important part of the CRO world: the "clinical research associate" or "CRA". CRAs are are the CRO's meat-and-potatoes, boots-on-the-ground, core, etc. after contracts have been signed. A CRA is the person who actually goes to clinical trial "sites" (e.g. a doctor's office, a hospital, or, even a dedicated facility for clinical trial participants). The CRA goes to each trial site, talks to the doc, makes sure participant recruitment is going well, checks/writes documentation, and follows up as the trial goes on. In other words, the performance of the CRAs strongly & directly affects the performance of the clinical trial.
Anyway, my top issues with clinical trials today are:
(1) Poor patient recruitment - this is a major problem. There is, of course, lots of variability between trials and CROs with respect to recruitment, but it should be telling that "trial rescue" is often done (at least where I worked) as a result of the incumbent CRO's inability to recruit enough patients.
(2) (Related) Limiting pay to study participants: although it's up to the institutional review board's interpretation (IRB, an ostensibly independent entity that's supposed to sign off on the trial's ethics), good clinical practice (GCP) guidelines states that paying participants too much would be "undue influence". I have no idea how this plays out in practice, but I think it's a bad guideline. Most of the other parts of GCP are good things - stuff like informed consent and not coercing subjects - but this one is (again, in my opinion) probably raising trial costs through recruitment delays or total non-recruitment, even accounting for the extra cost of paying participants more.
(3) Clinical Research Associate (CRA) training & hiring: this isn't unique to the CRO industry but it seems like a bigger problem than in other industries. I've seen US median salary ranges from $70,000 to $120,00 from publicly available sources but my time in the industry rarely demonstrated numbers that high. Although CRAs don't treat patients directly, it is a role that requires medical knowledge, attention to detailed paperwork, and good communication skills. There seemed to be a lot of cutting corners on training & hiring CRAs - e.g. hiring highly experienced CRAs from India and putting them into entry-level roles in the US on the cheap. I don't mean to say there aren't great experienced CRAs from India who are also great when dropped into an entry-level CRA role but doing so as a matter of unofficial policy is a bad idea: India's drug industry has a different regulatory environment (although there is significant cooperation between FDA and Indian manufacturing facilities) and, perhaps more importantly, there will be a wide variation in communication skills simply because english is a second language (or, in the case of India in particular, "Indian english" has taken on a life of its own).
(4) Over-promising, under-delivering - true of any contractor even outside the CRA industry. Even when working on a fixed price basis, there are perpetual "change orders" and arguments over change orders.
Side-note: if you want to learn more about the process by which drugs come to market, I strongly recommend New Drugs by Lawrence T. Friedhoff. It's short and the HN crowd won't have any problem understanding. Amazon link: esel2k ↗ I would add to your list of problems: Decades of easy-money milk-the-cow style companies (Often CROs) with very outdated systems with folks sittings in cheap countries to do some data change requests and manual tests. Truely horrible and outdated tech, mixed with bad salaries, high turnover and bad management.
I used to work for a company were because of timezone issues a wrong date was entered, CRA requested correction (edit manually, sign, scan) and wrong assumptions etc we had to change 7 times the whole thing and send CRA onsite and recorrect the mess. In short: Inefficient mess. selecsosi ↗ An interesting company I talked to a while back working on (1) in the spaced is deep6.ai. Interesting approach and there is a lot of work that can be done here sourcing patients more effectively. I think they said something like only 10% of studies are able to be carried out due to lack of patient recruitment. FrontierPsych ↗ what do you mean by ""Indian english" has taken on a life of its own"
I left that industry quickly and can say I that others are better. Lab tracking is way better with diagnostic samples with private labs or when mixed with good logistics partners. It is not everywhere as bad as this.
TLDR: add outdated tech, greed and bad management
Intellectually, I can understand the need for control groups, but I still think it's immoral. When you stare at a spreadsheet and see 70% of the control group is dead because some random number generator sorted them there like hell's own sorting hat, and the 98% of the people getting the drug are alive, you have no business talking about statistics. That graph will haunt me til my dying day.
Also, with the appropriate trial design, you can stop the control and transfer patients once you see these big differences. Same thing should happen if you see the opposite, e.g. killing 70% of your experimental group.
We killed 70% of the control group. Doomed by a random number generator.
If that was truly the case, then - as the parent hinted - it was the trial design that was the problem, and not the practice of having control groups.
It is pretty common to stop trials early when it is obvious that the treatment works. Conversely, it is easy to think that "yea of course the new treatment works" despite the evidence not being there. The need for robust analysis must also be respected.
- being able to draw causal conclusions
- being able to adjust against placebo effects
A lot of clever people have tried to come up with ways of doing away with control groups. But ultimately, the best we can achieve is to stop early, as soon as the trial has a clear outcome. I do perhaps think this has become more common in recent times though, so perhaps the study you were involved in was at a time when early stopping wasn't really "the done thing".
But it still beats "studies" done 100 years ago, when you might give someone a cough mixture, see that they improved (if they died, let's just ignore that), and conclude that it was the cough mixture that did it!
That must be really hard: if you wait for 95% confidence, you are selecting for 5% noise. If you repeatedly re-measure for 95% confidence, you strongly select for random noise.
Not many medical advancements provide such an anomalously strong signal (98% survival versus 30% survival).
You can't use standard methods for early stopping - as you rightly point out, you get gibberish if you naively keep peeking at a growing data set. Instead, you have to use statistical methods that explicitly adjust for the repeated sampling in early stopping trials. This does make early stopping more complicated to analyse than a trial with a pre-determined duration.
It is harder to have the same statistical confidence of efficacy and safety in such studies, but clinical researchers try to address those issues by varying e.g. dosage quantities, time in between doses, etc.
Source: place I work is currently doing studies of this nature, and in general such studies seem to be well-understood and accepted by the FDA.
The FDA will usually push for blinding if possible (sometimes it is not). They will also usually push for a randomization zed control with standard of care. They way the FDA views it (and I agree), is that control patients are not harmed because they are still getting the same care they would outside of the trial.
It is usually unethical to have a treatment free arm. However, this has its own problem, where if you keep using equivalence comparisons, the end of the chain might not actually be any better
I suspect there are some people with different philosophies that that sit better with this sorta thing.
It seems like you feel guilt not only for the harm you cause, but the harm you fail to prevent. Do you apply this logic to other parts of your life?
How do you feel about the trolley problem, were you have to kill some to save others?
How do you feel about the moral imperative of doctors to do no harm versus a utilitarian approach of maximize lives saved?
If you can prevent harm to others, then not doing so is just being an a$$. The trolley problem is just counting souls and really doesn't happen much in real life. The true problem is thinking the only choices are us/them versus everyone. I'm not a doctor so I have no idea how their ethics applies to their professional decisions.
We can't eliminate suffering and dying of untreated people but it's generally considered ethical to eliminate suffering and dying of many while withholding that treatment from a subgroup of the study ("allowing them to die" by not taking action, rather than the "causing them to die" by taking action).
as long as that selection is done randomly and "fairly" I think it's an entirely acceptable risk. There have been trials that were truly badly run, and people died and suffered more than necessary, due to mistakes (often amateur ones) in the protocol. I'm more concerned about those types of deaths.
I don't know how bit the groups were, but if you have 500 people in the control group and 150 die, and you save 1,000,000 per year, that is a great trade-off.
Plus, when you go into this experiment, you know you it is an experiment and know in advance it is a 50/50 proposition that you will get the drug/placebo, but at least you have a 50% chance. Better than no chance. And personally, which normally I hate personal experiences, but personally, I wouldn't mind giving up my life in a test if I know I might have a 50% chance of getting the drug, but if not, I save 1,000,000 people a year (or whatever it is).
It's like being in the army. If you are the commander, the general, you might have to send 5,000 men charging up a hill that you know is well-defended and 80% will die, but you are using it as a diversionary tactic and take 80,000 men towards your real objective and you win it, then too bad for the 5,000 men. It's just the way things are. And if someone doesn't have the stomach to be the general, then they shouldn't be the general and look for another line of work.
That's how I see it.
Some designs do include interim analyses and stopping rules.
However in my view the real scandal is we are still using NHST for clinical trials. We should be continuously updating a sensible prior for the effect size and approving the drug/stopping the trial when we have sufficient evidence one way the other.
As it is, the results of many underpowered trials are essentially thrown away because p > .05, which is stupid. This says nothing about the balance of evidence for the efficacy of an intervention… only the inertia and innumeracy of many clinicians preserves the tradition.
Statistical analyses need to be pre-defined and powered to measure an effect.
I've been on clinical trials where the innovator drug showed a fantastic effect early on, then two overall survival curves crossed a year later.
I used to be in enterprise software sales and consulting and have been involved for RFIs/RFPs across multiple industries. I don't know a single person who thought the process was sensible or efficient. Unless you have an person whose job is dedicated to responding to the questions, it's basically just a game of passing a hot coal around until you have something you can submit.
A lot of the big tenders are incredibly wasteful, with lots of money being wasted on them.
We have an aggressive sales pipeline process. We prune our sales pipeline hard, and ensure we are spending money where we can win.
Result, we win about 70% of tenders that go to the end - we save money by recognising where we can't win and walking away early.
On a related note, we are great fans of the 'pre-proposal'. You should be able to get the gist of any proposal down in 2 sides. That should be enough to get buy in from stakeholders on all sides. Only when everyone agrees on the pre-proposal do you then write the full proposal - so you save effort (and make the client happy because they don't have huge amounts of boilerplate to read!)
For example: most clinical studies have no way to track samples going from sites to labs beyond a hope, prayer, and constellation of unconnected spreadsheets. The status quo is to write a 50 page lab manual and include a wall of text and bullets outlining a list of multiple stakeholders that need to be emailed a tracking number each time a sample is shipped. Not to mention there are days you really shouldn't ship samples. Labs have no idea when samples are going to show up. Seriously, samples just show up, they open the box expecting a piece of paper to identify the samples.
An incredible amount of perfectly good, usable samples get LOST or DESTROYED due to the fact that there's no way to track any of these samples or coordinate stakeholders. We're talking critical samples, like blood from kids with cancer. It's absolutely ridiculous.
Compare this sob story to the fact that I know in realtime where all the dumb shit I bought on Amazon is. The cognitive dissonance here is beyond comparison!
+ + +
About 8 years ago my brother and I were given a window into this problem. Being Amazon FBA sellers ourselves, we thought, "hey now that can't be too hard to fix" and thus began the wild journey of Slope [0]. What we learned working on this problem is that clinical research sites literally get crapped on left and right with ridiculous contracts & protocols written by people who don't understand the clinical setting or logistics in general. With each research site running 30-50 clinical studies and each of those studies having its own catalog of kits, shippers, and other inventory, it's no wonder why clinical research is so expensive! And this is just supply chain folks.
Slope started by making an inventory and sample management software to address the needs of sites and have worked backwards into selling this platform to Sponsors and CROs. While I have your attention, we're hiring a head of engineering now and will have a JD up for a product manager next week [1]. Thanks for reading!
[0] https://www.slope.io/ [1] https://www.slope.io/jobs/head-of-engineering
All the bureaucratese in the RFPs and heavy-duty lawyering just add, unnecessarily and hugely, to the cost. At bottom, though, you have:
1) A lot of human beings who have to be recruited and paid to be subjects
2) Medical institutions that have to administer the drugs or run the devices
3) A sufficient amount of elapsed time for effects and side effects to manifest
4) A reporting system that's trustworthy
So at best you could cut (puts finger to the wind) half the cost? Surely not 90%.
There is a weird perversion of incentives in the industry driven under the guise of "quality" and "safety" that from a software engineer's perspective, looked pretty bad to me because of how manual and error prone the process is.
The problem is that the people who have the knowledge aren't incentived to make it more efficient or cost effective; the purpose of many of these folks is to simply extract rent from the system and blame regulations and "patient safety" concerns.
At an MES startup, one of the industry stalwarts (starts with V) was more or less using mafia tactics to force the startup to use their software or they would voice no confidence with pharma companies where big V was already entrenched.
One of my passion projects would be to build an open-core clinical trial document management system with a focus on automation and templatization of common processes and workflows. It's insane how much waste this system has.