I looked at a few of these (FDA approved AI products) recently because I was curious as to what the instrument was allowed to do and sure enough it was more of a smart instrument that told you, for instance, how large a volume was.
These products don't, at the moment, actually diagnose a condition or recommend a treatment which is interesting.
One of the constraints for AI treatment has always been authority and responsibility. To put it another way, if the AI completely messes up and recommends something terrible, whom do you sue?
I am hopeful that we (as a society) will sort out these issues with self-driving cars and apply some of those rules to medicine.
It's tough to assign blame and responsibility to systems that are inherently probabilistic. ML algorithms output probabilities so even as the algorithms get incredibly good, unlikely scenarios will inevitably occur in large enough samples. This isn't a problem when the task is to tag your Facebook photos but is a problem when driving a car in the real world.
>It's tough to assign blame and responsibility to systems that are inherently probabilistic.
I hear this all the time nowadays, but everything has always been probabilistic. Most of the time, people just don't realize that all problems are fundamentally the same from the perspective of how to approach risk.
Physical systems are probabilistic. Environmental conditions are probabilistic. And we've been trusting our safety to physical products operating in the real world for a long time.
Whether physical or ML or whatever ... There's always some process that you can't characterize to your liking. So you explore mitigation options until you can make a decision about whether to accept the risk. Standards and regulations enforce that you went through those steps to get a reasonably trustworthy result.
Then liability law applies in the usual way. Actuaries and accountants tell you what it's going to cost you, and market research tells you how much it might sell for.
Inability to control or understand what you built doesn't absolve you of any fault or responsibility. All it does is change your risk acceptance criteria.
I think it is a little wrong-headed to emphasise the litigation aspect. Something that I think people don't appreciate is that as a patient, you want a real person to take responsibility for your problem because that is the basis for a favourable outcome and a satisfying interaction. This applies to every service in the end, whether you are ordering food at a restaurant, trying to get a plumbing problem fixed or being treated for cancer.
Any corporation, for-profit or not, generally seeks to obfuscate their responsibilities where possible (one could argue that this is the explicit purpose of a corporation - to be a virtual entity which cannot really be held accountable). Tobacco companies are the best and ongoing examples of this. Creating legal protection for AI such that accountability is somehow removed or diminished is not desirable in my opinion. It will lead inevitably to a situation where healthcare becomes analogous to trying to get technical support for your printer. Contrast this with the what happens currently, where you walk into a clinic or a hospital with a problem and meet a bunch of real people who listen to you. Of course there are massive problems with healthcare, and many people have unpleasant or dangerous experiences, but it can definitely be made worse if we aren't careful.
I've done a fair amount of work in the area, and one of the biggest lessons is "even if you can diagnose, you shouldn't"
We likely won't have to deal with the legal dilemmas for a while for a simple reason: doctors _hate_ when non-doctors diagnose, and you can't build a diagnostic tool without doctors. The tools simply won't get built to any real quality standard.
There's an easier path, interestingly. It's much more effective to just give docs the info they need. A system that could do the following would be the holy grail: "Patient entered ER with [chest pain, shortness of breath], and has a history of [hypertension, diabetes]. They are on [X, Y medication], but have not filled their prescription in [Z weeks]." No diagnosis, just the right info at the right time.
The expert system guys in the 80s -- and this is before my time -- came to a similar conclusion, but for the reasons I am laying out (responsibility).
They also decided it was better to have a computer make a suggestion to a doctor than to give it unvarnished to a patience, sort of assisting the doctor.
I won't disagree with experts (yourself, or them), but there is the issue of scaling it. Consider a poor person in a poor country for whom the alternative is not a delayed appointment with a real person but no advice at all.
It's an interesting time for AI/ML/pattern rec in medical devices.
People have been shipping such systems since at least the mid 90s (e.g. CADe) but it's always been a bit of a difficult area from a regulatory point of view.
If you go back a couple decades, or really even only one depending on the panel, it was fair to describe the FDA as an organization that understood hardware but didn't really understand software. On top of this, a lot of their approach was focused on controlled, reproducible results. So anything probabilistic or even worse, continually learning, becomes difficult to frame.
Many people I've talked to without direct experience seem to assume the 13485 process as a whole is needlessly bureaucratic, but in my experience it's mostly holding yourself to a higher engineering standard that some other industries, and works out ok. This willingness to engage with technologies can be frustratingly slow, but it's mostly reasonable.
13485 is relatively bureaucratic, but it's not required for FDA per se and there are things you can do. What is more problematic is IEC 62304 or similar, which embeds a very particular SDLC into the development approach. It doesn't really fit AI/ML, it barely fits modern software development anyway.
You can avoid it for class I, but generally 13485 is becoming more harmonized and required.
I don't think 62304 is too hard to adapt; i haven't done the process work in a while but unless the latest rev radically changed things it isn't very prescriptive, not nearly as much as you suggest; I've seen agile and Agile variants done. It is a sign of the slowness of the process that it's only recently become required, and reflects 15-years ago thinking, but it's fine.
You are right that a lot of modern development processes need a fair bit of tweaking, but it's mostly because they aren't nearly careful enough for device use. This may be a bit of a burden on pure software devices, but it's probably the right trade off overall.
This is why the big shift in thinking for most developers will be the 14971 (risk management) adoption, not 62304 itself (IEC 62304 punts to ISO 14971 for this part, but it's common with hardware dev)
Interesting. I do agree 62304 seems to mostly codify what we would consider best software principles anyway.
Your statement about 14971 being the big shift in thinking is interesting. With a bit of clinical context, do you think developers can write code with risk management in mind?
> With a bit of clinical context, do you think developers can write code with risk management in mind?
Absolutely.
Some people with more of a hardware bent are probably used to FMEA - but that requires a design to evaluate. Risk management approach (should) starts at the other end, while you are still understanding what your product should do.
If you do 14971 right (and iteratively), it will help you pin down key requirements up front that will help you avoid missteps that are costly to recover later.
One of the big shifts is that it makes you think about uncommon/incorrect usages earlier, as well as thinking about how other systems could mess with you.
I agree risk management should start at the patient and work backwards to the individual units that themselves can be inputs into an FMEA. When the design gets a bit further along, you can perform an FMEA and repeat the process several times over the course of the design. I am always surprised at how many more risks can be identified by doing the process iteratively and frequently.
Do you recommend any tools that have helped you perform risk analysis?
Thanks! We think so too. It's a daunting task to go from ISO13485 or IEC62304 to a set of process documents. We hope to continue improving RDM to lower the barrier for researchers and startup founders in the space. We're about to add templates for a low-overhead QMS within the next week or two.
> It's a daunting task to go from ISO13485 or IEC62304
It can be a bit overwhelming, especially the first time, and especially from scratch.
I think for most people entering this space from a software point of view, the thing to get right first is some form of requirement management then later doc control. The QMS docs will come later too, but those two have the biggest impact on the engineering team itself.
Agreed. The standards are designed to be impervious to time and the implementations used. This has the unfortunate consequence of being quite abstract and difficult to grasp at first.
For example "configuration management" was meaningless the first time I read it but really just translates to "version control"
I think having an open source example will really help the concepts make sense to engineers implementing this for the first time.
Configuration management is more than version control, it's how you control your entire deployment, VC is one part.
VC can be a huge part of this if you are software only, but it also helps sometimes to remember that these standards were written with hardware in mind.
Another thing that can help with understanding is to start from what they are trying to achieve (talking to someone experienced here helps) and work backwards.
For example, the idea that if I pull a version of your release software off a machine at a hospital, you should be able to demonstrate to me that you know exactly how it was produced and ended up there. The source code, the build machines, the packaging, etc. You should be able to tell me what other version exist and where they are. If I need you to notify everyone with a particular configuration of an important update you can do that.
Stuff like this leads you to the configuration management language pretty naturally.
62304 does not limit your choice of sdlc or agility. You just have to be able to read it.
In generally, Developing with regulatory oversight and a real QA department is just very different. The agile manifesto speaks of "working software" over "documentation". It does not speak of code. It speaks about working software. In a regulated business "working software" includes a forest worth of hopefully digital paper and the code for your customers and oversight. The paperwork is part of the output here and not part of the self inflicted documentation.
That's a good way of putting it, but I would avoid pinning this on "regulated industry"
Rather, in good engineering practice the design and process, etc. is part of the product. Regulated industries just put some expectations on what that means. This can be done well or poorly, but it's basically the intent.
Don't read that the wrong way; I don't think every product benefits enough from good engineering practice to make it worthwhile - it certainly adds some friction and makes even "fast iteration" slower. On the other hand, it's absolutely the right way to approach things with high stakes for failures.
Regulatory requirements usually does box you into a waterfall sdlc. However, that does not stop you from practicing an iterative sdlc internally.
I like to conceptualize it as having a "waterfall shell with an iterative core"
It is not uncommon for us to have new requirements added in the middle of a release. However, once safety testing and clinical validation are performed, changes are pushed to the next version unless a huge safety issue is detected.
As far as regulatory submissions go and the design outputs that get stored in your QMS, the SDLC appears waterfall because there are clearly defined design inputs, design outputs, and verification / validation checkpoints. In practice, however, the actual process can be agile.
I would word it differently, that the set of documents in your DHF, MDR, 510(k) filing, etc. are identical regardless of the methodology you use to create them.
Some of the language around them sounds "waterfall-y" but you are perfectly welcome to iterate on these documents. The documents themselves are more about engineering practice than process, and are an output of whatever process you use, much like the software is.
1. I wonder how many times the test set can be used on "incremental changes in future versions of the model" before losing statistical validity.
2. This article describes their process, but not the FDA's process. Are there specific regulatory requirements for ML models beyond their four types of reports?
1) AFIAK there are no hard and set rules for this. I think this would have to be the manufacturer's judgement call. Good point though, with enough time you may end up just over-fitting to the test set.
What is your tooling regarding CI/CD and documentation maintenance?
We thought we’d use off the shelf software to do (COTS ;)) but it seems incredibly complex and rigid for a SaMD that’s just, in a way, a simple function.
36 comments
[ 0.20 ms ] story [ 83.5 ms ] threadThese products don't, at the moment, actually diagnose a condition or recommend a treatment which is interesting.
One of the constraints for AI treatment has always been authority and responsibility. To put it another way, if the AI completely messes up and recommends something terrible, whom do you sue?
I am hopeful that we (as a society) will sort out these issues with self-driving cars and apply some of those rules to medicine.
I hear this all the time nowadays, but everything has always been probabilistic. Most of the time, people just don't realize that all problems are fundamentally the same from the perspective of how to approach risk.
Whether physical or ML or whatever ... There's always some process that you can't characterize to your liking. So you explore mitigation options until you can make a decision about whether to accept the risk. Standards and regulations enforce that you went through those steps to get a reasonably trustworthy result.
Then liability law applies in the usual way. Actuaries and accountants tell you what it's going to cost you, and market research tells you how much it might sell for.
Inability to control or understand what you built doesn't absolve you of any fault or responsibility. All it does is change your risk acceptance criteria.
Any corporation, for-profit or not, generally seeks to obfuscate their responsibilities where possible (one could argue that this is the explicit purpose of a corporation - to be a virtual entity which cannot really be held accountable). Tobacco companies are the best and ongoing examples of this. Creating legal protection for AI such that accountability is somehow removed or diminished is not desirable in my opinion. It will lead inevitably to a situation where healthcare becomes analogous to trying to get technical support for your printer. Contrast this with the what happens currently, where you walk into a clinic or a hospital with a problem and meet a bunch of real people who listen to you. Of course there are massive problems with healthcare, and many people have unpleasant or dangerous experiences, but it can definitely be made worse if we aren't careful.
We likely won't have to deal with the legal dilemmas for a while for a simple reason: doctors _hate_ when non-doctors diagnose, and you can't build a diagnostic tool without doctors. The tools simply won't get built to any real quality standard.
There's an easier path, interestingly. It's much more effective to just give docs the info they need. A system that could do the following would be the holy grail: "Patient entered ER with [chest pain, shortness of breath], and has a history of [hypertension, diabetes]. They are on [X, Y medication], but have not filled their prescription in [Z weeks]." No diagnosis, just the right info at the right time.
They also decided it was better to have a computer make a suggestion to a doctor than to give it unvarnished to a patience, sort of assisting the doctor.
I won't disagree with experts (yourself, or them), but there is the issue of scaling it. Consider a poor person in a poor country for whom the alternative is not a delayed appointment with a real person but no advice at all.
https://karpathy.github.io/2019/04/25/recipe/
In particular, I can relate to the author's suggestion to "visualize everything."
People have been shipping such systems since at least the mid 90s (e.g. CADe) but it's always been a bit of a difficult area from a regulatory point of view.
If you go back a couple decades, or really even only one depending on the panel, it was fair to describe the FDA as an organization that understood hardware but didn't really understand software. On top of this, a lot of their approach was focused on controlled, reproducible results. So anything probabilistic or even worse, continually learning, becomes difficult to frame.
The FDA guidance doc/ action plan (link is broken in article and on FDA site: it's here https://www.fda.gov/medical-devices/software-medical-device-...) is actually pretty encouraging, because they took community feedback from the original request for feedback (https://www.fda.gov/media/122535/download) and seem to have handle that process reasonably.
Many people I've talked to without direct experience seem to assume the 13485 process as a whole is needlessly bureaucratic, but in my experience it's mostly holding yourself to a higher engineering standard that some other industries, and works out ok. This willingness to engage with technologies can be frustratingly slow, but it's mostly reasonable.
I don't think 62304 is too hard to adapt; i haven't done the process work in a while but unless the latest rev radically changed things it isn't very prescriptive, not nearly as much as you suggest; I've seen agile and Agile variants done. It is a sign of the slowness of the process that it's only recently become required, and reflects 15-years ago thinking, but it's fine.
You are right that a lot of modern development processes need a fair bit of tweaking, but it's mostly because they aren't nearly careful enough for device use. This may be a bit of a burden on pure software devices, but it's probably the right trade off overall.
This is why the big shift in thinking for most developers will be the 14971 (risk management) adoption, not 62304 itself (IEC 62304 punts to ISO 14971 for this part, but it's common with hardware dev)
Your statement about 14971 being the big shift in thinking is interesting. With a bit of clinical context, do you think developers can write code with risk management in mind?
Absolutely.
Some people with more of a hardware bent are probably used to FMEA - but that requires a design to evaluate. Risk management approach (should) starts at the other end, while you are still understanding what your product should do.
If you do 14971 right (and iteratively), it will help you pin down key requirements up front that will help you avoid missteps that are costly to recover later.
One of the big shifts is that it makes you think about uncommon/incorrect usages earlier, as well as thinking about how other systems could mess with you.
Do you recommend any tools that have helped you perform risk analysis?
This document has some useful tips:
AAMI TIR45:2012 (R2018) Guidance On The Use Of AGILE Practices In The Development Of Medical Device Software
Also, we have an open source offering that includes an IEC62304 compliant software plan. You can check this out here:
https://github.com/innolitics/rdm/blob/master/rdm/init_files...
It can be a bit overwhelming, especially the first time, and especially from scratch.
I think for most people entering this space from a software point of view, the thing to get right first is some form of requirement management then later doc control. The QMS docs will come later too, but those two have the biggest impact on the engineering team itself.
For example "configuration management" was meaningless the first time I read it but really just translates to "version control"
I think having an open source example will really help the concepts make sense to engineers implementing this for the first time.
VC can be a huge part of this if you are software only, but it also helps sometimes to remember that these standards were written with hardware in mind.
Another thing that can help with understanding is to start from what they are trying to achieve (talking to someone experienced here helps) and work backwards.
For example, the idea that if I pull a version of your release software off a machine at a hospital, you should be able to demonstrate to me that you know exactly how it was produced and ended up there. The source code, the build machines, the packaging, etc. You should be able to tell me what other version exist and where they are. If I need you to notify everyone with a particular configuration of an important update you can do that.
Stuff like this leads you to the configuration management language pretty naturally.
In generally, Developing with regulatory oversight and a real QA department is just very different. The agile manifesto speaks of "working software" over "documentation". It does not speak of code. It speaks about working software. In a regulated business "working software" includes a forest worth of hopefully digital paper and the code for your customers and oversight. The paperwork is part of the output here and not part of the self inflicted documentation.
Rather, in good engineering practice the design and process, etc. is part of the product. Regulated industries just put some expectations on what that means. This can be done well or poorly, but it's basically the intent.
Don't read that the wrong way; I don't think every product benefits enough from good engineering practice to make it worthwhile - it certainly adds some friction and makes even "fast iteration" slower. On the other hand, it's absolutely the right way to approach things with high stakes for failures.
I like to conceptualize it as having a "waterfall shell with an iterative core"
It is not uncommon for us to have new requirements added in the middle of a release. However, once safety testing and clinical validation are performed, changes are pushed to the next version unless a huge safety issue is detected.
This really isn't true, at least not for this context.
As far as regulatory submissions go and the design outputs that get stored in your QMS, the SDLC appears waterfall because there are clearly defined design inputs, design outputs, and verification / validation checkpoints. In practice, however, the actual process can be agile.
I would word it differently, that the set of documents in your DHF, MDR, 510(k) filing, etc. are identical regardless of the methodology you use to create them.
Some of the language around them sounds "waterfall-y" but you are perfectly welcome to iterate on these documents. The documents themselves are more about engineering practice than process, and are an output of whatever process you use, much like the software is.
1. I wonder how many times the test set can be used on "incremental changes in future versions of the model" before losing statistical validity.
2. This article describes their process, but not the FDA's process. Are there specific regulatory requirements for ML models beyond their four types of reports?
2) The FDA's guidance on ML models is still in flux. Please see https://www.fda.gov/medical-devices/software-medical-device-...