http://symcat.com/conditions/lumbago does not seem correct to me. I typed back pain and got linked to "I may have lumbago" which talks about mastectomy.
this is pretty cool. The "find a medical practice" tool is a great use of data on care quality. kudos for making this data easily accessible to the general public!
We started working on this during our ER rotation and were surprised how many people were showing up concerned about things that were ultimately minor complaints (eg cough, sore throat). Many of them had no primary care doctor or just didn't know there were alternatives to the ER. Very little info on the Internet helps these people so we're trying to fill that gap.
I was surprised at the results. I was disappointed that there wasn't some gentle friendly persuasive advice about going to see a real doctor quickly. Yes, I know there's a disclaimer on the front page, but I feel this is a missed opportunity.
You're going to tap into that hard to reach market of people who might have a serious problem but who are reluctant to see someone. You have a great opportunity to educate them; to let them know that there's probably nothing to worry about but that early intervention is crucial; and that this is what happens at these types of examinations.
But: It's a really cool tool, and I look forward to seeing how it develops!
Yeah, we have about 1000 symptoms, but that is an example of one we haven't incorporated yet. We'll add it, thanks for your help!
That's exactly where we're going. We want to direct patients to the right doctor/level of care and are incorporating triage guidelines towards that end.
I think it needs a lot of work. The interface is nice but it's missing a lot of symptoms and rules. I think it is repeating expert system work that has probably already been done.
In it's final result it should make a mention of urgency in seeking healthcare and what to do in the meantime.
We'll be adding symptoms based on user input analysis and building out the symptom synonym engine.
True, there has been a lot of work in medical expert systems. DXPlain is an old one that tends to come up a lot in our conversations. We have several Hopkins medical informatics specialists advising us and I would say that our algorithm is novel.
Moreover, past attempts have attempted to codify a medical expert's knowledge (eg let's ask this smart doctor we know to determine disease prevalence). We calculated it from clinical data and can continue to learn as more people use it.
At a bare minimum I would except the site to default to HTTPS (with no warnings). Got an "Untrusted Connection" when trying to use HTTPS. That's not a good sign if you want people to enter personal medical information.
The fix seems to have been to copy the definition of Chronic Back Pain. But those have different diagnoses in your system (I'm seeing Lumbago as 14% and Chronic Back Pain as 10% for the same symptoms)
You are right - she doesn’t. My wife, a 4th year med student blanket opinion of self-diagnostic websites for the general public is they are trying to kill their customers. And more and more I agree.
If you are ill and you don’t know what’s going on, see a doc (or a nurse practitioner etc). Uninsured (yeah - that’s me till 2 months ago.) - then it will cost a packet. Which is good as you’ll be encouraged to figure out what’s normal for your body - which is what needs to happen anyhow.
Remote diagnostics is a hard game even if a doc is doing it. A rules engine just isn’t up to it. It’s not just a game of checklists. This thing might be better (It’s down right now) - but nothing is as good as being in a room with the patient, having some EQ, asking questions, and being able to view symptoms with your own 2 eyes.
Sorry that it's down. Getting a lot of HN traffic.
I tend to agree with your wife (I'm a 4th year med student at Hopkins), which is why I'm building my own.
I think it's easy for websites to just say "if you are concerned, see a doctor" but I also believe that to be the problem. Obviously, if it were costless (time, money, opportunity cost) people would go see a doctor, but they in reality are weighing those costs. We'd like to make it easy for people to weight the risk vs. the benefits of seeing a doctor.
Moreover, for too many people "seeing a doctor" means going straight to the ER. There they wait for 4 hours (nat'l average) and are frequently told to follow-up with a doc anyway. We want to let others know that there are other options (eg urgent care centers) and let them know when such care is clinically indicated.
I don't know. The advantage of such expert systems is that they have access to the entire database, never forget anything and can recalculate all relevant probabilities on the fly. Your doctor can't do that.
Although the linked site failed completely (as far as I can tell, because its buggy!), the WebMD that was linked elsewhere correctly diagnosed my condition in less than 2 minutes. It took a real MD 2 hours+ and that included an (unnecessary) X-ray.
I don't think it's comparable to remote diagnosis being hard - that is very dependent on how doctors recall information, and visual information or feedback may be very important to them. An expert system has no such limitations.
Glad to hear it worked out for you! Unfortunately, for every person that are helped by medical information, there are 5 who are caused unnecessary anxiety. We want to give patients access to data that lets them know when they should be concerned and when they should not be concerned. Microsoft has a great study on this (try Googling "cyberchondria microsoft research")
We're still building out our recommendations, so you can expect ours to also alert people to see a doctor when appendicitis is likely.
For example, I can click Arm -> Armpit -> Lump or Bulge and see the possible conditions.
Armpit doesn't even exist in your system. Which would be fine if I already knew how to self diagnose a swollen lymph node. Of course, if I didn't, I would still need a doctor to tell me what search term to use on your site.
I can understand if the system is early in development, but if it doesn't do anything for the layman why would my doctor care if I saw it?
We've started with the most common symptoms, but are building it out more. We'll add the armpit, thanks for the suggestions!
The AI component is the fact that we train our algorithm based on clinical data from real patients (started with CDC data -- 500k at the moment and counting as people use the site).
I've had bad experiences with the WebMD symptom checker like most people I've spoken to. Try typing in "chills" and you get "Lyme disease, acne, Bubonic plague" (no joke).
WebMD makes money on advertising, so they unfortunately direct you to the pages that are going to keep you on the site for longer. You may find what you have using their symptom checker, but it is really optimized to keep you clicking.
Can you be a bit less general about your algorithm? Without giving any specifics that you may wish to keep to yourself of course. What distinguishes it from statistical methods for example?
Who is the intended audience for the symptoms and conditions directories? Are the numbers next to the entries probabilities of having that symptom or condition for the general population?
Ditto with everything related to menstruation-- although actually I have a sister/brother who's transitioning from female to male and is still menstruating. So, uh, there's some complexity there.
This has the same problem as hypochondriac internet forums.
I put in "tired" and got a list of: depression, diabetes, hypothyroidism, atrial fibrillation,anxiety ,multiple sclerosis, congestive heart failure and gastroesophageal reflux disease.
If I click on "increased risk," I get: valley fever, muscular dystrophy, hemolytic anemia, intracranial abscess, thalassemia, gallbladder cancer, polycythemia vera, and autonomic nervous system disorder.
Nowhere does it ask me if I happened to get enough sleep...
Maybe they need a big button that says "Are you really worried right now?" and if you click it, tell you logical steps and routine you need to be following long-term for the results to "feel more normal" - otherwise to not use to self-diagnose.
The data comes from ED and outpatient doc visits so it "assumes" a certain level of concern from the user at the outset. However, the algorithms can learn so that less severe diagnoses ("lack of sleep", "caffeine withdrawal", etc) will rise to the top as more people indicate that is the problem.
Couldn't you just add a prior probability that someone has any problem? Going to the ER is much more difficult than typing something into a web app, so you should use your expert knowledge to adjust that.
It would also seem like a better approach would be to have a prior that's much lower for the internet and is never updated based on user searches, then continue hand-entering real records for quality control purposes. Otherwise, sooner or later some internet community (4chan, SA, etc) will get carried away and start manipulating your system to make everyone think they're dying of elephantiasis.
Agreed, and that's a very reasonable place to start for conditions that are not challenging diagnoses but rather explanations for common symptoms (eg too little sleep, bumped my knee, etc).
WRT abuse, we're going to require the log-in and user consent before collecting any data and allow people to maintain, in effect, their own medical record. We can monitor for abuse (eg someone who seems to visit a hospital thousands of times / day).
You are looking for the old-country-doctor AI. That is the one that would also catch, unlike this site, my dry throat and itching eyes as probably being me not drinking enough water so far today.
I'm sure that there is a great service behind this link, but I completely ignored it at first because of the "your doctor doesn't want you to see this" message. Seems like the acai (sp?) berry ads that are scattered all over the internet.
Haha. My co-founder and I went back and forth about the right message for our HN title. I completely agree with you but he won the argument. Does anyone know of a good A/B testing site for HN titles? ;)
I think its worthwhile to add some more self-consistency tests to this thing.
You can enter symptoms in the first question, and then move on to the second question. At this point, regardless of the timing selected, some of the symptoms drop off.
In my case, at the end, "What are my chances of having..." reported 0% for the correct diagnosis. This is not good.
What is worse: the condition is in your database and described perfectly there, including the details that would allow for a correct diagnosis.
Did you try starting from each condition, and measuring how often you get the diagnosis right? Correcting for the prevalence of each condition as needed - in my case its a common one so there didn't seem to by any reason not to get it right.
Also, it would be nice if the conditions page would tell you how often this condition happens (if needed by asking age group/sex).
We did a standard 80/20 training/testing trial. We found that the correct diagnosis was in the top 5 of the generated possibilities 50% of the time.
What will be helpful is to update the algorithm from user input. I'd be interested in hearing about your particular case. We use symptom/disease prevalence in the calculations, so it is usually great at getting common conditions. Maybe you can leave a comment using the feedback form on the website and we can look into why it missed your diagnosis?
I would recommend allowing a person to add multiple symptoms up front on the landing page instead of just one. Requiring a postback and then having it to calculate the probabilities under "I may have" after entering only one symptom when the person already knows they plan on entering 2 or 3 more is a waste of resources on the system end as well as a significant delay on the user's end.
That's a fair suggestion. The question is if "rewarding" people with a list of diagnoses from just 1 symptom is worth the cost of resources. My fear is that in the process of having people enter an arbitrary number of symptoms and then having to click a "next" button, they will lose their interest, but we'll test that.
I think "House" should end with him and his team being replaced by a computer. It will need to be a robot (like the one on Lost in Space, ideally), not a web page, but symcat has the right idea. It would also be cool if the robot was a master game-player, finally putting House in his place.
You do know this is the final season of House? Would you rather see them continue to turn Chase into a Mini Me version of House (getting scruffy, painful injury, ignoring protocol) or witness a battle for relevance against a robot with smarts like the Watson computer? I'll take the robot battle.
The UK NHS has a pretty good expert system, like I assume this is supposed to be.
I entered (but do not have) "bloody feces" as a test case, it gave me 4 options for symtoms that they think I meant, none even anywhere close. Hint: if they're black and stinky you need immediate medical help, if the blood is red and sparse it's probably piles - THIS IS NOT MEDICAL ADVICE.
Re: NHS Direct, I've seen it and agree that it's a cool tool. It's pretty much restricted to triage and doesn't give you a great sense of what you have, though. Also, only helps in the UK. :(
There's a good paper on it that explains most people feel that it just directs them to a primary care physician. We'd like to offer value to those people as well and match them to exactly the physician who can help them (ie OKCupid for patients-doctors).
88 comments
[ 3.2 ms ] story [ 153 ms ] threaddesign needs some work
I was surprised at the results. I was disappointed that there wasn't some gentle friendly persuasive advice about going to see a real doctor quickly. Yes, I know there's a disclaimer on the front page, but I feel this is a missed opportunity.
You're going to tap into that hard to reach market of people who might have a serious problem but who are reluctant to see someone. You have a great opportunity to educate them; to let them know that there's probably nothing to worry about but that early intervention is crucial; and that this is what happens at these types of examinations.
But: It's a really cool tool, and I look forward to seeing how it develops!
That's exactly where we're going. We want to direct patients to the right doctor/level of care and are incorporating triage guidelines towards that end.
Thanks!
In it's final result it should make a mention of urgency in seeking healthcare and what to do in the meantime.
True, there has been a lot of work in medical expert systems. DXPlain is an old one that tends to come up a lot in our conversations. We have several Hopkins medical informatics specialists advising us and I would say that our algorithm is novel.
Moreover, past attempts have attempted to codify a medical expert's knowledge (eg let's ask this smart doctor we know to determine disease prevalence). We calculated it from clinical data and can continue to learn as more people use it.
"Application Error
An error occurred in the application and your page could not be served. Please try again in a few moments.
If you are the application owner, check your logs for details."
If you are ill and you don’t know what’s going on, see a doc (or a nurse practitioner etc). Uninsured (yeah - that’s me till 2 months ago.) - then it will cost a packet. Which is good as you’ll be encouraged to figure out what’s normal for your body - which is what needs to happen anyhow.
Remote diagnostics is a hard game even if a doc is doing it. A rules engine just isn’t up to it. It’s not just a game of checklists. This thing might be better (It’s down right now) - but nothing is as good as being in a room with the patient, having some EQ, asking questions, and being able to view symptoms with your own 2 eyes.
I’m probably just thinking that /very/ large sacrifices may be needed, even just for diagnostics.
-- Heavily edited
I tend to agree with your wife (I'm a 4th year med student at Hopkins), which is why I'm building my own.
I think it's easy for websites to just say "if you are concerned, see a doctor" but I also believe that to be the problem. Obviously, if it were costless (time, money, opportunity cost) people would go see a doctor, but they in reality are weighing those costs. We'd like to make it easy for people to weight the risk vs. the benefits of seeing a doctor.
Moreover, for too many people "seeing a doctor" means going straight to the ER. There they wait for 4 hours (nat'l average) and are frequently told to follow-up with a doc anyway. We want to let others know that there are other options (eg urgent care centers) and let them know when such care is clinically indicated.
Learning medicine is all about learning diagnostic algorithms
Although the linked site failed completely (as far as I can tell, because its buggy!), the WebMD that was linked elsewhere correctly diagnosed my condition in less than 2 minutes. It took a real MD 2 hours+ and that included an (unnecessary) X-ray.
I don't think it's comparable to remote diagnosis being hard - that is very dependent on how doctors recall information, and visual information or feedback may be very important to them. An expert system has no such limitations.
We're still building out our recommendations, so you can expect ours to also alert people to see a doctor when appendicitis is likely.
http://symptoms.webmd.com/coresc/landing#/introView
For example, I can click Arm -> Armpit -> Lump or Bulge and see the possible conditions.
Armpit doesn't even exist in your system. Which would be fine if I already knew how to self diagnose a swollen lymph node. Of course, if I didn't, I would still need a doctor to tell me what search term to use on your site.
I can understand if the system is early in development, but if it doesn't do anything for the layman why would my doctor care if I saw it?
The AI component is the fact that we train our algorithm based on clinical data from real patients (started with CDC data -- 500k at the moment and counting as people use the site).
I've had bad experiences with the WebMD symptom checker like most people I've spoken to. Try typing in "chills" and you get "Lyme disease, acne, Bubonic plague" (no joke).
WebMD makes money on advertising, so they unfortunately direct you to the pages that are going to keep you on the site for longer. You may find what you have using their symptom checker, but it is really optimized to keep you clicking.
http://www.ncbi.nlm.nih.gov/pmc/articles/PMC1307157/
http://en.wikipedia.org/wiki/Clinical_decision_support_syste...
http://en.wikipedia.org/wiki/Computer-aided_diagnosis
Who is the intended audience for the symptoms and conditions directories? Are the numbers next to the entries probabilities of having that symptom or condition for the general population?
And not that it's particularly important, but this entry doesn't seem correct: http://symcat.com/conditions/arsenic-poisoning (it discusses bacteria rather than the element).
Honestly, Google. We need to make sure that it is easily indexed and understood by crawlers. Some med students have found it useful though.
Will fix it, thanks!
I put in "tired" and got a list of: depression, diabetes, hypothyroidism, atrial fibrillation,anxiety ,multiple sclerosis, congestive heart failure and gastroesophageal reflux disease.
If I click on "increased risk," I get: valley fever, muscular dystrophy, hemolytic anemia, intracranial abscess, thalassemia, gallbladder cancer, polycythemia vera, and autonomic nervous system disorder.
Nowhere does it ask me if I happened to get enough sleep...
Maybe they need a big button that says "Are you really worried right now?" and if you click it, tell you logical steps and routine you need to be following long-term for the results to "feel more normal" - otherwise to not use to self-diagnose.
It would also seem like a better approach would be to have a prior that's much lower for the internet and is never updated based on user searches, then continue hand-entering real records for quality control purposes. Otherwise, sooner or later some internet community (4chan, SA, etc) will get carried away and start manipulating your system to make everyone think they're dying of elephantiasis.
WRT abuse, we're going to require the log-in and user consent before collecting any data and allow people to maintain, in effect, their own medical record. We can monitor for abuse (eg someone who seems to visit a hospital thousands of times / day).
It's a good title for gossip newspapers. It's not good for HN.
You can enter symptoms in the first question, and then move on to the second question. At this point, regardless of the timing selected, some of the symptoms drop off.
In my case, at the end, "What are my chances of having..." reported 0% for the correct diagnosis. This is not good.
What is worse: the condition is in your database and described perfectly there, including the details that would allow for a correct diagnosis.
Did you try starting from each condition, and measuring how often you get the diagnosis right? Correcting for the prevalence of each condition as needed - in my case its a common one so there didn't seem to by any reason not to get it right.
Also, it would be nice if the conditions page would tell you how often this condition happens (if needed by asking age group/sex).
What will be helpful is to update the algorithm from user input. I'd be interested in hearing about your particular case. We use symptom/disease prevalence in the calculations, so it is usually great at getting common conditions. Maybe you can leave a comment using the feedback form on the website and we can look into why it missed your diagnosis?
It didn't ask me if I was MSM and/or had unsafe sex previously. As such, there was no HIV seroconversion illness mentioned in the diagnosis list.
The recommendation (to see a professional) was correct.
I entered (but do not have) "bloody feces" as a test case, it gave me 4 options for symtoms that they think I meant, none even anywhere close. Hint: if they're black and stinky you need immediate medical help, if the blood is red and sparse it's probably piles - THIS IS NOT MEDICAL ADVICE.
Is this an alpha test?
There's a good paper on it that explains most people feel that it just directs them to a primary care physician. We'd like to offer value to those people as well and match them to exactly the physician who can help them (ie OKCupid for patients-doctors).