Excellent, thank you for creating this. Just learning that this data is available. Only California? Are the other states not making the data available in the same fashion as CA?
Also, was there a personal trigger for you to create this?
Thank you. This data should be available in all states, but each state has a proprietary site I have to write a new scraper for and write a new data pipeline for. I'm thinking about open-sourcing the project and gathering community support for the rest now that I have a good pattern set.
Yes, I was harmed by a doctor and the medical board dismissed my complaint. This made me curious about what the board actually does take action for, but this wasn't searchable because all that was available were PDF scans that weren't digitized/OCR'd. I also noticed some documents that were referenced in doctors' profiles had disappeared. As part of this, I archived every PDF available on the wayback machine to protect against that going forward.
I also want to give credit to a similar site that I took inspiration from called 4patientsafety.org, which did some of the same stuff I have here, but completely manually. I chatted with the creator of that and he agreed making an automated version would help because they fell behind updating it due to how intensive the manual process was.
Is there a link to the community where this is being developed? I was unable to find one on the site itself. I'd be interested in writing a scraper for my state and the digitization pipeline.
Thank you. I'm going to refine the site and data a little more over the next week and maybe do a second state myself to make sure my pattern generalizes well before opening up the repo, but I've now made a discord https://discord.gg/bXdk4CuR7B for anyone interested in contributing.
Thank you for offering to help. I'm going to refine the site and data a little more over the next week and maybe do a second state myself to make sure my pattern generalizes well before opening up the repo, but I've now made a discord https://discord.gg/bXdk4CuR7B for anyone interested in contributing.
> I archived every PDF available on the wayback machine
How did you tell archive.org to store the PDFs? It doesn't seem to have worked in all cases, since when I went to check out the single "order vacating automatic cancellation of certificate", there was no snapshot yet.
I have to admit I missed a few hundred out of about 18k PDFs. I didn't keep track of which ones I archived on my initial pass, so I need to rerun the whole script which takes about a week with Web Archive's rate limit, so I've just been putting it off. I was also thinking people who visit them can click the "archive" button in cases it's not yet archived to crowdsource the effort, like you seem to have done.
I have done a lot of research in medicine due to my poor experiences with even top specialists. Ignoring symptoms, denying side effects of meds I took, even though those were present on the label.
Do you have a discord?
Any plans on allowing people to comment on physicians?
Any plans of adding how much doctors have received from corporate interests?(as this data is available)
Sorry to hear you've also had poor experiences. I too had to delve into Pubmed to figure out what was going on with me after the doctor harmed me.
Thank you for offering to help out. I do want to refine the site and data a little more over the next few days and maybe do a second state myself to make sure my pattern generalizes well before opening up the repo.
Thank you. I'm going to refine it a little more over the next week and maybe do a second state myself to make sure my pattern generalizes well before opening up the repo, but I've now made a discord https://discord.gg/bXdk4CuR7B for anyone interested in contributing.
Thank you. I'm going to refine the site and data a little more over the next week and maybe do a second state myself to make sure my pattern generalizes well before opening up the repo, but I've now made a discord https://discord.gg/bXdk4CuR7B for anyone interested in contributing.
Open sourcing this would be a great idea and a lot of people would happily contribute, it already looks great. The biggest secret is that most doctors do more harm than good when a patient has anything worse than a cold. The psychopaths who couldn't make it as CEOs or serial killers end up becoming doctors, in charge of people's health.
Thank you. I'm going to refine the site a bit more over the next week and maybe do a second state myself to make sure my pattern generalizes well before opening up the repo, but I've now made a discord https://discord.gg/bXdk4CuR7B for anyone interested in contributing.
I also partially read a random one [0], about a doctor who had his license revoked for 3 years. Tearing a patient's nerve because he lost a tool and other worse cases (page 17)
>Respondent admits that there were billings for 35 treatments apiece for two undercover law enforcement officers when each had only five visits. Further, he acknowledged that had he known the true facts, he would have investigated further. However, had he been vigilant, respondent does not believe that he would have detected the problem because patient signatures, as evidence that the patient received treatment, were forged after he issued his reports.
I didn't realize that undercover agents were/are doing sting operations for insurance fraud. (And I don't understand the part about forged signatures...)
I've read some others with undercover agents when federal money is involved in the care like the patient using Medicaid and the doctor repeatedly uses that as a ticket for billing unnecessary services.
This is a fantastic tool, I have no doubt that it could help de-obfuscate the misconduct side of medicine, which most people have no idea even exists. Thank you for working on this!
One pet peeve, though. Some entries have "Arbitration Award" listed as an action. However, these usually mean that the physician won an arbitration case (i.e. the ruling was in their favour). Listing them without any differentiation can potentially negatively bias a person's opinion against a perfectly honest doctor.
This may be a character flaw on my part, but I’d be a little bit worried or at least curious about a doctor who’s had a lot of arbitration cases relative to their career length, even if they’ve won them.
Thank you. Here [1] is the source for one, which states "An arbitration award is a payment for damages and does not necessarily reflect that the physician's medical competence is below the standard of care. The Medical Board reviews all such reported arbitration awards and action is taken only if it is determined that a violation of the Medical Practice Act occurred. The Medical Board is prohibited by law from releasing a copy of the arbitration award report or any other information concerning the award." I am still unsure if that means the award was in the doctor's or patient's favor, and because the report is secret, it may not be possible to find out, but please let me know if you know how I could find out.
I also want to mention that patients are forced into arbitration via the contract for service they initially sign [2] that many don't read (whether due their inability to understand it or time constraints on needing immediate care and limited availability of doctors). However, I think it's a shady practice that hurts patients [3] and it would be worth keeping the data showing doctors taking advantage of it regardless. The main excerpt from the article supporting my thoughts is "Doctors’ attorneys are more likely to develop relationships with the arbitrators, they say, who, in turn, may give the plaintiff only a token award to get continued business from doctors."
Something patient-initiated should always be possible. Already patients take to Yelp and Google Maps to leave reviews, but they are often taken down by the doctor/business. I've enabled comments on doctors' profiles as a first step toward this, but you make me think I could improve it by allowing submission of new doctors worldwide.
2. Do you think this is because the URL stores the filter params, so every time you change a filter it's a new address? Do you have any examples of similar sites and how they behave? I can change this if it's not standard.
Thank you! This is such a useful site! Probably only possible in US since any other country restricts access to such records only for law enforcement which is a shame.
Thank you. I agree that even though in the US this stuff is not as available as it could be, it still beats most of the world. That's why I think allowing for patients to report the information is so important. I've taken the first step toward that by enabling comments on each doctor's profile.
This is great to see. Some feedback: I entered the exact name of a doctor I knew would be in the database, formatted as "firstName lastName". The result was buried under 20 other less relevant matches. I had to search as "lastName, firstName", for it to come back as the first result.
Thank you. I'll fix asap. The data source only gives me the names as complete names in "last, first middle" format, and I didn't want to take risk parsing them since there are inconsistencies, but I'll work it out. For now, I've clarified the format in the placeholder of that field.
Yea, a more exact search that can parse out first name and last name would be the way. If I search for JOHN SMITH, and there are actually none in the database, I think I'd rather just see no results than 50 different Johns, none with last name Smith.
Thank you for the recommendation. I plan to enable geo-filtering via the map. I'll need to figure out what is the best granularity to do so at—maybe county. I might be able to give an option for the granularity too.
(commment: pity all states don't publish this in machine-readable formats. Id say this is more important than the monthly food report in the nnewspaper.)
From the other comments it doesn't look like the data source is machine readable. It looks like there's automated OCR of PDFs. A similar project was doing the same but with human labor
There is movement on the price/cost front. I found one tool that provides this data in a more searchable way (turquoise.health) and used it with limited success. I believe the upstream data is public as well, but cant remember the name of the repository OTOH.
I marked the map feature beta because I noticed the geocoding wasn't accurate for all addresses. I'll get on that. I'm hesitant to just exclude ones not in the US though because it's possible a doctor has a California license while primarily practicing in another country, just like that case.
In that case I think it's a confusion over "Jerusalem" in her address and "Jerusalem" in the name of the tagged place, which is "Chapelle Notre-Dame-de-Jerusalem".
65 comments
[ 4.0 ms ] story [ 148 ms ] threadAlso, was there a personal trigger for you to create this?
Yes, I was harmed by a doctor and the medical board dismissed my complaint. This made me curious about what the board actually does take action for, but this wasn't searchable because all that was available were PDF scans that weren't digitized/OCR'd. I also noticed some documents that were referenced in doctors' profiles had disappeared. As part of this, I archived every PDF available on the wayback machine to protect against that going forward.
I also want to give credit to a similar site that I took inspiration from called 4patientsafety.org, which did some of the same stuff I have here, but completely manually. I chatted with the creator of that and he agreed making an automated version would help because they fell behind updating it due to how intensive the manual process was.
How did you tell archive.org to store the PDFs? It doesn't seem to have worked in all cases, since when I went to check out the single "order vacating automatic cancellation of certificate", there was no snapshot yet.
https://web.archive.org/web/20231009140931/https://www2.mbc....
I have done a lot of research in medicine due to my poor experiences with even top specialists. Ignoring symptoms, denying side effects of meds I took, even though those were present on the label.
Do you have a discord?
Any plans on allowing people to comment on physicians?
Any plans of adding how much doctors have received from corporate interests?(as this data is available)
Thank you for offering to help out. I do want to refine the site and data a little more over the next few days and maybe do a second state myself to make sure my pattern generalizes well before opening up the repo.
I've now made a discord here https://discord.gg/bXdk4CuR7B for anyone interested in contributing.
Comments on doctors should already be enabled via Disqus if you scroll to the bottom of a profile page.
Yes, I definitely want to add that data!
And the higher volume Open Data Portal dataset: https://data.wa.gov/Health/Health-Care-Provider-Credential-D...
https://web.archive.org/web/0/https://www2.mbc.ca.gov/PDL/do...
On another note, there are some really interesting ones like this one [1] starting on page 9 which reads like crime novel.
[1] https://web.archive.org/web/20230926165052/https://www2.mbc....
[0] https://web.archive.org/web/20230925064351/https://www2.mbc....
>Respondent admits that there were billings for 35 treatments apiece for two undercover law enforcement officers when each had only five visits. Further, he acknowledged that had he known the true facts, he would have investigated further. However, had he been vigilant, respondent does not believe that he would have detected the problem because patient signatures, as evidence that the patient received treatment, were forged after he issued his reports.
I didn't realize that undercover agents were/are doing sting operations for insurance fraud. (And I don't understand the part about forged signatures...)
I too was harmed by a doctor and I'd really like to see this cover all states in the USA. I signed the linked petition on change.org.
One pet peeve, though. Some entries have "Arbitration Award" listed as an action. However, these usually mean that the physician won an arbitration case (i.e. the ruling was in their favour). Listing them without any differentiation can potentially negatively bias a person's opinion against a perfectly honest doctor.
So you might want to examine a doctor's record vs the median for that kind of doctor...
I also want to mention that patients are forced into arbitration via the contract for service they initially sign [2] that many don't read (whether due their inability to understand it or time constraints on needing immediate care and limited availability of doctors). However, I think it's a shady practice that hurts patients [3] and it would be worth keeping the data showing doctors taking advantage of it regardless. The main excerpt from the article supporting my thoughts is "Doctors’ attorneys are more likely to develop relationships with the arbitrators, they say, who, in turn, may give the plaintiff only a token award to get continued business from doctors."
[1] https://search.dca.ca.gov/details/8002/G/77959/9df8d122bd5b5...
[2] https://law.justia.com/codes/california/2007/ccp/1295.html
[3] https://www.latimes.com/archives/la-xpm-2000-apr-24-he-22804...
1. When I search by "Malpractice Settlements", I get a "Minified React error #31" on every result I click on.
2. You're doing something horrible with the back button - I clicked on one result and had to click the back button 10 times to get out of your site.
1. Fixed
2. Do you think this is because the URL stores the filter params, so every time you change a filter it's a new address? Do you have any examples of similar sites and how they behave? I can change this if it's not standard.
(commment: pity all states don't publish this in machine-readable formats. Id say this is more important than the monthly food report in the nnewspaper.)