I accidentally started a movement – Policing the Police by scraping court data
The idea quickly evolved into a real goal, to make good on the promise of free and open policing data. By freeing policing data from antiquated and difficult-to-access county data systems, and compiling that data in a rigorous way, we could create a valuable new tool to level the playing field and help provide community oversight of police behavior and activity.
In the almost 3 years since the first post, something amazing has happened.
The idea turned into something real. Something called The Police Data Accessibility Project. (https://www.pdap.io)
More than 2,000 people joined the initial community, and while those numbers dwindled after the initial excitement, a core group of highly committed and passionate folks remained. In these 3 years, this team has worked incredibly hard to lay the groundwork necessary to enable us to realistically accomplish the monumental data collection task ahead of us.
Let me tell you a bit about what the team has accomplished in these 3 years.
Established the community and identified volunteer leaders who were willing and able to assume consistent responsibility.
-Gained a pro-bono law firm to assist us in navigating the legal waters. Arnold + Porter is our pro-bono law firm.
-Arnold + Porter helped us to establish as a legal entity and apply for 501c3 status
-501c3 status granted
-We've carefully defined our goals and set a clear roadmap for the future
-Hired first full-time staff.
-PDAP was awarded a $250,000 grant by The Heinz Endowments
So now, I'm asking for help, because scraping, cleaning, and validating 18,000 police departments is no easy task.
The first is to join us and help the team. Perhaps you joined initially, realized we weren't organized yet, and left? Now is the time to come back. Or, maybe you are just hearing of it now. Either way, the more people we have working on this, the faster we can get this done. Those with scraping experience are especially needed. The second is to either donate, or help us spread the message. The more donations, the more data we can gather. I want to thank the r/privacy community especially. It was here that things really began.
TL;DR: I accidentally started a movement from a blog post I wrote about policing the police with data. The movement turned into something real because of r/privacy and hackernews: (Police Data Accessibility Project). 3 years later, the groundwork has been laid, non-profit established, full-time staff hired, and $250,000 in grant money and donations so far!
Scrapers so far Github https://github.com/Police-Data-Accessibility-Project/Scraper... Discord if you would like to join the efforts: https://discord.com/invite/wMqex8nKZJ
*This is US centric
201 comments
[ 761 ms ] story [ 1914 ms ] thread[0] - https://github.com/Police-Data-Accessibility-Project/PDAP-Da...
I don't know whether the goal is to find bad cops who have been fired and get hired one town over and try to prevent those hirings? Or if the idea is to find people who have been arrested and ruin their lives? Or if the goal is simply to make the data available and let people use it for whatever they wish, from deepfakes to erotic fan fiction.
Data wants to be free. The data is the end goal. Public data should be public, etc all sound great. But, I've seen the mug shot database ruin people's lives, and eventually its founders had mugshots of their own.
Scraping and publishing is one thing, but knowing the goals is even more important because it would let me know why we're scraping, what we need to scrape, how to make that available, etc.
What problem are you trying to solve? And Why?
Police data is not easily searchable? Okay, so what? What good is it to make this available? What uses could it have? Even if the goal is just to build it and see how people use it, it would be helpful to know that.
I strongly suggest you consider working backwards. It's been 3 years. What does 3 years from now look like? What is the press release? How do you know you were successful, what does that mean?
Availability is nice. It could lead to transparency. That may lead to accountability. Is that the goal? Right now this data is hard to access. If you succeed it will be easier to access. And so what? What changes in the world could come because of this? If you fail, what lost opportunity do we mourn? You gotta have some sort of easily conveyed reason for doing this. If the data accessibility alone is the end goal, that's fine, not enough for me, but at least make that clear and convince me that's worth the effort.
That said, if you head to https://pdap.io you’ll find the most concrete explanation we have. It still needs to be clearer, but it’s more specific than the docs.
Amazon, for all its faults, has this process well defined. These links may be helpful to you. Good Luck
https://coda.io/@colin-bryar/working-backwards-how-write-an-...
https://medium.com/intrico-io/strategy-tool-amazons-pr-faq-7...
A nice video: https://www.youtube.com/watch?v=aFdpBqmDpzM
Do everyone a favor and go back to wherever you came from and stay there quietly.
wtf are you smoking? Do you think that activism doesn't involve marketing, and do you think that people don't need to market themselves in order to find volunteers? Do you call out charities for advertising, or call out employment ads for sounding like they're desperate for employees?
> You say it's turned into something real but I'm struggling to find it in the sea of marketing.
Did you just ignore the URLs?
Also, I recommend using Playwright instead of Selenium. I'm in the Discord now and will be hanging out, so looking forward to chatting more about this and contributing.
Do you mind giving us brief on what kind of data you are collecting and highlight any interesting findings so far?
I do have a few questions too:
1. Will this scale? One problem with scrapers is that they break when people update their website. I'm imagining this problem multiplied by 18,000 and compounded by each scraper potentially being written by a different volunteer.
2. Where are the scrapers getting run?
3. How do the documents that the scrapers collect get transformed into usable data?
4. It seems to me like a scalable solution would be a standard to report data, a law to compel police departments to follow that standard, and then a system to collect that data and make it available. Do you work with police departments at all on data reporting?
2. You can run them locally. We're not running the scrapers anywhere, or storing extractions anywhere.
3. This is a big, big question. Right now, the answer is dependent on the use case. Rather than trying to make the world's biggest database, we're going to respond to community needs and build this kind of thing as it comes up.
4. https://measuresforjustice.org/ is doing something like this! We're interested in creating incentives for police departments to make their data more accessible and transparent.
What I would expect to see is something like:
1. Here's what data we want from each police department each day. Here's what value you should use to indicate that data is not available.
2. Here's a list of police departments. Write a scraper. If it passes tests to show it's generating valid data, and code review, we will run the scraper in a daily basis storing the output in this database.
3. Here's how you can query our database.
We've only had paid staff for about 6 weeks. We've run several experiments and started from scratch a few times over the years. We've been slowly inching the project forward in our spare time; I was the only volunteer for much of that time, and I can't even code!
1. Sounds great, we're building something like this.
2. Even making a list of police departments is a big challenge. We've made a good start but have work to do.
3. Yep, soon we'll have something to query.
If you wouldn't mind reviewing https://news.ycombinator.com/newsguidelines.html and taking the intended spirit of the site more to heart, we'd be grateful.
Though I come from an uncommon political background and that colors how I interface with and view technology, I do generally feel I've approached the site from a place of genuine, calm, and engaged discourse on the whole.
---
Your phrase 'uncommon background' made me want to write down some more general thoughts for a bit, but just ignore it if it's not of interest.
We don't have any problem with uncommon backgrounds—we welcome them. Conversation gets better when it happens across differences—so long as people can remain curious. The trouble is that curiosity comes under strain as backgrounds diverge, differences increase, and people have less in common. The risk of the connection 'snapping' and the thread degenerating gets higher. This risk is greater online than it is in person, where there are more channels of information to draw on and also more constraints on how we treat each other.
When things 'snap' and then degenerate, we have no choice but to intervene as moderators, not to take a side on the topic, but literally to moderate the kinetic energy that breaks out. The alternative would be to let it destroy the forum, and that wouldn't do any good for anyone.
The person with an uncommon background—the one who holds a deviant or contrarian view, relative to the majority—inevitably comes under additional pressure when expressing themselves. Their risk of being misunderstood is higher, the likelihood of someone showing up to support them is lower, and there's a good chance that they'll attract a flurry of shallow majoritarian responses. This doesn't happen because people have bad intentions—it happens because of statistical mechanics. But it feels like the others have bad intentions; we're not designed to feel statistical mechanics.
When that happens, it's hard not to snap. The person with the minority view, being under additional pressure, often lashes out at the rest in a way that is against the rules of the site and that we have no choice but to moderate. They get labeled as the 'bad' one, but that's not really fair—the snappage is as much a consequence of the pressure differential as of any personal lapse. Most people would 'lapse' in such a situation. It's really a shared problem, but the majority gets to feel angelic while the other holds the bag.
I see this a ton on HN across every sort of 'minority' you can imagine—the obvious demographic minorities, of course, but also a long tail of less obvious subgroups. It's like a massively parallel greatest hits album of social psychology experiments.
Dismayingly often, we end up having to ban the account that lashes out for repeatedly breaking the site guidelines, even while sympathizing with their situation because of the dynamics I've just described. Then often then lash out at the mods for siding against them, accuse us of bias, and so on. In reality we may well personally agree with them, and even if not, we sympathize with their position—but it's not possible to communicate that.
Some of this, of course, is what minorities have always known—they're held to a higher standard in an unfair way. But it's interesting that one can derive this from the mechanical conditions of an internet forum.
The open question is whether there's a way out of the unfortunate tradeoff here, which is that moderating to keep kinetic energy at tolerable levels—that is, moderating flamewars so the forum doesn't burn to a crisp—means favoring the mediocre majority with its predictable views. HN is a good place to look for a way out, because both poles of the tradeoff—flamewar and lameness—are bad for curiosity, and curiosity is the one thing we're trying to optimize for ( ben174 ↗ could you elaborate exactly where the filth is here? I think I get it, but I'm not super-well versed enough to understand what the motive would be here. ALittleLight ↗ Edit: Seems like this criticism doesn't apply to this project. I think that's good. What I wrote below is just an explanation of why someone might view crypto as a red flag in a charity project. josh-pdap ↗ That's one of the many reasons why we abandoned the crypto idea after some initial optimism and research. kristintynski ↗ Nothing about PDAP is web3, or crypto based, though some community members have suggested it (especially last year among all the Web3 hype.) Turning into a web3 company is not happening. wongarsu ↗ It certainly set off my alarm bells for people who try to do pointless blockchain stuff for personal profit (or I guess fun and street-cred/CV-lines). josh-pdap ↗ The post you quoted lists potential avenues for using web3 tools. You may notice we have not gone down any of those avenues. We aren't doing crypto stuff. bb88 ↗ Yeah, I'm glad you went this route. kristintynski ↗ PDAP is not crypto based, and there are no plans to make it so. The message you pasted here was about a meeting taken last year with someone who was proposing this. It isn't something the community or leadership was interested in. PDAP is a non-profit, not a crypto startup. yuan43 ↗ I don't understand the claim you're making in this response. Can you condense it to 1-2 sentences (and cite source w/ link if available)? [dead] ooouups ↗ [dead] multjoy ↗ josh-pdap ↗ I think if a project like ours ever did use blockchain, it would be behind the scenes as a part of the product; decentralization and transparency are key parts of our ethos! However, it's too unstable and as you can see by the comments, mentioning "web3" costs a lot of credibility in many, or maybe most, communities online. jovial_cavalier ↗ Floating the idea of having a DAO governance system =/= "crypto grift." It's a legitimate way of trying to ensure that control of a distributed system remains distributed. It doesn't even sound like they've pursued the idea. You can, in theory, have a non-profit DAO. coding123 ↗ a vast number of people converting to marxism. (not me) dang ↗ Yikes. Other users have pointed out a bunch of things already but I need to add that this sort of name-calling and personal attack* is against the rules and spirit of this site, and is the sort of thing we ban accounts for. Regardless of how right you are or feel you are, please don't do it again. [deleted] ↗ (comment deleted) [deleted] ↗ (comment deleted) coding123 ↗ > pathetic crypto grift elsewhere please. joshenders ↗ Oh wow, that profile I created 10 years ago and have never once revisited has a BTC wallet address. You sure got me. pennaMan ↗ Do you really lack the self-awareness to comprehend how massively hypocritical your series of comments is? And you're double downing on it? It's truly remarkable (in the most negative way possible) what political tribalism does to people... joshenders ↗ 10 years ago, I kicked the tires of a (beta, at the time) identity proving service built by a colleague and added all the fields including a crypto wallet. The assumption that it is for personal “donations” is conjecture. I don’t need your money or acceptance. Thanks joshenders ↗ Also, maybe you should read this https://news.ycombinator.com/newsguidelines.html before trying to dox “hypocritical” witches. pennaMan ↗ Maybe you should read this, I was not talking about your keybase account :) https://news.ycombinator.com/item?id=32903946 joshenders ↗ While I appreciate web3 brainrot backpedaling, I don’t see how this is related to explicit rule violation of this site of personal attacks. pennaMan ↗ >Take your pathetic crypto grift elsewhere please.
---
I think the basic idea is that this web3 crypto stuff is pretty scammy. It would be like finding out the not-for-profit you were thinking about working with also sells timeshares - maybe it is, against all odds, legit, but still not a good look and kind of a red flag.
Another take on it is - if I want to volunteer at a place I don't expect to get paid. If I'm working for payment, I don't want my payment to be in NFTs. So, the web3 intrusion into this idea is unnecessary and doesn't fit for either volunteers or employees.
That doesn't mean it has to be the case here. But at first glance a DOA seems more like a detriment here (police can outspend citizens) and NFTs are NFTs, no explanation needed I suspect. Suggestion 3 might have merit, but storing the data in the blockchain (instead of just some hashes for timestamping) makes it look like some overambitious vanity project again.
We did indeed spend some time looking into web3 and peer-to-peer, and ultimately decided it's not for us. Most of the energy there is spent trying to make money / scam people. Peer-to-peer is cool too but has its own risks.
Thanks for checking out the discord, though!
It's one thing if you want to have a crypto wallet to accept donations -- that's not that controversial. It's another to drink the web3 kool-aid and base your organization's future on that.
https://knish.io/ is a good example of people using blockchain to make real products. I don't know that it's ready though.
Recently there has been an unreasonable amount of hostility towards anything that even mentions crypto or any related technology. I wonder what the source of it is.
If you'd please review https://news.ycombinator.com/newsguidelines.html and stick to the rules when posting to HN, we'd appreciate it.
* especially against a new user—greeting newcomers with a torrent of abuse is really bad
On your keybase.io page you list a BTC donation address.
Triple down on the hypocrisy, now that's impressive!
"Making public information public" is a good tagline too.
Do you know what kinds of work people did with the data? It seems to me one of the best ways to address BART crime would be to support the impoverished and desperate people who don't have any recovery or mental health support, but that work is slow...
Then there's email and other messaging for lower priority things, and phone calls for stuff that is sensitive.
Listening is not illegal, but recording or redistributing "is illegal", however, it's not clear whether it's actually illegal or it just depends how and when you use it and what you do with it. There was a kid here that brought it up with the police and was harassed over it, I believe he made a website to broadcast it over a web page and was given a stern telling off. Which tbh is fairly valid as it has horrific amounts of PII in it.
To best mimic the success of the original post, it would be good to be able to identify people doing the most damage to a neighborhood so we could help them first with tax relief, food, birth control, whatever would help the most.
That would be the police.
Yes, but the fact that we have to monitor them proves they aren't up to the job. The people need to do this, and we should use scraping data to prevent _ALL_ crimes.
My point about stopping ALL crime with data is a very clear statement that has validity and merit.
Without understanding the problems and ways police harm communities, throwing data at a problem is unlikely to result in good outcomes. Through the lens of “crime” harm is hard to separate from illegality. Driving with expired plates because you can’t afford it until next month is illegal, but it isn't causing the same amount of harm as burying that person under court fees and fines until they’re homeless. Unless you collect the right data, understand the data's meaning, data is useless.
Some starting places on harm police do to communities, if you’re interested:
- podcast: season 3 of Serial https://serialpodcast.org/season-three
- article: https://www.huffpost.com/entry/over-policing-of-america_b_44...
- book: the end of policing https://en.m.wikipedia.org/wiki/The_End_of_Policing
What would be an example of a core data set they are trying to compile? police involved shootings? police budgets? everything.
> Our mission is to make data from every U.S. police agency accessible via a single public resource.
More precisely:
> There are over 18,000 police organizations, and each has a unique way to publish information.
i.e. Police are publishing data? Let's organize it and make it easily accessible.
"Police data" is incredibly vague. What types of data are most interest and available from most of every agency?
If the only answer to what kind of data is "police data" then I'm not sure if I should care, support, or contribute.
Is this data on how many toilet paper roles departments purchase or police involved shootings? neither?
Certainly you can see where the question is coming from right?
That said, we still have work to do pointing people in this direction and helping them understand why. This whole thread is going to really affect our website and docs :)
I have gone through some of the public info for my department so I am curious.
Are police publishing arrests by time and geolocation? Quarterly stats by call or response? Then yes.
Hell, I'm sure if they are publishing coffee consumption numbers, that would go up as well.
I obviously don't know, and neither do you.
I don't know why you feel the need to weigh in on my question with unhelpful non-answers.
TL:DR; If you want to write scrapers: go for it! Run your scraper, share the results in Discord and with your friends, and talk about the process. We'll be listening, and it will help us build tools to support this important work.
A few things to clarify:
a. The source of truth for "what are we doing right now" and "how can I contribute" is https://docs.pdap.io/.
b. Empowering people who write scrapers is a part of our broad mission of "police data accessibility", but we have some foundational work to do first! Right now our primary project is creating a database of police agencies and data sources. This will help people understand what kinds of data are available, at which agencies, with which steps to access it. It will also help us create archives of the primary sources, so that if they get taken offline we can still go back and scrape them.
c. What we have realized in the past few years: there are already a ton of people writing and using web scrapers for their day to day work. They are as decentralized as our police system. Our scrapers repo will reflect that. We shouldn't all rely on one library, or even one language. The people who need the data are most motivated to maintain scrapers, and we expect that maintenance will be ad-hoc and as-needed for the immediate future. In most cases, data already published on the internet is useful to local users as-is.
d. If you have a question you'd like to answer about the police, here's the investigation process:
1. Determine whether public data exists to answer your question. Use google to find the appropriate agency, and see what they're publishing. 2. Determine how it can be accessed; do you need to make a FOIA request? Is there a URL? 3. If there's a URL, determine whether you need to write a scraper to access the records. Often, the records can simply be downloaded. 4. Write and run a scraper, if you need one! 5. If there's not a URL, make a records request for the public information. This is a long and complicated process. 6. Share the data with your friends.
This means that scrapers are helpful and necessary some of the time; but not always, and not as the first step. We're trying to help with steps 1, 2, 3, 5, and 6. The theory is that writing scrapers is something people can easily slot in and help with; and that, depending on what question you're trying to answer, two scrapers for the same data source might look wildly different.
Scrapers are an important part of the ecosystem, but they're one piece of the puzzle.
Josh, I can't reply to you directly, but yeah... you can see the comments, but most people can't, unless they have enabled "showdead" in their preferences.
Not sure if this is some false positive on anti-spam (new account, lots of comments in a short time on a single story) or if it's someone with a vendetta. @dang this is a little weird...
(I know, this site doesn't have magic @, but also i think the site moderator probably has scripts searching for mentions/summons anyway. I would anyway...)
* https://news.ycombinator.com/newsfaq.html
> What does [dead] mean?
> The post was killed by software, user flags, or moderators. Dead posts aren't displayed by default.
I suspect that it is a false positive. Maybe email the mods for help? hn@ycombinator.com
[0] https://news.ycombinator.com/newsfaq.html
You say in your FAQ "We aren't a watchdog—our activism is data collection and accessibility, not analysis or research."
Can you note any instances of other people using your data for analysis or research?
They're not cleaned up and ready for release, but I could do that if it's useful to the project.
At least a few more counties in my state use the same software.
Most of our scrapers are written by inexperienced volunteers so this is probably still quite valuable as is. If it works, we’ll take it.
You could also raise a hand as a human data source; perhaps one day someone will be looking for jail data from these counties and you could offer up some way to contact you. I gave my contact info to the person you replied to. Use it if you want!
I have it republishing on a very basic site with an easier way to view the arrests.
The software I'm scraping is called "Justice Solutions" (findtheinmate.com) and it looks like it was built without any knowledge of design OR usability.
Could you share what you’ve made somehow? josh.chamberlain@pdap.io or discord are likely the best way.
Would also be curious to hear what made you take this on.
I'm not here defending the police, or denigrating the project, just playing devils advocate. What happens if the police just ignore you?
I’d like to aim higher than a quotable statistic for a politician, in any case :)
We've come a long way since that post in terms of strategy and focus. Most of that time was spent with between 1 and 3 volunteers, working a couple hours a week.
Transparency is a good goal in itself, I think. People are already using this public data, we're just trying to make it more accessible.
"Policing the police" was the original phrase used on reddit, but if you look at our website (https://pdap.io), that's not a phrase we use.
This seems to indicate the data will always and only be used to tell a preferred narrative.
Despite the downvotes your example is good I think it's safe to say the 95% male to female ratio is likely to be down to males more likely to be involved in violent incidents than females. No one really has a problem with this until skin colour comes into it. As a society though tackling the cause of why males get into violent confrontations seems like a no brainer.
Why isn't it plausible that different groups are different in some ways? For instance, when it comes to testosterone, this seems to be the science.
https://www.ncbi.nlm.nih.gov/books/NBK20759/
> In fact, African American men have higher exposure to testosterone, the main biologically potent circulating androgen, than their Caucasian and Asian counterparts, beginning in the in utero period. African American women have testosterone levels that exceed those of Caucasian women by 50% or more in early pregnancy, an exposure that has been hypothesized to permanently alter the “gonadostat,” the hypothalamic-pituitary-testicular axis, in African American male offspring relative to Caucasians. African American men during young adulthood also have substantially higher circulating testosterone levels than their Caucasian counterparts (approximately 13 to 15% difference at age 20 years). Although this difference appears to dissipate with age, African American men still have slightly higher testosterone levels than Caucasians (≈3% higher) at age 40 years.
For example, here's a larger study showing NO difference in testosterone between black and white men: https://pubmed.ncbi.nlm.nih.gov/17456570/
Conversely, I've never heard of a single study anywhere claiming that women have more testosterone than men. QED
I guess the question is why people don't think that correlation holds true by race (or culture) as well. The percentages match up. For instance, white males (and black males and hispanic males) are actually over-represented as demographic cohorts who are victims of police shootings, whereas asian males and women of all races are under-represented. This tracks exactly with violent crime rates.
Perhaps we should address this a gender problem more than a racial problem. At least, that's what the data tells us.
The National Crime Victimization Survey says yes. Also any article you see trying to debunk FBI crime stats but doesn't mention the NCVS (and how the NCVS largely corroborates the FBI stats) is either ignorant or willfully deceiving you.
I have used it in different levels of govt to map managers to financial line items, to applications, corporate entities, projects, contract counterparties, platforms, techs, machines, ip addrs, vulnerabilities, etc. Developing a clear and addressable ontology of huge organizations with tens of thousands of people and devices is probably my one of my more useful skills. The main use case for graphs to me is patchy data, where you have a pile of incomplete metadata in dispirate spreadsheets and you need to find coherent paths through all of it.
I won't be in touch because I know what those people are capable of, but if graphs haven't accelerated your work already, you have some really epic times ahead!
When you say “I know what those people are capable of” do you mean something as ominous as that sounds?
It was a lot of work to find data on policing nationwide, because the question really was "Is the sheriff doing a bad job, or do bad things happen sometimes?"
After some hard work trying to identify other cities with similar socioeconomic circumstances and populations, it became clear that our local sheriff was actually better than average, and that much of the outrage was fabricated.
That's also when I learned that many people don't want to listen to statistics unless they agree with their own preconceptions.
This has been my experience with bodycam footage, I've found that there's been quite a few heavily protested police involved shootings that when looking over the footage and the facts of the situation, were by the book and completely justified, yet no matter how many times you say to someone "you do know there's footage of the entire event, uncut and unfiltered", it doesn't seem to matter.
EDIT: I just remembered what my throwaway username is.
https://www.youtube.com/c/PoliceActivity/videos (For most of the videos, which show people dying, you must be logged into a youtube account to pass the "violent content" prompt)
Maybe 9 out of 10 videos are "good kills". Officer says something like "hey, stop committing a crime", the suspect says something along the lines of "fuck you pig!", pulls a gun, and is swiftly shot to death by the 50 SWAT guys surrounding him. These videos get around 100K views, and the comments are full of hooting MAGA types.
1 out of 10 show an officer doing something completely inexcusable, (Shouting "I love being racist!" and then hitting an infant with a baseball bat) and the video has 100M views.
The tens of thousands of comments then act out the exact same conversation every single time: the red team says something like "but it happens very rarely" and the blue team is outraged that it happens at all. They don't want the rate to be low, they want it to be zero. This is how you end up with blue-aligned media earnestly and in good faith calling for the police and all prisons to be abolished: https://www.nbcnews.com/think/opinion/abolishing-police-pris... https://www.newyorker.com/news/our-columnists/the-emerging-m...
I say "conversation" up there, which is of course incorrect. They're talking right past each other, since they come from two separate and completely divorced epistemological universes.
I'm not sure what the problem is with that. Of course mistakes happen but it should be the goal.
The red team also seems to have less of those "oopsie I just killed you" moments, so perhaps that might color things there.
And then you go on to the far end that calls for abolishing police/prisons and make that the general team blue zeitgeist. There's low hanging fruit of reform that needs more focus, e.g., not using police for mental health emergencies.
Anyway, your framing of the situation as a non-conversation is kind of meta -- it feels like you talked right past the problems.
My point was that the comment about divisiveness between teams red and blue in this regard was divisive as well in how it was framed.
Police are granted incredible powers and have historically not had the best oversight, let alone relations with "minorities".
There are real problems and they can be addressed but that will never happen as long as we can't even agree that the "what" exists, let alone if the "how" is correct.
There's obviously violent scenarios that can only be addressed with violence, but bringing that on should be a last resort.
And for that to happen we need to understand it.
And for that to happen we need to talk about it.
Sure, you can dose him up with Haldol and solve that particular emergency. What happens when he does it again next week? And the week after that?
Persistent severe mental illness is interesting in that it's a new problem. For all of human history, severe mental illness was a death sentence. Either you'd get beaten to death by your largest neighbor for acting weird, or your village would exile you and you'd starve to death in the forest.
The current compromise we've arrived at is that being crazy is not actually a crime, so you cannot be permanently imprisoned for it. So low-functioning schizophrenics cycle through mental hospitals-- they climb a lamp post and urinate on passers-by, get committed and then are put on antipsychotics, sober up, survey their surroundings and rationally conclude that inpatient mental health facilities are really awful places to be, (basically no rights or privacy, surrounded by crazy people, can't wear or handle any object that's on a very short list of approved suicide-incompatible things) and then check themselves out, which they are legally allowed to do. Then they go off their meds, (also legally allowed) and it's back to the lamp post...
This continues until they accidentally walk into traffic, or are shot by a cop. It's a miserable compromise that persists because all the alternatives are even less palatable.
It should be cheaper than prisons and more humane as well. The homeless crisis begs for this.
Just got to work out trivial details like how to not have that power be abused...
The correct answer is to legalize, tax, and regulate all drugs and to create "mental health courts" that can administer the involuntary institutionalization of those that need it.
What should the goal of the police department regarding killing innocent people? Should they aim for a dozen per quarter? Per-capita weighted?
Incremental changes have been hard because it gets down to the police policing themselves, and that has shown to be a failure.
Again, one of the "simplest" reforms would be to have other professionals deal with mental health crises.
A bigger reform with much more value would be ending the War on Drugs but that's way higher up the food chain. It is germane though, in that the whole point of said war was to give police more opportunities to oppress "others" (i.e., "minorities" and "hippies").
All of these things have a pattern in common where the majority of the improvement can be had reasonably cheaply and capturing the last small improvement has huge costs that are usually much more costly than the benefit of eliminating the last bit and the population of potential events is huge. Road deaths are an illustrative example. Adding speed limits, traffic lights, crosswalks, moderate enforcement, etc. all dropped road deaths precipitously in the USA. If the USA changes its model from cars first to people first there are still some moderately costly infrastructure changes they could make to get their numbers down to Netherlands levels, which are still not zero. Beyond that you are left with hugely costly heavy enforcement among steadily more dystopian and invasive methods that will likely improve the number but not get it all the way to zero. It is really unlikley that these last improvements cost less than the benefit they incur between the direct cost of implementation and the negative effects on the total population of driving events.
On top of all this all those large costs you are incurring for very small gains come at the expense of being able to spend those limited resources on other things that likely have better cost to improvement ratios. You are foregoing a larger decrease in something else bad for society for small decreases in your target. In the road deaths example above that final small drop in road deaths might be coming at the cost of large drops in crime if you deployed those police there instead of on road deaths, as an example.
Maybe your writing was not clear enough, m'kay?
And now you play word games with my intent. I did not advocate for extreme measures -- simply to have clear targets of and rules of engagement; to identify workable solutions.
You're manufacturing slippery slope reductions of a concept and therefore dismissing the concept.
What should the goal of police departments of pet dogs killed by cops? Should they aim for 10,000? A million? Just a handful? A logical answer is "we want to avoid killing any pet dogs in the course of service" (also known as zero).
Now they could take absurd actions to avoid that (leaving the scene whenever a dog is present), or they could add that to standardized training so that they're better prepared to deal with that situation.
So without providing real-world examples of you concerns you are just offering up florid conjecture.
https://en.wikipedia.org/wiki/Adrian_Schoolcraft
It seems to me that the only way to explain this is the lack of publishing of damaging videos by the youtube account -- i.e. the usual thing you need to pay attention to in the social sciences -- a) who chooses to release the data, b) the conditions under which they choose to release the data.
On the side of the YouTube channel -- based on the patreon channel, it seems to me that they are most likely not part of the police, or the judicial system. Thus they must obtain body camera footage from freedom of information requests.
This means that that information request is subject to filtering on the side of law enforcement, who can almost arbitrarily choose what videos they want to release -- while there are guidelines present, they really only apply if you can prove them being broken, which... due to the nature of FoI requests... you can't...
So, just as a baseline, legal advice holds that data that is currently being pursued legally should be held off from release to a general audience, and court cases can take many years to be processed. So it seems to me that that is one main reason that videos that show misconduct by police offers would be failed to be released, or would otherwise be redacted.
On top of that, you have whether officers themselves working in the police administration care to release the footage. It seems reasonable that an officer may be subject to workplace-based social pressure, and not wish to release footage of wrongdoing by one of his coworkers, it also seems reasonable that in some cases, they might indeed feel departmental pressure to not release footage that displays such wrongdoing, so that the department as a whole does not come under flak. You have cases in the UK where officers themselves deleted videos that would prove wrongdoing on their part[3]. Either way, this is inherently impossible to prove either way.
And then you have whether or not the police officer was recording at all, regardless of what regulations state. There have been a few cases recently where police officers thought that their body camera was off, and used that time to break the law[4][5]. Indeed, in some states, it's entirely up to the officers whether to turn them on in the first place, based on what they consider as "an incident".
And then finally, as a youtube channel accepting donations, they are heavily incentivized to draw "engagement" and game the algorithm, so what they release is not just going to be based both on the political opinions of those within the organization, but also will heavily cater to whatever established audience they have, to ensure that each video is liked and that they gain subscribers, so they can drive donations to keep on doing what they are doing.
So to me it seems that this isn't as nearly cut-and-dry as you assume to think that it is. At the very least, a random youtube channel that releases police video, cannot be thought of as a proper or correct sample from which to draw correctly proportioned information from -- as we can see, there are many reasons why it would misrepresent the number of cases of each involved. While research in this area is perhaps uncomfortable for people to accept, it broadly shows that police -- at least how they behave at the moment -- are universally flawed. I myself would prefer to trust the data.
[0]: https://twitter.com/equalityAlec/status/1571898316295643136
[1]: https://resistancelab.network/our-work/taser-report/index.ht... (Disclaimer: I know some of the people who worked on this. Interestin...
> They don't want the rate to be low, they want it to be zero
Well, yes. "Thou shalt not kill" does not have an "unless you're a cop" footnote. There is no acceptable number of your children you would tolerate the shooting of.
As someone that cannot even vote in the US (and is therefore neither red nor blue) - that seems _eminently fucking reasonable_. This kind of thing just does not happen in the UK.
I'd wager there have been fewer police killings (let along just shootings) in the past two decades in the UK than just this weekend in the US.
For anyone that actually believes in small, limited government, the idea that government agents can wander around shooting effectively at will is so ridiculous that anyone claiming to be "red team" should hang their heads in shame.
https://www.independent.co.uk/news/uk/home-news/chris-kaba-p...
>Chris Kaba: Protests held across UK after unarmed black man shot dead by police. Crowd of more than 1,000 demonstrators brandished ‘fight racism’ signs following the death of unarmed 24-year-old
Maintaining scrapers for 18k county websites and PDs is no small task and looking through the docs for PDAP, it seems like this is still a very open question.
The 80000 Hours podcast has an interview with the (non-technical) creator of OWID. I seem to recall some interesting stories about them getting emailed PDFs with COVID data and such.
I had the same question as you, and I was hoping to find ideas in the comments. It seems like the kind of thing that's both inherently messy and scrappy yet if you don't get at least somewhat organized it can't scale.
Update: link to the podcast episode page with quotes, transcripts, etc. https://80000hours.org/podcast/episodes/max-roser-our-world-...
It's interesting that even one of the largest still uses manual execution for almost all of their pipelines (at least in the covid data project[1]). This [2] seems like the bulk of their data importers (scrapers) but most are still operating as manual jobs. I guess with open-source data work, hours and minutes don't matter as much, and being a few days behind the latest data is acceptable.
1 - https://docs.owid.io/projects/covid/en/latest/data-pipeline.... 2 - https://github.com/owid/importers
Here’s the pitfalls I see you falling into:
(1) seriously, what data are you collecting? “Everything” isn’t a great answer (who’s supposed to use ‘everything’, anyway? “Anyone”?). “Apples-to-apples police misconduct statistics” is a good one.
(2) it’s important to clarify 1 because you need to know who you’re serving, and why. Different activists need different data. “Have all data” sounds good until you need to decide how to allocate your resources.
(3) more deeply, data is the land of edge cases. Even just with police misconduct, you need to get DEEP to rigorously compare seemingly-simple stats like “# of unjustified police killings”. If you don’t start narrow, you’ll never show value. If you don’t show value, nobody will ever care you exist.
When I look at the data you’ve collected, it ranges from annual reports, to municipal contact info, to crime stats. What’s important to collect at scale? To whom? What do they need it for?
Again - great, ambitious idea! But $250k goes fast. Show value before it runs out!
1. Agreed. Our strategy for this isn’t clear on the website, I guess, but we do have one. It’s to focus on depth in geographic areas. This is because context is critical, and because most of the users we talk to are operating locally with municipal or county level data. So it’s more important to have every data source we can possibly find relevant to Pittsburgh than it is to have every arrest record in every municipality. Or at least, it’s more immediately useful to people.
That said, most people seem to contribute data sources from where they live. I think little microcosms will spring up where people take stewardship of maintaining information about their chosen geo or subject areas. Not too far down the roadmap, Milestone 2 for the PDAP heads.
2. I will take it as a next step to make this strategy clear and say why. We want to basically allow the community to make its own to do list: what kind of question are you trying to answer? That creates a “bounty” for data which can be fulfilled by an altruistic volunteer, another member of your team, etc.
3. Yes. We’re not trying to do apples to apples comparisons of departments yet, partly because it’s so absurdly difficult and you don’t know where to start. Why would you undertake a 12 hour research project to compare St. Louis and Minneapolis incident reports if you don’t have a use case? Instead we’re focusing on what we DO know we need: complete local data, town by town / county by county.
The data we collected reflects the nature of our early experiments, which were scattered. This airtable prototype is maybe 2 weeks old, next up is helping people understand where to focus.
The idea for demonstrating value is also local. I’m working with groups in Pittsburgh (where we are based, and where our funding came from) to make ourselves indispensable to them. I’m hoping to turn the $250k into a handful of killer local case studies in this year, rather than marking 0.1% progress toward a national vision.
Thanks again for giving me the practice explaining this stuff. I hope I’m making any kind of sense, and of course happy to hear where I’m still wrong.
Maintaining thousands of scrapers for different formats seems like a nightmare, and it won't take long for departments to learn they can slightly tweak the format of their reporting to cause extra work for you.
On the plus side, working with all this data probably makes you all very qualified to advise on developing standards.
Measures for Justice is working on developing standards: https://measuresforjustice.org/
I could imagine a revolving door between people working in the regulatory bodies and the industry they regulate.
So far this year, 177 LEO officers have died in the line of duty. Our gratitude should go to all.
https://www.odmp.org/search/year/2022
Uber reports 59 crash-related driver deaths in 2018, and that's just one gig economy company.
Also note US Postal workers suffered 5800 dog bits in 2020. Note this does not include other delivery services.
https://www.jdsupra.com/legalnews/dangers-delivery-drivers-f...