Also it's not unconstitutional in principle, but the range where it would be allowable is fairly narrow. If I understood it correctly, it would be permissible in similar circumstances as those that allow wiretapping
The only problem with Minority Report (aside from torturing the precogs, I mean) was treating the prevented murder as a crime and punishing it. If they had just used it as an intervention tool ... well, there'd be no movie.
Editorialized title unfortunately. The legislation of two German states to allow automated data analysis ... is unconstitutional. They'll need to narrow the scope and specify thresholds when to apply the data (and will probably do well to limit application and storage to not run the risk to have it struck down again).
It's not a broad decision that automated data analysis for crime-prevention is unconstitutional.
This is such a German judgement. So the police are using big data to predict who will commit crimes, which is sort of dystopian. But what the court finds unconstitutional about it is that they're using computers.
"Automated data analysis or interpretation requires justification under constitutional law."
I'm sure something is lost in translation. But even assuming a lot of context, I find it hilarious. Germany's highest court seems to have stumbled into avoiding a dystopia not because they don't want some biased ML model bullying minorities, but because they plain and simple don't like all this computer nonsense. (Grumble grumble.)
Requiring humans to sift through the data makes it easier to limit its use.
You either need a lot more budget (= oversight) to hire all the people to do the work, or do less of it.
Sure computers also cost money but you can keep using them as long as you pay the (already budgeted) power bill and (also already budgeted) maintenance/upkeep.
On the contrary, computers follow their programming. They don't cut corners to make their workload easier, they don't make more mistakes when they are overworked and tired. When you don't have enough resources to make the computer follow it's programming, you also don't have the resources to use the computer.
Keeping things manual means you're adding in all the errors that are unavoidable with humans, and limiting use by constraining resources introduces even more errors. Appropriate oversight is an additional cost on top of the labor you already can't afford, running without adequate oversight is not only possible but strongly selected for.
Furthermore, whereas with computers you can theoretically go through their programming and say "ah this is why a crime was predicted" and the model which generated that insight could be potentially challenged and changed, with a human you'll never really know if there is an issue, and your options for dealing with issues are incredibly limited.
If you can't trust a computer to do math, you certainly can't trust a human to do it.
More efficiency isn't necessarily better. There's a balance of power, and for instance, allowing different databases to be linked increases the power of those with access. Exponentially. That's why it's heavily limited.
And the new programs using neural networks, if allowed to be used, would bring with them even more issues coming from their black box behavior, which would encourage a lack of accountability (doubtful that programmers would volunteer to be directly responsible for all their mistakes !)
The argument has nothing to do with efficiency, except insofar as human inefficiency makes abuse of power more likely. The problem is there isn't fundamentally a balance of power, having unlinked databases doesn't allow the "good" databases to shield you from the "bad" databases, it just increases the potential for errors to arise. I'll amend my previous closing argument, if you wouldn't trust someone with a knife, why would you trust them with a whole cutlery set? We're not trying to make the system more efficient, we are trying to eliminate potential failure points, which every human involved is. If it is more efficient, that's just gravy on top.
If you are really concerned with who is in a position of power, consider the difference between a small team of maybe a dozen people working behind closed doors versus software which could be made as public as the laws it is meant to uphold and evaluated by anyone on earth. Any issue, be it from malice or incompetence, would be plain as day.
And if you don't like black box neural networks, I have some terrible news about the human brain.
There is no technical reason why the creators of an AI should be any less accountable for a flaw in their product than the producers of any other technology. This wouldn't generally be the programmers themselves, but it would be the companies employing those programmers. If a car company for example becomes aware of a serious flaw in their product and they don't take appropriate steps to fix it, they're liable. Likewise for most things. The creators of these systems don't need to volunteer to be responsible for how they perform, they are compelled. There have been countless tools added to law enforcement's utility belt over the centuries like radar guns and dna matching, and every legal system in the world has dealt with evaluating their efficacy and potential for bias. There is nothing about an AI that makes it incompatible with these existing safeguards.
I believe that if a crime is sufficiently bad that we should prosecute for it, it absolutely merits the use of scarce resources. And if we can't muster up the resources to get humans to do the investigation, accusing, trying, etc it must not be that bad of a crime.
Once algorithmic prosecution is enabled, we could easily see 100x as many accusations happen. That's the blessing of automation. 100x is actually a low bar. Many automations make millions more things possible.
I don't think it would be a good world with a million times as much law enforcement with basically all of it automated.
Again, you're fundamentally missing what I'm saying. No one is arguing that humans require too many resources. I am saying that a computer, even if it were just as resource intensive as a human program, is a fundamentally preferable option.
We do not live in a world where the more crime there is the more resources are available to deal with it. If anything, reality is the opposite: there is a vicious cycle where lack of government resources leads to crime and crime decreases available resources. It's trivially simple to imagine a scenario where crime gets so bad that we stop dealing with certain violations to conserve resources to deal with bigger issues. But that's irrelevant.
What matters is that each and every person involved is a potential source of issues. Anyone can be tired, or scared, or distracted, or biased. This is not due to inadequate resources, though that may exacerbate the problem. This is the human condition - we are fallible beings. And in situations like this mistakes can ruin or end lives. Humans need oversight because of this fallibility, but the overseers are likewise fallible. The larger an organization is, the more places there are for something to fall through the cracks and for a mistake to go unnoticed and unresolved.
Algorithmic prosecution might make accusations 100x cheaper, but you won't have 100x more accusations because there isn't 100x more crime to accuse people of. You're not going to see a massive spike in murders or bank robberies because of this algorithm, and the crime never having occurred is a really solid alibi.
Sure if it's easier to investigate crimes then maybe some of the crimes we are currently tolerating because we simply don't have the manpower to deal with them would get investigated, but if we're ignoring a crime because we think the crime is not an issue, we can certainly keep ignoring it. The difference is the choice is made by society, not forced upon it by circumstance.
You fear more law enforcement because you are plainly aware of how bad current law enforcement can be. This isn't more law enforcement, this is better law enforcement.
I strongly disagree that increased law enforcement capabilities will lead to ever more perfect policing. I'm pretty sure that's what you're arguing.
When we switched from toll booths with people in them to automated tag reading, this increased the ability to create toll roads. By making the cost of enforcement cheaper, it's enabling.
If you make the cost of law enforcement lower, that won't make law enforcement work harder at solving murders. It'll mean that whole swaths of things that previously weren't practical to police will become practical and even economic. Especially if there's an automated fining system.
Computers don't fix the flaws in systems. They magnify them, by enabling those systems to be scaled way, way up.
> I strongly disagree that increased law enforcement capabilities will lead to ever more perfect policing. I'm pretty sure that's what you're arguing.
That is not what I'm arguing. I am saying that if law enforcement capabilities are separate from the safeguards against their misuse, then a cut to resources will lead to a degradation of the safeguards much faster than the capabilities.
Computers aren't better because they decrease the cost of law enforcement, they are better because their safeguards can't be independently cut, at least not without substantial, coordinated, and publicly acknowledged effort. Sure they might be used to investigate more crimes, but they will investigate those additional crimes with the safeguards in place.
Automating toll roads may have made it easier to create toll roads, but that hasn't lead to the mass conversion of roads to toll roads. It has eliminated the possibility of toll booth worker saying "the sign is wrong, you owe an additional $2" and pocketing the difference. Now either everyone gets charged the toll they're supposed to, or everyone is charged an incorrect amount and there is a massive public outcry leading to the error being fixed.
If the police start prosecuting lots of things that were previously impractical, then either there is large public outcry and we change the laws to make legal something that never should have been illegal in the first place, or people believe it's a law that should be enforced in which case it's good that now the police are able. This is, in my opinion, a massive improvement over selectively enforcing a law against those who are too powerless in society to fight against it under the guise of "conserving resources".
Computers are not scaling up the existing system, they are a fundamental change to the system. I'm the first to acknowledge that this new system will still have flaws, and sure those flaws will be magnified by scale, but they are different and in my opinion at least, much more tractable flaws.
The idea that any policing algorithms will by definition be open and auditable and whatnot is a lovely one. I would welcome that.
We already have some situations where algorithms are used in policing. Like for breathalyzers. The code is not open source. The audit chain is not inspectable.
To say that new algorithms will be held to higher standards because you want them to be is laudable. But it seems like you might be engaging in magical thinking.
If you could make a strong argument why the system will use new algorithms differently than the existing ones, I'd love to hear them. But remember, the incentives for the people who allowed closed source breathalyzers, closed source voting machines, etc haven't changed yet.
If their incentives need to change in order for us to get open, auditable algorithms, that's going to have to happen before we start using them, not after.
- computers are fast and cheap
- humans are slow and expensive
- slow and expensive = rarely used technique
- rarely used technique = less chance of privacy invasion
Therefore forced human processing protects privacy.
It's a privacy concept that's quite often discussed. Pre-computers we had a certain level of privacy afforded to us that was due to the cost/speed of humans processing large amounts of data. Computers have stripped that away.
It doesn't matter if there are human processing errors as that comes out at court time.
Everything else you're saying is also kinda wrong, unfortunately. You aren't looking at the problems caused by computer efficiency, just somehow computers are magically better because they follow the same rules every time.
Firstly, the data is usually bad and messy. Like super bad and messy. Computers can't tell when data is obviously wrong, so innocent people can get caught in "computer says no" trap. So you're wrong computers are better because of that. They have no common sense.
Secondly, there ARE 100x more crime than get prosecuted. You're dead wrong about that. Most of the time that low level law breaking is skipped. Allowing mass processing means you can politically target people if you so desire with ease. Because all you have to do is look them up on the computer.
Third, computers don't forget. In the human world, when task force 1 flags you, you'll probably be forgotten. In the computer world you get a few black marks and that's it, you become ostracised from society. e.g. US's no fly list, China's social credit system.
Fourth, the computer rules can be bad. A human "black box" can (and will) explain itself. A lot of computers cannot. The Dutch recently had a big scandal because they flagged anyone with a foreign name. The UK prosecuted thousands for fraud because of a computer glitch (yes, thousands, see point 1). Any rules generated from ML is generally racist and bigoted. Etc.
I get GP's point, I'm arguing it's incorrect. Specifically slow and expensive != rarely used.
Once you accept that your privacy is going to be violated either way, the computer is the obviously preferable option.
Human processing errors are not guaranteed to come out at court time, and humans can and will hide their mistakes.
Getting caught in a "computer says no" trap fundamentally requires the human using the computer to also not be able to tell when the data is obviously wrong. Computers are not supposed to be the source of common sense in either system, they are a tool.
There isn't 100x more crime that isn't being prosecuted due to lack of resources. There are lots of crimes like jaywalking that our legal system leaves to discretion. 99% of the time, a person crossing the street is a non-issue. Even if you do it right in front of a cop and it would take zero resources to prosecute, it doesn't get done because we as a society don't believe it should be. We make it a crime because there is that 1% of cases where it is a problem, and we need to be able to deal with those cases, but distinguishing which is which can not be readily codified in law. The only way that AI will change this is that the decision on whether to prosecute jaywalking will be consistent.
I don't see why computers not forgetting is a bad thing. We already have the means of preserving data for long periods of time. The decision on whether records should be maintained or deleted should be made based on the circumstances, not on the faultyness of the hardware. I mean if you really wanted to, you could always program a computer to delete data at random, but it's just a dumb idea.
Human rules can be bad. A human black box can not spit out an error log showing the particular sequence of neurons that activated that led to its decision. It can generate a statement, but whether that statement correctly explains it's decision making process can not be determined reliably. For example a human can shoot someone because of the color of their skin, and then state that it was because they felt they were in danger despite not actually feeling that way. Detecting such a glitch is extremely difficult, and no one has a reliable method of fixing it. Falsely suspecting people because their name is foreign is a well known bug in humans. Racism and bigotry is likewise a common issue with human neural networks. When you consider the countless multitudes of erroneous decisions made by humans over the millenia, an AI falsely flagging mere thousands is an immense improvement.
That's not correct. In the same paragraph it states that in general a digital analysis of information is necessary for increasing police effectiveness and that this is suitable under constitutional law.
The issue that the constitutional court is pointing out, is that there is no appropriate threshold for using such analysis software. In other words they that that you can't use is for anything, just when it is really necessary and proportional.
As a German native I think you're a bit off both on translation and based on understanding of the foundations of the German constitution.
No constitutional law expert myself, trying to contextualize in the hope it helps:
Core pillars of the German constitutional system are informational self-determination (i.e. personal data has a very high degree of protection under the constitution, see Germany's history) and proportionality (in using its powers, the state must choose methods appropriate to specific situations, obv. to hinder misuse of power - see Germany's history again).
So the ruling in essence, as I understand it, says among other things:
1. Automated data analysis needs to respect informational self-determination of anyone (i.e. no data mining per se on people who are not suspects for a given type of crime being investigated)
2. Automated data analysis needs to be proportional, meaning that the purpose of running any analysis "for the benefit of the public" must be reasonably well established and the analysis must have a strict goal.
I.e. if John stole something, you have to analyse the data such that you can credibly prove that identification for theft was the purpose.
Starting with looking for thieves but ending up identifying X movie pirates, Y weed consumers etc. is not proportional and crosses the right to informational self-determination.
So in the above example, the movie piraters and weed smokers might have not had an official investigation into their offenses on record, so by law the presumption of innocence must hold - you cannot do blanket investigations without reasonable cause under German law.
In the final paragraphs, the text actually states under which conditions automated data analysis would fullfil constitutional requirements:
1. The legislative needs to codify which data points exactly can be used for a given offense where offenders are to be identified with data analysis. Laws also need to explicitly codify limits of usable data.
2. If automated data analysis is performed by authorities, they must document the reason for initiating an analysis, the methods chosen, the criteria for labeling the outputs as "guilty/innocent", document the entire procedure and be able to release all of it to the public domain so that public data protection officers - or anyone else - can audit the results.
All of this sounds super reasonable to me.
To be clear, automatic data analysis for the purpose of extreme threats to national or public security are exempt if I understand correctly. These would be cases where it is proportional to perform such analyses for the greater good.
But you cannot do mass number crunching on ordinary citizens.
> personal data has a very high degree of protection under the constitution, see Germany's history
Forward to recent years, not an insignificant amount of people start to hate this data protection nonsense because they have to pass documents across even government offices, which could happen automatically in a digitized world, while adtech always manages to track all the things.
That's not a data protection problem, however. No data protection law forbids digitization of government services. It's a common excuse, though.
The primary issue limiting digitization of government services is federalization. A lot of software is just incompatible, outdated, broken. There's no central agency that can enforce standards across the states or (gasp!) municipalities - every city essentially can use whatever and buy wherever. There are no economies of scale leveraged, no central standards, few qualified technical and management personnel on the buyers side. And the federal government has no mandate to enforce any of this. Contrast that with the UK, which created gov.uk as the central authority on digital citizen services.
> That's not a data protection problem, however. No data protection law forbids digitization of government services. It's a common excuse, though.
It's a super weird one as well. There's no extra special data protection rules for digital data vs. data on paper.
Are they admitting they violate privacy laws already? Or do they want to change the information flow while going paperless? If so, why, and why is it necessary to go digital at all?
In some cases (taxes mostly) digital data has been associated with government-wide identifiers that make profiling citizens across all government agencies possible and easy. Linking even more data automatically and without asking the affected citizen to those identifiers rightfully gets pushback.
Paper often uses the same identifiers, but being paper, isn't automatically linked to all the other papers in other filing cabinets somewhere...
Yup, it should not be surprising that the 1978 SAFARI scandal that resulted in the creation of the French data protection authority happened when the government attempted (and had to stop) to link various then paper databases into a centralized one through a single social security number.
You are correct, by now there are some standards. But look at the clusterfuck that the current digitization initiative is: for each service to be offered, a different municipality is responsible. Instead of all municipalities funding a common shared institution. We’re seeing entirely predictable results. :sigh:
Yes, but actually that isn't due to data protection. It is due to the lazy bureaucracy using data protection as an excuse to have the citizens do their work for them.
Thanks for taking the time to provide some local perspective.
As a fellow European, what I find fascinating about this is that we all mostly want to accomplish similar things:
1) The government shouldn't go on fishing expeditions for citizens breaking the law
2) Avoid oppressive laws
3) Avoid unexplainable, opaque and secret laws, of which arguably "ML model says no" is a really nasty example
In most countries, the debate about this is only tangentially related to privacy, but in Germany it's all about privacy.
By being so obsessed with privacy, Germany seems to avoid a lot of nasty effects of technology. On the other hand, y'all still don't have Street View, couldn't buy VR headsets for multiple years and getting two government offices to exchange your information is basically impossible. (And I always forget that I need to have physical cash when in Germany, like in 1992.)
I am guessing that the German POV is that those things were worth it to avoid the bad side effects, but it's also the case that other European countries have avoided the bad side effects without having such blanket bans on large scale data processing.
I have no comparison when it comes to statehood and legislature with other EU countries, but I can definitely tell you that German federal bureaucracy as well as civil servant system is insanely complex and the absolute primary reason for absolute drag on technology adoption.
Essentially it's death by a thousand cuts (or bureaucracies) plus powerful career civil servants appointed for live and not incentivized to change at all.
The root of these systems is directly attributable to the horror of the Third Reich and seeing how the Nazis corrupted centralized power and weak bureaucracy.
By design, things like large scale technology adaption in public administration is hard in Germany.
It is 100% intentional and not driven by the economy etc. but mostly the legal setup.
I don't particularly mind it in the case of Street View or VR headsets, the world does imo not need either to function properly.
It sucks in general though of course.
It also will only ever change at a glacial pace, unfortunately.
> plus powerful career civil servants appointed for live and not incentivized to change at all
Hum... My experience is that if you appoint good people and give them confidence they won't get fired, that people will go out of their way to make things change (into whatever is in fashion for a bureaucracy, today we are moving from strict legalism into formal cost/benefit based decisions).
Are you sure there isn't any strong systemic effect incentivizing them not to change?
> Are you sure there isn't any strong systemic effect incentivizing them not to change?
There's a strong incentive not to make any decisions, because decisions could be wrong, and being wrong could get you blamed. You'll never get fired for being wrong, but you might get moved to a different post that is somewhere else where nothing ever happens (and you can't do damage, I suppose).
German bureaucracies are trying to cover each and every single case so that state employees don't have to think and will reliably return the same result given the same inputs. Unfortunately, you rarely ever have the exact same inputs, so you'll need to add more code to fix edge cases, and at some point one person can't keep all your rules in their head, so you need two people, now they need to communicate and it goes on and on.
The other part is very much a cultural thing, state administrations are well-known for being cushy jobs where you won't have to do much and attract the kind of person that wants that job. Every now and then someone will see the state of bureaucracy in Germany, think "I'll help!" and get themselves hired, and then burn out within half a year of trying to change the culture around them.
I'm only half joking: the best thing to happen to most small-ish bureaucracies in Germany would be to have a team building exercise with an accident that leaves everyone unable to work for 2 years so that there's enough time to build up a new team, install a new culture and clean up the processes. When the old ones are ready to come back, you can slowly introduce them into healthy teams, so that they'll take on the new culture.
> German bureaucracies are trying to cover each and every single case so that state employees don't have to think and will reliably return the same result given the same inputs.
Hum, yes, that's strict legalism. It was the fashion a few decades ago.
Do you have insight on the inner working of your local bureaucracy? Because it's hard to believe in a country as large as Germany people are so outside of the curve. Maybe you are seeing the outside appearance of a power structure that evolved in some past decades, and that won't tell you anything about what people are trying to do, just about what people are allowed to.
(Or maybe Germany public service is almost all composed of people nearing their retirement and with old ideas. It's not that unlikely that an entire government is skewed in age.)
I don't have any insight in my local branch (I've only lived here for two years, but from the little contact I did have, it doesn't look different from where I've previously lived). I've done some consulting for a trade union that covers parts of public administrations and what I learned from helping them organize files surrounding the labor environment in ministries and local agencies looked like it could've well been the inspiration for that Asterix comic dealing with Roman bureaucracy.
I have a friend who works on a large federal digitalization project and he's ready to quit because it drains the life out of you for those same reasons (rules abound, nobody wants to make a decision, and expensive consultants and lawyers who are brought in to take the blame -- only they don't care for taking the blame, they want to bill as much as possible, and so everything takes forever). Yet he won't, because he believes that it's important work and will help make things better, and so he'll suffer and probably snap in a few years down the line.
To me, it looks like a culture issue. I'm a freelancer and have worked with quite a few teams over the past 20 years and some teams are just broken because of culture. They're places where nobody cooperates willingly and nobody takes ownership of anything, where fear stalks the offices and ideas go to die. And then you have places where people are proactive, and will handle a task and when an additional thing pops up a week later, they'll say "I've anticipated that and made sure it's easy to add" and things just work, because everybody works so that the team's future version has an easy time. I'm sure a broken culture can be fixed somehow, but I don't know if you can do it gradually because new members will adapt to the existing culture and perpetuate it.
Germany does, in fact have street view. There were delays and pauses, and more places blurred than in most countries but it's available, at least for major cities.
Dont you think they should teach law first, in order to protect kids from other criminals?
Manipulative people of all ages exist, just look at how judges dress up in fancy dress wearing grey wigs and black or red garbs in order to subconsciously protect power and wisdom, at least here in the UK they do.
Its also why the UK police wear black, psychological projection of power, but laws don't recognise this subtle manipulation of the state. I know German police wear green, and I think the French police wear blue, which gives you an insight into how they want to be perceived. The US police also wear black to project power and status.
So with the fact that no country as far as I know teaches law apart from maybe Iran and other middle eastern countries, how can anyone claim to live in a democracy if they are not kept up to date with law changes, let alone debate these changes, because the public are resource burned and hand over their decision making to their democratically elected representative?
What a con trick people get tricked into that also gets people reinforcing and justifying their vote decision.
Likewise whilst some chemical consumption can alter behaviour, alcohol perhaps being the most widely known, its less widely known what prescription meds and illegal meds will do to the body and thus behaviour.
And here is the point, knowing the state dont teach people what these different chemicals will do, again how can anyone claim to live in a democracy, if they are not informed in order to make an informed decision or give informed consent.
Its extremely lazy if not deceitful of the legal system to assume that once someone turns 18, they can give informed consent to anything, knowing how widely businesses especially big businesses have things like planned obsolescence built into products and services, in order to carry out their own form of law and order.
Perhaps the reason why a definition of consciousness doesn't exist is because this would frame and lock down the narrative on informed consent, which is what much of the law hinges on in order to award punishments of sorts.
In some respects the state treats the public like a dog, it throws it a stick, the dog chases after it, and happens to fall off a cliff with the owner blaming the dog for its mistake.
A simple example to combat the intellectual warfare some parts of society like to rein down on the public.
This is such a German judgement. So the police are using big data to predict who will commit crimes, which is sort of dystopian.
Police in the United States are close to this already. Chicago's police department used to brag about its data-crunching ability to predict where crime would happen. At the time, it was just by area, but it was stated that with enough data, it could be narrowed down to the block. It's not that big a leap to extrapolate that to the individual.
My guess, though, is that it didn't work, and the city was just parroting the promises made by the salespeople of whatever system they bought for this task. If it worked, then police would preemptively flood certain areas where they know crime is about to happen. But, as is seen on television most nights, and in the complaints of the aldermen, the police mostly seem to react to crime, rather than prevent it. The use of social media by crime mobs has only made it worse.
I no lawyer but a friend of who is said that there are differences in how countries practices law. The nordic countries in Europe are concerned with the process. The assumption being that if the process is right the judgment will be just.
In southern Europe one is more concerned with the outcome, as long as the judgment is just, then there is wiggle room in the process.
This seems like a judgment that focuses on process and not outcome.
It is unconstitutional because the law was written too broadly. The court said two things. It's fine to use this for highly legally protected goods, e.g., protection of body, life, or freedom. However, the same high standards need to apply like for hidden surveillance measures currently used.
Well, every model has its errors, and intrusive criminal "prevention" is self-fulfilling.
Just imagine, everyday 6AM an officer come at your door to check your apartment "to prevent" your crimes, suddenly every random guy in the street looks suspicious and spying on you, then some officers just randomly walks by your workplace, police stop you everytime you travel... This would eventually lead to an insult to the mafia guy and then boom you have already commited contumacy and the model was right! You are a criminal and you should be punished!
But would I have done it if the police weren’t following me? What if the computer is identifying at risk paranoid schizophrenics likely to be pushed over the edge by noticeable increases in their personal surveillance?
Frankly that seems more plausible than telling the future
I don’t make any claim at all for its capability. But if you load a dataset into some sort of ML algorithm right now, with the data coming from current policing statistics, you are going to be perpetuating some pretty dark stuff. Police target various groups and minorities.
I speak German and I have just read the text, it is vastly complicated court and lawyer speak.
But it is clear that this legislature is pushing for crime prevention based on aggregated data from computer systems(where from, exactly?)
The arguments are that a manual extraction and collection would be too slow and that this should be limited to only a handful of severe potential crimes. As a prevention rather than mitigation technique.
This is quite bad.
Now the objection is not purely because the data is coming from computers, but because it will reveal data on people tangentially connected to the targets.
This is so wrong in so many ways and the people who are pushing from this have learned nothing from the stasi and nazi regimes, or perhaps they are nostalgic.
It is very clear that these things are pushed by rather smart minds, matter experts and this makes it clear they do not care about the innocent peoples privacy, as they could not possibly be so ignorant to overlook this matter.
Justice even the conventianal way spawns injustices, now the lazy government wants to automate and scale their procedures. W.t.h.
It isn't just about their privacy but also their very freedom. Being in some police database might very well land you in the crosshairs of some investigation and related niceness like searching your home, taking you in for questioning and keeping you there because you didn't say something...
Anyone who ever end up an their radar would probably never be removed from that,and never even know.
It is highly offensive that the German government pretends it does not know where the crime is happening.
How about they deal with a few of these clans numbering 100s of people? This can be done , has to be done the convential way, as these clans will not be as stupid as leave digital traces, as computer illeterate as they might be.
The government has not even defined who they are targeting, they just use the commonly thrown around terms.
> It is highly offensive that the German government pretends it does not know where the crime is happening. How about they deal with a few of these clans numbering 100s of people? This can be done , has to be done the convential way, as these clans will not be as stupid as leave digital traces, as computer illeterate as they might be.
>
> The government has not even defined who they are targeting, they just use the commonly thrown around terms.
The government is just after the really easy convictions on the cheap. Traditional investigation is complex and expensive, needs personnel, etc. Sifting through vast databases for easy wins is far cheaper. Catching big fish also isn't opportune, because that would mean politicians being responsible for incarcerating their buddies, like in Cum-Ex. In related news, the most common reason to be incarcerated in Germany: Using public transport without a ticket... Most convictions for the least amount of effort.
I ran a free forum service once, long time ago.
Some teens created a forum where they posted Heil Hitler and whatnot.
I couldn't oversee over 2000 forums.
One day I got a phone call, the Kriminalpolizei.
Long story short, I had to write a script that extracts IP addresses and time of posts. That took me 4 hours which they didn't pay me.
> It is very clear that these things are pushed by rather smart minds, matter experts and this makes it clear they do not care about the innocent peoples privacy
I was working that this behaviour by FAANG will normalise it, and will lead to our governments doing the same.
Same 'smart people' work for government and fang, they are gonna behave the same way.
Skipping ahead to the fact that automated data analysis is not prohibited per se, just needs to be reasonably targeted, the two most interesting paragraphs are the last ones for me.
Basically the court mandates that any future laws passed for legalizing automated data analysis must explicitly
1. For any given crime to be automatically analysed, define the data model and define appropriate analysis techniques or methods.
Those laws should be interesting reads.
2. For any one analysis happening, document and later publish to the public domain the reason for initiating an analysis, the methods chosen, the criteria for labeling someone as "guilty/innocent" - basically the entire process so that the analysis is auditable end to end.
Again, this is not a suggestion in the ruling, but a requirement.
This is really good news for personal liberties in my book.
Stark contrast to US/ China data mining on citizens.
An earlier poster wrote:
> This is so wrong in so many ways and the people who are pushing from this have learned nothing from the stasi and nazi regimes, or perhaps they are nostalgic.
Whereas you say:
>Skipping ahead to the fact that automated data analysis is not prohibited per se, just needs to be reasonably targeted, the two most interesting paragraphs are the last ones for me.
I think this reads like the court curtailed reasonably broad data grabbing behaviors such as those Snowdon has revealed happened illegally and are still happening in the US. It still permits effective, focused/targeted data mining.
So at least the court learned from the Nazi/Stasi times but also from the times of left-wing terrorism ("Rasterfahndung" during the Bader-Meinhoff terrorist gang) - i.e., that these techniques are effective yet must not be applied to broadly. If this reading is correct (I'm not a legal expert), then the main danger is that lawmakers are incapable (or unwilling) to implement requirements (1.) and (2).
I agree fully, I am struggling to imagine how they can go and push such a legislature infront of this governing body with a straight face.
This is more dangerous than most people would think, once something like this is codified into law, many eu countries will follow suit, just for good measure.
What are the motives and who benefits from this?
This idea was born somewhere and writing all the drafts costs a lot of money, who was footing this bill and why?
> whenever an individual decision is taken on the basis (even partially) of an algorithm, the administration must [...] explain [using easy to understand language]†, at the request of the individual, how the relevant algorithm works [...], by providing the following information:
the degree and mode of contribution of the algorithm to the decision making;
the data processed and its sources;
the processing settings and their weighting applied to the situation of the data subject; and
the operations carried out by the processing.
† a very important detail that this translation for some reason omits ??
Now, something that has been bugging me since then, especially in the «AI» context that was also being discussed more or less at the same time by the legislators, is that so far this is impossible to do for neural network based programs, since the final program is not based on any algorithm (as commonly understood), or at least not one that even the programmers themselves could understand, much less distill into a commonly intelligible form !
For eg: denying service from a German retailer depending on the ML prediction result of the transaction being classified as fraud probable(Germany has pay by invoice), would also be constitutional or not?
Or credit score/risk rating using ML, trained on a feature extracted from parameters like Zip code, ethnicity could also classified as automated data analysis for prevention of criminal acts, right?
In general a retailer can refuse any customer for any reason (unless you're discriminating against a protected class)
The reason they do this isn't to fight crime, but to save the business from losing money. So I don't see how a ruling about preventing crime would apply.
I think it's crazy that people are considering "linking previously unconnected automated databases and data sources in analysis platforms and permitting systematic access of data across sources through searches." as something like Minority Report.
The truth is that Law Enforcement investigations are currently taking information from dozens of silos/databases, manually copy&pasting them into Excel and Word and then finally making a pdf or powerpoint to share the results. This is obviously a really inefficient way to do things and wastes a lot of taxpayer money.
A technical solution that would allow LE to crosslink data they already have is a no-brainer.
P.S: Very biased view, as I'm building/selling exactly this tool :)
> A technical solution that would allow LE to crosslink data they already have is a no-brainer.
A technical solution that is Open Source, that falls under federally mandated criteria for how it can be legally used, that automatically documents any and all access and makes the entire audit log part of any output result so that all happenings are easily auditable by 3rd parties... might be a no brainer.
Anything else I'm highly skeptical.
If going after syndicated and or international crime, fine.
First feature I'd like to see is linking Finanzamt with LE and go after money laundering - Germany being an absolute paradise to the detriment of everyone else.
If it takes manual copy paste to protect ordinary citizens from overreaching police, I welcome that technical limitation.
The one argument I rarely see against automated analysis is that any analysis has a false positive rate. If you hire 1,000 humans to do that analysis there's simply a limit to how many errors they can make in total.
If you introduce automation doing the same thing at a scale corresponding to 10,000× the human capacity, you will have 10,000× as many errors. This will violate Blackstone's ratio and erode trust in the system.
"Dieser Eingriffsanlass bleibt angesichts der besonders daten- und methodenoffen formulierten Befugnisse weit hinter der wegen des konkreten Eingriffsgewichts verfassungsrechtlich gebotenen Schwelle einer konkretisierten Gefahr zurück."
This roughly translates into "there has to be a sufficiently concrete danger to warrant such a ambiguously formulated infringement."
I truly appreciate this sentiment. Any collection or analysis of data "just because there might be something in there" has to be weighed against the right of everyone that is being data-mined.
This freaked me out about Minority Report when I first watched it: those precogs are essentially hacking into the future memories of everyone involved in _and by happenstance surrounding_ the lives of future criminals and their crimes.
Mass-scale datamining is effectively the real-world equivalent of telepathy. And I want neither precogs nor the government in my mind.
In the US - this should also be the case. It should fall under the 4th amendment as unreasonable search and seizure. But the NSA, and probably other agencies, has been doing it for years anyway.
There has to be a poe's law equivalent for gender issues at this point, if the article is anyway political someone's gonna mention it because uhh they gotta be mad at someone
Why? IMO we should use facial recognition, digital ID, a cashless society and everything at our disposal to stamp out crime.
It's the working class who pay the price at the moment. We get burgled, mugged and killed, meanwhile the wealthy lawyers and politicians can pontificate over constitutionality from their gated communities and private vehicles.
It's insane that the EU is trying to ban automatic facial recognition with the terrible AI act - https://eur-lex.europa.eu/legal-content/EN/TXT/?uri=celex%3A... - nevermind the sheer amount of bureaucracy it introduces for startups (a government register of "possibly harmful" AI - wtf?).
Bear in mind this is the same country that bans Google Street View (and on the road to ban nuclear power and GM crops).
People here support this ban because they think it's "harmful", but how long until they think some technology you rely on is also harmful? GPS? Phone cameras or Google Lens? Encryption - https://mullvad.net/en/blog/2023/2/2/stop-the-proposal-on-ma... ?
Technology is the future, and we should embrace it wherever possible - crime prevention, education, healthcare, nuclear fusion, genetic engineering, etc.
My best friend is a sex worker. I use cannabis to control my anxiety. Neither of those are legal. Looking historically, virtually every human rights movement required illegal direct action. Where do we fit into your cashless panopticon society?
The worst crimes I'm worried about are the ones that will never be illegal, the ones that benefit the powerful and rich. At-scale invasion of privacy is only one such tremendous crime, one against against human rights and dignity, but barely against the law.
> Bear in mind this is the same country that bans Google Street View (and on the road to ban nuclear power and GM crops).
> People here support this ban because they think it's "harmful", but how long until they think some technology you rely on is also harmful? GPS? Phone cameras or Google Lens? Encryption - https://mullvad.net/en/blog/2023/2/2/stop-the-proposal-on-ma... ?
Some of those are genuinely harmful (street view in particular for its privacy invasion - note it is not banned but simply very unpopular) so I'm not sure what your point is. Also, you should send your ideas to the Mullvad blog authors and get their input. I think you'll find that the overlap between people who support universal end to end encryption and people who support an all-seeing police state is very small, as they are fundamentally incompatible ideas.
> Technology is the future, and we should embrace it wherever possible - crime prevention, education, healthcare, nuclear fusion, genetic engineering, etc.
Technology is not always good all of the time. The past century and even past decade are full of examples of net harms caused by technology that could have been avoided if we developed them in a more intentional controlled manner.
A eccentric example to make the point: If there is a technology that can trigger a supernova of the sun simply by mixing water, sugar, gasoline, and bleach together in a certain ratio, then it would be extremely important we not conduct research into that technology until we have other technology that allows us to survive a supernova or can prevent that mixture from being created anywhere on Earth. Even if there are advantages of the technology (maybe a different ratio causes all humans to have 5x longer lifespans and eliminates all disease and suffering, and we gain intelligence allowing us to begin colonizing other planets within 3 years) the risk is just too high that it will be misused accidentally or maliciously until we have a way to handle it. https://en.wikipedia.org/wiki/Differential_technological_dev...
I find the claim that street view invades privacy or causes harm a bit odd, particularly since it long ago started blurring faces and license plates, and appears to do so reliably.
I'm receptive to the idea that scale, permanence and universal access can represent qualitative changes to the impact some kinds of information might have on people. I'm just not understanding how that applies to images of public streets.
When you have legislators chasing buzzwords and clout, you get a lot of grandstanding from them. They don’t understand the tech, which doesn’t really matter to them. What matters is they get to say they did something to protect the people.
In the medium-term, the cultural shear between the US and Germany will continue to be highlighted by decisions like this.
Germany leans heavily to more and more privacy because they have the experience in recent memory of what a government can do with vast information collection and centralized authority.
America leans lightly towards more and more centralization of data because they have the experience in recent memory of what individuals who "slip through the cracks" can do. So in Germany we see decisions like this and in the US we see police using privately-curated DNA databases to defrost the cold case of a serial killer, and they're lauded as heroes.
(It's worth noting, of course, that this is all relative. Relative to most of Europe, the US is starting from a place of massive decentralization and minimal individual-tracking. Some people still find social security numbers controversial).
> Germany leans heavily to more and more privacy because they have the experience in recent memory of what a government can do with vast information collection and centralized authority.
I theorize that in a hundred years or so, North Americans will have learned this lesson the hard way and will have atrocities in recent history, too.
I'm sad I won't see the day when Bluffdale is repurposed as a memorial to state surveillance, much like the Stasi headquarters are now in Berlin.
Can't find the study now, but there was a sociological survey of software engineers in the US and Europe, one really interesting thing was that the top social concern when implementing solutions for Americans was fairness, for Germans it was privacy. The other concern didn't even make the top 5 for either.
If china ever gets democratic, a new mighty privacy player will appear. Chinas citizens have the same nasty experience as east germans had with the stasi and the poles, chechs with the cheka. They have been spied upon, reported upon by there fellow citizen and have been victims of "zersetzung". Aka social engineering to destroy a citizen.
Hey hey hey, don’t leave us ex-Yugoslavians out! We had UDBA and political prisons and much of the rest. I hear it was “great” (too young to remember/experience myself).
Yes it wasn’t as bad as many other places. But bad enough for people to know there are things you can and can’t talk about in public. Growing up it was interesting to observe how my history class evolved from elementary to high school as news and declassifications trickled out.
Ah, do you have "Auskunftsrecht", were you can hand in your familys name at the states office, and they will crawl the Security office documentation for your name and give you all the information on all the people who brought shame upon themselves?
Bonuspoints: If you had family who was horrible in the Nazi times and the descendants then were horrible during Stasi times. And you know, anyone who can remember the name can crawl that info.. makes you wish, the ground could melt and swallow you.
Imagine that info got bundled into a family dharma app, were crimes would follow your family name through the generations. Nowhere on the planet, can you escape the murder spree of grandpas uncle.
Well no, because many (most?) of the people who were bad during nazi times simply got killed by the communists and put in unmarked mass graves. No records.
Many cultures have a simple "You are not guilty of crimes other people committed" rule baked into their law and culture to accommodate this, because what you're describing is also just known as "history."
In general, most people have at least one near ancestor that did something terrible. That makes none of the descendents terrible people.
Or us Portuguese - up until 1974 we had PIDE (translated: International and State Defense Police); so within living memory of many in power today. They did all the things any "good" oppressive police force of a semi-fascist to fully-fascist state does: receive anonymous complaints from conspirators, arrest people and torture them for reasons ranging from the political ("being a communist") to the moral ("being homosexual"), and generally make sure everyone's lives are too miserable and filled with fear to dream of overthrowing the powers-that-be.
Our concern with privacy is so prevalent that fairly recently the collection (and subsequent usage) of metadata by police was deemed unconstitutional[1]. This was met with a lot of complaints by investigators, and while there's some sectors pushing for a constitutional revision, this seems unlikely at the moment, but not entirely off the table.
Yes, I have read about the UDBA, they would follow opposition to abroad to kill them.
And the fallout after Yugoslavia fell apart had the same people and criminals unpunished, with the climax of a broad daylight execution of a prime minister.
I don't understand that last bit : in my understanding, the way that social security numbers are used in the US would be illegal in most (?) countries in Europe ?
For instance, in France specifically since the 1974 SAFARI scandal that resulted in the creation of the French data protection authority and assorted laws in 1978, and happened when the government in secret attempted to (and had to stop due to the outrage) link various (then paper) administrative files into a centralized computer database through a single social security number.
This concept was explored in a stephen spielberg movie. Minority report. Good movie. Watch it. Stars Tom Cruise and has mission impossible vibes.
In the end the government cancelled the program because of free will. A person always has the choice to change his actions because his actions aren't predetermined.
106 comments
[ 3.0 ms ] story [ 185 ms ] threadI think it's more like borderline presumption of guilt.
It's not a broad decision that automated data analysis for crime-prevention is unconstitutional.
"Automated data analysis or interpretation requires justification under constitutional law."
I'm sure something is lost in translation. But even assuming a lot of context, I find it hilarious. Germany's highest court seems to have stumbled into avoiding a dystopia not because they don't want some biased ML model bullying minorities, but because they plain and simple don't like all this computer nonsense. (Grumble grumble.)
You either need a lot more budget (= oversight) to hire all the people to do the work, or do less of it.
Sure computers also cost money but you can keep using them as long as you pay the (already budgeted) power bill and (also already budgeted) maintenance/upkeep.
Keeping things manual means you're adding in all the errors that are unavoidable with humans, and limiting use by constraining resources introduces even more errors. Appropriate oversight is an additional cost on top of the labor you already can't afford, running without adequate oversight is not only possible but strongly selected for.
Furthermore, whereas with computers you can theoretically go through their programming and say "ah this is why a crime was predicted" and the model which generated that insight could be potentially challenged and changed, with a human you'll never really know if there is an issue, and your options for dealing with issues are incredibly limited.
If you can't trust a computer to do math, you certainly can't trust a human to do it.
And the new programs using neural networks, if allowed to be used, would bring with them even more issues coming from their black box behavior, which would encourage a lack of accountability (doubtful that programmers would volunteer to be directly responsible for all their mistakes !)
If you are really concerned with who is in a position of power, consider the difference between a small team of maybe a dozen people working behind closed doors versus software which could be made as public as the laws it is meant to uphold and evaluated by anyone on earth. Any issue, be it from malice or incompetence, would be plain as day.
And if you don't like black box neural networks, I have some terrible news about the human brain.
There is no technical reason why the creators of an AI should be any less accountable for a flaw in their product than the producers of any other technology. This wouldn't generally be the programmers themselves, but it would be the companies employing those programmers. If a car company for example becomes aware of a serious flaw in their product and they don't take appropriate steps to fix it, they're liable. Likewise for most things. The creators of these systems don't need to volunteer to be responsible for how they perform, they are compelled. There have been countless tools added to law enforcement's utility belt over the centuries like radar guns and dna matching, and every legal system in the world has dealt with evaluating their efficacy and potential for bias. There is nothing about an AI that makes it incompatible with these existing safeguards.
Once algorithmic prosecution is enabled, we could easily see 100x as many accusations happen. That's the blessing of automation. 100x is actually a low bar. Many automations make millions more things possible.
I don't think it would be a good world with a million times as much law enforcement with basically all of it automated.
We do not live in a world where the more crime there is the more resources are available to deal with it. If anything, reality is the opposite: there is a vicious cycle where lack of government resources leads to crime and crime decreases available resources. It's trivially simple to imagine a scenario where crime gets so bad that we stop dealing with certain violations to conserve resources to deal with bigger issues. But that's irrelevant.
What matters is that each and every person involved is a potential source of issues. Anyone can be tired, or scared, or distracted, or biased. This is not due to inadequate resources, though that may exacerbate the problem. This is the human condition - we are fallible beings. And in situations like this mistakes can ruin or end lives. Humans need oversight because of this fallibility, but the overseers are likewise fallible. The larger an organization is, the more places there are for something to fall through the cracks and for a mistake to go unnoticed and unresolved.
Algorithmic prosecution might make accusations 100x cheaper, but you won't have 100x more accusations because there isn't 100x more crime to accuse people of. You're not going to see a massive spike in murders or bank robberies because of this algorithm, and the crime never having occurred is a really solid alibi.
Sure if it's easier to investigate crimes then maybe some of the crimes we are currently tolerating because we simply don't have the manpower to deal with them would get investigated, but if we're ignoring a crime because we think the crime is not an issue, we can certainly keep ignoring it. The difference is the choice is made by society, not forced upon it by circumstance.
You fear more law enforcement because you are plainly aware of how bad current law enforcement can be. This isn't more law enforcement, this is better law enforcement.
When we switched from toll booths with people in them to automated tag reading, this increased the ability to create toll roads. By making the cost of enforcement cheaper, it's enabling.
If you make the cost of law enforcement lower, that won't make law enforcement work harder at solving murders. It'll mean that whole swaths of things that previously weren't practical to police will become practical and even economic. Especially if there's an automated fining system.
Computers don't fix the flaws in systems. They magnify them, by enabling those systems to be scaled way, way up.
That is not what I'm arguing. I am saying that if law enforcement capabilities are separate from the safeguards against their misuse, then a cut to resources will lead to a degradation of the safeguards much faster than the capabilities.
Computers aren't better because they decrease the cost of law enforcement, they are better because their safeguards can't be independently cut, at least not without substantial, coordinated, and publicly acknowledged effort. Sure they might be used to investigate more crimes, but they will investigate those additional crimes with the safeguards in place.
Automating toll roads may have made it easier to create toll roads, but that hasn't lead to the mass conversion of roads to toll roads. It has eliminated the possibility of toll booth worker saying "the sign is wrong, you owe an additional $2" and pocketing the difference. Now either everyone gets charged the toll they're supposed to, or everyone is charged an incorrect amount and there is a massive public outcry leading to the error being fixed.
If the police start prosecuting lots of things that were previously impractical, then either there is large public outcry and we change the laws to make legal something that never should have been illegal in the first place, or people believe it's a law that should be enforced in which case it's good that now the police are able. This is, in my opinion, a massive improvement over selectively enforcing a law against those who are too powerless in society to fight against it under the guise of "conserving resources".
Computers are not scaling up the existing system, they are a fundamental change to the system. I'm the first to acknowledge that this new system will still have flaws, and sure those flaws will be magnified by scale, but they are different and in my opinion at least, much more tractable flaws.
We already have some situations where algorithms are used in policing. Like for breathalyzers. The code is not open source. The audit chain is not inspectable.
To say that new algorithms will be held to higher standards because you want them to be is laudable. But it seems like you might be engaging in magical thinking.
If you could make a strong argument why the system will use new algorithms differently than the existing ones, I'd love to hear them. But remember, the incentives for the people who allowed closed source breathalyzers, closed source voting machines, etc haven't changed yet.
If their incentives need to change in order for us to get open, auditable algorithms, that's going to have to happen before we start using them, not after.
His point is:
Therefore forced human processing protects privacy.It's a privacy concept that's quite often discussed. Pre-computers we had a certain level of privacy afforded to us that was due to the cost/speed of humans processing large amounts of data. Computers have stripped that away.
It doesn't matter if there are human processing errors as that comes out at court time.
Everything else you're saying is also kinda wrong, unfortunately. You aren't looking at the problems caused by computer efficiency, just somehow computers are magically better because they follow the same rules every time.
Firstly, the data is usually bad and messy. Like super bad and messy. Computers can't tell when data is obviously wrong, so innocent people can get caught in "computer says no" trap. So you're wrong computers are better because of that. They have no common sense.
Secondly, there ARE 100x more crime than get prosecuted. You're dead wrong about that. Most of the time that low level law breaking is skipped. Allowing mass processing means you can politically target people if you so desire with ease. Because all you have to do is look them up on the computer.
Third, computers don't forget. In the human world, when task force 1 flags you, you'll probably be forgotten. In the computer world you get a few black marks and that's it, you become ostracised from society. e.g. US's no fly list, China's social credit system.
Fourth, the computer rules can be bad. A human "black box" can (and will) explain itself. A lot of computers cannot. The Dutch recently had a big scandal because they flagged anyone with a foreign name. The UK prosecuted thousands for fraud because of a computer glitch (yes, thousands, see point 1). Any rules generated from ML is generally racist and bigoted. Etc.
Once you accept that your privacy is going to be violated either way, the computer is the obviously preferable option.
Human processing errors are not guaranteed to come out at court time, and humans can and will hide their mistakes.
Getting caught in a "computer says no" trap fundamentally requires the human using the computer to also not be able to tell when the data is obviously wrong. Computers are not supposed to be the source of common sense in either system, they are a tool.
There isn't 100x more crime that isn't being prosecuted due to lack of resources. There are lots of crimes like jaywalking that our legal system leaves to discretion. 99% of the time, a person crossing the street is a non-issue. Even if you do it right in front of a cop and it would take zero resources to prosecute, it doesn't get done because we as a society don't believe it should be. We make it a crime because there is that 1% of cases where it is a problem, and we need to be able to deal with those cases, but distinguishing which is which can not be readily codified in law. The only way that AI will change this is that the decision on whether to prosecute jaywalking will be consistent.
I don't see why computers not forgetting is a bad thing. We already have the means of preserving data for long periods of time. The decision on whether records should be maintained or deleted should be made based on the circumstances, not on the faultyness of the hardware. I mean if you really wanted to, you could always program a computer to delete data at random, but it's just a dumb idea.
Human rules can be bad. A human black box can not spit out an error log showing the particular sequence of neurons that activated that led to its decision. It can generate a statement, but whether that statement correctly explains it's decision making process can not be determined reliably. For example a human can shoot someone because of the color of their skin, and then state that it was because they felt they were in danger despite not actually feeling that way. Detecting such a glitch is extremely difficult, and no one has a reliable method of fixing it. Falsely suspecting people because their name is foreign is a well known bug in humans. Racism and bigotry is likewise a common issue with human neural networks. When you consider the countless multitudes of erroneous decisions made by humans over the millenia, an AI falsely flagging mere thousands is an immense improvement.
The issue that the constitutional court is pointing out, is that there is no appropriate threshold for using such analysis software. In other words they that that you can't use is for anything, just when it is really necessary and proportional.
No constitutional law expert myself, trying to contextualize in the hope it helps:
Core pillars of the German constitutional system are informational self-determination (i.e. personal data has a very high degree of protection under the constitution, see Germany's history) and proportionality (in using its powers, the state must choose methods appropriate to specific situations, obv. to hinder misuse of power - see Germany's history again).
So the ruling in essence, as I understand it, says among other things:
1. Automated data analysis needs to respect informational self-determination of anyone (i.e. no data mining per se on people who are not suspects for a given type of crime being investigated)
2. Automated data analysis needs to be proportional, meaning that the purpose of running any analysis "for the benefit of the public" must be reasonably well established and the analysis must have a strict goal.
I.e. if John stole something, you have to analyse the data such that you can credibly prove that identification for theft was the purpose.
Starting with looking for thieves but ending up identifying X movie pirates, Y weed consumers etc. is not proportional and crosses the right to informational self-determination.
So in the above example, the movie piraters and weed smokers might have not had an official investigation into their offenses on record, so by law the presumption of innocence must hold - you cannot do blanket investigations without reasonable cause under German law.
In the final paragraphs, the text actually states under which conditions automated data analysis would fullfil constitutional requirements:
1. The legislative needs to codify which data points exactly can be used for a given offense where offenders are to be identified with data analysis. Laws also need to explicitly codify limits of usable data.
2. If automated data analysis is performed by authorities, they must document the reason for initiating an analysis, the methods chosen, the criteria for labeling the outputs as "guilty/innocent", document the entire procedure and be able to release all of it to the public domain so that public data protection officers - or anyone else - can audit the results.
All of this sounds super reasonable to me.
To be clear, automatic data analysis for the purpose of extreme threats to national or public security are exempt if I understand correctly. These would be cases where it is proportional to perform such analyses for the greater good.
But you cannot do mass number crunching on ordinary citizens.
Yay in my book.
Forward to recent years, not an insignificant amount of people start to hate this data protection nonsense because they have to pass documents across even government offices, which could happen automatically in a digitized world, while adtech always manages to track all the things.
The primary issue limiting digitization of government services is federalization. A lot of software is just incompatible, outdated, broken. There's no central agency that can enforce standards across the states or (gasp!) municipalities - every city essentially can use whatever and buy wherever. There are no economies of scale leveraged, no central standards, few qualified technical and management personnel on the buyers side. And the federal government has no mandate to enforce any of this. Contrast that with the UK, which created gov.uk as the central authority on digital citizen services.
It's a super weird one as well. There's no extra special data protection rules for digital data vs. data on paper.
Are they admitting they violate privacy laws already? Or do they want to change the information flow while going paperless? If so, why, and why is it necessary to go digital at all?
Paper often uses the same identifiers, but being paper, isn't automatically linked to all the other papers in other filing cabinets somewhere...
1) https://www.fitko.de/ 2) https://www.xoev.de/ 3) https://www.xrepository.de/
As a fellow European, what I find fascinating about this is that we all mostly want to accomplish similar things:
1) The government shouldn't go on fishing expeditions for citizens breaking the law
2) Avoid oppressive laws
3) Avoid unexplainable, opaque and secret laws, of which arguably "ML model says no" is a really nasty example
In most countries, the debate about this is only tangentially related to privacy, but in Germany it's all about privacy.
By being so obsessed with privacy, Germany seems to avoid a lot of nasty effects of technology. On the other hand, y'all still don't have Street View, couldn't buy VR headsets for multiple years and getting two government offices to exchange your information is basically impossible. (And I always forget that I need to have physical cash when in Germany, like in 1992.)
I am guessing that the German POV is that those things were worth it to avoid the bad side effects, but it's also the case that other European countries have avoided the bad side effects without having such blanket bans on large scale data processing.
Essentially it's death by a thousand cuts (or bureaucracies) plus powerful career civil servants appointed for live and not incentivized to change at all.
The root of these systems is directly attributable to the horror of the Third Reich and seeing how the Nazis corrupted centralized power and weak bureaucracy.
By design, things like large scale technology adaption in public administration is hard in Germany.
It is 100% intentional and not driven by the economy etc. but mostly the legal setup.
I don't particularly mind it in the case of Street View or VR headsets, the world does imo not need either to function properly.
It sucks in general though of course.
It also will only ever change at a glacial pace, unfortunately.
I've thought about emigrating more than once. :D
Hum... My experience is that if you appoint good people and give them confidence they won't get fired, that people will go out of their way to make things change (into whatever is in fashion for a bureaucracy, today we are moving from strict legalism into formal cost/benefit based decisions).
Are you sure there isn't any strong systemic effect incentivizing them not to change?
There's a strong incentive not to make any decisions, because decisions could be wrong, and being wrong could get you blamed. You'll never get fired for being wrong, but you might get moved to a different post that is somewhere else where nothing ever happens (and you can't do damage, I suppose).
German bureaucracies are trying to cover each and every single case so that state employees don't have to think and will reliably return the same result given the same inputs. Unfortunately, you rarely ever have the exact same inputs, so you'll need to add more code to fix edge cases, and at some point one person can't keep all your rules in their head, so you need two people, now they need to communicate and it goes on and on.
The other part is very much a cultural thing, state administrations are well-known for being cushy jobs where you won't have to do much and attract the kind of person that wants that job. Every now and then someone will see the state of bureaucracy in Germany, think "I'll help!" and get themselves hired, and then burn out within half a year of trying to change the culture around them.
I'm only half joking: the best thing to happen to most small-ish bureaucracies in Germany would be to have a team building exercise with an accident that leaves everyone unable to work for 2 years so that there's enough time to build up a new team, install a new culture and clean up the processes. When the old ones are ready to come back, you can slowly introduce them into healthy teams, so that they'll take on the new culture.
Hum, yes, that's strict legalism. It was the fashion a few decades ago.
Do you have insight on the inner working of your local bureaucracy? Because it's hard to believe in a country as large as Germany people are so outside of the curve. Maybe you are seeing the outside appearance of a power structure that evolved in some past decades, and that won't tell you anything about what people are trying to do, just about what people are allowed to.
(Or maybe Germany public service is almost all composed of people nearing their retirement and with old ideas. It's not that unlikely that an entire government is skewed in age.)
I have a friend who works on a large federal digitalization project and he's ready to quit because it drains the life out of you for those same reasons (rules abound, nobody wants to make a decision, and expensive consultants and lawyers who are brought in to take the blame -- only they don't care for taking the blame, they want to bill as much as possible, and so everything takes forever). Yet he won't, because he believes that it's important work and will help make things better, and so he'll suffer and probably snap in a few years down the line.
To me, it looks like a culture issue. I'm a freelancer and have worked with quite a few teams over the past 20 years and some teams are just broken because of culture. They're places where nobody cooperates willingly and nobody takes ownership of anything, where fear stalks the offices and ideas go to die. And then you have places where people are proactive, and will handle a task and when an additional thing pops up a week later, they'll say "I've anticipated that and made sure it's easy to add" and things just work, because everybody works so that the team's future version has an easy time. I'm sure a broken culture can be fixed somehow, but I don't know if you can do it gradually because new members will adapt to the existing culture and perpetuate it.
https://goo.gl/maps/MDRCK7upDPPiPKqa6
Manipulative people of all ages exist, just look at how judges dress up in fancy dress wearing grey wigs and black or red garbs in order to subconsciously protect power and wisdom, at least here in the UK they do.
Its also why the UK police wear black, psychological projection of power, but laws don't recognise this subtle manipulation of the state. I know German police wear green, and I think the French police wear blue, which gives you an insight into how they want to be perceived. The US police also wear black to project power and status.
So with the fact that no country as far as I know teaches law apart from maybe Iran and other middle eastern countries, how can anyone claim to live in a democracy if they are not kept up to date with law changes, let alone debate these changes, because the public are resource burned and hand over their decision making to their democratically elected representative?
What a con trick people get tricked into that also gets people reinforcing and justifying their vote decision.
Likewise whilst some chemical consumption can alter behaviour, alcohol perhaps being the most widely known, its less widely known what prescription meds and illegal meds will do to the body and thus behaviour.
And here is the point, knowing the state dont teach people what these different chemicals will do, again how can anyone claim to live in a democracy, if they are not informed in order to make an informed decision or give informed consent.
Its extremely lazy if not deceitful of the legal system to assume that once someone turns 18, they can give informed consent to anything, knowing how widely businesses especially big businesses have things like planned obsolescence built into products and services, in order to carry out their own form of law and order.
Perhaps the reason why a definition of consciousness doesn't exist is because this would frame and lock down the narrative on informed consent, which is what much of the law hinges on in order to award punishments of sorts.
In some respects the state treats the public like a dog, it throws it a stick, the dog chases after it, and happens to fall off a cliff with the owner blaming the dog for its mistake.
A simple example to combat the intellectual warfare some parts of society like to rein down on the public.
Police in the United States are close to this already. Chicago's police department used to brag about its data-crunching ability to predict where crime would happen. At the time, it was just by area, but it was stated that with enough data, it could be narrowed down to the block. It's not that big a leap to extrapolate that to the individual.
My guess, though, is that it didn't work, and the city was just parroting the promises made by the salespeople of whatever system they bought for this task. If it worked, then police would preemptively flood certain areas where they know crime is about to happen. But, as is seen on television most nights, and in the complaints of the aldermen, the police mostly seem to react to crime, rather than prevent it. The use of social media by crime mobs has only made it worse.
Predictive policing https://en.wikipedia.org/wiki/Predictive_policing
Predictive policing in the United States https://en.wikipedia.org/wiki/Predictive_policing_in_the_Uni...
Gunfire locator https://en.wikipedia.org/wiki/Gunfire_locator
- Shot spotter
https://en.wikipedia.org/wiki/ShotSpotter
Radiolocation https://en.wikipedia.org/wiki/Radiolocation
Trilateration https://en.wikipedia.org/wiki/Trilateration
https://youtu.be/zb5RJyrk4gc?t=1m05s
Fusion center: https://en.wikipedia.org/wiki/Fusion_center
ResponsePod, Camp-in-a-Container https://shiftpod.com/responsepod/our-2020-catalog
Zello Work https://support.zello.com/hc/en-us/articles/230748827-How-do...
Sangoma Meet https://www.sangoma.com/products/communications-services/tea... : "Use the Sangoma Meet app to join a video call, or join audio-only by calling the dial-in number in the meeting invite."
https://www.networkworld.com/article/3134324/beyond-911-othe...
https://988lifeline.org/
Clean language interviewing, coaching https://en.wikipedia.org/wiki/Clean_language
This seems like a judgment that focuses on process and not outcome.
I have probably butchered what my friend said.
Just imagine, everyday 6AM an officer come at your door to check your apartment "to prevent" your crimes, suddenly every random guy in the street looks suspicious and spying on you, then some officers just randomly walks by your workplace, police stop you everytime you travel... This would eventually lead to an insult to the mafia guy and then boom you have already commited contumacy and the model was right! You are a criminal and you should be punished!
Frankly that seems more plausible than telling the future
And I think that is ok...
But it is clear that this legislature is pushing for crime prevention based on aggregated data from computer systems(where from, exactly?)
The arguments are that a manual extraction and collection would be too slow and that this should be limited to only a handful of severe potential crimes. As a prevention rather than mitigation technique.
This is quite bad.
Now the objection is not purely because the data is coming from computers, but because it will reveal data on people tangentially connected to the targets.
This is so wrong in so many ways and the people who are pushing from this have learned nothing from the stasi and nazi regimes, or perhaps they are nostalgic.
It is very clear that these things are pushed by rather smart minds, matter experts and this makes it clear they do not care about the innocent peoples privacy, as they could not possibly be so ignorant to overlook this matter.
Justice even the conventianal way spawns injustices, now the lazy government wants to automate and scale their procedures. W.t.h.
It isn't just about their privacy but also their very freedom. Being in some police database might very well land you in the crosshairs of some investigation and related niceness like searching your home, taking you in for questioning and keeping you there because you didn't say something...
Anyone who ever end up an their radar would probably never be removed from that,and never even know.
It is highly offensive that the German government pretends it does not know where the crime is happening. How about they deal with a few of these clans numbering 100s of people? This can be done , has to be done the convential way, as these clans will not be as stupid as leave digital traces, as computer illeterate as they might be.
The government has not even defined who they are targeting, they just use the commonly thrown around terms.
The government is just after the really easy convictions on the cheap. Traditional investigation is complex and expensive, needs personnel, etc. Sifting through vast databases for easy wins is far cheaper. Catching big fish also isn't opportune, because that would mean politicians being responsible for incarcerating their buddies, like in Cum-Ex. In related news, the most common reason to be incarcerated in Germany: Using public transport without a ticket... Most convictions for the least amount of effort.
Feels like a reasonable description of modern society run by zeitgeist of MBAs.
“I want to extract the most value for the least amount of effort on my part.”
I was working that this behaviour by FAANG will normalise it, and will lead to our governments doing the same.
Same 'smart people' work for government and fang, they are gonna behave the same way.
Skipping ahead to the fact that automated data analysis is not prohibited per se, just needs to be reasonably targeted, the two most interesting paragraphs are the last ones for me.
Basically the court mandates that any future laws passed for legalizing automated data analysis must explicitly
1. For any given crime to be automatically analysed, define the data model and define appropriate analysis techniques or methods. Those laws should be interesting reads.
2. For any one analysis happening, document and later publish to the public domain the reason for initiating an analysis, the methods chosen, the criteria for labeling someone as "guilty/innocent" - basically the entire process so that the analysis is auditable end to end.
Again, this is not a suggestion in the ruling, but a requirement.
This is really good news for personal liberties in my book.
Stark contrast to US/ China data mining on citizens.
Whereas you say: >Skipping ahead to the fact that automated data analysis is not prohibited per se, just needs to be reasonably targeted, the two most interesting paragraphs are the last ones for me.
I think this reads like the court curtailed reasonably broad data grabbing behaviors such as those Snowdon has revealed happened illegally and are still happening in the US. It still permits effective, focused/targeted data mining.
So at least the court learned from the Nazi/Stasi times but also from the times of left-wing terrorism ("Rasterfahndung" during the Bader-Meinhoff terrorist gang) - i.e., that these techniques are effective yet must not be applied to broadly. If this reading is correct (I'm not a legal expert), then the main danger is that lawmakers are incapable (or unwilling) to implement requirements (1.) and (2).
This is more dangerous than most people would think, once something like this is codified into law, many eu countries will follow suit, just for good measure.
What are the motives and who benefits from this? This idea was born somewhere and writing all the drafts costs a lot of money, who was footing this bill and why?
(scroll down to 3.1.(g) Public sector)
https://www.mondaq.com/france/technology/1059760/artificial-...
> whenever an individual decision is taken on the basis (even partially) of an algorithm, the administration must [...] explain [using easy to understand language]†, at the request of the individual, how the relevant algorithm works [...], by providing the following information:
† a very important detail that this translation for some reason omits ??Now, something that has been bugging me since then, especially in the «AI» context that was also being discussed more or less at the same time by the legislators, is that so far this is impossible to do for neural network based programs, since the final program is not based on any algorithm (as commonly understood), or at least not one that even the programmers themselves could understand, much less distill into a commonly intelligible form !
For eg: denying service from a German retailer depending on the ML prediction result of the transaction being classified as fraud probable(Germany has pay by invoice), would also be constitutional or not?
Or credit score/risk rating using ML, trained on a feature extracted from parameters like Zip code, ethnicity could also classified as automated data analysis for prevention of criminal acts, right?
Or this is only applicable to police?
The reason they do this isn't to fight crime, but to save the business from losing money. So I don't see how a ruling about preventing crime would apply.
The truth is that Law Enforcement investigations are currently taking information from dozens of silos/databases, manually copy&pasting them into Excel and Word and then finally making a pdf or powerpoint to share the results. This is obviously a really inefficient way to do things and wastes a lot of taxpayer money.
A technical solution that would allow LE to crosslink data they already have is a no-brainer.
P.S: Very biased view, as I'm building/selling exactly this tool :)
A technical solution that is Open Source, that falls under federally mandated criteria for how it can be legally used, that automatically documents any and all access and makes the entire audit log part of any output result so that all happenings are easily auditable by 3rd parties... might be a no brainer.
Anything else I'm highly skeptical.
If going after syndicated and or international crime, fine.
First feature I'd like to see is linking Finanzamt with LE and go after money laundering - Germany being an absolute paradise to the detriment of everyone else.
If it takes manual copy paste to protect ordinary citizens from overreaching police, I welcome that technical limitation.
Very biased, as I am an ordinary citizen.
If you introduce automation doing the same thing at a scale corresponding to 10,000× the human capacity, you will have 10,000× as many errors. This will violate Blackstone's ratio and erode trust in the system.
This roughly translates into "there has to be a sufficiently concrete danger to warrant such a ambiguously formulated infringement."
I truly appreciate this sentiment. Any collection or analysis of data "just because there might be something in there" has to be weighed against the right of everyone that is being data-mined.
This freaked me out about Minority Report when I first watched it: those precogs are essentially hacking into the future memories of everyone involved in _and by happenstance surrounding_ the lives of future criminals and their crimes.
Mass-scale datamining is effectively the real-world equivalent of telepathy. And I want neither precogs nor the government in my mind.
It's the working class who pay the price at the moment. We get burgled, mugged and killed, meanwhile the wealthy lawyers and politicians can pontificate over constitutionality from their gated communities and private vehicles.
It's insane that the EU is trying to ban automatic facial recognition with the terrible AI act - https://eur-lex.europa.eu/legal-content/EN/TXT/?uri=celex%3A... - nevermind the sheer amount of bureaucracy it introduces for startups (a government register of "possibly harmful" AI - wtf?).
Bear in mind this is the same country that bans Google Street View (and on the road to ban nuclear power and GM crops).
People here support this ban because they think it's "harmful", but how long until they think some technology you rely on is also harmful? GPS? Phone cameras or Google Lens? Encryption - https://mullvad.net/en/blog/2023/2/2/stop-the-proposal-on-ma... ?
Technology is the future, and we should embrace it wherever possible - crime prevention, education, healthcare, nuclear fusion, genetic engineering, etc.
My best friend is a sex worker. I use cannabis to control my anxiety. Neither of those are legal. Looking historically, virtually every human rights movement required illegal direct action. Where do we fit into your cashless panopticon society?
The worst crimes I'm worried about are the ones that will never be illegal, the ones that benefit the powerful and rich. At-scale invasion of privacy is only one such tremendous crime, one against against human rights and dignity, but barely against the law.
> Bear in mind this is the same country that bans Google Street View (and on the road to ban nuclear power and GM crops).
> People here support this ban because they think it's "harmful", but how long until they think some technology you rely on is also harmful? GPS? Phone cameras or Google Lens? Encryption - https://mullvad.net/en/blog/2023/2/2/stop-the-proposal-on-ma... ?
Some of those are genuinely harmful (street view in particular for its privacy invasion - note it is not banned but simply very unpopular) so I'm not sure what your point is. Also, you should send your ideas to the Mullvad blog authors and get their input. I think you'll find that the overlap between people who support universal end to end encryption and people who support an all-seeing police state is very small, as they are fundamentally incompatible ideas.
> Technology is the future, and we should embrace it wherever possible - crime prevention, education, healthcare, nuclear fusion, genetic engineering, etc.
Technology is not always good all of the time. The past century and even past decade are full of examples of net harms caused by technology that could have been avoided if we developed them in a more intentional controlled manner.
A eccentric example to make the point: If there is a technology that can trigger a supernova of the sun simply by mixing water, sugar, gasoline, and bleach together in a certain ratio, then it would be extremely important we not conduct research into that technology until we have other technology that allows us to survive a supernova or can prevent that mixture from being created anywhere on Earth. Even if there are advantages of the technology (maybe a different ratio causes all humans to have 5x longer lifespans and eliminates all disease and suffering, and we gain intelligence allowing us to begin colonizing other planets within 3 years) the risk is just too high that it will be misused accidentally or maliciously until we have a way to handle it. https://en.wikipedia.org/wiki/Differential_technological_dev...
I'm receptive to the idea that scale, permanence and universal access can represent qualitative changes to the impact some kinds of information might have on people. I'm just not understanding how that applies to images of public streets.
Germany leans heavily to more and more privacy because they have the experience in recent memory of what a government can do with vast information collection and centralized authority.
America leans lightly towards more and more centralization of data because they have the experience in recent memory of what individuals who "slip through the cracks" can do. So in Germany we see decisions like this and in the US we see police using privately-curated DNA databases to defrost the cold case of a serial killer, and they're lauded as heroes.
(It's worth noting, of course, that this is all relative. Relative to most of Europe, the US is starting from a place of massive decentralization and minimal individual-tracking. Some people still find social security numbers controversial).
I theorize that in a hundred years or so, North Americans will have learned this lesson the hard way and will have atrocities in recent history, too.
I'm sad I won't see the day when Bluffdale is repurposed as a memorial to state surveillance, much like the Stasi headquarters are now in Berlin.
https://www.stasi-unterlagen-archiv.de/mfs-lexikon/detail/ze...
Link is german: Shows how denunciation aka controlled leaking of private life like affairs was used, to destroy the lifes of resisting citizens.
PS: FB and Google could in theory use similar instruments against citizens that put pressure on them. Nobody could ever catch them at it.
If the ccp ever gets swept away and replaced by a new better china, the resulting parties will outdo one another to promise freedom and privacy.
Yes it wasn’t as bad as many other places. But bad enough for people to know there are things you can and can’t talk about in public. Growing up it was interesting to observe how my history class evolved from elementary to high school as news and declassifications trickled out.
https://en.wikipedia.org/wiki/State_Security_Administration_...
https://en.wikipedia.org/wiki/Category:Political_repression_...
Bonuspoints: If you had family who was horrible in the Nazi times and the descendants then were horrible during Stasi times. And you know, anyone who can remember the name can crawl that info.. makes you wish, the ground could melt and swallow you.
Imagine that info got bundled into a family dharma app, were crimes would follow your family name through the generations. Nowhere on the planet, can you escape the murder spree of grandpas uncle.
In general, most people have at least one near ancestor that did something terrible. That makes none of the descendents terrible people.
Our concern with privacy is so prevalent that fairly recently the collection (and subsequent usage) of metadata by police was deemed unconstitutional[1]. This was met with a lot of complaints by investigators, and while there's some sectors pushing for a constitutional revision, this seems unlikely at the moment, but not entirely off the table.
[1] - https://verfassungsblog.de/how-the-data-retention-legislatio...
And the fallout after Yugoslavia fell apart had the same people and criminals unpunished, with the climax of a broad daylight execution of a prime minister.
For instance, in France specifically since the 1974 SAFARI scandal that resulted in the creation of the French data protection authority and assorted laws in 1978, and happened when the government in secret attempted to (and had to stop due to the outrage) link various (then paper) administrative files into a centralized computer database through a single social security number.
In the end the government cancelled the program because of free will. A person always has the choice to change his actions because his actions aren't predetermined.
just like once upon a time it was constitutional to own slaves and now it isn't
just think about the monetary cost-savings! (nevermind the human misery)