She has good points. However, when practically implementing this, I could see the "any measure that becomes a target ceases to be a good measure" effect biting back: I imagine many companies would suddenly become very eager to insist that their algorithms are in fact not algorithms at all but "simply computer code" or such - and a lot of hair-splitting will ensue about what exactly is or isn't an algorithm...
(I'm not arguing this shouldn't be done, only that it would be messy)
Well you could technically demand the companies show the source to this hypothetical organization. AFAIK the FAA will take a look at the software that runs on planes, and how well you can trace requirements, whether the requirements are tested. Still Boeing and Airbus don't have their code on GitHub.
Don't get me wrong. I think it's possible, but it would be so costly that you would need to close nearly every software business in the US.
Safety critical software costs about 100x as much money/time to develop as compared to non safety critical software. "Algorithms" are everywhere, so certification costs would make any software shop as costly as automotive and aerospace. That's just completely misguided. Countries without such restrictions would take over the global software market quickly.
> I imagine many companies would suddenly become very eager to insist that their algorithms are in fact not algorithms at all but "simply computer code" or such
All you need to do is make human(s) accountable for decisions taken. It doesn't matter what is fed as input into the decision making process.
Maybe a good idea, maybe not, but targetting "algorithms" is certainly too broad. Everything that does anything is an algorithm. Will we have to get government authorisation before we can quicksort?
This should be restricted to anything making decisions based on personal information, if it is to be implemented at all.
Also:
> What digital ecosystem do you personally live in? Apple, Google, Microsoft?
> [...] There’s a big difference between the way that Google deals with their photos and the way that Apple deals with their photos. Apple photos are your photos. You keep them. When they do facial and image recognition on them, they use meta-level features of your data. The way that they’re communicating is in the most private way that they possibly can while still collecting the data they need.
You don't need to live in some company's digital ecosystem. I don't. I own my own photos, and Apple are not getting any facial recognition features out of them.
For someone who is clearly a strong proponent of privacy, it's disappointing that she doesn't even bring up the prospect of opting out of these companies.
> it's disappointing that she doesn't even bring up the prospect of opting out of these companies.
Hear hear. The lack of this in such articles makes me feel this is less of a grassroots boycott and more of a political hatred for consumer information tech companies.
I don't think I've ever heard anyone at Uber being held liable after the Uber car crash (in which Uber disabled the software's capability to do an emergency stop, which is certainly negligent).
"Maybe a good idea, maybe not, but targetting "algorithms" is certainly too broad. Everything that does anything is an algorithm. Will we have to get government authorisation before we can quicksort?"
I agree that the word "algorithm" is used in an unusual meaning here, but I don't think we have a good alternative. It's probably a lost cause at this point, like with "hacker". Possible option is to use qualifier, like "decision-making algorithm" (similarly to how people now use "blackhat hacker").
"For someone who is clearly a strong proponent of privacy, it's disappointing that she doesn't even bring up the prospect of opting out of these companies."
I find her FDA analogy very compelling. (Despite the fact that I largely agree with Cyberspace Manifesto.) I mean, without FDA, it's very difficult to opt out of a restaurant that serves bad food, without somebody getting hurt. Maybe you can say, well, then don't go to restaurants and make your own food. Sure, then you will eventually end up removing yourself from civilization entirely.
The reason it is different to the FDA is because a restaurant can't exist without serving food, whereas a company providing a camera app can exist without running facial recognition algorithms on the photos that are taken.
And "opting out" of the data-harvesting companies isn't akin to removing yourself from civilisation. You can still have photos, you just keep them on your own storage and don't give them to data-harvesting companies.
> For someone who is clearly a strong proponent of privacy, it's disappointing that she doesn't even bring up the prospect of opting out of these companies.
Because for many it simply isn't an option. A lot of people need to interact with those platforms for their work or to stay in touch with their social circle. Even if not, it requires technical expertise to find and use alternatives, something that a lot of people don't have.
There was a time when basic literacy required a level of expertise that most people didn't have, the answer wasn't to start highly regulating those who do have the expertise, the answer was for people to learn.
That analogy does not work. The discussion is about regulating those who can exploit the lack of expertise. Whether or not you have expertise is orthogonal.
To stay with the example, in a world where few people can read but dealing with knowledge written in books is essential for many, it would make sense to regulate lecturers.
> This should be restricted to anything making decisions based on personal information
It strikes me that the most obvious thing the US needs is some version of GDPR. It's a much better framework than some vague "regulator of algorithms".
The book contains more then the private personal data topic that we discuss here, a topic mentioned that I never seen here on HN is the algorithms used in justice.
I listen the audio book, some things that I learned and I did not know before:
- there are algorithms used in justice to determine some risks, so you getting a lower bail or a longer sentence is decided by an algorithm and some of this algorithms can be proprietary,racists and unfair.
- I do not recall all the details but a judge was identified and his web surfing was identified only by using the ad tacking systems, this was done by some reasearchers but some bad guys could potentially blackmail important people
- there were some interesting stories about people that used genetic test services, if your relative use those you are part indirectly in the data collection
- I forgot the details but in some case after some lawsuit the proprietary algorithm was revealed to be an Excel spreadsheat and it had some mistakes or unfair data in it
There were some good examples where AI strengths were combined with human strengths and the results were improved.
Large problems happen when the human blindly trusts the computer and almost drive over cliffs.
> Large problems happen when the human blindly trusts the computer and almost drive over cliffs.
I remember when car navigation systems were a new thing and there were frequent news reports of people who drove into rivers or lakes because the navigation system told them to cross the water, but not that the crossing happens by ferry (rather than via a bridge).
Other case mentioned was with face recognition, imagine a coop trusts the AI that tells him that you are a dangerous criminal, you could get killed if you are unfortunate because the cop could assume the computer is right and you are very dangerous and never thinks about the false positives
> You don't need to live in some company's digital ecosystem. I don't. I own my own photos, and Apple are not getting any facial recognition features out of them.
Workers, patients, and customers don't get this choice. I work on hospital systems that use "AI" and we wouldn't survive a proper code audit for what we claim saves lives. We get audited by the FDA, but they aren't combing through our code - just the documentation that we write.
Sure, we describe the algorithms put together by our researchers, but even the rank-and-file developers mostly just consume the algorithms that decide what to tell Doctors taking care of the patients.
For now, at least, Doctors and Nurses mostly use our software as a retrospective tool. I know we're not selling it that way and I worry about the day folks are using this system in place of their years of experience.
The statistics we put out say our software is x% better than the Doctor at making these decisions, is that true? Who knows, but an audit to make sure we're not selling snake oil might save lives.
> Everything that does anything is an algorithm. Will we have to get government authorisation before we can quicksort?
This just sounds like a strawman argument. If you say, 'our proprietary algorithm will save thousands of lives from malpractice,' you better be ready to back that up with an audit. It better not be a bridge that's going to fall apart when an eighteen wheeler travels across it for the first time.
A code-level government audit of all life-critical software sounds impractical. There are already millions of lines of code in production on FDA approved devices (e.g. pacemakers?) so I am sure that standards and certification processes around this already exist.
> Maybe a good idea, maybe not, but targetting "algorithms" is certainly too broad. Everything that does anything is an algorithm. Will we have to get government authorisation before we can quicksort?
Maybe I'm being paranoid, but I fear the "You know what I meant!" paradigm.
"Of course you won't have to register every algorithm. You know what I meant!"
"Of course you won't have to submit every program to approval. You know what I meant!"
"Of course you don't have to worry if you got sufficient consent to use that information. You know what I meant!"
And so on. The gag, of course, is that nobody really knows what They meant, only that there's now this massive chill upon the land because nobody wants to be the first test case for the system to demonstrate What They Meant on.
You can have a law that only affects algorithms used by the state in education,medicine, police and justice, this algorithms should be fully open or at least properly checked to be correct. Also the people using this algorithms need to be trained not to blindly trust the computer, they need to learn the failure cases.
> It’s never been quite clear, she says, whether the phrase—which is frequently the entire output of a student’s first computer program—is supposed to be attributed to the program, awakening for the first time, or to the programmer, announcing their triumphant first creation.
It seemed perfectly clear to me. "hello world" is the purposed program talking to the user of the computer on behalf of the service provider who wrote the application. The second lesson was then to usually request user input from that user.
> We need to make algorithms transparent, regulated, and forgiving of the flawed creatures that converse with them.
Do we? Do we really? Pray tell why...
> you could just put any old colored liquid in a glass bottle and sell it as medicine and make an absolute fortune. And then not worry about whether or not it’s poisonous. We stopped that from happening because, well, for starters it’s kind of morally repugnant. But also, it harms people. We’re in that position right now with data and algorithms.
No, we aren't. "Algorithms" is being abstracted here to mean, presumably, large scale high cost data collection infrastructure and logistics and business. That's not algorithms, it's something else.
> You can harvest any data that you want, on anybody.
No, you cannot.
> You can infer any data that you like, and you can use it to manipulate them in any way that you choose.
No, you cannot.
> you can roll out an algorithm that genuinely makes massive differences to people’s lives, both good and bad, without any checks and balances.
Presumably you mean sell optional, usually free services that people might want to leverage in return for access to their data. In reality there are only a dozen or less companies with reach on the scale that you might even start considering this a problem. Of course not using their services or applying various legislatively controlled options on those platforms protects you fairly well. Also not "algorithms"
Perhaps there is some idle minded conflation with the risk associated with data breaches?
> A regulatory body that can protect the intellectual property of algorithms, but at the same time ensure that the benefits to society outweigh the harms.
Insidious at best. The "algorithm" behind academic consideration of how to employ themselves by intervening in something that doesn't need intervention is certainly something I wish we could "regulate".
The SEC does things like banning behaviors and requiring transparency, both of which would help keep a lid on "algorithms" without really stifling innovation, vs the FDA model where it decides if things are safe before allowing them to be used.
I guess food regulation is like that compared to drug regulation.
"I went to go and give a talk in Berlin about this paper we’d published about our work, and they completely tore me apart. They were asking questions like, “Hang on a second, you’re creating this algorithm that has the potential to be used to suppress peaceful demonstrations in the future. ... I’m kind of ashamed to say that it just hadn’t occurred to me at that point in time. Ever since, I have really thought a lot about the point that they made."
I thought this was about algorithms and data structures which made me curious.
But the article is about how intentionally or accidentally sloppy data analysis or processing should be regulated if it can have a large impact on people or lead them to erroneous conclusions. Same could probably be suggested for using amateur statistics used to back up claims, any claims.
> You can harvest any data that you want, on anybody
Fear mongering. There are already laws in place about certain kinds of data such as GDPR, PCI or HIPAA. HIPPA violations can include prison time. Please be specific with what data and processing of such data causes harm and why rather than saying, "computers are scary."
But companies tried as much as possible to work around GDPR, using stupid tactics like popups with the option to not allow data collection hidden on hard to reach settings page.
To evaluate whether this will work in practice I looked to the actual FDA and the Patent Office. The FDA: budget issues, understaffed, susceptible to political attacks. The Patent Office: budget issues, complete lack of ability to make fair and balanced decisions on software patents (e.g., the linked list is patented by Google, https://patents.google.com/patent/US7028023B2/en). I also am concerned with the broadness of algorithms. Better might be if we restricted it to algorithms in applications which could affect human safety or financial transactions. Still, given the past examples, I'm doubtful this would work well in practice.
Wouldn't that put unnecessary regulations + bureaucracy into the path of innovation?
Like copyrighting/patenting algorithms and ever-greening comes to my mind. Though anyone with a PC can comeup with a kickass algo, while not the same as coming up with a blockbuster drug.
Anecdote:
I once suggested a change to the java standard library to change a O(n) algo into a O(1) algo. Now, more than 10 years later, it's still the O(n) (I just checked the std lib's source code). Nobody cares.
52 comments
[ 0.17 ms ] story [ 123 ms ] thread(I'm not arguing this shouldn't be done, only that it would be messy)
Safety critical software costs about 100x as much money/time to develop as compared to non safety critical software. "Algorithms" are everywhere, so certification costs would make any software shop as costly as automotive and aerospace. That's just completely misguided. Countries without such restrictions would take over the global software market quickly.
All you need to do is make human(s) accountable for decisions taken. It doesn't matter what is fed as input into the decision making process.
This should be restricted to anything making decisions based on personal information, if it is to be implemented at all.
Also:
> What digital ecosystem do you personally live in? Apple, Google, Microsoft?
> [...] There’s a big difference between the way that Google deals with their photos and the way that Apple deals with their photos. Apple photos are your photos. You keep them. When they do facial and image recognition on them, they use meta-level features of your data. The way that they’re communicating is in the most private way that they possibly can while still collecting the data they need.
You don't need to live in some company's digital ecosystem. I don't. I own my own photos, and Apple are not getting any facial recognition features out of them.
For someone who is clearly a strong proponent of privacy, it's disappointing that she doesn't even bring up the prospect of opting out of these companies.
Hear hear. The lack of this in such articles makes me feel this is less of a grassroots boycott and more of a political hatred for consumer information tech companies.
Why would you exclude safety-relevant systems, e.g. in the autonomous driving / autonomous flying / industrial control space?
If you are negligent or malicious, I'm pretty sure you're already liable for any harm caused.
I agree that the word "algorithm" is used in an unusual meaning here, but I don't think we have a good alternative. It's probably a lost cause at this point, like with "hacker". Possible option is to use qualifier, like "decision-making algorithm" (similarly to how people now use "blackhat hacker").
"For someone who is clearly a strong proponent of privacy, it's disappointing that she doesn't even bring up the prospect of opting out of these companies."
I find her FDA analogy very compelling. (Despite the fact that I largely agree with Cyberspace Manifesto.) I mean, without FDA, it's very difficult to opt out of a restaurant that serves bad food, without somebody getting hurt. Maybe you can say, well, then don't go to restaurants and make your own food. Sure, then you will eventually end up removing yourself from civilization entirely.
And "opting out" of the data-harvesting companies isn't akin to removing yourself from civilisation. You can still have photos, you just keep them on your own storage and don't give them to data-harvesting companies.
Because for many it simply isn't an option. A lot of people need to interact with those platforms for their work or to stay in touch with their social circle. Even if not, it requires technical expertise to find and use alternatives, something that a lot of people don't have.
To stay with the example, in a world where few people can read but dealing with knowledge written in books is essential for many, it would make sense to regulate lecturers.
It strikes me that the most obvious thing the US needs is some version of GDPR. It's a much better framework than some vague "regulator of algorithms".
- there are algorithms used in justice to determine some risks, so you getting a lower bail or a longer sentence is decided by an algorithm and some of this algorithms can be proprietary,racists and unfair.
- I do not recall all the details but a judge was identified and his web surfing was identified only by using the ad tacking systems, this was done by some reasearchers but some bad guys could potentially blackmail important people
- there were some interesting stories about people that used genetic test services, if your relative use those you are part indirectly in the data collection
- I forgot the details but in some case after some lawsuit the proprietary algorithm was revealed to be an Excel spreadsheat and it had some mistakes or unfair data in it
There were some good examples where AI strengths were combined with human strengths and the results were improved.
Large problems happen when the human blindly trusts the computer and almost drive over cliffs.
I remember when car navigation systems were a new thing and there were frequent news reports of people who drove into rivers or lakes because the navigation system told them to cross the water, but not that the crossing happens by ferry (rather than via a bridge).
Workers, patients, and customers don't get this choice. I work on hospital systems that use "AI" and we wouldn't survive a proper code audit for what we claim saves lives. We get audited by the FDA, but they aren't combing through our code - just the documentation that we write.
Sure, we describe the algorithms put together by our researchers, but even the rank-and-file developers mostly just consume the algorithms that decide what to tell Doctors taking care of the patients.
For now, at least, Doctors and Nurses mostly use our software as a retrospective tool. I know we're not selling it that way and I worry about the day folks are using this system in place of their years of experience.
The statistics we put out say our software is x% better than the Doctor at making these decisions, is that true? Who knows, but an audit to make sure we're not selling snake oil might save lives.
> Everything that does anything is an algorithm. Will we have to get government authorisation before we can quicksort?
This just sounds like a strawman argument. If you say, 'our proprietary algorithm will save thousands of lives from malpractice,' you better be ready to back that up with an audit. It better not be a bridge that's going to fall apart when an eighteen wheeler travels across it for the first time.
Maybe I'm being paranoid, but I fear the "You know what I meant!" paradigm.
"Of course you won't have to register every algorithm. You know what I meant!"
"Of course you won't have to submit every program to approval. You know what I meant!"
"Of course you don't have to worry if you got sufficient consent to use that information. You know what I meant!"
And so on. The gag, of course, is that nobody really knows what They meant, only that there's now this massive chill upon the land because nobody wants to be the first test case for the system to demonstrate What They Meant on.
It seemed perfectly clear to me. "hello world" is the purposed program talking to the user of the computer on behalf of the service provider who wrote the application. The second lesson was then to usually request user input from that user.
> We need to make algorithms transparent, regulated, and forgiving of the flawed creatures that converse with them.
Do we? Do we really? Pray tell why...
> you could just put any old colored liquid in a glass bottle and sell it as medicine and make an absolute fortune. And then not worry about whether or not it’s poisonous. We stopped that from happening because, well, for starters it’s kind of morally repugnant. But also, it harms people. We’re in that position right now with data and algorithms.
No, we aren't. "Algorithms" is being abstracted here to mean, presumably, large scale high cost data collection infrastructure and logistics and business. That's not algorithms, it's something else.
> You can harvest any data that you want, on anybody.
No, you cannot.
> You can infer any data that you like, and you can use it to manipulate them in any way that you choose.
No, you cannot.
> you can roll out an algorithm that genuinely makes massive differences to people’s lives, both good and bad, without any checks and balances.
Presumably you mean sell optional, usually free services that people might want to leverage in return for access to their data. In reality there are only a dozen or less companies with reach on the scale that you might even start considering this a problem. Of course not using their services or applying various legislatively controlled options on those platforms protects you fairly well. Also not "algorithms"
Perhaps there is some idle minded conflation with the risk associated with data breaches?
> A regulatory body that can protect the intellectual property of algorithms, but at the same time ensure that the benefits to society outweigh the harms.
Insidious at best. The "algorithm" behind academic consideration of how to employ themselves by intervening in something that doesn't need intervention is certainly something I wish we could "regulate".
I guess food regulation is like that compared to drug regulation.
"I went to go and give a talk in Berlin about this paper we’d published about our work, and they completely tore me apart. They were asking questions like, “Hang on a second, you’re creating this algorithm that has the potential to be used to suppress peaceful demonstrations in the future. ... I’m kind of ashamed to say that it just hadn’t occurred to me at that point in time. Ever since, I have really thought a lot about the point that they made."
Not enough apparently
Can you expand?
But the article is about how intentionally or accidentally sloppy data analysis or processing should be regulated if it can have a large impact on people or lead them to erroneous conclusions. Same could probably be suggested for using amateur statistics used to back up claims, any claims.
Fear mongering. There are already laws in place about certain kinds of data such as GDPR, PCI or HIPAA. HIPPA violations can include prison time. Please be specific with what data and processing of such data causes harm and why rather than saying, "computers are scary."
https://en.wikipedia.org/wiki/Independent_Institute#Position...
Like copyrighting/patenting algorithms and ever-greening comes to my mind. Though anyone with a PC can comeup with a kickass algo, while not the same as coming up with a blockbuster drug.
can be done via De Bruijn graphs as explained here: https://incubaid.wordpress.com/2012/01/24/number-of-trailing...