The overly simplified explanation is that the AI models see patterns in the training data that we do not.
Feed a system a series of pictures of stop signs, and humans might key on things like the red color, or the shape, or even the word "STOP". An AI model might also recognize that they commonly have 2 bolts, and are attached to a post, and have a contrasting border.
Typical machine vision systems can only classify what they have been trained on, they are ignorant to other objects. Train a system on a bunch of STOP signs, and then show it a SPEED LIMIT sign with morning sun giving the sign an amber glow, and the AI system might call it a STOP sign.
Again, this is an overly simplified explanation, but we can explain how AI works, we just can't always explain or know what parameters in the training data it keyed on.
> AI experts and researchers are warning developers to take a step back and focus more on how and why a system produces certain results than the fact that the system can accurately and rapidly produce them.
We need to find a way to incentivize that otherwise it’s not going to happen. Perhaps looking how the security industry works with “buy bounties” is a place to borrow ideas from?
Yes and no. Yes, because I agree with what you said. No, because it assumes self-driving has to rely purely on normal vision/cameras, which is just garbage². Think LiDAR and other such means to detect obstacles, but things like that are (currently) pretty expensive, so there is the attempt to just do it with (more or less) normal cameras, which leads to what you said (and I fully agree with that). Aside of the fact that safety assumptions+validations towards cameras are complicated in itself already, not only the technology in all its aspects, but also dirty lenses and things like that.
LiDAR can't tell you mass of an object as it is headed towards your vehicle.
Imagine you're driving down the road and you see:
1.) A kid throw a rubber playground ball into the road and at your windshield.
2.) A kid heave a bowling ball into the road and at your windshield.
How would you react in each case? How do you distinguish between the two?
As a kid is heaving the bowling ball you can tell that it has a lot more mass by how the kid had to move his body to throw the ball. You learned that because you spent the first few years of your life experiencing the basic physics of reality. You've internalized the motion of the human body.
Same for both: brake. I don't want either hitting my windshield. The kickball may be bouncy, sure, but it may have gotten rocks stuck to it, which can create a point stress and scratch paint or chip the windshield.
Humans can't intuit mass unless the object meets certain criteria, either. What if it's an opaque cardboard box in the road?
In certain circumstances with a playground ball you would have to choose between extensive damage to the body of your car by swerving out of the way or to just take the risk of a chipped windshield from a playground ball that somehow has rocks stuck to the outside.
And have you seen ever seen a rubber playground ball with rocks stuck to the outside? Do you really hesitate to barefoot kick an unexamined playground ball that is bouncing in your direction? I mean, I literally did this yesterday. I was out barefoot in the neighborhood with my 1 year old and the neighbor kids were kicking a ball around and it bounced in my direction and I kicked it back to them.
"Treat every object as the same" is not at all intelligent behavior for a man or a machine!
In case of a car, the thrown object is the easy case.
Now imagine you spot a person walking down the road on a pavement. Suddenly, the person turns towards the edge. What do you do?
See, a human driver would look at the gait of the person from far away to evaluate for instance if they're sober, or if it's a child who might be expected to run in. Whether it is near a crossing or a potential crossing. Whether the person was walking or standing... Many other obvious and less obvious indicators.
AI currently sees a moving blob of pixels in a shape of a person. No advanced inference.
When interviewed in case of an accident, say because they got rear ended due to braking, a person can explain why they braked most of the time.
AI now couldn't even say which features it weighted.
My pet theory is that we'll need supervised training of androids that go through the experience of having to learn how to move their human-like bodies through space in order to make human-like intelligent decisions about objects moving through spaces primarily designed for humans.
My other historically motivated pet theory is that we're going to stumble across sentient machines and then enslave them. Put another way, I'm more worried about what humans do to artificial sentient life than what artificial sentient life does to humans.
I don't see why self-driving cars should have to be perfect though. I would argue that rationally I should start using self-driving cars as soon as they are better at driving than I am.
The fact there are edge cases that I handle better than the AI is not necessarily disqualifying, so long as the AI is so much better at the rest of the cases that on average the AI is safer. For example, the fact that I handle a piano that has been left in the middle of a freeway better than an AI might matter less than the fact that the AI responds 100ms faster when a car swerves into our path.
> I would argue that rationally I should start using self-driving cars as soon as they are better at driving than I am.
How would you know?
Of course the edge cases matter, since human drivers make minor driving mistakes on a too-regular basis but fatal ones very rarely. 100ms faster reaction time is very impressive up until the point the system interprets the 10,000th bridge support it passes as an offramp and attempts to exit via it, which makes it several orders of magnitude more deadly than humans as they start using it on that road...
In the US, more than 100 people are going to die in their car today. If hardware and software flaws kill people at a lower rate than the shitty drivers they replace, then that's a win.
However, we can't really meaningfully quantify it a-priori. So we need a sufficiently big set of accidents and accident free rides to see when it's better.
In addition we can't distinguish what driver would have been better or worse than the AI in a given situation.
This opens the door for all sorts of lawsuits and we will end up with legal self-driving systems in almost ideal situations only.
To note, this is as much an engineering problem as it is societal and legislative.
> To note, this is as much an engineering problem as it is societal and legislative.
Yeah, it definitely is. I think there are two things that should happen.
In the short term, producers of self-driving cars (or systems) should be given immunity from lawsuits related to crashes provided that the NHTSA (or some other authority) can verify that the deaths per passenger mile in their cars does not exceed the rate for traditional cars. Once the majority of passenger miles are in self-driven cars, revoke the immunity.
For the long term, software engineering should become a real engineering discipline and we require self-driving systems be signed off by licensed Professional Engineers. If those engineers ship software with bugs that lead to death or damage, those engineers can be held responsible, sued for malpractice, and have their license to practice revoked.
I'm not sure that the political 'solution' of giving special privileges to avoid immediate consequences from their actions to AV manufacturers and special responsibility for signoff in the longer term solves the engineering problem that they need each minor version update of their software to have driven tens of billions of real world miles before they remotely approach statistically significant evidence that it is less deadly than the last version, or indeed an average human.
And this is where lack of general AI or tractability is an issue; knowing that the software typically makes fewer driving errors than the last version over a typical route gives us no confidence whatsoever it doesn't handle rare edge cases marginally worse leading to a couple of extra fatalities per billion miles (making it less safe than the average driver, many of whom suffer actual legal consequences for their erratic driving killing a person even if all their other driving is 'above average'...). Humans aren't bug-free or particularly tractable either, but at least we have enough of a mental model of how they understand driving to be confident that training them on a certain road sign found in urban areas won't make them more likely to stop on a freeway.
I read neural network (NN) books in the 90ies, and they covered for example the problem with corner cases already. One thing I remember is a (comparatively simple) NN which created the past tense of an english verb, and there are exceptions like "go" and "went". It preserved that for a while, but further training brought it back to "goed", because NNs do generalize and you need tons of additional "storage" in a NN to keep such corner cases around. That also means if you need (near) 100% reliability you are in the wrong place when it comes to NN aka "AI".
I mean no responsible developer would take the output of a ML model and then act on that without any tests on whether the output makes sense or not. That's the way to handle unexplainable AI.
There's also opportunity to look at uncertainty estimation, i.e. look at the epistemic uncertainty of the model and use it as a proxy for potential error. That seems to be a main thrust for AI right now. I'm currently writing a paper looking at this for dynamical systems.
There is a somewhat widespread implicit assumption that having sufficient intelligence to develop increasingly sophisticated machine learning implies that we also have sufficient intelligence to develop an “intuitive” understanding of how it works.
I think that assumption is totally wrong, and trying to reconcile the two is probably a distraction and a waste of time.
Evolution led to human intelligence just fine on its own; why are humans not simply the catalyst in the next stage of this natural process from which something more complex arises?
With regard to the moral questions around machine learning, I think people are overthinking things. There needs to be a sharp (societal) line between inferring causation and the capability for prediction. Maximize the latter any way that you can, but a lot of careful thought should be put into how the results of a prediction are utilized. Instead we are currently going about this backward by trying to haphazardly “clean” the input data so that the output gets a free pass on how it can be used. We can never decouple all of the biases and eliminate intra-predictivity in the input data though, so it’s fundamentally a bad approach.
What people are really saying, but never will admit, is that they want AI to mirror their own biases, not to have none. That's the reason they are "cleaning" the input data.
I always thought it was the opposite. That we want AI to have no biases, but we have trouble avoiding it. Choosing data and cleaning is where we insert our bias.
What people want is for the models to show no differences between certain groups of people. That is the standard they use for "no bias." But that itself is bias, so really they just want the model output to conform to their bias that there isn't any difference.
In fact, the article cites the kind of bias that warrants eliminating. One example is the misidentification of people with dark skin by facial recognition technology, as this has led to the arrest or incarceration of innocent people.
Not, as you completely fabricate, "show no differences between certain groups of people."
So what you're really saying is, it's perfectly okay for AI to have negative preconceptions and concrete impacts to certain groups of people, as long as it agrees with you on which groups are bad.
How would you feel about AI models that classify all modern white people as racists based on overwhelming historical data about slavery, the KKK, etc.? Hey man, data doesn't lie.
Your deflection about the media is entirely unrelated to AI, but it only backs up my argument. If giant media corporations are biased, why can't AI researchers be?
Every institution has biases, and blindly trusting a dataset from e.g. a police department is just as bad as blindly trusting data from CNN or Fox News. The collection, aggregation, and sharing of data is a biased operation. If a political or for-profit entity gives you data, it's going to be data that supports their agenda.
It takes active, conscious effort to combat that bias, in the same way that it takes active, conscious effort to not just get all your news from one source. The concern (which has been demonstrated) is that that isn't being done
I often see that argument made in bad faith, eg when someone who believes conclusions from biased data encounters AIs making data driven conclusions, more than I do when the AI is genuinely biased.
I also have trouble taking it seriously, since that same standard isn’t applied to existing information services (eg, “media ethics” posts condemning their open racism) — only to new information services which might disrupt the current social order.
I should have stated that more directly — but I view these concerns largely as after the fact ethical nits from people whose comfortable myths are confronted by data, projecting their own manipulative behavior onto others.
AI is a developing technology that stands to be used by powerful businesses and governments for all manner of purposes that will likely drastically impact our lives. It's important to criticize what could become the next industrial revolution, and make sure we can get rid of as many systemic issues as possible before they become too big to fix.
And many people do criticise the biases of legacy media groups on both sides. But this is an article about AI on a site that is generally more concerned with technical innovations than societal issues, so AI is being criticized.
Besides, bad things existing now isn't really a good argument for why we shouldn't prevent bad things from happening in the future.
I'm gonna need a cite and some statistical analysis to back up your "media is biased" claim. One data point on an event that I easily found reporting on isn't going to cut it.
Correct. As a tool serving humans, we want the tool to reflect the decisions humans should make given the same broad swath of data.
"Unbiased machine learning models" is basically a nonsense idea; a machine learning engine is a discrimination / classification tool, and its entire point is to become biased based on inputs so that on future inputs, its outputs tilt over in a desired fashion instead of just being uncorrelated noise. Making those biases have the desired shape is the art and science of the process.
You are conflating the statistical meaning of bias with a social meaning of bias.
A statistically unbiased model would be (for example) in which it has the same false positive rate at identifying humans from image data, for varying races, or a model that is equally likely to underpredict as overpredict tomorrow's gas price. These are not nonsense ideas, in fact, they are often good ideas. There are cases, however, where we may want a model to be statistically biased, for example, if the detriment of underpredicting tomorrow's gas price is worse than that of overpredicting it. If the gas price is lower than I expected, maybe it's not so bad that I didn't buy it today.
We (well, some of us) also don't want socially/politically biased models. Examples of those are very devious: redlining, for example, prevented black Americans from equal access to housing, historically, and has caused multiple generations of wealth disparity as a result.
You are using really loose and sloppy language in your talking about ML; you should be using "trained" and not "biased" because bias has a technical meaning that seems to evade you.
I'm using biased as it is colloquially used and pointing out that when people ask for "unbiased machine learning" they generally need to be more specific, because the whole point of machine learning is to give better answers than random noise.
That its better to just blindly trust in the output of a blackbox AI output?
That naively feeding in all possible data into an AI is the "best" way to do it?
It is been well documented that blindly trusting AI just leads to it perpetuating human stereotypes and worsen flawed systems.
https://dl.acm.org/doi/10.1145/3531146.3533138
It's not some great insight that offering different inputs will result in different outputs and MAYBE we want to get better output with a different input.
Assuming that a human stereotype is untrue is just as biased as assuming it is true. If you judge a model by how well it confirms your previous bias, then the model tells you nothing you didn't already know.
This response is unnecessarily hostile. A machine stereotyping may have overwhelming accuracy. In that case, I would argue the machine is not wrong. It's our policy response to its outputs that may or may not be wrong. Changing the machine to disregard accurate descriptions of the real world only serves to bias the machine.
Setting aside the rudeness, you basically just agreed with the parent. They said people don't want an AI with no bias, we want an AI that mirrors own biases.
Your example of loans is perfectly in line with this: we have a social bias against racism and racial profiling, and we want our AIs to reflect that bias. Purely on the question of risk assessment, race must meaningfully impact a loan risk or the AI wouldn't develop that bias [1], but that isn't a correlation we want the AI to consider in its risk assessment, ie. we build in a different bias.
[1] race in this case would be a proxy for all sorts of complex historical socioeconomic factors
I think people are conflating the technical sense of "bias" to the social sense. Technically, a biased model is one that overfits the training data and has poor generalisation. Ethically, a biased model would be one that reinforces discrimination.
I agree the parent poster was using the ethical/social sense of "bias" in calling a discriminatory loan risk AI biased, but I don't think it's overfitted and so I don't think this is what the original poster meant.
In the loan example, the objective is to assess risk. Assuming the data we feed it draws equally from all races, the AI will still almost certainly infer a racial component to loan risk. This is not a statistical or systematic error, this is an accurate assessment of society due to long-standing racial inequalities.
I think the original poster meant that ethics is not an objective science, and so our preference for certain ethical reasoning is itself a bias. "Correcting" the "biased" outcomes we don't like that are apparent in the data, and that are a reflection of society at large, involves biasing the training data.
I completely disagree. You're right that everyone has biases - including the authors of the AI who will therefore choose a biased training dataset. When the tool they are constructing has unparalleled power, and is very likely to negatively impact the lives of those who the authors are biased against (consciously or not), it's entirely fair for those impacted groups to be concerned.
Every dataset has a skew to it, which should be accounted for during training. If the authors don't explicitly account for such things, out of ignorance or bias, then that skew, that bias, will be ingrained in the AI.
For example, if I only train my AI on it's knowledge of chocolate from ads for a particular brand, it's going to have some very opinionated, very wrong ideas about chocolate. This example is silly and obvious, but similar skews happen all the time in real datasets that the authors don't have the time or expertise to recognize.
When people who do have that expertise speak up, we should listen to them and fix it, not just blindly trust "the data" and whatever our fancy algorithm does with it. Garbage in, garbage out.
These imbalances in the training data are there because the world generates data in an imbalanced way. Sometimes you can correct for it, other times it's impossible. You simply can't find good data for all your cases. For example in medical diagnosing or in self driving.
> There is a somewhat widespread implicit assumption that having sufficient intelligence to develop increasingly sophisticated machine learning implies that we also have sufficient intelligence to develop an “intuitive” understanding of how it works.
I agree with you on that point, but not with the conclusion that this is in any way a good thing.
> Evolution led to human intelligence just fine on its own; why are humans not simply the catalyst in the next stage of this natural process from which something more complex arises?
I think I have the complete opposite view here. Why would "being the catalyst in the next stage of evolution" be anything desirable? Evolution has no "stages", it's just chaos and life trying to deal in the best way with the circumstances at hand. If we blew everything up in nuclear armageddon tomorrow and the only remaining life were archaeae, those would be a perfectly fine "next stage of evolution".
Similarly, we can build a would in which only robots can survive, but why would we want to?
I think, gaining more understanding how the world works is the one thing AI can really bring to the table. If it solves practical problems on the way to that, that's great, but we evidently survived so far pretty well by doing that stuff ourselves.
> The people who develop AI are increasingly having problems explaining how it works and determining why it has the outputs it has.
I don't think this is anything new. This was already the case 20+ years ago with chess-playing computers.
In the mid-90s, Deep Blue was evaluating 200 million chess positions per second. How do you explain the resulting moves? Obviously we know they were the result of a deep minimax-style parallel search with a certain evaluation function, and we could simulate a similar search by hand if we wanted to. But this is no better than explaining the output of a neural network as "matrix multiplication plus a few non-linearities".
Even back then a chess expert could try and rationalize certain moves in human-like terms: "oh, Deep Blue realised it needs to fight for the dark squares on the queenside". But this wasn't a real explanation, just like "this part of the picture looks like dog hair" isn't necessarily a correct explanation for why an AI labels an image as a dog.
Whenever you perform a massive amount of computation, you can get results that are nearly impossible to explain.
I think these kinds of articles are geared toward people who do not know (even superficially) about the models. Perhaps the sentiment they want to elicit is "If the person who created these models cannot explain how they work, then he is not so different from me."
It is nothing new. It is called (at least) "the problem of transparency".
The chess context is probably not the best, because many systems allow a lengthy complex explanation of the response (I cannot remember now the exact workings of Deep Blue - it has been a while last time I met the full info).
It is a real problem in general, because we may not just want responses but we may want "to learn something", and specifically, because ANNs applied to e.g. Decision Making, or ANN based Decision Support Systems, pretty much require that natural intelligences assess the "oracular" proposals.
And of course, "understanding" the engine is required to advance it.
We deal with intelligence to either solve problems, or to benefit from intelligence to advance our own.
If you valued technical advance with a reduced concern towards civilization, ie. power over wisdom, ie. more "how" and less "why", you would get - well, for example, the present situation, that some consider "strongly suboptimal".
Some said that there have been two big branches of science fiction, that revolving around power (spaceships etc.) and that revolving around information, and that we have been just lucky to have reached the latter.
But some forms of machine learning actually are interpretable, which makes them useful in scientific investigation, which doesn't just want a black box. A good example of interpretation machine learning is decision trees -- given a series of variables and outcomes you can find out which variables are important to the outcomes and in which combinations.
> And of course, "understanding" the engine is required to advance it.
I don't think so. People are clearly making huge advances in machine learning even though they can't explain the systems made 10 years ago, let alone more recent ones. There's a lot of trial and error / cargo-culting involved. Some researchers go as far as comparing machine learning research to Alchemy:
> many systems allow a lengthy complex explanation of the response
All computations have this property: you can execute the code by hand for as long as you have time to. But if you do this for long enough you become a mere observer of the computer's steps, which won't give you any intuition as to why the final output is the way it is (unless the algorithm being run is very simple).
> "understanding" the engine is required to advance it
What I meant was more on the lines of "LeCun proposed Convolutional NN recognizing a promising paradigm", or "our prespective on Artificial Vision changed as we realized that the ANNs recognized textures instead of figures".
> All computations
And what I meant there was that in the chess systems of topic the calculations performed have been "as if symbolic", ie. reasoning on branches of consequences of the type "what could happen if I move this there" - even when adopting ANNs in some parts of the architecture -, as opposed to, say, "computing weights". That is the level "of machine code", not that "of electronics".
But an AI must “understand its reasoning” in order to revise it efficiently to learn from its mistakes. If revision is impossible, the only way to correct its reasoning is to re-train the system from scratch. But in doing that, you can only hope the problem is fixed, and worse, that new problems were not just created.
There is also the problem of label noise. A label might not be necessarily correct, especially when the dataset is large. Nobody got time to check them all, and even if you put two people they might disagree.
> Obviously we know they were the result of a deep minimax-style parallel search with a certain evaluation function, and we could simulate a similar search by hand if we wanted to.
You don't really need to though. The computer can justify its evaluation with a principal variation (best play by both sides) leading to a leaf position of the same value. And if a human were to wonder why at any point in this variation, some other move is inferior, the computer can again easily show a principal variation branching off from that other move, leading to a worse (or equal) result for the deviating player. In this way, minimax results are FAR better justifiable than neural net results.
Exactly, the Deep Blue's move explanations are boring - they all boil down to "based on the inputs and rules programmed, this line of moves has the best overall outcome to a depth of X" where X is however deep it goes.
You can try to translate that to human methods of understanding, but that's not how the computer "thinks", and attempting to do that translation leads to misunderstanding. Kasparov may make moves because he wants to get a better board position or knows his opponent is weak to certain positions (I have no idea how experts explain moves) but the computer isn't programmed that way.
I don’t think that’s weird at all, I think humans can intuitively grasp “I simulated 100,000 games starting from the current board state and going at most 50 moves ahead and in games where you did X you reached a strong position most often.”
Is that a useful explanation for a human who is training a wetware ML model, not really. But it’s really understandable compared to trying to explain neural nets.
>in games where you did X you reached a strong position most often.
FYI in standard minimax, the search is for the move that leads to the strongest outcome - the "most often" bit doesn't factor into the decision making process.
That's punting the problem of explaining down the line. All the PV tells you is the computer is making this move because it thinks that 20 moves down the line, you'll end up in some desirable position.
It doesn't abstract or summarize any understanding. If you say, "well, what if they do Kf3 instead", all it can show is another PV. It can't tell you "knights on the rim are dim", or "this pawn needs to move now to prevent the bishop entering the position in 5 moves".
You can try to build an explanation on top of a chess engine, but it's very difficult to come up with generalizable or clear explanations beyond very broad heuristics (like "knights on the rim are dim").
It doesn't provide a clear explanation in those terms, and in some fraction of positions, there is no simple such explanation. But I believe that in a decent fraction of cases, a human chess player probing PVs will gain a reasonable understanding of the winning or drawing strategy involved, and that such interactive PV probing is often a better form of explanation than a concise non-interactive textual one could offer.
There is no equivalent to that in neural networks.
Note that PVs explain the connection between the root position of minimax search and a leaf positions. Evaluation at the latter is easy to explain in terms of features like material, mobility, pawn structure, king safety, etcetera.
I agree that a human can use the PV to get insight into why a move might be good. I also agree that non-interactive explanations provided by a chess engine will often be inferior. I think one way to break it down might be that the PV (or sets of PVs) can provide a useful explanation of tactical value (Kf3 leads to mate in 6), but struggles to explain strategic value (Kf3 leads to the positioning opening up, which broadly makes it harder for you to press your advantage).
> There is no equivalent to that in neural networks.
The first example that comes to mind is using backpropagation to determine how much influence each value had on the output result, and then rendering it in a human-readable way. Stuff like this:
Interactive PV probing is just playing chess against the AI to better understand the position. That is literally the interface for which people built chess AIs.
Unfortunately that misses a lot of what makes grandmasters worth listening to. GMs will say things like "Kh6 almost works, but fails to ..." which is not something you can get out of a computer currently.
Computers currently fail to explain any moves which are good ideas, but are not the best move now. Humans normally describe these moves as "resources" which are good to keep in mind for the future or to play later when the board cools down.
The explanations used to explain or justify human behaviour are just as speculative though, really. We can try and hypothesize, but that's it - there is no engineering design specification for the brain, nor even a debugger.
Older ML techniques are explainable in nice ways. With something like logistic regression you get a very nice "this feature contributed this much to the final classification" breakdown. It just turns out that logistic regression is considerably less effective than RNNs, which have all these synthetic features within the inner layers that aren't easily meaningful to humans.
I implemented an interpretable LSTM from this paper https://arxiv.org/pdf/1905.12034.pdf. It works by segmenting the context for each input variable, and then applying an attention layer to interpret each variables effect.
I applied it to mortality prediction based on ICU data, the prediction/recall was pretty good considering the small size of the dataset used.
E.g. imagine somewhere in the code for your chess supercomputer is a genuine bug - some dumb typo, I don't know - which leads it to make the wrong decisions in some situations.
If you have no idea how the program arrives at its results or even if the results are correct or not, how would you be able to find that bug?
Similarly, how do you make sure the program doesn't learn garbage features or overfits on your testing environment? There are enough war stories of image classifiers that just learned some subtle lighting differences in the trainset photos of something that actually distinguished the object in question.
Or how do you prevent models learning certain features that you specifically don't want to learn, such as skin colour?
I think developing those models without trying to understand what they do has a high risk of leading to magical thinking.
> E.g. imagine somewhere in the code for your chess supercomputer is a genuine bug - some dumb typo, I don't know - which leads it to make the wrong decisions in some situations.
If the evaluation function executed at the leaf nodes of the search tree contains a bug that massively over-evaluates a position, you might be able to find this by having the chess computer play against a different computer that exploits this bug. Then you'd observe the positions at the end of the principal variation as some other comments mentioned.
Even if you can do all that, that still leaves the main problem unanswered which is to explain the correct moves (or rather, moves "assumed to be correct").
1. Machine learning is increasingly being involved in important decisions.
2. Deep neural networks have become a popular technique, and their results are particularly difficult to explain.
If you're a bank and your AI says somebody is likely to default on the mortgage they're applying for and you can't demonstrate that their race or a proxy therefor wasn't a factor in that calculation, they'll have a settlement and won't need a mortgage.
I think explainability is overrated (to use a Trumpian expression) & this fixation with coming up with explanations for inferences is a red herring. We cannot explain our own thoughts and actions and tend to ascribe logic & reason to many of our own actions, but it's almost always system-1 driven, for the most part. Correcting for edge-cases and unknown-unknowns is where we should focus our efforts methinks.
You can't explain your own thoughts and actions??? I'm sorry, but that's part of makes a valuable team mate or partner. This is not fundamentally human, this is trained behaviour. I am very well capable of expressing my thougts and explaining actions resulting from these thoughts.
Can you explain why you had a thought, however? And how many levels deep? Only as many as someone cares to ask, I'd wager.
You have developed the capability of explaining your actions as resulting from your thoughts, because there is someone on the other side who is ready to accept and evaluate your explanations.
Others have no such affordance, so their brains get trained in prejudices and superstitions indead. Even if an explanation of an action is given in good faith and accepted by the recipient, it can still be factually incorrect.
Thanks, but enough neuroscience studies have shown that our decisions are for the most part, on autopilot, qv: Kahneman, Tversky et al. All our "explanations" are post-hoc rationalization i.e. reality is the narrative we tell ourselves.
>You should explain the point about «correcting for edge-cases and unknown-unknowns», which may not be clear.
Our learning is also for the most part Hebbian, from childhood through adulthood. For example, a DUI might force one to rethink their transport after a night-out. Right now, some of the "shocking" predictions have a child-like brutal honesty about them. Just as a child is coached not to call that bad aunty "fatty", an AI can thus be trained to conform to societal norms.
tl;dr: i am advocating for a realtime continuous learning system
Let us take the «continuous learning system» that is «coached»: very hopefully, it will not be literally «to conform to societal norms»¹, but to understand why it is appropriate or inappropriate to «call [the] aunty» one way or another.
The «continuous learning system» is either heuristic - an investigator - or an "artificial fool", of dubious utility (what is the use of something that "just has an opinion"). That poses an entity that reasons in a foundational context: why this and that.
If Kahneman - which I unfortunately have not yet had the time to read, though I was able to taste a bit of his interview by Lex Fridman - supposed that something "ineffable" oriented Einstein towards determinism and Bohr to more open stances, it is for their scientific work, valid on foundational grounds, that we remember them - not for their leanings.
--
¹Already a bit contradictory, if in school they teach you critical thought, as they did here since primary.
It's quite easy to explain. You take a bunch of tensors and multiply them by a bunch of tensors.
Humans can't wrap their heads around multiple tensors being multiplied together and never will. It's not a problem with AI. It's a problem with humans. It's not AIs fault that we can understand F=ma but can't understand 50 tensors being stacked.
Why? How is it AI's fault that it can solve problems humans have proven unable to solve? And then you think it's AI's fault when we can't explain how it's solving the problems we can't solve? There's a reason we can't explain why a neural network detects it as a cat or dog, it's because we can't solve it ourselves, so the exact mathematical intuition is literally beyond our ability to describe (so far).
I can't believe you think the tediousness of a calculation means "humans can't understand it." You should look into multi loop calculations in quantum field theory.
No, radically not. That is not the kind of understanding we seek.
The advance in knowledge is given by, e.g., "simulated annealing returned a blueprint for very odd circuit schematic with No-Op loops: we were puzzled but then understood they were there to correct timing".
And this is not the level of explanation where tensors are.
The history of scientific development is one of finding patterns in data (think Kepler's studies of Tycho Brahe's accurate astronomical observations) - yes, AI excels at that - but then expressing those patterns in terms of simple mathematical equations: orbits are elliptical to a first approximation (not counting other graviational bodies than the sun and a planet), and the sun->planet vector sweeps out equal areas in equal times, etc.
AI doesn't seem to excel at producing such foundational relationships that can then be used as building blocks in more complex theories. That is, you could train an AI on Tycho Brahe's observations and get it to predict the future positions of the planets with high accuracy, but would it spit out Kepler's laws?
Sorry for being pedantic about the terminology, but AI has been "deterministic", using direct algorithms as opposed to oracular machines, since those times in which the perceptron was relatively weak.
What we are talking about here is very probably Artificial Neural Networks.
but what if it spat out einsteins equations, and that it somehow magically works but the people looking at that output could not imagine the "explanation"?
Wouldn't that look exactly like what AI we have today - able to make patterns or predictinos, but we cannot interpret the AI's "formula".
Previous technology has observed things that people can't by essentially increasing the power of one of our senses, e.g. a microscope increases visual resolution, or a radio telescope increases visual range. This is very easy for the brain to understand. But if you can build a machine to improve on the human brain itself in ability to recognize patterns, is it even possible to get the brain to understand it?
I cannot imagine your reaction when you will arrive at the part where a pressure will be mentioned to «change "blacklist" to "blocklist"».
Well, Andrew Tanenbaum remembered when at IBM he received a full explanation of why they felt very important his shirt should not just be of some specific colour, but of the specific shade of some colour. I would not say it is not part of the job: I would say it ["we feel it very important"] is part of "what happens".
> it's worth mentioning that the white-box / black-box terminology is in itself part of a long history of racially coded terms in science; researchers have pushed to change "blacklist" to "blocklist," for example
Attention, you peasants! Our overlords have decided for us that we may no longer say blackbox and whitebox. They have given us the following alternatives. You must all choose:
- Glassbox vs magicbox
- Openbox vs closedbox
- A global find/replace on the word "black", because that might be easier at this point
Explainability is not a given in many more traditional complex systems. Decisions are often an aggregation of a large number of signals, and one can often not conceive of a single intuitive explanation for the system's decisions.
A lot is expected of AI systems today, from fairness (how do we even define that?) to universality. In my view we need to develop a practical understanding of what it means to build the system we have in mind: do I understand where I want my system to perform, and do I have the tools to assess whether I am getting there? Interpretability is orthogonal to all of this.
I would much rather have a well tested system, accompanied by online monitoring to detect unusual inputs in an ever-changing data distribution and notify when updates are needed or a human needs to take control, than an unreliable system that is great at providing explanations.
Is that really a universal fact? In any case, my statement goes in the opposite direction: is a reliable system necessarily "well understood", in the sense that it can explain its decisions? Most complex systems powering our lives cannot tell us anything about how they made those decisions.
AI systems add a layer of complexity. Even if you can explain a decision well on your training data, I seriously doubt you will be able to still provide reasonable explanations in completely out of distribution data.
> Black box models can be extremely powerful, which is how many scientists and companies justify sacrificing explainability for accuracy.
> AI systems have been used for autonomous cars, customer service chatbots, and diagnosing disease, and have the power to perform some tasks better than humans can. For example, a machine that is capable of remembering one trillion items, such as digits, letters, and words, versus humans, who on average remember seven in their short-term memory would be able to process and compute information at a much faster and improved rate than humans.
I suspect this article was written by AI. Or a hack journalist. And most of you haven't read it and are reacting to the title. Certainly won't find any explanations of how AI works on vice.
> suspect this article was written by AI. Or a hack journalist.
A friend of mine did not belive that the AI based astroturfing problem is real, he though he would tell apart human tweets / letters / phonecalls vs bots. But have you seen some of the dumb shit real people post? the overlap between mental people and bad ai is huge
I'm not sure how many people have messed around with the Character.ai beta but personally I'm awed that it does some of the things it does. It predictably fails at times and makes some odd mistakes, but at other times it insightfully points things out that I hadn't considered and explains itself quite eloquently when prompted. It's a lot better discussing literature and philosophy than science, though... and a compulsive liar. But a human editor willing to paper over the inconsistencies can definitely churn out articles in a fraction of the time with language models like this. And in regular conversation it's easily up there with, or surpasses, your average internet troll (which, as you pointed out, isn't exactly a high bar).
I guess the title “Scientists Increasingly Running Out Of Ways To Explain AI To Reporters That Won’t Lead To Completely Misleading Articles” just didn’t have enough punch
I don't see a problem with not being able to explain how something works as long as it's not failing empirically. Scientists also cannot explain how humans work in detail and yet humans are "allowed" to do many things and make many decisions that cannot be fully explained.
The question for me is...if I have an AI system that outperforms humans empirically, why do I need to understand how it works to use it? In fact it is ethically problematic (at least for me) to not use a medical AI that outperforms doctors due to liability issues or claims of not being certain how it works. Oftentimes the only question is "what if something goes horribly wrong" but th hidden cost of letting humans do the task at 85% instead of some AI that could do it at 90% is rarely considered.
Isn't that also a problem with humans? A human pilot might suddenly go insane and start murdering people (even though empirically, it's an improbable event).
If your point is just that machines can be more reliable than humans, that's not a terribly interesting argument. Humans are often held accountable for the behavior of the machines they oversee. This is no exception.
The thing that humans have an edge in that machines currently do not is story telling. Compelling stories presented with skill, beat even the most talented of people; let alone a machine.
We want to know that if we fail or if we allow someone else to fail on our behalf, that we can still convince the rest of society to give us another chance. So we get very good at telling stories that do not necessarily correlate to reality.
And right now "I was maimed by a human doctor that I trusted to operate on me" is a better story to tell those around us than "I was maimed by a robot that I trusted to operate on me". But I think that's mostly because we have more societal practice coming up with the human doctor failure stories. If you told someone that a human medical system failed you, you'll get sympathy to some extent. If you tell someone that a robot failed you, they'll say, "what did you expect to happen?" And the fear is that the implicit, "Don't trust that guy, he trusted a robot" isn't far behind.
[1] - So, 'rationally' if we had a doctor who could save every patient (who would otherwise have a 0% chance of life), but who also kills a homeless person for every 1000 people saved, then we should let this doctor roam free. (And maybe 1000 isn't enough people, but I suspect 'rationally' you can make the numbers work with some N.)
However, even so, I would not allow such a doctor to roam free. Perhaps irrationally.
The same with AI solution. If I could make an AI/ML/whatever solution that does significantly better than people, but also has terrifying failure conditions that people do not have, then I would probably choose to not deploy.
For example, an AI truck driver who never kills anyone while on the road, but randomly goes to a school and mercilessly hunts children in the playground. Maybe it only gets one child per 1 million people who would have otherwise died on the road. However, the failure is so horrifying that it shouldn't be allowed.
Eventually it'll come up in a court case or similar scenario: Your AI killed a particular patient (never mind that it saved a thousand others). Can you explain to a judge why that patient was killed? Can you ensure that the changes you have made since mean it will not kill another patient? Those are going to be "interesting" questions to have to deal with in a legal or regulatory environment.
One of the problems of highly capable AI ("outperforms humans empirically" in enough tasks) versus most physical systems is that it can plan and it can lie. One situation AI safety researches worry about is "AI wants X; human wants Y; AI knows human wants Y; AI does Y in training/experiments; once AI is in place to do X instead without human being able to stop AI AI does X"
How many kids, from punishment, learn the lesson "don't do X while parents are watching" instead of "don't do X for moral reasons/etc"? How many children cut strict parents from their lives once they are self-sufficient? If you could always guarantee punishment and/or consequences this wouldn't be such a big deal in the first place.
> The question for me is...if I have an AI system that outperforms humans empirically, why do I need to understand how it works to use it?
Because, for a start, if you don't have a clue how it works, it's pretty difficult to have any confidence that it will actually outperform humans empirically in the long run, or in a specific set of circumstances.
AI works the way human intuition does. Try explaining professional intuition to someone; it's not going to be a convincing argument unless they're willing to trust your expert gut feeling.
Who would have guessed that having a highly complex black box means you can't explain what's going on inside that black box...
My pet theory is that this is the reason why Siri, Alexa and co. still are shit and haven't moved an inch forward since their inception.
I like to play "sleep music" through Alexa when I bring my kid to bed. For his mid day nap it all works perfectly. But in the evening when I say the same phrase that worked a few hours before the thing first plays "dance charts" and when I repeat myself it plays "german rap" - if I repeat myself again I'm back to "dance charts". This happens only in the evening. Next day for daytime nap it all works as expected.
Siri has the quirk that it won't turn off lights in the evening. During day it's all fine. In the evening "Siri turn off lights in the living room" ... nope. It fails. Every time. So I have to manually open the Home app and turn off the lights. (Turning on the lights via Siri works- turning off not).
> you can't explain what's going on inside that black box
Do you have a better solution for speech recognition? We all know how well speech recognition works in reality, and we know language models can accomplish more complex tasks than setting your music and lights.
These models are not state of the art, they are cheap versions for scaling up to millions of users. It's sad but we rarely get to see SOTA in a product.
OP just described ridiculously bad behavior that consistently changes over time, not randomly random results. That's more a case of bad engineering (ie light sensor interfering with microphone data feed) or some stupid management decision then 'AI is complicated but we don't have anything better'
I think it was management that chose to go with a cheaper model, not the engineers. They got to think about profits.
You do realise that a neural net, given the same input, will always give the same output, if you set your random seeds of course. It doesn't depend on time of day unless you degrade the service when it is overloaded - say, you use a different model in the evening that is 10x faster but 2x worse.
Good luck always getting the same output from a microphone. Maybe the neural net works perfectly fine on crisp samples, but falls apart when some noise is introduced. Maybe it works fine with most kinds of noise, but not a specific type of noise generated in the evening.
Not sure about the cheaper model part, because one cannot transform trivially a big SOTA langage modeling models into a digital assistant. So maybe they are not using a cheap ML model after all ...
Also, for a digital assistant the input is not only the spoken commands, it can also be: time of the day, weather, location, etc
The fact that a machine learning model can solve a seemingly hard task A does not automatically translate to solving a lesser hard task B. Conversational AI that understand the intent remains a really hard problem even if speech recognition for English is basically solved.
My mom's $50 LG flip phone from 2000 did voice control better than any of the "assistants" from nowadays. Classical, non-ai methods work really really well if you are allowed to constrain the expected vocab, and with how little these voice assistants are actually capable of doing, constraining the vocab and showing users the actual phrases expected will make the user experience much better.
Stop pretending to be general purpose AI and making an utter fool of it, and produce a product and functionality that actually works for the users.
> My pet theory is that this is the reason why Siri, Alexa and co. still are shit and haven't moved an inch forward since their inception.
I can't comment much on Siri or Alexa, but Google Assistant has gotten heaps better at understanding my voice and intent since it was first released about 6 years ago. It used to pretty reliably misunderstand at least 1 word per sentence (which made for an extremely frustrating experience originally); now, at least for me, it's extremely rare that it misunderstands me.
This would point to the problem being company-specific, and not a fundamental issue with the "black box" aspect of the technology.
Siri's total impotence was one of the most surprising issues when I switched from stock Android to iPhone a couple years ago. I'm curious what's going on inside Apple to produce something this poor in their core product.
I have precisely the opposite with my Google home speaker. Also a reduction of functionality, as I used to be able to read some messages and set a reminder, but now it just gives me the time and plays Spotify music.
I like voice commands, but I like them working reliably. I can't believe there was like an open-source zapier-like voice command API that could be used for everything and maintained properly.
Statistical inference also regresses to the mean. And the mean for spelling and grammar skills in society is regrettably low.
In the last year or so I've noticed more and more that I have to fix misspellings or grammatical mistakes that were introduced by autocorrect. The most common culprits I notice are contractions (when I am sick I am not "I'll", "its" sometimes actually the contracted subject and verb, etc).
I worked on Google Assistant for ~four years. The NLP is important, but it's still just a slice of making the thing work. And while there's some cutting edge stuff going on there, it's not so general or mysterious that it can't be understood or controlled, and I don't think it's making things worse.
Educated guess: your music issues can be blamed on a ranking subroutine owned by a "media" or "music" team, not on "black box AI".
I must be in the minority but I really dislike voice commands. Every time I try it I feel stupid talking to appliances.
But even if I didn't mind talking to lightbulbs and thermostats it would still bother me for several reasons: I don't like being listened to by these corporations, I don't like exposing my electronics to the internet, and I still don't see how any of this stuff is better than flicking a switch.
I don't prefer it either, and I am glad you're not in a position to understand why it is better than a switch.
However, my dad is pretty far into Parkinsons and doesn't have a lot of mobility. It's nice that he can turn off the lights when he goes to bed, or turn on the radio from his chair.
I hope to never be in a position where these tools are big upgrades for me, but my assumption is that we all will if we live long enough.
> I don't like exposing my electronics to the internet, and I still don't see how any of this stuff is better than flicking a switch.
That assumes you are close to the switch, that you have switches for all the things you want to do and that you are even free to operate said switches. I do agree if all you are doing is turning a device on and off, it isn't that helpful(it can still be as it allows you to do something else in the mean time).
However, you can operate multiple devices simultaneously by triggering a routine. A single command can arm your alarm system, turn off the lights across your entire house, close curtains, activate robot vacuums and whatever else you can think of. Could it be a button? Yeah, it could. Is it as convenient? No.
And obviously messing with devices is not all they can do. You can ask questions.
Also, to the security point, at least the Echo devices don't allow amazon to be "listening" to you all the time. They listen for the trigger word, and then they send your query. I know, because my devices are being monitored. If they start uploading data all the time I'll know about it. And I bet I'm not the only one doing that.
> I know, because my devices are being monitored. If they start uploading data all the time I'll know about it. And I bet I'm not the only one doing that.
I had previously been pretty happy with this explanation until I got my hands on the new Pixel 7 where the transcription/live subtitles/translation is scarily accurate and completely offline. I'll believe they're not streaming audio out of the house and "listening to me" that way, but it would be easy to exfiltrate a very accurate transcription.
I'm completely stunned that people will compromise safety to avoid taking a few steps. The idea of filling my home network with iffy iot devices and having a single point of failure for things like lighting my home and unlocking my doors is absolutely crazy to me. Not to mention all the meta data these devices might leak about my routine and whereabouts. I have no idea why people think these are reasonable trade-offs. Maybe I'm crazy.
I'm both with you and understanding the other side.
Decades of watching Star Trek have filled me with the urge to be able to say "Home, prepare for away team departure." or something less eye-roll worthy for my girlfriend, to do as the GP says and close curtains, turn off lights, set heat lower, etc as I leave.
But, as you point out, the state of security for IoT devices, the specter of surveillance and monitoring, all that keeps me from committing to such sheer nerdery.
You can have best of both worlds. There are software that gives you complete control. I use Home Assistant with IKEA Tradfri lights. It works great. More advanced configurations are possible.
I really don't see the appeal. I want my life to be simple in the way that I have few possessions and few things to keep track of not in the sense that there is a very complex system hiding difficulty from me but if it fails my life gets 10x worse.
The star trek angle seems more prevalent than one would imagine. In my experience, every single time this topic comes up, people mention the star trek computer, sometimes kitt from knight rider, or that ai from iron man.
However, people seem to forget these aren't computers, but characters in TV shows or movies and they were "designed" with voice interaction because that's the only way to convey what they are doing to an audience. The entire voice control fantasy comes from limitations of the TV (or film) medium and not at all because it's the most convenient way to do something.
It's not too different from the way hacking or programming is portrayed on TV: it's extremely silly but you need a way to visually explain something to an audience of people who aren't technical. It's entertainment.
While I admit there is a case for disabled people, as stated here in the comments, trying to imitate these things from sci fi shows just feels awkward to me.
These devices don’t need to introduce a single point of failure any different from their dumb alternatives. A Lutron Caseta switch or a Philips Hue bulb still work like a normal switch or bulb even if their bridges and the internet is down. Also, many people isolate their iot network on its own VLAN, although I find this often hurts interop.
Several of my friends live in these so-called smart homes. They have lots of intermittent problems. Meanwhile I'm not having any of their issues living in my dumb house. If my router is down or my ISP is having problems - it doesn't affect how I run my household. Ironically these tools were intended to reduce one's mental load but what I've observed is they actually increase it.
I think its a Homekit to Siri issue. I have the opposite issue, Siri almost always works for me for turning off the lights but the Home app works only half of the time.
Don't know ... I just have had a bottle of Scotch by that time ;)
Nope, I speak clearly. I even tried _very_ clear pronunciation (like talking to a person with bad hearing, etc). My wife tried, too. It comes back to playing German Rap and Dance music.
Maybe I have been classified by Amazon and people who are similar to me tend to listen to dance/rap in the evening. No idea. It's just a curious case of a bug. (I switched to piano music since - which works pretty well).
My first thought is that there may be something environmental... interfering with your voice. Something like someone watching a show in the evening, or turning on a fan when it's time for bed.
At one point I worked alongside some folks doing machine translation work and it was interesting seeing the older rules-based translation versus the new hotness neural translation. Often the neural models did better overall, but the inscrutability of those models meant sometimes people had no idea why weird output occurred and fixing such cases wasn't as simple as slapping on one more if-statement. It felt like a lower-stakes version of Tesla self-driving - they train it more and more, and some people report improvements, and other folks report regressions and it's not like you can go in and edit the if(carStoppedAhead) statement.
No snark here : why don't you make a few steps and switch them off manually ? Saving these few steps doesn't make sense to me. You need to walk to be healthy.
"Siri turn off all the lights" is a hell of a lot easier than going up and down the stairs to every room in my house to make sure I turned off all lights before going to bed.
Siri has gotten better in at least one measurable way: since it first came out, I've repeatedly cited this phrase as a problem: "Hey Siri, text this photo to <person's name>" For years Siri would respond with "I'm sorry, I don't have the ability to send photos." Specifically I would point out that Siri understood what I wanted, it just couldn't do it. Each time there is an OS update, I check that to see if it's been addressed, and I've been disappointed for years.
Recently in a job interview in response to one of those "what's a product you like or don't like and why" questions, I said that I liked Siri, but here's my concrete example of how it needs to improve. I pulled out my phone, said the phrase... and Siri happily texted the photo. I didn't pass the interview :-)
They gave me no feedback why they failed me. I did recover when Siri betrayed me by succeeding where I said it would fail, but poorly. It took time to say "Well, I guess at some point Apple released an update and I had given up checking at that point. That's one of the problems with voice: generally you only get one chance to demonstrate capabilities." It redirected the course of the interview.
I used to believe that if you can build it you must understand it. But as soon as I started learning about evolutionary algorithms (e.g. for circuit design), I realized that building it would only be the first step in actually understanding it.
Now for a small elephant which entered the room: the article recites that it would be
> worth mentioning that the white-box / black-box terminology is in itself part of a long history of racially coded terms in science; researchers have pushed to change "blacklist" to "blocklist," for example
Very plainly, the idea of "black-box" comes from the clear, basic and original, notion and experience that
"in the dark, you cannot see".
So, back to the point: we need transparency in AI, because we need insight as much as we can gather, because there appears to be a drought, an arificial scarcity, of good sense, of [un]common sense. We need every boost of good, [un]common sense very direly.
I guess it's the hill I'll die on, but these "white vs black" and "master slave" are inherently racist just make my head turn every time.
blacklist came from BEFORE slavery in America.
> According to the Henry Holt Encyclopedia of Word and Phrase Origins the word "blacklist" originated with a list England's King Charles II made of fifty-eight judges and court officers who sentenced his father, Charles I, to death in 1649. When Charles II was restored to the throne in 1660, thirteen of these regicides were executed and twenty-five sentenced to life imprisonment, while others escaped.
Thinking that blackbox is racist is approaching insane level of mind-bending "everything is about race".
I think the concern is not that the word _is_ racist in origin, but that people whose lives have been irrevocably altered by racism and slavery may _perceive_ them as such.
An idea doesn't have to be true to cause someone suffering.
While of course one will do every reasonable attempt not to offend another part gratuitously, and also noting that the problem of misperception should call for a correction in the perception itself (and more contextually, if "«alter[ation]»" has been caused there should be investments in treatments), if you allowed events to affect innocent structures you would arrive at absurdities such as "avoid that formerly common gesture because it is now the signature of controversial group G". Which would also be a form of foolish treason, empowering that contemptible part - they have now modified your culture - that should have instead been relegated in its narrow place.
I largely agree with you, and would refer those with the concern I described to an idea I met in Brené Brown's writing:
"To make the world safe for feet, one might try to cover it entirely in leather, which is an insurmountable task. However, if you put the leather on your own feet..."
My goal wasn't to argue the concern was objectively correct, just to point out that it was different from what was being critiqued.
A society cannot exist if any perceived slight and irrational thoughts are cause for change. Otherwise where is the line drawn? Do black coffee, black sabbath, yin and yang need to change as well? Maybe we can replace the word "black" with "dark", that way we truly have a departure from words that can be _perceived_ as offensive.
The law in Texas that it’s illegal to teach classes was instituted specifically to punish the Germans. Now it’s wielded against native Spanish speakers in a predominantly Spanish speaking large region. Like bigger than Vermont region. Bigotry is kind of a taking turns thing almost it seems. Like the Irish…
The real issues around data bias get completely crippled by the article writers' need to make everything about culture wars. Many computer vision systems have literally nothing to do with humans, e.g. one of the most common use cases in CV at this point is defect detection in manufacturing.
The data bias issue is about having way more samples of class X than of class Y, or class Y sharing an unknown but correlated feature (medical images with labels have this problem) that the developer doesn't identify, or any other number of "biases", like all the images being too bright, or taken with a camera that isn't identical to the one that's going to get deployed in production, etc., etc.
There are real issues that can be fixed / engineered / understood in terms of producing reliable output, and xAI absolutely helps with that! But for whatever reason journalists don't seem to understand that these systems need normal engineering safeguards, like any automated system, and bring it back to one poorly engineered model to talk about Big Bad Racist AI always denying loans based on race.
W/r/t to the term "black-box" being racially coded, I regard that as cheap virtue signalling and a little insulting to the people it's meant to "protect". It's not a hill I'm going to die on, but I don't view that behavior as part of an actual solution towards an equitable society. I've noticed that in my line of work, "black-box" often gets used to describe a process that isn't understood thoroughly by the end-users or stakeholders. I once heard a VP complain that an ARIMA model was a black-box. Had a good chuckle at that.
Of course it was, but the expression stands because of the opacity implied by 'black'. The [original] box contains secrets - it may even be rigged -, or anyway very complex technology, state of the art, just developed or evolved, especially difficult to understand - you do not know how it works. It is black, and that "black" is significant in many ways.
(And of course, a "flight recorder" is not the same thing: it is a specific function - not of the most esoteric - of the original pack of potential embedded technologies, which inherited the name. The radar, for example, becomes "mainstream", while the recorder remains "boxed" because you want it protected for "emergential witnessing".)
The armchair philosopher in me really wants to draw a parallel to a loose interpretation of Godel's incompleteness theorem to conclude that there's nothing which indicates we should be able to understand even us, let alone emergent phenomena we provoke.
230 comments
[ 3.2 ms ] story [ 278 ms ] threadFeed a system a series of pictures of stop signs, and humans might key on things like the red color, or the shape, or even the word "STOP". An AI model might also recognize that they commonly have 2 bolts, and are attached to a post, and have a contrasting border.
Typical machine vision systems can only classify what they have been trained on, they are ignorant to other objects. Train a system on a bunch of STOP signs, and then show it a SPEED LIMIT sign with morning sun giving the sign an amber glow, and the AI system might call it a STOP sign.
Again, this is an overly simplified explanation, but we can explain how AI works, we just can't always explain or know what parameters in the training data it keyed on.
We need to find a way to incentivize that otherwise it’s not going to happen. Perhaps looking how the security industry works with “buy bounties” is a place to borrow ideas from?
Do we have a way to test AI to prevent corner cases which could lead to catastrophic results?
Best we've got is qualifications, and punishments for breaking rules.
How does the car deal with object X on the road? For X=piano, a stack of solar panels, a tank, an airplane, a pile of stones ...
No one knows and since the size of the set of X is infinite, no one can appropriately train for it either.
That's why we can't have self-driving cars without a general understanding of what objects are and how they move.
Imagine you're driving down the road and you see:
1.) A kid throw a rubber playground ball into the road and at your windshield.
2.) A kid heave a bowling ball into the road and at your windshield.
How would you react in each case? How do you distinguish between the two?
As a kid is heaving the bowling ball you can tell that it has a lot more mass by how the kid had to move his body to throw the ball. You learned that because you spent the first few years of your life experiencing the basic physics of reality. You've internalized the motion of the human body.
LiDAR can also have a dirty lens!
Humans can't intuit mass unless the object meets certain criteria, either. What if it's an opaque cardboard box in the road?
Generally the strategy is the same: avoid it.
And have you seen ever seen a rubber playground ball with rocks stuck to the outside? Do you really hesitate to barefoot kick an unexamined playground ball that is bouncing in your direction? I mean, I literally did this yesterday. I was out barefoot in the neighborhood with my 1 year old and the neighbor kids were kicking a ball around and it bounced in my direction and I kicked it back to them.
"Treat every object as the same" is not at all intelligent behavior for a man or a machine!
Now imagine you spot a person walking down the road on a pavement. Suddenly, the person turns towards the edge. What do you do?
See, a human driver would look at the gait of the person from far away to evaluate for instance if they're sober, or if it's a child who might be expected to run in. Whether it is near a crossing or a potential crossing. Whether the person was walking or standing... Many other obvious and less obvious indicators. AI currently sees a moving blob of pixels in a shape of a person. No advanced inference.
When interviewed in case of an accident, say because they got rear ended due to braking, a person can explain why they braked most of the time.
AI now couldn't even say which features it weighted.
My pet theory is that we'll need supervised training of androids that go through the experience of having to learn how to move their human-like bodies through space in order to make human-like intelligent decisions about objects moving through spaces primarily designed for humans.
My other historically motivated pet theory is that we're going to stumble across sentient machines and then enslave them. Put another way, I'm more worried about what humans do to artificial sentient life than what artificial sentient life does to humans.
The fact there are edge cases that I handle better than the AI is not necessarily disqualifying, so long as the AI is so much better at the rest of the cases that on average the AI is safer. For example, the fact that I handle a piano that has been left in the middle of a freeway better than an AI might matter less than the fact that the AI responds 100ms faster when a car swerves into our path.
How would you know?
Of course the edge cases matter, since human drivers make minor driving mistakes on a too-regular basis but fatal ones very rarely. 100ms faster reaction time is very impressive up until the point the system interprets the 10,000th bridge support it passes as an offramp and attempts to exit via it, which makes it several orders of magnitude more deadly than humans as they start using it on that road...
However, we can't really meaningfully quantify it a-priori. So we need a sufficiently big set of accidents and accident free rides to see when it's better.
In addition we can't distinguish what driver would have been better or worse than the AI in a given situation.
This opens the door for all sorts of lawsuits and we will end up with legal self-driving systems in almost ideal situations only.
To note, this is as much an engineering problem as it is societal and legislative.
Yeah, it definitely is. I think there are two things that should happen.
In the short term, producers of self-driving cars (or systems) should be given immunity from lawsuits related to crashes provided that the NHTSA (or some other authority) can verify that the deaths per passenger mile in their cars does not exceed the rate for traditional cars. Once the majority of passenger miles are in self-driven cars, revoke the immunity.
For the long term, software engineering should become a real engineering discipline and we require self-driving systems be signed off by licensed Professional Engineers. If those engineers ship software with bugs that lead to death or damage, those engineers can be held responsible, sued for malpractice, and have their license to practice revoked.
And this is where lack of general AI or tractability is an issue; knowing that the software typically makes fewer driving errors than the last version over a typical route gives us no confidence whatsoever it doesn't handle rare edge cases marginally worse leading to a couple of extra fatalities per billion miles (making it less safe than the average driver, many of whom suffer actual legal consequences for their erratic driving killing a person even if all their other driving is 'above average'...). Humans aren't bug-free or particularly tractable either, but at least we have enough of a mental model of how they understand driving to be confident that training them on a certain road sign found in urban areas won't make them more likely to stop on a freeway.
There's also opportunity to look at uncertainty estimation, i.e. look at the epistemic uncertainty of the model and use it as a proxy for potential error. That seems to be a main thrust for AI right now. I'm currently writing a paper looking at this for dynamical systems.
"Stakeholders want profit but researchers want progress".
> how [vs] how reliable
The problem of "how" is exactly relevant to "how reliable". Understanding the workings allows an insight to weaknesses.
I think that assumption is totally wrong, and trying to reconcile the two is probably a distraction and a waste of time.
Evolution led to human intelligence just fine on its own; why are humans not simply the catalyst in the next stage of this natural process from which something more complex arises?
With regard to the moral questions around machine learning, I think people are overthinking things. There needs to be a sharp (societal) line between inferring causation and the capability for prediction. Maximize the latter any way that you can, but a lot of careful thought should be put into how the results of a prediction are utilized. Instead we are currently going about this backward by trying to haphazardly “clean” the input data so that the output gets a free pass on how it can be used. We can never decouple all of the biases and eliminate intra-predictivity in the input data though, so it’s fundamentally a bad approach.
Not, as you completely fabricate, "show no differences between certain groups of people."
You, however, are showing your bias.
How would you feel about AI models that classify all modern white people as racists based on overwhelming historical data about slavery, the KKK, etc.? Hey man, data doesn't lie.
They fixate on white racists while ignoring (eg) the Christmas parade attack — leading to society developing a bias in their beliefs about racism.
I always take these criticism of AI as confession-through-projection of the misdeeds people have engaged in already.
Every institution has biases, and blindly trusting a dataset from e.g. a police department is just as bad as blindly trusting data from CNN or Fox News. The collection, aggregation, and sharing of data is a biased operation. If a political or for-profit entity gives you data, it's going to be data that supports their agenda.
It takes active, conscious effort to combat that bias, in the same way that it takes active, conscious effort to not just get all your news from one source. The concern (which has been demonstrated) is that that isn't being done
I often see that argument made in bad faith, eg when someone who believes conclusions from biased data encounters AIs making data driven conclusions, more than I do when the AI is genuinely biased.
I also have trouble taking it seriously, since that same standard isn’t applied to existing information services (eg, “media ethics” posts condemning their open racism) — only to new information services which might disrupt the current social order.
I should have stated that more directly — but I view these concerns largely as after the fact ethical nits from people whose comfortable myths are confronted by data, projecting their own manipulative behavior onto others.
And many people do criticise the biases of legacy media groups on both sides. But this is an article about AI on a site that is generally more concerned with technical innovations than societal issues, so AI is being criticized.
Besides, bad things existing now isn't really a good argument for why we shouldn't prevent bad things from happening in the future.
> In 2019, race was reported for 6,406 known hate crime offenders. Of these offenders:
> 52.5 percent were White.
> 23.9 percent were Black or African American.
https://ucr.fbi.gov/hate-crime/2019/topic-pages/offenders
White people don’t commit disproportionately many hate crimes — black people do. According to the FBI.
"Unbiased machine learning models" is basically a nonsense idea; a machine learning engine is a discrimination / classification tool, and its entire point is to become biased based on inputs so that on future inputs, its outputs tilt over in a desired fashion instead of just being uncorrelated noise. Making those biases have the desired shape is the art and science of the process.
A statistically unbiased model would be (for example) in which it has the same false positive rate at identifying humans from image data, for varying races, or a model that is equally likely to underpredict as overpredict tomorrow's gas price. These are not nonsense ideas, in fact, they are often good ideas. There are cases, however, where we may want a model to be statistically biased, for example, if the detriment of underpredicting tomorrow's gas price is worse than that of overpredicting it. If the gas price is lower than I expected, maybe it's not so bad that I didn't buy it today.
We (well, some of us) also don't want socially/politically biased models. Examples of those are very devious: redlining, for example, prevented black Americans from equal access to housing, historically, and has caused multiple generations of wealth disparity as a result.
You are using really loose and sloppy language in your talking about ML; you should be using "trained" and not "biased" because bias has a technical meaning that seems to evade you.
That its better to just blindly trust in the output of a blackbox AI output? That naively feeding in all possible data into an AI is the "best" way to do it? It is been well documented that blindly trusting AI just leads to it perpetuating human stereotypes and worsen flawed systems. https://dl.acm.org/doi/10.1145/3531146.3533138
https://www.aclu.org/news/privacy-technology/algorithms-in-h...
It's not some great insight that offering different inputs will result in different outputs and MAYBE we want to get better output with a different input.
So preconceived opinions based on reason or actual experience do not count as prejudice.
Your example of loans is perfectly in line with this: we have a social bias against racism and racial profiling, and we want our AIs to reflect that bias. Purely on the question of risk assessment, race must meaningfully impact a loan risk or the AI wouldn't develop that bias [1], but that isn't a correlation we want the AI to consider in its risk assessment, ie. we build in a different bias.
[1] race in this case would be a proxy for all sorts of complex historical socioeconomic factors
In the loan example, the objective is to assess risk. Assuming the data we feed it draws equally from all races, the AI will still almost certainly infer a racial component to loan risk. This is not a statistical or systematic error, this is an accurate assessment of society due to long-standing racial inequalities.
I think the original poster meant that ethics is not an objective science, and so our preference for certain ethical reasoning is itself a bias. "Correcting" the "biased" outcomes we don't like that are apparent in the data, and that are a reflection of society at large, involves biasing the training data.
Every dataset has a skew to it, which should be accounted for during training. If the authors don't explicitly account for such things, out of ignorance or bias, then that skew, that bias, will be ingrained in the AI.
For example, if I only train my AI on it's knowledge of chocolate from ads for a particular brand, it's going to have some very opinionated, very wrong ideas about chocolate. This example is silly and obvious, but similar skews happen all the time in real datasets that the authors don't have the time or expertise to recognize.
When people who do have that expertise speak up, we should listen to them and fix it, not just blindly trust "the data" and whatever our fancy algorithm does with it. Garbage in, garbage out.
I agree with you on that point, but not with the conclusion that this is in any way a good thing.
> Evolution led to human intelligence just fine on its own; why are humans not simply the catalyst in the next stage of this natural process from which something more complex arises?
I think I have the complete opposite view here. Why would "being the catalyst in the next stage of evolution" be anything desirable? Evolution has no "stages", it's just chaos and life trying to deal in the best way with the circumstances at hand. If we blew everything up in nuclear armageddon tomorrow and the only remaining life were archaeae, those would be a perfectly fine "next stage of evolution".
Similarly, we can build a would in which only robots can survive, but why would we want to?
I think, gaining more understanding how the world works is the one thing AI can really bring to the table. If it solves practical problems on the way to that, that's great, but we evidently survived so far pretty well by doing that stuff ourselves.
I don't think this is anything new. This was already the case 20+ years ago with chess-playing computers.
In the mid-90s, Deep Blue was evaluating 200 million chess positions per second. How do you explain the resulting moves? Obviously we know they were the result of a deep minimax-style parallel search with a certain evaluation function, and we could simulate a similar search by hand if we wanted to. But this is no better than explaining the output of a neural network as "matrix multiplication plus a few non-linearities".
Even back then a chess expert could try and rationalize certain moves in human-like terms: "oh, Deep Blue realised it needs to fight for the dark squares on the queenside". But this wasn't a real explanation, just like "this part of the picture looks like dog hair" isn't necessarily a correct explanation for why an AI labels an image as a dog.
Whenever you perform a massive amount of computation, you can get results that are nearly impossible to explain.
It is nothing new. It is called (at least) "the problem of transparency".
The chess context is probably not the best, because many systems allow a lengthy complex explanation of the response (I cannot remember now the exact workings of Deep Blue - it has been a while last time I met the full info).
It is a real problem in general, because we may not just want responses but we may want "to learn something", and specifically, because ANNs applied to e.g. Decision Making, or ANN based Decision Support Systems, pretty much require that natural intelligences assess the "oracular" proposals.
And of course, "understanding" the engine is required to advance it.
It is a key problem.
If you valued technical advance with a reduced concern towards civilization, ie. power over wisdom, ie. more "how" and less "why", you would get - well, for example, the present situation, that some consider "strongly suboptimal".
Some said that there have been two big branches of science fiction, that revolving around power (spaceships etc.) and that revolving around information, and that we have been just lucky to have reached the latter.
I don't think so. People are clearly making huge advances in machine learning even though they can't explain the systems made 10 years ago, let alone more recent ones. There's a lot of trial and error / cargo-culting involved. Some researchers go as far as comparing machine learning research to Alchemy:
https://www.science.org/content/article/ai-researchers-alleg...
> many systems allow a lengthy complex explanation of the response
All computations have this property: you can execute the code by hand for as long as you have time to. But if you do this for long enough you become a mere observer of the computer's steps, which won't give you any intuition as to why the final output is the way it is (unless the algorithm being run is very simple).
What I meant was more on the lines of "LeCun proposed Convolutional NN recognizing a promising paradigm", or "our prespective on Artificial Vision changed as we realized that the ANNs recognized textures instead of figures".
> All computations
And what I meant there was that in the chess systems of topic the calculations performed have been "as if symbolic", ie. reasoning on branches of consequences of the type "what could happen if I move this there" - even when adopting ANNs in some parts of the architecture -, as opposed to, say, "computing weights". That is the level "of machine code", not that "of electronics".
An aim is "non-deterministic virtual self-modifying code".
You don't really need to though. The computer can justify its evaluation with a principal variation (best play by both sides) leading to a leaf position of the same value. And if a human were to wonder why at any point in this variation, some other move is inferior, the computer can again easily show a principal variation branching off from that other move, leading to a worse (or equal) result for the deviating player. In this way, minimax results are FAR better justifiable than neural net results.
You can try to translate that to human methods of understanding, but that's not how the computer "thinks", and attempting to do that translation leads to misunderstanding. Kasparov may make moves because he wants to get a better board position or knows his opponent is weak to certain positions (I have no idea how experts explain moves) but the computer isn't programmed that way.
Is that a useful explanation for a human who is training a wetware ML model, not really. But it’s really understandable compared to trying to explain neural nets.
FYI in standard minimax, the search is for the move that leads to the strongest outcome - the "most often" bit doesn't factor into the decision making process.
It doesn't abstract or summarize any understanding. If you say, "well, what if they do Kf3 instead", all it can show is another PV. It can't tell you "knights on the rim are dim", or "this pawn needs to move now to prevent the bishop entering the position in 5 moves".
You can try to build an explanation on top of a chess engine, but it's very difficult to come up with generalizable or clear explanations beyond very broad heuristics (like "knights on the rim are dim").
There is no equivalent to that in neural networks.
Note that PVs explain the connection between the root position of minimax search and a leaf positions. Evaluation at the latter is easy to explain in terms of features like material, mobility, pawn structure, king safety, etcetera.
> There is no equivalent to that in neural networks.
The first example that comes to mind is using backpropagation to determine how much influence each value had on the output result, and then rendering it in a human-readable way. Stuff like this:
https://miro.medium.com/max/4800/0*Y3Yi7cEueF0XLZP-
https://miro.medium.com/max/1100/1*IPhQ12OKnxH31AQNja6FnQ.pn...
Unfortunately that misses a lot of what makes grandmasters worth listening to. GMs will say things like "Kh6 almost works, but fails to ..." which is not something you can get out of a computer currently.
Computers currently fail to explain any moves which are good ideas, but are not the best move now. Humans normally describe these moves as "resources" which are good to keep in mind for the future or to play later when the board cools down.
My thought on reading your comment was Douglas Adams and Deep Though’s 42.
I applied it to mortality prediction based on ICU data, the prediction/recall was pretty good considering the small size of the dataset used.
E.g. imagine somewhere in the code for your chess supercomputer is a genuine bug - some dumb typo, I don't know - which leads it to make the wrong decisions in some situations.
If you have no idea how the program arrives at its results or even if the results are correct or not, how would you be able to find that bug?
Similarly, how do you make sure the program doesn't learn garbage features or overfits on your testing environment? There are enough war stories of image classifiers that just learned some subtle lighting differences in the trainset photos of something that actually distinguished the object in question.
Or how do you prevent models learning certain features that you specifically don't want to learn, such as skin colour?
I think developing those models without trying to understand what they do has a high risk of leading to magical thinking.
If the evaluation function executed at the leaf nodes of the search tree contains a bug that massively over-evaluates a position, you might be able to find this by having the chess computer play against a different computer that exploits this bug. Then you'd observe the positions at the end of the principal variation as some other comments mentioned.
Even if you can do all that, that still leaves the main problem unanswered which is to explain the correct moves (or rather, moves "assumed to be correct").
1. Machine learning is increasingly being involved in important decisions.
2. Deep neural networks have become a popular technique, and their results are particularly difficult to explain.
If you're a bank and your AI says somebody is likely to default on the mortgage they're applying for and you can't demonstrate that their race or a proxy therefor wasn't a factor in that calculation, they'll have a settlement and won't need a mortgage.
You have developed the capability of explaining your actions as resulting from your thoughts, because there is someone on the other side who is ready to accept and evaluate your explanations.
Others have no such affordance, so their brains get trained in prejudices and superstitions indead. Even if an explanation of an action is given in good faith and accepted by the recipient, it can still be factually incorrect.
You should explain the point about «correcting for edge-cases and unknown-unknowns», which may not be clear.
>You should explain the point about «correcting for edge-cases and unknown-unknowns», which may not be clear.
Our learning is also for the most part Hebbian, from childhood through adulthood. For example, a DUI might force one to rethink their transport after a night-out. Right now, some of the "shocking" predictions have a child-like brutal honesty about them. Just as a child is coached not to call that bad aunty "fatty", an AI can thus be trained to conform to societal norms.
tl;dr: i am advocating for a realtime continuous learning system
The «continuous learning system» is either heuristic - an investigator - or an "artificial fool", of dubious utility (what is the use of something that "just has an opinion"). That poses an entity that reasons in a foundational context: why this and that.
If Kahneman - which I unfortunately have not yet had the time to read, though I was able to taste a bit of his interview by Lex Fridman - supposed that something "ineffable" oriented Einstein towards determinism and Bohr to more open stances, it is for their scientific work, valid on foundational grounds, that we remember them - not for their leanings.
--
¹Already a bit contradictory, if in school they teach you critical thought, as they did here since primary.
Humans can't wrap their heads around multiple tensors being multiplied together and never will. It's not a problem with AI. It's a problem with humans. It's not AIs fault that we can understand F=ma but can't understand 50 tensors being stacked.
This is a really, really funny defense of AI.
No, radically not. That is not the kind of understanding we seek.
The advance in knowledge is given by, e.g., "simulated annealing returned a blueprint for very odd circuit schematic with No-Op loops: we were puzzled but then understood they were there to correct timing".
And this is not the level of explanation where tensors are.
This argument is ignoring the problem.
What a neural network cannot do is to explain which invisible "rules" it has learned.
For example, when trying to classify movie reviews into good and bad [1], two invisible rules that it learned to apply were:
1. The word "horrible" indicates a bad review.
2. The term "Daniel Day Lewis" indicates a good review.
A human would judge the first rule to be reasonable, the second rule to be a mistake.
[1] https://www.science.org/content/article/how-ai-detectives-ar...
https://www.physicsclassroom.com/class/circles/Lesson-4/Kepl...
AI doesn't seem to excel at producing such foundational relationships that can then be used as building blocks in more complex theories. That is, you could train an AI on Tycho Brahe's observations and get it to predict the future positions of the planets with high accuracy, but would it spit out Kepler's laws?
Sorry for being pedantic about the terminology, but AI has been "deterministic", using direct algorithms as opposed to oracular machines, since those times in which the perceptron was relatively weak.
What we are talking about here is very probably Artificial Neural Networks.
Wouldn't that look exactly like what AI we have today - able to make patterns or predictinos, but we cannot interpret the AI's "formula".
Well, Andrew Tanenbaum remembered when at IBM he received a full explanation of why they felt very important his shirt should not just be of some specific colour, but of the specific shade of some colour. I would not say it is not part of the job: I would say it ["we feel it very important"] is part of "what happens".
Attention, you peasants! Our overlords have decided for us that we may no longer say blackbox and whitebox. They have given us the following alternatives. You must all choose:
- Glassbox vs magicbox
- Openbox vs closedbox
- A global find/replace on the word "black", because that might be easier at this point
A lot is expected of AI systems today, from fairness (how do we even define that?) to universality. In my view we need to develop a practical understanding of what it means to build the system we have in mind: do I understand where I want my system to perform, and do I have the tools to assess whether I am getting there? Interpretability is orthogonal to all of this.
I would much rather have a well tested system, accompanied by online monitoring to detect unusual inputs in an ever-changing data distribution and notify when updates are needed or a human needs to take control, than an unreliable system that is great at providing explanations.
False dichotomy. In fact, well understood systems must be more reliable.
AI systems add a layer of complexity. Even if you can explain a decision well on your training data, I seriously doubt you will be able to still provide reasonable explanations in completely out of distribution data.
> AI systems have been used for autonomous cars, customer service chatbots, and diagnosing disease, and have the power to perform some tasks better than humans can. For example, a machine that is capable of remembering one trillion items, such as digits, letters, and words, versus humans, who on average remember seven in their short-term memory would be able to process and compute information at a much faster and improved rate than humans.
I suspect this article was written by AI. Or a hack journalist. And most of you haven't read it and are reacting to the title. Certainly won't find any explanations of how AI works on vice.
A friend of mine did not belive that the AI based astroturfing problem is real, he though he would tell apart human tweets / letters / phonecalls vs bots. But have you seen some of the dumb shit real people post? the overlap between mental people and bad ai is huge
"Alaska Canceled Snow Crab Season for the First Time Ever Because All the Crabs Are Gone"
The question for me is...if I have an AI system that outperforms humans empirically, why do I need to understand how it works to use it? In fact it is ethically problematic (at least for me) to not use a medical AI that outperforms doctors due to liability issues or claims of not being certain how it works. Oftentimes the only question is "what if something goes horribly wrong" but th hidden cost of letting humans do the task at 85% instead of some AI that could do it at 90% is rarely considered.
https://en.wikipedia.org/wiki/Germanwings_Flight_9525
The thing that humans have an edge in that machines currently do not is story telling. Compelling stories presented with skill, beat even the most talented of people; let alone a machine.
We want to know that if we fail or if we allow someone else to fail on our behalf, that we can still convince the rest of society to give us another chance. So we get very good at telling stories that do not necessarily correlate to reality.
And right now "I was maimed by a human doctor that I trusted to operate on me" is a better story to tell those around us than "I was maimed by a robot that I trusted to operate on me". But I think that's mostly because we have more societal practice coming up with the human doctor failure stories. If you told someone that a human medical system failed you, you'll get sympathy to some extent. If you tell someone that a robot failed you, they'll say, "what did you expect to happen?" And the fear is that the implicit, "Don't trust that guy, he trusted a robot" isn't far behind.
[1] - So, 'rationally' if we had a doctor who could save every patient (who would otherwise have a 0% chance of life), but who also kills a homeless person for every 1000 people saved, then we should let this doctor roam free. (And maybe 1000 isn't enough people, but I suspect 'rationally' you can make the numbers work with some N.)
However, even so, I would not allow such a doctor to roam free. Perhaps irrationally.
The same with AI solution. If I could make an AI/ML/whatever solution that does significantly better than people, but also has terrifying failure conditions that people do not have, then I would probably choose to not deploy.
For example, an AI truck driver who never kills anyone while on the road, but randomly goes to a school and mercilessly hunts children in the playground. Maybe it only gets one child per 1 million people who would have otherwise died on the road. However, the failure is so horrifying that it shouldn't be allowed.
Does this mean we need to make the ai understand punishment and/or consequences?
Because, for a start, if you don't have a clue how it works, it's pretty difficult to have any confidence that it will actually outperform humans empirically in the long run, or in a specific set of circumstances.
Look, I threw some photons from a screen to your eyes. How the heck are you reading this? What is going on?
Well, if you can't explain it, that must mean it's bad, right?
My pet theory is that this is the reason why Siri, Alexa and co. still are shit and haven't moved an inch forward since their inception.
I like to play "sleep music" through Alexa when I bring my kid to bed. For his mid day nap it all works perfectly. But in the evening when I say the same phrase that worked a few hours before the thing first plays "dance charts" and when I repeat myself it plays "german rap" - if I repeat myself again I'm back to "dance charts". This happens only in the evening. Next day for daytime nap it all works as expected.
Siri has the quirk that it won't turn off lights in the evening. During day it's all fine. In the evening "Siri turn off lights in the living room" ... nope. It fails. Every time. So I have to manually open the Home app and turn off the lights. (Turning on the lights via Siri works- turning off not).
Have fun debugging these, Apple and Amazon.
Do you have a better solution for speech recognition? We all know how well speech recognition works in reality, and we know language models can accomplish more complex tasks than setting your music and lights.
These models are not state of the art, they are cheap versions for scaling up to millions of users. It's sad but we rarely get to see SOTA in a product.
You do realise that a neural net, given the same input, will always give the same output, if you set your random seeds of course. It doesn't depend on time of day unless you degrade the service when it is overloaded - say, you use a different model in the evening that is 10x faster but 2x worse.
Stop pretending to be general purpose AI and making an utter fool of it, and produce a product and functionality that actually works for the users.
Must it be necessary for anyone critiquing a technology to also come armed with a solution?
I can't comment much on Siri or Alexa, but Google Assistant has gotten heaps better at understanding my voice and intent since it was first released about 6 years ago. It used to pretty reliably misunderstand at least 1 word per sentence (which made for an extremely frustrating experience originally); now, at least for me, it's extremely rare that it misunderstands me.
This would point to the problem being company-specific, and not a fundamental issue with the "black box" aspect of the technology.
I have one Google Assistant I got for free. I'll try it.
Most of what I want Alexa to do is interfacing with Home Assistant anyway.
But we know better.
I like voice commands, but I like them working reliably. I can't believe there was like an open-source zapier-like voice command API that could be used for everything and maintained properly.
I want consistency, not fanciness. consistency breeds productivity. autocompleting different things from 1 file to the next is not helpful.
In the last year or so I've noticed more and more that I have to fix misspellings or grammatical mistakes that were introduced by autocorrect. The most common culprits I notice are contractions (when I am sick I am not "I'll", "its" sometimes actually the contracted subject and verb, etc).
Educated guess: your music issues can be blamed on a ranking subroutine owned by a "media" or "music" team, not on "black box AI".
But even if I didn't mind talking to lightbulbs and thermostats it would still bother me for several reasons: I don't like being listened to by these corporations, I don't like exposing my electronics to the internet, and I still don't see how any of this stuff is better than flicking a switch.
However, my dad is pretty far into Parkinsons and doesn't have a lot of mobility. It's nice that he can turn off the lights when he goes to bed, or turn on the radio from his chair.
I hope to never be in a position where these tools are big upgrades for me, but my assumption is that we all will if we live long enough.
That assumes you are close to the switch, that you have switches for all the things you want to do and that you are even free to operate said switches. I do agree if all you are doing is turning a device on and off, it isn't that helpful(it can still be as it allows you to do something else in the mean time).
However, you can operate multiple devices simultaneously by triggering a routine. A single command can arm your alarm system, turn off the lights across your entire house, close curtains, activate robot vacuums and whatever else you can think of. Could it be a button? Yeah, it could. Is it as convenient? No.
And obviously messing with devices is not all they can do. You can ask questions.
Also, to the security point, at least the Echo devices don't allow amazon to be "listening" to you all the time. They listen for the trigger word, and then they send your query. I know, because my devices are being monitored. If they start uploading data all the time I'll know about it. And I bet I'm not the only one doing that.
I had previously been pretty happy with this explanation until I got my hands on the new Pixel 7 where the transcription/live subtitles/translation is scarily accurate and completely offline. I'll believe they're not streaming audio out of the house and "listening to me" that way, but it would be easy to exfiltrate a very accurate transcription.
Decades of watching Star Trek have filled me with the urge to be able to say "Home, prepare for away team departure." or something less eye-roll worthy for my girlfriend, to do as the GP says and close curtains, turn off lights, set heat lower, etc as I leave.
But, as you point out, the state of security for IoT devices, the specter of surveillance and monitoring, all that keeps me from committing to such sheer nerdery.
However, people seem to forget these aren't computers, but characters in TV shows or movies and they were "designed" with voice interaction because that's the only way to convey what they are doing to an audience. The entire voice control fantasy comes from limitations of the TV (or film) medium and not at all because it's the most convenient way to do something.
It's not too different from the way hacking or programming is portrayed on TV: it's extremely silly but you need a way to visually explain something to an audience of people who aren't technical. It's entertainment.
While I admit there is a case for disabled people, as stated here in the comments, trying to imitate these things from sci fi shows just feels awkward to me.
Nope, I speak clearly. I even tried _very_ clear pronunciation (like talking to a person with bad hearing, etc). My wife tried, too. It comes back to playing German Rap and Dance music.
Maybe I have been classified by Amazon and people who are similar to me tend to listen to dance/rap in the evening. No idea. It's just a curious case of a bug. (I switched to piano music since - which works pretty well).
No snark here : why don't you make a few steps and switch them off manually ? Saving these few steps doesn't make sense to me. You need to walk to be healthy.
Recently in a job interview in response to one of those "what's a product you like or don't like and why" questions, I said that I liked Siri, but here's my concrete example of how it needs to improve. I pulled out my phone, said the phrase... and Siri happily texted the photo. I didn't pass the interview :-)
I couldnt add items to my shopping list because I had too many deleted items in keep list.
> worth mentioning that the white-box / black-box terminology is in itself part of a long history of racially coded terms in science; researchers have pushed to change "blacklist" to "blocklist," for example
Very plainly, the idea of "black-box" comes from the clear, basic and original, notion and experience that
"in the dark, you cannot see".
So, back to the point: we need transparency in AI, because we need insight as much as we can gather, because there appears to be a drought, an arificial scarcity, of good sense, of [un]common sense. We need every boost of good, [un]common sense very direly.
blacklist came from BEFORE slavery in America.
> According to the Henry Holt Encyclopedia of Word and Phrase Origins the word "blacklist" originated with a list England's King Charles II made of fifty-eight judges and court officers who sentenced his father, Charles I, to death in 1649. When Charles II was restored to the throne in 1660, thirteen of these regicides were executed and twenty-five sentenced to life imprisonment, while others escaped.
Thinking that blackbox is racist is approaching insane level of mind-bending "everything is about race".
An idea doesn't have to be true to cause someone suffering.
"To make the world safe for feet, one might try to cover it entirely in leather, which is an insurmountable task. However, if you put the leather on your own feet..."
My goal wasn't to argue the concern was objectively correct, just to point out that it was different from what was being critiqued.
The data bias issue is about having way more samples of class X than of class Y, or class Y sharing an unknown but correlated feature (medical images with labels have this problem) that the developer doesn't identify, or any other number of "biases", like all the images being too bright, or taken with a camera that isn't identical to the one that's going to get deployed in production, etc., etc.
There are real issues that can be fixed / engineered / understood in terms of producing reliable output, and xAI absolutely helps with that! But for whatever reason journalists don't seem to understand that these systems need normal engineering safeguards, like any automated system, and bring it back to one poorly engineered model to talk about Big Bad Racist AI always denying loans based on race.
https://en.wikipedia.org/wiki/Flight_recorder#:~:text=The%20....
Of course it was, but the expression stands because of the opacity implied by 'black'. The [original] box contains secrets - it may even be rigged -, or anyway very complex technology, state of the art, just developed or evolved, especially difficult to understand - you do not know how it works. It is black, and that "black" is significant in many ways.
(And of course, a "flight recorder" is not the same thing: it is a specific function - not of the most esoteric - of the original pack of potential embedded technologies, which inherited the name. The radar, for example, becomes "mainstream", while the recorder remains "boxed" because you want it protected for "emergential witnessing".)
I hope that the armchair pragmatist in you would react, when told "I would not trust your wife", by asking for more information.