I wasn't aware of this and I'm happy it was posted. However I have to point out it contains one of the most strained analogies I've seen recently:
> We are saying that there might be a recipe for the cake, but regardless of the mixers you have available, you may not be able to make the desired cake. Moreover, when you try to make the cake with your mixer in the kitchen, you will end up with a completely different cake.”
> In addition, to continue the analogy, “it can even be the case that you cannot tell whether the cake is incorrect until you try it, and then it is too late,” Colbrook says. “There are, however, certain cases when your mixer is sufficient to make the cake you want, or at least a good approximation of that cake.”
It is terrible, they literally can't explain what they mean. They don't say what it means to "compute the A.I. network". Makes me think it is a bogus story, some academic runoff.
To "compute a neural network" is a long-established way to say "train a neural network", which in turn is a long-established way to say "find a set of weights for the neural network that maximises its accuracy".
The idea is that a neural net is a kind of data structure used in AI, like a decision tree or a decision list (like a decision tree but it's a list). There are different algorithms that can "compute", i.e. construct, a decision tree from data. In modern parlance we say that the decision tree is "trained". Same goes for neural nets, except the network itself is typically constructed beforehand, and manually (we refer to it as the "architecture" of the neural net) and only its weights need to be tweaked until it has a good accuracy- at which point we say the training algorithm has "converged".
It's all a bit confusing because in common parlance there is little distinction made between a neural net's network (its architecture), the algorithm that trains the neural net by finding the weights that minimise its error (backpropagation) and the neural net with trained weights (the "model"). Sometimes I wonder if this distinction is clear in the minds of people who actually train those things.
Btw, the study is solid and meaningful. It's a theoretical result. More of those are needed in machine learning, we got plenty of empirical results.
Thanks. What reason do they give that these network weights cannot be computed?
I am pretty hip to modern AI workings, and that is what I assumed they meant by compute. What I know also tells me that "compute" as a concept is lacking for what we might later call holistic AI, and if, for instance, they are conjecturing strictly SGD proved networks, then I maintain my position that they are merely noodling, with AI and terminology alike.
Intelligence is not entirely computable; nor is occurrence entirely probabilistic. Respectively, there is randomness at work, and dynamic systems that produce novelty.
I'm not going to pretend to be especially knowledgeable in AI/Machine Learning/Neural Nets but I read something a few years ago that helped my perception of the limits of that tech: AI is not capable of explaining why it makes a decision that it does. This was a big flag to me in how much I trust things like full self driving - a car is making decisions with a trained cloud of intelligence that can't explain itself. I'm happy to see scientists and mathematicians trying to find the limits of what the tech can do, it's interesting to model it after how a human mind works, but we don't even understand human minds well enough to try to replicate how they learn intelligence. The world is closer to continuous numbers not discrete numbers etc. AI is still interesting enough to keep building on, Github's Copilot has saved me time just yesterday with very helpful autocompletes on some refactoring I was doing. More of that please - but let me be in the captains seat, deciding whether I want to accept the AI's decision or not.
I wish companies like Tesla would accept this and shift some of the self driving budget to better battery tech. That's the need of the hour with fuel prices rising like they are and global warming looming over us.
Well humans can explain critical aspect of what they doing while teaching other humans, furthermore they can improve explaining while teaching to be more effective in real time in current context, such as using analogies, stories, etc.
That is not the kind of explanation that is needed in AI systems. When people talk about "explainable AI", they literally just mean systems that can answer the kind of question that a human would be able to answer.
That's because a question that a human cannot answer is very likely to have an answer that a human will either not be able to understand, or will have to work very hard to understand... which is no better than no explanation
I think what people are looking for in ‘explainable AI’ is: when the AI makes a bad decision, they want to be able to look into the neural network and say ‘there: that neuron being set to that value is what made the AI mistake the cyclist for a drop kerb’. Then we can fix the value and the AI will not make that mistake again.
But when an AI gets sufficiently complex of course there won’t be explanations that make sense for those kinds of errors, because just like a human the AI is integrating lots of different bits of information that it has learned are important and it has limited capacity and sometimes it just gets its attention focused on the wrong thing and it just didn’t see the guy, okay?
Demanding that AI be explainable is fundamentally demanding that it not be intelligent.
Just to be clear, a neural network is not "an AI". "AI" is the name of the research field. We don't have "AIs" as in Science Fiction yet, and neural networks don't do anything "just like a human". When people in AI research talk about "attention" in neural networks, that's just an anthropomorphic, and quite unfortunate, name for a specific technique in training neural nets. It doesn't mean that a machine vision system has the ability to focus its attention in the same way that humans do.
That out of the way, there are AI approaches that can explain their actions just fine without going dumb. For example, I posted this comment earlier:
about an AI system called Nook that recently won a tournament against 8 human champions of the card game Bridge. In Bridge, players must be able to explain their moves, so an AI player without the ability to explain its decisions can't play a full game of Bridge.
> AI is not capable of explaining why it makes a decision that it does.
I know it's different, we also don't always know why make certain choices. We just pick one and say something to reassure ourselves. "I have a good feeling". A real AI will need to this type of "intuition".
Think of it like human intuition. We haven't developed left-brain logic AI yet, but we are getting closer to training AI to outpace a trained human. They can look at a picture and say "That's a dog". That's an intuitive thought, not a logical one. A car could say "this situation feels dangerous", and ask the drive to take over, or it could just react and steer intuitively. It might not be able to reflect on its actions yet, but that doesn't mean it couldn't learn to drive.
Not that I think we WILL get to full self driving any time soon, but the car not being able to explain itself doesn't prevent us from getting there.
I don't think anyone does video to steering wheel in one network. But they do create a 3d model of the environment from sensors and do planning on top of that. So there is the explanation - what the car thought it saw.
You're right for self driving cars of course. Have you seen this paper? I have not had a chance to read in detail yet, but it seems to be looking at direct steering control from video with a NN:
> AI is not capable of explaining why it makes a decision that it does.
So, yes this is true, but also not the full story. At this point we don't have neural networks that are also capable of explaining their reasoning, but what we CAN do is do a lot of introspection with that network. There is an entire field called AI Explainability that seeks to probe the network in various ways to help humans understand what is happening. Remember that you have total control over the network, and you can run inference thousands of times, or run pieces of the network, or feed test data in to the network.
I am a casual observer of the field but I see this "AI can't explain itself" thing thrown around a lot by people who don't know about the extensive research being done in explainability.
Also Tesla has a massive testing infrastructure that checks their network for regressions. So they will know if it suddenly starts failing in some area before they release it. Obviously this is new and complex tech so it is not perfect.
But I think self driving is important for their business, and they are probably investing heavily in both batteries and AI. And fully self driving electric taxis could eliminate the need for many people to own an ICE car at all.
>> AI is not capable of explaining why it makes a decision that it does.
That's not true for "AI". It's true for a particular kind of AI system, which are collectively known as "black box" approaches. Deep neural nets for example, are a "black box" approach because they can't explain their decisions, as you say.
There are other AI approachs besides deep learning. Recently, a system based on Inductive Logic Programming, a form of machine learning for logic programs, beat 8 human champions in the card game of Bridge:
In Bridge, players must be able to explain their plays to their opponent, and the AI Bridge player in the article above, Nook, was specifically designed to have this ability _and_ play better than human champions.
Btw, lest this is perpetually misunderstood:
AI ⊆ machine learning ⊆ neural networks ⊆ deep learning
Being able to "explain oneself" is a poor metric. Neural networks are not simulated human reasoning systems, they don't work in a way that is parallel to how we would see and explain something. Asking for an explanation is asking for a different kind of reasoning system. Every once in a while there is a "no such thing as tacit knowledge" article - to me this is the dual.
Anyway, IMO is what's more important is demonstrating an AI system can be trusted for the task it's doing (for example by being clear that the data it's predicting on is similar to what it has encountered before), and using the predictions responsibly (like not making a final high value decision based solely on model output). These things are not trivial and areas of ongoing research. But I think they will be more fruitful for developing useful AI vs trying to "explain" something in a way that will satisfy a human checker.
(Also, incidentally, there is work showing that much of explainability - for example feature maps - is just building something that produces output a human wants to see. It doesn't actually or necessarily correspond to how the NN calculated it output)
>a slight alteration in the data they receive can lead to a wild change in outcomes.
In my experience, it's not the alteration that creates an unpredictable outcome; it's the lack of nuance. Accuracy requires an insane amount of details, and it appears that some people might be waking up to that fact that the brain is better suited to handle those nuances.
I work with NLP and have discovered advantages in leaving some decisions up to the person using AI rather than the computer. One advantage being accuracy :-).
23 comments
[ 5.3 ms ] story [ 67.4 ms ] thread> We are saying that there might be a recipe for the cake, but regardless of the mixers you have available, you may not be able to make the desired cake. Moreover, when you try to make the cake with your mixer in the kitchen, you will end up with a completely different cake.”
> In addition, to continue the analogy, “it can even be the case that you cannot tell whether the cake is incorrect until you try it, and then it is too late,” Colbrook says. “There are, however, certain cases when your mixer is sufficient to make the cake you want, or at least a good approximation of that cake.”
The idea is that a neural net is a kind of data structure used in AI, like a decision tree or a decision list (like a decision tree but it's a list). There are different algorithms that can "compute", i.e. construct, a decision tree from data. In modern parlance we say that the decision tree is "trained". Same goes for neural nets, except the network itself is typically constructed beforehand, and manually (we refer to it as the "architecture" of the neural net) and only its weights need to be tweaked until it has a good accuracy- at which point we say the training algorithm has "converged".
It's all a bit confusing because in common parlance there is little distinction made between a neural net's network (its architecture), the algorithm that trains the neural net by finding the weights that minimise its error (backpropagation) and the neural net with trained weights (the "model"). Sometimes I wonder if this distinction is clear in the minds of people who actually train those things.
Btw, the study is solid and meaningful. It's a theoretical result. More of those are needed in machine learning, we got plenty of empirical results.
I am pretty hip to modern AI workings, and that is what I assumed they meant by compute. What I know also tells me that "compute" as a concept is lacking for what we might later call holistic AI, and if, for instance, they are conjecturing strictly SGD proved networks, then I maintain my position that they are merely noodling, with AI and terminology alike.
Intelligence is not entirely computable; nor is occurrence entirely probabilistic. Respectively, there is randomness at work, and dynamic systems that produce novelty.
I wish companies like Tesla would accept this and shift some of the self driving budget to better battery tech. That's the need of the hour with fuel prices rising like they are and global warming looming over us.
How exactly does your brain do route planning?
How do you pick which lane to turn down in the parking lot when looking for a space?
Why did you get off at exit 14?
Wait, this isn’t even your turn, why did you go left here?
I mean, when you get down to it, why are you even driving to this deadend job?
But yeah, sure, humans can ‘explain’ their behavior, which is why we can trust them.
That is not the kind of explanation that is needed in AI systems. When people talk about "explainable AI", they literally just mean systems that can answer the kind of question that a human would be able to answer.
That's because a question that a human cannot answer is very likely to have an answer that a human will either not be able to understand, or will have to work very hard to understand... which is no better than no explanation
But when an AI gets sufficiently complex of course there won’t be explanations that make sense for those kinds of errors, because just like a human the AI is integrating lots of different bits of information that it has learned are important and it has limited capacity and sometimes it just gets its attention focused on the wrong thing and it just didn’t see the guy, okay?
Demanding that AI be explainable is fundamentally demanding that it not be intelligent.
That out of the way, there are AI approaches that can explain their actions just fine without going dumb. For example, I posted this comment earlier:
https://news.ycombinator.com/item?id=30872400
about an AI system called Nook that recently won a tournament against 8 human champions of the card game Bridge. In Bridge, players must be able to explain their moves, so an AI player without the ability to explain its decisions can't play a full game of Bridge.
Not that I think we WILL get to full self driving any time soon, but the car not being able to explain itself doesn't prevent us from getting there.
https://arxiv.org/abs/2106.08314
So, yes this is true, but also not the full story. At this point we don't have neural networks that are also capable of explaining their reasoning, but what we CAN do is do a lot of introspection with that network. There is an entire field called AI Explainability that seeks to probe the network in various ways to help humans understand what is happening. Remember that you have total control over the network, and you can run inference thousands of times, or run pieces of the network, or feed test data in to the network.
I am a casual observer of the field but I see this "AI can't explain itself" thing thrown around a lot by people who don't know about the extensive research being done in explainability.
Also Tesla has a massive testing infrastructure that checks their network for regressions. So they will know if it suddenly starts failing in some area before they release it. Obviously this is new and complex tech so it is not perfect.
But I think self driving is important for their business, and they are probably investing heavily in both batteries and AI. And fully self driving electric taxis could eliminate the need for many people to own an ICE car at all.
That's not true for "AI". It's true for a particular kind of AI system, which are collectively known as "black box" approaches. Deep neural nets for example, are a "black box" approach because they can't explain their decisions, as you say.
There are other AI approachs besides deep learning. Recently, a system based on Inductive Logic Programming, a form of machine learning for logic programs, beat 8 human champions in the card game of Bridge:
https://www.theguardian.com/technology/2022/mar/29/artificia...
In Bridge, players must be able to explain their plays to their opponent, and the AI Bridge player in the article above, Nook, was specifically designed to have this ability _and_ play better than human champions.
Btw, lest this is perpetually misunderstood:
AI ⊆ machine learning ⊆ neural networks ⊆ deep learning
There is a whole field of "explainable AI" - presumably in contrast with (the very popular) "opaque, inexplicable AI."
Personally I like decision trees, because you can trace the reasoning.
Actually the AI is mostly just for object recognition, and the navigation and decision making are simple rules-based systems.
Anyway, IMO is what's more important is demonstrating an AI system can be trusted for the task it's doing (for example by being clear that the data it's predicting on is similar to what it has encountered before), and using the predictions responsibly (like not making a final high value decision based solely on model output). These things are not trivial and areas of ongoing research. But I think they will be more fruitful for developing useful AI vs trying to "explain" something in a way that will satisfy a human checker.
(Also, incidentally, there is work showing that much of explainability - for example feature maps - is just building something that produces output a human wants to see. It doesn't actually or necessarily correspond to how the NN calculated it output)
Has that been proven to be a fundamental limitation yet?
In my experience, it's not the alteration that creates an unpredictable outcome; it's the lack of nuance. Accuracy requires an insane amount of details, and it appears that some people might be waking up to that fact that the brain is better suited to handle those nuances.
I work with NLP and have discovered advantages in leaving some decisions up to the person using AI rather than the computer. One advantage being accuracy :-).