Why would anyone use machine learning if not to create systems they don't understand ? That's the whole point of doing it. Not having to understand complex system to understand -or even predict- them is a very good thing indeed. I don't know anything about how to mathematically analyze pixel values to read text, but I can create such a formula using machine learning. After creating the formula I know nothing more about it. So the same person can create programs to read text, transcribe audio, predict loan repayment, ...
And when it comes to the bias in the data - bias in the system thing, well, same thing goes for human judgement. Another way of putting this would be garbage in-garbage out. Humans might be able to tell you of bias in the inputs, rather than just using it, but this ability rapidly deteriorates when the dimensionality of the inputs rises. You can tell if there's something weird about combinations of 3 numbers, but you can't tell if there is something off about 150 number sequences.
Perhaps. But the inability to verify or predict output is a real concern, especially in the nascent stages of applying the tech to an increasing number of critical situations.
Along these lines, I think these classes of algorithms might have much broader applications and could possibly solve problems that we don't even have good definitions for yet. I'm particularly curious to see how the field develops new techniques for systems that start off with poor quality results but evolve with new data to produce better results.
btw, I vouched for and upvoted the parent. It makes an excellent point and, frankly, I'm baffled by the downvotes/dead status.
This sounds far fetch and sci fi, but intelligible machines are particularly of importance in avoiding machines becoming self aware unbeknownst.
However, I think it's also possible that we may never be able to create machines that are self aware as people are unless we do allow them to build without our understanding of them. There could be a recursive relationship in that an intelligence is not capable of understanding itself.
We could just wait until we completely understand how the human brain works, then replicate it directly. It would take longer but be (supposedly) safer than going for a novel but faster approach involving machine learning, genetic algorithms and the like.
I think self-awareness is still so far off that it is not the most pressing problem.
The more serious problem is that once we make life-changing choices using machine learning models, the models should be just and accountable. Obviously, these are vague terms, but in terms of machine learning models they can have concrete meanings:
* Just: choices should be made on what we consider (morally) relevant inputs.
* Accountable: choices should be traceable to inputs and parameters (or eventually training data).
E.g. suppose that we find drones that stun people with a weapon acceptable. Such a drone would be just if it decides to stun a person because they have a weapon. Such a drone would be unjust if it decides to stun a person because they have a particular skin color [1].
Now, suppose that such a drone stuns a person robbing a supermarket with a fake weapon. Regardless of whether we consider this to be just, we probably want to know why the drone misrecognized the gun as real (the model should be accountable).
[1] There may be a non-uniform prior distribution p(weapon_use|race). But I hope we all agree that using such prior distributions would be absolutely unfair towards the individual.
This gets down to the notion of EU "Right to Explanation." We can make all the anti-bigotry laws we want, it won't do any good if people can lock illegal profiling into machine-learning algorithms and excuse bad effects with "The machine said do it, so I did it."
Well, we hope it is. I don't think we can say that with much confidence though unless we understand what the computers are doing. Take some massive datacenter with petabyte sized deep networks doing neural networks. Think of the dimensionality of those matricies. Do you really know what's going on??
Think of the dimensionality of those matricies. Do you really know what's going on?
As someone who uses deep networks for language processing, I completely agree. IMHO this implies that we should be careful making life-changing decisions using such models.
Though, I am not sure whether legislation at this point is useful either. The lack of understanding of new technology in legislative branches typically leads to bad laws.
To be fair, we don't know what going on in that bundle of several billion interconnect neurons between everyone's ears; what the OP refers to as "accountability" is actually process transparency, and we certainly don't have process transparency for individual humans the way we do for human institutions. We do have accountability though--the ability to recognize mistakes and learn from them (with incentives, i.e. rewards and punishments). Machine learning can also have this sort of accountability--if you fuck up, the fuckup is another data point to train on, perhaps weighted for the seriousness of the fuckup.
Given that humans all recognize that there is a bed, it's almost certainly the case that humans also looked at the bed when first scanning the image. We don't require a human show the breakdown of it's visual processing cortex for every decision, why do we want to require it of a machine?
Normal software is split into logical modules and layers, which you can understand on their own and then use that to reason about code at higher and higher levels. Many ML models do not offer anything comparable.
Obviously no one has a full mental model in their head of everything going on in their phone (for example) at a given moment in time. But when it really matters, and there is a specific question to answer, and a programmer with experience writing the kind of software in question has full access to all of the source code, its possible to get an answer to why a program behaved in a particular way.
I'm a programmer, and this is what I do every day.
And in those rare cases where knowledge of the specific hardware is required, there are experts who understand the hardware and can answer the relevant questions.
So for machine learning approaches, what is the equivalent of the "source code"? What resource is there to examine to determine how the system will respond to a given set of inputs, and why? Evidently, there are not clear answers, at least for some machine learning approaches.
To start with, most computers are running about 20-30 microcontrollers. No source code, most of the time even no access to the firmware.
Phones have a bit fewer, but I think you'll find at least about 10.
A simple cheap SD memory card has an embedded core, usually ARM7TDMI. Even most lithium batteries have an integrated microcontroller.
I'm also a programmer, I do bare metal embedded, kernel drivers and such. I often don't have anyone else to ask from, except occasionally hardware guys. They're also often stumped.
So in a high profile case, like the Tesla Autopilot fatality, if the behavior was traced back to a specific micro-controller, it might be just as hard to determine why the system behaved as it did, as with a complex neural network? Even with a company's reputation at stake, it's not possible to hire a consultant with a deep understanding of obscure micro controllers?
I only deal with the source code level, don't really have experience digging into the hardware level.
We rely on many layers of more or less leaky abstractions.
Components have always a set of intended behavior and some, often unknown or poorly understood, set of unintended behavior.
It's very important to understand that abstractions mask both intended and unintended behavior.
You deal with unintended side the best you can, with sanity checks and so on. But some of it inevitably leaks through.
Those outcomes that result from multiple systems being individually in good state but collectively causing undesired behavior are often very tricky to understand. On the flip side, when you do understand them, you tend to gain new surprising insights into the whole system.
Also timing makes understanding hard. When it comes to a complex network of systems, it's hard to concurrently reproduce the state where the effect occurs.
> So in a high profile case, like the Tesla Autopilot fatality, if the behavior was traced back to a specific micro-controller, it might be just as hard to determine why the system behaved as it did, as with a complex neural network?
If Tesla has good, exhaustive, event logging facilities, a "black box", I guess they'll be able to figure out what happened by analyzing all of the available data with the systems knowledge they have.
Anyways, if it's not systematic, why should Tesla autopilot mistake be handled so differently from a human being making a mistake?
You can't analyze why a human being caused an accident either. The best you can usually do is to develop scenarios "theories" based on the available evidence, but that's it.
I don't think analyzing why a human caused an accident is important either. However, being able to fix the bug is very important. If a programmer can identify the problem in their code, they can fix it. But how can one identify the problem in a neural network and fix it? At least right now, understanding an (actually useful) neural network might as well be impossible
Just look at the Toyota acceleration lawsuit, where a consultant spent months trying to figure the system out and still ended up with a probabilistic explanation (basically, the software uses a bunch of bad practices that could result in bad effects that could include the alleged accelerator problem).
I think it's fair to say that we don't have systems anymore that could be 100% understood by a single person. Your smartphone for example, I think it could be 100% understood by a crowd of engineers just big enough to fill a modest concert hall.
Way, way back, I had an 8 bit computer that I built myself from chips and solder. The "permanent" memory was 16kB; I knew most of that code pretty well. Even then the CPU itself required quite a bit of study to comprehend.
Is knowing, for example, that there's a race triggered by a new optimization in compiler version X that happens under conditions Y and Z really a qualitatively different type of understanding than "the weights are such that the ANN computed this incorrect output for these inputs?"
In both cases we can only fix a particular instance of an error and effectively have no way to show that similar errors don't exist, or even that we didn't introduce new bugs with the fix.
Yes. The matrix is stored. In fact, there has been a number of visualization tools for neural nets to help track down which weights contribute to which results.
Additionally, the weights are constant in production, which makes analyzing production issues just a matter of logging the inputs and the outputs. Of course, for most convolutional nets, the inputs can get quite large (for example, a camera feed).
The difference is that all human written systems was once understood by a human. ANN on the other hand creates unlabeled spaghetti which gets impossible to reason about if you scale that up to megabytes. You could try to fix a bug by changing one of the values, but that would ruin the ANN since all the values were chosen to be optimal. Thus the only way to make it better would be to change lots of unlabeled values at the same time and that is something no human can do. The only plausible way to fix a bug in an ANN is to incorporate said bug in the training set.
> The only plausible way to fix a bug in an ANN is to incorporate said bug in the training set
My point was not something about manipulating weights directly vs. augmenting the training set. The analogy is roughly, find a misclassification : add particular input to training set :: find a bug : add a patch/test case. I'm playing devil's advocate, of course, and there are obvious, important differences.
> The difference is that all human written systems was once understood by a human
But that's not one of them! In practice, we create software systems by getting some rough intuition about an interface, wiring together some components, then checking the results empirically. Basic components that everything else is built on, C compilers for instance, are based on ambiguous, even contradictory specifications.
> find a misclassification : add particular input to training set
Unfortunately, adding that particular input to the training set may mess up the ANN when given a different input (i.e. you fix one bug but create one or more new bugs, like a game of whackamole but where people die)
Right. What we're talking about here are tools. We can't understand or debug the electrons moving through the chip. But we build systems using languages and we build tools to translate execution back into those languages (Debuggers) Or tools in the execution to allow us to understand what happened later (Logging, metrics.)
I think the key problem right now is that there are not too many models for which tracing the execution lets us know much about what's happening. A single decision tree. A linear model. But add ensembles to trees or layers to linear models and the ability to understand what's happening through logging disappears.
In this sense I think we're essentially in the pre-assembly language state of machine learning. For most programming languages, we assume correctness as a given. If it produces the wrong result on correct inputs, that's really bad.
With machine learning, we don't have perfect correctness. Aka accuracy. We focus so much on improving accuracy that we rarely consider the trade-off in debuggability or interpretability.
I think this will have to change. Advances in NLP and image stuff from deep learning are very exciting. Ignoring the evil AI concerns, I think the best step will be trying to understand better why the models did what they did. I think it will be a while before we can just ask the model and get a reliable answer. In the meantime, I think we'll work on a kind of meta programming where we integrate models better with their broader context. An example might be sequence to sequence models that have a way to expose sentence fragments and documents in the training set that influenced a word choice. Or automatic labeling of clusters in image representation layers that have semantic meaning to a human. There's some interesting work already on debugging tools for image classification that I really liked. On the phone so I'll try to provide the link later.
The trouble is it all seems very domain specific. Figuring out how do it in a way that's not so domain specific is i think a very interesting thing to work on.
Of course you do. You've presented a trivial Python program; an equivalent in neural networks would be a single scalar input multiplied by a single weight to produce a single output. That's just about as easy to understand without having to run it.
No, the equivalent in neural networks is a network which is trained to always output "Hello, world!", and I dare say that such a network would be pretty hard to reason about compared to the python equivalent.
Actually, the minimal neural network that always outputs that string would basically have a weight vector equivalent to the ASCII coding of the string, so it would be equally easy to reason about compared to the Python equivalent.
You are comparing apples to oranges. The python line is a good representation of a part of a larger program, while the neural network you are talking about is not a good representation of any part of a larger neural network. You wont get that solution if you train it normally without knowing how to solve the problem yourself. Try setting up a network with 3 layers of nodes and 120 nodes in each layer and train it to output "Hello, world!" using a standard training method, you wont get something easily comprehensible from that.
You're neglecting the emergent effect of neural networks. Fully understanding the output of one or more individual neurons hardly tells you anything about how the network behaves has a whole. From that standpoint, neural networks are nearly impenetrable. With human written software you can at least trace through the source code and get an idea of what it's doing. Imagine being handed a large matrix of numbers, being told it represents a neural net, and then being asked to find out what it does. Good luck!
Having been a programmer for thirty-some years now, I can assure you, impenetrable emergent effects in human-written software are very much a thing, and fully understanding the operation of one or more individual lines of code doesn't necessarily tell you anything much about how the program behaves as a whole. Other things equal, there is some tendency for human-written software to be easier to understand, of course, but it's very far from always true.
Think of it this way: if it were impossible to understand opaque, highly emergent systems designed by nonhuman processes, we might as well shut down every life sciences lab and toss all the biology textbooks into the recycle bin. In reality, of course, we can figure out how biological systems work, if we care to put in enough effort, and artificial neural networks are not only orders of magnitude simpler, but have the enormous advantage that we can run them on digital computers, which at least gives us full access to all the raw code and data.
Yes, but a 100 lines Python program is about as simple to understand as this one. Understanding the equivalent neural network requires advanced tooling and lots of hours of labor.
Now, a 1000 lines Python program is generally comprehensible with a little effort. The equivalent neural network is completely opaque.
You don't understand software. You have the source, and then you go read it to understand the little slice you want to understand. Then you forget it and move on unless you have to understand that part again and again. Then you might remember that little piece :) that's it.
I think the author has read that Tesla use "deep learning" to make their auto pilot more accurate and presumed that that means a deep learning algorithm is running in the car.
What I would guess what they do is run deep learning algorithms on the data produced by the car's systems back at their HQ to design more efficient and complete systems, which they then update all of their cars to use.
I have heard Dr. Andrew Ng point out the "well known" fact (in the computer vision community) that nobody, including Dr. David Lowe himself, really understands why SIFT works either. So at least for those of us in the computer vision industry, this isn't maybe such a surprise.
Isn't that the essence of this particular approach though? To "learn" a solution in ways us humans would not conceive of? It's a clear consequence that we cannot easily follow that path of "reasoning" easily.
This is the main difference between Machine Learning and Data Mining; in Machine Learning it's the Machine that learns, in Data Mining it's you (and the machine).
Maybe we'll see the emergence of a profession kind of like clinical psychology for diagnosing strange behavior in trained pattern-recognizing machines :)
Asimov thought there will be robopsychologists. Maybe he was right. Although, his robots had a common foundation laid out by the Three Laws of Robotics, and they didn't consume gigabytes of data to learn.
> and they didn't consume gigabytes of data to learn.
Didn't they though? I'm pretty sure a couple of cameras for vision would mean they consume a couple of gigabytes rather quickly. That doesn't include sound, motor stress feedback and any other included sensors.
Asimov's robots would have consumed massively more data to learn. Of course, it wouldn't have required the data be curated and classified by humans.
The problem is I think that we don't actually know how to program an AI. Neural networks only sidestep the problem by being easier to program by trial and error, but it's nothing more than emulating a different architecture and brute forcing its programming.
But you see the process of training is a solution of the problem "how to program AI". For a machine to be intelligent it has to form it's own model of the world. This model is necessarily somewhat opaque because it includes representations which the machine has created for itself.
An intelligent machine that is interpretable in a straightforward manner is an impossibility, it is a contradiction in terms.
You can teach the machine to answer questions about it's internal state in order to analyse it's function or you can apply your own mind to painstakingly deduce it's operation. But I contend you cannot have a generalised rubric for determining it's operation.
I think there is an economic threshold for the need to know something versus the effort to reverse engineer and find out exactly how that thing happened. (don't xkcd this, randall!)
I think that's exactly the point. Evolution produced our brains, and no one understands them yet. But evolution was able to produce them because it cares about success-testing, not understanding.
I think we limit ourselves quite a bit in the mainstream software industry by insisting that everything be hand-crafted by humans and understandable. (Not to mention how short we fall of making software "understandable").
By "limiting" ourselves this way we can go forward much faster than evolution can. We create designs in hours what would take evolution millions of years to achieve. Also, it's a mistake to treat products of evolution as some kind of gold standard. It's literally worst design that could still work. We can do better, but we do it via different means.
Another thing is that understanding is a key component of trust. Animals come to mind here. We can't understand their firmware per se; by observing them over thousands of years, both passively and actively (through domestication, training regimens, etc.), we've learned to expect some things from them but not others. So I can be sure that e.g. my cat won't kill me in my sleep. But I can absolutely not trust it to go somewhere when I want it to go there, or to stay in place when I want it to stay, or even to eat food when I give it. We want to write software that will bear much greater, more important responsibilities than any animal in history. So we need to make them more understandable than animals if we're to ever trust them with those responsibilities.
Sure, I agree. Evolution did not take us from horse and buggy to planes and autonomous cars in 100-ish years.
But when we push up against the limits of the problems we can solve with software, it's usually because it has outgrown our ability to understand it.
That's exactly what is happening with this new burst of progress in machine learning. We're "teaching" the computer to do things, and we've let go of the ability to directly understand how the computer "represents its learnings."
I think there's an existing analogy here to service animals. You can train a sheep dog to manage the herd, you can train a seeing eye dog to stop it's owner at the lights. You can improve their training, etc. but you'll never really ever know why they sometimes don't follow the expected script.
That isn't necessarily a bad thing. You get more done with them than you would with out them.
The analogy is with anything with a brain, not just animals. Take a poorly trained human and put him in a decision-making position, he will probably make bad decisions. But no amount of training will guarantee you that he won't make any. The learning data set is just too big to be sure you won't have any bias.
"LIKE GOOD STUDENTS, COMPUTERS NEED TO BE ABLE TO SHOW THEIR WORKING"
Is working used as a noun here or is that a typo? Just never heard that use of the word working. Like ya, you can see I'm working vs you can see my working.
Here in Singapore, "working" is the standard term used in schools to refer to the written steps in solving a maths problem (as ajb said). I guess this means it's simply BrE (which Singapore English is a cousin of). I know Americans use "work" for the same purpose instead.
> Using a gerund as a noun seems very unconventional and incorrect almost by definition.
In English, gerunds are nouns (by definition). In "his singing was terrible", singing is a noun.
Furthermore, gerunds do often become used to refer to physical objects/results of actions: "writing", "reading" (e.g. readings for a class), "drawing", "booking" (of a ticket), "crossing" (on a road), "wiring", "etching", and others. Using "working" to refer to the written steps of working out a maths problem is just another example in that vein.
It is a little strange to my ear (possibly more common in British English), the OED has it all the way back to Chaucer though. More often I think of it as mechanism ("the workings of the economy are mysterious"), like clockwork(s).
Or, overfitting strikes again. Except that here the overfit is buried in a bunch of layers. Still -- eventually, especially by reversing the models to generate instead of discriminate, it is possible to be more transparent about what's going on.
I have to agree that the EU, despite normally being a bunch of martinets, is likely getting it right here. It's too easy to construct and overfitted DNN and have it produce bizarre biases that affect patients, payrolls, or patents. There needs to be an appropriate level of transparency baked in as a precondition for licensable IP, IMHO.
77 comments
[ 3.8 ms ] story [ 140 ms ] threadAnd when it comes to the bias in the data - bias in the system thing, well, same thing goes for human judgement. Another way of putting this would be garbage in-garbage out. Humans might be able to tell you of bias in the inputs, rather than just using it, but this ability rapidly deteriorates when the dimensionality of the inputs rises. You can tell if there's something weird about combinations of 3 numbers, but you can't tell if there is something off about 150 number sequences.
btw, I vouched for and upvoted the parent. It makes an excellent point and, frankly, I'm baffled by the downvotes/dead status.
However, I think it's also possible that we may never be able to create machines that are self aware as people are unless we do allow them to build without our understanding of them. There could be a recursive relationship in that an intelligence is not capable of understanding itself.
The more serious problem is that once we make life-changing choices using machine learning models, the models should be just and accountable. Obviously, these are vague terms, but in terms of machine learning models they can have concrete meanings:
* Just: choices should be made on what we consider (morally) relevant inputs.
* Accountable: choices should be traceable to inputs and parameters (or eventually training data).
E.g. suppose that we find drones that stun people with a weapon acceptable. Such a drone would be just if it decides to stun a person because they have a weapon. Such a drone would be unjust if it decides to stun a person because they have a particular skin color [1].
Now, suppose that such a drone stuns a person robbing a supermarket with a fake weapon. Regardless of whether we consider this to be just, we probably want to know why the drone misrecognized the gun as real (the model should be accountable).
[1] There may be a non-uniform prior distribution p(weapon_use|race). But I hope we all agree that using such prior distributions would be absolutely unfair towards the individual.
As someone who uses deep networks for language processing, I completely agree. IMHO this implies that we should be careful making life-changing decisions using such models.
Though, I am not sure whether legislation at this point is useful either. The lack of understanding of new technology in legislative branches typically leads to bad laws.
No, so its probably a bad idea to trust such a system to make important, consequential decisions.
Tagging faces in photos, recommending books or movies, probably fine.
Making an independent decision to fire a weapon, probably very bad idea.
Maybe the machine wasn't sure if most important thing in the question is "window" or "cover", so it looked at a bed looking for bed cover?
Last computer system I can somewhat fully understand is OCS Commodore Amiga. CPU, chipset and software.
Obviously no one has a full mental model in their head of everything going on in their phone (for example) at a given moment in time. But when it really matters, and there is a specific question to answer, and a programmer with experience writing the kind of software in question has full access to all of the source code, its possible to get an answer to why a program behaved in a particular way.
I'm a programmer, and this is what I do every day.
And in those rare cases where knowledge of the specific hardware is required, there are experts who understand the hardware and can answer the relevant questions.
So for machine learning approaches, what is the equivalent of the "source code"? What resource is there to examine to determine how the system will respond to a given set of inputs, and why? Evidently, there are not clear answers, at least for some machine learning approaches.
Phones have a bit fewer, but I think you'll find at least about 10.
A simple cheap SD memory card has an embedded core, usually ARM7TDMI. Even most lithium batteries have an integrated microcontroller.
I'm also a programmer, I do bare metal embedded, kernel drivers and such. I often don't have anyone else to ask from, except occasionally hardware guys. They're also often stumped.
So in a high profile case, like the Tesla Autopilot fatality, if the behavior was traced back to a specific micro-controller, it might be just as hard to determine why the system behaved as it did, as with a complex neural network? Even with a company's reputation at stake, it's not possible to hire a consultant with a deep understanding of obscure micro controllers?
I only deal with the source code level, don't really have experience digging into the hardware level.
Components have always a set of intended behavior and some, often unknown or poorly understood, set of unintended behavior.
It's very important to understand that abstractions mask both intended and unintended behavior.
You deal with unintended side the best you can, with sanity checks and so on. But some of it inevitably leaks through.
Those outcomes that result from multiple systems being individually in good state but collectively causing undesired behavior are often very tricky to understand. On the flip side, when you do understand them, you tend to gain new surprising insights into the whole system.
Also timing makes understanding hard. When it comes to a complex network of systems, it's hard to concurrently reproduce the state where the effect occurs.
> So in a high profile case, like the Tesla Autopilot fatality, if the behavior was traced back to a specific micro-controller, it might be just as hard to determine why the system behaved as it did, as with a complex neural network?
If Tesla has good, exhaustive, event logging facilities, a "black box", I guess they'll be able to figure out what happened by analyzing all of the available data with the systems knowledge they have.
Anyways, if it's not systematic, why should Tesla autopilot mistake be handled so differently from a human being making a mistake?
You can't analyze why a human being caused an accident either. The best you can usually do is to develop scenarios "theories" based on the available evidence, but that's it.
http://www.edn.com/design/automotive/4423428/Toyota-s-killer...
Way, way back, I had an 8 bit computer that I built myself from chips and solder. The "permanent" memory was 16kB; I knew most of that code pretty well. Even then the CPU itself required quite a bit of study to comprehend.
There are pretty significant limits to our conscious, logical understanding of the world.
In both cases we can only fix a particular instance of an error and effectively have no way to show that similar errors don't exist, or even that we didn't introduce new bugs with the fix.
Can we actually do this, for complex ANNs?
Additionally, the weights are constant in production, which makes analyzing production issues just a matter of logging the inputs and the outputs. Of course, for most convolutional nets, the inputs can get quite large (for example, a camera feed).
My point was not something about manipulating weights directly vs. augmenting the training set. The analogy is roughly, find a misclassification : add particular input to training set :: find a bug : add a patch/test case. I'm playing devil's advocate, of course, and there are obvious, important differences.
> The difference is that all human written systems was once understood by a human
But that's not one of them! In practice, we create software systems by getting some rough intuition about an interface, wiring together some components, then checking the results empirically. Basic components that everything else is built on, C compilers for instance, are based on ambiguous, even contradictory specifications.
Unfortunately, adding that particular input to the training set may mess up the ANN when given a different input (i.e. you fix one bug but create one or more new bugs, like a game of whackamole but where people die)
I think the key problem right now is that there are not too many models for which tracing the execution lets us know much about what's happening. A single decision tree. A linear model. But add ensembles to trees or layers to linear models and the ability to understand what's happening through logging disappears.
In this sense I think we're essentially in the pre-assembly language state of machine learning. For most programming languages, we assume correctness as a given. If it produces the wrong result on correct inputs, that's really bad.
With machine learning, we don't have perfect correctness. Aka accuracy. We focus so much on improving accuracy that we rarely consider the trade-off in debuggability or interpretability.
I think this will have to change. Advances in NLP and image stuff from deep learning are very exciting. Ignoring the evil AI concerns, I think the best step will be trying to understand better why the models did what they did. I think it will be a while before we can just ask the model and get a reliable answer. In the meantime, I think we'll work on a kind of meta programming where we integrate models better with their broader context. An example might be sequence to sequence models that have a way to expose sentence fragments and documents in the training set that influenced a word choice. Or automatic labeling of clusters in image representation layers that have semantic meaning to a human. There's some interesting work already on debugging tools for image classification that I really liked. On the phone so I'll try to provide the link later.
The trouble is it all seems very domain specific. Figuring out how do it in a way that's not so domain specific is i think a very interesting thing to work on.
Take this python program:
Do you know what it will do without running it? You have no chance of knowing that about a program created by machine learning.Think of it this way: if it were impossible to understand opaque, highly emergent systems designed by nonhuman processes, we might as well shut down every life sciences lab and toss all the biology textbooks into the recycle bin. In reality, of course, we can figure out how biological systems work, if we care to put in enough effort, and artificial neural networks are not only orders of magnitude simpler, but have the enormous advantage that we can run them on digital computers, which at least gives us full access to all the raw code and data.
Now, a 1000 lines Python program is generally comprehensible with a little effort. The equivalent neural network is completely opaque.
But yeah, let's be pedantic.
We don't understand computer systems. Like you said, We won't even understand software.
What we understand is a small slice of it and we make mistakes even with that. Or fail to consider some cases or scenarios.
Then we put those small slices together to form a module and make the same mistakes again.
What I would guess what they do is run deep learning algorithms on the data produced by the car's systems back at their HQ to design more efficient and complete systems, which they then update all of their cars to use.
Was anyone else bothered by the phrase "show their working"?
That's the kind of thing that grates on me because it's just slightly wrong. When I grew up it was always "show your work", not "working".
Is this a regional thing?
Didn't they though? I'm pretty sure a couple of cameras for vision would mean they consume a couple of gigabytes rather quickly. That doesn't include sound, motor stress feedback and any other included sensors.
Asimov's robots would have consumed massively more data to learn. Of course, it wouldn't have required the data be curated and classified by humans.
An intelligent machine that is interpretable in a straightforward manner is an impossibility, it is a contradiction in terms.
You can teach the machine to answer questions about it's internal state in order to analyse it's function or you can apply your own mind to painstakingly deduce it's operation. But I contend you cannot have a generalised rubric for determining it's operation.
I think we limit ourselves quite a bit in the mainstream software industry by insisting that everything be hand-crafted by humans and understandable. (Not to mention how short we fall of making software "understandable").
Another thing is that understanding is a key component of trust. Animals come to mind here. We can't understand their firmware per se; by observing them over thousands of years, both passively and actively (through domestication, training regimens, etc.), we've learned to expect some things from them but not others. So I can be sure that e.g. my cat won't kill me in my sleep. But I can absolutely not trust it to go somewhere when I want it to go there, or to stay in place when I want it to stay, or even to eat food when I give it. We want to write software that will bear much greater, more important responsibilities than any animal in history. So we need to make them more understandable than animals if we're to ever trust them with those responsibilities.
But when we push up against the limits of the problems we can solve with software, it's usually because it has outgrown our ability to understand it.
That's exactly what is happening with this new burst of progress in machine learning. We're "teaching" the computer to do things, and we've let go of the ability to directly understand how the computer "represents its learnings."
That isn't necessarily a bad thing. You get more done with them than you would with out them.
Is working used as a noun here or is that a typo? Just never heard that use of the word working. Like ya, you can see I'm working vs you can see my working.
Simply ‘work’ reads much better—is it really frequently used as ‘working’ in mathematics‽
> Using a gerund as a noun seems very unconventional and incorrect almost by definition.
In English, gerunds are nouns (by definition). In "his singing was terrible", singing is a noun.
Furthermore, gerunds do often become used to refer to physical objects/results of actions: "writing", "reading" (e.g. readings for a class), "drawing", "booking" (of a ticket), "crossing" (on a road), "wiring", "etching", and others. Using "working" to refer to the written steps of working out a maths problem is just another example in that vein.
Your examples are good, though the only ones that are moderately common in American English seem to be “drawing” and “crossing”.
FTFY
"LIKE GOOD STUDENTS, COMPUTERS NEED TO BE ABLE TO SHOW THEY'RE WORKING"
heh.
I have to agree that the EU, despite normally being a bunch of martinets, is likely getting it right here. It's too easy to construct and overfitted DNN and have it produce bizarre biases that affect patients, payrolls, or patents. There needs to be an appropriate level of transparency baked in as a precondition for licensable IP, IMHO.