Snake oil. This is not really explaining components of models as much as it's characterizing sensitivity to certain types of inputs, with humans labeling those types. It doesn't scale and it doesn't explain
In a way you are right, but on the other hand there are two parts of explainability: one is the model explanation, and one is the class explanation. Often times the explanation of the class is what people really need (I classified this as a Husky since there is this dog area and snow area on the image).
This doesn’t help in audits if ML models (where we want to understand why the model attended to these areas). On the other hand: are humans really able to explain their inner workings of the brain, or aren’t we as well describing what kind of input features motivated our decisions?
> On the other hand: are humans really able to explain their inner workings of the brain, or aren’t we as well describing what kind of input features motivated our decisions?
That's really an interesting question. Even mathematical proofs are not really 100% rigorous. They are really about convincing others and it's really dependent on the others knowledge. As an engineer I would require much more detailed steps than a mathematician to accept a proof (if we engineers cared about that, of course).
So, to summarize, an explanation is really an ill defined concept.
3 comments
[ 4.9 ms ] story [ 22.7 ms ] threadThat's really an interesting question. Even mathematical proofs are not really 100% rigorous. They are really about convincing others and it's really dependent on the others knowledge. As an engineer I would require much more detailed steps than a mathematician to accept a proof (if we engineers cared about that, of course).
So, to summarize, an explanation is really an ill defined concept.