In many cases, it depends on the context. For self driving cars, most companies only use neural networks for components of the cars and an explainable algorithm on top of that that interprets data from the components. If there was a crash, they can tie it down to "the neural network parsing the LIDAR did not classify the stop sign correctly, so the driving algorithm did not stop" without needing much interpretability of the neural network directly. If it's an end-to-end system, such as the nvidia car which connects the camera inputs directly to a wheel output, then explainability of that network is much more important.
There was a poll I saw on twitter a while ago (can't seem to find now) that had asked if users would prefer to be treated by an AI doctor with low explainability but 99% accuracy, or high explainability but only 70% accuracy. I think context is key here as well. If the expert doctor working in tandem with the explainable system is better than the non-explainable system, you'd want the explainable system. But if the non-explainable system is better even than experts, I think most people would want the non-explainable system.
Your AI self driving car doesn't need to explain itself, it just needs to arrive safely. An AI that merely supports a human decision maker also might not, depending on how its applied. But the AI that might deny your access to treatment for a disease, reject your mortgage application or recommend to a patrolling police officer that he should pull you over, and not the next car -- those AIs should explain themselves.
This is easy to explain. The driver is at fault. Explainability is what we are debating.
Becauae there’s nothing to explain. The software is broken and needs to be fixed, and if the problem is severe and the manufacturer knowingly tries to cover it up, that manufacturer is culpable of negligence.
The software has no more agency than a cruise control.
When these processes kill people we very much do somethint about them.
The second statement kinda denies the first. If broken software causes a severe or expensive problem, people generally expect an explanation. That is what a post-mortem is generally for.
I have to be able to debug anything I work with; debugging is necessary.
>> "the neural network parsing the LIDAR did not classify the stop sign correctly, so the driving algorithm did not stop"
It's important to look into why the stop sign was mis-classified, even if the classifier is a subcomponent.
I see these training approaches as a tool that makes machines search the space of potential solutions for certain types of problems more efficient than humans. The next logical step is to try and understand these solutions and improve upon them.
Consider adversarial input. Right now we cannot tell which classes of adverserial inputs can exist for a given NN. We can only try to find representative examples. If you had a good enough understanding of how the NN works internally, you have a chance to derive the full set of adversarial inputs or - maybe -prove their absence.
Now perhaps we could design another set of algorithms (NN generated?) that would present the right type of explanation to each. Frankly, I'm not very interested in the layman's explanation, because what they want to hear is so disconnected from what's happening that it often seems irrelevant. That said, the placebo effect is real so you want the doctor and patient convinced that they are doing the right thing... so maybe it's more important (edit to add) for actual medical outcomes?
Maybe? You'd have to convince me in areas as medicine. Why are you recommending this treatment path? "Dunno. The computers says so."
Mind you, the doctor may well not in reality have a lot more solid basis for his recommendation. But at least there's the appearance of logical justification.
If you then have treatment based on those tests, that's really a doctor being assisted by an AI at that point.
For that matter, freedom from daily blue screens and reboots is not needed to use an operating system.
All sorts of non-AI software has confusing states and quirks, yet is in wide use.
> "In particular, machine minds that cannot explain themselves, or whose detailed operation is beyond the realm of human language, pose a problem for criminal law."
It doesn't says that the AI is NOT useful, but that it can't be used for the liability that it creates.
> "Dr Datta feeds the system under test a range of input data and examines its output for dodgy, potentially harmful or discriminatory results." ... " If the randomisation of sex produces no change in the number of women offered jobs by the AI, but randomising weightlifting ability increases it (because some women now appear to have “male”abilities to lift weights), then it is clear that weightlifting ability itself, not an applicant’s sex, is affecting the hiring process."
This is the most interesting part of the article. As it shows that it's possible to test the system for unlawful decisions without actually understanding how it thinks.
This isn't as simple as it seems because a supposed job requirement, while superficially neutral, may be unnecessary. For example, police departments used to have height requirements. But height requirements discriminate against women. Do you need to be tall to be a police officer? Maybe it's really about projecting authority? Okay, how do you measure that?
Making job requirements neutral is a rather fuzzy area. What is the job really about? Who decides?
Any variable that's not independent of gender can be used to (imperfectly) detect gender. It may not be easy to figure out if machine learning is using it for something real or just as a substitute gender-detector. Randomizing gender while leaving everything else the same isn't going to detect this.
How can the manufacturer comply, if they don't understand why the AI makes that decision, and how to change its decision-making process? This isn't just about accountability, it's also about the manufacturers' inability to control the behavior of a black-box AI system.
If the manufacturer explains to the judge that they don't really know what's going on inside the AI, the judge won't be inclined to cut them a break.
For example, can any AI expert tell me: what is the exact date when Target corporation crossed the threshold of being able to face match 20% of its customers across different stores with 20% accuracy? How about 50% (for both of the different numbers, to keep it simple). How about 80%? 90%?
That’s just one example. If you’re tempted to reply with insights about face recognition, that’s not the point. The point is more that these numbers for ANY real world AI task (not just face recognition) are generally not shared, much less explained in the real world, even if would be possible to do so in an academic setting.
So yeah, going off what you said, all these people saying “don’t worry, nothing to worry about with AI” haven’t the slightest clue what AI is up to outside of whatever bubble they are in.
This is just false. The reason neural networks seem so mysterious is because in a fully connected neural network a large portion of the interactions (non zero w_ij in the weight matrix) are completely spurious. We look at the fully connected network topology and we throw our hands up. We can apply algorithms to trim out the spurious weights and what we’re left with is a logical circuit that we can analyze. Show an electrical engineer the circuit diagram of a 3 bit added and she’ll know exactly the function. Add a bunch of spurious circuits to that same diagram and of course it’s not going to make sense.
Like for this image of Grandma, these cells are recognizing it, so the others are spurious. Oops, for this other image of Grandma, a different subset are recognizing it.
If, say, the same 90% of the structure is not involved in recognizing any instances, then why wouldn't we have made the whole thing 10X smaller in the first place and trained that much smaller network? Or why don't we make it 10X smaller now? Trim away the dead weight and make it lighter and faster.
Distilling a problem to its essence. Isn’t that the hardest step in explaining it?
So under the status quo, what happens in the case of an aberrant or problematic decision and how does society cope with this lack of interpretability?
We try to piece together a plausible, retrospective narrative by looking at fact patterns and taking into account the education, experience, actions, explanations, and rationalizations of other humans. (Trusting these accounts is further complicated by human traits such as self-interest, emotion, and cognitive biases.)
There exist entire professional specialties devoted to this problem (litigation, internal investigations, police detectives, accident boards) who spend a lot of time on this. Even so, much of the time we still don't really know why exactly people made the decisions they did.
This inconvenient fact does not stop us from employing humans, and society has not collapsed under the weight of the liability issues.
Maybe one day we'll write a deep net that captions other deep nets. Who will build the supervised learning training set though?
If you’ve played against a dan player a few times, you might have had the experience of them ignoring a move you made and play elsewhere. That’s what happened. It’s also probably the moment you lost the game...
Not so sure it's possible for anyone to seriously claim to be able to explain the inner workings.
Biggest problem with these I can see is that there is more than one way to arrive at the given answer.
Simplistic analogy, let's say the equivalent nodes were represented by the value 20. Was this number a result of 5 * 4, 4 * 5, 10 * 2, 10 + 10 etc..
Right now, it doesn't matter, we're all just happy that 20 is correct and works for our use case.
Maybe that will be good enough but I doubt it.
How do you apply the same logic to a neural net? It has no comprehension of desirable outcomes, or what its choices actually entail and never can.
There is no way to tell a neural net that a human is not a signpost directly, so it knows in the future no humans are signposts. The only thing you can do is train it on a larger data set and hope it "gets it".
You can tell a human being that a human is not a signpost once, and they can abstract it to all situations.
NNs (don't call it "AI", it's just one approach) are simply in a pre-scientific phase.
As are our own brains.
This seems like the fundamental difference between AI and human intelligence at the moment. Our intelligence is based on our social prowess. We're always competing with, and exchanging ideas with our peers. AI should be modeled based on this.
AI 2.0: I helped people yell at each other more.
AI 3.0: I helped guide the rockets.