Trust versus understanding - preventing draconian overrule in ML

A conversation with Patrick Hall - Senior Director of Product at H2O.ai

It’s not something I’d ever thought about, but it sounds obvious when Patrick explains that “’trust’ and ‘understanding’ are different technical problems” in machine learning.

“Basically: explanations lead to understanding, which leads to the ability to appeal decisions – all slightly different from trust.”

Check out his paper: “Guidelines for Responsible and Human-Centred Use of Explainable Machine Learning.”

He’s working on “all sorts of interesting things in highly scaled machine learning; mostly ‘interpretable’ and ‘explainable’ ML.”

“I’m just more and more aware that these automated systems for making decisions are becoming more common. From decisions about music to show to you, or whether it’s something much more serious (credit card, prison). They’re becoming much more common.”

He talks about unravelling black boxes, torturing black box models. He advocates ‘white box’ or ‘grey box’ models “for any high stakes scenario”.

“I don’t think it’s a good idea to use black boxes for really high impact decisions that affect people’s lives.”

Documenting software is important, and we discuss the grave reality of unintended consequences.

“We don’t want to be caught. It’s a whole new domain for liability” -- I foresee algorithm lawsuits and ask if that’s happened over there; Americans are notorious for suing over hot coffee, let alone having one’s career, freedom and financial capability negatively affected unfairly due to an algorithm (that a company doesn’t really understand); but, he “can’t comment on whether that’s happened yet”.

He does, however, cite a “famous algorithm, COMPAS, that was used and is probably still used to give judges and parole boards a risk score about whether someone will commit a crime again in the future.”

I didn’t know about that. A black box is already in control of some people’s freedom. How do you feel about that?

“There’s a documented case in New York Times. Because of a black box, where a person was held in prison wrongly. It was very hard for them to appeal.”

Turns out it’s hard to sue a black box …

“That said, it hasn’t yet scared off innovation – there doesn’t appear to be friction from a legal perspective in many areas.”

Yet. I saw a chart about the growing investment for machine learning; it appears to be gaining momentum, most notably in the medical industry. I think the industry would benefit from communicating worst case scenarios so it’s not such a shock when an algorithm kills the odd person. = Expectation management 101.

“There’s an implied trade-off between accuracy and an algorithm’s comprehensiveness, but this is being questioned– some people say it doesn’t exist.”

(He cites Cynthia Rudin for dismissing the Accuracy/Interpretability trade-off: “Please stop explaining Black Box Models for High-Stakes Decisions”.)

“Complex models may be more accurate in static data sets, but this may or may not be the case in the wild …

...Christopher Molnar (and others) [quantifiably] showed a roughly straight line between interpretability and accuracy.”

He moves on to discussing issues of accuracy on static test data sets versus their ongoing accuracy in real world. Personally, I think Tesla is on the money with their up-front goal of designing a system that will be able to handle long tail events. If the system doesn’t truly ‘understand’, as Elon puts it, it’s useless.

I ask about his vision for ML as a technology and what it might solve for the planet. For this question, I always reference the optimised creation and allocation of resources – economics 101 on supply and demand: growingly scarce resources (happening) for growing demand (happening) will result mass unaffordability for the 7 billion people on the planet. Oh wait, or is it 8 billion now?

“This is my take being a kind of cynical medium-term player in the software and analytics market: these technologies have been deployed at the margins of businesses for decades. Marginal gains in terms of saving or increasing revenue. But a hospital is a hospital. Not a data science firm. They charge money for making people better. A power company charges people for power.”

“The longest term most successful applications of this tech. is marginal increase in value. The potential for machine learning is largely to optimise things on the edges … and maybe optimise everything. So, yes, it could make life better for everyone. But as we draw close to the precipice, it could make it worse.”

How, I ask?

Hacked. Unfair. Bias. Wrong. “A combination of these problems.”

(In an email follow up, he clarifies a point on fairness: “I'm just not sure that what American academic elites think about fairness will translate to even middle America (/main St. America) and then how can these ideas work in other cultures with completely different notions of privacy, social responsibility etc.”)

My interpretation here is clear; the elite will be building the algorithms that will impact (optimise, control and allocate) such incredib