The Glass Box

The machine shows the reasons behind its decision, not just the decision, so the operator can check it, trust it or override it.

Patriot missile battery

March 22, 2003 · Iraq–Kuwait border

A British Tornado was flying back to base after a mission over Iraq when a US Patriot missile battery's computer classified it as an incoming anti-radiation missile, a weapon that homes in on radar. The aircraft's identification system wasn't working, and the criteria programmed into the Patriot for that kind of missile were broad enough to include it. The battery's crew, following their rules of engagement for self-defense, fired. Both men on board the Tornado were killed.5,6

Anti-pattern: Verdict Without Reasons

The system gave the battery's crew an answer, anti-radiation missile, but not the evidence behind it. They couldn't see that the classification rested on criteria a friendly aircraft with a broken transponder could meet.56 An answer without reasons leaves only two choices, trust it completely or ignore it, and under time pressure people trust it.

The Glass Box

Therefore: Show the few reasons behind each recommendation, in terms the operator can check against what they can see, and record what they override.

Simplified illustration.

Without

Forklift aisle check

The forklift has checked aisle 7.

With the pattern

Forklift aisle check

The same check, on a display that shows its working.

Operators check the reasoning against what they see, and override when the machine is missing something.

The mechanism

inputs → decision logged

  1. 01InputsSensors, history, thresholds
  2. 02InferenceModel weighs the evidence
  3. 03 · Key momentRecommend + why"X, because of A, B and C"
  4. 04Operator judgesAgree, question or override
  5. 05Decision loggedWith the reasons behind it

At 03: Surface the drivers: sensor confidence, historical match, and which threshold was breached.

At 05: Overrides feed back, so disagreement becomes signal instead of noise.

On this page

Design considerations

  • Give a few reasons, not the whole model. Dumping everything into the interface produces the same rubber-stamping as showing nothing, so the reasons have to be few and specific to the decision in front of the operator.
  • Make every reason checkable. Each one should be something the operator can compare with what they can see or know. A blocked camera means something to a person standing in the aisle; a confidence score on its own does not.
  • Show how sure the machine is about each reason. A confident answer resting on one weak input should look different from one resting on several strong ones — the projection level of the transparency model is where that uncertainty belongs.1
  • Say what else the evidence fits. "No identification reply" also fits a friendly aircraft with a broken transponder, which is what it was.5 Naming the alternative is often the most useful reason of all.
  • Record what operators override. Repeated disagreement in the same situation is the clearest sign the model is wrong, which is what Learning Loop does with it.

In the field

Observed · Defense. The Patriot battery and Tornado ZG710: the classification criteria were "very broad", the aircraft's IFF did not reply, and the crew had about a minute to decide. The Board found that with a wider picture of the airspace they "would then have been more likely to identify ZG710 as a friendly track, albeit one without a working IFF" — the reasons existed, but not where the crew could reach them.5

Sources

An explanation only helps if the operator can check it against something they can see for themselves.

  1. Situation Awareness–Based Agent TransparencyJessie Y. C. Chen et al. · 2014Paper
  2. Agent Transparency for an Autonomous Squad MemberMichael Boyce, Jessie Y. C. Chen, Anthony Selkowitz and Shan Lakhmani · 2015Paper
  3. Trust in Automation: Designing for Appropriate RelianceJohn D. Lee and Katrina A. See · 2004Paper
  4. The “Problem” of Automation: Inappropriate Feedback and Interaction, Not “Over-Automation”Donald A. Norman · 1990Paper
  5. Military Aircraft Accident Summary: Royal Air Force Tornado GR Mk4A ZG710UK Ministry of Defence, Directorate of Air Staff · 2004Incident
  6. Tornado GR4A ZG710 (Accident Summary) — written statement, 14 May 2004Ivor Caplin, Parliamentary Under-Secretary of State for Defence · 2004Incident