Return of Control

When the machine takes control back, it does so visibly and only when the person agrees, and it never quietly fights what the person is doing.

Driverless forklift at a bottling plant Draft

December 9, 2015 · Washington State

A water bottling plant moved pallets with driverless forklifts that navigated by laser. When a forklift sensed something in its way it stopped, and as soon as the obstruction was gone it started again on its own, unless someone had pressed an emergency stop. A dock coordinator went into the warehouse after a forklift alarm went off. A coworker who heard noises over the radio found him slumped against one of the vehicles, and then the forklift started moving again. The 45-year-old died after its raised forks came down on him.2

Anti-pattern: Resumes on Its Own Draft

The forklift treated a stop as a pause. Once its sensors cleared, it took control back by itself, with no one choosing to hand it back and no one confirming the space was safe.

Return of Control

Therefore: When a person takes control, the machine stops acting and says so, and it resumes only when the person deliberately hands control back.

DraftSimplified illustration.

Without

Driverless forklift · Aisle 4

The forklift is working the aisle. You are about to walk into it.

With the pattern

Driverless forklift · Aisle 4

The same forklift, on a machine that holds.

Stopping is visible and resuming needs a deliberate human act, so the machine never takes control back on its own.

The mechanism

human has control → machine resumes

  1. 01Human has controlThey have taken over, or never left
  2. 02Machine stands backVisibly, and without arguing
  3. 03Conditions are metThe state the machine needs to resume
  4. 04 · Key momentHuman confirmsA deliberate act, not a timeout
  5. 05Machine resumesAnnounced, once, not repeatedly
On this page

Design considerations

  • Stand back the moment the person acts. One correction from the operator should stop the machine, not pause it until the next cycle. After the 737 MAX accidents, the fix was to let the system act only once per event.5
  • Make it obvious who's in command. At any moment, the operator should be able to tell whether the machine or they are flying.
  • Resume only on a deliberate act. Industrial emergency stops require a manual reset at the point where the stop happened.1
  • If the machine resumes on its own, show the condition. Super Cruise resumes when a visible condition is met, and its light bar tells the driver what that condition is.8
  • Give the person the last word. A disagreement between a person and a machine should end in a decision, not a fight.

In the field Draft

Observed · Aviation. China Airlines 140. The automation kept trimming nose-up against the pilots' nose-down inputs, and nothing on the flight deck showed the contest.3

Observed · Aviation. Lion Air 610 and Ethiopian Airlines 302. On Lion Air 610 the trim moved more than thirty times, the pilots trimming back against it each time; using the electric trim paused the automated system, which then reset and pushed the nose down again.67 All 189 people on board Lion Air 610 died, and all 157 on Ethiopian 302.4

Observed · Automotive. Super Cruise's light bar shows when it will resume: it pulses blue while the driver is steering, and the driver centres the car in the lane and waits for green before the system takes the wheel again.8

Observed · Industrial. ISO 13850 keeps an emergency stop in force until someone resets it by hand, at the place where the stop was triggered.1

Sources

The two interface examples sit at opposite ends. Super Cruise resumes on its own once a visible condition is met;8 an industrial emergency stop won't resume at all until someone acts at the machine.1 Neither is simply right. What they share is that the person can always tell which of the two is in command, and that's what turns a disagreement into a decision instead of a fight.

  1. ISO 13850 emergency stop resetIDEC, via FarnellStandard
  2. Warehouse Worker Crushed by Forks of Laser Guided Vehicle (FACE Report 71-171-2018)Washington State Department of Labor & Industries · 2018Incident
  3. Aircraft Accident Investigation Report — China Airlines Airbus Industrie A300B4-622R B1816, Nagoya Airport, April 26, 1994Aircraft Accident Investigation Commission, Ministry of Transport, Japan · 1996Incident
  4. Assumptions Used in the Safety Assessment Process and the Effects of Multiple Alerts and Indications on Pilot PerformanceNational Transportation Safety Board · 2019Incident
  5. Maneuvering Characteristics Augmentation SystemWikipediaArticle
  6. Lion Air jet's flight data show 'tug of war' over controlClaims Journal · 2018Article
  7. Taking the next steps while awaiting the MAX returnSatcom Guru · 2019Article