Capability Disclosure
Before the operator hands over control, the machine says where it can't run, and refuses to engage outside those conditions.
Autopilot runs on a road it wasn't built for
May 7, 2016 · Williston, Florida
A Tesla Model S was steering itself at 74 mph in a 65 mph zone on a divided highway west of Williston, Florida, a road with intersections and cross traffic. The owner's manual said Autosteer was meant for limited-access roads, but the car let it run here anyway. Of the 41-minute drive, the system steered for 37 minutes, and over all of it the car detected the driver's hands on the wheel for a total of 25 seconds. When a tractor-trailer turned left across the highway ahead, neither the car nor the driver braked, and the car passed under the trailer, killing the driver.3
Anti-pattern: Fine Print Limits
The car's limits weren't hidden. The manual restricted Autosteer to highways and limited-access roads, in a ten-page section with sixteen warnings and four cautions.3 And the car knew what road it was on, from GPS, map data and speed-sign recognition.3 It engaged anyway. The boundary existed in the owner's manual and nowhere in the machine's behavior. A limit the operator has to remember is a limit the operator will eventually forget, usually on a day when everything else is going well.
Capability Disclosure
Therefore: Make the system enforce its own limits, and when it refuses, name the condition it's missing.
Simplified illustration.
Without
Driver assistance
Pick either road. The control does not care which.
With the pattern
Driver assistance
Autosteer unavailable:
this road has cross traffic.
Available on limited-access
highways.
The control is there. It will not engage here, and it says why.
The machine holds the line itself, so operating outside what it was built for stops being something an operator can do by accident.
The mechanism
declare the envelope → re-check while running
- 01Declare the envelopeWhere it works, what it can't see
- 02Check the conditionsIs this one of them?
- 03 · Key momentOffer or withholdEngagement available only inside
- 04Name the blockerSay which condition is missing
- 05Re-check while runningWithdraw when the envelope ends
At 01: Name conditions the operator can recognize from the seat (this road, this load, this weather), not a sensor specification.
On this page
Design considerations
- Name the missing condition when you refuse. Refusing outright is honest but unpopular, and a system that refuses without saying why just reads as broken. “Autosteer unavailable: this road has cross traffic” keeps the operator’s trust; a greyed-out icon loses it.
- Describe limits in the operator's terms. This road, this load, this weather, not a sensor specification.
- Enforce the limit, don't just describe it. If the machine can tell it's outside its conditions, it shouldn't rely on the operator remembering the manual.3
- Keep checking while running. Conditions change mid-task; when the machine leaves its envelope, it says so and hands back control with time to spare. That handback is Graceful Degradation's job, and this pattern is what tells it where the edge is.
In the field
Observed · Automotive. Tesla, Williston: the manual restricted Autosteer to highways and limited-access roads, and the firmware placed no restriction on road class at all.3
Observed · Automotive. GM's Super Cruise offers hands-free driving only on roads GM has mapped, so the limit is enforced by the system rather than stated in a manual.4
Related patterns
- Precedes · 03Graceful DegradationDisclosure draws the envelope before anything starts; degradation is what happens when the machine reaches that edge while running.
- Complements · 05Shared Mental ModelDisclosure is where the human's model of the machine starts. Keeping it current once the machine is running is the other pattern's job.
- Complements · 11Readiness CheckDisclosure says where the machine will run; this says whether the human it is counting on is actually there. Both are preconditions checked before engagement rather than claims made in a manual.
