Explainable autonomy: test what people understand
An explanation is useful when it helps someone anticipate a system’s behavior. A polished description can still mislead if it presents a plausible story instead of the reasons relevant to the decision.
G-ATAI analysis
For an AI product, define the decision a reviewer needs to understand and identify the evidence that can support that explanation. Keep generated narrative distinct from measured inputs and model outputs. Test explanations on failures as well as routine cases. The interface should acknowledge when the available evidence cannot account for a recommendation.
Our proposed evaluation asks people to predict what the system will do in a new situation before and after seeing an explanation. Compare the accuracy of those predictions with a version that offers no explanation. Confidence alone is a weak measure because an elegant but incorrect story may increase it. Use the results to improve review and debugging while preserving an independent process for testing the underlying system’s behavior.
Questions before deployment
- Does the explanation reflect evidence from the actual decision?
- Can reviewers predict failures more accurately after reading it?
- Are confidence and prediction accuracy measured separately?
Independent commentary inspired by MIT News. No affiliation or endorsement is implied.
