‘Context’ matters in AI decision making, says ICO

Factors such as the urgency of the decision, its impact, and significance might outweigh a data subject’s wish to know more about the decision-making process, suggesting that a “one size fits all” approach to explaining AI-generated results is unworkable.

Such an attitude might benefit the AI industry, which says that while it “felt confident” it could technically explain the decisions made by AI, such “explainability” can be hampered by cost, “commercial sensitivities,” and the potential for “gaming” or abuse of systems.

The lack of a standard approach to establishing internal accountability for explainable AI decision systems also emerged as a challenge for developers, said the Information Commissioner’s Office (ICO).

The findings appear to be at odds with the concerns of data regulators, particularly in Europe, who want to ensure that transparency, fairness, and accountability “remain core” to how personal data is used in AI decision-making systems.

European Data Protection Supervisor Giovanni Buttarelli told attendees at the 40th International Conference of Data Protection and Privacy Commissioners in Brussels last October about the need to understand the ethics behind increased AI usage and how technologies use data to inform decision making. “We are fast approaching a period where design, deployment, and control of new technologies and technological processes are delegated to machines,” he warned.

Last year the U.K. government asked the ICO and The Alan Turing Institute, the country’s national institute for data science and artificial intelligence, to produce practical guidance for organizations to assist them with explaining AI decisions to the individuals affected. The initiative is called “Project ExplAIn.”

In an interim report released June 3, the ICO found that the importance of providing explanations to individuals—and the reasons for wanting them—changes dramatically depending on what the decision is about.

Based on feedback from “citizen juries”—effectively, members of the public who heard evidence from AI experts—jurors felt that explaining an AI decision to the individual affected was more important in areas such as recruitment and criminal justice than in healthcare, for example.

This was because jurors wanted more detailed explanations about decisions made in a recruitment or criminal justice context so they could challenge them, learn from them, and check they had been treated fairly.

In healthcare settings, however, jurors preferred to know a decision was accurate rather than why it was made.

The ICO said the findings showed there is a “need for improved education and awareness around the use of AI for decision making.” It added that there is also a need for board-level buy-in on explaining AI decisions, while a “standardized approach to internal accountability to help assign responsibility for explainable AI decision systems” is also necessary.

In the interim report, the ICO outlined a number of areas likely to be in the first draft of the Project ExplAIn guidance, which it aims to put out for consultation this summer before releasing a final version in autumn.

It includes setting out the legal requirements around AI decisions, including data protection and other relevant regimes, as well as enshrining corporate responsibility in the three key principles of transparency, context, and accountability.

The guidance will also likely include sections on organizational controls (such as the reporting lines around AI decision making, as well as policies on training and risk management) and technical controls on data collection and use.

The first draft will also feature a section on “explanation delivery” outlining the “contexts” in which data subjects will be told about how decisions are made.