New AI checklists address complexity and accountability

During the past 50 years the aviation industry has lowered the ratio of fatal accidents per 1 million flights 16-fold, from 6.35 in 1970 to just .39 in 2018. A big driver of this dramatic improvement has been implementation of standardized checklists that all pilots use to ensure safety. Now, TM Forum members are taking a page from the aviation playbook by developing AI Checklists that can help companies manage the complexity of AI, and use it safely and responsibly with ongoing accountability.

“Aviation has built a learning culture,” says Rob Claxton, Chief Researcher at BT and Leader of TM Forum’s AI Management Standards project. “Every accident gets investigated, and safety recommendations are made. There is an ability to propagate that information across the industry so that everyone benefits from it, and this knowledge isn’t lost. It’s part of the safety management systems that have been developed, and standardization has played a big part in achieving consistent performance across aviation globally.”

This approach is used in healthcare as well, with surgeons adopting checklists to improve surgery outcomes by reducing deadly errors and infections. The process is explained in Atul Gawande’s The Checklist Manifesto. Gawande helped develop the World Health Organization’s Safe Surgery Checklist, which is pictured below.

Managing complexity

“The checklist is a way of managing the complexity of the domain you’re operating in,” Claxton explains. “Deploying AI at scale has become equally complex, so we can’t just rely on individuals’ knowledge to make it safe and deliver benefits.”

The idea of TM Forum’s new AI Checklists is “to help people do the right thing in the right moment and make sure they’re not forgetting critical steps,” he says. “But it’s important that they don’t become tools for simply checking compliance, because as soon as that happens, they become a stick to threaten people as opposed to a tool to help.”

The AI Checklists focus on six stages of AI adoption: Procurement, Pre-development, Post-development, Deployment, In-life and End of life. The graphics below explain the aim of each checklist and show the steps to complete in each stage.

The power of simplicity

Claxton points out that while some checklist steps may seem simple, implementing them can be effective and powerful. For example, as part of the Safe Surgery Checklist, everyone in the operating room or theatre must introduce themselves before commencing the surgical procedure.

“This is so that everyone knows who is present and what skills are present in the room,” he explains. “That kind of icebreaking moment turns out to be quite important when things start to go wrong, because people have already met and understand the skills that are available.”

CSPs can benefit in the same way. For example, the Pre-development Checklist includes a prompt to make sure people stop to think about the data that they’re going to use to build the system. “Is it appropriate quality? Do they actually know where it came from?” Claxton explains. “This is the first step in establishing inputs into the system, but time and time again, I've seen that completely sidestepped.”

Also in the Pre-Development stage data must be tested for coverage and accuracy to address the potential for bias and poisoning. The ImageNet database, which contains millions of images used to train computers to recognize visual concepts, illustrates why this is important. Recently, researchers found that algorithms trained with the data were making assumptions about people shown in images because derogatory labels, such as racist terms, had been added intentionally.

“[ImageNet] is the foundation of many of the image machine vision models in use today, and they all depend on a data set whose quality is now known to be problematic,” Claxton says. “If you don’t know where your data has come from, how do you know it has not been poisoned in some way?”

What’s next?

TM Forum’s initial AI checklists have been approved by the collaboration team so that companies can begin using them as they deploy AI systems, but they are “dynamic” documents that will be continuously tested and updated. Indeed, the team would like to get input from other TM Forum members and people outside the Forum to improve them.

“Checklists shouldn’t be completely fixed,” Claxton says. “Ideally, they should be something dynamic and adaptable by different organizations with different practices.”

If you would like to learn more about the AI checklists or to find out how you can contribute feedback, please contact Aaron Boasman-Patel, TM Forum’s VP of AI & Customer Experience.