Handling data and AI models when closing the loop

This article was written by members of TM Forum’s Closed-loop Anomaly Detection and Resolution Automation project. Authors include: Tayeb Ben Meriem, Senior Standards Manager, Orange Emmanuel Otchere, Chief Architect, Huawei; Amandeep Singh, Technology Architect, IBM; Sharath Prasad, Data Scientist IBM; Utpal Mangla, VP & Senior Partner, IBM.

This is the third blog in the series from the TM Forum Closed-loop Anomaly Detection and Resolution Automation (CLADRA) project. The first blog covered how closed-loop management can help communications service providers (CSPs) tackle network challenges and improve customer experience. The second blog covers closed-loop Patterns with focus on OODA (Observe-Orient-Decide-Act) approach and shares logical architecture and it’s mapping to OODA layers.

This third blog covers the core processes dealing with data and AI Models as part of CLADRA Logical Architecture and the way it is mapped with AI closed-loop-driven ODA (Open Digital Architecture) as prescribed in GB1022. However, the intention is to apply this mapping to Autonomous Network Architecture Framework as well, once available. In both cases, the goal is to identify communalities and gaps to prevent duplications and use those AI closed-loop capabilities as common generic enablers to be used and consumed by all those architectural frameworks.

AI models and training

AI models are the brain of the closed-loop automation process and data is the fuel of the models to make it operate within a cognitive business and cognitive service assurance fashion.

There are a number of different steps involved in creating, training and operationalizing these models. These include:

• Data collection


• Data composition


• Training of models


• Validation and testing of models


• Deploying and monitoring models

To help CSPs streamline efforts in their AI journey, and maximize the benefits of AI closed-loop automation, we propose that the above steps must be subjected to the AI Checklists assessment exercise as described in IG1239.

The data collection process consists of collecting data from resources and services. The resources consist of both physical infrastructure such as networking and storage systems as well as virtual infrastructure such as firmware, operating systems etc. Services include functional services that are executable containers or applications; software programs or an implementation of software that performs a function for the organization, user, service, or other resource. The data collected from these systems include operational data, performance data, usage data and context data.

The data composition process performs the crucial role of converting the incoming data into valuable information and actionable knowledge. The data composition process is responsible for building information across data sources, and consists of cleaning, formatting and organizing data. The components in this layer allow for connection to, and managing of the data sources as well as all required data conversions.

Once the data is collected, cleaned, transformed, and composed, it is ready to be used to train a model. The AI systems needs to be trained to identify and resolve any issues. Depending on the use case, the training data could be time series data, trouble tickets, run books and a variety of other data sources. A number of techniques may be applied to train a model including machine learning and deep learning algorithms. Supervised learning algorithms such as neural networks, k-nearest neighbor, and decision trees are often used. Unsupervised algorithms such as K-means, autoencoders, principal component analysis and hypothesis tests-based analysis are also employed.

Once the model is trained, one needs to validate the models which involves measuring the quality of the model and fine-tuning the model hyperparameters. The validation dataset is used to evaluate and compare performances of multiple trained models and decide which works best. Once the best model is selected, it is tested on the testing dataset. There are several evaluation metrics such as accuracy, precision, recall etc.

Once the model is tested, it is deployed. Deploying a machine learning model means to integrate a machine learning model into an existing production environment where it can take in an input and return an output. In the closed-loop Automation context, the purpose of deploying your model is so that you can make the predictions from a trained machine learning model available to other systems to then take informed actions on various target systems such as network devices/ operating systems/ applications etc. Systems and data change with time, so a model created yesterday may not be accurate. It is therefore essential to ensure models remain accurate, free of bias and can accommodate for changing data environments. Therefore it is important also to monitor these deployed models to ensure the models remain fair, explainable and compliant.

AI Closed-loop-based ODA Intelligence Management Framework

Figure 1 (below) depicts the ODA Intelligence Management Framework as described in GB 1022. In this Framework, AI, autonomics and cognitive capabilities are built-in per design in the Intelligence Management functional block, and in each of the three ODA horizontal functional blocks (Party Management, Core Commerce Management, and Production).

Both types of decision making elements, or decision elements (DEs / AI models), are either logically centralized or distributed control software logics (DEs) that operate in different timescales, but interwork harmoniously in realizing these autonomic behaviors: Self-configuration, self-optimization, self-healing of managed entities.

Figure 1: ODA intelligence management (AI closed-loop-based ODA Architecture) Ref GB 1022

Mapping between CLADRA and AI closed-loops-based ODA

In Table 1 below, we compare and contrast CLADRA (IG1219) with ODA Intelligence Management (GB1022)

Table 1: Mapping between CLADRA and AI closed-loop-based ODA

Takeaway #1

We can see ODA and CLARDA Frameworks are aligned. Both are multilayer (hierarchical and nested AI closed-loop-based frameworks composed with two layers: Slow control loops and fast control loops

Mapping CLADRA use cases with AI closed-loop-based ODA architecture

This mapping can be broken down into two categories:

• CLADRA use cases pertaining to AI closed-loop-driven business assurance. It can be hosted by ODA Party Management and ODA Core Commerce Management functional blocks)


• CLADRA Use cases pertaining to AI closed-loop-driven service assurance. It can be hosted by ODA Production functional block

Takeaway #2

ODA Intelligence Management makes a clear separation between “cognitive business assurance”, and “cognitive flexible catalog assurance” in one hand, and “cognitive service assurance” in the other. This offers two categories of ‘platforms’ for hosting and operationalizing the two categories of CLADRA use cases 1) “AI closed-loop-driven business assurance use cases” 2) “AI closed-loop-driven service assurance use cases”

Mapping of AI closed-loop-based ODA Functional Architecture with CLADRA Logical Architecture

Figure 2 depicts such a mapping. We indicated in both architectures the “OODA” steps by their respective symbols: Observe (Ob); Orient (Or); Decide (D); Act (A); Security (S); Automation (Au)

Figure 2: Mapping of AI closed-loop-based ODA Functional Architecture with CLADRA Logical Architecture

Takeaway #3

It is a high level mapping in the sense that we considered only the ‘layer’ levels without considering component levels associated with each layer. We notice there is a nice mapping between CLADRA Logical Architecture and AI closed coop-based ODA Functional Architecture

Looking forward

AI Closed-Loop Automation Reference Architecture (TR284) captures key items for future consideration to be addressed in the next iterations/sprints of the CLADRA project. Item 1) titled “Mapping to ODA” – which constitutes the main content of this blog, along with data and AI models – aims to provide an initial high-level scoping. The next blog in this series will be dedicated to scoping the API view of AI closed-loop-driven ODA intelligence management.

We invite you to join our current study on closed-loop automation, with further details to be found here.