China Mobile details how it attained TM Forum's Level 4 of autonomous networks within its network operations center, an achievement that has delivered sizeable operational efficiencies and secured the company a TM Forum Excellence Award.
China Mobile achieves Level 4 AN in network operation center with intelligent agents
Who: China Mobile and Huawei
What: Automated operations and maintenance and reached level 4 of autonomous networks in its network operating centers (NOCs)
How: Developed automated and intelligent agents and co-pilots for operations and maintenance, supported by investment in accurate telco generative AI (GenAI) and AI models and the use of TM Forum’s frameworks and other assets.
Results:
The size of China Mobile’s network holds up a useful magnifying glass to the impact network faults have on both customers and a company’s bottom line, as well as how AI agents can help address them.
With 578 million 5G customers and just over 1 billion total mobile subscribers, China Mobile runs the world’s largest 5G network by number of subscribers. It also lays claim to operating the largest 400 Gigabits per second optical transport network (OTN) and SRv6 backbone worldwide, as well as the biggest telco-based, single-cluster intelligent computing center.
The extent of its network management needs has given it good reason to be a pioneer in autonomous networks: By 2023 it had reached AN level 3.2 across network-based high-value scenarios. Its next target was to reach level 4, which represents a shift from human-defined automation processes to true autonomous decision-making.
Attaining Level 4, however, depends on end-to-end automation of the network operations center (NOC) in high-value scenarios, and particularly those of fault management and customer complaint handling. China Mobile’s aim was to come as close as possible to eliminating manual intervention in operations and maintenance (O&M) by reducing the number of site visits by engineers and technicians to one per ticket.
In doing so the company saw an opportunity to ensure rapid service activation. “The complete IP backhaul activation process consisted of 54 steps, 41 of which required manual involvement, resulting in prolonged response times, increased error probability and elevated operational expenses,” says Yao Yuan, Project Manager of Autonomous Networks, China Mobile.
Just as importantly it wanted to be able to identify and address faults before they impacted customers. When it came to IP backhaul fault monitoring and resolution alone, in 2023 China Mobile processed 30,000 fault tickets each month in Guangdong, with backend O&M experts spending an average of 45 minutes on each one. And since those backend O&M experts needed to interact with personnel on-site, each ticket typically took between 30 to 100 minutes to resolve.
The company’s answer was to develop an end-to-end, highly autonomous NOC – also known as a dark NOC – where the need for human oversight of critical functions is eliminated. It did so using a range of TM Forum assets and its success won the company the TM Forum Excellence Award for Autonomous Networks.
China Mobile tested network automation solutions in two provinces before national rollout: Guangdong, which is China’s most populated province with more than 126 million inhabitants, and Zhejiang province, which has a population of approximately 65 million. In China Mobile’s Guangdong subsidiary, the focus was on dealing with IP backhaul fault management and home broadband customer complaint handling. Teams in the Zhejiang subsidiary, meanwhile, prioritized RAN fault management and core network fault management.
The company developed two different types of AI agents: role-based copilots and scenario-based agents. These use natural language and intelligent knowledge Q&A to work on fault management and customer complaint handling across radio access, core networks, IP backhaul and home broadband.
Agents live or die by the accuracy of the AI that supports them. Keen to limit inaccuracies and GenAI hallucinations, while at the same time automating the use and upkeep of knowledge graphs, China Mobile together with Huawei built a large AI model with more than 10 billion parameters and specialized algorithms. It also developed autonomous closed-loop operation toolchains that draw on collaboration between large models, which offer the benefit of generalization, and small AI models, which bring specialization.
Other considerations included variations in business models and data patterns between China Mobile’s provincial subsidiaries, which required teams to adapt and train telecoms GenAI / AI models for big data analysis using local data. When it came to the next step of scaling solutions nationally, China Mobile “mapped out a comprehensive view of its operation and maintenance scenarios and established an evaluation system”, according to Yuan. Focusing on a high-value scenario, it “selected provincial subsidiaries and collaborated with equipment, network and OSS vendors to implement a bottom-up deployment practice. This approach allows for rapid replication and promotion from the city level to the provincial level and nationwide,” explains Yuan.
China Mobile also needs to gauge its success. For this, it uses TM Forum’s framework of key business value indicators (KBI), key effectiveness indicators (KEI) and key capability indicators (KCI) as a basis for measuring quality, efficiency and effectiveness, as shown in the graphic below.
China Mobile says developing ‘intelligent Q&A’ and ‘intelligent analysis’ for real-time queries has helped it improve:
Both Guangdong Mobile and Zhejiang Mobile report improvements to fault management and reduced MTTR (Mean Time to Repair), which in turn has enhanced customer experience. In addition, the company reduced operational manpower by the equivalent of a total of 5,500, people per year. Around 75% of the manpower reductions were made in network operations, with customer operations accounting for the other 25% and represented approximately 10% of the total workforce in these areas.
China Mobile made extensive use of TM Forum assets throughout. Indeed, “without TM Forum assets, the entire process would have been significantly longer, if not impossible,” according to Yao Yuan.
For example, China Mobile deployed GB921 Business Process Framework (eTOM) to define the scope of the work in partnership with Huawei. It then drew on GB1022 ODA Functional Framework for a clear set of functions for defining business requirements, as well as principles for decoupling and integration that simplified management and operations.
The AN Journey methodology defined in IG1218F AN Framework, has helped China Mobile move through assessment, gap analysis, solution design, and implementation, as it builds out a dark NOC.
During the implementation stage, TM Forum’s process framework (eTOM) served as a common language for China Mobile, Huawei, and other parties, unifying the understanding of business requirements and processes.
China Mobile and Huawei have also fed what they learnt from building a dark NOC back into TM Forum standards. This includes contributing an Agent and Co-pilot design to ‘IG1345 Embracing Generative AI in Telecom’, as well as:
Guangdong Mobile
IP backhaul faults:
Home broadband complaint handling:
Zhejiang Mobile
During 2023, Zhejiang rolled out RAN fault management in Hangzhou, the capital city of Zhejiang province. This initial deployment resulted in more than 100 maintenance engineers using the RAN fault management system daily, closing more than 8,000 fault tickets per month across approximately 20,000 sites and over 100,000 cells in the city. Since then, Zhejiang Mobile has achieved the following results across the province:
RAN fault management in the city of Hangzhou:
Core network fault / complaint handling in the province of Zhejiang
Guangdong Mobile
In Guandong, network automation has delivered improvements in:
After over a year of deployment and operation in the field, China Mobile has started replicating its network automation solutions nationwide.