Globe Telecom takes service assurance to the public cloud

• Who: Globe Telecom and MYCOM OSI
• What: Implemented cloud-native service assurance suite and migrated the mission-critical capability to the public cloud - a first in Southeast Asia, while also evolving use cases and processes from network performance to service quality management
• How: Leveraged TM Forum’s Open Digital Architecture, Information Framework and Open APIs, part of the Open Digital Framework
• Results: Reduced routine performance data extraction and analysis time and effort by 70%; eliminated all manual activity for routine report generation; root-cause  identification efforts reduced to near-zero; improved availability and performance

One of the challenges of service assurance is being able to proactively spot the network issues that impact customer experience. To solve this challenge, Philippines’ mobile network operator Globe Telecom and its vendor partner MYCOM OSI leveraged the power of the public cloud and shifted focus from network performance to service quality – and beyond. Since 2016, Globe Telecom has been on a roll in terms of subscriber growth, averaging 13%+ subscriber net adds. While this is good news for any mobile operator, it does present a challenge in maintaining excellent customer experience while the subscriber base grows rapidly.

Meanwhile, Globe Telecom had established partnerships across multiple consumer and enterprise verticals but lacked the business agility and automation to support its ambitious time-to-value targets. The root problems underlying these service assurance challenges came down to two basic things: As Globe Telecom expanded its coverage and offerings, it realized that maintaining outstanding customer experience required converged, cross-domain network and service management (enabling rapid evolution through DevOps and continuous integration and delivery – CI/CD). This would require integrating systems across its entire network and IT estate, as well as supporting cross-functional use cases across network, service, customer and IT operations. For its consumer and enterprise vertical partnerships, Globe Telecom needed a containerized application architecture supporting DevOps and CI/CD innovation to eliminate infrastructure deployment time and costs, as well as the ability to leverage rapidly evolving open source technologies, standards and APIs. To that end, Globe Telecom has been working with MYCOM OSI since 2016 to transform its network and IT operations using MYCOM OSI’s cloud native Experience Assurance and Analytics (EAA) solution.

By the end of 2016, Globe Telecom was able to manage the entire end-to-end network from a single application, with end-to-end service visibility across all components. In 2017, Globe Telecom introduced workflow automation and open loop automation by integrating MYCOM OSI’s EAA ProActor automation application. All of this produced significant system total-cost-of-ownership savings, reducing routine performance data extraction and analysis time and effort by 70% (on average from 6 hours to 2.75 hours), eliminating all manual activity for routine report generation (which took up an average of five hours per report), and supporting the significant increase in data processing requirements that accompanied the rapid subscriber growth, with over 2 billion input data records processed per hour.

1. A proliferation of silos. In 2016, Globe Telecom had more than 10 systems supporting network performance management alone. Report generation could take a week. Identifying and resolving service issue root causes could take days, leading to revenue loss and churn. Identifying and contacting impacted customers was complex and time consuming.
2. A lack of cross-functional support systems supporting ICT convergence, which meant delays in resolving issues spanning functional boundaries.

The automation application also powered automated root-cause analysis use cases and subsequent remediation orchestration, leveraging real-time cross-domain network topology monitoring capabilities to enable faults and other issues to be quickly correlated to their root-causes across all network domains from a single pane of glass. Consequently, mean time to repair (MTTR) for issues that could negatively affect customer experience was significantly reduced, with more issues pro-actively resolved before they had a chance to impact customers.

Shifting to services

However, up to this point, Globe Telecom’s service assurance capabilities were still focused primarily on network performance management. Evolving the solution to support service quality management (SQM) required a different approach. Thus, in 2019, Globe Telecom migrated MYCOM OSI’s EAA application suite to Amazon Web Services (AWS), adding IT network topology and fault data, as well as data lake integration.

The diagram above depicts the implementation strategy for end-to-end service monitoring. The introduction of zero-touch root cause analysis using advanced analytics reduced root cause identification efforts to near-zero. The solution also automated identification of root causes – even when they originate in IT systems – and of remediation, so the operator can avoid a scenario with multi-hour outages that would, of course, lead to revenue loss and churn. Globe Telecom was able to leverage a number of TM Forum standards to accomplish this, not least because MYCOM OSI’s solution is compliant with those standards, including TM Forum’s Open Digital Architecture (ODA), an industry-agreed blueprint which sets a new vision for operational and business support systems (OSS/BSS), and acts as a de facto standard for the design of open digital platforms.

Globe Telecom also leveraged several TM Forum Open APIs to support system integration and service management, including: To hear how MYCOM OSI incorporates Open APIs read the My API Story: Assuring the networks of tomorrow Globe Telecom was also able to leverage ODA and TM Forum’s Zero-Touch Orchestration, Operations and Management (ZOOM) Project for OSS/BSS, as MYCOM OSI’s EAA platform is containerized to incorporate ODA-aligned componentization. This supported rapid expansion of IT components in service monitoring and management. The EAA platform also includes the Unified Network Inventory and Topology (UNIT) product, which provides real-time dynamic inventory and catalog-driven modelling aligned to the TM Forum Information Framework (SID). The framework provides standard definitions for all the information that flows through the enterprise and between service providers and their business partners.

Addressing the challenges The decision to move Globe Telecom’s service assurance to AWS might surprise people who still think of the public cloud as being a risky alternative to private clouds. While big names like Amazon Web Services, Microsoft Azure and Google Cloud Platform have a good track record in offering public cloud services and SaaS, some CIOs still get a bit nervous about security when it comes to public cloud versus a private cloud they can control themselves. But this is changing and more CSPs are migrating at least parts of their OSS/BSS to the public cloud simply because the benefits are too attractive to resist.

• Trouble Ticket Management API for the creation of trouble-tickets for workflow automation
• Performance Management API for the dynamic discovery of performance metrics, ultimately supporting root cause analytics
• Alarm Management API for the ingestion of fault management (alarm) data from external systems
• Service Problem API for the integration with service problem management systems

Another benefit of using the public cloud is that it supports Globe Telecom’s DevOps processes. EAA is a cloud native system comprised of microservices, so it’s easier for Globe Telecom to work together with MYCOM OSI and come up with new solutions and use cases. Adopting DevOps innovation reduced capability delivery times from months to weeks. That’s not to say the public cloud doesn’t present security issues that need to be resolved – but they are resolvable. Globe Telecom, MYCOM OSI and AWS tackled this issue in the design stage by defining a penetration testing regime that covers system and operational security testing, alongside using features such as direct connect and dedicated virtual private cloud architecture.

Putting Globe Telecom’s service assurance capabilities in the public cloud presented other challenges such as latency, although in this case it was more a matter of geography: The nearest AWS availability zone for Globe Telecom is Singapore, which is 2,000 km away. This meant that latency-critical functions would need to run closer to the network. Consequently, Globe Telecom adopted a hybrid cloud approach with certain service assurance functions, in particular data collection, running locally. “Some of the service assurance workload is done in the network with localized data collectors in our own private cloud,” Howsare explains. “Then that data is pushed into the public cloud, where it’s then analyzed by the EAA solution.” A third challenge relates to scalability. Rapidly growing CSPs such as Globe Telecom struggle to even predict the resource requirements over the next 12 months, let alone five years. To solve this, Globe Telecom, MYCOM OSI and AWS built a deeply integrated infrastructure orchestration capability that auto-scales compute, storage and connectivity capacity across three AWS data centers, enabling the system to provide up to 99.99% availability.

Beyond service assurance

Meanwhile, Globe Telecom has plans for 2020 to add new capabilities that have just been introduced for MYCOM OSI’s EAA platform, including QoS-driven orchestration, virtual network function cloud monitoring, real-time performance monitoring and closed loop automation. But perhaps the most interesting upcoming feature is AI and machine learning-driven predictive analytics that will enable predictive service assurance. This effectively means predicting when a fault is going to happen in the future, predicting when a KPI threshold is going to be crossed in the near future, and then preemptively resolving the root cause of that fault, or that threshold crossing, before it impacts the customer.

Hear Dirk Michel, Senior VP Cloud Business & Solutions Architecture, discuss the launch of the AWS-hosted Assurance Cloud which is used at Globe Telecom.