Six years ago, 13 of the world’s largest communications service providers (CSPs) joined forces to craft a plan for network functions virtualization (NFV), but since then implementation has been painfully slow, particularly among CSPs that don’t have a greenfield for deployment. As it turns out, testing and integrating virtual network functions (VNFs) from multiple suppliers is extremely difficult, but it’s getting easier thanks to an important ongoing TM Forum Catalyst proof-of-concept project called Joint Agile Delivery (JAD).
The Catalyst team led by Huawei has created a common environment, language and application program interfaces (APIs) for testing, which they are contributing back to TM Forum and plan to contribute to open source through the Open Network Automation Platform (ONAP). Standardization of testing is a critical step toward easing VNF deployment headaches when multiple suppliers are involved.
“The biggest investment in time and money is always in testing – in the resources for the test, the lab equipment and the time of the people involved,” says Frank Massoudian, Principal Architect, R&D System Engineering, Huawei Technologies. “If you can make that more efficient, it’s key.”
That was then
In the old days network equipment suppliers used to sell an entire solution that they tested before delivering it to CSPs. Now CSPs want to buy VNFs from multiple suppliers, and they have to do the integration and testing themselves (or hire a systems integrator to do it). It’s a complex, costly endeavor.
“No single party can test or assure their contribution in isolation,” Massoudian says. “Collaboration is the obvious solution, but it’s only possible if we standardize the processes and use a platform to enable the collaboration; it cannot be a loose collaboration.”
It’s also important for CSPs to retain control of the process, which is why AT&T, Orange and Telecom Italia have been eager to sponsor the project.
“The JAD Catalyst is contributing three important outcomes for us,” says Michel Valette, Tests and Diagnostics Domain Manager, Orange. “First is understanding the behavior of a collaboration platform…to understand the processes between actors. Second is the test model – we need to have standardization of the language and metadata, and testing is a key issue because we need qualification during the development phase and during the assurance process. And last but not least, we need test APIs.”
The third phase of the project, which focuses on building the collaborative test environment and developing the APIs, was demonstrated at TM Forum’s Innovation InFocus in Dallas last week. The first two phases focused on validating and assuring network services (interconnected virtual and/or physical network functions, or service function chains).
Phase 2 of the project won an award at TM Forum Live! for outstanding use of TM Forum assets. In this phase, the team used artificial intelligence and machine learning to predict the behavior of network services and based on that information, either trigger service reconfiguration through closed-loop assurance, or trigger a change request to the VNF supplier through wide-loop assurance.
What exactly is JAD?
JAD is a platform and a set of processes that are being applied using DevOps principles and Agile practices, which include continuous integration and testing. In this case the ‘product owner’ is the CSP, which delivers the requirements for all the parties involved.
“We’re doing work throughout the whole lifecycle of a VNF, so from requirements to development to testing the [network]services through to delivery and even assurance in production,” Massoudian explains. “We have a workflow engine that drives the processes and keeps track of them so that we have standardization in processes, and we have an integrated desktop that’s open to all of the participants so that they can all see what’s happening at any point in time.”
Participants in this phase of the Catalyst include: Huawei, which is the overall integrator and leader of the project; Infosys delivering service assurance; Neural Technologies providing orchestration; and Spirent and Tech Mahindra providing test execution. Neural Technologies is replacing IBM, which provided orchestration in the first two phases of the project.
“The beauty of this ecosystem is that we’ve had companies leave us and others come in to fill their roles, and that shows the openness of the architecture,” Massoudian says. “We are really proud of the fact that we haven’t had to change the platform when participants change.”
This graphic shows how the process works:
- It starts with a request to a VNF supplier, either for a new VNF the CSP wants to onboard or to resolve a problem that’s occurring in an existing VNF.
- The supplier delivers the VNF package containing a descriptor indicating what testing is needed, the test plans for the function, a test environment and all other information needed to onboard it.
- The CSP unpacks the package and analyzes it to see what sort of testing is needed. If it’s a new VNF that needs to be onboarded, it has to go through full acceptance testing.
- The next step is verifying the VNF as part of a network service. The red buttons on the diagram show each location where an open application program interface (API) is needed. The Catalyst team is working on a total of eight APIs, four for test management and four for test execution.
- Once the VNF has been certified as part of a network service, it can move on to the service assurance phase. In this phase, the network service is auto-configured and self-heals where possible. If it can’t heal itself, it sends a request for a change to a VNF supplier and the whole process starts again.
Speaking the same language
As part of the Catalyst project, the team has created a Joint Agile Delivery Language (JADL), a domain-specific language designed to create test cases, APIs and environments.
“Without standardization in this area, it becomes virtually impossible to integrate because things come from multiple vendors, each with their own ways of testing and their own APIs,” says Edward Pershwitz, Principal Architect for R&D Tools Infrastructure, Huawei Technologies. “We’re trying to generalize the domain and then create a language that will decouple a logical representation of the test from the actual concrete VNF or physical network function. Multiple vendors are able to contribute their technology, their resources and test execution platforms running in multiple languages [for example, Java, Tcl or Python], and it is all supported by this architecture.”
Here’s a look at the JADL components:
For the demonstration in Dallas, the team showed the new test management APIs at work. Test execution APIs are still under development and will be demonstrated at TM Forum Live! Asia in Singapore next month. The test management APIs are already available publicly on swaggerhub, an API collaboration community. The next step is contributing them back to the Forum as Open APIs and to open source.
“We want to work with ONAP to align or harmonize the TM Forum approach to test metadata and test language within the ONAP context,” Valette says.
This could happen as part of an additional phase of the Catalyst or as a new project at TM Forum Live! in May, or it could happen through a TM Forum workstream in the Zero-touch Orchestration, Operations and Management (ZOOM) project or the API collaboration team.
Getting ready for a VNF marketplace
The work this Catalyst team is doing lays the foundation for VNF marketplaces, as envisioned by another award winning Catalyst project called Enabling the digital services marketplace with automated onboarding.
“Before you can get anything to a marketplace, you need to certify it, and in order to do that you have to perform testing,” Massoudian explains. “You can’t have all these companies doing their own testing and then bringing it to the operator for more testing.”
Jenny Huang, Lead of OSS/BSS Standards Strategy Group, AT&T, and Co-leader of the ZOOM project, says she is looking forward to the next steps of the JAD project.
“The ability to share and reuse test cases and a test environment, and to be able to compare the results consistently are essential in an ecosystem environment,” she says. “The contributions of the JAD Catalyst of a domain specific test language, a testing metamodel, and testing APIs help to achieve this goal. These could be a real value-add to complex projects such as ONAP and the digital marketplace.”