At TM Forum’s Innovation InFocus in Texas this week, Frank Massoudian, Principal Architect, R&D System Engineering, and Edward Pershwitz, Principal R&D Tools Infrastructure Architect, both of Huawei will present the Catalyst project they led to help achieve test case standardization across the value fabric. Their joint article gives you some insight before their talk.
In virtualized/hybrid networks, operators are developing solutions alongside multiple vendors (a combination we call the value fabric) using ever-evolving standards. In this environment, verifying and integrating solutions is challenging and can involve:
- duplication of work – leads to a lengthy time to market;
- multiple suppliers – whereas in the past operators had to deal with two to three suppliers, now there are many making integration more complex;
- standards are still maturing – can’t use simulations for testing; and
- limited available resources – test activities must contend with other business needs.
To do things once, and efficiently, we need common test cases to determine whether solutions work correctly. This includes a common framework for how test cases are written, and a common way of defining the production environment.
The Joint Agile Delivery (JAD) Catalyst was developed by TM Forum Members to address this need. Catalysts are projects, developed collaboratively by TM Forum members using the Forum’s best practices and standards (ZOOM in this case), to create innovate solutions to common challenges. Orange, AT&T, and Telecom Italia sponsor this particular Catalyst, while Huawei works with fellow participants Spirent, Infosys, Tech Mahindra and Neural Technologies to design the ecosystem needed for standardizing tests.
We specify the following building blocks for standardizing tests:
- a standardized test model;
- a standardized language, including concrete and abstract syntax;
- an execution engine(s) (a type of software tester);
- a test case framework;
- a test execution platform; and
- an integrated development environment (IDE).
Test case elements
A standardized test case model describes a set of elements that can be combined to design, maintain and execute the test cases. Where there are multiple tests, many of these elements can be reused, so it’s important to design and maintain them as separate entities. The following five test case elements are part of the standardized test case domain model:
Resources – each test case has a set of resources it needs to communicate with the system being tested. The test case is required to declare all abstract resources (the concepts behind how something works) it needs to use. Once declared, abstract resources are turned into instances (for example, data structure, function, method) through the framework.
Execution flow describes the steps of involved in a test case and the order they’re executed in.
Higher-level functions implement actions of the test case that are commonly repeated, but outside the main focus of the test. These actions hide low-level messaging and allow the test case developer to pick and choose the level of granularity for different parts of the execution flow.
Test data – when executing a test case, using data not defined in the execution flow is necessary. This includes resource-specific data, such as resource attributes (used to provide resource information) provided through the resource API; and more general test data, such as defaults that are provided as a separate test data model.
Test case environment – we can separate test case resources into roughly two groups: those that must be allocated for exclusive use by individual test cases, and those that a group of test cases can share.
If a test activity requires multiple test cases to be executed using a set of shared resources, they need to be allocated and provisioned before any of the test cases can run, as we cannot use these shared resources at the same time. Different test cases use them for the duration of their execution and then release them for other test cases within the same test activity to use. For example, mobile devices registered to the wireless network are shared as part of the activity, but can only be allocated to individual test cases.
Both sets of resources are required for test activities to be executed. Together, they form the test case environment.
Test suites and call models
In a typical test scenario, test cases are not executed by themselves. They run as a part of either a test suite or a call model. Test suites aggregate test cases that verify a particular set of functional requirements. We can execute test cases in a test suite once we consider success and failure per test case.
Call models aggregate test cases that verify non-functional requirements, such as performance, robustness, etc. Test cases in a call model execute repeatedly, each at their own rate. Together, they emulate real network traffic where multiple activities happen at the same time. Successes and failures go by how the entire call model is executed; a certain number of individual test case failures are expected.
Test standardization is key to reducing duplication and to using resources efficiently by: reusing test cases, sharing them across the value fabric and clearly communicating of the intents and results of test cases.