Virtualization

Dynamic APIs: Negotiating change

Given that network functions virtualization (NFV) has been at least partially motivated by the interest of service providers to act more like software companies, moving away from an infrastructure-based present mode of operations and looking at the concept of ‘dynamic’ application program interfaces (APIs) provides a path to a future mode of operations.

In our recent award-winning Catalyst at TM Forum Live! 2015, we demonstrated the use of dynamic APIs to flexibly communicate real-time product and service information between eco-system partners in order to improve the optimization, resilience and business agility of the end-to-end solution.

In our Catalyst presentation, our CloudNFV represented a management and orchestration (MANO) platform that provided a northbound API based on the TM Forum’s REST Open Digital API standards. The CloudNFV API abstracts NFV and software-defined networking (SDN) complexity, simply exposing a catalog of services and resources to operational and business support systems (OSS/BSS) systems, which were represented by DGIT and Comverse respectively. Watch the video below for a conversation about the Catalyst.

Since change can be anticipated in multi-partner, multi-technology, multi-protocol environments, it was understood that simply agreeing to a static contract and encoding it on both sides of the relationship would be brittle; moreover, it would preclude the type of service velocity and business agility envisioned by the Forum’s Zero-Touch Orchestration and Operations Management (ZOOM) working group and ETSI NFV.

So, building on our previous contribution of a metamodel, we extended the REST Open Digital standard to express a consistent method for communicating change between all the engaged parties. The method involved adding link, metadata and constraint information to the interface specification and thus the API payload. By establishing the method in advance, the parties could anticipate changes and react to them as they occurred, triggering a notification for a human intervention or even an automated response leading to updates and upgrades of services and resources.

Our recommended approach to dynamic APIs requires little change, introduces minimal overhead and does not obligate a partner to be dynamic, however it offers the potential for an early return on NFV and SDN with minimum impact on existing OSS/BSS investments. This concept of dynamic APIs provides a path to a future mode of operations.

IT departments expose functionality via software interfaces to a wide variety of applications, microservices, systems, databases, devices and machines. The use of such APIs has exploded with the growth of distributed computing and is driven by the popularity of the web, cloud and now Internet of Things (IoT). It logically follows that the more we want to interact with a ubiquitous distributed software-defined world, the more interfaces we need.

So what are APIs?

According to Wikipedia, an API expresses a software component in terms of its operations, inputs, outputs, and underlying types. An API defines functionalities that are independent of their respective implementations, which allows definitions and implementations to vary without compromising the interface. A good API makes it easier to develop a program by providing all the building blocks. A programmer then puts the blocks together.

While this description provides a concise encapsulation of the nature of an API and the objective of facilitating composability of software development ‘building blocks’, most of us recognize that the real-world is not quite as plug-and-play as the definition suggests.

APIs: the wild-wild west

There is no singular API standard. The designs of interfaces vary based on how they are specified, communicated, and what they expose and how they expose it. While REST APIs have eclipsed more formal SOAP/WSDL based interfaces (which supplanted CORBA), REST APIs are far from a formalism. Developers have progressively traded up for lighter-weight, less formal protocols for convenience, but the net result is that every API is a snowflake. Each implementation of an interface is unique – APIs are as diverse as the functions they expose.

Wonderful, but how do developers cope if everything that is connected is a ‘one-off’ manual integration. Moreover, how do the resulting composite applications constructed from these building blocks deal with change?

Since all businesses are now digital businesses, we need to advance interface design in order to realize the vision of service-oriented architecture and achieve the vision of truly composable enterprise solutions. We need a pattern for dynamic APIs.

APIs: a business perspective

At its essence an interface is implemented as a contract between two engaged parties that defines the agreement or ‘shared-understanding’ established up front between the provider of the API and its consumers. The API describes concepts and policies, much like a legal contract provides terms and conditions. The provider’s technical implementation details (e.g. underlying data and code objects) are generally hidden; which is why APIs and services are often referred to as black-boxes.

The API only keeps working as long as the shared-understanding is maintained, otherwise the contract is broken. Generally no one willfully seeks to break an API that is in use. However, the technical implementation details naturally evolve over time (i.e. data and code objects are updated, upgraded and replaced), which may require changes to the agreed Interface concepts and policies.

If we step back for a second and consider contracts generally, we know that agreements between engaged parties are routinely amended. Standard legal terms generally anticipate change by specifying a means for amendment. No business person wants to re-negotiate a contract from scratch for every change if it can be avoided – it would require lawyers and be unnecessarily expensive and time consuming – instead the parties focus on managing the specific change at hand.

However, with APIs there are no such standard mechanisms to accommodate change. All concepts and policies are set in advance (static schema) and the back-end technical details are hard-wired (static implementation), which leads to tightly-coupling, and thus a brittle realization of a contract. As a result, when the provider of an API makes a modification the agreement is abruptly broken, even if the change is relatively immaterial to the shared-understanding. The business is interrupted, and now IT is required to manually re-integrate the interface.

APIs: an embarrassment of riches

As organizations become ever more distributed, dynamic and diverse their business processes are becoming increasingly complex composites integrating with engaged parties across a value fabric, but IT simply can’t keep up with the demand for manual API integration (and re-integration), it simply does not scale. This problem is at the heart of the tension between the business and IT.

Services and APIs are only loosely-coupled in conceptual architecture, but not in the real-world where people still have to manually discover and integrate them at the application and process layers. This is the reason why service orientation has yet to deliver service velocity and ‘business Agility’.

From that high-level enterprise architecture perspective (as opposed to the bottom up IT view) it’s not useful or scalable to be stuck manually considering individual APIs. It would be better to have an abstraction, a model for looking at all of these endpoints as objects that expose functionality (nodes), but hide their complexity and automate management of the end-to-end API contract lifecycle (relationships) in order to advance our service orientation architectures.

If legal contracts can anticipate change, when it isn’t even a routine occurrence, we can agree that IT, which is highly-dynamic should be able to anticipate and adapt to new requirements. APIs should be ‘living documents’ so they can safely evolve over time.

Dynamic APIs: a recommendation

Below is a matrix representing EnterpriseWeb’s maturity model for Dynamic APIs (Level 0-3). Level 0 represents the mainstream status quo (fully-manual/disruptive). Levels 1-3 allow for an API to advertise new/modified features in its payload (the body of its messages) so that a consuming system client can be notified of the changes and react to them.

 

EnterpriseWeb Dynamic API Maturity Model
InterfaceInitial ImplementationChange characteristicsNotes
Conventional APIs Level 0Static a priori Contract between partiesTightly-coupled syntactic integrationChange to interface is not programmatically supported, contract is brokenLevel 0 represents conventional API management as broadly practiced.
Dynamic APIs Level 1 (Notification)Static a priori Contract between parties with ability to include: (i) metadataTightly-coupled syntactic integrationChange to interface is not programmatically supported, contract is broken
a) can automate notification of change
Level 1 is a known practice with limited capabilities and implementations.
Dynamic APIs Level 2 (Documentation)Static a priori Contract between parties with ability to include: (i) metadata; and (ii) linksTightly-coupled syntactic integrationChange to interface is not programmatically supported, contract is broken
a) can automate notification of change
b) can facilitate changed mappings between systems
Level 2 builds incrementally on Level 1, but still doesn’t address the fundamental problem.
Dynamic APIs Level 3 (Automation)Static a priori Contract between parties with ability to include: (i) metadata; (ii) links; and (iii) constraintsLoosely-coupled, semantic interoperabilityChange to interface is programmatically supported, contract is adaptable
a) can automate notification of change
b) can facilitate changed mappings between systems
c) can fully automate mappings between systems to maintain synchronization
Level 3 changes the game.

 

Note that the API itself is not dynamic. Dynamism is represented by actual behavior, a property of an actor (human or system). What makes a dynamic API ‘dynamic’ is expanding the scope of its contract to define an interaction-model between the engaged parties that is designed to cope with change (e.g. “on update you can expect this behavior from my system”). This definition should be semantically understood by both human and system clients. In effect, it supports the ability for the actors to ‘negotiate’ change during the lifecycle of the agreement.

Level 1 provides for simple notification for a human intervention (semi-automatic/disruptive). Level 2 provides notification and documentation to improve human intervention (semi-automatic/disruptive). Level 3 adds constraints to dynamically maintain synchronization between systems (fully automatic/non-disruptive).

Dynamic APIs: low cost, high return

While EnterpriseWeb supports the declarative modeling of dynamic APIs and a runtime for using them, dynamic APIs are not a product; they constitute a pattern that decouples interfaces from implementations to support resilient applications and processes.

This separation of concerns is very powerful. Instead of encoding all intelligence into a fixed interface at design-time, which makes it static, non-responsive and brittle, the interface can now describe its change semantics and this allows for dynamic, flexible and adaptable implementations. Interfaces are transformed into living documents, which we refer to as ‘Active Contracts’.

Conceptually, the Active Contract is a promise of a capability; it expresses ‘intent’, with related service level agreements and policies rather than a single explicit/static representation.

We believe that widespread adoption of the dynamic APIs pattern would be a general good, helping every organization to facilitate non-disruptive API evolution, regardless of technology, and providing for a loosely-coupled form of interoperability that supports service velocity and business agility.

We are working within the TM Forum to encourage adoption of these concepts and we are hopeful that members will collaborate to specify a standard and well documented model that will be supported across domains, providing much needed design consistency.



    Advertisement:
    Share.

    About The Author

    Managing Director

    Dave is the founder of EnterpriseWeb (www.enterpriseweb.com). His company offers an award-winning application platform for real-time data-driven processes. EnterpriseWeb delivers transformational solutions worldwide in industries from Telecom to Life Sciences. Dave has made a career of building, growing and turning around companies. Beyond being an entrepreneur, he is an inventor, the author of academic papers on “context-aware” computing, blogger and a regular conference speaker (CloudExpo, SemTech, EDW, GoTo Con, TM Forum, BPMnext, etc). Dave represents EnterpriseWeb in the TM Forum, most notably in the Zero-touch Operations, Orchestration and Management (ZOOM) working group. His company has won two consecutive Catalyst awards at TM Forum Live! for its Carrier Virtualization solution “CloudNFV”. Dave is also a member of the Industrial Internet Consortium and leads the Integration and Orchestration Team, which is providing guidance on next gen software architectures.

    Back to top