Digital data needs new consistency models

This is the first of a two-part series on using digital technologies to transform your business. Is there anything that is not digital today? Just about everything now has ‘digital’ monikers – be they names of businesses, products, services or even organization titles. So why leave data behind? Data has thus far been labeled as ‘big’. How about calling it ’digital data’? Do you ever wonder why YouTube never seems to run out of space? Or why Facebook never seems to go down for an upgrade, yet constantly churns out new features? How do pages on Amazon.com load at the snap of a finger, despite having millions of products? While there’s no doubt that these organizations run stellar software and are engineering marvels in their own right, the data they hold, is not always consistent. Don’t believe me? Try this – pull up your Facebook or LinkedIn newsfeed from a browser on your laptop, and then from a browser on your phone. Very likely, they’re not in the same order. What’s more, some of the articles appearing on one device may not show up on the other. But fear not – your data will be eventually consistent!

The price of strict consistency

A highly responsive digital channel with page load times of under a second is one of the key determinants of user engagement. This, in turn, has a direct impact on an organization’s top and bottom lines and on the success or failure of any digital strategy. Achieving a highly responsive system entails segmenting your data and realizing that not all data is alike. Our industry is highly attuned to treating every piece of information like an ATM transaction – it has to be strictly consistent end to end, right from the user interface all the way down to the systems of record. Should an API call fail somewhere in the chain of transactions, we abort the entire transaction.

Such strict consistency comes at a price – it impacts performance. It is incredibly common to see chains of API calls spanning several stacks of systems and myriad technologies. And the longer the chain of APIs, the more the likelihood of failure and the slower the overall response times will be.

Digital transformation is often defined as the reinvention of an organization through the use of digital technology. So one may infer that the use of a certain category of technologies should be considered as being truly digital. Enter distributed computing. Right from cloud to Hadoop, distributed systems are a suite of technologies that are highly resilient, self-healing and can achieve massive scale in a highly automated manner, through horizontal scaling. More so, these systems are geared to deliver an ’always-on’ and highly responsive user experience. However, this requires a trade-off between data consistency and performance.

Relax and embrace change

Newer data consistency models are geared towards performance. Strict consistency can be relaxed by the causal and eventual consistency models, each decreasing in order of strictness, but with a greater performance payoff in turn, respectively. The data is considered as causally consistent, as long as the causal relationship between pieces of information is maintained, for example, the comments within a newsfeed must be in order, since they’re usually conversations between people and are causally related. However, the order of newsfeeds themselves don’t really matter, since each concerns a separate topic. These will be eventually consistent, much like the Facebook or LinkedIn newsfeeds. Understanding and embracing these consistency models is the first step towards achieving a truly always-on and highly responsive digital experience. All too often, we tend to get stuck, mandating that every piece of information needs to be strictly consistent end to end. What if you could relax the consistency needs and, instead, give users the illusion that the data is always consistent? How does one embrace eventual consistency and deliver a digital experience atop legacy systems? I’ll address these questions and how to define ‘digital technologies’ in part two of the series. Stay tuned.