Author name: Anindya Sharma

Data is a competitive edge, only if we can harness knowledge from it. Successful businesses will drive decision-making based on the trends, anomalies, variances they see in the data. Unfortunately, many companies may not even know the underlying currents and hidden trends in data, which may be critical to their businesses. This results in missed opportunities or expensive mistakes.   According to the Harvard Business Review, in 1958, corporations listed in the S&P 500 had an average stay of 61 years. By 1980, numbers had declined sharply to 25 years. In 2011, the average tenure dropped to 18 years. Failure to understand and act on hidden signals regarding the business can be catastrophic. Data is at the heart of digital disruption, and it is only accelerating at an unprecedented pace.   Businesses of all sizes need to measure the “as is” (the current state of affairs) before making future changes. There needs to be an agile approach to data harmonization and analysis at speed to understand data signals. Existing cloud data infrastructure makes it easy to start experimenting with data at a low start-up cost. Agile data operations integrated into continuous integration and continuous delivery DevOps methodology facilitate businesses adapting to changing external factors in real-time without expensive rework.   Thus, investments into cloud data infrastructure such as Microsoft Azure is key to uncovering the “known unknowns” and discovering the “unknown unknowns” in the data. Net net, Investments into the cloud data infrastructure increases chances of success in a highly competitive and dynamic business climate of the new digital world.

On average, a typical mid-size enterprise has hundreds of applications, databases, and unstructured file sources. A consumer-facing application may also have thousands to millions of end devices (IOTs). The data residing in such applications create ubiquitous data islands that do not capture a complete business view. To get a 360 idea of the business, enterprises need to combine data from various sources into a harmonized structure. Often companies create data warehouses, data marts, etc., to integrate the applications/databases to get this complete picture. While these traditional approaches lead to incremental value for the business, they are too slow and lose relevance over time. While the need to harmonize data still exists, the techniques for traditional ETL are no longer adequate. Businesses need to look into modern methods of ingesting data in raw form and defining analysis techniques at run time. In other words, traditional techniques of ETL put too much burden on moving extensive data through the network while performing computations, thus slowing it down significantly. In the new world, data is ingested as is (ELT), and compute performed closer to data, thus resulting in faster analysis at scale.   Enterprises can break down these silos of data or applications by adopting new data integration techniques deployed on on-demand, unlimited compute on scalable cloud platforms.

The proliferation of applications (legacy, cloud), IOTs, databases, unstructured files (images, videos, etc.) has created an enormous gap between data creation and the ability to value out of that data. Hence the enterprises started to store data on low-cost data lake storage. Without a proper way to catalog, clean, and prep such data, it is impossible to get value out of creating such storage. Some teams embark on hiring big teams to curate these vast piles of data using various data governance tools, but those efforts continue to fall short. Thus the swamp of structured and unstructured data continues to get bigger and bigger. This results in a huge execution gap between data creation and data analysis.   There need to be automated and intelligent ways to label, clean, and curate data, hence drain the data swamp. Data cleansing and governance tools should offer and employ rule-based artificial intelligence techniques for data cleansing and curation. The good news is that with the advent of cloud technologies, access to sophisticated machine learning algorithms is getting easy with very little upfront investment. The enterprise data governance team can experiment with various machine learning techniques (supervised or unsupervised learning) to bridge the execution gap and drain the data swamp.