Why is Data Lineage important?
The number (and diversity) of data sources, the number of data items, and the complexity of applications make Data Lineage analysis a daunting challenge. However, without data lineage, every department within an organisation is essentially an independent silo with little visibility of each other’s metadata, processes and data definitions. It is commonplace, that the same data is digested and interpreted in many ways (and ultimately reported) diversely between departments of the same organisation!
If you think about the millions of bits of data which enter and flow throughout an organisation daily, it would be an exhaustive task to track and manage these differences between departments when an issue is identified. In fact, it would almost be impossible to identify such issues, unless an overarching department exists which oversees the data governance across all departments of the organisation.
A major side effect of poor data lineage is the inability for an organisation to quickly act and respond when situations such as; new business opportunities arise, responding to competitor actions in the market or (perhaps even more scarier) responding to a regulatory demand.
In many instances, the outcomes of poor data governance are loss in competitive advantage, hefty compliance fines and reputational impact.
Data Lineage looks to answer the critical questions that are required by effective data management. Where did the data originate? When was it captured? Who is using the data? Why is it meaningful? What are the data points it has passed through? How has the data changed as it passed through those points? And,
importantly, is the quality of the reported data to be trusted?
To answer these questions one must be able to trace up and down the data stream, from one data point to another without gaps, and be able to properly trace any errors back to their source and ensure reported
data is sufficiently accurate and complete in order to be in compliance.
This is especially relevant for the finance industry where compliance has been the main trigger for innovation in the traditional approach to data lineage in banking.
What is the traditional approach?
Lets take the example of a bank where the volume of critical data elements runs into the trillions. Banks simply cannot identify every single piece of data for reporting, and instead must identify the most important
data to follow through their systems.
The benefit of data lineage in such a scenario is straightforward: it simplifies the complexity of vast amounts of information and provides visibility of the data’s flow through various data points from source
to destination for governance, compliance and business impact.
But exacerbating data lineage is the issue of technology. For example, in every bank or large financial institution there is so much different technology that the data interacts with – various platforms and applications. Often, these technologies do not talk to each other because they do not speak “the same language,” which means someone needs to manually connect the dots, often using multiple spreadsheets.
Nixora and ASG Technologies step in
Nixora and ASG Technologies have taken a somewhat different approach, which is an automated approach that maps data lineage in an information system and detects movement of information from one data store to
another. The tool builds a knowledge base by scanning metadata and reconciles that metadata into an integrated information model that supports full forwards and backward navigation.
ASG has developed over 220 scanners for different technologies that allow the technologies to talk to each other. Which means data lineage is no longer a matter of manually identifying important data and following
it through each of the systems. They have instead mapped the technologies so that all of the data contained within those technologies is connected up automatically.
What are the benefits?
All stakeholders in your organisation need to know that reports accurately reflect the data. If there is a discrepancy and, lets say, there is a request for information on its source, and how it has moved and
changed through the organisation, such an investigation could take months.
Now, with ASG’s Enterprise Data Intelligence solution, if a problem exists and no matter where it exists, it takes only hours to identify the problem and find its source because everything has been automatically
tracked through the flow of people and processes.
Moreover, if your organisation would like to build a new product, such as cross-selling life insurance and banking, your campaign need not start on the ground floor. Because you already know where to obtain the
information and have everything you need to know about your customers, you can better understand and make quicker decisions about where you can cross-sell, or where not to.
Make no mistake about it! ASG’s data lineage is revolutionizing the way banks and large financ