Metadata Management Guide

What is Metadata Management?

Metadata, in simple terms, is data about data. Metadata Management refers to managing, maintaining and governing the metadata of an organization. Metadata serves many purposes, but one of the most important is enabling users to search required data  and get additional details about it such as its origin, latency, owner, quality etc. This ability can go a long way in establishing organization agility and ensuring quality insights.

Types of Metadata

Metadata in an organization can be categorized into these three categories:

• Physical/ Technical Metadata – Metadata about the physical assets in your data ecosystem, such as tables, columns, ETL jobs, reports, etc., all form the technical metadata. 
• Business Metadata –This is the logical layer on top of physical metadata. It consists of business terms corresponding to physical data assets. E.g. Loan_ouotstanding can be a field in a table, but what does it mean, how is it calculated, who owns it, etc., represent the business metadata.
• Operational Metadata –These are the transactional details associated with your data assets and include details like when was the last time data was refreshed, its size, hierarchy, current status (active/archived), etc.

How doe it help?

Until a few years ago, data would primarily be consumed by the IT team in an organization. However, today businesses are actively adopting self-service analysis to turn around faster insights. Not only that, increasingly,the data access teams are getting more diverse – Data scientists, data engineers, business analysts all need timely access to reliable data without worrying about its quality, duplicity, latency. 
This is only possible if you have a centralized metadata repository, providing straightforward answers to users about their data sourcing needs.

Mentiond below are some of the benefits of metadata management:

Data quality
Metadata management tools provide details about the datasuch that it can be understood and more readily consumed by the organization. It provides the details of what, who, when, where, why, and how questions about the data assets.
The streamlined approach of creating new and accessing existing data assets results in reduced duplication of data and improved data quality

Efficient Resource Utilization
Data repositories can blow up over the years and incur significant expenses to manage and maintain. Metadata tracking helps monitor the assets and keep track of used or unused and obsolete assets. The knowledge can lead to trimming the data fat and conserving resources

Reduced cost
Data preparation and management can take up to 80% of analytics projects implementation time. The Unified, reliable metadata repository enhanced with business metadata makes it easier to find the data required for undertaking a project. This ease of finding data and reuse of already transformed data cuts down the efforts leading to a reduction in cost.

Improved Organization Agility
Metadata management makes organizations more agile by reducing reliance on IT and other resources. To locate a data asset, a user does not have to call 10 different individuals and can instead just search it through an easy search interface

Components of Metadata Management

Some of the key components of metadata management are as follows:

Metadata Repository/ Data Catalog
Data Catalog or metadata repository is the heart of metadata management.
It can be best explained through the working of a  library catalogue. When you want to borrow a book from the library, you go through the library catalogue to discover whether the book is available, its edition, its publication, its category etc. This helps you decide if it’s the book you were looking for. The library catalogue also tells you the book’s location so that you know how to get it.
Similarly, a data catalogue can help you find a particular data point/field in the vast data repositories across the organization. In addition, it can also help answer the question that you may have about that piece of data, such as –  its size, origin, latency, owner, sensitivity and much more

Business Glossary
A business glossary is a list of business terms and their definitions created and maintained as part of the metadata repository. This ensures the same definitions are used company-wide when analyzing data. 

Data Lineage
Data lineage refers to the data’s ‘line of descent’. In other words,lineage provides the journey of a data point from its source to the destination.This includes all the transformations the data underwent along the way. Data lineage forms the basis for performing Impact analysis which helps you understand the inter-dependency between data assets in the ecosystem,

Impact analysis
Any good business identifies reports, data elements, and end-users affected before implementing a change. Data lineage software helps teams visualize downstream data objects and measure the impact of the change. 
Data lineage lets you see how business users interact with data and how a change would affect them. It helps businesses understand the impact of a particular modification and allows them to decide if they should follow through.

Impact Analysis
Data lineage helps you answer the following questions:

• What was the source for the creation of the given data point?
• What all transformations were done on that point
• Where all is ghi data point being used
• Who was responsible for modification?
• When was the change made?

Metadata Search
A single integrated metadata repository makes it easier to find the data that the user is looking for. The metadata management tools can search by different parameters such as business definition, owner, data source, etc. It often provides a google-like interface, making data search a piece of cake.

Applications of Metadata Management

Create self-service ecosystems 
A centralized metadata repository makes all your data assets easily discoverable. Users looking for data can search their way to the required data rather than contacting multiple people to know where they can find the relevant data. These reduced dependencies help an organization to be more agile and efficient.

Effective Data Management  
Data Sources, warehouses and marts are living breathing systems. They continuously change to align with business requirements. Teams managing this ecosystem need to know how a small change impacts the rest of the objects such as ETL jobs, Reports, other dependent tables, etc. The insight can help them implement all those changes in a planned manner rather than waiting for reports and ETL jobs to fail to figure out the impact. Metadata management solutions, particularly impact analysis capability streamline management of the data ecosystem.

System upgrades and Migration
A metadata management solution helps expedite the data migration and upgrade initiatives. While upgrading or migrating, it’s essential to understand which datasets are relevant and which have become obsolete or redundant over the period of time. Operational metadata reports tell you how the system is being used, what are the duplicate datasets, what data has not been accessed in years and so on, helping the migration team plan the effort more efficiently.

Data governance: 
Different parts of the world have different data protection and privacy requirements e.g. GDPR in Europe. The failure to comply can lead to millions of dollars in penalty. Metadata management solutions help you identify sensitive PII and other data, track its access and usage and create compliance reportsensuring compliance.

Establish Credibility of Warehouse and Marts
There is often a tussle between IT and other functional departments concerning the accuracy of the data in the data warehouse. There may be a situation e.g. where data in the warehouse doesn’t match with the finance records. Metadata management can be used in these situations to track the  Data lineage and justify how the warehouse data has been transformed over the lifecycle, clarifying the confusion

Metadata Management Tools

Alation
Alation is an industry-leading data cataloguing solution. It supports the organization’s self-service needs by providing a single integrated data catalogue. Users can quickly locate the data of interest by leveraging the natural language searchinterface.It has pre-built connectors for a wide variety of data sources the capabililty to build custom connections for lesser-known components.
Alation can monitor and profile data to ensure the highest data accuracy. Additionally, Alation can also offer insights on how people create and share information pulled from their raw data lakes.It has demonstrated impressive results in helping customers improve productivity, save cost and adopt agility making it one of the most sought after cataloguing solution
    
Manta
Manta is a leading automated data lineage solution. It helps you to get a holistic view of your data flow to boost your data governance, achieve DataOps, and carry out Migration smoothly. Manta’s natively supported scanners for database, ETL, reporting and analytics tools, Modelling tools and natively supported languages helps the organization create the most comprehensive data lineage collection. This data lineage can also be fed into third-party metadata management tools such as Alation, Collibra, etc. to enhance data governance and cataloguing.

Collibra
Collibra is a data intelligence platform committed to governance, quality and privacy. It has been named a leader by Gartner for metadata management solutions.Collibrademocratizes access to data by providing an interface to quickly find, understand and access the data they need across all of your data repositories, ETLs, applications, BI and analytics tools. 
Its automated data lineage helps you track the data journey across the multiple available data layers in the organization.