Metrics for metadata management

Tejbir Singh

Metrics for metadata management

Hi All, 

We are trying to figure out metrics to monitor metadata management program. In your views, what are the most crucial metrics for metadata management to begin with?

Michelle Knight

RE: Metrics for metadata management
(in response to Tejbir Singh)

In my view, good metadata management lends itself to good training and understanding with the data as well as good data quality. It will vary with company and industry; but, here are some ideas. I would look at the amount of retraining that needs to be done due to misunderstanding concepts in a specified time period. Talk with the company trainers and track how often they use the metadata in training for data related tasks and analysis. Looking at how often data needs to be cleaned and statistics on how well new data is entered may be a good start. Does anyone else have any ideas?

william burkett

RE: Metrics for metadata management
(in response to Michelle Knight)

Personally, I think "metadata metrics" is far too general a term.  Any relevant metrics would depend on what the metadata is about and what the organization is trying to achieve.  

Michelle Knight

RE: Metrics for metadata management
(in response to william burkett)

It can be general but it also can be useful. I agree that it depends on what the organization is trying to achieve, tied to its data strategy.

However, process metadata about data warehouse operations automatically gets collected by many systems.  If there is a way to use metadata management metrics to make this more effective, say for security purposes, then that would be beneficial. See this article for more on process metadata.

Also, metrics can help when implementing and maintaining metadata. The Department of Interior plans on tracking the adoption of metadata tools across the bureau and office. Are these the most crucial metadata management metrics? That will be determined over a period of time.

I would argue without some kind of good objective metadata managment metric data organizations are left with subjective metadata management metric data. With just subjective feedback, it becomes hard to justify spending time and money in managing any kind of metadata because of uncertainty on how it helps the business. This type of metadata management could also get lost if  an organization focuses on only the relevant terms . It would be hard to point to metadata management's value. So, I think figuring out some relevant metadata management metrics, even if it is just when someone logs into the Business Glossary page, would be helpful.

Edited By:
Michelle Knight[All DATAVERSITY Members] @ Oct 04, 2019 - 10:53 AM (America/Pacific)

Tejbir Singh

RE: Metrics for metadata management
(in response to Michelle Knight)

Thanks a lot Michelle for your inputs. 
We have started capturing metrics around two metadata capabilities:
1. Business Glossary: In order to create common and consistent business vocabulary across the enterprise, we have enabled the Data Stewards to use Business Glossary enabled through a centralized tool-set. Key metrics we have started measuring are: 
Number of Business Terms; Number of technical assets associated to business terms
2. Metadata Enrichment: In order to provide proper business context to the data assets across the enterprise, we have provisioned the enrichment of technical assets by adding descriptive metadata e.g. Business Name, Business Description, Security Classification', PII and PCI-DSS flags etc. through a centralized tool set. Key metrics we are monitoring are:
Total data assets cataloged in the tool; %age of data assets described (Business Name and Description is available); % age of data assets enriched (All metadata fields are populated)

These metrics will help us keep the track of progress happening around enriching data assets with proper metadata and also to gauge the success of metadata program adoption across different lines of businesses 

Merrill Albert

RE: Metrics for metadata management
(in response to Tejbir Singh)

The other thing about metrics is you need to make sure that you're not promoting the wrong behaviours.  For instance, I've seen people track the number of terms defined.  What they got was terms defined badly.  There was a rush to get the task done and they didn't care what the definition was as long as they had something to check the box.

Michelle Knight

RE: Metrics for metadata management
(in response to Tejbir Singh)
That sounds good, Tejbir. I like how there is a clear purpose for the metadata program metrics. I like how you are taking advantage of the centralized toolset too.
 
Two points to consider.

 

One is there a general agreement as to what counts as metadata enriching the data assets? For example, how are Business Glossary terms weighted so that not just any term would be added to make the metrics increase?

 

Two, is there a brief audit in place to check how well the current metadata metrics work? In case there needs to be any tweaking or to handle any criticism brought forth.

Thank you so much for sharing, Tejbir. As time permits, let us know how it works for your organization.

Michelle

Michelle Knight

RE: Metrics for metadata management
(in response to Merrill Albert)

Agreed, Merrill. Hopefully with more business case studies we can see examples about what works given that businesses context. Only one way to find out is to start somewhere and see how well it meets the goal as stated while not detracting from necessary business operations.

Tejbir Singh

RE: Metrics for metadata management
(in response to Merrill Albert)

Yes, that's so true to have proper qualitative checks in place. In enterprise tool, we have functionality of crowd-sourcing where users can like or dislike the business terms, descriptive metadata etc. which helps to keep track that the metadata getting enriched is of  proper use and not just a result of pressure to have more numbers.

Tejbir Singh

RE: Metrics for metadata management
(in response to Michelle Knight)

Yes Michelle, 

We have published guidelines around best practices to create Business Terms, enriching descriptive metadata etc. We also have a review process where we assess metadata for each LOBs Critical Data Elements and share our feedback with opportunities to enhance the metadata further.

Michelle Knight

RE: Metrics for metadata management
(in response to Tejbir Singh)

Awesome. Let us know how it goes. There is a lot I and others can learn from you.

Scott Flory

RE: Metrics for metadata management
(in response to Tejbir Singh)

I know this is a little late on the thread, but I just joined the forum and saw this post first.

Another metric I would want to see, to measure the success of metadata, would be reporting/analytics users indicating their level of understanding of the data they are using/seeing.  So, a survey of sorts asking about their understanding of data they have used in the last 2 weeks (or some other period).  You could ask a few key questions to get an idea of how much data they have interacted with, and what level of understanding they felt like they had of it and whether that understanding was supported by metadata, or not.  Surveying a subset of users every few weeks could start to build a picture of the ultimate success of metadata (users understanding their data as it is defined).

Just a thought.  I like the other metrics in terms of usage of the tools and but those are more performance metrics for the people tasked with building up the metadata rather than ways to measure the efficacy of the metadata itself.

Tejbir Singh

RE: Metrics for metadata management
(in response to Scott Flory)

Thanks Scott, that's a great way to gauge the success of Metadata Management program. Crowd sourcing is another way to evaluate the quality of descriptive metadata where users (Stewards + Analytics community) can endorse or provide their comments on the descriptive metadata. This will help to provide proper business context to data assets.

Michelle Knight

RE: Metrics for metadata management
(in response to Merrill Albert)

I agree with Merrill that the metrics need to promote good behaviors not just concentrating on a number. I think metrics need some subjective information as to how well the objective measurements are meeting the metadata management strategy.

Michelle Knight

[login to unmask email]

Freelance Production Assistant

Freelance Data, Technology and Science Writer

Michelle Knight

RE: Metrics for metadata management
(in response to Scott Flory)

That sounds like a great idea for a corporate trainer or development person to do. When I worked for Symbeo, an automated accounts processing firm, the company had a training/brush up form. The larger training goal was stated on the form, by the trained. The trainee completed it by checking the boxes whether or not he/she understood the material and how to apply it. Someone was also there to do an observation (So observing someone at work using the Metadata would be helpful too).

If compounding errors were made then the same form template was used for a brush-up.

All training forms and brush-ups were kept to better suit the overall training objectives of the company.

I think something like this could work as some metrics for metadata management.

Michelle Knight

[login to unmask email]

Freelance Production Assistant

Freelance Data, Technology and Science Writer

Eric Dodson

RE: Metrics for metadata management
(in response to Tejbir Singh)

Going back to an earlier post about key glossary metrics you've started with, like "Number of Business Terms; Number of technical assets associated to business terms," it may help to address the quality vs. quantity concern by distinguishing between published (or final) terms and unpublished terms that have been created, imported, or otherwise under development.

If you combine that with a strong review and approval workflow process, then in theory, you will have ensured the quality, completeness of attributes, and the association of technical assets of the terms that are approved and published.

Then perhaps you could shift your metrics towards Number and % of Business Terms Published vs. Unpublished, and Number of Business Terms Published Last Week (or Last Month), Number of Business Terms that Received Positive/Negative Feedback Last Week, Number of Unpublished Business Terms Needing a Steward to be Assigned, etc.