Master data diagnosis: the signs your company needs data cleansing

A master data diagnosis is the process of measuring, with criteria and not by impression, the real state of a company's master data base: how many customer, supplier and material records are duplicated, how many have a mandatory field left blank, how many are outside the descriptive standard or have the wrong tax classification. It is the first step before any data cleansing project, because without measuring you can't know the size of the problem, and without knowing the size of the problem you can't justify the fix to leadership.

In practice, it is rare for a company to ask for a master data diagnosis on its own initiative. What usually happens is that the problem shows up first as a symptom somewhere else: a duplicate purchase, a report that doesn't reconcile, a supplier approved twice under similar CNPJs (Brazil's company registration numbers). Only then does someone ask: is this just an isolated case, or is the entire master data like this?

The signs that the company's master data needs cleansing

Anyone who has taken part in an SAP project or an MDM implementation recognizes the signs quickly. The first is repeat purchasing: the same material being bought under different descriptions, from different suppliers, because nobody looks at the master data before creating a new code. The second is the report that doesn't match: procurement says there are three hundred active suppliers, finance says two hundred and eighty, and nobody knows which number is right, because each area applies a different filter to the same base full of inconsistencies.

Another classic sign is supplier approval time growing for no apparent reason, because the team spends more energy checking whether that supplier already exists in the system under another name than evaluating the supplier itself. A new ERP, such as SAP S/4HANA, being implemented on top of a base that was never cleaned also counts as a sign, carrying into the new system the same duplication and the same classification errors that existed in the old one, only now with a much larger investment at stake.

And there is the quietest sign of all, which appears when the company starts using artificial intelligence or BI more seriously. AI doesn't question the master data, it learns from what is there. If the material master has three different codes for the same screw, the model will treat them as three different items, and the consumption, inventory or purchase forecast analysis comes out wrong without anyone noticing where the error came from.

Why waiting to cleanse is expensive

The cost of putting off cleansing is not visible day to day, and that is exactly why the problem grows. It shows up scattered: a bit in idle stock, a bit in extra tax paid because of a wrong tax classification, a bit in analyst hours spent manually fixing what the system should have blocked at entry. Industry studies, such as those cited by Gartner, estimate that poor-quality data costs a mid-size to large company close to thirteen million dollars a year on average, counting rework, wrong decisions and lost operational efficiency. The exact number varies, but the direction is always the same: bad data doesn't show up as a single line on the balance sheet, it spreads across several areas at once.

The point few people notice is that the cost of cleansing doesn't go away if the company does nothing, it just changes form. It becomes manual rework, decisions made in the dark, audits that take longer, and eventually the ERP project that slips because data migration became the bottleneck in the schedule.

Who is responsible for identifying and acting on the master data

Identifying that the master data needs cleansing is not the job of a single area. In practice, it is usually procurement or supply chain that feels the problem first, because they deal with duplicate suppliers and poorly classified materials every day. Finance feels it at close, when numbers don't match between systems or areas. IT feels it when it needs to migrate or integrate systems and discovers the base isn't in any condition to be moved without cleansing first.

Whoever should take the decision to act, however, is the data owner of the affected domain, with the support of an executive sponsor, because fixing master data at scale requires budget, time and, usually, accepting to temporarily block some processes to put the house in order. Without that sponsorship, the diagnosis becomes an interesting report that gets shelved.

How a master data diagnosis and cleansing works in practice

The path most used in the market follows a similar logic, regardless of the tool chosen. First comes the diagnosis, measuring duplication, inconsistency and empty fields on the company's real base, not on a sample. Then comes standardization, applying taxonomy and descriptive rules so everyone names the same thing the same way. Next, deduplication, merging repeated records into a single golden record per customer, supplier or material. Then validation and enrichment, cross-checking the records against external sources to fill in what is missing and fix what is wrong. And lastly, and perhaps the most forgotten step, continuous governance, with rules at system entry and an approval workflow, so the clean base doesn't get dirty again within a few months.

That last point is what separates cleansing that lasts from a one-off rework. Cleaning once without changing the entry process is like tidying the house without fixing the leaking faucet. For those who want to better understand how this process works in practice, including a free initial diagnosis to measure the real situation of the material, supplier or customer master, 4MDG details the full methodology for [master data cleansing](https://4mdg.com.br/pt/saneamento-de-cadastro).

Which tools are used in master data cleansing projects

Technology alone doesn't solve it, but it speeds up and sustains what the process defines. MDM platforms such as SAP Master Data Governance, Informatica, Stibo Systems, Reltio, Semarchy and Profisee usually automate a good part of the deduplication and the application of standardization rules, especially when natively integrated with ERPs such as SAP, TOTVS and Oracle. For the specific part of data quality, tools such as Ataccama and Informatica Data Quality continuously measure whether the base is within the defined standard, raising alerts before the problem grows again. In parallel, the use of artificial intelligence to cross-check records against external bases and suggest duplicate matches is becoming standard in the most mature projects, reducing the manual work of those who used to compare line by line in a spreadsheet.

How to learn about data quality and master data cleansing

Anyone who wants to specialize in the subject will find a good theoretical foundation in the DAMA-DMBOK, which treats data quality as one of the central areas of the data management discipline, and can pursue the CDMP certification to formalize that knowledge. But the part that really teaches the most is practice: taking part in a real data cleansing project, even a small one, quickly shows why a deduplication rule that looks simple in theory needs fine-tuning when applied to a real base, full of exceptions and history accumulated over years.

Career opportunities in data quality and cleansing

Professionals who know how to run a master data diagnosis, understand the causes of duplication and propose a correction plan are among the most sought after today in data governance, because it is knowledge that combines technical command with an understanding of the business process. This applies to those starting out as master data analysts who want to grow into data stewards, and also to BI and data professionals who want to stop only consuming the base and start influencing its quality before the problem reaches the report.

Main questions about master data diagnosis and data cleansing

How long does a master data diagnosis take? It depends on the volume of records and the number of domains assessed, but an initial diagnosis, done with a suitable tool, usually takes a few weeks to show the real size of the duplication and inconsistency problem.

Is master data cleansing the same as data migration? No. Migration moves the data from one system to another. Cleansing fixes the data itself, and should happen before the migration, so the old error isn't simply carried over into the new system.

Does cleansing have to be done all at once, across the whole base? Not necessarily. Many companies start with a more critical domain, such as suppliers or higher-turnover materials, and expand later, which reduces risk and shows results faster to sustain investment in the next phases.

Once cleansed, does the base stay clean on its own? No, unless the company implements quality rules at system entry and an ongoing governance process. Without that, duplication and master data errors start growing again within a few months.

MDM Academy: deepen your knowledge of master data

MDM Academy is the data governance and master data school of 4MDG, created to train professionals who understand not only data quality theory but how to apply it day to day in companies that use SAP, TOTVS, Oracle and other ERPs. There, topics such as master data diagnosis, deduplication, standardization and continuous governance are covered with real examples from procurement, supply chain and master data, to train professionals able to run this kind of project in practice, not just describe it.

Those who want to go deeper can get to know MDM Academy or join the Data Club, the community that brings together master data and governance professionals swapping experience on the same problems discussed here.

Conclusion

Master data that accumulates duplication and inconsistency doesn't warn you before it turns into losses, it just shows up gradually, disguised as repeat purchasing, reports that don't match and system projects that slip. The difference between companies that suffer from this for years and companies that fix it is almost always the same attitude: measure before acting, with a real diagnosis, instead of continuing to treat an isolated symptom as if it were an exception.

In your company, has anyone stopped to measure how many duplicate records exist today in the customer, supplier or material master data?