Deduplication: when to trust the algorithm and when to call a human

Deduplicating means deciding that two records are the same thing in the real world. Part of that decision is mechanical and should be automated without hesitation. Part of it is judgment, and automating it produces the worst possible outcome: silent data loss.

The art is in knowing where the boundary lies.

What the algorithm solves on its own

Deterministic matching: same strong key, normalized. Same CNPJ (Brazil's company registration number), same employee ID, same barcode. There is no doubt to resolve here, and human review only adds cost and delay.

The part that demands care is normalization before the comparison. Strip punctuation, standardize case, remove double spaces, decide what to do with accents. Once that is done, merge automatically and log it.

One exception worth naming: same strong key with incompatible data — the same CNPJ with another company's legal name — is not a duplicate, it is a typo in the key. That case goes to review.

Where fuzzy matching helps

Without a strong key, similarity comparison comes in: distance between strings, known nicknames, common abbreviations, similar addresses.

The result is not a yes or no, it is a score. And that is why the most important design decision is not which algorithm to use, but which two thresholds to adopt.

Above the upper threshold, merge automatically. Below the lower one, ignore. Between the two, a human queue.

How to choose the thresholds

Not by intuition, and not by the vendor's default. Build a sample of a few hundred pairs that you yourself classified as duplicate or not, run the comparison, and see where the errors are.

Set the upper threshold at a point where false positives are practically absent in the sample — a false positive merges two distinct entities, and undoing that is expensive or impossible. The lower threshold can be more tolerant: a false negative leaves a duplicate in the database, which is bad, but reversible.

The asymmetry is deliberate: the two errors do not have the same cost.

When to call a human

A score in the middle band. A strong key with incompatible data. Records with financial or tax activity on both sides — there the merge has an accounting effect and needs someone who will answer for it. And any case where the two records belong to different owning areas: the decision is a business one, not a data one.

The human queue has to be dignified

This is where most projects fail. The queue arrives with ten thousand pairs, no context, no priority order, and nobody works on it.

Sort by impact — value moved, recency. Show the two records side by side with the differences highlighted and the activity history. Offer three actions: merge, choosing the survivor; mark as distinct; escalate.

And record the human's decision back into the sample. That is what lets you recalibrate the thresholds over time, and it is the only way for the queue to shrink instead of grow.

What to never automate

A merge without a reversal trail. Regardless of the score, keep what was there before. At some point — an audit, a lawsuit, a complaint — someone will ask where the old record went, and "the algorithm decided" is not an answer.