Skip to content

Stockertim Dictionary · Artificial intelligence · Technology

Deduplication

Also called de-duplication, dedupe, duplicate removal

In short

Finding and removing repeated records in a data set, after first deciding which fields must match for two rows to count as the same record.

  • Beginner
  • Process
  • Checked

In detail

Deduplication removes rows that describe the same thing more than once. Duplicates creep in when files are merged, when an export is run twice, when a customer signs up with two email addresses or when a system retries a failed upload. Left in, they inflate counts and totals and distort averages.

The key decision comes first: what makes two rows the same record. Exact duplicates match in every column and are easy to find. Harder cases match on some fields but not others, such as the same order number with a different timestamp, or the same person with "St" in one address and "Street" in another. Fuzzy matching can catch near-duplicates but also produces false matches, so its results are reviewed.

An AI assistant can help write the matching rules and the formulas or code that apply them, and can explain why a pair was flagged. The analyst decides which rule fits the question and which copy of a duplicate to keep, such as the most recent or the most complete.

Good practice is to keep the original data untouched, record the rule used and the number of rows removed, and check a sample of removed rows by eye before moving on.

Worked example

An analyst cleans 2,400 customer rows from a merged export. Two rows count as duplicates when the order number and the email address both match.

  1. Duplicates found under the rule: 130
  2. Rows after deduplication: 2,400 - 130 = 2,270
  3. Duplicate share: 130 / 2,400 = 5.42%
  4. A person checks 20 of the 130 removed rows and confirms all 20 were true repeats.

The working file has 2,270 rows, and the rule, the count and the sample check are written in the analysis log.

In practice

Deduplication is routine in customer lists, sales extracts, survey responses and merged reports. Teams run it before counting or totalling anything, and keep the removed rows in a separate tab or file so the step can be undone or audited.

Watch out

  • Two rows that look alike may be genuinely separate, such as two identical orders on the same day.
  • Deleting duplicates in the only copy of the data makes the step impossible to check or reverse.

Learn it properly

Official sources

Facts last checked on 3 November 2026.

Education, not advice: the Stockertim Dictionary explains what terms mean. It never recommends buying or selling anything.

English