Duplicate CSV rows are not always repeated records. Two people can have the same name. One customer can place several orders. A repeated email address can represent a shared inbox. Removing rows based on one familiar-looking column can erase legitimate information.
Before cleaning a file, write down what one row represents. That sentence determines which duplicate test is useful.
Start with the row's meaning
For a sample orders file, one row might represent an order line rather than an entire order. The order ID would then repeat whenever the order contains several products. Treating that repeated ID as an error would be the wrong rule.
A full-row duplicate is a different case: every compared field is identical. Even then, check whether the source intentionally records repeated events. A repeated measurement or transaction can be meaningful if its occurrence is not uniquely represented elsewhere.
| Row represents | A possible identity question |
|---|---|
| One customer | Which stable customer key does the source provide? |
| One order line | Do order ID and line number together identify it? |
| One event | Is there an event ID, or only repeated-looking values? |
These questions help define a rule. They do not prove that the suggested fields are unique in your file.
Separate duplicates from conflicts
Two rows with the same key and the same values may be redundant. Two rows with the same key and different addresses require a decision. Deleting one arbitrarily can keep the outdated record.
Choose a survivor rule only when the source supports it. “Keep the latest update” requires a reliable update field and a clear interpretation of its timestamp. “Keep the first row” merely depends on file order unless that order has a documented meaning.
Data Quality Rule Checker can test supported uniqueness and other declared rules. It reports failures against those rules; it does not decide whether two customers are actually the same person. Its unique rule compares exact cell text and ignores empty cells, so add a required rule when blanks also count as errors.
Review removals before replacing the dataset
CSV Cleaner offers Remove duplicate rows, which starts unchecked. When selected, it compares every cell after the earlier cleanup steps and keeps the first matching row. Trimming surrounding spaces is on by default; optional letter-case changes can also make previously different rows match. Review these choices before removing anything. The tool does not select a survivor by a key or by the latest timestamp.
If that full-row comparison does not express your intended rule, use a process that does. Keep the source and review the removed or conflicting candidates where your workflow allows it. Compare counts, but also inspect representative records. A smaller file is not evidence of a better dataset.
For the orders example, preserving repeated order IDs may be exactly right. The useful result is a file whose rows still represent the intended entities or events, with a documented reason for every class of removal.