Before removing repeated lines, decide whether case and surrounding spaces belong to the value. AB-12, ab-12 and AB-12 can represent one untidy entry or three distinct identifiers. The right rule depends on the system that owns the list.
Remove Duplicate Lines compares whole lines. It can keep the first copy, keep the last copy, or keep only values that occur once. It does not merge changed records by a customer ID or another field.
Keep a copy and choose the matching rule
Paste one item per line. Leave Ignore letter case off for case-sensitive codes. Turn it on only if the receiving system treats upper- and lowercase spellings as the same value.
Ignore leading and trailing spaces changes the comparison key; it keeps the surviving line's original text. It does not remove every space from the output or collapse spaces inside a line. If the output also needs trimming, do that as a separate, reviewed cleanup.
Blank lines follow the duplicate rule too. Remove empty lines drops empty comparison keys from the survivors. A line containing spaces becomes an empty key when surrounding-space comparison is enabled. Repeated blank separators cannot reliably preserve every group in a deduplicated list.
Work through a small example
Start with these five lines:
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
With both comparison options off, all five lines remain. Ignoring surrounding spaces merges the Ana pair and leaves four lines. Ignoring case as well merges the Bo pair, leaving three.
With Keep the first copy, the result keeps [email protected]. Keep the last copy keeps [email protected]. With Keep only lines that appear once, both repeated groups disappear entirely and only [email protected] remains. That last mode answers a different question from keeping one representative.
One value repeated three times counts as one repeated value and two removed lines when you keep one copy. These totals measure different things.
Review before sorting or importing
Check the input, output and removed-line counts. Inspect a known repeated value and a value that must stay distinct. Text Compare in Lines mode can show the removals against your saved original.
Sort the survivors only if order is irrelevant. Sorting a list first can change which spelling survives a first-copy or last-copy rule. The deduplicator offers sorting; Sort Lines provides a separate step when you want to inspect the cleanup before changing order.
If identical-looking values remain, check tabs, non-breaking spaces and punctuation. Remove Extra Spaces and Line Breaks can help, but keep wrapped-line joining off for a one-item-per-line list.
Limits and later use
The tool allows 5,000,000 characters and 200,000 lines. Separate batches cannot find a duplicate shared between batches. Use a workflow that can compare the complete dataset if global uniqueness is required.
Save the cleaned result separately and record the options. Removing a repeated email address does not validate it, establish consent or confirm that two differently written names belong to the same person. Review those questions in the system that manages the records.