Remove Duplicate Lines
- 7
- Lines in
- 5
- Lines out
- 2
- Removed
- 2
- Distinct repeats
oslo: 2 timescopenhagen: 2 times
Deduplication removes repeated lines and keeps the order of what remains, which is the difference between it and sorting a list to group duplicates together. It also reports which lines repeated and how many times, so you can see what was actually in the data rather than just how much smaller it got.
How it works
Each line is reduced to a comparison key, and the first line producing a given key is kept while later ones are dropped. What goes into that key is what the options control.
- Case sensitivity: with it off, "London" and "london" are the same line. Off is the right default for names and addresses, on is right for identifiers and code.
- Ignore surrounding spaces: with it on, a line with a trailing space matches the same line without one. Trailing whitespace is invisible and is the most common reason a duplicate is missed.
- Keep first or keep last: keeping the last occurrence is what you want when later rows in an export supersede earlier ones.
Order is preserved. Lines come out in the sequence they first appeared, so a list that was already in a meaningful order stays in it.
Examples
A list with repeats
Lines
Helsinki, Oslo, Copenhagen, Oslo, Stockholm, Reykjavik, Copenhagen
Result
7 in, 5 out, 2 removed: Oslo and Copenhagen each appeared twice
The first Oslo and the first Copenhagen are kept in their original positions; the later ones go.
A duplicate hidden by a trailing space
Lines
"apple" and "apple " (with a trailing space)
Result
Matched as duplicates only when "ignore surrounding spaces" is on
Strictly these are different strings. Almost always the space is an accident of copying, which is why the option defaults to on.
Keeping the last occurrence
Lines
status: draft / status: review / status: draft
Result
Keeps the second "status: draft", at the end
For a log or an append-only export where the latest entry wins, keeping the last occurrence is the correct reading of the data.
Frequently asked questions
Does it change the order of my lines?
No. Lines come out in the order they first appeared. That is the main reason to use this rather than sorting, which groups duplicates together but destroys any meaningful order the list already had.
Why is a line I can see twice not being removed?
Almost always trailing whitespace or a different case. Turn on "ignore surrounding spaces" and turn off "case sensitive", between them they catch nearly every duplicate that looks identical on screen but is not identical as a string.
What does keeping the last occurrence do differently?
It keeps each line at the position of its final appearance rather than its first. Use it when later entries supersede earlier ones, as in an append-only log where the newest status is the real one.
Can it tell me what the duplicates were before removing them?
Yes, the "what repeated" panel lists every line that appeared more than once with its count, up to twenty distinct lines. Read that before you copy the result if the duplicates themselves are what you are investigating.
Will it remove duplicate words inside a line?
No, it compares whole lines. For repeats inside a line, use the remove duplicate words tool, which works word by word.