Remove duplicate lines from a list

Cleaning a list without opening a spreadsheet

Repeated lines creep into every kind of list: a mailing list built from several exports, a set of keywords collected over months, a column of URLs, a log file, or a list of names typed by different people. Spreadsheets can remove duplicates, but only after you import the data, pick the column and check what the tool did to the rest. Pasting the list here and copying the result back is faster.

Two ways to define "the same line"

By default two lines are duplicates only if they match exactly. Real lists are rarely that tidy, which is why there are two comparison options. Ignore upper/lower case treats Apple and apple as one item. Ignore spaces at line edges treats apple and apple as one item, which matters because a trailing space is invisible and is the most common reason duplicates survive a clean-up.

Keep the first copy, or the last

The result keeps the order in which lines first appear, so your list stays recognizable. If you paste an ordered log where the newest entry is the one you care about, switch to keeping the last occurrence instead. You can also sort the result alphabetically, with numbers ordered naturally (item 2 before item 10), or by how many times each line repeated.

Find the duplicates instead of removing them

Sometimes the useful answer is which lines repeat. Choose "Only lines that repeat" to list them once each, or "Only lines that appear once" to get the entries that have no duplicate at all. Below the result you also see a ranking of the most repeated lines with their counts.

What it does not do

The tool compares whole lines. It will not spot two lines that are similar but not equal (for example a misspelling), and it does not look inside a line, so for a spreadsheet with several columns, copy the single column you want to check.