Skip to content

Remove Duplicate Lines

Text ToolsCleanup, Sorting and Lists
Unique lines —
Lines in
—
Lines kept
—
Duplicates removed
—
Empty lines removed
—

Paste your list and every repeated line is dropped, keeping only the first copy of each in the order it appeared. A six-line list of apple, banana, apple, cherry, banana, apple becomes three lines: apple, banana, cherry. You can also ignore letter case, trim stray spaces and delete empty lines.

About this tool

Merged mailing lists, keyword exports and log extracts tend to carry the same entry many times. This tool reads your text line by line and keeps the first occurrence of each line, so the order you started with survives, unlike a sort-then-dedupe in a spreadsheet. Matching can ignore letter case, which matters for email addresses, and can ignore spaces at either end of a line, which often sneak in from copied cells. A table lists the lines that repeated most and how many times each appeared. The comparison is whole-line only: two lines that differ by a single comma or an inner space are treated as different.

How to use it

  1. Add the list

    Paste one item per line into the box, or open a .txt or .csv file from your device.

  2. Choose how lines match

    Tick the case option to merge Apple and apple; keep space trimming on to ignore stray leading or trailing spaces.

  3. Decide on empty lines

    Leave Delete empty lines ticked to drop gaps, or untick it to keep one blank line.

  4. Copy the unique lines

    Read the counts and the table of repeated lines, then copy the cleaned list.

Examples

A fruit list with repeats

Lines to clean
apple banana apple cherry banana apple
Treat Apple and apple as the same line
No
Ignore spaces at the start and end of lines
Yes
Delete empty lines
Yes

Result Unique lines: apple
banana
cherry
Lines in: 6
Lines kept: 3
Duplicates removed: 3
Empty lines removed: 0

  1. Read 6 lines; matching is case-sensitive with leading and trailing spaces trimmed.
  2. Kept the first copy of each line in its original position: 3 unique lines.
  3. Removed 3 repeated lines and 0 blank lines.

Six lines hold three distinct fruits; the first apple, banana and cherry stay in their original order.

Emails that differ only in case

Lines to clean
Email@x.com email@x.com EMAIL@x.com
Treat Apple and apple as the same line
Yes
Ignore spaces at the start and end of lines
Yes
Delete empty lines
Yes

Result Unique lines: Email@x.com
Lines in: 3
Lines kept: 1
Duplicates removed: 2
Empty lines removed: 0

  1. Read 3 lines; matching is case-insensitive with leading and trailing spaces trimmed.
  2. Kept the first copy of each line in its original position: 1 unique lines.
  3. Removed 2 repeated lines and 0 blank lines.

With case ignored and spaces trimmed, three spellings of one address collapse into the first one.

Repeats with gaps between them

Lines to clean
a b a
Treat Apple and apple as the same line
No
Ignore spaces at the start and end of lines
Yes
Delete empty lines
Yes

Result Unique lines: a
b
Lines in: 5
Lines kept: 2
Duplicates removed: 1
Empty lines removed: 2

  1. Read 5 lines; matching is case-sensitive with leading and trailing spaces trimmed.
  2. Kept the first copy of each line in its original position: 2 unique lines.
  3. Removed 1 repeated lines and 2 blank lines.

The two empty lines are deleted and the second a is dropped, leaving a and b.

How it is calculated

kept lines = first occurrence of each distinct key; removed = lines in − lines kept − empty lines removed

key
the line used for comparison, lower-cased and trimmed when those options are on
lines in
every line in the input, split at line breaks (Windows and Unix endings both count)

Each line is turned into a comparison key, and a line is kept only when its key has not appeared earlier. Because the scan runs top to bottom, the survivor is always the first copy and the list keeps its order. When trimming is on, the kept line is also written without its outer spaces.

When not to use it

  • For near-duplicates such as typos or different spellings, which need fuzzy matching rather than exact comparison.
  • For CSV rows where only one column should decide what counts as a repeat.

Common mistakes

  • Leaving case matching on for email addresses, so John@site.com and john@site.com both survive.
  • Pasting comma-separated values on one line, which the tool treats as a single line.

Frequently asked questions

Which copy of a repeated line is kept?

The first one. Lines are read from top to bottom, and any later line whose comparison key matches an earlier one is dropped, so the cleaned list keeps the original order.

Does it sort the list?

No. The order of the surviving lines is exactly the order they first appeared in. Sort the output separately if you need it alphabetical.

Why were two lines that look identical both kept?

They differ somewhere you cannot see easily, such as a tab, a non-breaking space inside the line, or a letter in another case. Turn on case-insensitive matching and check for inner spaces.

How many lines can it handle?

Lists of tens of thousands of lines work in a normal browser, because each line is checked against a lookup of earlier keys rather than compared with every other line.

Is my list uploaded anywhere?

No. The text and any file you open are processed in your browser and are not stored after you leave the page.

Cite this tool

Using this tool in an assignment, article or report? Copy a citation, or a link for your website.

APA

Tools Directory Editorial Team. (2026, October 19). Remove Duplicate Lines. Tools Directory. Retrieved October 26, 2026, from https://tools.milunir.com/text-tools/cleanup-sorting-and-lists/remove-duplicate-lines/

MLA

Tools Directory Editorial Team. “Remove Duplicate Lines.” Tools Directory, 19 Oct. 2026, https://tools.milunir.com/text-tools/cleanup-sorting-and-lists/remove-duplicate-lines/. Accessed 26 Oct. 2026.

Chicago

Tools Directory Editorial Team. “Remove Duplicate Lines.” Tools Directory. Last modified October 19, 2026. Accessed October 26, 2026. https://tools.milunir.com/text-tools/cleanup-sorting-and-lists/remove-duplicate-lines/.

Link<a href="https://tools.milunir.com/text-tools/cleanup-sorting-and-lists/remove-duplicate-lines/">Remove Duplicate Lines – Tools Directory</a>