Remove Duplicate Lines: Free Online Tool

How to Remove Duplicate Lines

1. Paste your list or drop a file

   Paste text into the input box, one item per line, or drop a .txt or .csv file on the drop area. A file is read on your computer; nothing is sent anywhere.

2. Choose how lines are compared

   Leave “Trim whitespace” and “Ignore empty lines” on for most lists. Turn on “Ignore case” if “Apple” and “apple” should count as the same line. Turn on “URL mode” for lists of links.

3. Pick which copy to keep and how to sort

   Keep the first copy to preserve the original order, or the last copy if later entries are more up to date. Leave sorting on “Keep original order”, or sort lines alphabetically A–Z, Z–A, naturally (2 before 10) or by length.

4. Check the counts

   The result appears in the output box with three numbers: lines in, unique lines and duplicates removed. Press “Show duplicates” to see which lines repeated and how many times.

5. Copy or download the result

   Press Copy to put the cleaned list on your clipboard, or Download to save it as a .txt file. Short lists update as you type; for very large inputs press “Remove duplicates” to run it.

How This Duplicate Line Remover Works

The duplicate line remover splits your text at every line break (Windows CRLF, Unix LF and old Mac CR are all recognised) and builds a comparison key for each line. The key is the line itself, adjusted by the options you picked: trimmed if trimming is on, lowercased if case is ignored, and normalised if URL mode is on. The first time a key appears, that line is kept. Every later line with the same key is counted as a duplicate and dropped.

Lookups use a JavaScript Map (a hash table), so a million lines take about a second, not minutes. Large inputs run in a Web Worker, a background thread, so the page stays responsive while a progress bar shows how far it has got.

Your text never leaves your device. The page sends nothing you paste or drop over the network, stores nothing and logs nothing. Close the tab and the list is gone.

• Order of operations: split into lines → trim (optional) → drop empty lines (optional) → build the key (URL mode, then case) → keep first or last → sort (optional).
• The output always shows the original line, not the key. In URL mode, “https://www.example.com/page/” stays exactly as you typed it; it just also stands for the other versions of that link.

What Counts as a Duplicate Line?

A duplicate line is a line whose content is identical to an earlier line, character for character. “Apple” and “apple” differ in case. “apple” and “apple ” differ by a trailing space you cannot see. Text copied from a web page may contain a non-breaking space (U+00A0) instead of an ordinary one. These look the same on screen and are still different strings.

That is why a dedupe tool needs options. Trimming removes spaces, tabs and other whitespace at the start and end of each line, including non-breaking spaces, because JavaScript’s trim() follows the Unicode definition of whitespace. Ignoring case compares lowercased copies. Neither option changes the words in the middle of a line: “New York” and “New York” (two spaces) are still different.

Removing duplicates, sorting and finding duplicates are three separate jobs. Unix tools blur them because uniq only removes repeats that sit next to each other, so you sort first. This tool removes duplicates wherever they are, sorts only if you ask, and lists the repeats in the “Show duplicates” view.

How to Remove Duplicate URLs (Same Page, Different Link)

To remove duplicate URLs properly, you have to compare what the links point to, not their exact text. The same page can be written many ways. “example.com/pricing”, “https://example.com/pricing”, “https://www.example.com/pricing/” and “https://example.com/pricing?utm_source=newsletter” all load the same pricing page, but a plain text comparison sees four different lines. Lists built from browser tabs, bookmarks, analytics exports or crawls are full of these near-duplicates.

URL mode fixes this by building a normalised version of each link before comparing. It parses the line with the browser’s own URL parser, which follows the WHATWG URL Standard, then applies the rules in the table below. Lines that are not web links (anything that isn’t http, https, or a bare domain like example.com/page) are compared as plain text, so a mixed list still works.

Some differences are kept on purpose. Paths can be case-sensitive, so “/Page” and “/page” stay separate unless you also turn on “Ignore case”. Query values are kept, because “?id=1” and “?id=2” are usually different pages. If you are auditing redirects, turn URL mode off.

URL mode decides which lines are duplicates; it does not rewrite the links you keep. If you also want the kept links stripped of tracking parameters, run the result through the URL cleaner afterwards.

Examples: Deduplicating Real Lists

• Email list merged from two sign-up forms: “[email protected]” and “[email protected] ” appear in both. Turn on Trim and Ignore case, keep first, and 2 lines become 1, keeping the original spelling “[email protected]”.
• Tabs exported from three browser windows: “https://docs.google.com/document/d/abc/edit”, “https://docs.google.com/document/d/abc/edit#heading=h.1” and “https://docs.google.com/document/d/abc/edit?usp=sharing”. URL mode merges the first two (the fragment is ignored). The third keeps its ?usp parameter because usp isn’t on the tracking list, which is the safe choice.
• File names like “scan-2.pdf, scan-10.pdf, scan-1.pdf”: plain A–Z puts scan-10 before scan-2, because it compares character by character. Natural sort gives scan-1, scan-2, scan-10.
• A CSV export where whole rows repeat: each row is one line, so identical rows are removed. Rows that differ in any column are kept. For duplicates in one column only, use a spreadsheet (see below).

How to Sort Lines Alphabetically (and Other Sort Options)

To sort lines alphabetically, choose “A–Z” in the sort menu. The tool uses your browser’s language-aware collation (Intl.Collator), so “apple” sits next to “Apple” and accented letters sort next to their plain versions, unlike a raw byte sort.

Natural sort treats runs of digits as numbers, so “Chapter 2” comes before “Chapter 10”. Use it for file names, version-like strings and numbered items. Sort by length puts the shortest lines first and breaks ties alphabetically, which is useful for spotting truncated entries or picking the shortest URL. Z–A is simply A–Z reversed.

How to Remove Duplicate Lines in Excel, Google Sheets, VS Code, Notepad++ and Sublime Text

Excel: select the column, then go to Data → Data Tools → Remove Duplicates. Microsoft notes that this permanently deletes the duplicate values, so work on a copy (Ctrl+Z works straight away). In Excel for Microsoft 365, Excel 2021 and Excel 2024 you can instead use a formula that leaves the original alone: =UNIQUE(A2:A1000), or =SORT(UNIQUE(A2:A1000)) to sort the result.

Google Sheets: select the range, then Data → Data cleanup → Remove duplicates, choose the columns and whether the data has a header row. Google’s help page says cells with the same value but different letter case, formatting or formulas are treated as duplicates. Data → Data cleanup → Trim whitespace removes leading and trailing spaces first, though it doesn’t trim non-breaking spaces. The formula route is =UNIQUE(A2:A), or =SORT(UNIQUE(A2:A)).

VS Code: select the lines (or nothing, for the whole file), open the Command Palette with Ctrl+Shift+P (Cmd+Shift+P on Mac) and run “Delete Duplicate Lines”. The command arrived in VS Code 1.62. “Sort Lines Ascending” and “Sort Lines Descending” are in the same palette.

Notepad++: Edit → Line Operations → Remove Duplicate Lines removes repeats anywhere in the file, and “Remove Consecutive Duplicate Lines” only removes repeats that sit next to each other. The same menu has sorts that ignore case, sort as integers or decimals, or sort by length, plus “Remove Empty Lines”.

Sublime Text: Edit → Permute Lines → Unique removes duplicates and keeps the first copy. Edit → Sort Lines (F9) sorts, and Edit → Sort Lines (Case Sensitive) is next to it.

Spreadsheets compare cells, not lines, and they can dedupe by one column while keeping the rest of each row. That is the right choice for CSV data where only the email or ID column should be unique.

Remove Duplicate Lines From the Command Line

On macOS and Linux, the shortest way is sort -u input.txt > output.txt. It sorts the file and keeps one copy of each line. Add -f (sort -f -u) to ignore case. Note that plain uniq only removes duplicates that are next to each other, so uniq on an unsorted file usually leaves duplicates behind; that is why you see sort and uniq together.

To keep the original order, use awk: awk '!seen[$0]++' input.txt > output.txt. It prints a line only the first time it has been seen. For a case-insensitive version, use awk '!seen[tolower($0)]++' input.txt. To keep the last copy instead of the first, reverse the file, dedupe, and reverse again: tac input.txt | awk '!seen[$0]++' | tac on Linux, or tail -r instead of tac on macOS.

To see which lines repeat and how often, run sort input.txt | uniq -c | sort -rn. To list only the lines that appear more than once, use sort input.txt | uniq -d.

In PowerShell, Get-Content input.txt | Sort-Object -Unique | Set-Content output.txt sorts and dedupes. Microsoft’s documentation says Sort-Object -Unique is case-insensitive, so “localhost” and “LOCALHOST” count as one. Get-Unique works differently: it is case-sensitive by default, only compares each item to the next one, and so needs sorted input, as in Get-Content input.txt | Sort-Object | Get-Unique. PowerShell 7.4 added a -CaseInsensitive switch to Get-Unique.

Locale changes how sort orders lines. With a UTF-8 locale, GNU sort interleaves upper and lower case; with LC_ALL=C it sorts by byte value, so every capital letter comes before every lowercase one. Set LC_ALL=C when you need identical results on every machine.

How to Close Duplicate Tabs in Your Browser

If the duplicates you are fighting are open tabs rather than lines of text, some browsers can close them directly. Firefox added a “Close Duplicate Tabs” command in version 127: right-click a tab, or open the List all tabs menu at the end of the tab bar. Vivaldi shows a Duplicate Tabs section in its Windows panel with a “Close All Duplicate Tabs” button. Safari doesn’t close duplicates, but when you type an address that is already open it offers “Switch to Tab” instead of opening another copy.

Chrome has no built-in duplicate closer. Its Tab search (Ctrl+Shift+A, or Cmd+Shift+A on Mac) lists open tabs and filters them as you type a title or address, which makes it easy to spot and close five copies of the same page by hand. The address bar also suggests “Switch to this tab” when you type a URL that is already open. Browsers built on Chromium behave similarly, and extensions can fill the gap in all of them.

The route that works everywhere: copy your open tabs’ URLs as a list, paste them here with URL mode on, and reopen the unique pages with the bulk URL opener.

Duplicate Line Remover vs Other Methods

Limits and Troubleshooting

• Lines that look identical aren’t merged: one probably has a hidden character, such as a zero-width space, in the middle. Trimming only handles the ends of a line.
• Very large files: around a million lines or 50 MB works in a current desktop browser. Beyond a few hundred megabytes the tab can run out of memory. Use sort -u or awk for files that size.
• Big results are previewed: if the output is over about 5 MB, the output box shows the first 5,000 lines to keep the page fast. Copy and Download always include every line.
• CSV files are compared row by row. The tool doesn’t understand columns, quoted commas or headers, so a header row is treated as an ordinary line.
• URL mode merged links you wanted separate: turn off “Strip tracking parameters”, or turn URL mode off entirely when the scheme, www. or trailing slash matters to you, for example when checking redirects.

Removing duplicates can’t be undone in the output box, but your input stays untouched until you clear it. Keep the original until you have checked the counts.