How to Remove Duplicate Emails from a CSV List

You have a contact list in a CSV — two exports stitched together, or a signup file that people typed into themselves — and you know some addresses are in there more than once. The import screen will eventually tell you so, or worse, it will accept the file and email the same person twice.

Three Kinds of Duplicate, and Only One Is Obvious

“Duplicate” means three different things in a contact export, and a plain duplicate check catches only the first one.

The case difference is the one that surprises people. In the email standard, the domain part of an address follows normal DNS rules and is not case sensitive, while the part before the @ technically is: RFC 5321 says the local-part “MUST BE treated as case sensitive”, then immediately adds that “exploiting the case sensitivity of mailbox local-parts impedes interoperability and is discouraged”. In practice, mail providers deliver Ann@shop.com and ann@shop.com to one inbox, so for a mailing list you almost always want case ignored.

Stray spaces come from copy-and-paste and from spreadsheet cells. A leading or trailing space makes the value a different string, so a naive comparison keeps both rows — and the receiving system may trim the space on import, leaving two contacts that look identical in its own interface.

Why the Extra Copies Cost You Something

• You pay for them. Email and CRM platforms commonly price by the number of contacts you store, so duplicates are billed like real people.
• They use up your import allowance. Import tools cap the rows or megabytes you can send in one go, and duplicate rows spend that budget for nothing. See how to split a CSV to fit an import limit.
• Somebody gets the same email twice. That produces unsubscribes and spam complaints, the two numbers that decide whether your future mail is delivered at all.
• Your reports lie. One dead address listed four times counts as four bounces, and open rates are measured against an inflated recipient count.

Dedupe on the Email Column, Not the Whole Row

Comparing every column is the safe default for a file that was merged twice, but it is the wrong choice here. The same person signing up in January and again in March gives you two rows that differ in the date, the source and perhaps the spelling of their name. Compare only the column that is supposed to be unique, and keep the rest of the row as it is.

1. Open the remove duplicates tool and choose your CSV. It is read on your own computer and never uploaded.
2. For what makes two rows the same, choose these columns match and tick only your email column.
3. Tick Ignore upper and lower case and Ignore spaces at the start and end of values.
4. Tick Also save the removed rows to a separate file so you can check the result.
5. Press Remove Duplicates and download the cleaned file.

Those two folding options affect the comparison only. The row that is written out is copied as its original text, so Ann@Shop.com stays capitalised, and a ZIP code of 02134 or a long account number keeps every digit — which is not true of a round trip through a spreadsheet. See why Excel removes leading zeros for what that costs.

Which Copy Is Kept

The first one, unless you choose otherwise. Rows stay in their original order, and any later row matching one already seen is dropped. That matters when the duplicates are not identical: if the March signup carries a better phone number than the January one, keeping the first copy throws the newer detail away. If the file is in signup order, oldest first, choose Keep the last copy under Which copy to keep: the copy nearest the bottom stays instead, and the rows still come out in their original order. That setting reads the file twice, so it takes about twice as long, and one run handles about 8.4 million different addresses rather than about 33 million. If the file is in no useful order, sort it by date before you dedupe — in a spreadsheet, or by asking the exporting system to sort by date — and choose the copy to keep to match. Our tools do not sort; they are text tools, not a spreadsheet.

Excel’s Remove Duplicates, and Its Two Real Limits

Excel has this built in: select your range, then Data › Remove Duplicates in the Data Tools group, tick the columns to compare and click OK. Excel keeps the first occurrence of each value and deletes the rest, and Microsoft notes that data is removed from every column even if you ticked only one. It is a permanent deletion, so keep a copy of the sheet first; Ctrl+Z works only straight away. You also cannot run it on outlined data or data with subtotals — remove those first.

Two limits stop it being the answer for a big export:

• The row ceiling. A worksheet holds 1,048,576 rows including the header, and Excel never loads the rest of a longer CSV, so it cannot compare rows it has not read. See Excel’s row limit explained.
• Excel edits the other columns while you are in there. Opening a CSV converts values: leading zeros vanish, numbers longer than 15 digits are rounded, and codes such as 1-2 become dates. Microsoft is explicit that duplicate comparison “depends on what appears in the cell — not the underlying value stored in the cell”, so two IDs that differed in the file can look identical after conversion, and one of them is deleted.

Google Sheets: Data Cleanup and UNIQUE

In Sheets, run Data › Data cleanup › Trim whitespace first, then Data › Data cleanup › Remove duplicates, choosing the columns to compare and whether the range has a header row. For a list of distinct addresses without touching the original, =UNIQUE(A2:A) in a spare column does it, and =LOWER(TRIM(A2)) gives you a normalised column to compare on. Normalise it yourself rather than assuming a duplicate finder ignores case, and you know exactly what was compared. The cell ceiling Google publishes is 20 million per spreadsheet, which by simple division puts a 12-column contact file out of room at about 1.67 million rows, fewer once you add a helper column. Google also caps a spreadsheet created in or converted to Sheets at 100 MB, so a file with long values can fail before it reaches 20 million cells.

Check the Result Before You Delete Anything

The removed-rows file is the point of the exercise. Open it and the deleted rows are listed under the header. Two checks take a minute: the kept rows plus the removed rows should equal the original row count (see how to count the rows in a CSV), and nothing in the removed file should surprise you. If it contains rows whose email column is empty, that is worth knowing — every blank counts as the same value as every other blank, so all but one of the blank rows are removed.

Don’t Over-Dedupe

Some addresses look like duplicates but are not the same string, and some providers treat them as one mailbox while others do not. Gmail is the well-known case. On dots, Google’s help is explicit: “If the sender added dots to your address, you’ll still get that email”, so john.smith@gmail.com and johnsmith@gmail.com are one mailbox. On plus tags, Google’s Workspace help says you “can create variations of your email address where all messages arrive in your current inbox” by adding “a plus sign (+) and any word before the @ sign”, so john+news@gmail.com reaches the same person as john@gmail.com.

Those are Gmail’s rules, not email’s rules. RFC 5233 describes the “+detail” convention as site-specific and implementation-specific, and on an ordinary company domain ann.brown@acme.com and annbrown@acme.com can easily be two different people. Our tool compares the text as written, ignoring only case and outer spaces if you ask it to; it applies no provider-specific rules. That is deliberate. If you do want to collapse plus tags, do it knowingly: build a normalised column in a spreadsheet, restrict it to the domains you are sure about, and keep the original address for sending.

A Sensible Order of Operations

1. Merge the CSV exports into one file first, so duplicates across files become visible.
2. If characters or separators look wrong, fix the encoding and delimiter before anything else.
3. Check the file’s order if you want the newest copy of each contact kept: newest first suits the default, oldest first needs Keep the last copy.
4. Remove duplicates on the email column, ignoring case and outer spaces, keeping the removed rows.
5. Check the counts, then split the CSV into smaller files if it is over your import limit.

One thing to be clear about: RowSlice’s tools clean CSV text. They do not check whether an address exists — for that, use your email platform’s own verification step — and they do not edit individual records, which is a spreadsheet’s job; LibreOffice Calc is free and asks you, column by column, how to read a CSV when it opens one. If the cleaned list has to go to someone as a workbook rather than as a CSV, RowSlice’s CSV to Excel converter writes a real .xlsx from the CSV text, which keeps the leading zeros and long account numbers that a round trip through Excel would strip.