To remove duplicate rows from a CSV, decide which columns make two rows the same record, such as email, then keep one row per value of those columns. FindUtils CSV Dedupe does this in your browser: pick the key columns (or none, to compare whole rows), keep the first or the last copy, and download the cleaned file plus a second file holding every removed row. The kept rows stay in their original order, and the cells are written back exactly as they were. The file is not uploaded.
This guide covers choosing the key, the matching options, what happens to blank keys, and the kind of duplicate the tool does not try to find.
Why Exports Collect Duplicates
CRMs, newsletter tools, form builders and shops rarely stop a record from being added twice:
- The same person signs up twice, from two forms or two campaigns.
- Case and spacing differ.
[email protected]and[email protected]are the same address to a mail server, but not to an exact comparison. - An import runs twice and appends every row again.
- An append-only log holds one row per update, and you want the latest per customer.
Sending to that list means the same person gets the same message more than once.
How to Dedupe a CSV
Step 1: Load the file
Open CSV Dedupe and drop a .csv, .tsv or .txt file on the Input area, choose one, or paste the text. The delimiter is detected from the first line; set it under File format if the guess is wrong.
Step 2: Pick the key columns
Pick the columns that decide a duplicate, for example email. Pick several for a composite key, such as first_name + last_name + postcode. Leave all unpicked to remove only rows that repeat in every cell.
Step 3: Choose which copy to keep
Under When a row repeats, keep, choose First row to keep the earliest occurrence of each key or Last row to keep the latest. Then set Trim whitespace and Ignore case.
Step 4: Check and download
The status line shows how many rows were read, kept and removed. Check the Kept rows and Removed duplicates tabs, then download the cleaned file and, if you want a record, the duplicates file.
A Worked Example
This list has one person twice under the same email:
email,name [email protected],Ada [email protected],Bob [email protected],Ada Lovelace
With email as the key and the first copy kept, the cleaned file is:
email,name [email protected],Ada [email protected],Bob
With the last copy kept, the cleaned file keeps the newer name, and Bob stays in front because kept rows never change order:
email,name [email protected],Bob [email protected],Ada Lovelace
The duplicates file for that run holds the removed row under the same header: [email protected],Ada.
How Trim and Ignore Case Change the Result
Trim whitespace is on by default and Ignore case is off. Here is one column, email, with three spellings of one address:
| Trim whitespace | Ignore case | Rows kept | Why |
|---|---|---|---|
| On | Off (default) | 3 | The letters differ in case |
| Off | On | 2 | [email protected] keeps its spaces, so it differs from the other two |
| On | On | 1 | All three reduce to [email protected]; the first, [email protected], is kept |
The kept row is written exactly as it was, capitals and spaces included. The options change only the comparison.
For email addresses, Ignore case is usually safe. For names, IDs and codes it usually is not: AB-1 and ab-1 may be different products.
What Happens to Rows With an Empty Key?
By default, a row whose key cells are all empty is kept as unique. Ten subscribers with no email address are ten different people, not one. Choose Collapse into one under Blank keys to deduplicate empty keys like any other value, which keeps one of them. A cell of only spaces counts as empty.
This applies to key columns only. With no key picked, whole rows are compared, and two completely identical rows are duplicates whatever they contain.
Exact Matching Only: What It Will Not Find
CSV Dedupe removes rows whose keys match exactly after the optional trimming and case folding. It does not guess:
| Row A | Row B | Duplicate? |
|---|---|---|
[email protected] | [email protected] | Only with Ignore case on |
[email protected] | [email protected] | Yes with Trim on (the default) |
Jon Smith | John Smith | No: different text |
+44 20 1234 | 020 1234 | No: phone formats are not normalised |
[email protected] | [email protected] | No: different addresses |
Deciding that two differently written records belong to one person is fuzzy matching, and the tool does not attempt it. Rows it leaves alone stay in the cleaned file for you to review.
CSV Dedupe vs Spreadsheet Remove Duplicates
| Feature | CSV Dedupe | Spreadsheet "Remove duplicates" |
|---|---|---|
| Choose key columns | Yes | Yes |
| Keep the last copy | Yes | No: the first occurrence is kept |
| Separate file of removed rows | Yes | No: rows are deleted in place |
| Ignore spaces and case | Checkboxes | Needs helper columns or formulas |
| Values changed on the way in | Never: the text is written back as read | Opening a CSV can drop leading zeros or turn values into dates |
| Fuzzy matching | No | No |
Common Mistakes
Keying on a name column. Two people can share a name. Use email or an ID, or combine name with a second column.
Keeping the first copy of an append-only log. In a log where later rows are newer, keep the last copy.
Deduping one export when the duplicates span two. Combine the files first with CSV Merger, then dedupe the merged file.
Using it on a plain list. For one value per line with no header, Duplicate Line Remover is simpler.
Use It From Code
The same logic is the csv_dedupe tool on the FindUtils REST API and MCP server. Text sent there is processed on the server rather than in your browser.
curl -X POST https://api.findutils.com/api/tools/csv-dedupe/execute \
-H "Content-Type: application/json" \
-d '{"csv":"email,name\[email protected],Ada\[email protected],Bob\[email protected],Ada L\n","keys":["email"],"keep":"last","ignore_case":true}'Other arguments: trim, blank_keys (distinct or collapse), delimiter and header. The response holds cleaned_csv, duplicates_csv, the counts and a preview of up to 50 removed rows. Input can be up to about 10 MB. See the API reference and the MCP reference.
Tools Used in This Guide
| Tool | Use |
|---|---|
| CSV Dedupe | Remove duplicate rows by key columns or whole rows |
| CSV Merger | Combine several CSV files before deduplicating |
| CSV Diff | Compare the cleaned file with the original by key |
| CSV Viewer | Read a large CSV as a table |
| Duplicate Line Remover | Remove repeated lines from a plain list |
| Dedupe Bookmarks | The same job for browser bookmarks |
FAQ
How do I remove duplicate emails from a CSV mailing list?
Load the file in CSV Dedupe, pick the email column as the key, turn on Ignore case, and download the cleaned file. Trim whitespace is already on, so addresses with stray spaces match too.
Which copy of a duplicate is kept?
The first occurrence by default, or the latest when you choose Last row. Either way, the kept rows stay in their original file order.
Does it change my data?
No. Cells are written back exactly as they were read, including spaces and letter case. Only which rows are kept changes.
Can it find near-duplicates like Jon and John?
No. Matching is exact after the optional trimming and case folding. Different spellings, typos and reformatted phone numbers are different values.
Is my file uploaded?
On the page, no. The file is read and processed in your browser tab.
Next Steps
Clean your export with CSV Dedupe. To see exactly which rows changed between the original and the cleaned file, compare them with CSV Diff.
