CSV Dedupe
BetaRemove duplicate rows from a CSV or TSV file by the columns you choose, keeping the first or last copy. Runs in your browser; the file is not uploaded.
- Free, no sign-up
- REST + MCP
- Updated
- Reviewed by Olgun Ozoktas
Input
No data yet — drop a file or paste CSV text
0 rows
Kept and removed rows will show here
Pick key columns after loading, or leave them empty to compare whole rows.
Runs in your browser; the file is not uploaded. Analytics and ads may load on the page.
Why Use CSV Dedupe?
CSV Dedupe removes repeated rows from a delimited table. It reads the text with the usual CSV rules: a field wrapped in double quotes may hold the delimiter or a line break, and a doubled quote inside it is a literal quote. Comma, semicolon, tab and pipe are detected from the first line, or you can choose one. The first row is treated as the header unless you turn that off, in which case the columns are named column_1, column_2 and so on. Blank lines are skipped and do not count as rows. A header name that appears twice is renamed with a suffix such as email_2 so every column can be picked.
A row is a duplicate when its key matches the key of another row. With no key columns picked, the key is the whole row, every cell it has. With key columns picked, only those cells are compared, so two rows with the same email but a different name or country count as duplicates. Before comparing, Trim whitespace ignores spaces at the start and end of each cell and Ignore case ignores letter case; the cells themselves are never changed. A row whose key cells are all empty is kept as unique by default, because an empty email is not the same person as another empty email; choose Collapse into one to treat empty keys like any other value.
Keep first row keeps the earliest occurrence of each key and Keep last row keeps the latest. Either way, the rows that survive are written in the order they had in your file, and the removed rows go to a second file in their original order too. Both files carry the header and use the delimiter the input was read with. The status line counts the rows read, kept and removed, and the Removed duplicates tab shows the first 50 removed rows with ignored spaces marked.
The page works on files up to about 10 MB. Larger inputs are processed in the background so the tab stays responsive. If a quoted field never closes, the page stops, names the line and removes nothing. To combine several files before deduplicating, use CSV Merger. To compare the cleaned file with the original by key, use CSV Diff. For plain lists that are not tables, use Duplicate Line Remover.
How it compares
Excel's Remove Duplicates and Google Sheets' Data cleanup both remove repeated rows by the columns you tick, keep the first occurrence and change the sheet in place. They do not give you a separate list of what was removed, and opening a CSV in a spreadsheet can change values on the way in, such as leading zeros in IDs or text that looks like a date. CSV Dedupe works on the text, writes every cell back exactly as it was, lets you keep the last occurrence instead of the first, and hands you the removed rows as their own file. In pandas, drop_duplicates(subset=[...], keep='first') does the same exact matching and also offers keep='last'; trimming and case folding need their own steps there, and here they are two checkboxes. None of these, this page included, does fuzzy matching: "Jon Smith" and "John Smith" are different values, and deciding that two differently written records are the same person is not something this tool attempts.
Tips for Deduplicating CSV Files
- Email addresses are usually safe to compare with Ignore case on; names and IDs usually are not.
- Trim whitespace is on by default, so " [email protected]" and "[email protected]" match. The cells are written out with their spaces, unchanged.
- Pick several key columns to build a composite key, for example first_name + last_name + postcode. A row is a duplicate only when every picked column matches.
- Keep last row is the one to use for an export where later rows are newer versions of the same record. The kept rows still come out in file order.
- The duplicates file uses the same header and delimiter as the cleaned file, so you can open both side by side or feed the duplicates to CSV Diff.
- Both downloads use LF line endings and quote only the cells that need it.