Remove Duplicate Rows from TSV
Drop a tab-separated file and get it back with duplicates removed. By default a row counts as a duplicate only if every cell matches; name one or more key columns and it dedupes on those instead, which is what you want when a record was exported twice with a different timestamp. The status line tells you how many rows were dropped so you can sanity-check the result before trusting it.
How to use
- Paste or drop the file. Row one is the header unless you tick no header row.
- Leave key columns blank to drop only rows that are identical across every cell.
- Or name the columns that define identity —
email, oruser_id,day— and everything else is ignored when comparing. - Choose which copy survives. Keep first preserves the earliest row in file order; keep last is the one you want when later rows are the fresher export.
- Read the status line — it reports the number removed — then copy or download.
Whole-row vs. key-column deduping
Whole-row deduping is safe and boring: a row has to be byte-identical (modulo the case and space options) to be dropped, so you can never lose information. It catches the common case of a file that was concatenated with itself, or an export that ran twice.
Key-column deduping is the one that actually cleans real data, and the one that can lose information. If you dedupe a payments file on transaction_id and two rows share an ID but differ in amount, one of those amounts is silently discarded. Before you dedupe on keys, it's worth running value counts on the key column to see how many collisions exist, and TSV diff afterwards against the original if the number removed surprises you.
The two options that trip people up
Ignore case treats [email protected] and [email protected] as the same key. Correct for email domains and country codes; wrong for anything case-sensitive like a base64 token or a Linux path. Ignore surrounding spaces trims each key value before comparing, which catches the classic "Berlin" vs "Berlin " pair that a spreadsheet round-trip introduces. Neither option modifies the output — the surviving row keeps its original spelling and spacing. If you want the values themselves cleaned, use trim columns first.
FAQ
Does it preserve the original row order?
Yes. Surviving rows come out in the order they appeared in the input — this is not a sort. With keep last, the surviving row is the last duplicate's content but it sits at the position of the first occurrence, so the shape of the file doesn't shift.
How do I see the duplicates instead of removing them?
Run value counts on the key column — anything with a count above 1 is duplicated, sorted most-frequent-first. That's usually the faster diagnosis than diffing the before and after files.
Can it dedupe columns rather than rows?
Not directly, but the trick is short: transpose the file so columns become rows, dedupe here, then transpose back. For simply deleting known-duplicate columns, delete columns is more direct.
Are blank rows removed?
Trailing blank lines are discarded on parse. Blank rows in the middle of the file are treated as data — all-empty rows are identical to each other, so whole-row deduping collapses them to one. Use remove empty rows if you want them all gone.
What's the shell equivalent?
For whole rows: (head -1 f.tsv; tail -n +2 f.tsv | sort -u) > out.tsv. For a key column, awk -F'\t' '!seen[$2]++' keeps the first occurrence and preserves order. The awk version is closer to what this page does.
Privacy
100% client-side. No upload. See the privacy policy.