Remove Duplicate Columns from a TSV
Duplicate columns are what joins leave behind. A merge of three exports gives you id, id_b and id_c holding the same values; a widened report has the customer name repeated in four places. Databases reject a table with two columns of the same name outright, and pandas quietly renames them. This drops the repeats — matching on the header, on the values, or on both.
How to use
- Paste or drop the file. The first row is the header.
- Choose what counts as a duplicate. By content is the safe default; by name is what you need before a database load.
- Keep the first or the last. Position is preserved either way — a kept column stays where it was, it doesn't move to the front.
- Read the status line. It names every column that was dropped, so you can confirm none of them was one you needed.
The three modes are genuinely different
Same values compares each column's cells top to bottom and drops any column whose values are byte-identical to an earlier one, regardless of what it's called. This is the honest test for redundancy: country and country_b holding the same 5,000 values really are one column. The risk is coincidence — two boolean flags that happen to agree on every row in a small sample look identical and are not. Check the row count before trusting it on a short file.
Same header name ignores the values entirely and keeps one column per distinct name. Use it when the goal is a loadable file rather than a smaller one; it will drop a column that shares a name but holds different data, which is exactly what you want when the second one is the join's leftover and exactly what you don't want if the two were meant to be different things and someone named them badly.
Same name and same values is the conservative option: a column has to match on both counts to go. Nothing is lost that could have been meaningful, and anything left over is a real conflict worth looking at.
What the checkboxes affect
Both apply to the comparison only — the kept column's values are written out exactly as they came in. Trim makes UK and UK compare equal, so a column that differs only in trailing whitespace is recognised as a duplicate. Ignore case does the same for UK versus uk, and for header names, which is the right setting before loading into a database that folds identifiers.
Ragged rows are handled by treating missing cells as empty, so a short row doesn't make two otherwise identical columns look different. If your file is ragged for a reason, fix it first with fix ragged rows.
FAQ
What about duplicate rows rather than columns?
That's remove duplicate rows, which works on the other axis and can key on specific columns.
Can I just rename the duplicates instead of dropping them?
Yes — header tools can rename columns, and the join tool applies a suffix automatically when it creates the clash in the first place. Dropping is right when the column is genuinely redundant; renaming is right when it isn't.
Does it detect a column that's a duplicate except for a few rows?
No — comparison is exact, so one differing cell means the columns are kept. That's deliberate: a near-duplicate usually means the two sources disagree, and finding out where is the actual task. Extract both columns with extract columns and run TSV diff on the result.
What happens to a column with no header?
Its name is the empty string, so in name mode all unnamed columns collapse into one. If a file has several blank headers holding different data, use content mode instead, or give them names first with header tools.
Privacy
100% client-side. No upload. See the privacy policy.