Add or Remove a TSV BOM
Three invisible bytes at the start of a file cause two opposite problems. Without them, Excel on Windows opens a UTF-8 TSV as Windows-1252 and turns every accented character into mojibake. With them, a Unix parser reads the first column name as id instead of id, so every lookup by that name fails for reasons nothing in the file explains. This page shows you which situation you're in and flips it.
How to use
- Drop the file. This tool reads raw bytes rather than decoded text, so the status line reports truthfully whether
EF BB BFis on disk. - Pick the direction. Remove for anything programmatic; add if the destination is Excel on Windows.
- Download. The saved file carries the BOM bytes, or doesn't, as you chose. Copy-to-clipboard can't preserve a BOM — use the download.
When you want a BOM
Exactly one case: the file will be opened by double-clicking it in Excel on Windows. Excel's importer has no encoding detection worth the name — without a BOM it assumes the system's legacy code page, and Zürich becomes Zürich. The BOM is the only signal it reliably honours. Google Sheets, LibreOffice, and Excel on macOS all cope without one.
Note that a BOM alone doesn't guarantee Excel parses tabs into columns; that depends on your locale's list separator. Opening via Data → From Text/CSV and choosing Tab explicitly always works, and needs no BOM at all — which is the better habit if you can teach the recipient one thing.
When you don't
Everywhere else. A BOM breaks Postgres COPY, MySQL LOAD DATA, most command-line pipelines, and JSON produced from the file. It corrupts the first header name in pandas unless you pass encoding='utf-8-sig'. Git treats it as content, so it shows up in diffs. And a BOM in the middle of a concatenation — which is what you get from cat a.tsv b.tsv when both files have one — leaves a stray zero-width character mid-file that nothing will explain to you.
The parsers on this site all strip a leading BOM before doing anything else, so tsvkit itself never trips over one. That's also why the detection here needs raw bytes: by the time text has been decoded normally, the evidence is gone.
FAQ
How do I check for a BOM without a tool?
head -c 3 f.tsv | xxd — if it prints efbb bf the BOM is there. file f.tsv also reports "UTF-8 Unicode (with BOM) text".
Is a UTF-16 BOM the same thing?
No. FF FE or FE FF means the file is UTF-16, not UTF-8 with a marker — the whole file is encoded differently and stripping two bytes won't help. Excel's "Unicode Text" export does this. Use convert encoding to get it into UTF-8 first.
Why does my pasted text always say "no BOM"?
Because clipboards carry decoded text, and the BOM is an encoding artefact that gets dropped in the process. That's not a bug in the tool — it's why the drop zone exists.
Can I remove a BOM from many files at once?
Not here. On the command line: sed -i '' $'1s/^\xef\xbb\xbf//' *.tsv strips it from the first line of every matching file.
Privacy
100% client-side. No upload. See the privacy policy.