UTF-8, UTF-16 and Windows-1252: Which One Is Your CSV?

Someone sends you a CSV, you open it, and the customer called José is now José. Nothing is broken and nobody made a typo — the file is being read with the wrong set of rules for turning bytes into letters, and this guide is about working out which set it actually needs.

What an Encoding Is

A file on disk is a long row of numbers, each between 0 and 255. A byte of 74 is not the letter J; it becomes the letter J only because the program reading the file consults a table that says so. That table is the encoding. The first 128 entries have been the same in almost every table since the 1960s, which is why plain English opens correctly everywhere. Disagreements start above 127, where accented letters, curly quotes, currency symbols and every non-Latin script live. Windows-1252 has room for 128 of them and no more; UTF-8 handles all of them by spending two, three or four bytes on a character instead of one.

That is the whole mechanism. When a file looks wrong, the bytes are almost always fine; two programs simply consulted different tables.

Why a CSV Cannot Tell You Which One It Uses

A .xlsx workbook, a PDF and a web page all have a header where the encoding is declared. A CSV file has no header. It is the first data row straight away, so there is nowhere to record anything about the file itself.

RFC 4180, the closest thing CSV has to a specification, does not settle it either. It notes that “common usage of CSV is US-ASCII, but other character sets defined by IANA for the ‘text’ tree may be used in conjunction with the ‘charset’ parameter” — and that parameter belongs to the MIME type used when a file is sent over a network, not to the file in your downloads folder. So every program that opens a CSV has to guess. Modern editors guess well. Excel, opening a file by double-click, does not guess UTF-8 unless the file carries a byte order mark: Microsoft’s own advice is that a UTF-8 CSV “can be opened normally if it was saved with BOM”, and otherwise you have to import it. Without that mark it falls back on your computer’s ANSI code page, and Microsoft warns that those “can be different on different computers” — which is why the same file can look right on your desk and wrong on a colleague’s.

Identify Yours from the Symptoms

You do not need a tool for the first pass. The damage pattern tells you which mistake was made. Each row below is the real byte sequence, checked by encoding and decoding the characters both ways.

The difference matters when you decide what to do. Mojibake like José is fully reversible: every original byte is still in the file and only the reading was wrong. The � case is not always reversible, because the program that produced it may have thrown the offending bytes away and written the replacement character in their place.

Chinese and Japanese Files

Older exports from East Asian systems often predate UTF-8. Windows knows Japanese Shift JIS as code page 932 and simplified Chinese as code page 936 — commonly called GBK, and listed by Microsoft as GB2312, its narrower ancestor. GB18030, code page 54936, is the later standard covering all of Unicode. The symptom differs from a European file: instead of a few odd letters you get long strings of unrelated characters, or nothing readable at all, because these encodings use two bytes per character and reading them as one gives nonsense rather than a near miss.

What a Byte Order Mark Is

A byte order mark, or BOM, is a short signature at the very start of a file that says which Unicode encoding follows. Microsoft’s own table of the marks is short:

For UTF-16 the mark does real work: the two bytes of each character can be stored in either order, and the file has to say which. For UTF-8 there is only one possible order, so, as Microsoft’s documentation puts it, “for UTF-8, the byte order mark is optional, since the bytes may only be in one order.” It serves purely as a label — which is why whether to include one is a decision about who receives the file, not about correctness.

Excel is the reason it is not optional in practice. Microsoft’s support page on opening UTF-8 CSV files says a UTF-8 file opens correctly in Excel if it was saved with a BOM; without one you have to route the file through Power Query or the Text Import Wizard instead of double-clicking it. That is why our converter ticks the BOM box by default.

The catch is everything that is not Excel. Many databases, scripts and import APIs read the three bytes as ordinary characters at the start of the first field, so a column named id arrives as id and a header match fails for no visible reason. If your file is going to a program rather than to a person, leave the BOM off.

How to Check an Encoding Yourself

1. Notepad, on Windows. Open the file and choose File › Save As. The Encoding box next to the Save button shows what Notepad thinks the file is, and it distinguishes UTF-8 from UTF-8 with a BOM, which is exactly the distinction that matters below. Change nothing and press Cancel; you only wanted to read the box.
2. VS Code. The encoding appears in the status bar along the bottom. Click it and you get Reopen with Encoding and Save with Encoding. Reopening is the useful one: try Windows-1252 on a file full of é and watch the accents come back, which confirms the diagnosis before you convert anything. The same bar shows the line ending, CRLF or LF.
3. A terminal, on macOS or Linux. Run file --mime-encoding data.csv. The short flag differs by platform — on macOS it is file -I data.csv, on Linux file -i data.csv — but --mime-encoding works on both and prints only what you want. Expect answers like utf-8, utf-16le, us-ascii or, for a Windows-1252 file, unknown-8bit: the tool can tell that the file is eight-bit and not Unicode, but it cannot name which of the dozens of single-byte tables it is. That is the guessing problem in one word.
4. PowerShell, to see the actual bytes. Format-Hex -Path data.csv prints the first bytes as hex next to their characters, so a UTF-8 BOM appears unmistakably as EF BB BF followed by . Stop it after the first line with Format-Hex -Path data.csv | Select-Object -First 2.
5. LibreOffice Calc, if you would rather click than type. It shows a Text Import dialog every time it opens a CSV, with a Character set dropdown and a live preview. Change the setting and watch the preview: when the accents look right, that is your encoding.

What Our Converter Does, and What It Cannot Do

The CSV encoding tool detects UTF-8 with or without a BOM, UTF-16 LE and BE, and Windows-1252, and lets you override the guess with ISO-8859-2, Shift JIS or GBK. Output is always UTF-8, with the BOM as a checkbox. It can also change the delimiter between comma, semicolon, tab and pipe, and set line endings to CRLF or LF. Nothing is uploaded: the file is read in chunks in your browser, so file size is limited by your free disk space rather than by an upload cap.

What it cannot do is undo a mistake already written to disk. If a program read José as José and then saved the file, the characters é are genuinely in the file now, and a faithful converter keeps them faithfully. The same goes for a file full of �: those bytes were discarded and no tool can invent them. The fix is upstream — request the export again, and ask for UTF-8.

It also only ever handles CSV text. It will not produce an .xlsx file, open an Excel workbook, edit cells, chart or sort. If the encoding is right but everything still lands in column A, that is a delimiter problem, covered in why your CSV opens in one column in Excel.

Which Encoding to Send

UTF-16 is left off on purpose. It is a perfectly good encoding, and Excel’s Unicode Text save format produces it, but it doubles the size of Western-language files and many import tools reject it. If you receive UTF-16, convert it to UTF-8 and move on.

One Habit That Prevents Most of This

Check a file the moment it arrives. Open it, find an accented name, a curly apostrophe or a currency symbol, and confirm those characters specifically. Fixing an encoding on delivery takes a minute; finding it after the file has been merged, deduplicated and imported means redoing all of it. If you are combining exports from several systems, convert each one to UTF-8 before you merge them — a merged file with two encodings inside it cannot be fixed in one pass.

Related reading: CSV UTF-8 vs CSV in Excel shows the bytes each of Excel’s four CSV formats writes, how to convert a CSV to Excel without breaking your data covers the import route that keeps both your encoding and your values, and stop Excel removing leading zeros covers the other half of what Excel changes while it opens a file.