A CSV file does not say what encoding it is in

In October a directory management console was found to have two CSV writers. Not two formats, not two features: one format, written twice, by two pieces of code in the same application. The import side offered a blank template to fill in. The list side offered Export selection. One of them began its file with three bytes and one did not, and, according to the issue filed against it, that was enough for Excel to show an account's accented name as Hélène.

The issue is linagora/twake-directory-manager#77, opened 18 October 2026 by the project's automated QA and closed on the 22nd. Its title is the whole bug:

"Export selection" writes a CSV without UTF-8 BOM (unlike the import template): accented values garble in Excel

What makes it worth a post is not the missing bytes. It is that the fix was not to add them.

There is nowhere in the file to put the answer

Here is the smaller half of the problem, and it is not a bug in anything. These are the first sixteen bytes of two files written for this post, with the same three columns and the same two accented rows, in the same encoding:

nobom_lf.csv    75 69 64 2c 67 69 76 65 6e 4e 61 6d 65 2c 64 65
bom_lf.csv      ef bb bf 75 69 64 2c 67 69 76 65 6e 4e 61 6d 65

Both are UTF-8. The second one carries a mark that some readers take as saying so; the first one says nothing, because there is no field in which to say it. A CSV file is rows of bytes with separators in them. It has no header block, no schema, no declaration, and so nothing in the first file distinguishes it from the same names written in a single-byte encoding. The bytes 48 c3 a9 6c c3 a8 6e 65 are Hélène in UTF-8 and Hélène in Windows-1252, and the file does not contain the sentence that would tell you which was meant.

This post quotes no standard. Every claim below is either a measurement, or a quotation from an implementation's own source at a named commit, because what decides how a file is read is the implementation that reads it.

The one place it was declared was already right

The interesting part of #77 is what the broken code looked like. This is exportSelection in src/components/EntityList.ts at commit e3c4527, the parent of the fix:

  /** Hand the selection to the browser as a CSV file. */
  private exportSelection(): void {
    const rows = this.entries.filter(([id]) => this.selected.has(id));
    const header = this.columns.join(',');
    const body = rows
      .map(([, entry]) =>
        this.columns
          .map(name => csvCell(text(entryValue(entry, name))))
          .join(',')
      )
      .join('\n');
    const blob = new Blob([`${header}\n${body}\n`], {
      type: 'text/csv;charset=utf-8',
    });

Read the last two lines. The broken version declared the encoding. It said charset=utf-8, it said it correctly, and the file still arrived in a spreadsheet as mojibake. The declaration is attached to the Blob object inside the page, and what the download writes to disk is a name and a sequence of bytes. The media type stays behind. Whatever the spreadsheet used to decide how to read those bytes, the one true statement the program made about them was not available to it.

The fixed version, at 4176f2a, declares exactly the same thing: its Blob is built from a different string, but the type: 'text/csv;charset=utf-8' beside it is untouched. A declaration that comes through a fix unchanged was never the thing that was broken.

So the reader decides, and the reader is not even the program

Four independent implementations, seven reader configurations between them, all run here on 11 November 2026 over four files: byte-order mark or not, LF or CRLF. Each cell below is the first given name in the file, which was written as Hélène.

Reader	no BOM	with BOM
Python 3.12.3 csv, encoding="utf-8"	Hélène	Hélène, first column named '\ufeffuid'
Python 3.12.3 csv, encoding="utf-8-sig"	Hélène	Hélène
Python 3.12.3 csv, encoding="cp1252"	Hélène	Hélène, first column named 'uid'
DuckDB 2.0.0, read_csv_auto	Hélène	Hélène
LibreOffice 24.2.7.2 headless, default CSV import	Hélène	Hélène
LibreOffice 24.2.7.2 headless, --infilter="CSV:44,34,1,1"	Hélène	Hélène, first column uid
DuckDB-Wasm 1.33.1-dev45.0, read_csv_auto	Hélène	Hélène

Line endings changed nothing, in any reader, in any row of that table. The encoding changed everything, and nothing in the file selected it.

Two of those rows are the reported bug, reproduced in two unrelated programs. Python told to read the bytes as Windows-1252, and LibreOffice told to import them with its single-byte character set, both produce Hélène: character for character, the string the issue says Excel shows. Without Excel that is as close as we can get to the report, and close enough to say the mechanism is understood rather than guessed at.

Then there is the sharpest version of the same point, which needs no second program at all. The same file, the same four lines of Python, the same interpreter, run twice with one environment variable changed:

$ python3 read.py
getpreferredencoding(False) = UTF-8
open() no encoding -> ['hmoreau', 'Hélène', 'Opérations']

$ LC_ALL=C PYTHONUTF8=0 python3 read.py
getpreferredencoding(False) = ANSI_X3.4-1968
open() no encoding -> ERROR UnicodeDecodeError 'ascii' codec can't decode byte 0xc3 in position 34

Nothing about the file changed between those runs, and the program did not change. The answer to "what encoding is this CSV in" moved from UTF-8 to US-ASCII because of a shell variable. In the second case the file became unreadable rather than wrong, which is the better of the two outcomes and is available only by luck.

A byte-order mark is a signature, not a declaration

The fix in #77 was to write the mark. That is the right call for this application, and it is worth being precise about why, because "add a BOM" is not a general answer and five of the seven configurations in the table above read the mark-less file correctly.

The clearest statement of what the mark actually is ships with Python, in the docstring of the codec named after it. This is encodings/utf_8_sig.py, in the interpreter that produced every Python row above:

""" Python 'utf-8-sig' Codec
This work similar to UTF-8 with the following changes:

* On encoding/writing a UTF-8 encoded BOM will be prepended/written as the
  first three bytes.

* On decoding/reading if the first three bytes are a UTF-8 encoded BOM, these
  bytes will be skipped.
"""

Note the shape of that. utf-8-sig is a different codec from utf-8, not a flag on it, and the suffix is sig, for signature. The mark is not part of the encoding. It is a convention sitting in front of the data that a reader may or may not know to look for, and a reader that does not look for it does not fail: it succeeds, and returns the mark as content. In Python's case the content it lands in is the first column name.

utf-8      fieldnames[0]='\ufeffuid'   row['uid'] -> KeyError 'uid'
utf-8-sig  fieldnames[0]='uid'         row['uid'] = 'hmoreau'

A KeyError on uid is a column-name bug. It gives no hint that an encoding convention is involved, and it is caused by the three bytes added to prevent an encoding bug. When the reader guesses wrong anyway, the mark is the first thing to break visibly: uid, in both of the single-byte readers above. The marker meant to stop the misreading becomes its first symptom.

The reason it is nonetheless correct here is specific and local, and it lives in the same file as the writer. src/csv.ts at the fix commit holds both halves of this application's convention, and the reading half strips the mark:

export function parseCsv(text: string): CsvTable {
  const source = text.replace(/^\uFEFF/, '');

Its doc comment says so in words: "A byte-order mark is dropped, CRLF and LF both end a row, and a row with no content at all is skipped." So a file this app writes is a file this app can read, mark included. That is a property of the pair, decided by the people who wrote both, and recorded in the only place it can be recorded, which is the code rather than the file.

The fix was to delete the second writer

Here is the part worth taking away. The application already had a correct writer. It is writeCsv, in src/csv.ts, and its doc comment is the convention stated as a promise:

/**
 * Write rows as a CSV file, with the separator given.
 *
 * @param rows header row first
 * @param delimiter separator
 * @returns the file content, with a byte-order mark so a spreadsheet reads
 *   it as UTF-8
 */
export function writeCsv(rows: string[][], delimiter = ','): string {
  return (
    '\uFEFF' +
    rows
      .map(row => row.map(value => csvCell(value, delimiter)).join(delimiter))
      .join('\r\n') +
    '\r\n'
  );
}

exportSelection did not call it. It built the same format itself, with a join(',') and a '\n', so it did not get the mark. The template button, in src/components/EntityImport.ts, hands writeCsv's output to a download helper whose Blob declares the same text/csv;charset=utf-8: two files, one declaration, and only the bytes differed. The commit that closed #77 is titled "Write the exported selection as the import's template is written", and what it does is route the export through the function that already existed. The changelog entry records the same thing in the user's terms: "Export selection writes its CSV with a byte-order mark and CRLF line endings, as the import's template: a spreadsheet read accented values as mojibake".

So the defect was not an absent byte. The defect was a second implementation of a format whose rules live nowhere except in implementations. Once the encoding has to be decided by the writer, the number of writers is the number of decisions, and the second one drifts without anything going red: the file was valid, the tests passed, and the application could read its own output back perfectly, because its reader strips the mark either way.

The drift was not even confined to the encoding. In the old code the data cells went through csvCell, which does two jobs: it quotes a cell holding a separator, a quote or a line break, and it puts an apostrophe in front of one a spreadsheet would run as a formula. The header row did not: it was this.columns.join(','), raw. With column names like uid and displayName that changed nothing in any file, which is exactly why nobody noticed: the divergence was waiting for a column name that needed one of those jobs. Routing the whole table through writeCsv closed it too, because writeCsv maps every cell of every row including the first. A second writer does not usually diverge in one way. It diverges in as many ways as nobody checked.

Where TableRune fits

TableRune reads CSV in the browser, so it sits on the reader side of this, and the honest summary is that it does not guess because it does not decide. Read at origin/prod commit 7fd53e6, importCsv in apps/web/src/lib/engine/duckdb.ts registers the file bytes in DuckDB's virtual filesystem and runs CREATE OR REPLACE TABLE ... AS SELECT * FROM read_csv_auto(file), with no options. There is no encoding handling in the repository at all: searching the TypeScript, Svelte and Rust sources at that commit for a byte-order mark, FEFF, utf-8-sig or an encoding option returns nothing outside one unrelated comment about protobuf encoding and one about PNG encoding.

So the behaviour is DuckDB's, and it was measured on the build the app actually ships rather than a native DuckDB. DuckDB-Wasm 1.33.1-dev45.0, which is what pnpm-lock.yaml resolves and which reports engine v1.5.1, driven through the package's Node bundle with TableRune's own call sequence: all four files give the three columns uid, givenName, department and the row hmoreau, Hélène, Opérations. The mark is stripped, UTF-8 is assumed, the column name is uid and not \ufeffuid, and CRLF and LF are equivalent. That is the good case: a reader with one rule, which applies it and never consults a locale.

The export side is where TableRune sits on the writing side of this post, and it is worth saying plainly. exportTable in the same file runs COPY <table> TO <file> (FORMAT CSV, HEADER). Measured on that same shipped build, that produces:

75 69 64 2c 67 69 76 65 6e 4e 61 6d 65 0a 68 6d 6f 72 65 61 75 2c 48 c3 a9 6c c3 a8 6e 65 0a
"uid,givenName\nhmoreau,Hélène\n"

UTF-8, no byte-order mark, LF line endings. That is byte for byte the shape of the file #77 is about. It is read correctly by every reader in the table above except the two that were told to use a single-byte character set, and it is the kind of choice that is defensible right up until someone opens the file in the one program that decides by locale. That is flagged here rather than called a bug, because the program that would settle it is Excel, which we have not run.

What this post does not establish

• No Excel. Excel was not run, on Windows or anywhere else. Every statement about what Excel does is the issue's and the fix commit's, attributed to them; the issue was filed by the project's automated QA, which ran in Chromium. The Hélène string is reproduced here in Python and in LibreOffice, which shows the byte-level mechanism is the one described, not that Excel behaves as reported.
• No specifications. RFC 4180, the Unicode standard and the Python, DuckDB, LibreOffice and Microsoft documentation are neither quoted nor cited. Where a rule is stated above it is quoted from an implementation that was run, or from source at a named commit.
• LibreOffice through the converter, not the dialog. Every LibreOffice result is soffice --headless --convert-to ods, with LANG and LC_ALL both empty. The interactive import dialog, which is where a person would meet this, was not exercised, and a different locale may give a different default.
• DuckDB-Wasm under Node, not in a browser. It is the version and engine TableRune ships, and the call sequence TableRune uses, run through the package's Node runtime rather than in a Web Worker.
• TableRune read, not run. Its import and export behaviour here is source at 7fd53e6 plus measurements of the engine it depends on, not a browser session.

The practical version

If you write CSV, the encoding is a decision you are making whether you know it or not, and it belongs in one function. One writer, with the convention in its name or its docstring, and every caller going through it. The cost of a second writer is not the first bug; it is that the first bug is invisible to your own tests, because your own reader was written by the same people and forgives it.

If you read CSV, assume nothing from the file, because there is nothing in it to assume from. Pick the reader setting that tolerates both cases where one exists: on these measurements utf-8-sig read all four files correctly and plain utf-8 did not, which makes the tolerant choice free. And if a column name ever fails to match by one invisible character, look at the first three bytes before anything else.

Related, on the same format: a CSV file does not say what type anything is, which is the delimiter and number half of this, measured the same way.