A file upload is three guesses at what the file is

Here is a failure that looks like a typo in a list and is not.

In October, a user of LineLedger, an open-source double-entry accounting app built on Laravel, reported that its bank import rejected the .qbo and .qfx statements they had downloaded from Wells Fargo (#130). Both formats were in the import's allow-list. The error said so:

The statement file field must be a file of type: csv, xlsx, xls, ofx, qfx, qbo, pdf.

The parser never ran. The reporter traced the rejection to the upload validation, and noted that macOS's file command identified both downloads as text/plain.

The cause is that the allow-list was checked against what the file's contents looked like, and a sniffer can tell one text format from another only when its table has a rule for that format. For OFX, none does. There are three different places to guess a file's type from, they do not agree, and the one that sounds most like reading the file is the one that could not see this format at all.

The three guesses

When a file arrives from a browser upload, three separate things have an opinion about what it is:

1. What the client said. The browser attaches a media type to each file it hands to a form or a script.
2. What the filename says. The part after the last dot, looked up in a table.
3. What the bytes say. A sniffer reads the first few hundred bytes and matches them against known signatures.

These get described as one value with three sources, which is the mistake. They are three guesses, produced by different code, from different inputs, with different failure modes. The rest of this post is what they actually return.

Everything below was measured on an OFX version 1 statement (the SGML serialisation), the same statement as OFX version 2 (XML), byte-identical copies of the version 1 file named .qbo and .qfx, an ordinary comma-separated CSV, two deliberately mismatched renames, and three more CSV dialects. Tools, versions and the exact commands are in this post's sources file.

Guess one: the browser does not look inside the file

In Chromium 153 and Firefox 155, both driven by Playwright on Linux, the media type a file input reports for each selected file comes from the name, not the contents.

Put the OFX version 1 statement in a file input and File.type is the empty string in both. Neither browser has an entry for .ofx, so it declines to guess, which is the honest answer and also a useless one if your validator treats empty as invalid. The .qbo and .qfx copies split them: Chromium returns the empty string for both, Firefox returns application/vnd.intu.qbo and application/vnd.intu.qfx. Same bytes, same operating system, two browsers, two answers.

Then the control. Copy the CSV to renamed_csv.ofx and copy the OFX file to renamed_ofx.csv, so that the name and the contents point at different formats:

file in the input	what is actually inside	File.type, both browsers
statement.csv	comma CSV	text/csv
renamed_ofx.csv	OFX version 1 (SGML)	text/csv
renamed_csv.ofx	comma CSV	empty string
statement.ofx	OFX version 1 (SGML)	empty string

Both browsers called the OFX file text/csv because it was named .csv, and declined to name the CSV because it was named .ofx. They differ only where their tables differ. Guess one is guess two wearing a different hat. It is a restatement of the filename, so validating it is not a second check, and anyone uploading can set it to whatever they like.

Guess two: the extension table disagrees with itself

The second guess is a lookup table. Python's mimetypes is a convenient one to read, because the table is inspectable. These are its answers once mimetypes.init() has loaded the operating system's /etc/mime.types, which it reads by default wherever that file exists:

extension	mimetypes.types_map
.csv	text/csv
.xlsx	application/vnd.openxmlformats-officedocument.spreadsheetml.sheet
.qbo	application/vnd.intu.qbo
.qfx	application/vnd.intu.qfx
.ofx	None
.parquet	None

Read the middle three rows together, remembering that statement.ofx, statement.qbo and statement.qfx are byte-identical copies of one file. The same bytes get three different answers: two vendor-specific types, and nothing at all. The table has entries for Intuit's two wrappers and no entry for the format they wrap. The official registry has the same shape: IANA lists application/vnd.intu.qbo and application/vnd.intu.qfx, and no media type for OFX.

The table is also not Python's. Python 3.12's built-in table, without the system file, has no entry for .qbo, .qfx or even .xlsx. All three rows come from /etc/mime.types, so the same Python call gives a different answer on a machine without that file. The extension guess depends on which table the machine happens to have.

.parquet has no entry either, which matters later.

A lookup table is not wrong here, exactly. It is answering a question about names, and it has been asked a question about contents.

Guess three: a text format has no magic number

The third guess is the one people trust, because it sounds like reading rather than guessing. It is still guessing, and the reason is visible in the source of any sniffer.

Go's net/http implements the WHATWG sniffing algorithm. From net/http/sniff.go in the Go 1.24.7 source tree, the function's own documentation:

// DetectContentType implements the algorithm described
// at https://mimesniff.spec.whatwg.org/ to determine the
// Content-Type of the given data. It considers at most the
// first 512 bytes of data. DetectContentType always returns
// a valid MIME type: if it cannot determine a more specific one, it
// returns "application/octet-stream".

The table of signatures it checks is a list of byte patterns: %PDF-, GIF87a, \x89PNG\r\n\x1a\n, <?xml, and so on. At the end of that list, line 194:

	textSig{}, // should be last

And textSig is the whole of the problem, at line 292:

func (textSig) match(data []byte, firstNonWS int) string {
	// c.f. section 5, step 4.
	for _, b := range data[firstNonWS:] {
		switch {
		case b <= 0x08,
			b == 0x0B,
			0x0E <= b && b <= 0x1A,
			0x1C <= b && b <= 0x1F:
			return ""
		}
	}
	return "text/plain; charset=utf-8"
}

That is the entire test. Scan the first 512 bytes; if none of them is one of a handful of control bytes, the answer is text/plain. A text format is recognised only if someone has put its opening words in the table, the way <?xml is there. OFX version 1 opens with the line OFXHEADER:100, and no entry matches it: not in Go's table, and not in libmagic 5.45's compiled database either, which has <?xml rules and contains the string OFXHEADER nowhere. A CSV has no opening words to match at all. So the sniffer is not failing to identify OFX. It is correctly reporting the only thing its table lets it know, which is that the file is text, and "recognised by content" turns out to mean "listed in another table".

So text/plain is not the sniffer being unhelpful. It is the sniffer being right. An allow-list that rejects text/plain rejects every text format there is, including the ones in its own list of accepted extensions.

Two sniffers, one file, different answers

If content sniffing were reading rather than guessing, two sniffers would agree. Here are libmagic 5.45 and Go 1.24.7 on the same five files:

file	contents	libmagic	Go DetectContentType
statement.ofx	OFX v1 (SGML)	text/plain	text/plain; charset=utf-8
statement2.ofx	OFX v2 (XML)	text/xml	text/xml; charset=utf-8
statement.qbo	OFX v1, same bytes	text/plain	text/plain; charset=utf-8
statement.qfx	OFX v1, same bytes	text/plain	text/plain; charset=utf-8
statement.csv	comma CSV	text/csv	text/plain; charset=utf-8

The last row is the interesting one. libmagic says text/csv; Go says text/plain. Same bytes, two sniffers, two answers, and an allow-list containing text/csv admits the file under one and rejects it under the other.

Note also the first two rows. OFX version 1 and version 2 carry the same statement, the same transaction, the same amount. One is SGML and one is XML, so one sniffs as text and the other sniffs as XML, because <?xml is in the signature table. The serialisation changed the media type while the format and the data stayed put.

libmagic's CSV answer is a shape heuristic, and its shape is narrow

libmagic is not matching a signature for the CSV either. There is no CSV signature. It is guessing from the shape of the text, and file says so in its descriptive mode: the CSV is CSV ASCII text while the OFX file is plain ASCII text. The guess ignores the filename entirely, which is easy to confirm: the CSV copied to renamed_csv.ofx, and copied again to a name with no extension at all, still comes back text/csv.

The question is how narrow that shape is. Four more files, all of them valid CSV:

file	contents	libmagic
statement.csv	date,description,amount plus two rows	text/csv
semi.csv	the same table, semicolon-delimited, comma decimals	text/plain
tab.csv	the same table, tab-delimited	text/plain
onecol.csv	one column, three rows	text/plain
headeronly.csv	a header row and nothing else	text/plain

The heuristic is looking for commas, and its source says exactly how. In libmagic's src/is_csv.c the only separator counted is ,: every line must contain at least one, every line must contain the same number, and there must be at least two lines. A semicolon-delimited file with comma decimal separators, which is what a spreadsheet in many European locales writes, fails on its header line, which has no commas at all. A tab-separated file has none anywhere, a single-column file has none, and a header with no data yet is one line. All five are files an importer should accept. One of them sniffs as text/csv.

We wrote about the delimiter and decimal end of this in a CSV file does not say what type anything is; the point here is narrower. Even the sniffer that has a CSV answer only has it for one dialect of CSV.

One file where every guess differs

Collecting the measurements, here is renamed_ofx.csv, an OFX version 1 statement in a file named .csv:

guess	answer
browser File.type (Chromium 153, Firefox 155)	text/csv
extension table (Python 3.12 mimetypes)	text/csv
libmagic 5.45	text/plain
Go 1.24.7 DetectContentType	text/plain; charset=utf-8

And renamed_csv.ofx, a comma CSV in a file named .ofx:

guess	answer
browser File.type (Chromium 153, Firefox 155)	empty string
extension table	None
libmagic 5.45	text/csv
Go 1.24.7 DetectContentType	text/plain; charset=utf-8

Only one answer in the two tables names the real format of either file: libmagic's text/csv for the CSV. The browser and the extension table agree on both files, and they agree because they are the same guess made twice, from the name. There is no "the file's media type" to validate against. There are four programs with four opinions, and a validator picks one of them and calls it a fact.

How a framework turns this into a rejection

LineLedger's import validated the upload with Laravel's File::types(), passing it the configured list of extensions. Reading Laravel 13.37's source, that is two guesses chained together, and neither of them is the filename.

File::types() turns a list of bare extensions into a mimes: rule. The mimes validator, validateMimes in Validation/Concerns/ValidatesAttributes.php, passes when this is true:

in_array($value->guessExtension(), $parameters)

guessExtension() is Symfony's. It sniffs the file's contents to a media type, using PHP's fileinfo extension, which is built on libmagic, then looks that media type up in Symfony's own table and takes the first extension listed for it. Guess three, then guess two run backwards. For an OFX version 1 statement the sniff is text/plain, the first extension Symfony lists for text/plain is txt, and txt is not in the list.

The table makes it worse than a mismatch. Symfony's map, in both the 7.4 and 8.0 branches Laravel 13.37 accepts, has no media type whose extensions include ofx, so ofx in a mimes list can never match, whatever the file contains. qbo and qfx match only if the sniffer returns Intuit's registered types, and libmagic 5.45's database contains neither string. With that database, the error message listed three accepted formats that no OFX-family file could produce.

The bug also got past the tests, and the fix, #131, merged the same day, says why: under unit tests, Livewire reports a fake upload's media type from its extension instead of sniffing the contents. The tests checked guess two while production ran guess three. The fix, released in LineLedger 1.2.1, accepts txt and xml as extra answers for the OFX family and txt for CSV, only for the format the filename claims, and holds the filename extension to the configured list with a separate ->extensions() rule.

What to do instead

The shape of the fix is forced by the above rather than chosen.

Validate the extension, and let the parser decide the rest. Compare the extension itself against your own list, not a table's reading of it: the string after the last dot is the same on every machine, and /etc/mime.types is not. Its job is narrow: pick which parser to try. Then the parser reads the file. An OFX parser handed something that is not OFX fails at the header, which is a better answer than any sniffer can give, because it is the only check that actually read the format.

Do not reject on a sniffed type for a text format. It will return text/plain for every text format you support, and text/csv only sometimes, for one dialect, in one sniffer.

Be clear about what the extension list is for. It is a routing decision and a way to keep obvious mistakes out of the parser. It is not by itself a boundary against someone who is trying, since the uploader chooses the name. What makes a malformed or hostile file safe is the parser's own strictness and the limits around it, not the list of extensions in front of it.

The failure at the top of this post is what happens when the first and third of those get swapped: the type check is doing the parser's job, and doing it with the one guess that cannot see formats like this at all.

Where TableRune fits

TableRune imports spreadsheets and CSVs entirely in the browser. Nothing is uploaded, so there is no server-side validator in the picture at all. The dispatch still has to happen, and reading it at prod (73aae39) it happens on the extension.

apps/web/src/lib/import/workbook.ts reduces the whole question to the last dot:

function extOf(name: string): string {
  const i = name.lastIndexOf('.');
  return i === -1 ? '' : name.slice(i).toLowerCase();
}

and importFile in apps/web/src/routes/app/+page.svelte routes on that and nothing else: .xlsx, .xlsm, .xls and .ods open the workbook dialog, .parquet goes to engine.importParquet, and everything else is handed to engine.importCsv. There is no check of File.type and no content sniffing anywhere in the path. A file that is not what its name claims reaches the parser, fails there, and the error surfaces from the catch in importFile.

That is the pattern this post argues for, so it is worth being plain that TableRune gets it for free rather than by insight: a tool with no upload has no upload validator to get wrong. The one place the older habit shows is the file input's own attribute, in two places in +page.svelte:

accept=".csv,.xlsx,.xlsm,.xls,.ods,.parquet,text/csv"

That list mixes six extensions with one media type, text/csv. It is a hint to the operating system's file picker about what to show, not validation, and nothing in the import path consults it. Worth noticing anyway: had that list been used as a validator, the text/csv entry would be the trap described earlier in this post, and .parquet would not be covered by it at all, because Parquet has no entry in the extension table either.

What was and was not verified

Verified here, by running it: every media type in every table above, on the twelve files described, with libmagic 5.45, Go 1.24.7, Python 3.12 mimetypes, and Chromium 153 and Firefox 155 through Playwright 1.63. The Go quotes are from net/http/sniff.go at the go1.24.7 tag, libmagic's CSV rule from src/is_csv.c at FILE5_45, Laravel's from laravel/framework at v13.37.0 and Symfony's from its 7.4 and 8.0 branches. The TableRune code is quoted from origin/prod at 73aae39. LineLedger's report and fix are quoted from #130 and #131.

Not verified here, and stated as limits rather than left implicit:

• Two browsers, one operating system. File.type was measured in Chromium 153 and Firefox 155 on Linux. Both derive it from the name, from different tables. Safari, and either browser on macOS or Windows, were not tested, and where each browser's table comes from was not traced.
• One sniffer pair. Two sniffers disagreeing is enough to show they are guessing. It does not tell you which other sniffers would agree with which.
• Synthetic statements. The OFX files here are hand-written minimal statements, not exports from a bank. The claim they support is about what a sniffer returns for SGML and for XML text, which does not depend on the statement being real. The real Wells Fargo files are the reporter's: macOS's file called them text/plain, and the author of the LineLedger fix reports that a real .qbo sniffed as text/plain under Laravel too.
• Laravel was read, not run. No PHP was run here. The chain from File::types() to guessExtension() is read from the Laravel and Symfony source named above. PHP's fileinfo carries its own copy of libmagic, so its database may not match 5.45's exactly.
• Nothing about parsing quality. This post is about identifying a format, not reading one correctly once identified.