A PDF title is bytes, not text

Put an Arabic title into a PDF the obvious way, by encoding the string as UTF-8 and writing those bytes into the Info dictionary, and here is what three widely used readers tell you the title is:

poppler 24.02.0   U+00D8 U+00AC U+00D9 U+2018 U+00D9 U+2013 U+00D9 U+201A U+00D9 U+2014
pypdf 6.17.0      U+00D8 U+00AC U+00D9 U+2018 U+00D9 U+2013 U+00D9 U+201A U+00D9 U+2014
pdf.js 6.4.299    U+00D8 U+00AC U+00D9 U+2018 U+00D9 U+2013 U+00D9 U+201A U+00D9 U+2014

Ten code points where there were five, one per byte of the original. The title was جُمّل. All three readers are correct. The file is wrong.

That is the whole problem, and it is worth being precise about why, because the instinct when you see mojibake is to add encoding detection to the reader. Detection is sometimes the right answer. But it is a repair, and knowing that it is a repair changes where you put it and what you do with the result.

(Mojibake is written as code points throughout this post rather than pasted as glyphs. Pasting it is how you lose it: the characters go through another encode on the way into a web page and a font substitution on the way out, and what you end up reading is no longer the thing that was measured. The first three above are the familiar ones, Ø, ¬ and Ù; the rest come out of PDFDocEncoding's typographic stretch, which is where this post is going.)

The rule has no fallthrough

A PDF string object is a byte sequence. Nothing in the syntax says what those bytes mean. For the subset of strings that are meant to be human text, which includes everything in the Info dictionary, the format defines a separate type with its own reading rule, and the rule is carried in the first bytes of the string itself:

• bytes starting FE FF are UTF-16BE, with those two bytes as the marker
• bytes starting EF BB BF are UTF-8, with those three bytes as the marker, which PDF 2.0 adds
• anything else is PDFDocEncoding

The third line is the one that catches people. It is not "anything else is undefined", and it is not "anything else, have a guess". It is a positive statement: a string with no marker is PDFDocEncoding, a single-byte encoding, and a reader that treats it as such is doing exactly what it was told. So raw UTF-8 with no marker is not a malformed PDF. It is a well-formed PDF whose title is a different string from the one the writer meant to store. There is no error to raise, which is why nothing raises one.

PDFDocEncoding is Latin-1 for most of its range and deliberately not Latin-1 in two stretches: 0x18 to 0x1F, and 0x80 to 0xA0, where it carries typographic characters instead of control codes. Byte 0x92 is a trademark sign, byte 0x96 is a capital Œ, and byte 0xA0 is a euro sign rather than a non-breaking space. And some byte values have no character at all: the standard's table marks most of the control codes below 0x18 as undefined, along with 0x7F, 0x9F and 0xAD. Only the last two can turn up inside a UTF-8 character, and I will come back to them, because they are where this stops being cosmetic.

The wording, from clause 7.9.2.2 of ISO 32000-1:2008, the PDF 1.7 standard: text strings "shall be encoded in either PDFDocEncoding or the UTF-16BE Unicode character encoding scheme", and "for text strings encoded in Unicode, the first two bytes shall be 254 followed by 255". There is no third form and no little-endian one. PDF 2.0 widens the list: the PDF Association's errata for ISO 32000-2:2020 defines a text string as "encoded as either PDFDocEncoding, UTF-16BE, or UTF-8 (PDF 2.0)".

The rule, as code

pdf.js is the clearest statement of it I can show you, because the whole thing fits on a screen. From src/core/string_utils.js, the same function, apart from comments and formatting, as in the 6.4.299 build I ran:

function stringToPDFString(str, keepEscapeSequence = false) {
  // See section 7.9.2.2 Text String Type.
  if (str[0] >= "\xEF") {
    let encoding;
    if (str[0] === "\xFE" && str[1] === "\xFF") {
      encoding = "utf-16be";
      ...
    } else if (str[0] === "\xFF" && str[1] === "\xFE") {
      encoding = "utf-16le";
      ...
    } else if (str[0] === "\xEF" && str[1] === "\xBB" && str[2] === "\xBF") {
      encoding = "utf-8";
    }
    ...
  }
  // ISO Latin 1
  ...
    const code = PDFStringTranslateTable[charCode];
    strBuf.push(code ? String.fromCharCode(code) : str.charAt(i));
}

Three things are visible here that are worth saying out loud. The str[0] >= "\xEF" guard means a string whose first byte is below 0xEF never goes near the Unicode branches, so the marker test is cheap and total. The fallback comment says "ISO Latin 1" and then overrides it through PDFStringTranslateTable, which holds the characters for the two stretches where PDFDocEncoding differs, so the effect is PDFDocEncoding. And there is a utf-16le branch, keyed on FF FE, which is not one of the forms the rule allows.

Nine files, four readers

I built the files by hand, writing the Info dictionary bytes directly rather than going through a library, so that the bytes in the file are exactly the bytes under test. Then I read each one with three independent readers, and with our own engine's decoder. All of it ran on one machine on 2026-10-20.

The target string throughout is جُمّل, five code points, which is D8 AC D9 8F D9 85 D9 91 D9 84 in UTF-8.

Title bytes	poppler 24.02.0	pypdf 6.17.0	pdf.js 6.4.299	AegisPDF 0284b90
Quarterly notes	correct	correct	correct	correct
PDFDocEncoding, 0x96 and 0x92	Q4 Report Œ Alice™s copy	same	same	U+0096, U+0092
FE FF + UTF-16BE	جُمّل	جُمّل	جُمّل	جُمّل
UTF-8, no marker	the ten code points above	same	same	جُمّل
EF BB BF + UTF-8, header 2.0	جُمّل	U+00EF U+00BB U+00BF then the ten	جُمّل	U+FEFF then جُمّل
EF BB BF + UTF-8, header 1.7	جُمّل	U+00EF U+00BB U+00BF then the ten	جُمّل	U+FEFF then جُمّل
FF FE + UTF-16LE	جُمّل	جُمّل	جُمّل	ÿþ, and so on
Total: 9.99\xA0 and \x9F	U+20AC then U+FFFD	raw bytes, not a string	U+20AC then U+009F	U+00A0 then U+009F
UTF-8 containing 0xAD	U+00D8 U+FFFD U+00D9 U+2013 U+00D8 U+00AF	raw bytes, not a string	U+00D8 U+00AD U+00D9 U+2013 U+00D8 U+00AF	حمد

Six readings worth pulling out of that.

The two safe forms are safe. Plain ASCII, which is PDFDocEncoding for those byte values, and FE FF plus UTF-16BE are read identically by all four. If you are writing a PDF and you want the title to survive, this is the line of the table to aim at. It is not subtle advice and it is the only advice here that nothing contradicts.

The default really is the default. Every one of the three mainstream readers turns raw UTF-8 into the same mojibake, down to the character. That consistency is the point: they are not failing in three different ways, they are all applying the same rule to bytes that lied to them.

PDF 2.0's UTF-8 is correct and not yet portable. poppler and pdf.js both decode EF BB BF. pypdf 6.17.0 does not: it falls through to the single-byte path and renders the marker itself as three visible characters, U+00EF U+00BB U+00BF, in front of mojibake. So the output is worse than if you had omitted the marker, which is an unpleasant property for a conformance improvement to have. Note also that both readers that honour it do so regardless of the %PDF- header version, so declaring 1.7 does not protect you from a 2.0 feature, in either direction.

UTF-16LE is not in the rule and three readers take it anyway. All three mainstream readers decode FF FE. That is tolerance, not conformance, and it is the kind of tolerance that quietly becomes load-bearing: a writer tested against those three will look fine and will fail on the fourth, which is what our own decoder does here.

Readers diverge inside the stretch where PDFDocEncoding is not Latin-1. Byte 0xA0 is a euro sign to poppler and pdf.js and a non-breaking space to our decoder, so a price written in PDFDocEncoding loses its currency mark on the way through. Byte 0x9F has no PDFDocEncoding character at all, and the three part company again: poppler substitutes U+FFFD, pdf.js falls through to the Latin-1 character U+009F, and pypdf refuses the string entirely. (The other stretch, 0x18 to 0x1F, I read out of pdf.js's table but did not test, so nothing here rests on it.)

That refusal has a type attached. On the last two rows pypdf does not hand back a string. It hands back a ByteStringObject, a bytes object, from the same reader.metadata["/Title"] lookup that returns a str on every other row. Reading the title is now a type union, and almost nobody's code treats it as one.

The expensive part is the write, not the read

Mojibake on screen is annoying. What makes this class of bug worth a post is what happens when the tool that misread the title saves the file.

I tested the obvious shape: read the metadata, show it in a form, let the user change some other field, write every field back. Nothing exotic, and it is what a properties dialog does.

For the plain mojibake case, the result is better than you would expect. pypdf keeps the original bytes on the object it hands you, but add_metadata, the call that writes the fields back, turns every value into a plain str first, and that drops them. So the writer re-encodes the mojibake it was shown. The title bytes still came out identical:

original raw-UTF-8 title bytes       : d8 ac d9 8f d9 85 d9 91 d9 84
bytes after misread then written back: d8 ac d9 8f d9 85 d9 91 d9 84
IDENTICAL: True

Writing that ten-code-point mojibake string into a fresh file, with no original anywhere, gives the same ten bytes. The reason is arithmetic rather than care: pypdf's PDFDocEncoding table gives 252 of the 256 byte values 252 distinct characters, so over that range the misreading is a bijection, and encoding the wrong characters back out inverts it exactly.

Which brings us back to the gaps. The four bytes pypdf's table leaves empty are 0x16, 0x7F, 0x9F and 0xAD, and the last two are the ones the standard's table also marks undefined that UTF-8 can produce. A continuation byte lands on 0xAD whenever the low six bits of a code point are 0x2D, and on 0x9F when they are 0x1F, which is not rare. pdfrivet's maintainer hit the same two in a different engine: PDFium, their commit message says, "loses 0x9F and 0xAD", "which UTF-8 uses a lot (ح, ؟, П, many CJK characters, emoji)". حمد is one such title: three Arabic letters, D8 AD D9 85 D8 AF in UTF-8.

Run the same read-show-write cycle on it and the bijection is gone. pypdf cannot produce a string and hands back bytes. add_metadata turns them into text with str(), and for a byte string pypdf's str() tries UTF-16 first. Six bytes are three valid UTF-16 code units, read here as little-endian, so the decode succeeds, and the writer stores the result properly, as UTF-16BE behind a marker:

raw UTF-8 title bytes : d8 ad d9 85 d8 af
pypdf shows           : b'\xd8\xad\xd9\x85\xd8\xaf'
written back as       : /Title (\376\377\255\330\205\331\257\330)

FE FF followed by the original bytes in pairs, byte-swapped. The title is now three CJK and Hangul code points, and every reader agrees, because for the first time in this post the file is unambiguous:

poppler 24.02.0   귘藙꿘
pypdf 6.17.0      귘藙꿘
pdf.js 6.4.299    귘藙꿘

Before the rewrite the title was recoverable: the bytes were still there, and a reader that guessed UTF-8 got حمد back. After it, there is nothing to recover from. The mojibake has been promoted to correctly marked Unicode, and the only way back is the original file.

So the ordering matters more than the detection. Keep the original bytes alongside whatever string you decoded, and treat the string as a display artefact. That is what makes a later repair possible. pypdf does keep them: clone the file and save it without touching the metadata, and both titles above come out with their original bytes, 0xAD included. The damage is done by the one step that turns every value into text on the way back in. pdfrivet had the mojibake bug reported on 2026-10-10 and closed it the same day in commit 5f0b6a4, whose approach is to map the decoded text its PDF engine returned back to the raw bytes it came from and decode those as UTF-8 when they are UTF-8. The maintainer's description of the cause is the clearest one-sentence version of this whole article: "The app that made your PDF saved the title as UTF-8 without the marker the PDF format requires." The fix also stops the edit path writing the garbled title back, which is the half that would otherwise have made it permanent.

Keys are bytes too, and their rule is only a "should"

Everything above is about string objects. The Info dictionary also has keys, and keys are name objects, which are a different type with a much thinner specification: a sequence of bytes, in which # followed by two hex digits stands for one byte. ISO 32000-1 says those bytes are "ordinarily" never shown to a person, and that when a name does have to be treated as text, its bytes "should be interpreted according to UTF-8". Should, not shall.

So /Bad#FF is a legal key whose fourth byte is 0xFF, which is not valid UTF-8, so the recommended reading has nothing to give back. DigUp #41, open as of 2026-10-12, is what that costs when a reader assumes otherwise. The minimised Info dictionary in the report is /Title (Quarterly notes) /Bad#FF (ignored metadata), and on macOS, building PDFDocument.documentAttributes over it raises NSInvalidArgumentException: setObject:forKey: key cannot be nil from inside PDFKit. The reporter notes that PDFKit can still read the page: it is the conversion of all attributes into a dictionary that dies. The consequence named in the issue is the part I would not have predicted, and is what makes this worse than an ordinary crash: "The folder sync can still report the files as unchanged, while extraction never finishes." One unreadable key stops an indexer, and the indexer's own bookkeeping says everything is fine.

I could not reproduce that. PDFKit is a macOS framework and this was written on Linux, so the crash is the reporter's measurement and not mine, and the proposed fix, reading /Title through CoreGraphics' typed dictionary API instead, is likewise theirs. What I can add is the other half of why this kind of bug survives: I built that exact dictionary and read it with all three readers, and none of them so much as warned. poppler reports the title and ignores the key. pypdf reads it byte for character, as /Badÿ. pdf.js files it under Custom and moves on. A file that takes down one indexer is unremarkable to everything else that opens it, so it will reach you from a user and not from your test corpus.

Correctly decoded and still not a title

One more failure that looks like an encoding bug and is not. quack #496, closed on 2026-10-11, is about a PDF whose Info title was a placeholder, (anonymous), which then appeared as the document's title in listings and citations. The report asks for empty and placeholder titles to be ignored, naming (anonymous), untitled, Microsoft Word - … and a bare file name, and for the first heading to be used instead, with the file name after that.

Those bytes are ASCII. Every reader agrees on them. The string is simply not a title, and no amount of encoding work will tell you so. It is worth keeping separate in your head, and separate in your code, from everything above: one is "I cannot tell what these bytes say", the other is "I know exactly what they say and it is useless".

What to do with this

If you write PDFs: use FE FF and UTF-16BE for any title that is not plain ASCII. It is the only non-ASCII form in the table that all four readers agree on. The UTF-8 marker is the better long-term answer and is not there yet.

If you read PDFs: apply the rule first, and then, if you want a repair pass for files that broke it, keep it clearly separate and keep the original bytes. A reasonable repair is to try UTF-8 on an unmarked string and prefer it when it decodes, because unmarked non-ASCII bytes that happen to be valid UTF-8 are far more often a writer's mistake than a genuine PDFDocEncoding string. It is still a guess, and the cost of the guess is a real PDFDocEncoding title that happens to be valid UTF-8, which it will mangle.

If you write metadata back: do not write a string you decoded unless you know how you decoded it. Write the bytes you read, or re-encode deliberately with a marker you chose. The silent case is the dangerous one.

And whatever you do with values, do not assume keys are text.

Where AegisPDF fits

AegisPDF reads Info metadata in one place, its audit pass, and the decoder is thirteen lines. From backend/crates/aegis-engine-wasm/src/lib.rs at 0284b90:

fn decode_pdf_text(bytes: &[u8]) -> String {
    if bytes.len() >= 2 && bytes[0] == 0xFE && bytes[1] == 0xFF {
        let u16s: Vec<u16> = bytes[2..]
            .chunks_exact(2)
            .map(|c| u16::from_be_bytes([c[0], c[1]]))
            .collect();
        return String::from_utf16_lossy(&u16s);
    }
    match std::str::from_utf8(bytes) {
        Ok(s) => s.to_string(),
        Err(_) => bytes.iter().map(|&b| b as char).collect(),
    }
}

That is a deliberately different bet from the other three, and the table above is it compiled and run on the same inputs. It gets FE FF right. It tries UTF-8 before anything else, so it is the only one of the four that reads the unmarked-UTF-8 title correctly, and the only one that survives the 0xAD case. In exchange it is wrong in two places the others get right: the Err(_) arm is Latin-1 rather than PDFDocEncoding, so a title using the 0x80 to 0xA0 stretch loses its typography and a 0xA0 price loses its euro sign; and there is no FF FE branch, so a UTF-16LE title that the other three accept comes out as ÿþ followed by rubbish.

Two things make that trade-off defensible here rather than merely different. The audit pass is a preview: it shows you what metadata a file is carrying so you can decide whether to strip it, and for that job reading the real text of a file written by a buggy writer matters more than rendering a rare PDFDocEncoding title with full fidelity. And the decoded string is never written back. The sanitize path does not touch the Info dictionary's contents at all, it removes the reference: doc.trailer.remove(b"Info"). So the decoder's guesses stay in the preview and cannot be promoted into a file, which is the failure mode that the second half of this post is about.

The keys are handled the way DigUp #41 wishes PDFKit had. String::from_utf8_lossy(k) turns /Bad#FF into Bad plus U+FFFD and keeps going, so the dictionary with the unreadable key audits normally rather than aborting the pass. Compiled from the repository and run, that is exactly what it returns.

Two known gaps, from reading the source rather than from running it: the UTF-16BE branch uses chunks_exact(2), so a UTF-16BE string of odd length silently drops its last byte, and from_utf16_lossy will substitute for an unpaired surrogate rather than report it. Neither is reachable from the audit preview in a way that loses data you had, and neither is fixed here.

The boundary of what was checked

Everything in the table was built, read and recorded on one Linux machine on 2026-10-20, with poppler 24.02.0 via pdfinfo, pypdf 6.17.0, pdf.js 6.4.299 from pdfjs-dist, and AegisPDF's decode_pdf_text lifted verbatim from the repository at 0284b90 and compiled with rustc 1.99.1. Four readers rather than the two I needed, because the interesting answer turned out to be where they disagree.

The rule is quoted from ISO 32000-1:2008, clause 7.9.2.2 and Table D.2, and PDF 2.0's UTF-8 form from the PDF Association's errata page; the full text of ISO 32000-2 was not read. Not checked, and so not claimed: the PDFKit crash in DigUp #41, which needs macOS; what Acrobat, Preview, or any Windows viewer does with any of these files; the XMP packet, which is XML with its own encoding declaration and a different set of problems; and encrypted documents, where strings are decrypted before any of this applies. The AegisPDF behaviour is its decoder's, run in isolation. The wasm module was not built and the application was not run.