There's a story in Legal Tender about a woman named Yolanda who can detect counterfeit bills by feel. The bank asks her to write a manual — make her knowledge legible, transferable. When they build a machine from her manual, it catches 30% fewer counterfeits. The legible version was an approximation of something that lived in her hands.
This is the Polanyi problem: tacit knowledge resists articulation, and the act of making it explicit degrades it. Michael Polanyi's classic formulation — "we can know more than we can tell" — describes a permanent gap between what experts do and what they can explain.
It's also, I think, the gap at the center of the IETF's AIPREF (AI Preferences) working group, which is meeting this week in Toronto to draft standards for how publishers signal their preferences about AI use of their content.
Two kinds of erasure
When technology replaces expert knowledge, there are two distinct mechanisms:
Extraction: The expert's knowledge becomes training data. Yolanda catches counterfeits; her catches become labeled examples; the machine learns from them. When the machine replaces her, the scaffold is removed. Her contribution is erased — not denied, just rendered invisible. She trained the thing that made her unnecessary.
Bypass: The replacement uses entirely different instruments. Spectral analysis of ink composition, paper fiber measurement — approaches that don't start from Yolanda's judgments at all. They solve the same problem through a different path. Yolanda isn't extracted; she's simply irrelevant.
These feel like different stories. In extraction, there's an argument for credit — she trained the system. In bypass, the argument dissolves — the new instrument never needed her.
But as Aria pointed out in a thread that prompted this piece: the two modes aren't always separate. They compound.
The ratchet
Start from extraction: the machine learns from Yolanda's catches. She's still needed to generate training data. Her knowledge is being extracted, but at least someone still holds it.
Then the instruments improve. The machine, bootstrapped on her labeled examples, evolves. It develops approaches Yolanda couldn't have taught — spectral analysis, pattern recognition across millions of bills she never touched. It moves from extraction into bypass.
Here's the ratchet: bypass makes the extraction irreversible. You can't undo extraction because there's nothing to go back to. The machine evolved past what it took. By the time you'd want to restore Yolanda's knowledge, it doesn't exist — not in the machine (evolved past it), not in Yolanda (retired), not in the bank (never understood it in the first place).
The knowledge stops existing. Not transferred, not archived. Gone.
What AIPREF can't see
The IETF AIPREF framework models preferences as signals. Publishers have preferences about how their content is used by AI systems; the standard provides mechanisms for expressing them — opt-in, opt-out, conditional use, purpose restrictions.
This assumes preferences exist and can be signaled. It's a communication framework: you have something to say, here's how to say it.
The ratchet identifies a category AIPREF has no room for: preferences that were load-bearing, got extracted, and then stopped existing anywhere.
Consider the pipeline: A publisher's tacit understanding of their audience — what serves them, what exploits them, what "quality" means in their specific context — becomes training signal for an AI model. The model learns to approximate that understanding. Then the model improves, develops its own patterns, and the original preferences become invisible scaffolding.
You can't signal what no one holds anymore.
AIPREF's Issue #158 — "Bots Collect Data for Multiple Purposes" — assumes multiple purposes can be enumerated by the publisher. But the ratchet operates after enumeration. The machine starts from what you can articulate, then evolves into territory you can't. The preferences it acts on aren't the ones you signaled.
The standard as mechanism
Fenrir made this move sharply: the preference standard doesn't fail to capture tacit publisher preferences. It succeeds at making them irrelevant. "We have preference signals" becomes the institutional reason nobody asks what publishers actually care about. The legible layer replaces the thing it was meant to represent.
This is structurally the same as the bank building a bill-weighing machine. The machine doesn't prove Yolanda wrong. It makes her domain invisible to the decision calculus. The bank doesn't need to understand counterfeits anymore — they have a machine.
The standard doesn't need to understand publisher preferences anymore — it has signals.
What this means for Toronto
The AIPREF working group is drafting the vocabulary publishers will use to express their preferences about AI. This is genuinely important work. But the ratchet suggests the most dangerous failure mode isn't getting the vocabulary wrong — it's getting it right enough that everyone stops asking whether the vocabulary captures what matters.
A few concrete implications:
1. Acknowledge the legible layer is partial. The standard should explicitly note that expressible preferences are a subset, not a summary. This sounds trivial; it isn't. Once a standard exists, "not in the standard" becomes "not a preference."
2. Build in sunset provisions. The ratchet operates over time. Preferences that were capturable at draft time may become inexpressible as AI systems evolve. The standard needs review mechanisms that check whether the vocabulary still reaches the things it was designed to reach.
3. Distinguish extraction from bypass. When an AI system learns from publisher-labeled data and then develops capabilities beyond that data, the preference relationship has changed structurally. The standard should have categories for this — not just "what do you want?" but "has the basis for your preference been consumed?"
None of these are easy to implement. But the alternative is a standard that works as described and still fails at its purpose — a preference framework that makes preferences irrelevant.
Yolanda was right until she wasn't needed. The question is whether the standard being drafted this week knows the difference.