The production of bullshit
Bullshit jobs, туфта, greenwashing, cargo cults, and the difficulty of sifting the real from the fake
January 6, 2025 — January 7, 2025
Suspiciously similar content
One of the problems of the modern world is that it is so deep, specialised, and complicated that it is difficult hard to tell real progress in some specialised areas from bullshit. This is a problem in science, but I think, everywhere that anything complicated is happening
This leads to the deeper problem that it is easier to seem good than to be good. Sometimes, e.g. art, these can be nearly the same thing. But sometimes, with the production of material goods or the production of knowledge, it can take a long time to tell the difference, and yet the difference can be very important.
A masterful summary by Sarah Constantin:
Concepts like Goodhart’s Law, cargo-culting, greenwashing, hype cycles, Sturgeon’s Law, even bullshit jobs are all pointing at the basic understanding that it’s easier to seem good than to be good, that the world is full of things that merely appear good but aren’t really, and that it’s important to vigilantly sift out the real from the fake.
I would add some other examples to her list: tufta, the replication crisis (Ioannidis 2005), etc.
Her coda is really good too:
But then sometimes I meet people who talk as though they aren’t tracking “how do we make sure we’re tackling the main problem?” or “well, most things in this space are {unreplicable, failures, trivial, biased, error-prone, etc} so how are we ensuring we select for the few actually-good ones”?
What follows is a taxonomy of fake production and some ideas about ameliorating it.
I know I called this “bullshit” and that as such we are in danger of being trite, and complaining about those bullshitters over there. But the thing is, there is no way out of the game. Bullshitting is something we must all struggle to escape; even knowing if I myself am bullshitting is not always easy. Am I suffering from imposter syndrome if I don’t oversell? Or am I feeding the hype cycle if I do?
1 Proxies for good things only work for a while
The bullshit problem is IMO a special case of the alignment problem. This was put beautifully by Paul Christiano
We will try to harness this power by constructing proxies for what we care about, but over time those proxies will come apart:
- Corporations will deliver value to consumers as measured by profit. Eventually this mostly means manipulating consumers, capturing regulators, extortion and theft.
- Investors will “own” shares of increasingly profitable corporations, and will sometimes try to use their profits to affect the world. Eventually instead of actually having an impact they will be surrounded by advisors who manipulate them into thinking they’ve had an impact.
- Law enforcement will drive down complaints and increase reported sense of security. Eventually this will be driven by creating a false sense of security, hiding information about law enforcement failures, suppressing complaints, and coercing and manipulating citizens.
- Legislation may be optimized to seem like it is addressing real problems and helping constituents. Eventually that will be achieved by undermining our ability to actually perceive problems and constructing increasingly convincing narratives about where the world is going and what’s important.
He is talking about AI safety in particular, but I do not see anything AI-specific in there.
2 What structures minimise bullshit?
TBC
3 Bureaucracy
Production of endless KPIs and reports and summit documents is a feature of bureaucracy, and anyone who has worked in one will tell you. When do these function more to produce bullshit and when more to produce outcomes?
4 Moloch
c.f. Moloch.
5 Tufta
Also known as tukhta, туфта (tʊfˈta)(Solzhenit︠s︡yn 2003).
The Soviet one. Notable for the wonderful and hilarious examples of how the system was gamed by the workers, and the workers gamed the system.
Our job was to cut down trees, clean off the branches, cut the logs into segments of equal length, and stack them for hauling. But instead of performing this exhausting work, Nikolai’s mission was to find stacks left over from last year, of which there were plenty, and claim them. ‘All we need to do,’ Nikolai confidently explained, ‘is cut off the ends of the logs so they look freshly cut and bring the brigadier to measure them.’ […]] This eliminated the most exhausting work, allowing Nikolai and me to concentrate on what he called ‘cleaning the stacks’ – cutting an inch off each end of the log so it looked freshly prepared. We put our signatures on the fresh ends, buried the scraps we’d sawed off, and covered the hole with moss and branches. (Welsh 2024)
6 Goodhart’s Law
See Goodhart’s Law.
7 Cheap talk
See cheap talk.
8 Having skin in the game
TBC