The Dangerous Illusion of AI Coding — Jeremy Howard

The person who invented the pre-training and fine-tuning paradigm powering every modern LLM thinks the way most people use AI to code is fundamentally broken. Not because the models are bad, but because the workflow destroys the very thing that makes a software engineer valuable: understanding.

The Episode

Jeremy Howard joins Machine Learning Street Talk for an 87-minute conversation that spans the origins of ULMFiT and modern transfer learning, a sharp critique of AI coding workflows, cognitive science research on learning and friction, the distinction between coding and software engineering, and questions about power centralization and AI risk. Howard is characteristically direct: he sees an industry sleepwalking into a competence crisis, producing more code than ever while comprehending less of what they’ve built.

ULMFiT: The Accidental Foundation

Howard traces modern transfer learning to a deceptively simple idea: language models pre-trained on large corpora could be fine-tuned for specific tasks with dramatically less data. His 2018 ULMFiT paper with Sebastian Ruder introduced the three-stage process (pre-training, domain adaptation, task fine-tuning) along with techniques like discriminative learning rates and gradual unfreezing that became standard.

The origin was prosaic. Howard was frustrated that NLP was “20 years behind computer vision” in leveraging transfer learning. He tried the obvious thing: take a language model, fine-tune it, see what happens. The results on text classification were so strong they changed the field. GPT and BERT followed within months, both building on the same paradigm.

“There’s been surprisingly little acknowledgment of this work. But that’s fine, I guess. The important thing is that the idea got out there.”

Howard draws a particular irony: the paradigm he helped create now powers tools whose usage patterns he finds deeply problematic.

He also describes his approach to discovering ULMFiT as characteristic of his broader philosophy. Rather than starting from theory, he worked empirically: trying transfer learning in NLP, observing results, iterating. He contrasts this with the dominant academic culture that “starts with the math and then tries to find a problem to apply it to.” His approach is Bayesian: each experiment updates his priors, and the goal is understanding how things actually work rather than proving a preexisting theory.

The Slot Machine

Howard’s wife Rachel Thomas provided the core metaphor: AI coding has all the hallmarks of gambling addiction.

Illusion of control. You craft your prompt carefully, configure MCPs and skills files. It feels like mastery. But you pull the lever, and the outcome is stochastic.

Intermittent reinforcement. Occasionally it produces perfect code, so you believe you’ve cracked it. That’s just probability doing what probability does.

Losses disguised as wins. You spend 6 hours getting AI to generate working code. Writing it yourself might have taken 2 hours, and you’d understand every line.

“The thing about AI-based coding is that it’s like a slot machine in that you have an illusion of control.”

Howard validates the metaphor from personal experience. After a 14-hour Claude Code marathon, he needed days to recover. Most of what he built: “garbage.” His research found only a “tiny uptick” in actual software output, despite everyone feeling their productivity had multiplied. The METR study showed even worse: measured productivity actually declined, while developers believed it had improved.

“No one’s actually creating 50 times more high-quality software than they were before.”

The slot machine metaphor also explains the emotional dynamics. The intermittent reward schedule creates a compulsive loop: the next prompt might be the one that works perfectly. This keeps developers engaged in a workflow that feels productive but often isn’t, analogous to how casinos keep gamblers at machines through carefully engineered variable reward schedules.

Coding vs. Software Engineering

This is the sharpest edge in the conversation.

LLMs are good at coding. Coding is essentially style transfer: converting specifications into target language syntax, interpolating within the training data distribution. It’s no different from translating C++ to Rust or natural language to Python.

Software engineering is something else entirely. It means designing systems that have never existed, understanding business logic, handling edge cases, anticipating maintenance problems six months out. These require going beyond the training data distribution, which is exactly where LLMs break down.

“They’re really bad at software engineering. And then I think that’s possibly always going to be true.”

Howard cites Fred Brooks’ 1986 paper “No Silver Bullet”: the promises of 4GL languages to eliminate programming were identical to today’s claims. Brooks predicted at most a 30% improvement because the essential complexity of software was never about typing speed.

He’s blunt about industry leaders’ blind spots. Dario Amodei and Elon Musk, he argues, don’t have recent software engineering practice. “I’m not sure Dario has ever been a software engineer.” They equate coding speed with software engineering capability, which is a fundamental category error. If this holds, every current AI capability benchmark based on coding tasks measures the wrong thing.

The Chris Lattner case illustrates the point. Anthropic showcased Claude writing a C compiler. Lattner (LLVM’s creator) pointed out the “clean-room implementation” replicated LLVM’s code patterns, including parts Lattner himself considers design mistakes. That’s not innovation; it’s sophisticated copying with inherited flaws.

Code Nobody Understands

The IPykernel story crystallizes the problem.

IPykernel started crashing randomly after upgrading from v6 to v7: over 5,000 lines of complex multithreaded code, all tests passing but crashing in production. Howard spent two weeks with the strongest AI tools, repeatedly attacking the problem until the bug was fixed.

The result: he owns the only correctly implemented v7 protocol Python Jupyter kernel. But nobody understands this code, including him.

“Here’s a piece of code that no one understands. Am I going to bet my company’s product on it?”

Memory leaks? Breakage from a minor protocol adjustment next year? Nobody knows. This isn’t a question of whether AI is useful (it genuinely solved a bug humans couldn’t). It’s a deeper engineering question: when AI generates code that works but nobody understands, we’ve entered uncharted territory.

Tim Scarfe highlights how the erosion is gradual. You start with 10% AI-generated code. Six months later a PR comes in where 60% is AI-generated. You lose connection with your codebase without realizing it. The organizational knowledge degrades in a way that’s invisible until a crisis hits.

Why Friction Is the Feature

Howard draws on cognitive science, particularly Robert Bjork’s research on “desirable difficulty.” The core finding: learning that feels hard produces stronger, more durable understanding than learning that feels easy.

This maps directly to his critique. When you struggle through a problem manually, you build mental models. When an LLM hands you the answer, you skip that struggle and gain nothing durable. Howard uses GPS navigation as an analogy: studies show people who use GPS develop worse spatial reasoning than those who navigate with maps. The convenience is real, but so is the cognitive cost.

He extends the principle through John Ousterhout’s framework:

“A little bit of slope makes up for a lot of intercept.”

Intercept is how much you can produce right now. Slope is how fast you’re growing. If you only chase maximum output at AI’s current capability limit, you’re optimizing intercept at the expense of slope. That’s a path to obsolescence.

Anthropic’s own research confirms the concern: Claude Code users “learned almost nothing.” This directly contradicts Dario Amodei’s optimistic narrative, using Anthropic’s own data against their CEO’s thesis.

Howard practiced what he preaches. He spent 10 years learning Chinese with Anki, cards always scheduled at the moment just before forgetting: “excruciating but the results are amazing.” Fifteen years later, without review, he still remembers. The problem with AI coding is precisely that there’s too little friction to form durable learning.

As a CEO, his management philosophy runs counter to Silicon Valley mainstream: “The almost only thing I care about is how much your personal human capabilities are growing. I don’t care how many PRs you ship, how many features you build.” His judgment of enterprise CEOs embracing AI coding is harsh: if their bet is wrong (which he thinks is likely), and they have no way to tell it’s wrong (because this isn’t something their MBA taught them), they’re driving their companies toward destruction.

LLMs Cosplay Understanding

Howard engages with the philosophical question of whether LLMs truly understand anything. He lands on the skeptical side, but with nuance. He acknowledges models can demonstrate intelligence “in restricted domains.” The ARC challenge showed that: if you place constraints into the problem, you can go faster toward a known goal.

But he draws a sharp line between interpolation within training distributions and genuine reasoning about novel situations.

“LLMs cosplay understanding things. They pretend to understand things.”

He references the Dennett-Searle debate but adds a practical dimension. Even agency and planning capabilities don’t solve the fundamental problem: “You can have all the intelligence and agency in the world, but if you don’t have the knowledge and the constraints, then you’re going in the wrong direction faster.” Models don’t actually know the world; they know patterns in text about the world.

This isn’t a dismissal of utility. Howard uses Claude Code himself. But the human must remain the one who understands the system, with AI as an accelerator for execution rather than a replacement for comprehension.

The Notebook as Counter-Model

Howard offers a concrete alternative: the interactive notebook, specifically nbdev (which he built). Real software engineering, in his view, happens in an exploratory loop: you poke at data, observe results, build intuition, gradually construct understanding.

In a notebook environment, exploration, implementation, testing, documentation, and examples all live in one place. This forces intellectual honesty: you can’t just generate code; you have to show it working on real inputs. NBDev ships with out-of-the-box CI integration, so the entire cycle of exploration, testing, and production deployment is unified.

He contrasts this with the typical AI coding workflow where code gets generated into files nobody explores interactively. The notebook preserves the friction that produces understanding while leveraging computational tools for speed.

“It literally disgusts me. I literally think it’s inhumane. My mission remains the same as it has been for 20 years, which is to stop people working like this.”

This is directed at the line-based terminal interface model: “a 40-year-old environment” where your entire interaction with code happens through prompts. His vision, inspired by Bret Victor’s work, is tools that create direct, visceral connections between humans and the objects they manipulate inside computers. His new product Solve It embodies this philosophy: humans and AI collaborating inside the interpreter.

His programming practice reflects the principle: “I basically never need to use a debugger because I basically never have bugs. Not because I’m particularly good, but because I build in small steps, and at each step I can see it working. Bugs have no room to survive.”

Power Centralization: The Real AI Risk

The conversation turns to existential risk. Howard and Arvind Narayanan co-authored a rebuttal to the 2023 AI existential risk statement signed by Hinton, Hassabis, and others.

His argument is structural, not about capabilities. It’s an “even if” argument: even if AI becomes incredibly powerful, concentrating control of that technology is the wrong response.

“Even if it’s going to be incredibly powerful, you still shouldn’t centralize all of that power in the hands of one company or the government.”

The autonomous-AI-destroys-humanity narrative, in Howard’s reading, conveniently justifies exactly what power-hungry actors want: centralization of AI in a single node they can capture. He draws historical parallels with writing, printing, and voting rights: each time a transformative technology emerged, incumbents argued it was too dangerous for the masses. Each time, society benefited from broad distribution, not restriction.

His immediate risk assessment focuses on something already happening: “enfeeblement.” People losing the capacity to become more capable because they’ve delegated cognitive tasks to AI. This connects directly to his coding concerns and represents, in his view, a more concrete and currently unfolding danger than speculative scenarios about autonomous AI.

On privacy, Howard offers a sober take: AI doesn’t create qualitatively new risks beyond what Google, Microsoft, and data brokers like Palantir already posed. The US government is prohibited from building large databases about citizens directly, but faces no restriction on contracting with companies that do exactly that. AI makes existing surveillance easier to use but doesn’t create a fundamentally new category of threat.

A Few Observations

This episode carries unusual weight because Howard has genuine authority on both sides. He literally created the paradigm powering current AI and has spent decades building software by hand. That dual perspective gives his critique a credibility that pure talk from either the AI-optimist or AI-pessimist camps lacks.

• The slot machine metaphor is precise. It captures why AI coding feels productive without being productive: the illusion of control plus random reward schedules create addictive workflows that don’t compound into understanding. Rachel Thomas naming this pattern is itself valuable: once you see the mechanism, it’s harder to be manipulated by it.
• Desirable difficulty applies to coding, not just studying. The struggle of debugging, tracing logic, reading error messages isn’t waste to eliminate. It’s the mechanism by which engineers develop the judgment that separates shipping code from building systems.
• The coding vs. software engineering distinction may be the most consequential claim. If Howard is right that LLMs are “possibly always” going to be bad at software engineering, then the entire industry narrative about AI replacing developers is measuring the wrong thing. The benchmark scores everyone celebrates are testing interpolation, not engineering.
• The risk hierarchy is inverted from mainstream. Howard puts deskilling (happening now, accelerating) above power centralization (structural, medium-term) above autonomous AI (speculative, long-term). Most policy and media attention flows to the bottom of this ladder while the top plays out in every developer’s daily workflow.
• “Even if” is a devastating rhetorical move. By refusing to debate whether AI will become supremely powerful and arguing that even the strongest bull case doesn’t justify centralization, Howard sidesteps the capability debate entirely and lands on democratic principles.