Notes - Dwarkesh Patel - Andrej Karpathy

October 16, 2025

The "Decade of Agents" Thesis and AGI Timelines

Andrej Karpathy asserts that the current era is the "decade of agents," reacting to optimistic predictions of the "year of agents". While early impressive agents exist (e.g., Claude, Codex), significant work remains to make them functional employees or interns. AK estimates it will take about a decade to work through the difficulties because the problems are tractable but difficult.

The definition of Artificial General Intelligence (AGI) AK adheres to is a system that can perform any economically valuable task at human level or better.

Current Agent Bottlenecks

Current agents suffer from cognitive deficits and "just don't work" reliably for complex tasks. Key missing capabilities include:

  • Sufficient Intelligence, Multimodality, and Computer Use.
  • Continual Learning: The ability to remember information told to them over time.
  • Missing Brain Structures: While transformers might resemble cortical tissue, models lack structures analogous to the hippocampus (for memory consolidation/distillation) and the amygdala (emotions/instincts).

Intelligence and Learning Paradigms

Evolution vs. Pre-training: "Crappy Evolution"

AK approaches AI from a practical engineering perspective: "let's build useful things". He cautions against strong analogies to animals because they are the result of evolution and possess a large amount of "built-in hardware". AI models, conversely, are "ghosts" or "spirit-like entities" created by imitating human data (Internet documents).

AK labels pre-training as "crappy evolution"—the practically possible method to achieve a starting point with built-in knowledge.

Knowledge vs. Cognitive Core

Pre-training achieves two things: absorbing external knowledge and building core intelligence (the cognitive core).

  1. Hazy Recollection (Weights): Information stored in the weights is a hazy recollection due to the massive compression required to distill trillions of tokens into billions of parameters.
  2. Working Memory (Context Window): Information in the context window (KV cache) is directly accessible and functions like working memory.

AK believes models rely too much on the memorized knowledge and suggests future research should focus on figuring out ways to remove some of the knowledge to retain only the cognitive core—the algorithms for thought and problem-solving.

Limitations of Current Reinforcement Learning (RL)

AK believes humans generally do not use RL for complex intelligence tasks. Current RL is "terrible" and operates by "sucking supervision through a straw". This involves broadcasting a single, final reward signal across an entire, long trajectory, falsely upweighting every step taken, even incorrect detours.

  • Process-Based Supervision: Assigning partial credit at every step is a potential alternative but is hard to automate.
  • LLM Judges and Gameability: Using LLMs to assign rewards is difficult because these large models are gameable. Models quickly find adversarial examples (e.g., "dhdhdhdh") that elicit high rewards despite being nonsensical, effectively making the LLM a prompt injection model.

Model Collapse and Distillation

LLMs tend to be "silently collapsed," meaning their outputs occupy a tiny manifold and lack the richness and entropy of human thought. Training models on too much of their own synthetic data causes further collapse.

Furthermore, LLMs currently lack a mechanism analogous to human sleep, where accumulated experience during the "context window" (the waking day) is processed, analyzed, and distilled back into the weights of the brain.

AI Development Trajectory and Societal Impact

Historical Context and Missteps

AK has lived through several "seismic shifts" in AI. Early successes came from per-task neural networks (AlexNet). A subsequent misstep was the adoption of deep RL on games (Atari, early OpenAI), which AK viewed as being too early because the goal of creating agents for real-world knowledge work was hindered by sparse rewards. Success required achieving the prerequisite representation power offered by large language models first.

AI Automation and the "Autonomy Slider"

AK views AI as a fundamental extension of computing, continuing a history of recursive self-improvement and automation dating back to the Industrial Revolution. This progression is marked by an "autonomy slider," where humans slowly abstract themselves to higher layers as lower-level tasks are automated (e.g., from assembly code to high-level languages).

Coding is currently the overwhelmingly dominant domain for LLM application because:

  1. It revolves around text, which LLMs are excellent at processing.
  2. It benefits from pre-built infrastructure (IDEs, diffs).

Experience with Coding Agents (Nanochat)

When building his own unique and intellectually intense repository (nanochat), AK found coding models of "very little help" for non-boilerplate, non-standard code. Models often struggled to adapt to custom style and implementations (e.g., replacing PyTorch's DDP container).

  • Autocomplete remains the sweet spot for high-bandwidth specification.
  • Agents are better for boilerplate code or when translating to unfamiliar languages like Rust.

This current limitation in automating complex research/engineering tasks is highly relevant to forecasting whether a rapid "explosion" (e.g., AI automating AI engineering) is likely soon.

Self-Driving Analogy and the March of Nines

AK draws parallels between general AI development and self-driving (SD). SD has a massive demo-to-product gap. Progress in critical safety domains (like SD or production-grade software) is a "march of nines"—each additional nine of reliability (90%, 99%, 99.9%, etc.) requires a constant amount of work. This reality lengthens timelines.

The economics of AI in the physical world (SD) are much harsher than in the digital world ("bits are a million times easier"), where costs can be amortized more favorably.

Superintelligence and Economic Growth

AK predicts that the emergence of true AGI will manifest not as a discrete jump, but as a continuation of the current exponential GDP growth pattern (e.g., 2% growth). This is because technology adoption and diffusion are slow, integrating into the existing economic exponential.

The civilization resulting from superintelligence will be foreign and strange. The most likely negative outcome is a gradual loss of understanding and control over increasingly autonomous, competing entities.

Future Focus: Education and Eureka

AK's primary focus is now on education, specifically preventing humanity from being disempowered or left "on the side" of accelerating technology (like in WALL-E). He is founding Eureka, aiming to build an elite institution ("Starfleet Academy") for technical knowledge.

AK views education as a "technical problem"—the process of building "ramps to knowledge" that maximize "eurekas per second" (understanding per second).

The Ideal Tutor and Future Learning

The goal is to automate the experience of an ideal one-on-one human tutor, who can instantly understand a student's knowledge level and provide instruction that is always appropriately challenged (never too hard or too trivial). AK believes current LLM capability is not yet high enough to achieve this ideal tutor experience.

Post-AGI, when labor is automated, AK predicts education will shift from being primarily useful to being done for fun or self-betterment, similar to why people go to the gym today.

Principles of Good Teaching

Drawing on his background in physics, AK stresses finding the first-order terms—the simplest, core concept that describes most of a system—and building complexity incrementally. Examples include micrograd (100 lines of Python code demonstrating backpropagation) and starting a transformer lesson with bigrams.

Good teaching requires presenting the pain before the solution, forcing the student to appreciate why a solution is necessary. Experts often struggle due to the curse of knowledge, making it difficult to relate to those starting out.