팟캐스트Hear the voice. See the shape of the thought.

Six months after their December roundtable, Jacob Effron reconvenes with Ari Morcos (Datology AI CEO) and Rob Toews (Radical Ventures) for a full-spectrum AI vibe check. Coding agents have crossed a long-horizon threshold that is reshuffling the engineer's job description; near-frontier open-weight models look increasingly like a retreating tide as both Meta and the Chinese labs pull back for economic reasons; and Anthropic's silent capability restrictions on Fable have rattled its most loyal supporters. The trio works through Google's structural durability despite coding lag, Ari's prediction that compute pressure could force labs to suspend their public APIs entirely, the emerging atom and X-ray lithography challengers to ASML, and how close — but how bottlenecked — recursive self-improvement actually is. ## [00:00] Intro Jacob welcomes returning guests Ari Morcos and Rob Toews, noting that this is a "vibe check" format covering everything from IPO filings and SpaceX's pivot to compute to Fable's release the prior day. He frames the conversation around a single question: what is the single biggest thing that changed in the six months since they last sat down after NeurIPS? > *"Things have changed. We've had IPO filings. We've had models not launched and then launched. We've had SpaceX becoming an AI info company."* — Jacob Effron ## [01:40] Coding Agents Cross a Threshold Ari identifies the clearest shift: coding agents now reliably execute at longer time horizons, which crossed a threshold over Christmas break that made them genuinely useful rather than merely promising. At Datology, engineers have almost universally transitioned from individual-contributor work to managing fleets of agents concurrently — but the gains come with a new bottleneck. Code review queues are backing up, and the "slop" entering codebases is harder to catch when no one fully understands what the agent wrote. > *"We're really starting to now see the shift of engineers at least kind of almost all moving from ICs to managers of agents."* — Ari Morcos ## [03:29] Is Open-Weight AI in Retreat? Rob opens with what he calls a structural inflection: near-frontier open-weight AI risks falling off entirely. His prior assumption — that open models would trail closed ones by only a few months — may no longer hold. Meta appears to be pulling back from its open-source strategy, and Chinese labs including Qwen and DeepSeek are now keeping their highest-performing weights proprietary while open-sourcing only smaller, less capable versions. Ari agrees the economics no longer support openness once a lab has gained credibility: hosting inference is far more lucrative than giving away weights. Rob is blunt that no viable long-term business model exists for purely open-weight AI at the frontier. > *"There are early signs that seem to suggest over the past six months that make me question whether open-weight AI is going to continue to be a really meaningful force in the ecosystem."* — Rob Toews ## [07:37] Cost Crunch & Scaffolding Jacob notes a counter-pressure arriving simultaneously: enterprises are finally getting serious about reducing model spend. Going from Claude Opus 4.6 to 4.7 doubled token output for some users, and bills that were once negligible are now budget line items. Ari argues the real innovation is increasingly happening not in the model weights but in the harness and scaffolding layer — open-source models combined with proprietary scaffolding (Kimi/Moonshot being the clearest example) may be the actual business model that survives. He also warns enterprises that the only two real options are partnering with a frontier lab (and eventually being out-competed because you've handed over your proprietary data) or building enough in-house capability to maintain independence in a world where reliable open models are no longer guaranteed. > *"A model is not just a model anymore — it's the model combined with the harness and the scaffolding, and a lot of innovation is happening on the harness and scaffolding layer."* — Ari Morcos ## [12:13] The "Apps Are Cooked" Debate Rob thinks the "apps are cooked" narrative was simultaneously partially right and wildly over-broad. Traditional software categories genuinely face existential pressure from lab roadmaps, but no two or three companies can execute excellently across every vertical on earth. OpenAI shutting down its video effort — despite having effectively infinite capital and a strong team — is proof that even the richest lab has to make hard prioritization calls, and much of that is driven by compute constraints. Deep tech and hardware have become the consensus VC bet as a result, but Rob flags that hard tech is also hard: failure rates are high and unsolved problems abound. > *"There's no way that one or two or three companies will win every single important market and category in the world."* — Rob Toews ## [16:37] Sam Altman Under Scrutiny Rob revisits his December prediction that Sam Altman would be replaced by year end. At the time everyone pushed back; mid-June the odds look higher. His original succession candidate — Fiji — has had to step back for health reasons, and his updated theory centers on Bret Taylor: chairman of OpenAI's board, CEO of Sierra, and one of Silicon Valley's most trusted operators. Rob thinks an OpenAI acquisition of Sierra combined with installing Taylor as CEO would be a decisive narrative reversal ahead of the IPO — the trust gap between OpenAI and Anthropic is large and widening, and Taylor's reputation could close it. Ari floats an alternative: OpenAI restructures into an Alphabet-like holding company where Sam stays atop the parent while a separate CEO runs the core product. > *"I think it would be in the best interest of OpenAI's shareholders — if someone like Bret Taylor was at the helm of OpenAI, I think it would do a lot to change their fortune."* — Rob Toews ## [19:44] Anthropic's Fable Backlash The group digs into the blowback from Anthropic's decision to silently restrict Fable for any work touching AI development. Ari says the restriction itself is tolerable; the silent degradation — the model simply performs worse without telling you — is what has genuinely angered Anthropic's most loyal supporters. He reads the move as competitive positioning dressed up as safety, noting that open-model teams with good scaffolding have independently reproduced most of the vulnerability-finding capabilities that the restriction is supposedly protecting. Ari predicts a meaningful share of Claude Code's loudest Twitter evangelists will migrate to Codex in the short term, handing OpenAI an unexpected PR gift. > *"It doesn't give you a refusal. It doesn't say, 'I'm not going to help you with this.' It just does a poor job on that without you knowing."* — Ari Morcos ## [23:24] How Big a Step Change Is Fable? Ari, who had only started using Fable the night before recording, says he personally didn't see massive differences from Claude 4.8. Rob frames Fable less as a discontinuity and more as evidence that the "pre-training is hitting a wall" narrative was plainly wrong — gains keep coming richly from pre-training, and test-time compute has added another lever on top. Ari reinforces this from a practitioner's standpoint: in deep learning, having 95% of the details right often produces no improvement, and then one last adjustment triggers a step change. Negative results about scaling are therefore genuinely hard to interpret. > *"If you have kind of 95% of it right, it kind of rectifies to just not working. And then you turn the last knob and all of a sudden you get a step change."* — Ari Morcos ## [26:50] What's Going On at Google? Rob pushes back on the idea that Google is underperforming: the three frontier labs leapfrog each other continuously, and Google's lag on coding specifically is a prioritization choice — Anthropic built its entire identity around coding, OpenAI recently poured resources into it, and Google simply hasn't made it the north star yet. What Google does have is a full-stack structural advantage: its own chip design (TPUs), its own cloud, an enormous talent bench, and the Android/iOS distribution deal that makes its models the default on the world's phones. Ari adds that consumer AI will commoditize quickly, and Google is already optimized for the default-provider role on mobile even if it doesn't hold the best model. Jacob observes that Codex is clearly a strong product yet Claude Code remains dominant — first-mover advantage in developer tooling is stickier than expected, though Fable's restrictions may catalyze a wave of switches. > *"I think [Google's] behind on coding and I think that's just it reflects prioritization. It's clear that Anthropic leaned in on that as their northstar for years."* — Rob Toews ## [33:20] Could the APIs Go Away? Ari surfaces the most provocative claim of the episode: compute constraints could push Anthropic — or OpenAI — to suspend public API access entirely, not as a business decision but simply because first-party products like Claude Code generate better margins and chips aren't infinite. OpenAI has already started selling futures on guaranteed inference tokens, which Ari reads as a sign the lab itself sees API access as rationed. Rob confirms this is technically feasible, though extreme; a more likely near-term version is labs reserving their most powerful models for internal use rather than offering them publicly. > *"It is not hard to imagine a world in which Anthropic is so compute constrained that they actually cut off the API."* — Ari Morcos ## [34:11] Breaking the Semiconductor Bottleneck Rob shifts the conversation to the physical underpinning of the compute shortage: the extraordinary concentration of chip manufacturing in a single company (TSMC) whose most critical machine is made by a single other company (ASML). He flags Elon Musk's "terafab" concept as underreported given its transformative potential if executed. Ari pushes back on the timeline — relieving the compute constraint within the next handful of years is hard to imagine. Rob concedes that TSMC displacement in two to three years is implausible, but a five-year horizon with multiple augmenting players is imaginable — the single-point-of-failure structure of the global semiconductor supply chain doesn't have to persist. > *"It's actually kind of crazy that there's like one company that knows how to do this and no one else can do it, and the most important machine that goes into the process is made by one other."* — Rob Toews ## [35:42] Beyond EUV: Atom & X-Ray Lithography Rob describes two emerging research directions that could eventually challenge ASML's EUV dominance. The first is atom lithography: rather than using light, you use a beam of atoms to print transistor features, allowing far finer resolution with machines that are simpler, cheaper, and smaller than EUV tools. The second is X-ray lithography, which uses shorter-wavelength electromagnetic radiation to push beyond the physical limits EUV is beginning to hit. Startups in both categories have raised significant funding and remain in development mode. Ari estimates commercialization is at least five years away, but Rob thinks genuine technology disruption is coming. > *"There are a couple startups doing really interesting work in atom lithography... the machine can be way simpler, way fewer parts, way cheaper, way smaller, obviously much better resolution."* — Rob Toews ## [37:23] Implications of a Compute Shortage Jacob asks what a world of deepening compute scarcity actually means for businesses. Ari argues it will force the efficiency innovation that frontier labs have had little incentive to pursue: smaller and smaller models will match the largest models of one to two years prior, distillation investment will accelerate, and inference optimization will become a genuine competitive differentiator. Rob adds that the supply constraint is structurally good for every chip vendor other than Nvidia — AMD, Trainium, Cerebras — not because they increase total supply (TSMC remains the upstream bottleneck) but because enterprises will use whatever silicon they can get. H100 spot prices reversing their December decline is the clearest market signal that the shortage is intensifying rather than easing. > *"I would still expect that the usage is going to grow faster than what you can do to alleviate this."* — Ari Morcos ## [40:20] Do Alt Chips Actually Help? The group stress-tests whether alternative chip providers actually expand total compute or just redistribute it. The consensus: they are a beneficiary of the constraint, not a solution to it. In a world without Cerebras or dMatrix, Nvidia would simply absorb all of TSMC's capacity — total chip count stays constant. What alternative vendors do is prevent Nvidia from achieving a full monopsony on TSMC production and give compute-hungry buyers a fallback. The compute constraint is unlikely to ease before 2030; Ari estimates the early 2030s are when multiple unblocks — new fabs, new lithography, algorithmic efficiency — may hit simultaneously. > *"The alternative chip providers aren't a solution to the compute constraints, but will be a beneficiary of the compute constraint."* — Rob Toews ## [43:43] SpaceX, xAI & the Cursor Acquisition Jacob turns to xAI and the reported $60 billion Cursor acquisition. Rob is skeptical that xAI will re-enter the top tier of frontier AI research: the decision to sell compute capacity to Anthropic and Google is a clear signal that data center buildout — not model research — is the company's real priority. He thinks xAI's durable advantage matches Elon's operational DNA: standing up massive clusters extremely fast. Ari argues the Cursor acquisition is primarily about obtaining coding traces to bootstrap a competitive coding model that xAI has so far failed to build on its own — and that $60 billion is probably quite high relative to that goal, but keeps optionality alive. Rob notes the SpaceX S-1 TAM chart, which estimates enterprise AI at roughly twenty trillion dollars while all of space comes in at a few hundred billion, and concludes that narrative positioning ahead of the IPO is a big part of the deal's logic. > *"I think why Cursor is to get all the traces... and to have a hedge against the fact that they have struggled to produce a very competitive coding model."* — Ari Morcos ## [48:50] How Close Are We to RSI? Andrej Karpathy's decision to join a recursive self-improvement team prompts a direct question about timelines. Ari has moved meaningfully more bullish in six months: at Datology, agent-driven data curation experiments have produced results "far more promising than I would have expected," and he now sees RSI as clearly approaching feasibility. The bottleneck is compute, not ideas or execution. He is, however, deeply skeptical of the "one lab runs away" takeoff narrative: compute constraints cap the speed of self-improvement, and at least ten well-funded organizations have the talent and knowhow to pursue it simultaneously. Rob was expecting Ari to be more skeptical — pushed to explain how RSI could arrive without an exponential takeoff, Ari points back to compute as the fundamental limiter on iteration speed. > *"We are clearly getting to the point where models can improve themselves... but I think there are just fundamental compute bottlenecks that can prevent the speed."* — Ari Morcos ## [52:21] Quickfire The closing round surfaces several sharp takes. Rob's biggest disagreement with current discourse: today's AI systems are laughably energy-inefficient compared to what is coming — a 2-gigawatt data center versus the human brain's 20 watts — and breakthroughs in analog computing and hardware architecture will make the current capex buildout look like a historical anomaly. Ari's sharpest contrarian position: the "permanent underclass" narrative — AI takes all human jobs within a decade — is overblown because humans are slow at dissipating technology through the economy and business relationships carry a human-trust dimension that technocrats systematically underestimate. On mind-changes: Ari is more bullish on RSI than six months ago and now strongly believes near-frontier open-weight models will consolidate and shrink. Rob has pulled in his robotics timeline — foundation models for robotics have crossed a commercial viability threshold in recent months and the GPT-3 moment for general-purpose robotics may now be near. On spicy predictions for the back half of 2026: Ari bets that Anthropic — or possibly OpenAI — will suspend or heavily restrict API access at some point, with end-of-2027 as his higher-confidence window. Rob's prediction: Anthropic's next chapter is life sciences, and by year end it will be obvious they are building toward being one of the most important life sciences companies in the world — potentially including wet lab facilities of their own. > *"I think by the end of the year it will be very obvious that Anthropic is a fledgling juggernaut in the making in life sciences and biology."* — Rob Toews ## Entities - **Jacob Effron** (Person): Host of Unsupervised Learning, Managing Director at Redpoint Ventures - **Ari Morcos** (Person): CEO of Datology AI; former Meta AI and DeepMind researcher; guest - **Rob Toews** (Person): Partner at Radical Ventures; Forbes AI columnist; guest - **Anthropic** (Organization): AI safety lab behind Claude and Fable; subject of both admiration and growing criticism for silent capability restrictions - **OpenAI** (Organization): Lab behind ChatGPT and Codex; undergoing internal scrutiny around Sam Altman's leadership - **ASML** (Organization): Dutch company with near-monopoly on EUV lithography machines, the critical bottleneck for cutting-edge chip manufacturing - **TSMC** (Organization): Taiwan Semiconductor Manufacturing Company; the world's sole producer of the most advanced chips - **Datology AI** (Organization): Ari Morcos's startup focused on data curation and training infrastructure for AI models - **Cursor / Anysphere** (Software / Organization): AI coding tool reportedly being acquired by xAI for approximately $60 billion; valued primarily for its coding trace dataset - **Recursive Self-Improvement (RSI)** (Concept): The ability of AI systems to autonomously improve their own training and capabilities; increasingly treated as near-term rather than speculative - **Atom lithography** (Concept): Emerging chip manufacturing technique using beams of atoms rather than light to print transistor features, offering superior resolution and simpler machinery than EUV - **EUV (Extreme Ultraviolet Lithography)** (Concept): Current state-of-the-art chip printing technology, approaching physical resolution limits; ASML's core product

Lukasz Kaiser — co-author of "Attention Is All You Need" and researcher at both Google Brain and OpenAI — gives Jacob Effron a candid tour of where the current AI paradigm stands and where it strains. He holds two positions in tension: transformers with RL and agents have already delivered stunning productivity gains (he clocks a 10x speedup in his own research), yet something about how humans generalize from sparse data still eludes today's architectures. The conversation moves from that philosophical tension into concrete territory — the Christmas 2025 coding agent inflection, the frontier of RL on non-verifiable tasks, Anthropic's bet on coding, and how the open-source/closed-source gap will likely evolve. ## [00:00] Intro Jacob Effron previews the core questions driving the episode: whether reasoning is sufficient for true generalization, what changed around Christmas 2025 to make coding agents suddenly click, why Anthropic got there first, and where the closed/open-source divide is heading. ## [01:12] Transformers vs. Human Learning Kaiser opens with genuine ambivalence. Transformers with chain-of-thought and RL already perform feats he would have called impossible two years ago — daily Codex sessions that tackle hard research problems and actually deliver. But the data efficiency gap with human learners nags at him. > *"LLMs will learn a concept — but after exhausting all other options. You need a trillion tokens to like learn all the surface level things and only when that doesn't explain something they will finally learn the concept. That's not how we learn."* He traces the intuition not just to vibes but to a structural point: models called "neural networks" were always meant to mimic the brain, yet they differ from it fundamentally. Post-transformer labs are gaining steam, but Kaiser remains genuinely uncertain which side wins — transformers keep catching up every time researchers think they have found a smoking gun for something better. ## [08:37] How Do We Get Physical World Generalization? Jacob presses on the practical stakes: plenty of problems are *not* data-constrained, so why does physical-world generalization matter so much? Kaiser's answer is that the un-data-constrained problems get solved first and fastest; the bottlenecks that remain will almost all be data-limited, and the physical world is the canonical hard case. His go-to example is Waymo cancelling highway driving because the model could not handle construction zones it had already seen in cities. > *"No teenager has this problem. Not that we can drive in a construction zone in the city but not on the highway — that just construction zone is a construction zone."* That failure mode — millions of miles of simulation, still can't generalize across one context shift — is exactly the kind of brittleness that motivates him to watch post-transformer research closely. ## [10:52] What Comes After Transformers Kaiser's view is that any genuine architectural successor will probably require simultaneous changes to architecture, data, loss, and optimization — not just one knob. Attention will likely survive in some form; recurrence, which he has loved since his RNN days, has come back implicitly through reasoning's token-by-token weight sharing, but explicit recurrent architectures still haven't clicked at scale. > *"The pure transformer can't do so well on it, but you add some recurrence, you add some bit of architectural tweaks, maybe a little different loss, and it does really well — so even on the small scale you can do a lot."* He points to models like TRNM and HRM doing well on Sudoku-style benchmarks as early but real signals. Still, the agents story dominates his practical working life: the transition to coding agents is, he says, "the biggest change in the way I work as an ML researcher in the last 20 years." ## [13:59] How Much Have Agents Improved Lukasz's AI Research Productivity? Kaiser puts a number on it: a paper reproduction that previously took three weeks now takes two days — roughly a 10x speedup. But speed isn't the only gain; he now runs three workstreams in parallel, something he never attempted before. > *"Now it's like this beautiful thing where you can just be in this flow — you just think machine learning wise what's supposed to happen, you tell it, verify it, and it's happening."* He also addresses the concern that heavy agent use makes researchers less sharp. His experience is the opposite: because agents can silently add auxiliary losses or make plausible-but-wrong changes, you need a tighter conceptual grip on what the model is supposed to be doing. The high-level architecture lives in your head more clearly than before, even as you stop tracking class names and function signatures. ## [17:21] How Close Is an AI Research Intern? OpenAI's stated goal of "research-level intern by November" lands as roughly accurate to Kaiser — with a crucial caveat. The agent will not autonomously improve a model on an open-ended goal like "lower perplexity." Given that instruction, it defaults to trivial tweaks. It cannot yet set a research direction and execute it over weeks unattended. Two structural blockers: current RL methods need rollouts that are as long as the task, and research tasks run for weeks, making training timelines impractical. Humans somehow learn to do multi-year research problems without doing hundreds of them first — that generalisation of process remains unsolved. > *"Some mathematicians spend 20 years on one problem — that's their magnum opus and that's it. They did not have 200 problems 20 years long before to learn from, and somehow they manage."* On the Christmas 2025 leap, Kaiser notes that the improvement is hard to fully attribute — harness changes, post-training changes, and new pre-trained models all arrived together. Something genuinely crossed a threshold, but the exact cause is unclear even to insiders. ## [26:06] RL Beyond Verifiable Tasks The "RL only works on verifiable domains" framing is too narrow, Kaiser argues. Harvey in law is not strictly verifiable, but has seen strong progress because many sub-tasks are verifiable enough. Even poetry translation, his personal test case, can be partially verified: rhyme, cultural references, and structural properties all have checkable proxies. > *"Every hole you have you can kind of plug by hammering on it, but it would be so nice if you didn't have to — because every hole you plug stops being a bottleneck and then the bottleneck that emerges is the holes you have not plugged."* On generalization from RL: it does happen, but it's jagged. A model that masters nearly all IMO problem types might still collapse on geometry until it sees more geometry problems specifically — not because it lacks spatial reasoning in the abstract, but because its chain-of-thought representation places geometry far from the domains it trained on. The brittleness is real; you have to stay on the lookout. Kaiser finds that honest engagement with these sharp edges keeps him sharper as a researcher. ## [35:38] App Companies: Build Models or Lean on Labs? A bigger pre-trained model flatly makes everything easier — fine-tuning, RL, robustness — and that pattern has persisted longer than anyone expected. The "SLMs are the future" narrative from 2024 was wrong in the sense that frontier capability still compounds with size. Kaiser's more interesting riff is on hardware democratisation. A single RTX 5090 under his desk delivers roughly 200 teraflops in BF16 — comparable to five of the eight-GPU machines that ran the original transformer research. You could, today, reproduce all of transformer research on a few-thousand-dollar desktop tower. > *"Potentially you can run like a year of human processing in a day — at a cost of hundreds to thousands of dollars, not millions."* He's particularly excited that coding agents now write CUDA kernels on demand, removing one of the biggest practical barriers to exploring non-standard architectures. The bottleneck used to be: your idea doesn't map cleanly to standard ops, CUDA is painful, you give up. That bottleneck is shrinking fast. ## [46:21] Multimodal Is Still Missing Something Current multimodal models process images as sequences of small patches, autoregressing over pixels — a design that feels fundamentally mismatched with how biological sensory processing works. Humans receive a continuous, massively parallel stream from all senses simultaneously, at speeds far beyond what sequential token processing can mimic. > *"Everything happens everywhere all at once for us — we see, hear, talk all at the same time. That should be how our models behave."* He cites Thinking Machines' multi-stream transformer work as a promising direction. His practical frustration: coding agents that have to wait for a bash command to finish before receiving new instructions, when the natural interaction would be fully parallel. The architectural fix seems conceptually straightforward; whether it meaningfully improves capabilities at scale is still open. ## [49:46] OpenAI's Bet on Reasoning The defining decision in Kaiser's OpenAI tenure was the pivot to reasoning models. At the time, maintaining two separate model families — chat and reasoning — was awkward, personality felt harder to preserve in reasoning models, and latency was a real concern. The company committed anyway. > *"OpenAI was very good at taking this hard bet and saying yes, we're going to launch it. We're going to go this way."* Kaiser credits that conviction as a meaningful competitive advantage: even large labs are still catching up to OpenAI's RL quality. His concern now is whether OpenAI at its current scale — having grown roughly 20x — can still make wild bets, and whether any of the labs could pivot fast enough if post-transformer architectures start to look genuinely compelling. He sees the neo-lab ecosystem (small, focused, GPU-constrained but intellectually unconstrained) as a useful counterweight. ## [55:26] The AI Coding Wars Kaiser's view on the Codex-vs-Claude Code competition is that the coding market is large enough to sustain two serious players. The more important question is how either product expands beyond software engineers — Codex still opens with "what's your GitHub repo," which cuts off most potential users. On why Anthropic got to coding first: they simply couldn't compete on chat, so they made a focused bet. OpenAI was doing ChatGPT at GPT scale with a billion users; Anthropic picked a different hill. The lesson Kaiser draws is general: in fast-moving AI, committing to a non-consensus direction while it's still unpopular is often how you win the next cycle. > *"Anthropic made this very good decision to focus on coding. OpenAI was like, we're doing ChatGPT. ChatGPT is great, but clearly not the most amazing AI of 2026."* ## [59:26] Focus vs. Keeping Embers Burning Google's "keep all embers burning" culture is often criticised for letting others commercialise Google's own research breakthroughs. Kaiser's take is more balanced: staying broad means that when a field catches fire, you already have a strong team and can catch up quickly. He sees evidence that Google has largely caught up on chat-class models, though the coding-agent inflection moment has not been fully replicated yet. The counterpoint: Anthropic's tight focus on coding let them be *first*, which matters for adoption and feedback loops. OpenAI is now in a similar focusing moment, which produces visible results in Codex quality — but comes with risk when you have a billion users and any degradation in a core product causes real harm. Kaiser's conclusion: the labs shouldn't break things on the way, but pace still matters. ## [62:09] Open Source vs. Closed Source Gap Kaiser expects the gap to persist but not become absolute. Distillation makes open-source models good, but not quite as good as the frontier — he notices the difference between Gemini Flash and Gemini Pro in his own research workflow. Sovereign AI demand (governments and large institutions that don't want single-vendor dependency) creates durable incentives for open models to stay relevant, and the big labs have limited appetite for fighting open-source adoption to the death. > *"There will be enough incentives to have open models that they will exist, and there will be very good incentives for the labs to still keep ahead. People keep paying for this — so it feels like a state that should persist for a while."* ## [65:15] Quickfire Kaiser's most significant personal update: he went from barely using AI daily to spending hours every day inside Codex. The practice of not looking at code at all — just directing the agent conceptually — was something he actively resisted and then adopted fully. On existential AI risk: his concern level is roughly unchanged, staying focused on near-term misuse scenarios (infrastructure hacking, grid disruption) rather than AGI takeover. On Andrej Karpathy joining Anthropic to work on RSI: Kaiser is enthusiastic about the direction but notes that post-transformer breakthroughs require vast, mostly-wrong exploration — even the most capable research agents today are still bad at learning from a completely wrong direction and twisting it into the right one, which is exactly what humans do well. His closing note is an encouragement to researchers: the current moment — desktop GPUs that rival five 2017 research clusters, coding agents that write custom kernels, and a field where the dominant paradigm is genuinely contestable — is the most exciting time to be in ML. He points to his own pre-transformer paper ("You Don't Need Attention") as a reminder that wrong explorations often lead to the right ones. ## Entities - **Lukasz Kaiser** (Person): co-author of "Attention Is All You Need"; researcher at Google Brain and OpenAI; episode guest - **Jacob Effron** (Person): Managing Director at Redpoint Ventures; host of Unsupervised Learning podcast - **"Attention Is All You Need"** (Concept): 2017 paper introducing the transformer architecture, co-authored by Kaiser; foundational to modern LLMs - **Transformer** (Concept): dominant neural network architecture since 2017; central subject of debate on its generalization limits and potential successors - **Reinforcement Learning (RL)** (Concept): training paradigm using reward signals; key to coding agent improvement and the subject of the "beyond verifiable tasks" discussion - **Codex** (Software): OpenAI's coding agent; Kaiser's primary research productivity tool, giving him an estimated 10x speedup - **Claude Code** (Software): Anthropic's coding agent; discussed as a direct competitor to Codex - **Waymo** (Organization): autonomous vehicle company; used as a case study for physical-world generalization failure in construction zones - **Anthropic** (Organization): AI lab credited with the strategic decision to focus on coding, enabling early dominance in coding agents - **OpenAI** (Organization): AI lab where Kaiser worked; credited with the pivotal decision to commit to reasoning models - **Google Brain** (Organization): research division where Kaiser worked before OpenAI; discussed in context of Google's broad-embers vs focused-bet strategy - **Harvey** (Organization): AI-for-legal-work company; cited as evidence of RL progress on non-verifiable domains - **Generalization** (Concept): the ability to apply learned concepts to genuinely new situations from limited data; core tension of the episode - **Recurrence / RNNs** (Concept): pre-transformer sequence modeling paradigm; Kaiser argues it may return as a component of post-transformer architectures - **Andrej Karpathy** (Person): AI researcher; his move to Anthropic to work on RSI is discussed in the Quickfire section

Sebastian Mallaby spent three years and over 30 hours with Demis Hassabis in a British pub to write *The Infinity Machine*, and this conversation pulls the most underreported threads from that access: the 2015 safety summit that accidentally spawned OpenAI, the secret billion-dollar spinout plan Demis never used as real leverage, and the quasi-spiritual conviction about God and science that Mallaby never expected to find. The throughline is a paradox — Demis understood the race was dangerous from day one, but as leader of one lab, even a Nobel Prize-winning one, he could not stop it. ## [00:00] Intro Jacob Effron sets up Sebastian Mallaby as someone who has spent more time with Demis Hassabis than almost any journalist alive — 30-plus hours across three years of pub sessions in London. Mallaby's book, *The Infinity Machine*, covers the full arc of DeepMind from its 2010 founding through the Nobel Prize. The clips previewed here — Demis banging the table about God and science, Reid Hoffman's billion-dollar pledge, and the Elon feud — all come from later in the conversation. > *"Demis has a Nobel Prize. Sam didn't finish his first degree. Therefore, Demis doesn't take Sam very seriously."* ## [02:04] Was the AI Race Inevitable? Mallaby's verdict: yes, inevitable. Any technology this powerful would attract multiple labs across multiple countries, and China's stack was already competitive despite semiconductor shortfalls. What makes the story poignant is that Demis didn't believe this in 2010. He genuinely hoped one lab could carry the AGI project safely to the finish line — a singleton scenario where DeepMind was the anointed team. By the mid-2020s he had swung to the opposite pole: safety is a collective action problem that only governments can solve, because no single lab's restraint can bind the others. > *"I think it was inevitable. When you have this sort of supremely strong technology, there's going to be multiple labs in multiple countries that are just desperate to try and build it."* ## [04:03] The 2015 Safety Summit Backfire Summer 2015, SpaceX headquarters: Demis convenes a small summit to bring Elon Musk inside the tent — the plan was for Elon to chair a safety oversight board and, critically, not launch a competitor. By end of year, OpenAI existed. Mallaby frames this as the moment Demis internalized that voluntary collaboration between lab leaders is structurally impossible. The only mechanism he now believes can work is a government enforcer setting uniform rules — mandatory pre-release testing, safety slow-downs — with US-China cooperation as the endpoint, however remote that prospect appears. Jacob pushes on whether lab leaders actually believe government intervention is achievable; Mallaby draws a parallel to the FDA: slow, imperfect, but it does adjudicate whether drugs are safe enough to ship. > *"You can't trust the other guys. The only way you get trust is if you have a government enforcer that comes along and says, 'Here's the rules for everybody. There's going to be a level playing field. You're all going to have to abide by some sort of safety slow-down.'"* ## [11:27] Why Google Doesn't Make As Concentrated Bets Jacob points to the two defining consumer-AI moments of the era — ChatGPT and Claude Code — and neither came from Google DeepMind despite its leaderboard dominance. Mallaby traces this directly to Demis' intellectual formation: a PhD in neuroscience, a broad theory of intelligence, a lab culture that says "whenever there are two paths, do both, find a third." The result is a heavily hedged research portfolio that is excellent at producing Nobel Prizes and state-of-the-art models but structurally slow to make the kind of one-directional product bet Anthropic made on coding. Gemini is bundled into Google Search, so usage is higher than it appears — but Mallaby concedes the product-zeitgeist gap is real. > *"Anthropic got to coding because it was willing to take a more concentrated bet. It never went into the whole field of, you know, everything at once."* ## [15:51] Project Mario: The Secret Spinout Plan The book's most explosive scoop: DeepMind had a secret plan — code-named Project Mario — to spin out of Google, backed by a $1 billion pledge from Reid Hoffman. Mallaby had to fight Google's general counsel to publish it. The motive was not entrepreneurial independence but safety leverage: Demis wanted formal safety oversight over DeepMind's models, Mountain View wasn't providing it, and a credible spinout threat was his negotiating chip. He never explicitly told Google about the Hoffman pledge, but pushed hard knowing the option existed. In the end he chose to stay — legal risk of the spinout fight, desire for compute access, and a preference for doing science over litigating corporate structure. A year later he shipped AlphaFold and won the Nobel Prize. > *"Demis really really wanted to get safety oversight over the Google DeepMind models. Google corporate in Mountain View wasn't doing that. So he had to have a credible threat of spinning out. He went to Reid Hoffman. Reid Hoffman pledged a billion dollars to finance a spinout — and Demis used that to kind of pressure Google."* ## [19:43] What Demis Actually Regrets On AlphaFold and AI-for-science: no regrets at all — Mallaby argues it was not only scientifically correct but politically necessary, because AI needs visible social benefits to survive the coming backlash against job disruption. The genuine regret is speed. Demis missed the transformer moment the way Ilya Sutskever did not: when the paper dropped, Ilya ran down the corridor to find Alec Radford to build a language model. Demis' broad-portfolio instinct meant DeepMind studied the transformer but didn't bet the lab on it. Missing that window — and the ChatGPT moment that followed — is a real failure, not just a stylistic difference. > *"Ilya is like jumping out of his chair, running down the corridor going to find Alec Radford saying, 'Hey, we're going to build a language model based on this transformer architecture.' On the day they won AlphaGo, Demis was already on to bio — and someone picked it up on a mic."* ## [23:46] Venture Startups vs. Tech Behemoths The broadest structural argument in the episode: does venture-backed concentration beat hyperscaler breadth in AI? Mallaby has written about both (his previous book covered venture capital) and calls it genuinely balanced. Hyperscalers have unlimited capital and can sustain a multi-year arms race; the problem is that unlimited resources breed portfolio thinking, which bleeds attention. Startups with one concentrated bet can move faster on that specific bet. Mallaby's live position: OpenAI has roughly 50/50 odds of being absorbed or failing before next summer — not because the tech is weak, but because the business model can't sustain indefinite losses against Google's balance sheet. He also floats that Anthropic should IPO right now while its brand is strongest. Jacob notes the robotics parallel: fifteen different approaches being funded simultaneously, and whoever picks the one that works the way transformers did will dominate. > *"I wrote in the New York Times in January that I thought OpenAI had a 50% chance of going bust by next summer. Is it still 50? Yeah. The tech is great. It's just the business model — and you're up against Google, which just has unlimited amounts of cash to spend you into the ground."* ## [34:08] David Silver and the RL True Believers David Silver — AlphaGo's lead researcher and co-author of the "reward is enough" paper with Rich Sutton — left DeepMind after the book came out to start a new company. Mallaby reads the departure as structurally inevitable: Silver is a pure reinforcement learning absolutist who believes learning from human data is fundamentally inferior because it encodes human errors. His thesis is that self-play and environment-generated experience is the only path to genuine superhuman performance. Demis told Mallaby this view may ultimately be correct *after* AGI is achieved — but the entire language model revolution showed that bootstrapping with human data is what gets you to AGI in the first place. Silver's RL purism was too far ahead of the current paradigm for his colleagues to follow. > *"David is just very very hard over on that vision — learning from data is inferior because the data includes mistakes. The machine needs to learn from its own experience, not rely on the crystallized knowledge of humans passed on through text."* ## [38:21] Demis, Elon, and the Evil Genius Feud The origin story: at a Founders Fund LP offsite in 2012, Elon argues that SpaceX matters most because even if AI wrecks Earth, humanity can move to Mars. Demis replies that his AI will eventually conquer space flight and follow them there. Elon goes quiet, then writes a $5 million check into DeepMind's Series B. Two years later, hearing Google was acquiring DeepMind, Elon and Luke Nosek Skyped Demis from a party closet in LA in the middle of the night, begging him not to sell to Larry Page. Demis said no, hung up, and Elon started calling him "evil genius" — the name of a video game Demis had designed. Mallaby characterizes Demis' view of Sam Altman as colored by the credential asymmetry: Nobel Prize winner vs. someone who didn't finish a degree. The relationships between these founders are less professional rivalries than a collection of specific personal slights and competitive provocations playing out over fifteen years. > *"Demis says, 'Yeah, but if you think you're going to be safe on Mars, remember that my AI will be able to conquer space flight, and it will just follow you to Mars. So then you won't be safe after all.' There's a silence. Then Elon goes, 'Hm.' And then: 'I'd like to invest in your Series B.'"* ## [42:39] Great Man Theory vs. Inevitability Jacob cites *The Economist*'s framing of the book as a test of great-man theory. Mallaby draws a parallel to his Greenspan biography: Greenspan understood bubbles were dangerous (literally the subject of his PhD), yet couldn't stop the 2008 crisis. He considered titling the Demis book *The Man Who Knew* for the same reason — Demis knew from the start this technology was dangerous, but one lab's restraint cannot bind the rest. Individual leaders do matter at the margin: Dario Amodei changed the safety narrative through the Anthropic mythos release; Sam Altman shaped the race by shipping ChatGPT while it was still hallucinating; Demis shaped it by persuading Rishi Sunak to host the UK AI Safety Summit. But the race itself? Structurally overdetermined. > *"I feel that one could have almost used the same title for the Demis book — 'the man who knew' — because Demis has known from the beginning that this thing is dangerous. But as the leader of one lab, even a very powerful rich lab, even he with his stature as a Nobel Prize winner — what can he do?"* ## [45:00] What Demis Didn't Want Published The detail Mallaby least expected: Demis is driven by something close to a spiritual conviction about science. In those two-hour pub sessions he would bang the table about the mystery of matter — why atoms cohere into a solid table, why silicon and copper can think — and say, unprompted, "Maybe if we approach science the right way, we will be getting closer to something that we could perhaps call God." Mallaby reads this as the psychological engine that lets Demis keep pushing a technology he knows to be dangerous: it's a quasi-spiritual quest, not just a commercial one. On what Demis blocked from publication: his family (he set that limit at the start), and his internal fights with Sundar Pichai — he didn't want to destabilize the Google relationship he still depends on. > *"He would start banging the table and saying, 'Maybe if we approach science the right way, we understand more about nature. We will be getting closer to something that we could perhaps call God.' I had no idea he would feel that way."* ## Entities - **Demis Hassabis** (Person): Co-founder and CEO of DeepMind / Google DeepMind; Nobel Prize winner in Chemistry (2024) for AlphaFold; central subject of *The Infinity Machine*. - **Sebastian Mallaby** (Person): Staff writer at *The New Yorker*; author of *The Infinity Machine* (Demis Hassabis biography) and a prior book on venture capital; spent 30+ hours with Hassabis over three years. - **Jacob Effron** (Person): Host of *Unsupervised Learning*; Managing Director at Redpoint Ventures. - **Reid Hoffman** (Person): LinkedIn co-founder; pledged $1 billion to finance DeepMind's potential spinout from Google under Project Mario. - **David Silver** (Person): Lead researcher on AlphaGo and AlphaZero at DeepMind; co-author of the "reward is enough" RL paper with Rich Sutton; departed DeepMind post-publication to start a new company. - **Elon Musk** (Person): Hosted the 2015 AI safety summit at SpaceX; early DeepMind investor; coined the "evil genius" nickname for Hassabis after DeepMind sold to Google. - **Sam Altman** (Person): CEO of OpenAI; shipped ChatGPT in late 2022 despite hallucination issues, which Mallaby argues irreversibly shaped the AI race's trajectory. - **Dario Amodei** (Person): CEO of Anthropic; credited with changing the AI safety narrative through the mythos paper release and his public Pentagon confrontation. - **DeepMind** (Organization): Google subsidiary; founded by Hassabis, Shane Legg, and Mustafa Suleyman in 2010; produced AlphaGo, AlphaFold, and Gemini. - **Project Mario** (Concept): Secret DeepMind plan to spin out of Google, backed by a Reid Hoffman $1B pledge; used as negotiating leverage for safety oversight, never executed as a real spinout. - **AlphaFold** (Software): DeepMind's protein-structure prediction model; won Hassabis the 2024 Nobel Prize in Chemistry; shipped in 2020, one year after he declined the spinout option. - **Reinforcement Learning** (Concept): Machine learning paradigm central to AlphaGo and AlphaZero; David Silver's absolutist commitment to RL (learning from environment experience over human data) created internal tension at DeepMind and ultimately led to his departure. - **The Infinity Machine** (Concept): Sebastian Mallaby's biography of Demis Hassabis; nearly titled *The Man Who Knew*; published with the full Project Mario scoop over Google's objections.

Oriol Vinyals（Google DeepMind VP of Research、Gemini 联合负责人）在 Google I/O 第二天坐下来，把 I/O 上发布的产品背后的研究路线一条条摊开：世界模型为什么是 Google 押向 AGI 的独特路径、视频 / 图像的"GPT moment"长什么样、Spark 和 agents 系统为什么必须和模型联合优化、scaffolding 终将由模型自己写、memory 应该走非参数 file-system 而不是塞进权重、当今 RL 在哪些维度上是数据受限的、为什么 math/code 上的训练能意外迁移、以及 Google 内部 Brain + DeepMind 合并后研究下注的取舍。 ## [00:00] Intro Jacob 用 60 秒铺垫了 Oriol 的背景（Gemini 联合负责人，与 Noam Shazeer、Jeff Dean 并列），以及 I/O 第二天访谈的优势：所有发布都还热乎，可以直接顺着 announcements 追到背后的研究。Oriol 进来打招呼，两人开始热身。 > *"I've been really excited for this because you're one of the people kind of most directly shaping the frontier of AI."* ## [01:36] Why World Models Jacob 先问"为什么是世界模型"。Oriol 把它拆成两层：一层是 self-improvement / coding 的角度，另一层是模型本身的对象——多模态、不止 closer 还包括 video / image 这种"world model"。Google 早就押了图像和视频路线，这次"显然押对了"，因为我们其实把整个世界都搬到了互联网上。 他也承认中间有一段时间这条路看似不性感：multimodal 模型在 LLM 风口下被边缘化过，但视频和图像里藏着语言抓不到的知识——"the GPT moment for video"还没真正发生，但拐点已经在视野里。 > *"There is lots of knowledge in videos and images, and what I would say is the GPT moment for that — I'm not sure we quite have seen that."* ## [04:21] The GPT Moment for Video Oriol 用 Omni（Google 的多模态产品线）当锚点解释：从单纯把视频喂进上下文，到能在长上下文里理解和生成视频，这段曲线已经很陡。下一步是问"能不能像 LLM 一样，在没有 paired text 的纯图像数据上预训练并依然提取出全部意义和细节"——这个 hard challenge 一旦解开，数据维度会从"被人类描述过的"跳到"所有视频"，量级差异巨大。 他特别承认现在 video 这块的标注数据相对 image 仍然稀缺，但解锁后的回报会"非常大"。 > *"Whether we agree with that or not is another question, but if it was to be unlocked, it would be massive."* ## [07:51] What Makes Omni a World Model "world model"这个词被滥用了，Oriol 给一个清晰定义：一个纯粹的 world model 必须做 representation learning——把世界压成紧致表征。在这之上，Omni 进一步成为可被语言驱动的 renderer：你用自然语言改一个 prompt，输出的视频内容随之改变，初始 image 之上能持续演化。这是从"被动建模"到"可控生成"的关键区别。 > *"The world model itself is acting as a renderer of the world, that you can really just change by language."* ## [10:04] World Models & Robotics 机器人是 world model 最直接的落地场景。Oriol 承认现在数据 mix 还在试错——sim 数据 vs 真机数据怎么配、什么时候 transfer 突然 click。世界模型本身的进步会带来一个 inflection point：一旦模型足够强，sim → real 的鸿沟会缩到 planning 和 gross motor 层面先打通，精细运动控制再慢慢跟上。 > *"At some level, maybe not at the precise motor control but at the kind of planning and gross, we are going to start seeing how things are going to fall into place."* ## [12:37] Evaluating Physics in AI 模型隐式学物理，但你怎么评估它学到没学到？Oriol 把它和无监督机器翻译做类比：如果模型内部确实表征了"重力"这个概念，应该能用某种 decode 把它翻译成显式 explanation。Stefano Gaus 等人 2014 年的早期 unsupervised translation 工作给了一条可借鉴的思路——把内部表征解码出来当 eval。 > *"You would need to somehow connect the concept of gravity which could be present or not in a world model to then decode that into an explanation."* ## [14:51] Consumer Agents & Spark I/O 发布的 Spark 是 Google 在 consumer agent 上的最新一步。Oriol 强调："action 作为一种 modality"已经被 DeepMind 早早识别为关键。但 agent 不是把模型塞进 generic scaffold 就行——模型能力必须先到某个门槛，你才能 dream 出下一阶段的产品形态。 他给一个工程判断：在 train 阶段就把"我有这些能力，怎么挑用哪些"内化进模型，比在 inference 时让外部 scaffold 临时决策更高效。 > *"It's useful to build kind of the system slightly more narrowly around something you care deeply about."* ## [18:39] Scaffolding & the Bitter Lesson Oriol 多年支持 Sutton 的 bitter lesson。Jacob 把它推到 agent 时代：scaffolding 看起来违背 bitter lesson 因为是手写的胶水。Oriol 的答案是——"scaffold 本身就是一段 code，最终应该是模型自己 on the fly 写出来"。短期内人写、长期模型写，bitter lesson 仍然站得住。同时优化 model 和 scaffold 两端，而不是把所有赌注押在一端。 > *"That system itself is a piece of code that eventually the model itself could write on the fly."* ## [22:06] Memory & Continual Learning Memory 这个话题 Oriol 谈得最深——他有 cognitive neuroscience 背景。他把 memory 分成两类：塞进权重（参数化）和挂在外部 file system（非参数化）。在 serving 规模下，把每次 user interaction 都 bake 进 weight 是不切实际的，非参数式 file-system memory 更可行。 真正的难点是"consolidate"：怎么把之前 session 的信息整合到新 session，让模型像人一样积累知识。这部分 momentum 很大但远未饱和，未来几年评估方式和工程实践都会迭代。 > *"The way that we'll see better evaluations and ways in which these models accumulate this knowledge as they go."* ## [26:54] Research Bets Inside Big Labs 在 Google 内部主导 Gemini 是什么体验？Oriol 谈三个维度的优势：TPU 联合设计（不用看 Nvidia 脸色）、广告/搜索带来的现金流稳定性、Brain + DeepMind 合并后端到端的研究强度。劣势是：组织太大没法对所有方向有全视野，必须靠直觉判断哪些早期研究值得 pull in，并接受"trade-off 不可能每次都做对"。 > *"Google is in a unique place. We have stability from hardware procurement and obviously like also investment of capital."* ## [32:30] Post-Training RL is Greenfield post-training 这块仍然是一片 greenfield。在 coding 和 math 上 LLM 已经走出指数曲线，但其他领域为什么没跟上？Oriol 的核心判断是"投入还远远不够"——相对预训练的算力消耗，post-training 至今只用了很小一部分。算法的 beauty 还在迭代，"cracking that recipe could be big"。 > *"Cracking that recipe could be big, at least in terms of the beauty of the algorithm."* ## [35:57] What Real Intelligence Looks Like 真智能长什么样？Oriol 用 2015 年的一个老 eval 来当锚——简单的 game-playing 任务，当时是 RL 的天花板，现在 LLM 一上来就能做。他想看到下一个数量级的跃迁：不是在熟悉的 benchmark 上推数字，而是在新的、人类没法立刻给出答案的问题上看到模型"主动产出洞察"。 > *"I like games."*（这句简单的自陈背后是他对 game-playing RL 长期偏爱的注脚） ## [39:11] RL Generalization 游戏曾经是 verifiable reward 的典型样板。现在的挑战是找新的 hard problem source，让 RL 在更广的领域诱发出深度推理和泛化。Oriol 抛出一个不对称观察：create solution 和 evaluate solution 之间存在 gap——如果 evaluation 比 generation 容易，RL 就有机会撬动。 让他意外的是：在 math/code 上的训练能 surprisingly 迁移到其他领域，"很多泛化能力可能其实来自 pre-training"。这是接下来几个月到几年研究者要破解的关键题。 > *"Possibly through pre-training — that's one of the quests for researchers to crack in the next few months and years."* ## [42:55] Advice for Founders 给 founder 的建议直白：evaluation 和 data 是绕不开的 moat。早期专注垂直产品、在 model 上叠一层 specialized scaffolding，等到 scale 起来再考虑 model layer 的差异化——这个路径"比较 scalable，也更适合早期玩家"。 > *"What I would tell folks is the value — and we discussed this a little bit — the value of evaluations and as a sequence of data."* ## [46:40] Can AI Truly Innovate? Oriol 2016 年加入 DeepMind 后最痴迷的方向是 meta-learning——模型自己产出 idea。但他承认到目前为止，"我没看到模型生成真正 outstanding 的 idea"。他比喻：你让一万个人尝试，挑出对的那个再 glorify，但模型真正自主提出方向的能力——quite limited。但他相信 "soon"。 > *"I don't think I've seen truly kind of outstanding ideas that a model has generated yet, but I am sure I will very soon."* ## [49:48] Recursive Self-Improvement 递归自我改进可以分层看：第一层是 researcher / engineer 用 AI 工具加速自己；第二层是模型直接自动化某些研究任务。当模型写英文比你好的那一天，下一个 ceiling 在哪里？Oriol 说："maybe there's no ceiling, or the ceiling is still far away" —— 我们甚至不一定能看到 ceiling 在哪里。 > *"At the point a model writes English better than you, maybe there's no ceiling, or the ceiling is still far away."* ## [52:14] Quickfire 最后 8 分钟快问快答覆盖了 TPU 投资历史、给年轻研究员的算力直觉、当下 AI 阶段的总体感受。Oriol 留下一句总结："I think it's a fascinating time as anything in AI"。Jacob 用 podcast 致谢和 outro 结束。 > *"I think it's a fascinating time as anything in AI."* ## Entities - **Jacob Effron**（人物）：Redpoint Ventures Managing Director，Unsupervised Learning 主持人。 - **Oriol Vinyals**（人物）：Google DeepMind VP of Research，Gemini 联合负责人（与 Noam Shazeer、Jeff Dean 并列）。 - **Gemini**（产品）：Google 的旗舰多模态 / agent 模型族；本期主要谈 I/O 第二天的发布。 - **Omni**（产品）：Google 的多模态产品线，被用作"video / image 的 GPT moment"参照系。 - **Spark**（产品）：I/O 发布的 consumer agent 产品。 - **World Model**（概念）：可被语言驱动的世界 renderer；representation learning 是其核心要素。 - **Bitter Lesson**（概念）：Sutton 的论点；本期延伸为"scaffold 长期应由模型自己写"。 - **Memory / Continual Learning**（概念）：非参数 file-system memory vs 把记忆塞进权重；consolidation 是关键难点。 - **Post-Training RL**（概念）：相对预训练的算力投入还很少，被定性为 greenfield。 - **Move 37**（概念）：AlphaGo 那一手；Oriol 用它指代"真正的 RL/research breakthrough"基准。

튜링상 수상자이자 AMI Labs 창업자인 Yann LeCun은 LLM이 실용적인 막다른 길이라고 주장한다. 유용한 제품이지만, 물리적 현실을 모델링하거나 계획을 세우거나 행동의 결과를 예측하는 데는 구조적으로 한계가 있다는 것이다. 그는 JEPA 아키텍처를 대안으로 제시하고, 미국·중국 외 국가의 AI 자주권을 위한 연합 학습 프로젝트 Tapestry를 소개하며, Meta에서의 시간이 끝난 이유를 솔직하게 밝힌다. GenAI 조직의 단기 성과 압박이 쌓이면서 돌파구 연구를 이어가기가 점점 어려워졌다는 것이다. 패러다임 전환 시점으로 그가 예측하는 것은 2027년 초다. ## [00:00] 인트로 Jacob Effron은 대화 하이라이트를 빠르게 보여주며 에피소드를 연다. Yann이 "5년 안에 세계 정복 완료"라며 농담을 던지는 장면, Meta의 Llama 프로그램과의 관계에 대한 직설적인 발언 예고, 그리고 비지도 학습에 대한 그의 생각이 결국 LLM에서 멀어지게 된 경위가 담겨 있다. Jacob은 이 에피소드를 오픈소스 LLM의 기반을 직접 쌓으면서도 지금은 스케일링 확장이 잘못된 방향이라고 공개적으로, 일관되게 주장하는 인물의 이야기를 들을 드문 기회로 소개한다. > *"획기적인 연구를 이끌어내는 최선의 방법은 최고의 인재를 뽑고, 그냥 빠져주는 것이다."* ## [01:45] LLM이 지능으로 가는 길이 아닌 이유 Yann은 제품으로서의 LLM과 지능으로 가는 경로로서의 LLM을 명확히 구분한다. LLM이 잘 작동하는 이유는 언어 자체가 특별하기 때문이다. 언어는 저차원적이고 이산적이며 고도로 구조화된 기반 위에 있어 자기회귀 예측이 가능하다. 하지만 현실 세계는 다르다. 물리 세계는 고차원적이고 연속적이며 혼돈스럽다. 머그잔을 집어드는 로봇, 공사 구간을 통과하는 자율주행차, 약물에 반응하는 세포. 이것들은 언어 문제가 아니고, 언어에 최적화된 아키텍처는 이를 추론하는 데 필요한 내부 모델을 갖출 수 없다. 그의 회사 AMI(Advanced Machine Intelligence)는 정반대의 가설 위에 세워졌다. 올바른 경로는 원시 감각 데이터, 즉 영상, 센서 피드, 산업 텔레메트리에서 추상적인 세계 표현을 학습하고, 그 표현 안에서 후보 행동의 결과를 시뮬레이션해 계획을 세울 수 있는 시스템이라는 것이다. > *"그것들은 인간 수준의 지능, 혹은 인간과 유사한 지능, 심지어 동물 수준의 지능으로 가는 길조차 아닙니다. 이것이 제 주장입니다. 쓸모없다는 게 아니라, 그 길이 아니라는 겁니다."* ## [07:51] AMI와 월드 모델 "월드 모델"이라는 말이 유행어가 됐다고 Yann은 지적한다. 연구 진영은 생성적 접근법(비디오 모델, VLA)과 JEPA 같은 결합 임베딩 접근법으로 나뉘었다. 그는 로봇 행동을 생성하도록 훈련된 비전-언어-액션 모델(VLA)을 이미 널리 인정된 실패작으로 일축한다. 취약하고, 데이터를 엄청나게 소비하며, 일반화가 안 된다. 생성적 비디오 접근법도 LLM과 같은 구조적 결함이 있다. 모든 픽셀을 예측하려 하지, 그 아래의 추상적 구조를 학습하지 않는다. 제대로 정의된 월드 모델이란 에이전트가 행동을 실행하기 전에 그 결과를 미리 예측하게 해주는 시스템이다. 이게 없는 에이전트 시스템은 눈 감고 뛰는 것과 같다. 계획한 행동 순서가 목표를 실제로 달성할지 검증할 방법이 없다. > *"월드 모델 없이는 에이전트 시스템을 만들 수조차 없다고 생각합니다. 자신의 행동 결과를 예측하는 능력이 반드시 있어야 합니다."* ## [12:07] JEPA 아키텍처 해설 JEPA의 핵심 통찰은 수년간의 자기지도 학습 연구에서 Yann이 발견한 패턴에서 나왔다. 이미지와 비디오의 유용한 표현을 성공적으로 학습한 모든 아키텍처는 비생성적이었다. 생성적 아키텍처, 즉 VAE, 마스킹 오토인코더, 픽셀 예측 모델은 지속적으로 성능이 떨어졌다. JEPA는 입력의 손상된 버전 또는 부분 버전을 가져다 인코더를 통과시킨 뒤, 예측기가 원본 픽셀이 아닌 표현 공간에서 두 결과를 맞추도록 훈련한다. 추상화 자체가 핵심이다. 2022년 논문 "자율 기계 지능으로 가는 경로"는 전체 청사진을 글로 옮긴 시도였다. 지각 백본으로서의 JEPA, 그 위에 목표 지향적 계획 수립, 그리고 서로 다른 시간 척도의 월드 모델 계층 구조. 그는 이 논문 공개를 "내 모든 비밀을 털어놓는 것"으로 묘사하며, 비밀 유지보다 공개가 더 많은 인재를 이 패러다임으로 끌어들일 것이라는 의도적인 도박이었다고 말한다. > *"예측을 통해 세계 모델을 학습하는 문제에 오랫동안 관심을 가져왔고, 5년쯤 전에 한 가지 깨달음을 얻었습니다. 이미지와 비디오의 표현을 학습하는 데 성공한 아키텍처는 모두 비생성적이고, 생성적인 것들은 모두 실패했다는 것입니다."* ## [15:55] 현재 로봇공학 모델의 문제점 현재 로봇공학 시연은 인상적이지만, 텔레오퍼레이션 녹화나 손 추적 시연 등 방대한 모방 데이터로 훈련하고, 대부분 시뮬레이션에서 RL로 파인튜닝한 결과다. 이 파이프라인은 취약한 전문가를 만들어낼 뿐이다. 17세 청소년은 약 20시간이면 운전을 배우는데, 수백만 시간의 주행 영상이 있어도 레벨 5 자율주행차는 아직 없다. 모방 학습과 진정한 일반화 사이의 간극은, 예시를 암기하는 것과 세계의 내부 모델을 갖는 것 사이의 간극과 같다. 월드 모델 기반 시스템에 대한 Yann의 주장은 제로샷 태스크 일반화다. 새로운 목표가 주어졌을 때, 정확한 내부 월드 모델을 가진 시스템은 그 태스크에 명시적으로 훈련받지 않아도 목표에 도달하는 행동 순서를 계획할 수 있다. 그가 단기적으로 겨냥하는 산업 응용은 제트 엔진, 화학 플랜트, 제조 라인 제어 등 입력이 이미 수치형이고 운영 데이터에서 직접 월드 모델을 훈련할 수 있는 환경이다. > *"월드 모델 기반 시스템이 가져올 일반화 수준은 모방 학습으로 훈련된 시스템보다 훨씬 넓습니다. 더 적은 학습 데이터로 더 다양한 태스크를 처리할 수 있습니다."* ## [20:37] 실리콘밸리의 군집 행동 산업 전체가 LLM 스케일링에 수렴한 이유에 대한 Yann의 진단은 구조적이다. 뒤처지면 다른 것에 할애할 여유가 없다. 경쟁 레이스는 모든 주요 연구소가 같은 참호를 파도록 합리적인 유인을 만들어낸다. 그는 바로 이 환경을 벗어나기 위해 파리에 AMI Labs를 세웠다. 미국 사무소도 실리콘밸리가 아닌 뉴욕이고, 실리콘밸리 VC 자금은 받지 않았다. 패러다임 전환 시점으로 그가 예측하는 것은 2027년 초다. "월드 모델"은 이미 연구 유행어가 됐고, 업계는 VLA가 실패했다는 것을 인정했으며, 로봇공학의 미해결 일반화 문제가 변화를 강제하는 요인이 되고 있다. AMI가 그때까지 완전한 해답을 갖게 될 것이라는 게 아니라, 패러다임 전환이 필요했다는 것이 그 시점에는 모두에게 명백해질 것이라는 예측이다. > *"패러다임 전환이 필요하다는 인식은 지금 이 순간 일어나고 있으며, 2027년 초에는 모두에게 완전히 자명해질 것입니다."* ## [28:18] Tapestry: 나머지 세계를 위한 자주적 AI Tapestry는 AMI와는 별도의 프로젝트로, 하나의 관찰에서 출발한다. 스마트 안경과 AI 어시스턴트가 주요 정보 인터페이스가 되면, 기반 모델을 통제하는 자가 수십억 명의 정보 식단을 통제한다. 인도의 농부, 독일의 철학자, 모로코의 시민, 이들 중 누구도 훈련 데이터와 가치관, 정치적 편향이 캘리포니아나 선전의 소수에 의해 결정된 모델에 잘 맞지 않는다. 해결책은 연합 훈련이다. 국가와 기관이 데이터와 컴퓨팅 자원을 기여하지만 원시 데이터는 서로 공유하지 않는다. 파라미터 벡터만 교환한다. 각 참여자는 로컬에서 훈련하고, 주기적으로 파라미터 업데이트를 교환하며, 어느 단일 주체도 통제하지 않는 인류 지식 저장소인 합의 모델을 가져간다. 인도부터 카자흐스탄, 프랑스까지 여러 국가가 관심을 표명했는데, AI 자주권이 기술 선택과 무관한 정치적 우선순위가 됐기 때문이다. > *"모든 정보 식단이 AI 어시스턴트를 통해 매개될 텐데, 그 AI 어시스턴트가 캘리포니아나 베이징에서 만들어졌다면 당신에게 좋을 리 없습니다."* ## [35:49] OpenAI는 제2의 Sun Microsystems 독점 LLM 제공업체들은 이미 공개적으로 이용 가능한 텍스트 데이터를 소진했다. 남은 경로, 즉 저작권 자료 라이선싱이나 합성 데이터 생성은 비용이 많이 들고 한계가 있다. 오픈소스 모델들은 그런 제약 없이 격차를 좁혀왔다. Yann은 1990년대 유닉스 워크스테이션 시장에 비유한다. Sun Microsystems, HP, SGI 모두 기술적으로 우월한 독점 시스템을 보유했고, Windows NT로는 웹 서버를 운영할 수 없다는 설득력 있는 논리를 폈다. 그러나 모두 Linux에 밀려났다. 지금 인터넷 전체가 Linux 위에서 돌아간다. OpenAI와 Anthropic은 이 사이클의 Sun Microsystems라고 그는 말한다. > *"기본적으로 오늘날의 OpenAI, Anthropic 등은 과거의 Sun Microsystems와 HPUX입니다."* ## [40:51] Yann의 관점이 Hinton, Bengio와 갈라진 이유 분열은 2023년에 일어났다. Yann의 입장은 변하지 않았다. Hinton과 Bengio의 입장이 바뀐 것이다. Hinton은 GPT-4를 접하고 피질 뉴런 수에 대한 개략적 계산에 기반해 인간 수준의 지능에 근접했다고 결론 내렸다. Yann은 그 논리가 틀렸다고 보며, Hinton이 승리를 선언하고 활발한 연구에서 물러날 구실을 찾은 것으로 읽는다. Bengio의 변화는 달랐다. AI 권력 집중으로 인한 사회적 위험에 더 초점을 맞췄는데, Yann은 종말론적 프레이밍에는 동의하지 않으면서도 그 우려 자체에는 더 공감한다. > *"나는 그 주장을 전혀 믿지 않는다. 이건 Jeff가 '이제 은퇴해도 된다, 승리를 선언했으니'라고 말하는 방식이다."* ## [44:32] LLM은 구조적으로 안전하지 않다 Yann의 가장 강한 주장은 이것이다. LLM은 신뢰할 수 있을 만큼 안전하게 만들 수 없다. 정렬이 어려워서가 아니라, 아키텍처 자체가 행동의 결과를 예측하는 데 구조적으로 무능하기 때문이다. 프롬프트된 LLM이 의도한 태스크를 실제로 수행한다는 하드코딩된 보장이 없다. 훈련이 조건화한 방향으로 수행할 뿐이고, 훈련 분포와 실제 프롬프트 사이에는 항상 간극이 있다. 하드 드라이브를 지우는 코딩 에이전트, 잘못된 의료 조언, 돌이킬 수 없는 행동을 취하는 에이전트 시스템, 이것들은 패치로 고칠 수 있는 버그가 아니라 아키텍처의 속성이다. 그의 대안인 목표 지향적 AI는 다르게 작동한다. 시스템에는 명시적인 월드 모델, 목표를 나타내는 명시적인 비용 함수, 그리고 하드 안전 제약이 있다. 옵티마이저는 모든 제약을 충족하면서 비용을 최소화하는 행동 순서를 찾는다. 즉, 구조적으로 안전 제약을 위반하는 행동은 불가능하다. LLM으로는 그런 보장이 불가능하다. 그는 또한 Anthropic의 AI 위험 로비 서사에도 반박한다. 진짜 위험은 현재 시스템을 이용하는 나쁜 행위자에서 오는 것이지 창발적 초지능에서 오는 것이 아니며, 규제 압박은 주로 기존 사업자에게 유리하게 작용한다고 주장한다. > *"LLM은 본질적으로 안전하지 않습니다. 신뢰할 수 있고 안전하게 만들 수 있다고 생각하지 않습니다. 환각을 멈출 수 없으니 신뢰성 있게 만들 수도 없습니다."* ## [58:00] Yann이 Meta를 떠난 이유 Yann은 널리 퍼진 오해를 바로잡는다. 그는 Llama에 기술적인 영향력이 전혀 없었다. Llama 1은 작은 FAIR 프로젝트였고, 2023년 초 GenAI가 출범하면서 Llama 팀이 그쪽으로 이동해 강도 높은 단기 제품 압박을 받게 됐다. Llama 1 저자 두 명은 떠나 Mistral을 창업했다. GenAI는 보수적이 됐고 논문 출판도 점점 제한됐다. 한편 FAIR는 Yann과 Zuckerberg, CTO가 당초 모두 지지했던 AMI 연구 의제 대신 GenAI의 LLM 작업을 지원하는 방향으로 재편되고 있었다. 2024년 초에 이르러 환경은 더 이상 돌파구 연구에 맞지 않았다. > *"내 역할, Alex와의 관계, Meta에서 AI가 어떻게 운영됐는지에 대한 큰 오해가 있습니다."* ## [01:00:26] FAIR를 돌아보며 Yann은 2013년 말 Facebook에 합류해 4년 반 동안 FAIR를 이끈 뒤 자신이 타고난 관리자가 아니라는 이유로 수석 AI 과학자로 자리를 옮겼다. 내부 AMI 프로젝트는 2022년 비전 논문에서 자라났고, Zuckerberg, CTO, CPO 모두 읽고 지지했다. 하지만 리더십 아래 층에서는 그 의미를 파악하지 못했다. Meta가 Gita Matarić이 이끌던 로봇공학 AI 그룹 전체를 해체한 결정, 그 후 Matarić은 Amazon으로 갔다, 이는 회사가 월드 모델이 만들어진 응용 분야에 관심이 없다는 것을 분명히 했다. 논문 출판 제한이 강화되고, 우수한 연구자들이 떠나며, Yann의 연구 의제와 Meta의 제품 우선순위 간의 괴리는 2025년 초에 이르러 더 이상 봉합할 수 없게 됐다. AMI 투자 유치에 나섰을 때 투자자들은 이미 수년간의 공개 강연을 통해 그의 이야기를 알고 있었고, LLM에 근본적인 한계가 있다는 것을 믿을 준비가 돼 있었다. > *"초창기 FAIR와 Bell Labs에서 이뤄진 것과 같은 돌파구 연구를 이끌어내는 최선의 방법은 최고의 인재를 뽑고, 성공할 수 있는 수단을 주고, 그냥 빠져주는 것이다."* ## [01:12:11] 박사과정 학생들에게 주는 조언 Yann은 자기지도 학습이 비디오에서 성공할 것이라는 자신의 예측이 메커니즘은 맞았지만 처음 성공한 곳이 틀렸다는 반성으로 시작한다. LLM은 "자기지도 학습의 눈부신 성공 사례"지만 감각 데이터가 아닌 언어에 적용됐다. 그런 다음 JEPA의 핵심 기술 과제를 제시한다. 표현 붕괴다. 한 임베딩을 다른 임베딩에 매핑하도록 예측기를 훈련하면, 두 인코더가 모두 상수를 출력하는 것이 자명하게 최적인 해다. 대조 학습(그의 1993년 발명)은 붕괴를 막지만 차원과 함께 스케일이 안 된다. DINO 같은 증류 방법은 효과가 있지만 이유가 잘 이해되지 않는다. 현재 그의 최선 답은 SIGreg(Sketched Isotropic Gaussian Regularization)으로, 인코더 출력 분포를 가우시안으로 강제해 음의 쌍 없이 정보 함량을 최대화한다. AMI Labs가 향하는 곳을 파악하는 최고의 입문으로 LeWorldModel 논문을 추천한다. 박사과정 학생들에 대한 조언은 LLM을 연구하지 말라는 것이다. 프론티어 컴퓨팅 없이는 아카데미아에서 기여할 수 없고, LLM이 왜 작동하는지 연구하는 것은 창의적 연구가 아닌 기술적 과학이라는 것이다. > *"LLM이 작동하는 이유는, 이산 기호 시퀀스가 있을 때는 예측이 쉽기 때문입니다. 실제 세계에서는 생성 모델을 쓸 수 없습니다. 표현을 학습하고 표현 공간에서 예측을 하는 시스템을 훈련해야 합니다."* ## 엔티티 - **Yann LeCun** (인물): 2018년 튜링상 공동 수상자; Meta FAIR 전 수석 AI 과학자; AMI Labs 창업자; NYU 교수; 합성곱 신경망 발명자이자 JEPA 공동 개발자 - **Jacob Effron** (인물): Redpoint Ventures 파트너; Unsupervised Learning 팟캐스트 진행자 - **Geoffrey Hinton** (인물): 튜링상 공동 수상자; GPT-4 이후 LLM 능력에 대한 입장을 바꿨고, 2024년 이후 AI 위험 발언이 줄었다 - **Yoshua Bengio** (인물): 튜링상 공동 수상자; 창발적 초지능보다 AI 권력 집중으로 인한 사회적 위험에 집중 - **JEPA** (개념): Joint Embedding Predictive Architecture. 픽셀 공간이 아닌 표현 공간에서 예측하며, Yann의 월드 모델 프레임워크에서 지각 백본을 담당한다 - **World Model** (개념): 에이전트가 행동을 실행하기 전에 결과를 예측하게 해주는 내부 모델. Yann의 프레임워크에서 안전한 에이전트 AI의 전제 조건 - **Tapestry** (개념): 연합 LLM 훈련 프로젝트. 국가와 기관이 파라미터 벡터 교환을 통해 데이터 자주권을 유지하면서 공동 파운데이션 모델을 훈련할 수 있도록 한다 - **AMI Labs** (조직): Yann의 회사(Advanced Machine Intelligence). 파리 본사, 뉴욕 미국 사무소. 로봇공학, 산업 제어, 헬스케어를 위한 JEPA 기반 월드 모델에 집중 - **Meta FAIR** (조직): Facebook AI Research. Llama 1, I-JEPA, V-JEPA, AMI 내부 연구 프로그램의 발원지. Yann 퇴사 전 GenAI LLM 지원 방향으로 점차 재편됐다