PodcastsHear the voice. See the shape of the thought.

Tara Agyemang from the Google Chrome DevRel team presents WebMCP, a proposed web standard that replaces the brittle screen-scraping loop today's AI agents run through — DOM parsing, accessibility tree analysis, screenshot pixel math, coordinate clicks — with a clean menu of named, typed, described tools the browser exposes directly. Two API paths cover most sites: a declarative API that auto-generates JSON schemas from HTML form attributes, and an imperative API for registering custom JavaScript tools with explicit execute blocks. A live demo buys concert tickets in exactly three tool calls, and the spec is already testable in Chrome 146 via a side-panel inspector extension. ## [00:15] The DOM-scraping problem: what agents go through today Buying two tickets to an Afro Beats Festival sounds simple. For a current AI agent it means: parse the full HTML DOM, walk the accessibility tree, take a screenshot, do pixel-coordinate math to find the button, click — and then discover an ad has loaded and pushed everything 200 pixels south. Agyemang walks through each step live using a Gemini-in-Chrome side panel against a demo ticket site, making visible just how many tokens and how many fragile inferences sit between a user's natural-language request and a form submission. > *"It can be brittle, and I don't even want to guess at how many tokens you probably just used trying to do something simple. It's probably a lot."* ## [03:02] Accessibility first: the prerequisite before WebMCP Before reaching for WebMCP, Agyemang flags a prerequisite: semantic HTML and solid accessibility standards are not optional groundwork — they are what makes a site legible to agents by default. Proper ARIA roles, meaningful labels, and logical DOM structure collapse much of the agent's interpretation work even without any new API. > *"Making your site accessible for everyone makes it accessible to AI agents by default."* ## [03:53] What WebMCP is: a structured tool menu for agents WebMCP is a proposed web standard (not yet finalized) whose core idea is to flip the information asymmetry: instead of every agent reverse-engineering what a site does, the site author declares a menu of tools — named, typed, described — that agents can call directly. Agyemang borrows the USB-C analogy: any conforming agent speaks the same protocol, and any conforming site answers back. > *"Instead of any agent guessing what your website does, you're kind of giving them a menu of tools that they can use to interact with your site."* ## [04:43] Demo: navigating a maze with WebMCP tools The first live demo uses a maze escape game built by the Chrome DevRel team, shown alongside the Model Context Tool Inspector — a Chrome extension that lists every tool the current page exposes. At page load only one tool exists: `start_maze_game`. After calling it, the tool list expands to directional move tools (`north`, `south`, `east`, `west`), a look tool, and item management tools. Agyemang then types freehand prompts ("right, up, right again"; "complete the maze") and the Gemini 1.5 agent maps each instruction to the correct tool call, iterating autonomously. The maze is deliberately navigable only through the agent interface — no clickable buttons exist — which makes the tool-call loop the only path through. > *"The AI agent should use my prompt, match it to the specific tools, so in this case, the move tool. It's taken my direction of down and right, matched that to the north, south, east direction, and sent that through."* ## [09:58] WebMCP vs MCP: client-side vs server-side The question Agyemang anticipates most: isn't this just MCP? The distinction is scope. MCP connects agents to server-side applications and data sources. WebMCP implements the tools portion of MCP but runs entirely in the browser — the browser window must be open, and all tool execution happens client-side in the page's JavaScript context. She likens the relationship to JavaScript and Java: inspired by, not interchangeable with. The practical implication is that WebMCP covers the slice of agent work that is inherently tied to what a user has in front of them: filling complex multi-step forms, navigating stateful UI flows, personalizing a shopping session based on what's visible on screen. > *"Web MCP allows engineers to provide tools to in-browser AI agents. And it's very specific for the client-side features."* ## [12:35] The two APIs: declarative and imperative WebMCP offers two implementation paths. The **declarative API** requires only a few new HTML attributes on existing form elements (`tool-name`, `tool-description`); the browser generates the full JSON schema automatically. A boolean `agent-invoked` attribute lets the server distinguish agent submissions from human ones. The **imperative API** is for anything more complex: developers call `registerTool()` with a schema object they build manually, attach a description with enough detail for an agent to choose it correctly, and write an `execute` block containing ordinary DOM JavaScript — validate input, call existing functions, manipulate state — then return a result object so the agent knows what happened. The imperative path is currently more common because most real-world flows go beyond a single form. > *"The execute block is essentially where you call normal JavaScript. So, maybe you already have functions that you're using that you can call in here."* ## [15:16] Demo: buying concert tickets in three tool calls Back to the original ticket-buying scenario, this time on the WebMCP-instrumented demo site. Agyemang types: "Buy two VIP tickets to Summer Vibes Festival." Gemini 2.0 (upgraded from 1.5 for this demo) makes exactly three tool calls: `search_concerts` to find the event by name, `open_concert_page` with the returned concert ID to navigate to the right page, and `purchase_ticket` with quantity and section parameters. The UI updates in sync at each step — section selector, quantity picker — and the agent pauses before final checkout, surfacing the total (£356) so the user can confirm. Agyemang notes this last manual confirmation step is intentional: for real-money transactions, the human should always see what's about to happen before the agent commits. > *"You spent £356. Great, I'll put that on the Google's credit card."* ## [17:46] Getting started: Chrome 146, the inspector, and how to give feedback WebMCP is in early preview on Chrome 146+. Agyemang recommends Chrome Canary to keep experimental flags isolated from a daily-use profile. Setup requires enabling the `chrome://flags/#web-mcp` testing flag, then installing the Model Context Tool Inspector from the Chrome Web Store. Two resources cover the rest: a sign-up blog post for the early preview program (gives access to initial docs, best practices, and example implementations) and a GitHub repository with all demos — including the maze — plus an eval CLI for automated testing against a site's declared tools. The API is changing week to week; Google is actively looking for friction reports and bug filings before the spec stabilizes. > *"We don't have to settle for these brittle screen-scraping processes that we have today. Instead, we can use Web MCP tools to turn every website into a high-performance API for agents."* ## Entities - **Tara Agyemang** (Person): Developer Relations Engineer on the Google Chrome team; presenter and WebMCP advocate; GitHub/X handle @taraojo. - **WebMCP** (Concept): Proposed web standard that exposes structured, typed tools from a web page to in-browser AI agents, eliminating DOM-scraping; still experimental as of Chrome 146. - **MCP (Model Context Protocol)** (Concept): The parent protocol WebMCP draws from; MCP connects agents to server-side applications, while WebMCP handles client-side browser tool exposure. - **Declarative API** (Concept): WebMCP implementation path using HTML attributes on existing form elements; browser auto-generates JSON schema. - **Imperative API** (Concept): WebMCP implementation path using `registerTool()` in JavaScript; supports arbitrary DOM logic in the `execute` block. - **Model Context Tool Inspector** (Software): Chrome side-panel extension built by Chrome DevRel that lists all tools a WebMCP-enabled page exposes; available in the Chrome Web Store. - **Google Chrome DevRel** (Organization): Google team building WebMCP, the maze demo, the inspector extension, and the eval CLI; manages the early preview program. - **Gemini** (Software): Google's AI model used as the in-browser agent in both demos; demo upgraded from Gemini 1.5 to Gemini 2.0 for the ticket-buying scenario.

Garrett Galow, head of product at WorkOS, built Studio to kill the "explain your question, wait for an engineer, get an answer, realize you need one more join, share a one-off in Slack" loop that plagues every company with non-technical stakeholders. Studio lets anyone query Snowflake, Linear, and Notion in plain English, get a live answer, and — crucially — turn that answer into a deterministic, reusable widget whose code runs directly against the data sources without involving the LLM again. The reliability comes from three engineering choices: preflight sequencing that injects schema context only when a tool is actually invoked, a layering rule that tells the model to explicitly distrust its own knowledge about WorkOS products and pull from primary sources, and a validation step that runs every generated Snowflake query before hardcoding it into a widget. ## [00:14] WorkOS and Today's Talk Galow opens with a 10-second company pitch — WorkOS is the enterprise platform layer that powers SSO and other developer-facing features for Cursor, Anthropic, and OpenAI — and immediately flags that he is not here to talk about that. The session is about how WorkOS operates internally and what they built to make the whole team, not just engineers, faster at answering questions about the business. > *"If you've ever logged into Cursor, you've used WorkOS — whether that was username password or you went through your enterprise IDP."* ## [01:02] The Slow Loop of Business Questions The problem Galow describes is familiar: a go-to-market or support teammate has a question, cannot write SQL themselves, has to explain it to an engineer, waits, gets a partial answer, asks for one more join, gets another partial answer, and eventually receives a one-off table in Slack that is immediately stale. Even Retool or internal dashboards fail here because they are built for a fixed question — the moment someone needs one extra filter or one extra column the whole request cycle restarts. > *"Someone has a question, often about the business. They may not be technical enough to go answer it themselves. They have to explain their question, why they need it answered, the context to answer it. They wait."* ## [02:33] Studio Demo: From Question to Live Dashboard Studio is an internal workspace (web dashboard plus Slack bot) backed by a LangGraph agent running Claude Opus, connected to integration proxies for Snowflake, Linear, and Notion. Galow fires off a live question: which content on the WorkOS marketing site drives the most new team sign-ups? The agent runs preflight checks, determines it needs Snowflake, pulls schema context at the moment of invocation, issues several queries, and returns a ranked table in roughly 90 seconds. The more interesting part comes after: he asks Studio to turn that answer into a reusable widget with time-slice filters. The widget is declarative JavaScript that calls the underlying APIs directly. On every subsequent run the LLM is not involved at all — it is just code re-executing queries against Snowflake. The on-screen result shows blog posts, changelogs, and docs ranked by conversion to sign-ups, filterable by content category. > *"A widget is basically like sandbox code that runs — it's both the UI, the APIs, and the query necessary to power a fully usable tool."* ## [07:34] Radar Support Widget: Self-Serve for the Support Team Galow walks through a second widget built for WorkOS's support team around Radar, their bot-blocking security product. When a customer asks "why did this user get blocked?", support reps used to pass around ad-hoc SQL queries or wait for a data engineering ticket. The Radar widget lets any support rep type in a customer email, and the widget re-runs its hard-coded queries live against the database, returning the full login-attempt history and whether each attempt was flagged. Support staff can build these widgets themselves: if a question is genuinely one-off, they get the answer ad hoc; if the same question keeps recurring, they build a widget and share it internally. No platform team involvement required. > *"Our support team can basically, if it's a one-off, get the question answered themselves; and if they're finding that they're actually asking the same question a lot, they can build these and then share them internally to other folks."* ## [09:55] Three Pillars: Sequencing, Layering, Validation The reliability section is the technical heart of the talk. Galow names three design choices that made Studio usable enough to hand to non-engineers. **Sequencing** — before doing anything, the agent runs preflight checks: are all integrations connected? Does it have enough context to answer the question? If not, it asks for clarification. Schema context for each data source is injected only at the moment a specific tool is invoked, not upfront, keeping the context window clean for the actual reasoning. **Layering** — the prompt stack has a base layer (Studio defaults), an org layer (shared rules), and a tool-edit layer (session-specific context). Crucially, the model is explicitly told to distrust its own knowledge about WorkOS's products, because model training data goes stale fast and the product changes constantly. It is directed to pull from internal docs and live data sources instead. **Validation** — every Snowflake query the agent writes is executed before being committed to a widget. A query can be syntactically valid SQL and return zero rows; if the agent does not notice that, the widget ships as broken. Running the query first catches that failure mode before it becomes a user-facing truth. > *"We tell the LLM to specifically distrust knowledge around our product — sometimes the model training is using outdated data. Our product changes very quickly. So we actually tell it: no, go for primary sources, look up data in our docs."* ## [12:54] Q&A: Schemas, Governance, Cross-Tool Queries, and Access Three audience threads surface practical design decisions. **Dirty schemas**: a questioner asks whether Galow had to clean up Snowflake before Studio could use it. He did not. The hard joins — customer entity to users, four levels deep — are encoded once in the Snowflake context block; the LLM learns the quirks from that description rather than from a tidy schema. No RAG database, no schema rewrite. The guidance block does need to encode filter-column discipline (e.g. "only pull non-deleted entities") because models miss those silently. **Widget governance**: an audience member raises the trust problem — a widget that generates a query incorrectly becomes a "truth" that no one ever questions. Galow acknowledges this is real but says the hit rate has been high enough in practice. Embedding data-quality rules directly in the context block (active status filters, soft-delete guards) removes most silent errors; the remaining ones tend to be large enough to be obvious. **Cross-tool widgets and architecture**: asked whether widgets can draw from multiple tools simultaneously, Galow confirms they can — a widget can call Snowflake and Linear in one interface. The widget is JavaScript; it makes the underlying API calls independently, and merging the data is just code. Once a widget is generated, it is entirely deterministic: no LLM call on refresh, no inference cost, no variability. **Access control**: per-user OAuth is the current model (each employee connects their own Snowflake and Linear credentials), which is awkward. WorkOS is building "org connectors" via their own Pipes product — one admin sets up a connection, then role-based rules govern what each user can read or edit within that connection. > *"The actual final product is very reliable in that regard. The LLM's not involved once the widget is developed — until I go back and say, 'Hey, can you make an adjustment to this widget?'"* ## Entities - **Garrett Galow** (Person): Head of product at WorkOS; built and presented Studio. - **WorkOS** (Organization): Developer platform providing enterprise SSO, bot-blocking (Radar), and third-party integrations (Pipes) to companies like Cursor, Anthropic, and OpenAI. - **Studio** (Software): WorkOS's internal natural-language workspace; lets any employee query Snowflake, Linear, and Notion and build reusable widgets. - **Snowflake** (Software): Cloud data warehouse used as WorkOS's primary internal analytics database. - **Linear** (Software): Issue-tracking tool integrated as a Studio data source. - **Notion** (Software): Knowledge-management tool integrated as a Studio data source. - **LangGraph** (Software): Agent orchestration framework used to drive Studio's LLM-tool interaction loop. - **Claude Opus** (Software): Anthropic LLM used inside Studio; chosen for quality at query-writing and reasoning tasks. - **Radar** (Software): WorkOS's bot-blocking and fraud-detection product; the Radar support widget is the showcase use case. - **Pipes** (Software): WorkOS's third-party integration product; being extended to power org-level connectors inside Studio. - **Convex** (Software): Used as Studio's session-state store to preserve widget and conversation history across sessions. - **Widget** (Concept): Studio's core output artifact — declarative JavaScript that calls data-source APIs directly, runs deterministically without LLM involvement on each refresh. - **Preflight sequencing** (Concept): Studio's practice of running tool-connectivity and context-adequacy checks before answering a query, then injecting schema context lazily at tool-invocation time. - **Layering** (Concept): Studio's prompt architecture stacking base defaults, org-level rules, and session-specific context, with an explicit directive to distrust stale model knowledge about WorkOS.

Two engineers from Anthropic's Applied AI team — Ash Prabaker and Andrew Wilson — walk through what it actually takes to keep a coding agent productive for five-plus hours: a year of model and harness co-evolution that took runs from 20 minutes to 12+ hours, and the internal harness recipe behind their one-shot app demos — a planner that writes deliberately vague specs, a generator and an adversarial evaluator that negotiate "done" into testable contracts, taste rubrics that make design gradable, and a debugging loop that is mostly reading traces by hand. A 35-minute audience Q&A covers Ralph loops, agent teams, traceability, and human-in-the-loop trade-offs. ## [00:00] Introduction and speakers Ash Prabaker opens with introductions: he and Andrew Wilson are engineers on Anthropic's Applied AI team, and the session grew out of a blog post the team published a couple of weeks earlier on agents that keep working for extended stretches. Companies love showing one-shotted-a-browser demos, he notes, but rarely share what's inside the harness — that gap is the agenda. Andrew takes history and shipped primitives; Ash returns for the experimental half. > *We're talking 5 6 hour plus kind of runs.* ## [01:21] Overview of long-running agents Andrew, a solution architect based in London, frames the year with a quote from Boris, Claude Code's creator, on the tool's first anniversary: a year ago Claude struggled with bash commands and string escaping; now nearly all of Claude Code is written by Claude Code, with runs lasting days. > *it could run for, you know, maybe 20 minutes at a time.* ## [02:29] Challenges: Context, Planning, and Judgment Three buckets explain why long runs are hard. Context: windows are finite, new sessions start with amnesia, coherence rots as the window fills, and models near the limit exhibit "context anxiety" — rushing to finish. Planning: models try to one-shot everything, build half a feature and stop, or run out of context mid-app. Judgment, the least intuitive: models are poor critics of their own output, declaring a half-baked feature done or shipping a button with no backend behind it. > *models are really bad at judging their own output* ## [04:14] Two approaches: Model updates vs. Harness evolution Fixes come from two directions. Bake ability into the weights — the METER chart (how long an agent completes 50% of tasks on a minimal scaffold) went from about 1 hour on Opus 3.7 to 12 hours on Opus 4.6 a year later. Or change the harness: the Agent SDK ships the core primitives — the agent loop, MCP tools, sub-agent delegation, claude.md, skills, slash commands, the permission system. Andrew's running observation: every model release shipped harness changes alongside it. > *when we've released a model we've always also released a lot of harness changes alongside the models* ## [05:58] Prehistory: Sonnet 3.5, Computer Use, and MCP Before Claude Code existed there were artifacts on Claude.ai, and Sonnet 3.5 — the first model that showed real coding promise because it could look at what it had built and iterate. Computer use added clicking, screenshots, and self-testing; the MCP spec gave it tools. > *That was quite an aha moment sort of pre-Claude code.* ## [06:34] The evolution of Claude Code February 2025: Sonnet 3.7 lands state-of-the-art on SWE-bench and Claude Code ships as a research preview — explicitly to learn how developers use Claude for coding and feed that back into the model. That sets the recurring trend: as models improve, harness pieces become unnecessary or evolve. By May, Opus 4 and Sonnet 4 manage their own context better and reach task completion without reward hacking; Claude Code goes GA with an SDK. > *the goal of Claude code was to better understand how developers use Claude for coding to inform future model improvements* ## [07:55] The Ralph loop technique An interlude on the Ralph Wiggum technique — Jeffrey Huntley published it last July, traction arrived around December. The simple version: feed a prompt into the CLI on a loop until the tasks are done. The real version has phases — plan the prompt into features, pick one task, start a fresh session with a clean context window. Its appeal is captured in Huntley's "deterministically bad in an undeterministic world." Anthropic's own plugin runs inside a single session instead, relying on compaction, max iterations, a safe word, and a stop hook. > *it's better to fail predictably than it is to succeed unpredictably* ## [09:49] Sonnet 4.5, Agent SDK, and checkpoints Sonnet 4.5 starts tracking its own token consumption — context-aware enough to manage the end of its window instead of panicking. Claude Code 2.0 introduces checkpoints for rewinding a session. The Claude Code SDK is renamed the Agent SDK because the team realized the harness generalizes beyond coding. Runs reach roughly 30 hours. > *we realized it's much more general purpose than actually just for coding* ## [10:49] Opus 4.5 and the role of sub-agents Haiku 4.5 and Opus 4.5 complete the family, and the economics shift: many sub-agents become affordable, and Opus 4.5 plans well — so Opus plans while Sonnet executes. Skills arrive with progressive disclosure (only frontmatter loads up front), and programmatic tool calling lets the model write code to chain tool calls and return just the final result instead of dumping everything into context. > *all of a sudden running many sub-agents became really economical* ## [12:05] First long-running agent patterns Around November the team published its first long-running-agents blog post. A human writes something vague — "create a Slack clone" — and an initializer agent breaks it into persistent artifacts: a feature list stored as featurelist.json (models overwrite markdown more readily than JSON), a progress file, a git repo, an init script. The harness loop then runs in fresh context windows: get bearings, run the init script as a smoke test, pick exactly one unfinished feature, implement, verify with Puppeteer, commit, repeat. > *the models might overwrite markdown files, whereas they're they're less likely to just overwrite JSON files* ## [14:20] Opus 4.6, Agent Teams, and server-side compaction Sonnet 4.6 offers near-Opus intelligence at Sonnet pricing and becomes the workhorse; Opus 4.6 is "very much an agentic model" — the METER figure jumps from ~4 to 12 hours on a minimal scaffold. Agent teams ship: sub-agents coordinate directly with each other and report to the main agent only when needed. Server-side compaction means sessions can effectively run indefinitely, and 1M context goes GA — nudging the design question toward fewer fresh sessions and one big window. Andrew's closing point: the harness doesn't vanish as models improve; gaps get filled by the harness, the model trains on that, and pieces get deleted. > *the harness doesn't just disappear as the models get better* ## [17:28] State-of-the-art harness patterns Ash polls the room — only two or three people have agents running in the background right now — then lays out the core pattern, borrowed shamelessly from GANs: a generator builds, a standalone evaluator grades, with adversarial pressure between separate context windows, system prompts, and jobs. The evaluator doesn't read diffs; it opens live pages with Playwright, clicks around, and hands critique back. Why doesn't an LLM evaluator just rubber-stamp LLM output? The gap they exploit: tuning a standalone critic to be harsh is tractable; tuning a builder to be self-critical is not — same as humans, where critiquing a meal is easy and cooking it is hard. > *The evaluator here isn't just reading diffs, but it's actually using playwright, um, to open live pages, click around, try things out* ## [21:30] Evaluating subjective output with rubrics Most people say you can't grade taste; the team disagrees — if you hold a strong enough opinion, write it down. Their rubric scores design, originality, craft, and functionality, weighted toward the first two since Opus 4.6 already handles functionality — the real fight is purple gradients and AI-slop aesthetics. Few-shot examples on reference sites calibrate the evaluator's taste to their own. The distinctive behavior this unlocks: when the generator keeps scoring low on originality, the GAN-style harness throws everything out and restarts — where a single loop would keep patching the same thing. > *most people say you can't grade taste, but, you know, we think you can if you have a a strong enough opinion on it and you just kind of write it down* ## [23:44] Introducing the 'Planner' role To go from nice pages to working apps they added one more role. The planner turns a one-line prompt into a deliberately high-level spec — a series of sprints — and explicitly does not plan granular technical details, because a wrong detail cascades through every sprint and magnifies over multi-hour horizons. Squint and it's a PM/IC/QA org chart. > *We just kind of gave each role its own kind of context window.* ## [25:04] The generator-evaluator contract The glue between generator and evaluator: before a single line is written, the two agents negotiate what "done" means. The generator proposes a feature and tests; the evaluator pushes back — scope too big, tests too weak, missed edge cases — via markdown files on disk until both agree. Grading then happens against that contract, not the planner's original spec. Ash calls this the key innovation the Ralph loop never had: nobody argues with the main loop. The proof is a "build a retro game maker" prompt run both ways. Solo loop: pretty screens, but in play mode the arrow keys and space bar do nothing. With the harness (~$200, 6 hours): the app names itself Retro Forge, builds a 54-color sprite editor, turns a vague "AI features" spec line into a working AI level assistant, and play mode has a live debug HUD, a running physics loop, and real collisions — the difference is entirely scaffolding. > *we have the two agents basically negotiate what done actually means* ## [31:28] Specificity in contracts and debugging traces What the evaluator actually catches is unglamorous: a FastAPI route-ordering bug that passes unit tests but breaks in prod, a Boolean logic bug on the delete key — found only because it uses the app. For the game maker, the agents settled on 27 contract criteria; vague criteria produce vague critiques the generator shrugs off. Ash is candid that out of the box, Claude is a bad QA agent — the same sycophancy that plagues LLM-as-judge had early evaluators filing "fix it later, might take 2 weeks" and moving on. There was no secret fix: the art was reading traces, finding where the model's judgment diverged from theirs, and tuning prompts — plus piping transcripts to files and having another agent grep them to close the loop. > *If you have vague criteria, you have vague critiques* ## [34:14] Adjusting harnesses as models evolve Is harness design dead? Ash's answer: learn each model's spiky behaviors and fill the gaps. Moving from Opus 4.5 to 4.6 they dropped context resetting entirely (4.6 has no context anxiety; one continuous session plus compaction suffices), dropped forced sprint decomposition (4.6 holds a 2-hour continuous build coherently), and moved the evaluator from every sprint to the end of each one-shot generation. The harness wasn't wrong — it was right for 4.5, and the frontier moved. Today's setup keeps the planner-generator-evaluator core, shares state through the file system, and runs at roughly half the previous cost — demonstrated by a DAW the harness built whose music was, by Ash's admission, trash, but whose app was thoroughly fleshed out. > *it was right for 4.5, the frontier moved* ## [37:56] How to build your own agent harness None of this requires Anthropic's internal harness. Auto mode covers the safe middle ground; custom sub-agents already exist as a primitive — give your evaluator a harsh system prompt and a detailed rubric; Playwright MCP or Claude for Chrome handles web apps, computer use handles native; skills package grading rubrics into the dev flow. > *there's nothing stopping you from just going ahead and building something similar to this kind of on your own* ## [39:01] Key takeaways for long-running agents The photo slide: self-evaluation is a trap — use an adversarial evaluator. Compaction does not equal coherence — lossy summaries drift; structured handoffs and clean contexts work. Subjective quality is gradable if you force yourself to write the standard down. And sit with the model reading traces — only then do you know which scaffold pieces to delete when the frontier moves. > *self-evaluation, very much a trap* ## [40:05] Q&A session Eleven audience members take the mics for 35 minutes. Highlights: evaluator tuning generalizes across projects when you target common model weak points (calibrate with "this is AI slop" examples). On Ralph loops and the model's "smart zone": with 1M context GA and 4.6's coherence, the team moved to one continuous session with compaction — but use your own evals. On watching agents work: Ash sees wanting to watch as a trust gap; the model now reads console errors and spots overlapping text itself. The 4.6 generation is strikingly willing to throw ten passes away and restart when it can't hill-climb the rubric — one evaluator got fed up and told the generator to delete everything. The planner stays out of the inner loop deliberately; the spec is re-inserted as a reference instead. For products that outlive the run, the harness leaves breadcrumbs — a learnings JSON ("tried this, found this bug, fix worked") plus high-level docs — enough for a human with Claude Code to pick up. Feeding the generator's context to the critic was tried and rejected: judging output alone beats muddying the two streams. Traceability remains mostly reading traces by hand ("you got to read the whole thing"), with Claude-over-traces as a first pass. And on human-in-the-loop sprint reviews: hooks can inject one, but the team optimizes for full autonomy — run ten generations, read the seven failures, tune the harness prompts, repeat. > *you got to read the whole thing* ## Entities - **Ash Prabaker** (Person): Engineer, Anthropic Applied AI team; presents the state-of-the-art harness patterns and Q&A. - **Andrew Wilson** (Person): Solution architect, Anthropic Applied AI (London); presents the model/harness history. - **Anthropic** (Organization): The speakers' employer; ships Claude models, Claude Code, and the Agent SDK. - **Claude Code** (Software): Anthropic's coding agent CLI whose one-year evolution frames the talk. - **Agent SDK** (Software): Renamed Claude Code SDK; ships the agent-loop primitives the harness builds on. - **Generator-evaluator pattern** (Concept): GAN-inspired split of builder and adversarial critic with separate contexts; core of the harness. - **Ralph loop** (Concept): Jeffrey Huntley's loop-a-prompt-until-done technique; precursor lacking an arguing counterparty. - **Playwright MCP** (Software): Browser-automation tooling the evaluator uses to test live apps.