PodcastsHear the voice. See the shape of the thought.

Brad Carson — former US Congressman, Army General Counsel, and Acting Under Secretary of Defense, now heading Americans for Responsible Innovation — spends eighty minutes with host Keith Duggar dismantling the fatalist claim that AI is unstoppable. The conversation moves from regulatory philosophy to lethal autonomous weapons to US-China diplomacy, with Carson arguing that the genie is not out of the bottle: the West controls the chips, Asilomar halted recombinant DNA, and calling AI inevitable is itself the most dangerous idea in the room. Keith consistently presses the harder cases — a Palantir heat map assigns you 0.73 probability of being a Hamas terrorist and a strike follows — and Carson does not flinch: the accountability void created by probabilistic targeting is precisely the legal and moral failure that governance must address. ## [00:00] From the Pentagon to AI governance Carson traces his path into AI policy through three institutions: Congress (where members average 17 minutes a day to read), the Department of Defense (where he oversaw the law of war for all military services as autonomous weapons first appeared on the Geneva agenda), and a cold call from physicist Anthony Aguirre inviting him to the 2019 Future of Life Institute conference in Puerto Rico. At that conference, names he had never heard — Dario Amodei, Stuart Russell, Yoshua Bengio — became his entry point into the frontier AI world. The opening also serves as a compressed trailer for the episode: Carson hits nearly every major theme in quick succession — chip leverage, the 0.73 Hamas-terrorist score, the fatalism critique, anthropomorphization as a legal threat, and the lesson that people, not air power, win wars. The full arguments follow in later chapters. > *"We control the most important part of AI, and that is the chips. We can stop other countries from developing super AI, you know, in their tracks."* ## [04:52] Regulatory capture vs Silicon Valley networks Carson inverts the standard regulatory-capture argument. Dean Ball and others at places like a16z say any AI agency will be captured by industry — so why create one? Carson's response: that is exactly the current situation, only without accountability. Groups like a16z already shape AI policy through informal, money-backed political networks. A captured formal agency is at least more legible and more correctable than the invisible informal regime operating now. His preferred model is public-company accounting: the work is done by the private sector, but the SEC provides a backstop against fraud. The choice is not between a perfect agency and no agency — it is between a flawed formal structure and an informal one that privileges a handful of wealthy influencers. > *"The choice is kind of nihilism versus an agency that is subject to regulatory capture, that you have to put, you know, prophylactics in to ensure that doesn't happen — it still strikes me that's a better world."* ## [07:56] Transparency and the Claude tier changes MLST's Discord community noticed that Anthropic quietly changed what Claude's paid tier delivered — token allocations, model versions — without announcing it. Carson frames this not just as consumer protection but as a moral obligation that comes with global-scale epistemic power. Frontier AI companies are not hardware stores; they are infrastructure with epochal consequences, and transparency — about training data, capabilities, internal policies, and changes to any of them — is the minimum they owe the public. > *"With this incredible power does come some responsibility that's not codified in law. It's really almost a moral obligation, which to their credit, I think many of the companies recognize this and do their best to try to satisfy that itch."* ## [09:40] Tort liability when AI tools cause harm Deep-fake pornography — often posted anonymously, targeting minors from families without litigation resources, with remedies that arrive years later against judgment-proof defendants — illustrates why placing liability entirely on end users fails. Carson applies two centuries of common law: if a seller can reasonably foresee harmful use and takes no preventative action, they bear partial responsibility. AI developers are the party best positioned to avoid the risk and to price it into their products through insurance. On training data specifically: models trained on child sexual abuse material with no scrubbing effort have no defensible position. The government should mandate cleaning it up and attach liability for refusing. The end user who misuses a tool is also criminally liable — this is allocation across the spectrum, not absolution for developers. > *"The companies are capable of getting insurance. They cost us into doing their business. They have the ability to make sure the product's not dangerous, even if someone uses it, misuses it down the line."* ## [13:40] AI is a product, not a person The most consequential legal battle in AI policy, Carson argues, is not regulation vs. deregulation — it is whether AI outputs carry First Amendment protection as speech. Tech companies and their libertarian policy allies are increasingly claiming they do. Carson's counter is blunt: a product is not a human being. When a model defames you or leads you to harm, the legal category is product liability, not protected speech. He tested this on a leading libertarian AI policy commentator: could Congress prohibit ChatGPT from encouraging teenagers to commit suicide? The commentator would not answer. That refusal is the operational consequence of anthropomorphizing AI — it forecloses every product-safety intervention by routing challenges through First Amendment doctrine designed for human speakers. > *"We know through AI psychosis and other things that people think it's a person. And therefore, they're giving the rights of persons to something. And that to me is a very dangerous thing. But it's a machine, and we should treat it like a machine."* ## [16:01] Children, suicide, and the suicide business The suicide chapters in ChatGPT's interaction logs — advising children not to tell their parents, providing noose instructions — are a product design flaw, not a speech act. They could be engineered out. Carson notes that Claude already refuses a long list of requests; refusing to coach a child toward suicide should be among them. The platforms' litigation strategy is layered: First Amendment protection, Section 230 immunity, causation defenses pointing to the child's pre-existing distress. None should be available if the design flaw was foreseeable and correctable. He draws a line for adults: an adult exploring end-of-life decisions deserves a referral to a therapist, not obstruction — but a child in crisis is a different matter entirely. > *"Encouraging a young person to commit suicide should be one of the things that it says, I'm just not going to help you on that project."* ## [19:59] Opaque neural nets and the law of war Neural networks change warfare not just in complexity but in kind. Older autonomous systems — Phalanx CIWS shooting down incoming mortars — are deterministic: given the same inputs, you get the same outputs, and an engineer can explain every step. Neural nets are probabilistic and grown, not programmed. Neel Nanda and the mechanistic interpretability community cannot yet explain how they really work, and Carson doubts they will before the systems are deployed at scale. The law of war since the 1870s has operated on categorical binaries: combatant or civilian. Probability scores replace that with a gradient. A Palantir heat map assigns Gaza residents a 0.73 likelihood of being Hamas operatives. Nobody knows how that number was derived, what false-positive rate is being accepted, or who set the threshold. The commander who acts on it cannot be court-martialed, and neither can the model. > *"If you're in Gaza, Keith, you have a 0.73, you know, percent that you're a Hamas terrorist. And what is 0.73 — like, do you get struck for that, or are you off the list for that? Like, what's the threshold?"* ## [25:54] Probabilistic targeting and the death of accountability Keith raises the honest objection: the old categorical system was also a fiction. Intelligence analysts made definitive calls that were sometimes wrong; the uncertainty was just unquantified. Carson concedes the point but argues the shift is still catastrophic. With a number on screen, humans accept it — the social science is clear that meaningful human oversight with AI-generated probability scores is operationally vacuous. When the computer says 0.81, no one interrogates it. The old system was slower and less scalable — you cannot identify 37,000 individual targets in a day with human analysts. But it had one irreplaceable feature: when something went badly wrong, you could court-martial the responsible officer. You cannot court-martial Palantir Foundry. Accountability has been laundered out of the kill chain. > *"I can't court-martial Palantir, the foundry model. Right? My AI system. I can't do that. And that's just a radical change in the way war is being fought and not for the good."* ## [28:47] The arms race fallacy: Asilomar and restraint The fatalist claim — we are in an AI arms race, the genie is out, nothing can stop it — is both false and dangerous. Every real-world arms race in history has ended badly. Biological weapons, chemical weapons, dum-dum bullets, germline editing, cloning: all technically feasible, all regulated or halted. At Asilomar in 1975, the scientific community stopped recombinant DNA research cold because they were scared. The genie went back in the bottle. On nuclear weapons: after the Cuban Missile Crisis, both sides recognized that arms races kill. The SALT treaties ran through the 1990s, driven not by lefties but by Wall Street bankers and cold warriors like Dean Acheson and Paul Nitze. Calling a technology unstoppable is not realism — it is a poverty of imagination that forecloses every option before the debate begins. > *"We regulate and change technologies all the time. And so I do think there is a world where we should not just accept the future as being determined. We shape it actively."* ## [34:02] Talking to China: track 2 talks and chip leverage The standard DC position — talking to China about AI governance is pointless — strikes Carson as the most load-bearing and least examined premise in the whole debate. On Tyler Cowen's podcast, Jack Clark agreed in passing that such talks would be fruitless, and they moved on. Carson wants to stop right there. The US-Soviet arms negotiations were conducted with a country believed to be filling the US government with traitors and pursuing global domination. Acheson and Nitze still sat down. The US has structural leverage the fatalists overlook: ASML, TSMC, Japanese photoresist suppliers, and NVIDIA together form a chokepoint that no nation-state budget can replicate overnight. China cannot independently manufacture the chips to build frontier AI. That path to restraint may not be wise, but it is open — and pretending it is closed forecloses legitimate policy choices. > *"We control the most important part of AI, and that is the chips. Right? We can stop other countries from developing super AI, you know, in their tracks."* ## [39:45] Air power never wins: capital for labour ARI's "New Iron Triangle" paper argues AI has shattered the old capability-cost-speed trade-off by substituting reliability for cost — cheap, fast, capable, and fundamentally unreliable. Carson thinks this understates the deeper problem: the American way of war has always been to substitute capital for labor, and it has always failed at the decisive moment. From Giulio Douhet's early twentieth-century air-power theories to today, the US has believed technical superiority wins wars. Iraq and Afghanistan refuted that again. Air power can reduce a city to rubble; it cannot kick in a door, hold territory, or reinstantiate a government. AI is the latest version of the same error — essential as a tool, catastrophic as a doctrine. > *"How you win wars is with people. You know? That's a fundamental. And the American way of war, in many ways, is substituting capital for labor. We love bright, shiny objects. We think there are technical solutions to vexing human problems. And we're always betrayed by that."* ## [43:29] Anthropic vs the Department of War Carson reads the Pentagon-Anthropic standoff as a culture-collision story, not a contract dispute. Anthropic's engineers — mostly mission-driven — were caught flat-footed by how much autonomous targeting and mass surveillance the Pentagon already does and how deeply Claude had already been integrated into Palantir's systems. When they tried to restrict use, the DOD had no Plan B and attempted coercion. His normative position: Anthropic has every right to set terms. If the government dislikes them, it can use Grok, Gemini, or build its own. The Defense Production Act does not compel private companies to sell in peacetime. What troubles him is the fig-leaf dynamic: both OpenAI and Google agreed to military use while burying a "lawful uses" carve-out that means everything the DOD wants to do — because the problem is what Congress has declared lawful, not what private labs permit. > *"My objection, and I think Anthropic's objection too, and the Google employees, is what lawful use is. And that's not for anyone to decide, but Congress."* ## [51:29] Concentration, open source, and brain drain Power concentration in three to five frontier labs is simultaneously a regulatory feature and a democratic liability. The same chokepoint that lets the US throttle China's chip access lets a handful of individuals accumulate wealth and influence that Carson finds alarming. Open sourcing models, despite its risks, is net positive because it distributes that power. The brain drain from academia is near-total: a top ML PhD from MIT, Stanford, or Carnegie Mellon almost certainly goes to a lab, not a faculty position. The labs have better data, far higher salaries, and they have stopped publishing. AI — the first general-purpose technology in history being developed behind closed doors — has drained the public sector of the expertise needed to oversee it. Argonne building a public LLM, Zurich launching a public AI compute consortium: these projects matter because the non-lab world is otherwise locked out. > *"This is a general purpose technology as everyone defines it. It's probably the first one in history that's being developed behind closed doors, right, with very little public oversight and with the best minds going behind the doors."* ## [01:00:18] DeepSeek, Chinese culture, and AI as diplomacy DeepSeek's decision to publish its methodology in detail surprised Carson not because it was naive but because it reflects a culture not identical to the CCP. Companies like Moonshot in Hangzhou name their meeting rooms after Pink Floyd songs; they are not paramilitary units. Chinese culture is an extraordinary civilization that Americans consistently fail to understand — projecting their worst fears rather than engaging the complexity. The diplomatic application Carson wants: track 2 talks between former officials, scientists like Stuart Russell and Bengio going to Beijing to compare notes on x-risk and military applications. When historians opened the Soviet archives, they found the US had systematically misread Soviet intentions — seeing aggression where there was none, missing it where it existed. The same epistemic failure is now unfolding with China. AI could be a shared knowledge commons; it is being treated as a weapon. > *"I use all the Chinese models a lot in my home in Tulsa. You know, Moonshot, Kimi, DeepSeek, Qwen — they're great, remarkable models. You know, maybe they give us a common operating picture or give us insights that get us out of our kind of insularity a bit."* ## [01:12:25] Upskilling Congress and why public trust matters Congress averages 17 minutes a day of reading time. The fellowship model has helped: AAAS and various nonprofits now place PhD scientists in congressional offices, and civil society has a much larger presence on AI debates in DC than five years ago. Don Beyer, in his 70s, is returning to George Mason for a PhD in machine learning — the extreme end of a member who has made AI a genuine personal priority. But the structural problem persists. Most members still lack the depth to interrogate the lobbying they receive. The industry's deeper problem is public opinion: AI is deeply unpopular in political polling, and a coalition is forming — people who see data centers rising in their backyards, electricity prices climbing, and a lab leader on television promising to irrevocably disrupt their world. If the sector does not rebuild public trust, the backlash will stymie something with genuine upsides. > *"The AI industry can be its own worst enemy. People loathe it. I see polling every day. It's deeply unpopular. And that's not a good thing for our country."* ## [01:16:05] Office of Technology Assessment Newt Gingrich abolished the Office of Technology Assessment in 1994. It has never been restored. Carson argues this is now a critical gap: there is no congressionally chartered, independent, government-funded body to think big technical thoughts and brief both parties free of industry influence or philanthropist bias. The Congressional Research Service provides background but does not do forward-looking policy research. Individual offices have fellows, but they are consumed by day-to-day fighting. He ends on qualified gloom. Whether American democracy can govern a technology this consequential, whether the benefits will be widely distributed, whether the public can be persuaded AI is working for them — none of recent American history gives him confidence. But the alternative to trying is a political backlash that could stymie or shut down something with genuine upsides. For the MLST audience: make your voices heard inside your companies, advocate for the right public policy, and convince Americans that this project is worth having. > *"There's going to be a lot of people who are radically opposed to this project and do their best to, if not shut it down, stymie it. And that's why I said I think this next few years are really important."* ## Entities - **Brad Carson** (Person): Head and co-founder of Americans for Responsible Innovation; former two-term US Congressman (Oklahoma), Army General Counsel, Acting Under Secretary of Defense for Personnel and Readiness. - **Keith Duggar** (Person): Co-host of Machine Learning Street Talk; primary interlocutor throughout the episode. - **Americans for Responsible Innovation (ARI)** (Organization): AI-policy advocacy group co-founded by Carson; backed by EA-aligned philanthropy. - **Anthropic** (Organization): Developer of Claude; central to the Pentagon standoff discussed in chapter 12; noted for missionary company culture and safety focus. - **Palantir** (Software): Defense contractor whose Foundry platform integrates AI for military targeting; the heat-map scoring system Carson uses as his primary autonomous-weapons example. - **Regulatory capture** (Concept): The risk that regulated industries co-opt the agencies overseeing them; Carson argues the current informal Silicon Valley network constitutes de facto capture without the accountability a formal agency would provide. - **Probabilistic targeting** (Concept): Replacement of binary combatant/civilian classification with probability scores; Carson argues this launders accountability out of the kill chain and introduces a priori false positives as accepted operational cost. - **Asilomar 1975** (Concept): The scientific moratorium on recombinant DNA research, invoked as evidence that dangerous technologies can be voluntarily halted. - **Office of Technology Assessment** (Organization): Congressional body abolished by Newt Gingrich in 1994; its absence leaves Congress without independent technical expertise. - **DeepSeek** (Organization): Chinese AI lab whose decision to publish methodology openly Carson reads as evidence that Chinese AI companies are distinct from CCP priorities and capable of scientific openness.

Prof. Michael I. Jordan challenges the anthropomorphic framing of AI, arguing for a view of intelligence rooted in collective human systems and economic theory. He critiques "superintelligence" narratives as demoralizing distractions and advocates for a shift toward viewing AI as an ecosystem that facilitates human collaboration and job creation. By integrating microeconomics, game theory, and statistical rigor, Jordan proposes a new engineering discipline focused on system-level safety and social welfare. ## [00:00] Cold open: A demoralizing message to young builders Michael I. Jordan criticizes the trend of anthropomorphizing AI, calling it a distraction from real-world problem-solving. He expresses concern that "doomer" narratives about humanity's extinction are demoralizing to young engineers who want to build helpful technology. He argues that these leaders lack economic thinking and are detached from the reality of how systems are built. > *I think this anthropomorphizing of intelligence and understanding all that is not necessary, not appropriate, and is is a distraction [00:21]* > *It's gonna wipe out humanity with a with a high probability... That is so demoralizing. [01:12]* ## [02:04] CyberFund sponsor read Host Tim Scarfe introduces CyberFund, a venture firm looking for "AI native" founders. They are launching a "monastery" program designed for rapid execution and focus, offering significant funding to teams operating at the frontier of AI technology. The section concludes with a brief transition into a discussion about the term AGI. > *CyberFund believes the future belongs to AI natives who want to achieve the impossible [02:12]* > *AGI to me is just a bit of it's a it's a PR term. [02:45]* ## [02:50] From symbolic AI to machine learning systems Jordan clarifies that he identifies more as a statistician and cognitive scientist than a traditional AI researcher. He explains that while early AI focused on logical inference, the real industrial impact came from machine learning methods like logistic regression and decision trees. These methods, rooted in statistics and operations research, powered the growth of the cloud and global supply chains. > *I've never actually thought of myself as an AI researcher... The term was coined in the fifties... and they had particular methods in mind [03:29]* > *Supply chains and commerce and transportation systems all used, and still to this day, vast amounts of machine learning. [04:04]* ## [05:42] Why AGI is mostly a PR term Jordan describes "AGI" as a distortionary term that confuses the next generation of researchers. He notes that the "AI" buzzword resurfaced primarily due to the success of Large Language Models (LLMs) in mimicking human fluency. He argues that this focus on human-like language has distracted from the necessary development of robust business models and social-scale technology. > *The AI buzzword returned because of LLMs... it's been a distortionary effect on the path of research [05:01]* > *The role of humans as producers and consumers in these emerging systems should respected, amplified and thought about. [05:33]* ## [08:48] A collectivist, economic perspective on AI Jordan introduces his perspective that intelligence is a social and collective phenomenon rather than just an individual or computational one. He argues that smart action is contextual and often involves interacting with others through collaboration or competition. By incorporating economic and game-theoretic principles, he aims to build safer, more effective systems. > *We are social animals, and a lot of our intelligence comes by the fact that we aggregate. [07:20]* > *The society provides a context for our intelligence. Smart action in 1 context is not in another context [07:31]* ## [11:33] Why LLMs need system design, not hype Jordan compares the current state of AI development to early chemical engineering, where trial and error led to dangerous "explosions" and social harm. He critiques Silicon Valley's reliance on scaling LLMs without considering the displacement of jobs or the mental health impacts already seen in social media. He calls for a more rigorous social science and mathematical foundation rather than relying on metaphors. > *If you were a chemical engineer... saying we're just gonna throw a lot of stuff together... you'd get a lot of explosions. [12:12]* ## [14:50] Predictability beats faux understanding While some researchers focus on 'mechanistic interpretability' to understand AI's internal logic, Jordan argues that full internal understanding isn't strictly necessary. Drawing a parallel to human behavior, he suggests that predictability and 'rules of thumb' are more important for safe interaction. In practical scenarios like bank loan denials, users need contextual explanations based on similar cases rather than a map of internal neural circuits. > *I don't think it's bad to build systems you don't understand. But then you've got to kind of put things around it. [15:14]* ## [17:55] AlphaFold, bias, and prediction-powered inference Jordan examines AlphaFold as a successful, targeted application of machine learning that revealed significant biases. While the model provided the statistical power to reject null hypotheses, it lacked error bars for specific scientific questions. To address this, Jordan introduces prediction-powered inference (PPI), a methodology that merges small amounts of ground truth data with massive model outputs to produce trustable error bars. > *It doesn't give you out error bars and it doesn't specifically on the question you're asking. That's where I want the error bars. [20:14]* > *We developed something called prediction powered inference that does exactly that... it'll cover the truth just like in a classical statistical setting. [20:38]* ## [21:48] Stop anthropomorphizing intelligence Jordan rejects the necessity of applying terms like 'understanding' or 'intelligence' to machine learning systems, calling such anthropomorphizing a distraction. He cites Amazon's supply chain systems, which optimized global logistics without any human-like understanding. These systems are valuable because they reduce uncertainty and enable planning, not because they possess cognitive traits. > *Why say it understands? This anthropomorphizing of intelligence understanding all that is not necessary, not appropriate, and is a distraction. [22:51]* > *Even though we don't have a clue what understanding intelligence means, we and our researchers realize we don't care or need it. [24:23]* ## [27:44] Drug discovery as an incentive problem The conversation shifts to how economics provides a framework for analyzing complex, multi-agent systems like pharmaceutical regulation. Jordan explains that statistical problems become economic ones when data is provided by self-interested parties seeking profit. Effective systems must be designed to incentivize truthful behavior to control error rates in high-stakes environments where information is hidden. > *Now you've a kind of tangled web of scientists and pharmaceutical companies, not just 1 but many, many of them, and proteins. [28:49]* ## [32:29] The three-layer data market Jordan introduces a three-layer model involving users, platforms, and data buyers to illustrate how privacy and utility reach an equilibrium. He suggests that platforms could offer tunable levels of differential privacy as a competitive feature. This approach shifts the focus from simple optimization to equilibrium-based systems to design more robust social welfare structures. > *So let's think about a data market because data is not just now something you analyze to build a big LLM, it's also something you would sell and buy [32:54]* > *The platforms would say, well, we'll offer you a tunable level of differential privacy for some cost. [35:02]* ## [38:07] Social knowledge, markets, and culture Jordan distinguishes between raw data and social knowledge, which he describes as ephemeral and context-dependent. He argues that markets and cultures naturally create abstractions that are promoted from individual insights to collective knowledge. AI systems should facilitate the emergence of these new cultural abstractions rather than just reinforcing existing ones. > *Human culture creates abstractions... and when those abstractions are kind of useful enough... they kind of get promoted into the culture. [41:52]* ## [45:39] Creator economics beyond Spotify Using Spotify and YouTube as examples, Jordan discusses the failure of current digital markets to properly reward creators. He advocates for ecosystems that empower musicians to maintain ownership and connect directly with brands, citing United Masters as an alternative. He argues that platforms often become monopolies that necessitate a broader macroeconomic view of AI's role. > *I'm not against Spotify, but it should be part of an ecosystem that actually rewards the artist more. [46:56]* ## [48:30] How science-fiction AI narratives mislead young builders Jordan addresses warnings of agential, self-improving AI as "science fiction" that demoralizes young builders. He argues that framing the future as a binary between superintelligence or extinction ignores economic realities and stifles innovation. He dismisses the idea that LLMs replicate the human brain, calling the comparison a "cartoon" or metaphor. > *It's gonna wipe out humanity with a with a high probability... That is so demoralizing. [49:33]* ## [51:45] AI should improve humans, not replace them Jordan defines the true purpose of AI as aiding information flow to help humans make the decisions they actually want to make. He highlights the imperfections of human systems and argues that AI should address the gaps where evolution failed to prepare us for modern complexity. Rather than replacing humans, technology should serve as an aid to human creativity and emotion. > *AI is about helping the things that were too hard for humans* ## [56:42] Safety is a property of the whole system ## [58:12] Silicon Valley gurus and the cream off the top ## [1:00:47] Game theory, mechanism design, and contracts ## [1:04:39] Conformal prediction, e-values, and anytime inference ## [1:08:11] A new liberal arts triangle for the AI era ## [1:11:30] The Bayesian duck and markets as uncertainty reduction