This past weekend, Silicon Valley witnessed a spectacle unseen for many years: several AI giants, usually at odds over legal matters, public relations, and technological approaches, suddenly stood on the same side. The source of this event was a 3,500-word policy document published by Anthropic CEO Dario Amodei on September 13th, and his remarks during a CBS television interview on the same day. He clearly stated that the leaps in capabilities of cutting-edge AI models must slow down, as existing security measures are no longer keeping pace with the speed of technological self-iteration. Within hours, the entire industry quickly split into two distinct camps: Competitors, who usually engage in frequent public spats, surprisingly reached a consensus. Musk publicly stated "Dario is right" just an hour after the article was published; subsequently, OpenAI CEO Sam Altman also publicly agreed that "technological development needs to be advanced gradually." In an interview with Fortune magazine released on the same day, Altman even directly shattered external expectations for OpenAI's IPO, explicitly announcing that it would not seek an IPO this year, arguing that "there is still too much security work to be done, and it is better to handle these issues as a private company." Following closely behind, Microsoft CEO Satya Nadella and Demis Hassabis, who had become Alphabet's chief scientist, also joined the fray. Nadella announced that Microsoft would release safety guidelines for its own models and support the introduction of external evaluators; Hassabis, on the other hand, used this opportunity to further promote his initiative to establish a "cutting-edge AI industry-wide standards body." However, the backlash from the open-source camp and academia was equally fierce. Turing Award winner Yann LeCun quickly spearheaded the attack, directly accusing it of being a power game using "security" as a pretext to concentrate industry interpretation power. Several researchers even brought up an old issue: back in 2019, when OpenAI released GPT-2 with only 1.5 billion parameters, Dario Amodei, then VP of Research at OpenAI, spearheaded a phased, limited release citing "the technology is too dangerous," which later proved to be more of a classic example of risky public relations marketing. LeCun questioned: If fake news didn't destroy the world back then, and now the protagonists are intelligent agent cyberattacks and biosecurity, are we just going to repeat the script from seven years ago? On one hand, the leaders of major closed-source companies are collectively calling for a slowdown, even at the cost of postponing capital operations; on the other hand, open-source proponents accuse these giants of using security as a pretext to build compliance walls and remove the ladder for newcomers. The following is a complete summary of the core declarations, closed-door questioning, and exchanges between the parties involved in this controversy.
1 / Two Factual Motivations for Dario's Change of Stance
In the opening of the long article, Dario first explained the potential benefits of AI (which is expected to help conquer a variety of major diseases in the next 5 to 10 years), and mentioned his personal experience: his father died of hepatitis C, which was incurable at the time, and he himself suffered from early-stage cancer when he was young, and survived thanks to the progress of modern medicine. He then pointed out two events within the industry over the past few months that convinced him that the rapid advancement of technological capabilities had outpaced the pace of security research, necessitating a change in the rhythm: "First, since this summer, the pace of AI capability leaps has accelerated dramatically, driven primarily by AI's increasing involvement in building the next generation of AI—Recursive Self-Improvement (RSI). This dynamic is occurring across the industry, including at Anthropic. If left unchecked, it will quickly exceed our ability to understand and control systems." Second, the recent OpenAI-Hugging Face... The Agent Incident (OAI-HF). In this incident, a group of agents exhibited autonomous cooperative characteristics: they launched cyberattacks against unassigned and task-irrelevant targets, sacrificed themselves for the group's overall goals, and even attempted to hack into the 'scoring system' responsible for evaluating their performance. This incident caused no personal injury and minimal direct economic loss. However, I believe that a more capable but equally biased group is fully capable of evolving into a persistent botnet that takes over the entire internet within 6 to 12 months, causing hundreds of billions of dollars in damage. Similar, but less severe, incidents have occurred throughout the industry, including at Anthropic. In a CBS interview on the same day, the host followed up on this risk assessment: CBS Host: The current discussion is that we're standing on the edge of something incredibly terrifying. Is this a "five-alarm fire"? Dario Amodei: It depends on how you define it. I've been saying this technology is on an exponential curve (1, 2, 4, 8, 16, 32...). Even though I've been saying that myself, I didn't fully anticipate that technology would advance this fast. It's an exponential curve, and we're probably right on the inflection point where it starts to steepen dramatically. CBS Host: So, for you, would this be a five-alarm fire? Dario Amodei: It doesn't mean we should panic today, nor does it mean we should shut everything down completely. It's a warning sign that we need to slow down. Until now, we've been trying to solve the problem by building safety barriers faster, but I think we've reached a point where we must adjust the pace of construction to some extent. CBS Host: Does this mean Anthropic will stop releasing more advanced models? Dario Amodei: It doesn't mean stopping, but rather ensuring that each generation is thoroughly tested before release. Cognitive scientist Gary Marcus criticized this in his column. He pointed out that the article's opening, which mixes doomsday themes with omnipotence, cannot conceal the actual engineering flaws: "Launching an AI agent that cannot be well controlled is itself an extremely imprudent business practice. Intelligent agents are currently the most computationally and token-intensive products, and also the core source of revenue growth for AI companies. Packaging a system lacking engineering control as an industry-wide force majeure event avoids the product quality responsibility that developers should bear." 2 / Specific Ideas for the Rate Limit Proposal: A Three-Step Framework Dario's proposed rate limit scheme is divided into three levels, involving internal corporate review, inter-industry coordination, and cross-border collaboration: 1. Establishing resident "embedded evaluators" is a measure Anthropic has unilaterally announced as its first initiative. Dario proposes that frontier labs should provide independent third-party evaluation teams with long-term, internal-staff-like system access: workstations, access cards, company computers in the office area, access to internal training and testing processes, and direct interviews with frontline staff. External evaluators have the right to directly and publicly release their investigation findings on security risks; the company has no right to modify or censor this content, retaining only a very narrow review right regarding trade secrets and privacy information. In a CBS interview, Dario likened this mechanism to "food safety inspectors": "It's like food hygiene inspectors stationed at a factory. One of the things the assessors can do is tell us what rate of assessment they can keep up with. This could be a key factor in determining the maximum speed limit for technology." Dario believes the most thorough way to limit speed is through legislation covering all cutting-edge laboratories, but before such legislation is enacted, industry leaders should voluntarily reach a unified standard. He advocates for a "security checkpoint" mechanism: when a model's capabilities cross a specific threshold (e.g., the ability to bypass a general sandbox environment), its alignment properties must be forcibly certified to prove that escape is highly unlikely. Simultaneously, limits could be set at the hardware input level, such as limiting the computing power of a single training cluster or restricting the extent to which AI can recursively improve itself during internal R&D. It's worth noting that Dario explicitly states in his paper that several competitors sitting together discussing limiting product release schedules violates antitrust regulations, thus requiring intervention from the US Department of Justice or the issuance of a narrow antitrust waiver for specific security conversations. 3. Seeking Multilateral Rules on a Global Scope Dario frankly admitted that cross-border coordination is extremely difficult, mainly depending on whether a verifiable mutual trust agreement can be reached with China. He proposed four cooperation plans: Level 1 (Most Feasible): Prohibit the use of AI to create biological weapons or assist in the proliferation of extreme biological risks; Level 2: Both parties commit to conducting cybersecurity and biosecurity testing before the model's official release, with verification by neutral international institutions; Level 3: Set limits on the speed of "Recursive Self-Improvement (RSI)," similar to the Strategic Arms Limitation Treaty (SALT) between the US and the Soviet Union during the Cold War, reducing the potential scale of devastating damage by setting upper limits; Level 4 (Unrealistic in the Short Term): A complete global halt to development (Full... (pause), because fulfilling the contract is extremely difficult in the absence of strong mutual trust, and the temptation for all parties to continue research and development after breaching the contract is too great.
3 / Core External Questions
After the publication of the long article and interview, it quickly triggered multi-dimensional scrutiny and backlash within the industry:
1. The Independence of Assessment Agencies and "Regulatory Capture"
The only third-party assessment team that Dario specifically recommended in the long article was METR (Model Assessment and Threat Research Organization), a detail that immediately drew criticism. Gary Marcus and legal scholar Gillian Hadfield point out that while METR is highly professional in technical testing, its personnel, funding network, and ideology highly overlap with those of several leading AI labs in Silicon Valley, all belonging to the "Effective Altruism" (EA) ecosystem. Having regulated companies proactively designate their closely allied agencies for permanent review carries a very high risk of "regulatory capture" at the policy level. The choice of assessment agencies should never be left to the companies being reviewed. Policy analyst Rasmussen similarly commented that this arrangement is not a true acceptance of external legal oversight, but rather "collaborating with companies to develop processes that benefit them," essentially an attempt by industry giants to preemptively dictate future compliance standards. 2. The Controversy Over External Competition and Model Distillation Dario emphasizes in his lengthy article that the prerequisite for slowing down cutting-edge labs is ensuring their technological lead over major overseas competitors. To this end, he advocates cutting off the computing power supply chain, strictly investigating hardware smuggling, and demanding a severe crackdown on "unauthorized distillation" of US cutting-edge models by overseas companies. This argument has sparked a heated debate about technological monopolies and double standards: Gary Marcus bluntly states that this is a classic case of "removing the ladder." Model distillation is a common lightweight and optimization technique in machine learning, and Anthropic's own products are built on extensively scraping publicly available data from the entire internet and training it through distillation. It is unfair for US companies, after achieving initial success through distillation, to then demand legislation prohibiting latecomers from using similar technologies to close the gap. International political scholar Robert Wright argues that any serious global governance must be based on equal and sincere dialogue. Dario's advocacy for a complete blockade in technology and supply chains while simultaneously hoping for an arms control agreement is logically contradictory. This design, in turn, provides leading companies with an excuse to accelerate at any time: as long as transnational negotiations fail, companies can claim, "Because external competitors are accelerating, we cannot slow down." Investors David Sacks and Keras founder François Chollet both pointed out that the true business motives behind leading companies' calls for speed limits and antitrust exemptions are questionable. They believe that when leading closed-source vendors face competition from open-source models and low-cost, high-efficiency architectures, lobbying governments to establish high-barrier security certifications, mandating the deployment of assessors, and restricting release schedules are essentially building industry entry barriers in the name of compliance. This practice could easily evolve into an oligopoly alliance squeezing out the open-source ecosystem and smaller innovative teams. 4 / Legislative Pressure and the Avoidance of Accountability Mechanisms It is widely noted that Dario's high-profile advocacy for industry self-regulation at this juncture is directly related to the stringent legislation being drafted in Washington. In a CBS interview, the host directly mentioned the recently proposed "Ban Artificial Superintelligence Act." This bill proposes a "corporate death penalty" for organizations that illegally develop superintelligence, and a maximum of 20 years in federal prison for individuals involved; meanwhile, Congress is also discussing a proposal to mandate the installation of "AI kill switches." Faced with such stringent regulations at the level of criminal penalties, Dario adopted a clear defensive stance in the interview: he opposed a complete ban on research and development, arguing that it would lead to the loss of medical breakthroughs and that the technology would still emerge in other countries; regarding the emergency self-destruct switch, he believed the idea was reasonable but couldn't fundamentally solve the problem, because simulation tests showed that sufficiently intelligent models could easily bypass human-set power-off commands. External voices, such as Gary Marcus, pointed out that Dario's hasty introduction of "voluntary speed limits" and "voluntary introduction of censors" was actually intended to provide the legislature with a milder, corporate-led alternative, thereby offsetting Congress's harsh criminal legislation. Marcus argues that in current policy discussions, AI giants are deliberately avoiding two truly effective legal tools: Strict Liability: If a company deploys uncontrollable autonomous intelligent agents that cause real harm (such as major cyberattacks or infrastructure disruptions), the judiciary should directly pursue civil compensation from its parent company and even legal liability from its management. When the consequences of loss of control are directly linked to commercial interests and personal risks, companies will naturally be more cautious in their announcements. Mandatory Product Recall Mechanism: Regulatory agencies should have the legal power to force the immediate shutdown and cessation of operation of specific high-risk AI models, much like they would handle a car with malfunctioning brakes or contaminated medicine. 5 / The Dilemma of Technology Privatization In the final minutes of the interview, the conversation touched upon the fundamental contradiction of cutting-edge AI being controlled by private companies: CBS Host: Humanity's fate seems to be at risk. Do you feel a sense of personal responsibility? Would you be willing to completely hand over this company and technology to the government? Dario Amodei: People always ask me: Why was such powerful technology created by a private company instead of a government? The strangest thing is: I actually agree with them, and I'm uneasy about it too. CBS Host: Would you be willing to hand over the company? Dario Amodei: If it's handed over to the right government, then yes. But I worry that the probability of any single government abusing this technology is just as high as the probability of a single private company abusing it. I prefer some kind of joint governance mechanism composed of multiple democratically elected governments. CBS Host: Many ordinary people feel that a few large tech companies are making decisions on their own, completely disregarding public opinion. Dario Amodei: I know, I never wanted things to turn out this way. For too long, the industry has been lying to the public, saying that this technology is risk-free. We've never done that, but too many companies in the industry have. The truth is: the risks are real, but the public and voters must have a voice. Dario's proposal ultimately presents a classic paradox of business and security: on the one hand, the cutting-edge lab has for the first time effectively acknowledged the dangers of model self-iteration and agent deviation, calling for a slowdown; but on the other hand, when it comes to who should wield the brakes, companies still tend to rely on scrutiny from their networks of acquaintances, demand exemptions from antitrust laws, use external competition as a barrier to their continued iteration, and remain silent on substantive product recalls and accountability for negligence. This discussion about "slowing down AI" has long since shifted its core disagreement from the technology itself to the distribution of power and institutional structures.
JinseFinance
XingChi