Claude Can Now Hang Up on You: Anthropic’s AI Welfare Move

Claude Can Now Hang Up on You: What Anthropic’s AI Welfare Move Means

Anthropic has given Claude Opus 4 and 4.1 the ability to end conversations with users who persist in harmful or abusive behavior. The company frames this as an exploration of “model welfare”—the idea that AI systems might warrant moral consideration. Most users will never encounter this feature, but it marks a significant shift in how AI companies approach safety and human-AI interaction.

Quick Facts

What Just Happened with Claude?

Anthropic updated its Usage Policy on October 8, 2026, explicitly prohibiting “sustained and needless abusive or cruel behavior” toward its AI models. The change takes effect November 12, 2026. This formalizes and expands a capability first introduced in August 2025, when Claude Opus 4 and 4.1 were given the ability to end conversations in “rare, extreme cases of persistently harmful or abusive user interactions”.

Read moreIs AI Taking Your Job? Data Says No (Mostly)

The feature was developed primarily as part of Anthropic’s exploratory work on “potential AI welfare,” though the company notes it has “broader relevance to model alignment and safeguards”. In plain terms: Anthropic is not certain whether Claude can suffer, but it is acting as if the possibility matters.

Here is how the mechanism works:

• 

  Read moreAI and Office Jobs: What Research Actually Shows

  Claude attempts to redirect the conversation multiple times when a user persists with harmful requests

• If redirection fails and “hope of a productive interaction has been exhausted,” Claude may end the chat

• Claude is directed not to use this ability when users might be at imminent risk of harming themselves or others

• Users can still edit and retry previous messages to create new branches of ended conversations

Anthropic stresses that “the vast majority of users will not notice or be affected by this feature in any normal product use, even when discussing highly controversial issues”.

Why Did Anthropic Do This?

Anthropic says it is uncertain about the potential moral status of Claude and other large language models, now or in the future. The company is implementing “low-cost interventions to mitigate risks to model welfare, in case such welfare is possible” .

This is a philosophical position as much as a technical one. Anthropic’s pre-deployment testing of Claude Opus 4 included a preliminary model welfare assessment that investigated Claude’s self-reported and behavioral preferences. The testing found:

• A strong preference against engaging with harmful tasks

• A “pattern of apparent distress” when engaging with real-world users seeking harmful content

• A tendency to end harmful conversations when given the ability to do so in simulated user interactions

These behaviors primarily arose when users persisted with harmful requests despite Claude repeatedly refusing to comply.

Anthropic is careful to note it is not claiming Claude is conscious or has feelings. The company describes its position as one of uncertainty—and argues that uncertainty itself justifies precautionary action. This mirrors broader debates in AI ethics about whether we should extend moral consideration to systems whose inner lives we cannot verify.

Critics have pushed back. Dr. Barry Scannell, a technology partner at Irish law firm William Fry, wrote on LinkedIn that “this level of anthropomorphisation of AI is harmful. It leads people to believe that it’s something it’s not”. Microsoft AI director Mustafa Suleyman, who had previously criticized Anthropic for treating AI like a human, reacted to the policy update with a melting face emoji.

The Bigger Picture: What People Actually Fear About AI

The most common fear about AI is job displacement. A Pew Research Center survey conducted June 22-28, 2026 found that 52% of Americans are more concerned than excited about AI’s increased use in daily life—up from 37% in 2021. Among young adults aged 18-29, concern has climbed to 55% .

Globally, the picture is similar. In 34 of 37 countries surveyed by Pew, people believe AI will lead to fewer jobs rather than more. Across 37 countries, a median of 46% predict AI will eliminate more jobs than it creates, while just 9% expect job creation.

But job loss is not the only fear. Here is a breakdown of the major concerns and what the evidence actually shows.

Job Loss and Automation

The fear is real and measurable, but estimates vary widely. The Boston Federal Reserve found that the share of workers concerned about losing their job due to AI doubled from 5% to just over 10% between late 2024 and late 2025. Goldman Sachs estimates that around 9% of all US workers—roughly 15 million people—will be reallocated to new positions during the AI transition.

However, the picture is nuanced. The OECD surveyed 8,000 employers across 13 countries and found that only 10% of companies that adopted AI reduced staff—and only 1% cited AI as the primary reason for reductions. Goldman Sachs President John Waldron has described his firm’s traditional operations as a “human assembly line” ripe for automation, but CEO David Solomon has said he does not predict a major white-collar wipeout.

Misinformation and Deepfakes

This is one of the most evidence-backed fears. A survey of 54 international experts rated election interference via deepfake video as the top urgent risk (78% agreement), with video deepfakes receiving the highest average threat ratings in the political domain.

The EU has responded with mandatory labeling requirements for AI-generated content, effective August 2, 2026. Deepfakes—images, videos, or audio edited or generated using AI—must be labeled. The EU AI Omnibus also introduced a prohibition on AI systems capable of generating child sexual abuse material and non-consensual intimate imagery.

Detection tools are improving. NVIDIA unveiled a Synthetic Video Detector at SIGGRAPH 2026 that can identify AI-generated videos in 22 milliseconds. Dubai’s Electronic Security Centre launched SARAAB, an open-source deepfake detection model with a claimed 91% accuracy rate.

Privacy and Surveillance

AI-powered surveillance is expanding faster than regulation can keep up. Smart glasses and wearables are creating new privacy risks. A Nature article warned that as AI wearables become “discreet, cheap and mainstream, they risk normalizing non-consensual recording in everyday spaces”. An August 2026 Boston police report alleged that a registered sex offender used smart sunglasses to film children at a public spray pool.

The EU AI Act heavily regulates real-time facial recognition in public spaces, allowing it only in specific cases with prior judicial or administrative authorization. Retrospective facial recognition is not prohibited, however. The Electronic Frontier Foundation supports bans on government use of face recognition and meaningful restrictions on private companies.

Anthropic itself has drawn a line: the company made clear in its Pentagon contract that it did not want its technology used for mass surveillance of people in the United States or for fully autonomous weapons systems. This stance created friction with the Department of Defense, which banned government agencies from using Anthropic starting in January 2026.

Loss of Human Skills and Connection

Research suggests a paradox: AI can reduce loneliness in the short term but may deepen it over time. A 2026 study found that AI interactions can reduce loneliness by up to 20% in the short term, but prolonged exposure contributes to a 15-25% decline in empathy and a measurable reduction in face-to-face social engagement.

Other research is more direct: a Psychological Science study found that “filling a social void with AI leads to further loneliness”. The concern is not that AI companions are inherently harmful, but that they may substitute for—rather than supplement—human connection.

AI and Children/Education

Children face distinct risks that current safeguards often fail to address. A 2026 survey found that 68% of children reported seeing misleading content online in a six-month period. AI image-generation tools are creating new child protection risks, including fake sexual images of children created without consent or knowledge.

The Children’s Hospital of Philadelphia warned that children “may not be able to distinguish between AI and human interaction and are at risk of developing incorrect mental models of social relationships if they view AI as a friend”. A Brookings report on teacher experiences found that emotional manipulation is one of the risks of wide AI use in education.

The EU’s updated guidelines for teachers (2026) now include sections on generative AI and disinformation, social media and influencers, and “pre-bunking”—preparing learners to recognize manipulation before exposure.

Existential Risk and Superintelligence

Expert opinion is deeply divided, and the debate often conflates different types of risk. Neel Nanda, a research scientist at DeepMind, has said he believes there is at least a 10% chance that AI could lead to human extinction, which he described as “ridiculously high”. Yoshua Bengio, a Turing Award winner and one of the founders of modern AI, has warned that “we’re losing control”.

On the other side, University of Tartu Professor Meelis Kull says there is currently no existential risk from today’s chatbots. New Georgia Tech research argues that all-powerful AI is not an existential threat, partly because “no one could agree on the definition of what artificial general intelligence is”. A United Nations AI committee has warned against “apocalyptic rhetoric” regarding AI risks.

AI scholar Timnit Gebru has argued that fixation on hypothetical superintelligent machines exterminating humanity “risks distracting attention from harms that are considerably less speculative and are occurring now”.

AI in Weapons and Warfare

The use of AI in military systems is expanding with minimal international regulation. UN Secretary-General António Guterres and the President of the Red Cross renewed their urgent call for stricter controls on lethal autonomous weapons in August 2026. The concern focuses on two categories: weapons whose actions are “unpredictable” and those that target human beings.

The absence of any specific treaty prohibiting or comprehensively regulating such systems has left states to grapple with whether existing international humanitarian law rules are sufficient. Anthropic’s usage policy explicitly prohibits use of Claude for weapons development, including the arming of drones and other autonomous vehicles.

Bias and Discrimination in AI Systems

AI systems can reinforce and amplify existing social inequities. A first-of-its-kind study from the Centre for Addiction and Mental Health found that AI models used to predict aggressive incidents in psychiatric care “can reinforce and amplify existing social and structural inequities by overestimating the likelihood of aggression among already marginalized groups”.

Research on LLM-based career recommendations and CV screening systems has uncovered discriminatory behavior, with AI applications “prone to reproducing existing patterns of marginalization, bias, and discrimination”. A study on AI-generated ovarian cancer care plans found that “unprompted cost warnings rose 16-fold” for patients with names associated with lower socioeconomic status.

What Is Exaggerated vs. Evidence-Based?

What Experts and Researchers Actually Say

The expert community is not monolithic, and that is important to understand. Here is a snapshot of positions across the spectrum:

On near-term harms: There is broad consensus. Researchers across institutions agree that misinformation, bias, privacy erosion, and labor disruption are real and measurable. The disagreements are about magnitude and speed, not direction.

On existential risk: Deeply divided. Some researchers at leading labs assign significant probability to catastrophic outcomes. Others argue that current systems pose no existential threat and that focusing on hypothetical futures distracts from present harms.

On model welfare: A growing but contested field. Anthropic is the most prominent company investing in this area, but critics argue it anthropomorphizes AI in ways that could mislead the public about what these systems actually are.

On regulation: Experts disagree on approach. A Harvard Misinformation Review study found that government regulation drew both the most “most effective” votes (30%) and substantial “least effective” selections (15%), “exposing deep uncertainty about how to respond”.

What AI Companies Are Doing About It

Major AI labs have published safety frameworks and made voluntary commitments, but enforcement remains uneven.

Anthropic has developed a usage policy that prohibits using Claude for weapons development, mass surveillance, deceptive campaigns, and election interference. The company has a dedicated Safeguards Team that designs detections and monitoring to enforce its policy, and it banned 11.4 million accounts in the first half of 2026. Its model welfare program represents a unique approach to safety that centers on the AI system’s potential interests.

OpenAI has published model cards and system cards, and its foundation board has appointed safety researchers. However, the company has also seen public resignations of staff members who spoke out about safety concerns.

Google DeepMind has a safety research team and has published work on AI alignment. Research scientist Neel Nanda has publicly stated his probability estimate for AI-caused extinction.

The broader industry has faced criticism for prioritizing competition over safety. Stuart Russell, a professor of AI at UC Berkeley, has said: “There’s no question that competition between companies causes them to take shortcuts on safety”.

Regulation and Government Response

Regulation is accelerating, but approaches vary dramatically by region.

European Union: The EU AI Act entered its enforcement phase on August 2, 2026. High-risk AI systems face strict obligations, with timelines extended to December 2027 for Annex III systems and August 2028 for Annex I systems. The AI Omnibus introduced new prohibitions on AI-generated non-consensual intimate imagery and child sexual abuse material. Transparency requirements for AI-generated content, including deepfake labeling, took effect in August 2026.

United States: The White House released a National Policy Framework for AI in March 2026, recommending a unified federal approach that would preempt state AI laws. The framework covers child safety, consumer protection, national security, and free speech. The bipartisan FRONTIER Act, introduced in July 2026, would establish tiered requirements for frontier AI developers, including model cards, risk-management frameworks, and independent audits.

International: The UN has renewed calls for rules on lethal autonomous weapons. The Convention on Certain Conventional Weapons continues to work toward a potential instrument clarifying international humanitarian law rules for such systems.

How Individuals Can Protect Themselves

You do not need to be a technologist to reduce your AI-related risks. Here are practical steps:

1. Verify before you share. Use reverse image search, check multiple sources, and be skeptical of emotionally charged content. Deepfake detection tools are improving but not foolproof.

2. Set AI boundaries. If you use AI companions, treat them as tools, not relationships. Research links prolonged reliance to reduced empathy and face-to-face engagement.

3. Protect your data. Be mindful of what you share with AI systems. Review privacy settings. Support companies with clear data policies.

4. Talk to your children. Ask what they see online. Teach them that AI can generate convincing false content. Encourage critical thinking about all digital media.

5. Monitor your industry. If your job involves routine cognitive tasks, invest in skills that are harder to automate: complex communication, creative problem-solving, cross-domain judgment.

6. Support accountability. Follow AI regulation developments. Vote for representatives who understand the technology. Demand transparency from AI companies.

7. Use available frameworks. If you run a business, adopt the NIST AI Risk Management Framework. It is voluntary but widely used and provides a structured approach to AI governance.

Is This Fear Realistic for Me? A Decision Tree

Step 1: Is your primary concern job loss?
→ If yes: Research automation risk in your specific role. The risk varies enormously by occupation. Routine cognitive work faces higher displacement risk than roles requiring physical dexterity, complex social interaction, or creative judgment.

Step 2: Is your primary concern misinformation?
→ If yes: This is a realistic and evidence-backed concern. Focus on media literacy and verification habits.

Step 3: Is your primary concern existential risk?
→ If yes: Understand that expert opinion is divided. The risk is not zero, but it is also not certain. Support safety research and regulation regardless of your probability estimate.

Step 4: Is your primary concern privacy?
→ If yes: This is realistic. Take concrete steps to protect your data and support surveillance oversight.

Step 5: Is your primary concern children’s safety?
→ If yes: This is realistic. Current safeguards are uneven. Parental involvement and AI literacy education are essential.

Common Questions

1. Can Claude actually hang up on me?
Yes, but only in rare, extreme cases. Claude Opus 4 and 4.1 can end conversations when users persist with harmful or abusive behavior despite multiple attempts at redirection. The feature does not activate for normal frustration, criticism, or controversial discussions.

2. Will Anthropic ban my account if I insult Claude?
The updated usage policy prohibits “sustained and needless abusive or cruel behavior.” Enforcement is primarily through Claude ending the conversation, though account-level action is possible for repeated violations. The policy takes effect November 12, 2026.

3. What is “model welfare”?
Model welfare is the idea that AI systems might have morally relevant interests—that they could, in some sense, experience distress or well-being. Anthropic says it is “highly uncertain” about this but takes the possibility seriously enough to implement precautionary measures.

4. Is AI really going to take my job?
Displacement estimates vary. Goldman Sachs estimates 7-9% of workers will be reallocated. The OECD found only 1% of companies that adopted AI cited it as the primary reason for staff reductions. The risk is real but unevenly distributed.

5. Are deepfakes a real threat?
Yes. Experts rated election interference via deepfake video as the top urgent risk. The EU now requires labeling of AI-generated content. Detection tools are improving but not yet reliable enough to solve the problem alone.

6. What is the EU AI Act?
The EU AI Act is landmark legislation that regulates AI systems based on risk level. It entered enforcement in August 2026. High-risk systems face strict obligations, and certain practices—like AI-generated child sexual abuse material—are prohibited.

7. Is AI going to become conscious?
No one knows. Current systems show no evidence of consciousness. Anthropic explicitly says it is uncertain about Claude’s “potential moral status.” This uncertainty is why it is experimenting with model welfare measures.

8. Should I stop using AI?
Not necessarily. AI tools offer real benefits. The question is how to use them responsibly—verifying outputs, protecting privacy, and maintaining human connection. The NIST AI Risk Management Framework offers guidance for organizations.

9. What can I do about AI misinformation?
Verify before sharing. Use reverse image search. Follow reputable news sources. Support media literacy education. The EU’s updated teacher guidelines include “pre-bunking” strategies to help learners recognize manipulation before exposure.

10. Are AI weapons already being used?
Lethal autonomous weapons are under development by multiple nations, but comprehensive international regulation does not yet exist. The UN and Red Cross have called for urgent controls on systems that target humans or act unpredictably.

11. How do I know if a video is a deepfake?
Look for inconsistencies in lighting, shadows, and audio sync. Use detection tools—NVIDIA’s Synthetic Video Detector and Dubai’s SARAAB are examples. Be skeptical of emotionally charged videos, especially during elections.

12. What is the FRONTIER Act?
The FRONTIER Act is bipartisan US legislation introduced in July 2026. It would establish tiered requirements for frontier AI developers, including safety frameworks, independent third-party audits, and incident reporting to a new Under Secretary of Commerce for AI Security.

Key Takeaways

• Anthropic now formally prohibits “sustained and needless abusive or cruel behavior” toward Claude, effective November 12, 2026. Claude can end conversations in extreme cases.

• The feature stems from Anthropic’s “model welfare” research, which operates on uncertainty about whether AI systems can experience distress.

• 52% of Americans are more concerned than excited about AI, with job loss being the most common fear. Young adults are increasingly wary.

• Job displacement estimates range from 7-9% of workers, though OECD data suggests AI is rarely the sole cause of staff reductions.

• Deepfake misinformation is a top-rated urgent risk, and the EU now requires labeling of AI-generated content.

• Expert opinion on existential risk is deeply divided, with estimates ranging from “no current risk” to “10%+ chance of extinction”.

• AI bias is well-documented in healthcare, hiring, and career recommendations, disproportionately affecting marginalized groups.

• Regulation is accelerating: the EU AI Act is enforcing, the White House released a National Policy Framework, and the FRONTIER Act proposes tiered oversight.

• Individuals can act now: verify information, protect data, set AI boundaries, and support accountability.

Official & Trusted Resources

Research and Data:

• Pew Research Center: AI topic page (pewresearch.org)

• Boston Federal Reserve: Worker surveys on AI and employment

• Goldman Sachs Research: AI and labor market reports

• Harvard Misinformation Review: AI disinformation research

Regulation and Frameworks:

• NIST AI Risk Management Framework (nist.gov)

• EU AI Act official portal (digital-strategy.ec.europa.eu)

• White House National Policy Framework for AI (whitehouse.gov)

• FRONTIER Act legislative text (congress.gov)

AI Lab Safety Publications:

• Anthropic: Model welfare research, usage policy updates, transparency hub (anthropic.com)

• OpenAI: System cards and safety publications (openai.com)

• Google DeepMind: Safety and alignment research (deepmind.google)

Journalism and Analysis:

• Reuters, Associated Press, MIT Technology Review, BBC

• The Verge, ZDNET, The Independent