What struck me when I read Anthropic’s September threat intelligence report was how much the company could piece together about its users’ activities. It could analyze exchanges, link accounts, reconstruct workflows, and assess the actors behind them. Detecting and disrupting the cyberattacks, surveillance, and other abuse described in the report has real value. The same cases also reveal how much an AI provider can observe. Report

The public evidence does not establish that Anthropic maintains a complete profile of every user, let alone that employees can browse conversations at will. Still, I find the concentration of capability unsettling: a private company holds powerful models, extensive user context, and systems for analyzing user behavior. We spend a great deal of time discussing how to stop people from misusing AI. How do we stop the companies that control it from misusing their position?

An assistant that also assesses its users

User profiling is nothing new. What is changing is the depth of information we provide. To get a useful answer, we explain our constraints, doubts, and reasoning. A technical discussion may expose a business plan; a career conversation may reveal family circumstances. Information once scattered across different settings now flows into one assistant. The provider receives not just our decisions, but how we arrive at them.

We share that context to get help. The platform also operates abuse detection, account investigations, and policy enforcement. The same information can help the assistant understand a task and help the provider assess the person behind it. Those uses require different permissions. Receiving the data should not make the boundaries between them disappear.

Anthropic describes using hierarchical summarization to condense individual interactions and analyze the summaries for account-level concerns. Its Clio documentation also distinguishes aggregate research from safety analysis: the latter can produce results linked back to individual accounts, accessible to a limited group of authorized staff. Safeguards · Clio privacy documentation

Analyzing interactions together can expose abuse that isolated requests would miss. But who decides when a user becomes an investigation subject, how far that investigation can reach, and when it ends? Safety explains the purpose. It does not establish the limits.

A conversation can be deleted. What about the judgment?

Analysis leaves more than transcripts behind. It can produce summaries, labels, and risk scores. Anthropic’s consumer policy says inputs and outputs flagged by automated safety systems as violating its Usage Policy may be retained for up to two years, and associated classification scores for up to seven. Source material and derived judgments can have different lifespans. Retention policy

Someone studying fraud is not necessarily committing it. Someone describing a situation may not be describing their own. If a discussion is misclassified and the resulting label continues to affect an account, how does the user correct it? That is a risk to guard against, not an incident established by the report. A provider does not need to understand someone completely to affect them. A mistaken interpretation can have consequences too.

Who constrains the provider?

I am willing to believe these systems were built to prevent abuse. But sound system design cannot depend entirely on an operator’s good intentions. Administrators make mistakes, systems are compromised, and companies change direction. Restricting ordinary employee access is one safeguard. The people authorized to expand data use or change access rules need constraints as well.

Those constraints should be concrete: separate authorization for individual investigations, access bounded by scope and time, records open to independent review, and a way to withdraw incorrect judgments from subsequent decisions. Privacy promises need mechanisms that can be checked.

Anthropic’s disclosure gives us a basis for this discussion. A company that can explain how it detects user abuse should also explain how it detects and limits abuse of its own capabilities. Greater capability brings greater responsibility. Beyond asking whether a provider deserves our trust today, we should ask whether we have made it sufficiently difficult for that provider to overstep tomorrow.