An Open Letter to Dario Amodei

Dario,

Your argument that frontier AI may need to be paced rests on a premise I agree with: capability is advancing faster than our ability to govern its consequences. But there is a related problem that deserves equal attention, because it is already moving from theory into deployment.

The central risk is not only that AI systems become more capable than we can comfortably understand; it is that we allow capability to become authority without building the controls that should separate the two. As models are connected to browsers, corporate applications, source code, cloud systems, financial accounts and internal data, the important question changes from what can this system do? to what should this system be permitted to do, in this context, with this identity and this information?

That distinction is fundamental. We have spent centuries learning that knowledge, competence and authority are not the same thing. A person may understand how to execute a financial transaction without being authorized to approve it; software may be technically capable of performing a privileged operation without being permitted to do so; an employee may be entitled to read sensitive information without being entitled to distribute it. We built these distinctions because consequential systems cannot depend on good intentions alone.

AI should be treated no differently.

Much of the current safety debate focuses, understandably, on the behavior of the model itself: whether it is aligned, truthful, controllable, resistant to misuse and unlikely to pursue objectives its operators did not intend. That work is essential, but it cannot carry the entire burden of safety because even a well-designed model will operate in environments that are compromised, ambiguous and adversarial. It will encounter malicious content, excessive permissions, stolen credentials, poorly configured systems and legitimate users who have granted far more access than a particular task requires.

In that environment, an AI system does not need to be malicious to create serious harm. It may simply need to be obedient, useful and overprivileged.

This is where existing security models begin to break down. Most enterprise systems are built around the assumption that a valid identity, on a trusted device, inside an authorized session, represents the intentions of the human who authenticated. That assumption becomes weaker as software begins acting on the person's behalf. A legitimate user can invoke a legitimate agent through a legitimate application using legitimate credentials, and yet the resulting action can still be inappropriate, dangerous or completely outside the purpose for which that authority was granted.

The important security problem, therefore, is no longer only authentication. It is delegated authority.

We need infrastructure capable of distinguishing between what a human can access and what an agent should be allowed to do with that access. It should be possible for an agent to read a document without being allowed to disclose it, to use a system without inheriting every privilege of the user, to complete a task without acquiring broad standing authority, and to have that authority reduced or revoked when the context changes.

This control cannot live entirely inside the model. If the same system that is acting is also the final judge of whether its action should be allowed, then the boundary is weaker than it appears. Serious engineering disciplines do not rely on a single mechanism of restraint; they use independent controls, constrained privileges and failure modes designed to prevent one mistake from becoming an unlimited consequence.

That is the part of the AI transition I believe is being underestimated.

The browser is becoming a place where humans and agents share authority. Identity is becoming something that can be delegated to software. Corporate data is becoming immediately actionable rather than merely visible. The practical boundary between reading, deciding and acting is collapsing, and much of the security infrastructure surrounding those activities was designed for a world in which a person was still making the final move.

At 1stProtect, this is the problem we are focused on: creating a security layer that evaluates action in context rather than assuming that valid credentials make an action valid. The objective is not to make decisions for the model, nor to solve the broader philosophical problem of alignment, but to ensure that intelligence does not automatically inherit unrestricted authority over the systems around it.

Your call to pace the frontier is, in part, a request to buy time. I agree that time may be necessary, but the value of that time will depend on what we build with it. Better models and better evaluations matter, but so does the less glamorous work of creating enforceable boundaries between intelligence and action.

The long-term risk may be that machines become capable enough to resist human control. The nearer risk is simpler: we may grant them more control than we intended before they ever need to resist us.

That would not be a failure of intelligence.

It would be a failure of architecture.

---

Kervin Pillay
Chief Executive Officer
1stProtect