AI Agents Are Becoming a New Layer in Software Architecture

For the last several years, most applications of generative AI have followed a fairly familiar pattern.

A user submits something. An application sends a prompt to a model. The model returns a response. The application displays or processes that response.

That model isn't disappearing, but I think we're beginning to see something considerably more important emerge.

AI agents are becoming an architectural layer of their own.

OpenAI's recently announced Agents API is a good example of how quickly this transition is occurring. Rather than asking developers to assemble every component required for an autonomous workflow, the platform provides infrastructure for long-running sessions, tool use, context management, sandboxed execution and multiple cooperating agents.

That distinction matters.

From API Call to Digital Worker

Traditional APIs are largely transactional.

Send input.

Perform an operation.

Receive output.

An agent can potentially operate very differently.

Give it an objective and it may need to inspect data, call several APIs, run code, evaluate the results, delegate portions of the task, recover from errors and continue working until it reaches an acceptable result.

That begins to look less like a conventional API request and more like a worker operating inside your system.

From an architectural perspective, that's a significant change.

Developers now have to think about concepts such as:

• What tools can the agent access?
• What data should it be allowed to see?
• How long can a task run?
• What actions require human approval?
• How should failed work be recovered?
• How do we observe what the agent actually did?
• Where does the agent execute?
• How do we prevent one bad decision from cascading through other systems?

These are infrastructure and systems-design questions, not prompt-engineering questions.

The Agent Harness Becomes Infrastructure

One of the interesting aspects of OpenAI's Agents API is the idea of separating the intelligence of the model from the infrastructure required to operate an agent.

The platform handles much of the orchestration surrounding long-running work, including context management and coordinating subagents.

That means application developers can spend more time defining business capabilities.

An application might expose tools such as:

getCustomer()

createInvoice()

checkInventory()

deployApplication()

queryTelemetry()

sendNotification()

The intelligence layer determines when and how those capabilities should be used.

The application still owns the capabilities and permissions.

That's an important architectural boundary.

APIs May Increasingly Be Designed for Two Audiences

Historically, APIs were primarily designed for software developers and other applications.

Increasingly, they will also be designed for agents.

That could influence API design significantly.

Agents benefit from clearly defined schemas, descriptive operations, predictable errors and narrowly scoped capabilities. In other words, many of the practices that already make a good API may become even more important.

We may eventually evaluate an API on two dimensions:

How easy is it for a developer to use?

And how safely can an autonomous agent use it?

Permission Design Becomes Critical

Giving an AI model access to a system isn't particularly interesting by itself.

Giving it permission to modify that system is where architecture becomes important.

A useful agent might be allowed to investigate a production problem automatically but require human approval before changing infrastructure.

Another might draft a customer refund but need approval before issuing one.

An engineering agent could create a pull request but not merge it.

This creates a very useful architectural pattern:

Agents investigate and recommend. Systems enforce policy. Humans approve consequential actions.

Not every application needs that exact model, but bounded authority is likely to become one of the central principles of production agent design.

Observability Will Matter as Much as Intelligence

Traditional applications produce logs, traces and metrics.

Agentic applications need those things too, but they also need an audit trail of reasoning and actions.

If an agent makes twenty tool calls while completing a task, developers will eventually need to answer questions such as:

What did it access?

What did it modify?

Why did it choose that tool?

Which step failed?

How much did the workflow cost?

Was a human involved?

Production AI will require considerably more than a good model.

It will require good operations.

The Bigger Architectural Shift

I don't think the most important development in AI is that models continue getting better at answering questions.

The larger shift may be that software systems are gaining a new kind of computational participant.

We've traditionally architected systems around clients, APIs, services, databases, queues and workers.

Agents may soon belong on that list.

And when they do, the engineering challenge won't simply be making them intelligent.

It will be making them reliable, observable, secure and appropriately constrained.

That is where AI stops being a feature.

It becomes architecture.