Every SRE team has the same story. Here is how we change the ending.

Every SRE has a version of the same story. The details change — the service name, the tool, the hour — but the shape of it is always the same.

An alert fires in the middle of the night. Someone gets paged. They log in half-asleep, open Splunk, open AppDynamics, open the Kubernetes dashboard, search Confluence for a runbook they half-remember, check GitLab for what deployed recently. Forty minutes later they have an answer. They apply a fix. They write an incident report at 3 AM that nobody reads until Tuesday. And somewhere in the back of their mind, they already know: this will happen again.

I have been in enough of those situations — and more recently, working across platforms running Splunk, AppDynamics, AKS, AWS, and GitLab — to recognise a pattern that goes beyond any single team or company. The tools are excellent. The process around them is broken. Every piece of evidence needed to diagnose the problem exists somewhere in the stack. The issue is that a human has to manually connect those dots, at the worst possible time, under pressure, often alone.

The question that kept coming back to me was simple: what if the system itself did the detective work?

Not a smarter alert. Not a better dashboard. An actual reasoning system — one that reads Splunk, queries AppDynamics, checks the Kubernetes pods, searches the Confluence runbooks, looks at what GitLab deployed recently, and comes back with a diagnosis and a proposed action. Something that does in four minutes what currently takes forty.

That is what this article is about. Not a concept — a real system I have been designing and building, grounded in how enterprise SRE teams actually work. Here is what I learned, what the architecture looks like, how every tool in a typical enterprise stack connects into it, and why I think this is the most meaningful shift in reliability engineering in the last decade.

The Real Problem Is Not the Technology

Here is the thing that surprises people when I describe this. The tools most SRE teams already have — Splunk, AppDynamics, Kubernetes, Confluence — are genuinely good. The data is there. The runbooks are there. The answers, more often than not, are already in the system.

The problem is the process. Specifically, the fact that it requires a human to manually connect all those dots, in real time, under pressure, often in the middle of the night. And that human — however experienced — is working sequentially through a checklist that a computer could work through in parallel, in seconds.

I have seen this play out the same way across teams large and small. The most common SRE story is not "we didn't have the data." It is "we had the data, but it took us 40 minutes to read it all and put the picture together." That gap — between the data existing and the diagnosis being formed — is exactly where an AI agent lives.

The numbers above are not projections. They are real results published by teams who have deployed this kind of system — from ClickHouse, Komodor, Microsoft, and others in 2025. The technology to achieve this exists today.

What Is an SRE AI Agent, Really?

I want to be precise here, because there is a lot of noise around "AI agents" right now.

An SRE AI Agent is not a chatbot that answers questions about your infrastructure. It is not a dashboard with AI-generated summaries. It is an autonomous software system that runs a continuous detect → investigate → diagnose → fix → verify loop, using a large language model as its reasoning engine and your existing tools as its hands.

"Think of it as your most experienced SRE — the one who knows every system, every past incident, every runbook — available at the speed of a computer, around the clock."

The agent watches your observability data. When something looks wrong, it starts asking questions across all your tools simultaneously — the way a seasoned engineer would, but in seconds instead of minutes. It reads the logs, checks the metrics, looks at what deployed recently, searches the runbook library, and forms a diagnosis. Then it either acts on that diagnosis automatically for safe, well-understood problems, or it asks you to approve its proposed action.

Every decision it makes is logged, explained, and reversible. The engineers stay in control — they just don't have to do the routine detective work anymore.

The Architecture — Five Layers That Work Together

When I designed this, I settled on a five-layer architecture. Each layer has one job, and nothing in a lower layer knows about the layers above it. This separation is what makes the system scalable and easy to change.

The key design decision I made was separating the orchestrator from the specialists. The Orchestrator Agent is the Incident Commander — it receives an alert, decides which specialist agents to dispatch, reads their findings, reasons over everything, and makes the final call. The specialist agents are narrow and focused: the Logs Agent only reads logs, the Kubernetes Agent only talks to the Kubernetes API. This keeps each piece simple and testable.

The other critical decision was using MCP (Model Context Protocol) as the connection layer between agents and tools. Think of MCP as the USB-C standard for AI tools — one protocol that connects everything. Splunk has an official MCP server. AppDynamics connects via REST. Kubernetes connects natively. Adding a new tool in the future means writing one small connector file, not rebuilding the agent.

How It Actually Works — The Eight-Step Loop

Here is the complete journey from a problem occurring to the system being healthy again. This happens automatically, in the background, for every single alert.

The step that surprises people most is the Safety Check. Before acting on anything, the agent evaluates the risk of its proposed action. Restarting a crashed pod is low risk — the agent does it automatically. Rolling back a deployment or scaling down infrastructure is medium risk — it sends a Slack message with an Approve or Reject button and waits for a human. Modifying a database or deleting resources is high risk — it escalates directly to the on-call engineer with a full briefing.

This graduated autonomy model is what makes the system trustworthy. You start with the agent asking permission for everything. Over time, as you see it making good decisions consistently, you expand the list of actions it can take automatically. Trust is earned incrementally, not assumed.

On Context Engineering

The biggest lesson I learned building this: it is not really an AI agent project — it is a context engineering project. What you put into the LLM's context window determines everything. The right logs, the right time window, the right runbooks — that is what produces 92% accuracy. Too much context and model quality drops non-linearly. The art is knowing exactly what to load and what to leave out.

Integrating With Your Existing Stack

One of the most common questions I get is: Do I need to replace my existing tools? The answer is no. The agent is designed to plug into whatever you already have. It does not replace Splunk or AppDynamics — it reads them. It does not replace GitLab — it creates issues and merge requests inside it. It does not replace Kubernetes — it talks to the Kubernetes API directly.

Here is how each tool in a typical enterprise stack connects to the agent:

The beauty of this architecture is that every tool speaks the same language — the MCP standard. Adding a new tool in the future means writing one small connector file, not re-architecting the agent. I have made this fully configurable through a YAML file, so enabling or disabling a tool is a one-line change with no code deployment needed.

A Real Example: What Happens When AppDynamics Fires at 2 AM

Let me walk you through exactly what happens the next time an incident fires — with your specific stack.

How to Build It — The Technology Choices

I deliberately kept the technology choices boring. Boring choices in infrastructure are good — they are predictable, well-documented, and easy to hand off to another engineer.

The entire agent is written in Python. The orchestration framework is LangGraph — it reached v1.0 stability in late 2025 and is now used in production by companies like Uber, LinkedIn, and Cisco. The LLM is Claude Sonnet for deep reasoning and Claude Haiku for fast triage classification. The web server is FastAPI for receiving webhooks. Tool connections use the MCP protocol.

The most important design principle I applied is: everything is configurable through YAML, nothing is hardcoded. Which tools are enabled, which LLM model to use, which actions require human approval, which services have SLO targets — all of this lives in config files, not in the code. This means you can change behaviour without deploying new code, and different environments (dev vs staging vs prod) have different configurations.

Biggest Architecture Mistake (And How to Fixed It)

In many projects, the initial idea is to build one large, all‑in‑one agent that can handle every task. However, this approach often leads to challenges. Large agents can run into LLM context limits, become slower, and struggle when they lose track of information.

A more effective approach is to break the system into smaller, specialized agents. Each agent focuses on a specific task, making the overall system faster, easier to test, and more reliable. An orchestrator then combines the outputs from these specialists.

This shift toward a multi‑agent architecture commonly improves performance significantly-often raising accuracy from around 65% to more than 90%.

Where to Host It

This is the question I get most often from people who already have a Kubernetes or AKS cluster running. The short answer is: host it on your existing Kubernetes cluster (AKS/AWS/GCP, etc.). You do not need a new platform. The agent runs as a Kubernetes Deployment, just like any other service in your stack.

One thing worth emphasising about running on AKS: since the agent runs inside the same Kubernetes cluster it monitors, it connects to the Kubernetes API natively without any network complexity. The agent pod gets a Kubernetes ServiceAccount with carefully scoped RBAC permissions — read-only access by default, write permissions only for specific namespaces and only for actions we have explicitly approved.

On Safety — The Part Most Articles Skip

Every article about AI agents eventually gets to this section and then rushes through it. I want to spend real time on it because safety is not a nice-to-have here — it is the whole reason leadership will let you build and deploy this in production.

The safety model I implemented has three layers.

First: risk scoring before every action. Every possible action the agent can take — restart a pod, rollback a deployment, scale a service — has a pre-assigned risk score from 1 to 10. The agent calculates the risk score of its proposed action before doing anything. Score 1–3 executes automatically. Score 4–7 requires a Slack approval. Score 8–10 always escalates to a human with a full briefing, no matter what.

Second: every action must have a rollback path. The agent is not allowed to take an action unless there is a documented way to undo it. This sounds obvious but it eliminates an entire class of dangerous actions from consideration before the agent even thinks about them.

Third: circuit breakers stop runaway agents. If the agent's remediation fails twice in a row for the same incident, it stops trying and escalates to a human. No infinite retry loops. No cascade of failed fixes making things worse.

The Rule I Never Compromise On

The agent is never allowed to delete resources, modify production database schemas, or change security configurations without a human physically approving the action with their identity confirmed. Some lines are not worth crossing for the sake of automation speed.

Monitoring the Agent Itself

Here is something that surprised me: the most useful tool for monitoring the SRE agent is the exact same tool it uses to monitor everything else — Splunk.

The agent logs every action it takes in structured JSON format — what it decided, why it decided it, what tool it called, what the result was, how long it took, and what it cost. All of these logs go to Splunk. I have a Splunk dashboard that shows me: incidents handled this week, auto-resolve rate, false-positive rate, average reasoning time per incident, and LLM API cost per incident.

For the deeper tracing — seeing exactly what the agent was thinking at each step — I self-host Langfuse on the same AKS cluster. Langfuse is like a flight recorder for AI decisions. Every LLM call, every tool invocation, every state transition is recorded with timing and cost. When the agent makes a wrong diagnosis, I can replay the incident in Langfuse and see exactly where its reasoning went wrong.

I also created an AppDynamics health rule on the agent's own health endpoint. If the agent goes down, AppD pages the team the same way any other production service would. The watchdog has its own watchdog.

Five Lessons I Wish I Had Known at the Start

1. Start in shadow mode, not production mode. The first thing you should do is run the agent alongside your team, where it investigates every incident and records what it would have done — but takes no action. Spend a few weeks comparing its answers to what your engineers actually did. This builds trust and reveals where the agent's reasoning needs improvement, before it has any real-world consequences.

2. The prompts matter more than the model. I spent weeks obsessing over which LLM to use. In the end, the factor that moved accuracy the most was the quality of the system prompt — specifically, the instructions about how to reason through an SRE incident and how to format its output. A well-prompted weaker model beats a poorly prompted stronger one every time.

3. Your Confluence runbooks are gold. The single biggest source of accuracy improvement came from loading all our SRE Confluence pages into the vector database. The agent stopped guessing solutions it had never seen before and started applying solutions that our own engineers had documented and proven. Knowledge that was locked in a wiki became an active part of every incident response.

4. Build depth before breadth. I tried to connect every tool at once in the first sprint. The result was a system that could talk to everything but understood nothing deeply. The second attempt: spend the first two weeks getting Splunk and Kubernetes deeply right. Only then add AppDynamics, GitLab, and the rest. Quality of integration matters far more than quantity.

5. Observability for the agent is not optional. You must be able to see exactly what the agent was thinking and why. Langfuse was the tool that made this possible for me. Without it, debugging a wrong diagnosis is nearly impossible — you are left guessing at the reasoning behind a decision that happened inside a black box. Log everything from day one.

Where This Goes From Here

We are at the beginning of a genuine shift in how reliability engineering works. The teams that take this seriously — not as a quick AI experiment but as a real system with proper observability, guardrails, and iterative trust-building — will have a compounding advantage. Every incident the agent handles teaches it something. Every runbook added to Confluence improves its accuracy. Every false positive it learns to filter makes the next on-call week a little quieter.

And I want to be direct about what this is not. It is not a silver bullet. It is not a replacement for strong engineers — if anything, it makes good engineers more powerful by removing the routine work and letting them focus on the judgment calls that actually need human thinking. It is not something you deploy in a weekend and hand over. Building a system like this takes thoughtful design, real testing, and a willingness to expand its autonomy slowly as it earns trust.

Done right, though, it genuinely changes the experience of being an SRE. Not just the metrics — the actual feeling of the job. Fewer interruptions. More time to build. Less time firefighting problems that a computer should have caught.

The SRE story that opens every article in this space — the bleary-eyed engineer, the 2 AM page, the 40 minutes of tab switching — that story does not have to end the same way anymore. The ones that do reach an engineer can arrive with a full briefing already prepared. Root cause identified. Evidence gathered across every tool in the stack. Action awaiting a single tap.

That is a fundamentally different experience of this work. That is what this is for.

If you are building something similar or thinking about where to start — I would love to hear from you. These systems are still evolving quickly and every implementation teaches something new. Connect with me on LinkedIn or Medium and share what you are working on.

Senior SRE & Platform Engineering Leader · Building scalable, configurable AI-powered infrastructure systems.

Follow on LinkedIn · Connect for SRE & AI engineering conversations

Originally published at https://www.linkedin.com.