How Circle K Cut DevOps Incident Triage to Under 15 Minutes with Amazon Bedrock

Introduction

Retail giant Circle K runs hundreds of micro‑services that power everything from point‑of‑sale terminals to loyalty‑program back‑ends. When a production incident occurs, every minute of downtime translates into lost revenue and a hit to brand trust. In early 2026 the company faced a recurring challenge: detecting and triaging incidents quickly enough to keep mean‑time‑to‑recovery (MTTR) below 15 minutes.

By pairing Amazon Bedrock with the AWS DevOps Agent, Circle K built an automated pipeline that turns Bedrock token‑usage spikes into actionable incident investigations. The result? A 75 % reduction in MTTR and 94 % root‑cause accuracy across the first 30 pilot incidents.

• --

The Problem: Cost‑Anomaly Detection & Slow Manual Triage

1. Late visibility of Bedrock usage – Bedrock only publishes InputTokenCount and OutputTokenCount metrics to CloudWatch minutes after a model call, making cost spikes a lagging indicator.
2. Manual log correlation – Engineers had to stitch together CloudWatch logs, VPC flow logs, and application traces to understand why a spike occurred.
3. Long triage cycles – Typical detection → investigation → remediation took 30‑45 minutes, far above the target SLA for a 24/7 retail operation.

These pain points aligned with the five‑step DevOps incident response lifecycle (Detect, Triage, Diagnose, Remediate, Continuous Learning) described by PagerDuty.

• --

The Solution Architecture

![Architecture diagram placeholder]

| Component | Role |
|-----------|------|
| Amazon CloudWatch Alarms | Monitors InputTokenCount/OutputTokenCount and fires on anomalous spikes |
| Amazon SNS | Sends alarm notifications to Lambda and Step Functions |
| AWS Lambda (Trigger Function) | Normalises the alarm payload and starts the DevOps Agent workflow |
| AWS DevOps Agent (running on Amazon Bedrock AgentCore) | Performs a read‑only first‑pass investigation – queries CloudWatch, Config, IAM, and code repositories |
| AWS Step Functions (Standard) | Orchestrates validation, human approval, and optional remediation actions |
| Amazon Bedrock Model | Provides the reasoning engine that correlates telemetry, deployment metadata, and run‑book knowledge |
| Human Approver (via Amazon SNS / Slack) | Reviews the agent‑proposed fix before any change is applied |

How It Works

1. Detect – A CloudWatch alarm detects a sudden token‑count surge (e.g., > 3σ above baseline).
2. Trigger – SNS forwards the alarm to a Lambda function that invokes the DevOps Agent on Bedrock AgentCore.
3. First‑Pass Investigation – The agent, using read‑only credentials, pulls:

• Recent CloudWatch logs


• Terraform/CloudFormation change history


• CI/CD pipeline status


• Service‑mesh topology from AWS App Mesh

4. Reasoning – The Bedrock model evaluates the data, generates a root‑cause hypothesis, and suggests a remediation (e.g., revert a security‑group change).
5. Human Approval – Step Functions pauses (waitForTaskToken) until an on‑call engineer approves or modifies the recommendation.
6. Remediate – Upon approval, a second Lambda executes the fix (e.g., roll back a security‑group rule) using least‑privilege IAM roles.
7. Learn – The outcome is logged to an audit trail and fed back into the agent’s knowledge base for future incidents.

• --

Implementation Highlights

• Token‑Count as Early Cost Signal – By treating InputTokenCount as a cheap, real‑time cost metric, the team could detect anomalies within minutes of occurrence (see AWS blog on token counting).
• IAM Split – Two separate roles: one read‑only for investigation and one write‑enabled for remediation, satisfying the principle of least privilege.
• Shadow Mode Testing – The pipeline ran in shadow for 3 weeks, replaying 15‑30 past incidents. Accuracy matched the post‑mortem root causes in 94 % of cases.
• Model Swappability – The Bedrock model is referenced via an environment variable, allowing quick swaps between Claude, Titan, or custom fine‑tuned models without code changes.
• Human‑in‑the‑Loop – Even when the agent fails, an SNS alert notifies a human, ensuring the page never depends solely on the AI.
• --

Results & Business Impact

| Metric | Before | After |
|--------|--------|-------|
| Mean‑time‑to‑triage | 30‑45 min | < 15 min |
| MTTR reduction | – | 75 % lower |
| Root‑cause accuracy | ~60 % (manual) | 94 % (agent‑assisted) |
| On‑call fatigue | High (multiple pager alerts) | Reduced by ~80 % |

Corey Quinn of The Duckbill Group summed it up: “You’re paying for the privilege of having AI do what your 2 AM on‑call engineer does, except it won’t passive‑aggressively Slack the team about it afterward.”

• --

Best Practices for Replicating the Pattern

1. Enable Bedrock token metrics – Ensure InputTokenCount and OutputTokenCount are published to CloudWatch.
2. Define a baseline – Use a moving‑average or statistical model (e.g., 3‑sigma) to set alarm thresholds.
3. Start in shadow – Replay historical incidents to fine‑tune the agent’s prompts and IAM policies.
4. Separate IAM roles – Keep investigation and remediation privileges distinct.
5. Human approval gate – Use Step Functions waitForTaskToken to pause workflows until a human validates the recommendation.
6. Continuous learning loop – Store investigation outcomes in a version‑controlled knowledge base for future model fine‑tuning.

• --

Conclusion

Circle K’s adoption of Amazon Bedrock and the AWS DevOps Agent demonstrates how AI‑driven telemetry analysis can transform incident response. By turning token‑spike metrics into an autonomous investigation pipeline, the company achieved sub‑15‑minute triage, 75 % faster MTTR, and 94 % root‑cause accuracy. The approach is fully extensible to any AWS‑native stack and provides a repeatable blueprint for organizations looking to modernize their DevOps incident‑management workflow.

• --

References

• How Circle K accelerated DevOps incident triage to under 15 minutes with Amazon Bedrock – AWS Blog
• Analyze Amazon Bedrock cost spikes with AWS DevOps Agent – AWS re:Post
• Build an On‑Call Incident Triage Agent on AWS – AWS Builder Center
• Incident Investigation with AWS DevOps Agent – AWS Tech Tales (YouTube)
• AWS Announces General Availability of DevOps Agent – InfoQ
• Circle K company profile – Tavily
• The Incident Response Lifecycle – PagerDuty