How Circle K accelerated DevOps incident triage to under 15 minutes with Amazon Bedrock

How Circle K Cut Incident Triage to Under 15 Minutes with Amazon Bedrock

Circle K runs a massive, cloud‑native retail platform that supports point‑of‑sale, loyalty, and EV‑charging services across 17,200 stores in 27 countries. As the number of services grew, the DevOps team found that each alert triggered a manual, 45‑60 minute investigation: pulling CloudWatch logs, querying OpenSearch, checking ECS task status, and hunting through GitHub runbooks. The repetitive steps slowed remediation and varied with the on‑call engineer’s familiarity.

To eliminate this bottleneck, Circle K built an automated triage system on Amazon Bedrock. By leveraging Bedrock’s managed foundation models, tool orchestration, and Retrieval‑Augmented Generation (RAG) knowledge bases, the team created an agentic workflow that gathers all relevant context and delivers a structured summary to Slack within minutes. The result: mean triage time dropped to under 15 minutes, a 70 %+ reduction.

---

1. Why Bedrock Was the Right Fit

• Managed model hosting – No need to run self‑hosted LLMs or maintain versioning pipelines.
• Unified API – Foundation models can be swapped without code changes, simplifying experimentation.
• Native AWS integration – Bedrock works with existing IAM, VPC, and logging setups, and its agents can call AWS services directly.
• Built‑in knowledge bases – The RAG feature stores internal product cards and runbooks, removing the need for a separate vector store.
• Data privacy guarantees – Bedrock does not share customer data with model providers, meeting Circle K’s compliance requirements for operational logs.

These capabilities let the DevOps team focus on prompt design and tool definitions rather than infrastructure overhead.

---

2. Architecture Overview

2.1 Initial Implementation with Bedrock Flows

1. Alert ingestion – Grafana pushes alerts to Amazon SNS, which forwards them to an SQS queue.
2. Lambda trigger – A Lambda function reads the payload and starts a Bedrock Flow.
3. Parallel agents

AWSResourceExplorerAgent – Queries ECS task status, ALB health, and CloudWatch metrics across multiple accounts. OpenSearchExplorerAgent – Retrieves recent logs and distributed traces from Amazon OpenSearch Service, using the Bedrock Knowledge Base for service‑specific context.

1. GitHubExplorerAgent – After the parallel agents finish, this agent searches the relevant GitHub repositories (code, IaC, recent commits) to pinpoint the root cause and suggest a fix.
2. AnalysisPrompt – A single model call that synthesizes the three agents’ outputs into a concise, actionable summary.
3. Slack notification – The final summary is posted to the on‑call channel, ready for immediate action.

2.2 Migration to Bedrock AgentCore

While Flows provided quick proof‑of‑concept, scaling required tighter control over state and tool usage. The team migrated to AgentCore, which offers:

• Persistent session memory for multi‑step reasoning.
• Fine‑grained tool permissions, reducing the surface area for accidental API calls.
• Better observability through built‑in tracing and logging.

The core workflow remained the same, but AgentCore’s modular agents now share a common context store, enabling smoother hand‑offs and lower latency.

---

3. Key Lessons Learned

| Lesson | Takeaway | |--------|----------| | Start with a narrow use‑case | Circle K focused on the most frequent alert types (ECS restarts, latency spikes). This limited scope made prompt engineering tractable and delivered early ROI. | Leverage RAG for internal knowledge | Storing service cards and runbooks in Bedrock Knowledge Bases eliminated the need for a separate vector DB and kept prompts short. | Parallelize independent data sources | Splitting log retrieval from resource health checks kept each agent’s context window within model limits and reduced overall latency. | Guard data with IAM and VPC endpoints | Restricting Bedrock’s access to specific AWS resources via IAM roles and VPC endpoints ensured compliance without sacrificing performance. | Iterate on prompts, not code | Most performance gains came from refining prompts (e.g., asking the model to prioritize recent error messages) rather than rewriting Lambda logic.

---

4. Practical Takeaways for Your Team

1. Identify repetitive triage steps – Map the manual workflow and isolate data sources that can be queried programmatically.
2. Prototype with Bedrock Flows – Use the visual Flow designer to stitch together AWS SDK calls and model prompts quickly.
3. Introduce a knowledge base early – Ingest runbooks, service diagrams, and error‑code mappings to give the model domain‑specific context.
4. Measure latency and cost – Track end‑to‑end processing time and Bedrock usage (tokens per incident) to fine‑tune model selection.
5. Plan for migration to AgentCore – Once the prototype proves value, move to AgentCore for better state management and observability.

By following Circle K’s approach—starting small, leveraging Bedrock’s managed tools, and iterating on prompts—organizations can dramatically shrink incident triage cycles, free engineers to focus on remediation, and improve overall service reliability.