The agent evaluation playbook: Built-in, DeepEval, and AutoEval on Amazon Bedrock AgentCore Evaluations

The Agent Evaluation Playbook: Built‑in, DeepEval, and AutoEval on Amazon Bedrock AgentCore

When an AI agent moves from prototype to production, the real question isn’t whether it runs—it’s whether it consistently delivers the business outcomes you need. Surface signals like latency or occasional user feedback miss the deeper behaviors that determine success: task completion, compliance, tool usage, safety, and more. Amazon Bedrock AgentCore provides a structured way to evaluate those behaviors using three tiers of evaluators—built‑in, DeepEval, and AutoEval—plus the ability to create custom judges.

---

1. Start with Business Outcomes, Not Metric Names

The most common pitfall is to pick a metric because it sounds good (e.g., “Helpfulness”) and then wonder why the scores don’t translate into real value. Instead, follow a three‑step mental model:

1. Define success in business terms.

Customer‑support agent: first‑contact resolution. Research agent: a complete, well‑sourced answer grounded in retrieved documents. Content‑moderation assistant: flag every violating item and nothing else.

1. Break the outcome into observable behaviors.

For first‑contact resolution, you need goal completion, coherent multi‑turn dialogue, compliance with data policies, and correct tool invocations.

1. Map each behavior to an evaluator at the appropriate level.

Trace‑level metrics capture per‑turn or per‑tool‑call actions; session‑level metrics capture outcomes that only make sense when the whole conversation is considered.

This mapping ensures that every score you collect directly reflects a behavior that matters to the business.

---

2. The Evaluator Catalog

AgentCore bundles three sources of evaluators:

| Source | What It Offers | Typical Use Cases | |--------|----------------|-------------------| | Built‑in | 13 AWS‑authored evaluators (e.g., `Builtin.GoalSuccessRate`, `Builtin.Helpfulness`, `Builtin.ToolSelectionAccuracy`). | Quick start, high reliability, common dimensions of response quality, tool use, safety, and trajectory checks. | | DeepEval | Managed open‑source library with metrics for task completion, conversational quality, tool use, and safety. | Multi‑turn agents where knowledge retention, conversation completeness, or bias detection are critical. | | AutoEval | Specialized signals such as security injection detection, over‑cautious refusals, and humor quality. | Agents that interact with external systems, need strict security guards, or have a brand‑specific personality. |

If a built‑in or DeepEval metric already covers a behavior, use it before building a custom evaluator. You can retrieve the full list with the `ListEvaluators` API, which returns both AWS‑provided and any custom evaluators you have defined.

---

3. Choosing the Right Level: Trace vs. Session

• Trace‑level evaluators run on each turn or each tool call. They are ideal for fine‑grained checks such as `ToolSelectionAccuracy` (did the agent pick the correct tool?) or `TurnRelevancy` (is the response relevant to the immediate user input?).
• Session‑level evaluators look at the entire conversation. Use them for outcomes like `GoalSuccessRate` (was the user’s overall goal met?) or `ConversationCompleteness` (did the agent address all user requests across the session?).

Mixing both levels gives a holistic view: trace metrics surface early failures, while session metrics confirm end‑to‑end success.

---

4. Extending with Custom Evaluators

When the catalog lacks a precise signal, you can create a custom LLM‑as‑a‑judge:

1. Select a base evaluator (built‑in or DeepEval) that most closely matches the desired behavior.
2. Provide a custom prompt or rubric that reflects your domain‑specific criteria.
3. Optionally swap the underlying model for a more specialized or cost‑effective judge.

The custom evaluator is registered via the AgentCore API and then used exactly like any other evaluator, preserving the same trace/session semantics.

---

5. Putting It All Together: A Sample Workflow

1. Identify the business outcome – e.g., “resolve a refund request in a single interaction.”
2. List observable behaviors – goal completion, correct tool usage (refund API), compliance with data‑privacy policy, professional tone.
3. Select evaluators –

- `Builtin.GoalSuccessRate` (session level) for overall success. - `Builtin.ToolSelectionAccuracy` & `Builtin.ToolParameterAccuracy` (trace level) for tool correctness. - `Builtin.Harmfulness` (session level) to ensure no PII leakage. - `DeepEval.TurnRelevancy` (trace level) for conversational relevance.

1. Run evaluations on a batch of production logs via the AgentCore `EvaluateAgent` API.
2. Analyze scores – flag sessions where any metric falls below a threshold, drill down to the offending trace, and iterate on prompt or tool‑selection logic.

---

6. Practical Tips

• Start simple. Deploy a handful of built‑in metrics first; they are fully managed and require no extra infrastructure.
• Layer depth gradually. Add DeepEval metrics for multi‑turn nuance, then AutoEval for security or personality checks.
• Automate alerts. Hook evaluation results into CloudWatch or an internal dashboard to surface regressions in real time.
• Version evaluators. Treat evaluator configurations as code—store prompts, rubrics, and model selections in version control so you can roll back if a change degrades performance.

---

Closing Thoughts

Effective agent evaluation is less about collecting a long list of scores and more about aligning each score with a concrete business behavior. Amazon Bedrock AgentCore gives you a ready‑made hierarchy of evaluators—built‑in, DeepEval, AutoEval—and the flexibility to craft custom judges when needed. By starting with clear outcomes, breaking them into observable actions, and mapping those actions to the appropriate trace‑ or session‑level metrics, you can turn raw numbers into actionable insight and keep your agents reliably delivering value at scale.