Route coding agent requests to the right model on Amazon Bedrock with NVIDIA NeMo Switchyard

Routing Coding‑Agent Requests with LiteLLM and NVIDIA NeMo Switchyard on Amazon Bedrock

Developers increasingly rely on coding agents (e.g., Claude Code, OpenAI Codex) to edit code, run tests, and iterate until a task is complete. Because a single task may involve dozens of model calls, the true cost is best measured as cost per solved task—the total spend divided by the number of tasks that pass their tests. Routing each request to the most appropriate model can dramatically lower that metric.

This guide shows how to deploy an open‑source LiteLLM gateway with the Switchyard routing plugin using the AWS Cloud Development Kit (CDK). The gateway runs on Amazon ECS/Fargate, calls models hosted on Amazon Bedrock, and lets you compare routing strategies (Stage vs. Random) against single‑model baselines.

---

1. Architecture Overview

``` [ Coding Agent ] → HTTPS request (model=\"switchyard\") → [ ALB ] → [ ECS/Fargate (LiteLLM + Switchyard) ] → [ Amazon Bedrock (Claude Opus / Nemotron 3) ] → [ Response back through ALB → Agent ] ```

• LiteLLM acts as an OpenAI‑compatible gateway. It receives a request whose `model` field is set to `switchyard` and forwards it to the Switchyard plugin.
• Switchyard decides which Bedrock model to invoke based on the selected routing strategy.
• Stage routing examines recent tool activity in the request history and picks either a high‑performance model or a lower‑cost efficient model.
• Random routing distributes traffic across a weighted pool of models, useful for A/B experiments.
• All traffic stays inside the VPC via a Bedrock Runtime interface endpoint, preserving security and low latency.

Key components:

• Amazon ECS/Fargate – runs a single container image (LiteLLM + Switchyard).
• Application Load Balancer – terminates TLS (ACM certificate) and forwards only the allowed API paths.
• AWS WAF – IP reputation, bad‑input, and rate‑limit filters.
• Amazon Secrets Manager – stores the gateway key.
• CloudWatch Logs & Container Insights – observability.

---

2. Prerequisites

| Requirement | Why it matters | |-------------|----------------| | Non‑production AWS account with Bedrock access to Claude Opus 5.5 and NVIDIA Nemotron 3 Super 120B | These models are used in the demo and support tool‑calling. | | AWS CLI v2, CDK bootstrapped, Node.js 20+, Docker (BuildKit), Python 3.11+ | Required for building and deploying the CDK stack. | | Owned domain (or public Route 53 hosted zone) and ACM certificate in the target region | Enables a secure HTTPS endpoint for the gateway. |

If you only need a quick functional test, you can skip the custom domain and use the default load‑balancer DNS name with a plain‑HTTP request.

---

3. Deploying the LiteLLM + Switchyard Gateway

1. Clone the sample repository

```bash git clone https://github.com/aws-samples/sample-route-coding-agent-requests-Amazon-Bedrock-nemo-switchyard.git cd sample-route-coding-agent-requests-Amazon-Bedrock-nemo-switchyard ```

1. Define the model group

Edit `profiles/litellm.yaml` to list the Bedrock models you want to route between, e.g., `claude-opus-5.5` (high‑quality) and `nemotron-3-super-120b` (cost‑effective). In `profiles/switchyard.toml`, set `strategy = "stage"` for stage routing or `strategy = "random"` with optional `weights = {model1 = 0.7, model2 = 0.3}` for random routing.

1. Bootstrap the CDK stack

```bash npm install -g aws-cdk cdk bootstrap aws://<ACCOUNT_ID>/<REGION> ```

1. Deploy

```bash cdk deploy --require-approval never ``` The stack creates: VPC with two AZs and a NAT gateway. Private ECS task definition with the LiteLLM image (v1.102.0) and Switchyard plugin (0.3.0). ALB, ACM cert, Route 53 record, and WAF rules. Secrets Manager entry for the gateway key (prefixed with `sk-`).

1. Verify

Check the CloudWatch log group `<stack-name>/litellm` for startup messages. Use `curl` to hit the health endpoint: `https://<hostname>/health/liveliness`.

---

4. Measuring Cost per Solved Task

The sample includes a simple benchmark runner (`benchmark.py`). It sends a series of coding‑agent style prompts to the gateway, records the number of attempts, token usage, and whether the final test suite passes.

```bash python benchmark.py --endpoint https://<hostname>/v1/chat/completions \ --model switchyard --tasks ./tasks.json ```

The script outputs:

• Total spend – summed from Bedrock usage reports.
• Successful tasks – those where the returned code passes the provided tests.
• Cost per solved task – `totalspend / successfultasks`.

Run the benchmark twice:

1. Routed group – using the Switchyard‑enabled LiteLLM endpoint.
2. Baseline groups – direct calls to each Bedrock model via separate LiteLLM groups (no routing).

Compare the two cost‑per‑task numbers to see the economic impact of routing. In typical experiments, stage routing can shave 20‑40 % off the baseline cost while maintaining comparable success rates.

---

5. Operational Considerations

• Security – Keep the gateway key in Secrets Manager and rotate it regularly. Restrict ALB access to known IP ranges or VPC endpoints.
• Observability – Enable Container Insights for CPU/memory metrics and set up CloudWatch Alarms on error rates.
• Scaling – The ECS service uses Fargate auto‑scale based on request count; adjust the target CPU utilization to match your workload.
• Model limits – Bedrock quotas differ per model; monitor usage to avoid throttling.
• Non‑production disclaimer – The Switchyard integration is experimental; validate in a sandbox before any production rollout.

---

6. Next Steps

• Experiment with other strategies – Switchyard also offers classifier‑backed routing (Task, Composite). Although not supported out‑of‑the‑box in the LiteLLM plugin, you can extend the plugin or invoke a separate judge model.
• Integrate with CI/CD – Automate benchmark runs after each code change to continuously validate cost efficiency.
• Add more models – Include additional Bedrock models (e.g., Titan, Llama 2) to broaden the routing pool.
• Fine‑tune thresholds – Adjust stage‑routing heuristics (e.g., number of tool calls) to better match your agents' behavior.

By deploying LiteLLM with the Switchyard router, you gain a flexible, cost‑aware gateway that lets coding agents automatically select the most appropriate Bedrock model for each turn, ultimately reducing the price per solved task while preserving developer productivity.