AWS sample uses NVIDIA NeMo Switchyard to pick Bedrock models for coding agents

AWS published a sample on 13 November 2026 that deploys the LiteLLM gateway with the NVIDIA NeMo Switchyard routing plugin on Amazon ECS. Coding agents can then have each request sent to one of two Amazon Bedrock models, and the sample measures cost per solved task.

What happened

In a post on the AWS Machine Learning Blog, AWS authors describe a CDK-deployed gateway that sits between a coding agent and Bedrock. The agent sends OpenAI-compatible requests with the model field set to switchyard. LiteLLM asks the Switchyard plugin which model should answer, then calls that model through the Bedrock Converse or ConverseStream API.

The post's central argument is about measurement. A coding agent works in tasks that may involve many model calls, so a low per-token price does not tell you what a finished task costs. AWS proposes judging routing by cost per solved task: total cost of the runs divided by the number of runs whose tests pass. Failed and retried attempts count toward the cost. For more on this framing, see OpenBeat's earlier coverage of measuring Bedrock agent cost per task and how request settings change inference spend.

Key details

• Routing strategies: Stage routing reads recent tool activity and results in the request history and picks one of exactly two candidates, a capable model and a cheaper efficient one. Random routing picks from at least two models with optional weights, intended for traffic splits and A/B comparisons. Neither calls a judge model.
• Not supported: Classifier-backed strategies such as Task and Composite routing need intermediate model calls, which this integration does not support.
• Agent requirement: Because Stage reads the history, the agent must send the full conversation history.
• Model pair: The prerequisites name Claude Opus 5.5 and NVIDIA Nemotron 3 Super 120B on Bedrock, tested in US East (N. Virginia). Other Regions need a check of model availability and tool-calling support.
• Versions: The sample installs switchyard-server 0.3.0 from crates.io and runs the LiteLLM v1.102.0 image.
• Alternative: The post says LiteLLM has also shipped a native Switchyard stage router as an auto-router strategy.

How the deployment is laid out

LiteLLM and the plugin run in one container on ECS with AWS Fargate in private subnets. An Application Load Balancer with an ACM certificate fronts it, AWS WAF applies IP reputation, known-bad-input and per-IP rate-limit rules, and only an IPv4 CIDR range you allow can reach it. The load balancer forwards just /v1/chat/completions, /v1/models and /health/liveliness and returns 404 for other paths, which keeps the LiteLLM admin UI and key management routes unreachable.

Inference goes through a Bedrock Runtime interface endpoint in the VPC, using the task's IAM role. The x-litellm-model-name response header shows which model answered, while the response body's model field stays switchyard. Logs go to a CloudWatch log group with 30-day retention.

Prerequisites

• A non-production AWS account with Bedrock access to the two models, bootstrapped for the AWS CDK.
• AWS CLI v2 with credentials and an explicit Region.
• Node.js 20 or later, npm, Docker with BuildKit, and Python 3.11 or later.
• A hostname you own, with a public Route 53 hosted zone or an ACM certificate in the deployment Region. A short plain-HTTP test needs neither.

Try it

The sample is cloned from the aws-samples GitHub repository named in the post. Its profiles pair litellm.yaml with switchyard.toml. In profiles/stage/litellm.yaml, both candidates share one model_name, so the client only ever asks for switchyard:

model_list:
  - model_name: switchyard
    litellm_params:
      model: bedrock/converse/global.anthropic.claude-opus-5-5
      aws_region_name: os.environ/AWS_REGION
  - model_name: switchyard
    litellm_params:
      model: bedrock/converse/nvidia.nemotron-super-3-120b
      aws_region_name: os.environ/AWS_REGION

List the capable model first and the efficient one second, since Stage infers roles from that order. The same file defines two single-model baseline groups, claude-opus-5-5 and nemotron-super-3-120b, so the sample's benchmark runners can compare the routed group against each model alone in the same deployment.

Why it matters

Routing only pays off if cheaper-model calls do not cause extra retries or unfinished tasks. The sample's baseline groups let you test that against your own workloads rather than relying on list prices. The pass rate that goes into the cost-per-solved-task formula comes from your own test suite.

The caveats are AWS's own. NVIDIA documents switchyard-server 0.3.0 as a demo and evaluation component, the LiteLLM integration is experimental, and the configuration may change between releases. AWS says the code is for non-production use and should be deployed in a non-production account after security and legal review.