AWS details UAEF, a framework-agnostic way to evaluate AI agents
AWS published UAEF on 11 November, an evaluation system that converts traces from several agent frameworks into one format and scores them with 36 built-in metrics across 7 dimensions. AWS says 12 organizations used it.
If you run agents on more than one framework, you can score them on one scale and compare versions before release. You would not need to rebuild metrics for each framework's trace format.
AWS published the Universal Agent Evaluation Framework (UAEF) on its Machine Learning Blog on 11 November. UAEF converts agent traces from different frameworks into one format and scores them with 36 built-in metrics across 7 dimensions.
What happened
In its own blog post, AWS describes the problem. Each agent framework records execution differently, so evaluation metrics and result storage often get tied to one trace format. Teams rebuild them when they switch frameworks.
AWS says customers at 12 organizations in telecom, healthcare and financial services used UAEF to evaluate agents and compare versions before release.
Key details
- Adapters: they read an agent's messages, tool calls and agent-to-agent handoffs. They convert them into one canonical format called AgentTrace.
- Supported setups: AWS names LangGraph, LangChain, Strands Agents and Amazon Bedrock AgentCore. Agents can run locally, behind an HTTP endpoint or on a managed runtime.
- PII screening: UAEF screens traces for personally identifiable information before data is stored.
- Metrics: 36 built-in metrics cover tool calling, response quality, responsible AI, performance, multi-turn conversation, multi-agent coordination and reasoning. Each returns a score from 0 to 1, where 1 is best.
- Two metric types: deterministic metrics reproduce exactly on repeated runs. LLM-as-a-judge metrics use a judge model on Amazon Bedrock. Custom metrics and RAGAS and DeepEval integrations are also supported.
- Storage: calling evaluate() or batch_evaluate() with persist=True creates an experiment. Amazon DynamoDB holds per-experiment metadata, and Amazon S3 holds the full results.
- Baselines: a run can be marked as the baseline for its experiment, so versions can be compared to detect regressions.
A dataset is the second input. Ground-truth answers are optional. Tool-calling metrics need expected tool calls, including order and arguments.
AWS also says judge scores are an evaluation signal, not a safety decision. It advises using Amazon Bedrock Guardrails for runtime enforcement.
Why it matters
Teams often cannot tell whether a prompt, model or tool change improved an agent. UAEF aims to answer that with the same metrics and stored baselines across versions.
The claims here come from AWS's own post. The source shows no independent test of UAEF. The post also says it points to runnable notebooks for each supported framework. The excerpt does not show them.
For related AWS agent work, see AWS's four layers of context engineering in Deep Insight and the serverless MCP server for data lake access.
FAQ
What is UAEF? UAEF is AWS's Universal Agent Evaluation Framework. It normalizes agent traces from different frameworks and scores them with built-in metrics.
Which frameworks does UAEF work with? AWS names LangGraph, LangChain, Strands Agents and Amazon Bedrock AgentCore. It also supports local agents, HTTP endpoints and traces in Langfuse.
Do I need ground-truth data? No. Ground truth is optional. Only metrics that compare against a reference use it, and tool-calling checks need expected tool calls.
Checked claims
| Claim | Status | Evidence |
|---|---|---|
| UAEF uses adapters to convert framework traces into a common format and scores them with 36 built-in metrics across 7 dimensions. | confirmed | it uses adapters to convert traces from each framework into a common format, then scores those traces with 36 built-in metrics across 7 dimensions. |
| AWS says customers across 12 organizations in telecom, healthcare and financial services used UAEF. | confirmed | Customers across 12 organizations in the telecom, healthcare, and financial services industries used UAEF to evaluate their AI agents |
| Results are stored in Amazon DynamoDB and Amazon S3. | confirmed | It stores the results in Amazon DynamoDB and Amazon Simple Storage Service (Amazon S3) |
| Every metric returns a normalized score from 0 to 1, where 1 is best. | confirmed | every metric returns a normalized score in the interval [0, 1], where 1 is best |
| AWS says judge scores should not gate a security control. | confirmed | Judge scores are evaluation signal, not a safety decision, and should not gate a security control. |
| Ground truth is optional and used only by metrics that compare against a reference. | confirmed | Ground truth is optional, and only metrics that compare the agent’s response against a reference use it. |
Found a mistake? Use Report above; an editor reads every one. Editorial policy
Question for readersHow do you currently check whether a prompt, model or tool change made your agent better, and what would you want from a shared evaluation layer?
Discussion · 0
No comments yet. Start the discussion.
Sign in to join the discussion.