12 November 2026 · 4 min read
AWS Open-Sources UAEF for Cross-Framework AI Agent Evaluation
AWS released UAEF under Apache 2.0, a framework-agnostic evaluation system with 36 metrics, hardened judge, and regression detection for AI agents.
What was announced
On 11 November 2026, Amazon Web Services open-sourced the Universal Agent Evaluation Framework (UAEF) under the Apache 2.0 licence [1][2]. The code is available on GitHub in the awslabs/uaef repository. UAEF provides a single set of metrics and a shared scoring engine that works across multiple agent frameworks, addressing a long-standing fragmentation in how teams measure agent performance.
Why cross-framework evaluation matters
Teams building AI agents often use different frameworks—LangGraph, LangChain, Strands Agents, Amazon Bedrock AgentCore—and each records execution traces in its own format [1]. Without a common evaluation layer, comparing an agent built on one framework with another, or even tracking whether a prompt change improved an agent, requires custom glue code for every combination. UAEF solves this by defining a canonical AgentTrace format and providing adapters that convert native traces from each framework into that format [1][2]. The adapters listed in the README cover LangGraph, LangChain, Strands Agents, Amazon Bedrock Agents, Amazon Bedrock AgentCore, and a generic JSON adapter for custom frameworks [1][2].
36 metrics across seven dimensions
UAEF ships with 36 built-in metrics grouped into seven dimensions: Response Quality, Tool Calling, Multi-Turn, Reasoning, Responsible AI, Multi-Agent, and Performance [1][2]. Each metric returns a score between 0 and 1.
Metrics fall into two scoring modes. Deterministic metrics compute scores directly from the trace and any supplied ground truth using code, so they produce identical results on every run. Judge metrics use a model on Amazon Bedrock to assess subjective qualities like relevance, safety, and faithfulness [1]. Judge scores can vary slightly between runs, and research cited by AWS shows that even consistent judges can exhibit systematic biases, such as favouring markdown formatting over plain text [1]. The UAEF documentation therefore treats judge scores as evaluation signal, not as a safety decision, and recommends Amazon Bedrock Guardrails for runtime enforcement [1][2].
Hardening the judge against prompt injection
A distinctive feature of UAEF is its approach to the AI model that grades agents. Because an agent might pick up adversarial instructions from web pages or documents and carry them into its output, a naive judge could be steered into awarding an inflated score. AWS notes that malicious prompt injection payloads in web content rose by 32% between November 2025 and February 2026, according to Google [1].
UAEF’s judge pipeline applies several controls: the scoring rubric is placed in the model’s system field while the user turn holds only the content being evaluated; untrusted content is escaped; responses are parsed strictly; and the judge must quote a verbatim span of the agent’s output as evidence for each score [1][2]. UAEF verifies that the quoted span exists in the agent’s output and rejects scores without valid evidence. The design reduces the risk of injection corrupting evaluation results, though the project’s security notes acknowledge that no delimiter is unforgeable against every model [1].
Regression detection and experiment storage
UAEF persists evaluation results as experiments. Metadata such as run counts and per-metric averages is stored in Amazon DynamoDB, while full JSON results and judge reasoning go to Amazon S3 [1][2]. When a new run completes, it is compared against a stored baseline. Changes below a configurable threshold—5% by default—are treated as noise. Larger drops are graded as minor (5–10%), moderate (10–20%), or severe (over 20%) and are grouped by dimension, so a team can see that, for example, Tool Calling held steady while Performance degraded [1].
The web interface shows per-metric and per-dimension charts, and an optional statistical report provides confidence intervals and outlier detection without another model call. An optional insights module can use a Bedrock model to suggest root causes [1][2].
Two ways to run UAEF
UAEF can be used as a Python library inside existing CI/CD pipelines or deployed as an AWS service with a low-code web interface. Both share the same evaluation engine and storage [1]. Datasets are supplied as CSV or XLSX files; only a Question column is required, with optional columns for ground-truth answers, context, expected tool calls, and session and turn IDs for multi-turn tests [1]. A PII screening step detects email addresses, phone numbers, and credit card numbers in traces before storage, with optional redaction [1][2].
Where UAEF sits in AWS’s evaluation landscape
AWS now offers several evaluation tools at different stages of development. Agent-EvalKit, introduced in June, works inside a developer’s AI coding assistant. Amazon Bedrock AgentCore Evaluations, generally available since 31 March, is a managed service with 13 built-in evaluators. In July, AWS released aws-bench, which tests how accurately agents complete real AWS tasks in isolated accounts [1]. UAEF complements these as a self-deployed, open-source option with adapters for multiple frameworks, baseline comparison, and graded regression detection built in.
What this means for developers
For teams running agents across different frameworks, UAEF provides a consistent scoring baseline that can survive changes in prompts, models, or tools. The framework adapters remove the need to write custom evaluation glue for each stack. The hardened judge pipeline lowers the risk that adversarial content in agent outputs will skew evaluation results, though developers should still treat judge scores as one signal among several and enforce safety policies separately. The regression detection with severity tiers gives teams a structured way to catch performance drops before they reach production. Because judge metrics require Amazon Bedrock, teams not using Bedrock can still use the library for trace normalisation and deterministic scoring [1]. Multi-tenant deployments require separate AWS stacks for each tenant, as UAEF does not provide tenant isolation itself [2].
Sources
- AWS Open-Sources AI Agent Evaluation Framework That Hardens Its AI Grader Against Prompt Injection
- AWS Details UAEF for Cross-Framework AI Agent Evaluation
- aws
- uaef
- ai-agents
- evaluation
- open-source
Researched and drafted with AI from the sources above.
Learn to build with AI
Hands-on tutorials on Cursor, Claude Code, AI agents and more.