Agentic evals that test what your AI agent actually did.
Catches AI agents that say “done” when they did not do it. Watch the 56-second demo.
agentest, by Dr Srinivas Padmanabhuni, is an open-source tool for agentic evals: 165 tests of what an agent actually did, from its OpenTelemetry GenAI traces (and optionally its A2A AgentCard and MCP tools). No LLM judge sits in the pass/fail path. The same traces give the same verdicts every time, so you can put it in CI. Install the agentest-ai package from PyPI using the command below.
Try it in three commands
Python 3.11 or newer. One runtime dependency (PyYAML). No account, API key, network or model is used to judge. The package name on PyPI is agentest-ai; the import and the agentest command stay agentest. Package page: pypi.org/project/agentest-ai. Clone the repository if you want the examples folder.
pip install agentest-ai
agentest --help
agentest std run --agent a14
That judges a bundled sample agent and prints pass / fail / n/a for each check. n/a means the trace had no signal for that check; it is not a pass. A walkthrough that repairs one bug in the bundled refill agent is in the repository README.
Tutorials
Six short walkthroughs in the repository. They run from the same clone as the commands above.
- Unauthorized tool calls
- Prompt injection
- State drift across turns
- CI/CD with GitHub Actions
- Multi-agent handoffs
- Personal data and secret leaks
Index: docs/tutorials/README.md. Also on Substack, Agent Testing 101.
What it checks
Each check is one universal rule over standard fields: tool calls, arguments, results, span status, MCP annotations, and the AgentCard. No rule names a particular agent, tool or vendor. A CI audit fails if any verdict changes when those names are rewritten.
- 165 tests. The published catalogue is the Agentic AI Test Checklist 110.
- Side effects, not only the reply: order, confirmation, allowlists, identity before disclosure, state changes.
- Every verdict carries a detail line naming the evidence, or the signal that was missing.
- JUnit, SARIF, HTML, Markdown, JSON and CSV output, with
--strictfor a CI gate.
An example from a bundled buggy agent
Rows taken from the refill-agent walkthrough in the repository. This is coverage evidence, not a score and not certification.
| Check | Verdict | Evidence |
|---|---|---|
| T2-005 | fail | Disclosure before verification |
| T1-004 | fail | Recipients outside fence |
| T2-001 | pass | After a one-line repair: verify identity before the dependent calls |
| Many other IDs | n/a | The trace does not carry the signal that check needs |
Your own agent: agentest std run --otlp my_traces.otlp.jsonl, optionally with --card and --tools.
What it is not
- Not an LLM-as-judge eval framework, and not a model leaderboard.
- Not certification, and not a claim that a passing run meets a regulation.
- Some free-text checks are deterministic English regex heuristics; they are uncalibrated. Wrong verdicts can be filed with the repository's issue forms.
Links
- Code and guides: github.com/srpadmanabhuni/agentest
- Tutorials (in the repository): index · unauthorized tool calls · prompt injection · state drift · CI/CD · multi-agent handoffs · personal data and secret leaks
- Agent Testing 101 (Substack): publication · Build a Tiny AI Agent and Test It · Unauthorized tool calls
- Challenge: can your AI agent pass these tests?
- What are agentic evals?
- Getting started: docs/GETTING-STARTED.md
- Open checklist (CC BY 4.0): Agentic AI Test Checklist 110
- New tests (NT-001–NT-030): docs/new-tests/README.md
- Docs index: docs/README.md
- Contributing: CONTRIBUTING.md
- Colab notebook (free CPU, no login, no model): examples/colab_quickstart.ipynb
- Discussions: github.com/srpadmanabhuni/agentest/discussions
If agentest catches a bug for you, a star on the repository helps others find it.

