Evals need to be end-2-end
Evals need to be E2E
If you build AI agents, sub-agents, and agent skills. You know you need evals; that can be tricky. If you run agents inside your organization in our internal systems and services, it's pretty simple to collect evals. IF others and external companies run your skills/agents, then the story is a bit different, as you might not have control of the whole execution context.
Its All under the testing umbrella
There are 3 things that have a very close proximity: tests, observability and now evals. Testing includes classical pre-production and post-production tests, like unit tests, end-to-end tests, integration tests, chaos testing, stress testing, property-based testing, fuzzy testing, CSS testing, contract testing, and more. These can be done before production and in lower environWhich can be done before production and lower environments or in production. Production testing also encompasses things like canary and observability, which are forms things like canary and observability which is a form of testing.

What about evals? Can you guess it? Evals are a form of testing too. So it's all tests in the end of the day. Evals are common in the machine learning, models solutions and also in agentic engineering with skills and agents.
Evals need to be E2E
IF your skill or agent/sub-agent it's self contained your evals might be centralized and focus on the english let's say it. However if your skills/agents call CLIs and systems or even APIs or APIS behind CLI and systems, them you need evals-2-end.

Just because an API exists it does not mean was designed well and can serve agents well. Agents can always workaround which is great but can result in slowness and poor user experience.
Agents can figure by calling your solutions via browser using playwright for instance or if the application is desktop they can use your computer and do clicks. IF there is an api or CLI(which call apis) agents can use that too. However are this things optimized?
How do you know your agent is not hitting a API that ends in a monolith or distributed monolith where we are missing database index or doing the wrong access patterns i.e missing batch support. You might not able to see that from the eval alone, but if the eval it's end to end them you can correlate all things.
How to get better
As AI models blend with engineering, as harness/CLIs become more popular we need more end more traces end-2-end and evals end-2-end otherwise we will be on the dark and might not even know user is having a poor experience.
To do better we can:
- Make sure all agents, sub-agents, custom commands and skills have evals.
- Make sure evals are connected with observability of apis and systems in production via traces.
- Make sure your agents/skills are being tested.
- Make sure when you do LLM as judge you eval your evals as well :-)
- You might only write the skill but the user will have the whole experience so test the whole system end-2-end not only your skill in order to feel the pain the user might feel.
Cheers,
Diego Pacheco
