Evals need to be end-2-end
Evals need to be E2E If you build AI agents, sub-agents, and agent skills. You know you need evals; that can be tricky. If you run agents inside your organization in our internal systems and services, it's pretty simple to collect evals. IF others and external companies run your skills/agents, then the story is a bit different, as you might not have control of the whole execution context. Its All under the testing umbrella There are 3 things that have a very close proximity: tests, observability and now evals. Testing includes classical pre-production and post-production tests, like unit tests, end-to-end tests, integration tests, chaos testing, stress testing, property-based testing, fuzzy testing, CSS testing, contract testing, and more. These can be done before production and in lower environWhich can be done before production and lower environments or in production. Production testing also encompasses things like canary and observability, which are forms things like canary and...