Three days after deploying a conversational agent, the phone call came. Wrong figures. Two business decisions already made on bad data. The agent had been returning incorrect numbers silently, confidently, for 48 hours. The system had no way to know it was wrong. This is the gap evaluation closes: Most teams shipping AI agents can tell you if their system ran. Very few can tell you if it ran correctly. This tutorial fixes that. We build a complete agent evaluation pipeline in Python with FastAPI, grounded in the observability foundations (logging and tracing) that make rigorous evaluation possible in the first place. Everything we build is open-source and self-hostable. No expensive SaaS platforms, no dollar-denominated subscriptions. What you will learn: By the end of this session, attendees will be able to: - Design an evaluation strategy for non-deterministic AI agents, choosing the right approach for each failure mode they need to catch - Implement rule-based evals that run automatically on every agent response, catching regressions before users do - Build an LLM-as-judge pipeline that uses a second model to score agent outputs for correctness, relevance, and quality at scale - Collect structured human feedback and integrate it into a continuous prompt optimisation loop, the practical alternative to fine-tuning - Instrument agents with structured logging and distributed tracing using OpenTelemetry, creating the data foundation that makes evaluation possible - Interpret evaluation results to diagnose root causes, separating prompt issues from tool-call failures from model limitations Why does evaluation matter more than monitoring? Logging tells you what happened. Tracing tells you how it happened. Neither tells you whether it was right. For AI agents, that last question is the only one that matters in production. We treat evaluation as the central engineering problem of shipping AI agents and cover three complementary approaches that work together: - Rule-based evals: deterministic checks (format validation, factual grounding against known data, output constraints) that run on every response - LLM-as-judge: automated scoring using a second model to assess quality, relevance, and correctness on dimensions that rules cannot capture - Human feedback collection: structured capture of real user signals that closes the loop and feeds prompt iteration Together, these form the evaluation loop every production AI system needs: observe, evaluate, improve, repeat. Key Takeaways Attendees leave with: 1. A working evaluation pipeline they can drop into any FastAPI-based AI agent 2. A decision framework for choosing between rule-based, LLM-as-judge, and human evaluation approaches 3. A prompt optimisation workflow driven by evaluation data, not guesswork 4. An open-source observability stack (structured logging plus OpenTelemetry tracing) configured for AI workloads 5. Patterns for debugging non-deterministic agent behaviour using evaluation data 6. A self-hostable architecture that scales without recurring SaaS costs Workshop outline (90 minutes) - The silent failure: why agents need evaluation, not just monitoring (5 min) - Observability foundations: structured logging for AI agents (10 min) - OpenTelemetry tracing for agent tool chains (10 min) - Rule-based evals: design and live implementation (15 min) - LLM-as-judge: building automated scoring (15 min) - Human feedback collection and the prompt optimisation loop (10 min) - Guided exercises: attendees instrument and evaluate their own agent (20 min) - Q&A and wrap-up (5 min) Prerequisites: Basic Python, familiarity with FastAPI or any web framework, some exposure to LLMs or AI APIs. Bring a laptop with Python 3.10+ installed.