How PMs Should Evaluate LLMs: A Practical Framework

The rise of generative AI and large language models (LLMs) has introduced unprecedented opportunities - and challenges - for product and engineering teams. As organizations integrate AI into their products, ensuring the quality, reliability, and functionality of these complex systems has become crucial. For product managers (PMs) and engineers tasked with building and optimizing AI-powered tools, developing a robust evaluation framework is no longer optional - it's essential.

This article distills insights from a workshop on evaluating LLMs effectively, providing a practical guide for AI PMs and technical practitioners to better understand, evaluate, and improve AI systems in production. Whether you're a seasoned AI product manager or a technical engineer looking to refine your workflows, this guide will help you tackle the fast-evolving demands of building reliable AI products.

Why Evaluations (Evals) Matter in AI Product Development

Evaluating AI systems is fundamentally different from testing traditional software. Unlike deterministic systems where outputs are predictable (e.g., 1 + 1 always equals 2), LLMs are probabilistic and non-deterministic, meaning they can produce varied outputs even when given the same input. This variability introduces unique challenges when assessing the quality and reliability of AI systems.

Key Characteristics of AI Evaluations:

1. Non-Deterministic Behavior: LLM outputs can vary, even with identical prompts, making testing less straightforward than in traditional software.
2. Agent Complexity: Many AI-powered applications rely on multi-agent systems, with each agent performing specific tasks that feed into a broader workflow. Evaluating these systems involves testing not only individual components but also their interactions.
3. Dependence on Context and Data: The performance of AI systems often hinges on the quality of the data and contextual information they are trained or provided with.

Evaluations (referred to as "evals") serve as the backbone for ensuring these systems perform reliably under real-world conditions. They help teams catch issues like hallucinations, poor contextual accuracy, and suboptimal user experiences before users encounter them.

The Five-Step Framework for Effective AI Evaluations

To bridge the gap between prototyping and production, PMs and engineers must build strong evaluation workflows. Here's a practical five-step framework outlined during the session:

1. Define the Purpose of Your Evaluation

Before diving into evals, clarify the objectives:

• What specific aspects of the LLM's performance are you trying to measure (accuracy, tone, contextual relevance, etc.)?
• Which business goals or customer needs are these evaluations tied to?

For example, you might want to evaluate:

• Whether the LLM generates friendly and engaging responses.
• The correctness of outputs when performing numerical operations.
• The system's reliability in tool-calling workflows (e.g., integrating APIs or external databases).

2. Build and Use Real-World Data Sets

Data is at the heart of effective evaluations. Collect a representative data set that mirrors real-world use cases for your product. This can include:

• Synthetic data generated by the LLM to simulate user interactions.
• Production data from actual user sessions.
• Edge cases where the model previously failed or produced undesirable results.

"Think of your data set as evolving over time", the speaker emphasized. Start small and expand as you encounter new edge cases or challenging examples. For example, in self-driving systems, teams progressively built data sets for straight roads, left turns, and left turns with pedestrians.

3. Prototype and Iterate on Prompts

The prompts you use to interact with LLMs play a critical role in shaping their outputs. PMs, especially, should take ownership of prompt design and iteration to ensure the system aligns closely with user expectations.

Using tools like prompt playgrounds can help you:

• Test prompts against multiple scenarios and variables.
• Make iterative improvements based on user feedback and evaluation results.
• Save and version your prompt templates for easy comparison.

For instance, a prompt might be updated to specify tone ("super friendly"), enforce brevity ("keep it under 500 characters"), or add business logic ("offer a discount if the user provides their email").

4. Evaluate at Scale Using Automated Systems

While manual testing or "vibe coding" might work during prototyping, it doesn't scale. To evaluate hundreds or thousands of examples, teams need automated eval workflows that leverage LLMs to judge outputs.

Three Common Types of Evals:

• LLM as a Judge: Use an LLM to classify or score outputs (e.g., "Does this text match a friendly tone?").
• Code-Based Evals: Write Python functions to validate outputs against known criteria, such as checking numerical accuracy or checking for specific phrases in a response.
• Human Annotations: Use human reviewers to label examples and validate the accuracy of automated evals.

By combining automated evals with human oversight, teams can scale evaluations while maintaining quality.

5. Continuously Improve Through Feedback Loops

Evaluations aren't a one-and-done activity. As your product evolves, so must your evaluation framework. Implement feedback loops to:

• Update data sets with examples from production where the system performs poorly.
• Refine eval prompts by incorporating user feedback and edge cases.
• Monitor changes to ensure new updates don't degrade existing functionality.

Practical Example: Evaluating an AI-Powered Trip Planner

During the session, the speaker demonstrated an AI trip planner that uses multiple agents to generate personalized itineraries. Here's how the evaluation process unfolded:

1. System Design: The trip planner comprised separate agents for handling budget constraints, local experiences, and itinerary generation, with outputs combined into a final plan.
2. Prompt Iteration: The itinerary agent's prompt was iteratively refined to make outputs shorter, friendlier, and more aligned with user preferences.
3. Production Data Testing: Synthetic data (e.g., various destinations, travel styles, budgets) was used to simulate real-world inputs and evaluate the system's reliability.
4. LLM as a Judge: Automated evals assessed whether outputs matched desired goals, such as offering a discount to users or maintaining a friendly tone.
5. Human Oversight: Human labels were used to cross-check the eval results, ensuring the LLM judge was functioning as intended.

This iterative cycle of tracing, testing, and improving highlights the collaborative nature of building and evaluating AI systems.

Key Takeaways

• AI evaluations are critical for ensuring reliability and quality in LLM-powered products due to their non-deterministic nature.
• Evals are like the new requirements document. Use them to define success criteria and acceptance tests for your product.
• Product managers play a vital role in prompt design and evaluation workflows, bridging the gap between user needs and technical implementation.
• Start small and iterate: Build curated data sets of edge cases and challenging examples, then expand as your system encounters new scenarios.
• Automated evals scale your testing, but human oversight is essential for maintaining accuracy and quality.
• Refine your evaluation framework continuously: Keep adding edge cases from production and revisiting eval prompts to align with evolving user needs.
• Collaboration is key: Engineers, PMs, and stakeholders should all work together to ensure the system meets both technical and business goals.

Conclusion

Building and evaluating AI-powered products is a complex but rewarding challenge. By implementing a structured evaluation framework, teams can navigate the unique difficulties posed by LLMs, ensure their systems perform reliably, and deliver value to users. For PMs and engineers alike, the ability to actively participate in these workflows - whether through prompt optimization, data curation, or evaluation design - will be an indispensable skill in the AI era.

Source: "Shipping AI That Works: An Evaluation Framework for PMs – Aman Khan, Arize" - AI Engineer, YouTube, Dec 26, 2025 - https://www.youtube.com/watch?v=2HNSG990Ew8

Related Blog Posts

• The Ultimate Guide to LLM Feature Development
• Reusable LLM Use Cases: Best Practices for Documentation
• How to Assess LLMs for Healthcare Applications
• How to Build Auditing Frameworks for LLM Transparency