A RAG system is not production-ready because the first demo answer looks right. Teams need a repeatable checklist that tests retrieval relevance, source coverage, freshness, citation faithfulness, access control, latency, and cost.
The evaluation set should include common questions, ambiguous questions, permission-bound records, stale documents, edge cases, and questions the system should refuse. These cases become release gates whenever prompts, chunking, models, or retrieval parameters change.
The practical goal is not perfect answers. It is visible failure modes: which sources were used, which checks passed, which risks remain, and whether the system is improving or drifting after release.