U2xAI
…
Quality, Cost & Token Economics

Quality metrics: accuracy vs usefulness vs trust Interview Questions

25 questions. All 25 carry a written answer.

  1. #1Define accuracy, usefulness and trust as three distinct measurable properties.ConceptFoundational
  2. #2Give an example of an output that is accurate but not useful.ConceptFoundational
  3. #3Give an example of a product that is useful despite being frequently wrong.ConceptIntermediate
  4. #4How would you measure trust in an AI feature?CaseAdvanced
  5. #5Explain why improving accuracy can decrease trust.ConceptAdvanced
  6. #6Describe the calibration problem: what happens when confidence does not match correctness?ConceptAdvanced
  7. #7How do you measure whether users over-trust your AI feature?CaseAdvanced
  8. #8What is automation bias and what product metric would surface it?ConceptAdvanced
  9. #9Design a survey instrument to measure perceived reliability.Artifact critiqueIntermediate
  10. #10Explain how a single high-profile failure affects trust disproportionately.ConceptIntermediate
  11. #11How do citations affect measured trust, and does that hold if the citations are wrong?CaseAdvanced
  12. #12What is the relationship between latency and perceived quality?ConceptIntermediate
  13. #13Describe how you would measure usefulness for a feature with no obvious ground truth.CaseAdvanced
  14. #14Explain the difference between helpfulness and harmlessness as product properties.ConceptIntermediate
  15. #15How would you decide between a model that is right 90 percent of the time and one that is right 85 percent but says when it is unsure?CaseAdvanced
  16. #16What metric captures a model's willingness to say it does not know?ConceptAdvanced
  17. #17Describe how you would measure quality for a creative generation feature.CaseAdvanced
  18. #18Explain how quality perception differs between novice and expert users.ConceptAdvanced
  19. #19Critique a quality report that presents a single accuracy number.Artifact critiqueIntermediate
  20. #20How does the severity distribution of errors matter more than the error rate?ConceptAdvanced
  21. #21What is the product implication of a model that fails confidently?ConceptIntermediate
  22. #22How do you build a trust recovery plan after a public quality incident?CaseAdvanced
  23. #23Describe the metric you would use to compare two models on the same task from a user's point of view.ConceptAdvanced
  24. #24Explain when you would deliberately reduce accuracy to increase usefulness.CaseAdvanced
  25. #25Argue that trust is the only quality metric that matters, then argue against it.InterviewAdvanced