Describe an eval for a multi-turn conversation rather than a single response.

A model can answer every single message in a chat correctly and still hand a customer the wrong account by the end of it.

How to answer this, stage by stage

Nobody is grading whether you can define a rubric. They are grading whether you know a chat can be right in every sentence and wrong as a whole.

Let's learn

Thurlow is the chat inside Hallowmere, a scheduling and invoicing tool for small health clinics. It walks a brand new customer through setting up their account, which plan, how many staff, which appointment types, instead of handing them a blank form.

Before Thurlow, a new customer filled out a six page setup form alone, or booked a call with a setup specialist. The form took about forty seven minutes on average, and about one signup in six never finished it, and needed a follow up call to get going at all.

Thurlow finishes the same setup in about nine minutes, as a normal back and forth chat, and cut that drop off number by more than half.

Checked every week against a five hundred turn held out set, graded one exchange at a time, does this reply make sense next to the message right before it, Thurlow holds around ninety six percent for months.

That number is not the problem. Ninety six percent right, reply by reply, says nothing about whether the account Thurlow hands over at the end still matches what the customer actually chose along the way. A chat can be right in every sentence and wrong as a whole.

Here's what that drop looked like on one real account. Saskia Verrico runs a two person physiotherapy practice and picked the Solo plan, thirty nine dollars a month, at turn three of the setup chat. Eleven turns later, while confirming staff and billing, a backend call that provisions the actual account timed out, quietly, about once every twenty tries, and defaulted new accounts to Team. Thurlow's reply at turn fourteen confirmed a Team account, seventy nine dollars a month, twice what she'd picked, without ever saying anything had changed.

In the sample Nkiru Hartigan, who owns Thurlow's eval, pulled that week: three hundred forty onboarding chats, seventeen ended on a plan that didn't match what the customer picked at turn three. All seventeen were quietly billed at the higher price.

Once one reply contradicted another, there was no dial left. Only a switch: restate everything, or trust it whole.

At its worst, a setup chat that quietly reconfigures your account behind a reply that still says it heard you is worse than the old paper form. A form never claims to remember what you told it, so you never trust it to.

What I'd leave alone: Thurlow's one-off answers outside onboarding, like fee questions or export help. Those really are independent, and grading them as a full chat instead of a single reply would just slow the review down without catching anything new.

The lesson: a model can be right in every single sentence and still be wrong as a whole conversation. Ninety six percent, reply by reply, told the team Thurlow was fine. It never told them whether the account at the end still matched the choice made at the start.

Now here is the same thing as a story

Read the short version above when you're in the room. Read this one when you want to feel why one contradiction was enough to change how Saskia typed every message after it.

The clinic Saskia Verrico runs has exactly two treatment rooms and one screen behind the front desk. She built both by hand over six years, the kind of person who can quote a client's insurance details from memory before she's finished saying hello.

When she signed up for Hallowmere, on a Tuesday between appointments, Thurlow met her instead of a form. It asked short questions. She gave short answers. Solo plan, just her and one contractor, two appointment types, assessment and follow up.

For the first six turns, she checked every reply carefully, force of habit from years of software that never remembered anything about her the second time she opened it. By turn nine, she'd stopped. Thurlow kept the plan straight, kept her name straight, and each new question already knew what she'd said two questions back. By turn twelve, she was answering in half sentences, the way you talk to someone who's actually listening.

Then, at turn fourteen, confirming staff and billing, Thurlow wrote back: "Great, your Team account is set, seventy nine dollars a month, ready for you and your team to get started." She read it twice. She'd picked Solo. Thirty nine dollars. Turn three, twenty minutes earlier, in the same chat.

Reply by reply, nothing ever looked wrong.

The flip itself was not gradual. One contradiction, and it never went back down.

She didn't call anyone. She didn't file a ticket. She just, for every question after that, started typing the whole answer again: "Solo plan, just me and one contractor, thirty nine dollars, in case that changed too." Every message, for the rest of setup, and every message in the two follow up chats she had with Thurlow in her first month.

The extra typing cost her maybe ninety seconds across the rest of that one chat. That was never the real cost. The real cost was that a woman who'd trusted the chat completely by turn twelve now treated it, from turn fourteen on, like a form that couldn't be relied on to remember anything at all, the exact thing it was built to replace.

Nkiru Hartigan, who owns Thurlow's eval at Hallowmere, heard about it sideways. A support teammate mentioned, in a stand up, that three refund requests that month were all the same shape: a customer swearing they'd picked Solo, billed for Team, no idea why.

Nkiru pulled the last month of onboarding chats and did the thing the weekly check never did: read whole conversations instead of single replies. Three hundred forty chats that week. Seventeen ended on a plan that didn't match what the customer had picked at turn three. Every single one of those seventeen chats, graded reply by reply, scored close to perfect. Nothing in any one exchange looked wrong.

The decision that opened the door went back to a design review eight months earlier, when Thurlow still only answered one-off questions, what's a processing fee, how do I export an invoice. Someone asked whether the weekly quality check should read whole conversations or just single replies. Whole conversations would cost more to build, need a person to log what a customer had actually chosen, and take longer to grade. Single replies were fast, cheap, and at the time, genuinely enough, because no answer back then depended on anything said earlier in the same chat. Nobody revisited that call once onboarding turned into one long chat with a memory to keep.

Run the same Tuesday again, with one change. The weekly check now grades whole chats: log what the customer picks as they pick it, check the finished account against it, flag any reply that contradicts an earlier one. The same provisioning timeout happens, the same week, on about the same one chat in twenty. The check catches it in the pre ship sample, an alert reading plan mismatch, four point eight percent of sampled chats, and blocks the release before a single real customer sees turn fourteen. Seventeen wrongly billed accounts become zero.

Saskia's next chat with Thurlow, months later, gets the same half sentence answers back from her. She never rebuilt that trust from a distance again.

One design let a chat be right, sentence by sentence, while quietly getting a customer's account wrong as a whole. The other design asks the one question that actually matters: does the ending match what she said at the start.

What I'd tell myself, back in that design review: the moment a chat is allowed to remember anything across turns, ask what a broken memory looks like to someone reading the transcript one reply at a time, because it looks like nothing at all. Nobody asked. That's on the room, not on Thurlow, and not on Saskia.

FLIPS, applied to eleven turns nobody compared

Not five guesses about what could go wrong in a chat. FLIPS names exactly one habit that snaps, and asks what decision made snapping the only option.

Five steps. Only the I step is genuinely hard to find.

Three things worth stating directly, since this is where the real judgment sits. The alternative the team could have tried first was tightening the per-reply rubric itself, adding a rule telling raters to flag any answer that looks inconsistent with earlier context. It loses because a rater grading turn fourteen alone has no way to see turn three; the rubric can only judge what's in front of it, not what came nine turns earlier. The AI specific failure worth naming by name is silent state loss inside a multi-step tool call: a backend service the chat depends on failed quietly and fell back to a default, and nothing about the visible conversation showed that a fallback had happened, so the transcript read as confident and consistent right up to the wrong ending. The guardrail is the whole-chat check itself, plus one more thing: the provisioning call now has to say, out loud in the chat, whenever it falls back to a default, instead of answering as if nothing happened. That guardrail isn't free. Logging a customer's real choices and grading full transcripts by hand costs more per week than an automated reply-by-reply script, and it runs slower, a real trade accepted on purpose: production billing correctness matters more here than eval turnaround time. And the bar it enforces was never zero mismatches, a chat handling this much signup traffic can't promise that. It's a probability bar, checked against the held-out sample: the check blocks a release when plan mismatches clear one percent of sampled chats, not a promise that Thurlow never times out.

And if you want to be sure it really works, try it somewhere else

Same five letters, a city permit office, and this time the memory breaks across days instead of across turns in one sitting.

Formwell is a chat the Drummuir city planning department uses so residents can start a building permit application, save it partway through, and finish later. Doireann Stancliffe is the senior permit clerk who used to walk every applicant through intake by phone herself, and now lets Formwell take the first pass.

The case for handing it over: for its first four months, Formwell finished about seventy percent of applications without Doireann touching them at all, and the ones that did reach her were mostly complete and correctly classified.

The case against it: about a third of applicants save an application partway through and come back a day or two later to finish it. Formwell's session only ever kept the raw chat log across that gap, not the actual facts pulled from it. On resume, it re-guessed the project type from a short recap message instead of the value the applicant had already given, and the recap step leaned toward "commercial" whenever the wording was even slightly ambiguous, because most people who paused and returned days later, in Formwell's early data, were contractors filing for commercial jobs.

Nothing in the resumed chat looked broken. The fact it needed had simply never been saved as a fact.

A homeowner who'd told Formwell, on day one, that she was reroofing her own garden shed came back two days later to attach a photo. Formwell resumed the chat, re-guessed her project as commercial, and issued her a commercial permit form: a six hundred dollar higher fee and an inspection requirement no shed in her yard needed. Doireann caught it only because the fee looked wrong to her on a routine review, not because anything in the chat looked broken.

Same method, different flip. Doireann didn't drift into double checking, the way Saskia did. She'd delegated intake to Formwell completely, the way a senior clerk hands work to someone she trusts. Once she found the misclassification, she didn't start spot checking Formwell's work. She reclaimed every multi day application by hand, read start to finish herself, on top of her existing caseload. Two people, in effect, doing the one job Formwell was supposed to do alone.

Same rank as before: score full sessions, across the pause, not single replies within one sitting. Formwell's per-reply grading, sixty questions a week, sensible answer to a sensible question, stayed close to ninety four percent the entire time. A held out set of a hundred fifty simulated pause and resume sessions told a different story: eleven percent came back with a project type that didn't match what the applicant said before the pause. After Formwell started saving the extracted facts themselves, not just the transcript, that number dropped to zero.

Swap the trigger and it still runs.
Speed: an interviewer caps you at ninety seconds. Skip straight to the fix, grade the whole conversation against what the person actually chose, never a reply on its own.
Cost: there's no budget this quarter to log ground truth choices by hand for every chat. Sample it, twenty chats a week logged properly beats scoring all of them one reply at a time and catching nothing.
The model got better, for real: say Thurlow's underlying model gets upgraded to a newer version. That's not proof its memory across turns held up. A better model can write a more convincing wrong answer just as easily as a right one.

Where people run it wrong.
They read a healthy weekly reply score as proof nothing's broken, and never once open a full transcript.
They treat a session that spans a pause the same as one that doesn't, when a pause is exactly where a stored fact gets lost instead of quoted back.
They fix a customer complaint by rewording one reply, when the reply was never wrong, the account behind it was.

How to use it live. Say the split out loud before answering: "is this asking whether one answer is good, or whether the whole conversation stays honest with itself." Naming the split buys a beat to pick the right check instead of guessing out loud in front of the interviewer.

Flashcards (tap any card to flip it)

Check yourself Score: 0 / 0