Describe how you would set a quality bar for a feature whose output is free text.

Trailscript is Thornwyck Travel's engine for writing a full day by day trip plan in plain text: restaurants, hours, how to get between stops, all of it. Griselda Vellacott owns its quality bar. Five months in, the app's own dashboard still said 4.1 out of 5. Then Hepzibah Niemczyk, three weeks into the support desk, read a complaint ticket out loud and asked a question nobody on the team could answer.

How to answer this, stage by stage

Nobody is grading whether you can name three things a quality bar might check. They're grading whether you know which one a traveler actually can't forgive.

Let's learn

Trailscript writes a full trip plan in plain text: a named restaurant for lunch, a train time for the day trip, a walking route between the two, five or six days at a stretch.

One itinerary reads like one piece of writing. It's actually five different kinds of claim stacked together, and only some of them are facts you can check.

Before Trailscript, a traveler put together a five day trip herself: a dozen browser tabs, a guidebook, a phone call or two to check a restaurant was still open. It took most of an evening. Trailscript writes the same plan in about ninety seconds, close to nine hundred of them a day now, about a hundred and thirty thousand across its first five months live.

Complaint tickets tagged "itinerary was wrong" came in at about the same rate the whole time, 412 of them over those five months. Split by hand, 239 were factual, a closed venue, wrong hours, a train that doesn't run. 111 were completeness gaps, a skipped day, a stated request the plan ignored. 62 were style complaints, clunky phrasing, a cold or repetitive tone.

Here's the turn. The overall number never moved much, so nobody went looking. What moved was which kind of complaint made up the total, and that shift was invisible from the dashboard alone.

Trailscript launched validated in 40 cities with data someone had actually checked. Three hundred more joined on a faster, rougher feed, and the writing never got any less confident about it.

What it costs at its worst: a traveler stops trusting the plan and starts re-checking every line herself, which is exactly the evening of work Trailscript was supposed to save her. Multiply that by three hundred cities where the underlying data was never as solid as the forty it launched on, and the product quietly gets worse in exactly the cities it was built to help most.

What I would leave alone: Trailscript's short one day city breaks in the original forty flagship cities. Coverage there is dense and checked often, so the accuracy risk barely exists. Building the same confidence gate machinery for those itineraries would just slow down cases that were never actually broken.

The lesson: a quality bar that gives one number can't tell you which promise it's actually keeping. Split the score before you trust it, or you find out which promise broke only after a traveler already has.

Now here is the same thing as a story

Read the short version above when you're in the room. Read this one when you want to feel why a closed restaurant, not a clunky sentence, was the thing that could have cost Thornwyck the trip after next.

Griselda Vellacott built Trailscript's first version in forty cities, the ones Thornwyck already had years of curated data on: real booking pages, real transit timetables, restaurant hours somebody had actually called and confirmed.

For the first three months, that carefulness showed. A traveler typed in five days in Lisbon and got back a plan with a named lunch spot for day two, a train time for the day trip to Sintra, a walking route between the two. Complaints trickled in, the odd typo, a museum closed on a Monday nobody had flagged. Nothing that shook anyone's confidence in the product.

Then Thornwyck asked Trailscript to cover everywhere Thornwyck sold trips, not just the forty flagship cities. Three hundred more went live over six weeks. The new cities didn't come with the same kind of data behind them. Trailscript pulled from a faster, rougher feed instead, one that kept listing a restaurant as open for months after it had actually closed.

Nobody decided this was risky on purpose. On paper, the rollout looked like more of the same good thing.

The complaint tickets kept arriving at roughly the same rate they always had, so the dashboard's blended score barely moved, 4.1 out of 5, week after week. What the dashboard couldn't show was what kind of complaint was landing. Factual complaints, a place that was shut, a train that didn't run, made up about a third of tickets in week two. By week twenty, they were well over half.

Hepzibah Niemczyk had been on the support desk three weeks. Reading through a batch of recent tickets during onboarding, she found one where a traveler had walked twenty minutes to a restaurant Trailscript called the best table in the neighborhood, only to find it closed, permanently, for two years. She brought it to Griselda with a plain question. "How do we know the places Trailscript names are actually still open?"

Two things that were both true at the same moment, and only one of them was useful to Griselda.

Nobody on the team had a real answer. Not because nobody cared. Because the only score Trailscript had ever kept, one to five stars, blended by a single reviewer, couldn't separate a wrong fact from a clunky sentence. Both just pulled the same number down a little.

Griselda pulled ninety days of tickets and split them by hand into three piles: wrong sentence, missing request, wrong fact. The wrong-fact pile hurt the most, and not by a little. Itineraries with a factual complaint scored 1.8 out of 5 on the follow-up survey. Style-only complaints still scored 3.9, mildly annoying, forgotten by the next trip. Forty four percent of travelers who hit a wrong fact didn't book their next trip through Thornwyck within ninety days. Only seven percent of the style-only group did the same.

That gap was the real finding, not the closed restaurant by itself. A clunky itinerary is a rough first draft. A wrong fact is a promise Trailscript can't take back once someone's already walked there.

The decision Griselda would take back sat right at the start, before Trailscript covered a single new city. The team built one dashboard score because it was faster to read and simpler to build than three. That was a fine call for forty cities Thornwyck already knew cold. It stopped being fine the moment coverage grew faster than the data backing it, and nobody had a way to see the gap opening until a new hire happened to read the right ticket.

Run the rollout again, with three scores from day one instead of one. The same three hundred new cities go live over the same six weeks. This time, the accuracy score drops on its own, fast, days after the rougher feed starts feeding in, not twenty weeks later when a new hire stumbles onto it by accident. Griselda pulls the accuracy number in week nine, not week twenty, and holds the newest cities to category-level suggestions until the feed catches up.

One design let a wrong fact hide behind a decent-looking average for five months. The other would have caught it in nine weeks.

What I'd tell myself, back when we shipped one blended score because it was simpler: simple was the right call for the size of team we were then. It just wasn't a call built to survive the day we started covering three hundred more cities than anyone had actually checked.

ORDER: what a free text bar has to protect first

Not a way to make three numbers sound rigorous. ORDER is what forces you to say, before any bar ships, which broken promise a traveler actually can't forgive.

One door swings back open the moment someone edits the sentence. The other stays shut, because the traveler already made the trip.

The order a gut call skips: check the model against real cases before any bar built on top of it gets written down.

Three things worth stating directly, since this is where the real judgment sits. The alternative Griselda's team piloted, and rejected, was one master checklist, twenty items, accuracy and completeness and style all folded into a single pass by one reviewer. Fifty itineraries into the pilot, some were passing with seventeen of twenty checks clear while failing two of three fact checks outright, because a blended checklist can average away exactly the failure that matters most. It lost, for the same reason the old one to five score lost. The AI-specific failure worth naming by name is a grounding gap dressed as good news: the faster feed behind the new three hundred cities lists a closed restaurant as open for an average of fourteen months after it actually shuts, and a model that states a claim with full confidence has no way to say it's less sure about this one unless something forces it to check its own source's age first. The guardrail is that sixty day confirmation window, any named claim resting on a source older than that, or with no confirming source at all, gets flagged instead of stated flat. And the trade-off is real and accepted on purpose. Itineraries in newer, thinner cities read a little less specific, a well reviewed noodle place near the station instead of a name and an address, and each verified claim adds a small amount of generation time. That's traded away on purpose, because the alternative is finding out about the next closed restaurant from a support ticket instead of a flag that cost a few extra seconds and nothing else.

The whole answer to this question, in one picture. Three needles that move on their own beat a single needle that can hide any of them.

And if you want to be sure it really works, try it somewhere else

Same five letters, a veterinary chain instead of a travel app, and this time the fragile claim isn't a restaurant's hours. It's a dosage line.

Mendscript drafts the after visit instructions Elderfen Veterinary Group's fourteen clinics hand pet owners on the way out the door: medication name, dose, how often, what to watch for, when to call back. Ferdinandine Sorbier owns its rubric. For seven months, one staff reviewer rated each note one to five, mostly on whether it read clearly and matched the visit type.

Dr. Evangelina Loxbury caught the near miss during a routine chart review: a printout for a forty two pound dog recovering from a neuter had pulled its pain medication dose from a cached template built for a much smaller weight bracket, understating the dose by half. Nothing had gone wrong yet. The owner hadn't left the building.

Same rank, different lever, mapped straight onto ORDER: the outcome here isn't "a well written note," it's an owner who follows the instructions exactly as written because they trust them. Reversibility splits the same way: a wrong dosage or a missing warning sign can't be undone once a pet's already been given the wrong dose at home, while a stiff, clinical sentence gets fixed by any follow-up phone call. Dependency is the same shape too: verify every dosage and warning line against the actual prescription record before scoring anything, not against how professional the note sounds. Evidence is a hand checked sample of the last sixty discharge notes against real prescription records, cheap, and already sitting in the system. And the rank lands the same way: dosage and warning claims verified against the prescription record first, as a hard gate, every required warning category present second, bedside tone last, worth polishing, never worth blocking a discharge over.

Different animal, same order. The claim a pet owner can't safely act on wrong ships first, not the sentence that only needs to sound kind.

Swap the trigger and it still runs.
Speed: an interviewer caps you at ninety seconds. Skip straight to the rank: verify dosage against the prescription record first, check required warnings second, polish tone last.
Cost: there's no budget to check every note this quarter. Check the highest risk categories first, pain medication and anything with a dosage number, before the rest.
The model got better, for real: say Mendscript's dosage accuracy climbs from occasional misses to near perfect. The rank doesn't move. Dosage verification still leads, it just clears the gate faster.

Where people run it wrong.
They score everything on one blended scale and never find out which promise is actually breaking.
They fix the loudest complaint instead of the one nobody can undo.
They wait for a serious incident to happen before splitting the score, instead of splitting it before anything happens.

How to use it live. Ask, before ranking anything: "which of these mistakes can the person fix themselves after the fact, and which one already happened by the time anyone notices?" That question sorts the three dimensions faster than any framework does on its own.

Flashcards (tap any card to flip it)

Check yourself Score: 0 / 0