When is a search box the correct answer to a request for a chatbot?

Fennelworth Transit Authority runs the buses and light rail for a mid-size metro area, about nineteen hundred employees between operators, station agents, dispatchers, and HR staff. FindDesk is the internal search tool that holds every one of their policies, safety bulletins, and maintenance manuals. Ianto Colbrun owns it. When Selamawit Girma, the VP of People Experience, asked him to swap FindDesk's search bar for a chatbot called AskFennelworth before the next town hall, he had six weeks to work out whether that was a good idea, or just a good demo.

How to answer this, stage by stage

Nobody in the room is grading whether you sound excited or suspicious about chatbots. They're grading whether you can tell a lookup from a real question, on the spot, and say why it matters.

Let's learn

What does a company's own help desk actually need: a box you type into, or someone to talk to?

FindDesk is the search bar Fennelworth's employees use to find a policy answer instead of emailing HR and waiting. Before FindDesk existed, three years ago, finding out something like a PTO rule meant emailing HR and waiting about a day for someone to look it up by hand. Now FindDesk answers most of those same questions itself, in well under a minute, across about fourteen thousand searches a month.

Almost every real question FindDesk gets is one of these two shapes. The shape decides which tool should answer it.

Ianto Colbrun owns FindDesk. Selamawit Girma, the VP of People Experience, came back from a conference having seen a slick chatbot demo and wanted FindDesk's search bar replaced with one, called AskFennelworth, in time for the next town hall, six weeks out. Her reasoning: fewer HR tickets, and a tool that looked like it belonged in the same decade as everything else the company had just bought.

Ianto's team built a pilot and opened it to forty North Line operators. It answered every question the exact same way: find whatever documents matched, then write a full sentence back. Friendly, confident, done.

Here's the part that's easy to miss. Getting a few more answers wrong wasn't the real issue on its own. The real issue was what people did with a wrong answer that sounded exactly as sure of itself as a right one. Nobody double checks a sentence that reads like policy. They act on it.

At its worst, that's two people believing FindDesk told them their extra time off never expired. It does. Forty hours, used by March thirty first, in the same document FindDesk had linked correctly for two years running. Both of them lost hours of paid time off they thought they still had, and it took Ianto's team three full workdays combined to trace both cases back to the same chatbot session and make it right. FindDesk had never once gotten that question wrong before AskFennelworth started answering it in its own words.

What I would leave alone: FindDesk also fields things like "when's the potluck signup close" or "what's on the cafeteria menu this week." Those stay chatty and machine-written, on purpose. A wrong answer there costs someone a shrug, not twelve hours of pay. The rule isn't that a chatbot is bad at lookups in general. It's that a wrong lookup answer has to be cheap before you let a model write it in its own words.

The lesson: a chatbot and a search box can answer the exact same question. The difference that matters is what a wrong answer costs, not which one feels more modern in a town hall demo. Check what the real questions look like before deciding which one gets built.

Now here is the same thing as a story

Use this version when you've got the extra two minutes. A number moves your head. A person's actual Tuesday moves your gut.

Ianto Colbrun has run internal tools at Fennelworth for five years, and he has one rule he doesn't break: never change something everybody depends on without pulling a real sample of what people actually ask it first. It's saved him from shipping at least two bad ideas that nobody else caught in time.

FindDesk itself has been steady for three years, the search bar nobody thinks about because it just works. Then Selamawit Girma came back from a vendor conference in March with a demo on her phone and a deadline in her calendar: swap it for a chatbot, AskFennelworth, before the next town hall, six weeks out.

Ianto started the way he always does, pulling a sample of real questions before touching anything. Six weeks isn't much runway, and Selamawit wanted a pilot ready for a demo, not a research project. He told himself he'd run the full two-week sample after the beta shipped, since it was "just forty people," and pulled a two-day sample instead, enough to sanity check the demo, not enough to catch anything rare. The pilot went out answering every question the same way, lookup or not, because that was the simplest thing to build in six weeks. Nobody flagged it. It felt done.

The two-day sample that felt like enough never got the chance to catch what a six-week one later found.

Then, in week five, two operators on the North Line mentioned to their union rep, in the same week, that they thought their unused time off just carried over on its own. Neither one filed a complaint. They just mentioned it, the way you'd mention the weather.

Payroll's year-end sweep runs automatically. It zeroed out anything over the forty-hour cap, same as it does every year. Both operators lost time off they thought they still had, twelve hours between them, and this time the union rep asked where they'd gotten the idea it didn't expire. Both of them said the same thing: they'd asked AskFennelworth.

Every step up to the fourth one worked exactly as designed. The fourth step is where a fact quietly stopped making it into the sentence.

I want to say the problem was the twenty percent wrong rate. It isn't really that either. Nobody at Fennelworth had a number in their head for how often AskFennelworth should be allowed to get it wrong. They just knew that a question about their own paycheck should never come from something that sometimes makes a detail up and never says so.

Back when the pilot got scoped, in the meeting where six weeks became the whole plan, somebody asked whether the bot should answer every question type the same way, or handle some differently. The answer, at the time, was that splitting it into two paths sounded like twice the work for a demo that only needed to look good once. So it wrote a sentence for everything. That was the choice.

Here's the replay. Same six weeks, same beta group. This time, any question that matches one document by itself gets the document quoted back with a link, no written sentence at all. The PTO question comes back reading exactly what the policy says: forty hours, used by March thirty first. It takes about the same thirty five seconds plain search always took. The fourteen percent of questions that really do need pulling two or three documents together still get a written answer, sources shown underneath, and it's right about eighty nine times out of a hundred. Six weeks later, replayed the same way: zero wrong answers on the lookup questions. Zero.

One design let the model answer every question the same confident way. The other design let it answer only the questions that actually needed it to think.

What I'd tell myself, in the meeting where six weeks became the whole plan: the two-path build was never twice the work. It was maybe a week more, for a bot that couldn't get a paycheck question wrong. I skipped the two easy days of sampling that would have told me that, because a demo felt more urgent than a sample. It cost twelve hours of somebody else's vacation to learn it back.

PICK: the test that decided FindDesk's front door

Not permission to keep every chatbot out, and not a reason to build one everywhere either. PICK only earns its place here if it turns "chat feels modern" into a real test.

One mistake costs a second try. The other costs real pay and a grievance, and it never looks like a mistake while it's happening.

Two shortcuts got tried and dropped before the real fix. Neither one was lazy, both just solved the wrong half of the problem.

The PTO cap and the cafeteria menu are both one-document lookups. Only one of them belongs on the danger side of the chart.

The trade worth saying out loud: routing lookup questions to search means FindDesk's front door won't feel like one smooth conversation. Some questions come back as a quoted line and a link, not a written paragraph. That's a real loss for how modern the tool feels in a town hall demo, and it's worth paying, because the alternative is a fast, friendly-sounding answer that's wrong on exactly the questions people trust it most to get right.

And if you want to be sure it really works, try it somewhere else

Same four letters, a library system instead of a transit authority, and the AI-specific risk moves from a PTO cap to a fine policy that gets copied into thirty four branches' training notes.

Four branches, one root question: does this question already have one right answer sitting in a document somewhere.

Thackwood County Library System runs StackWise, the internal portal its staff use across thirty four branches for circulation rules, interlibrary loan policy, and cataloging standards. Kaarlo Virtanen is the systems librarian who owns it. About 2,200 staff searches a month go through StackWise. A checked sample found 79 percent were single-answer lookups, like "how many renewals can a patron get on an interlibrary loan," and 21 percent genuinely needed pulling two or more policies together, like whether a fine can be waived for a patron who's also part of a school partnership account. Position: build search for the 79 percent, and save a chatbot for the 21 percent that really does need stitching sources together. Impact: a chatbot paraphrasing the fine cap for a lost children's book wrong, even once, risks getting copied straight into a training note that all thirty four branches follow, since staff trust a clean written answer more than they trust digging through the policy binder themselves. Cost asymmetry: a missed search costs a staffer a second try with different words. A wrong fee policy, copied into branch training notes, costs a corrected memo that has to go out to every branch and get read again. Building search first for the lookup-shaped 79 percent is cheap and works immediately. Letting a chatbot answer fee and policy questions with a written paraphrase risks a wrong number reaching thirty four branches before anyone catches it. Kill criteria: does the question have one document with the answer in it, or does it genuinely need two or more policies read together with real judgment involved? For the interlibrary loan renewal count, yes, one document, so it's search. For the fine waiver tied to a partnership account, no, it needs two policies and a judgment call, so a chatbot can draft it, sources shown, a person still signs off before it becomes a branch-wide answer.

Swap the trigger and it still runs.
Speed: an interviewer caps you at ninety seconds. Skip straight to the test: does this question already have one document with the answer in it, or not.
Cost: no time to tag a thousand real queries before a decision is due. Fine, ask whoever's requesting the chatbot for their own best guess at the split, and make them defend that number out loud.
The model got better, for real: say a newer version of the retrieval step starts citing the exact sentence, not just the document, and testing shows it stops dropping numbers and dates. That's the moment lookup questions can start moving to the chatbot too, not before.

Where people run it wrong.
They build the chatbot because it demos well, not because they checked what real questions actually look like.
They treat "the chatbot named its source" as the same as "the chatbot got the source right," when it can point at the correct document and still write the wrong number from it.
They fix one wrong answer by hand and call the whole thing safe, instead of checking whether the same mistake happens again on the next hundred tries.

How to use it live. If you're ever asked whether to build search or chat, buy yourself a second with one plain question, said out loud: "do most of these questions already have one right answer sitting in a document somewhere?" That question is the whole method, asked instead of assumed.

Flashcards (tap any card to flip it)

Check yourself Score: 0 / 0