How do you handle a pilot customer who wants custom behaviour that does not generalize?

How to answer this, stage by stage

Nobody is grading whether you can say yes or no to a pilot customer. They are grading whether you can rank the real options by what is hardest to undo, not by who is asking loudest this week. Eight moves get you there.

Let's learn

Say we build a tool that reads a company's expense reports and checks each line against that company's own spending rules. Cinch does this for Halsted Finance's customers: a receipt over a set amount needs a photo attached, a meal over a set amount needs a manager's note, a mileage claim gets checked against the standard rate. Before a tool like this, a company's own accounts payable clerk did all of it by eye, one report at a time, maybe forty a day.

With Cinch running, that same clerk reviews closer to four hundred reports a day, because the model clears the easy, obvious ones on its own and only hands over the ones that actually need a person to look.

Marrick Freight, three weeks into a pilot, runs three hundred and forty long-haul drivers who file about nineteen hundred meal line items a week from truck stops and diners on the road. Cinch's standard rule, built for Halsted's early customers, needs a receipt photo for anything over twenty five dollars. A sit-down meal at a truck stop often runs twenty eight to thirty four dollars, and a driver who has been on the road twelve hours rarely still has the paper receipt. So Cinch flags around four hundred and sixty of those meal line items a week, missing a receipt, over the line, and drops every one into a queue for Marrick's small finance team to check by hand.

Here is the turn. Those four hundred and sixty flagged meals are not really the problem. The real problem is what Marrick's finance team does next. They can trust the queue and wave the whole thing through without looking, which throws away the actual audit. Or they can open all four hundred and sixty by hand every week, which is close to the same fifteen hours a week Cinch was supposed to save them in the first place.

At its worst, this costs more than fifteen hours a week back. Say the fastest fix ships instead, a line hidden inside Cinch's shared prompt that lets driver meals skip the check whenever the customer is Marrick. That same line now sits inside the exact model path every other customer's audit runs through. Nobody outside the two people who wrote it knows it is there. The next time anyone touches that prompt for an unrelated reason, driver meals could quietly stop being checked at all, or the same shortcut could get copied for a different customer who never should have gotten it.

Nothing on this list means anything until you count how often the exception would actually fire. Building the real setting, and turning it on, both sit downstream of that one count.

The choice I would take back is not the twenty five dollar line itself. It is that Cinch's receipt rule was built as one number, sitting inside the shared prompt every customer's report runs through, instead of a real setting a customer's own contract could hold. That was a fine choice back when every one of Halsted's early customers was an office of consultants and accountants filing the same kind of report. It stopped being fine the day a trucking company showed up.

What I would leave alone: Cinch's mileage math never needs a Marrick-shaped exception. Every customer, trucking or not, reimburses mileage against the same standard government rate, so that part of the tool already works exactly the same for everyone, and touching it would only add risk nobody asked for.

The lesson: a customer's one-off request is not really a question about whether to say yes. It is a question about whether your product has a real setting for this at all, or only a shared number with nowhere else to live.

Now here is the same thing as a story

The short version is above. Read on if you want to feel how ordinary the Monday that almost broke this looked from Kaia's side of the pilot.

Kaia Denholm has spent two years turning Cinch from a demo into something forty finance teams now run their expense reports through every week. She is good at the part most product managers dread: sitting on a pilot call, hearing a customer describe a strange edge case, and knowing inside thirty seconds whether it is a real gap in the product or a one-off that will never come up again.

For the first three weeks of the Marrick Freight pilot, Kaia loved that call. Feyi Ostrow, who runs finance operations there, kept saying how much lighter her mornings felt. Cinch was clearing most of Marrick's non-driver expenses clean, hotel folios, fuel cards, office supplies, the exact same as it did for every other customer.

Then came the meal reports. On the Monday of week three, Feyi flagged eleven driver meals in a row that Cinch had sent to her queue, all in a single morning, all missing a receipt, all a few dollars over the line. By Wednesday it was a running joke in their shared pilot channel. By Friday it was not a joke anymore. It was forty five minutes of her one clerk's day, every day, undoing the exact thing Cinch was built to remove.

The trigger came the following Monday, small, almost nothing. Feyi wrote one line: "Can you just turn this off for us? Our drivers aren't lying about their lunch." No threat, no deadline, just a tired, reasonable ask from someone who had been patient for three weeks.

Here is where it almost went wrong. Reo, the engineer closest to the Marrick account, had a fix ready inside the hour: add a check right inside Cinch's shared prompt, if the customer is Marrick and the role is driver, skip the receipt rule under seventy five dollars. It would have shipped by end of day. It would have made Feyi's Monday. It probably would have helped the renewal call sitting on the calendar for that Friday.

Because it was not really a Marrick-only line. It was one condition inside the exact prompt every one of Cinch's forty customers' expense reports ran through, this week and every week after. A junior associate at a consulting firm claiming a seventy dollar client dinner with no receipt would clear the same gate, for a completely different reason, and nobody reviewing that report would ever know a rule built for a trucking pilot was the reason it slipped through.

A slower Monday for one clerk is a cost you can hand back the moment the evidence changes. A line buried in a shared prompt is not something anyone can point to and undo six months from now.

So Kaia pulled the fix before it merged. Two years earlier, when Cinch's founding engineers first built the receipt rule, the design meeting was short. Every one of Halsted's first six customers was an office of consultants, agencies, or law firms. Twenty five dollars, one number, baked straight into the prompt, was the fastest way to ship something that worked for all six of them. Nobody in that room was picturing three hundred and forty drivers eating at truck stops. Why would they. That customer did not exist yet.

What Kaia actually did that Monday: she gave Feyi's team a real lever instead, a bulk-review button on the flagged queue that let her clerk clear a whole week of driver meals in one pass, with one shared note, instead of opening each one alone. Not a fix to the model. A fix to the workflow around it. She tagged every meal that queue would have skipped, four hundred and sixty a week, and told Feyi plainly: give me three weeks, and if two more transportation pilots hit the same wall, I will build this properly, as a real setting, not a favor.

The replay, three weeks later: two more pilots, a produce distributor and a parcel courier, had asked Halsted's sales team for almost the identical thing, on the exact same meal-receipt rule. That was the number Kaia needed. She built a real setting, a receipt-and-cap rule keyed to a customer and a job role, checked outside the model call itself, tested against six weeks of Marrick's own flagged meals before it ever touched a live report. It cleared at a rate Halsted's compliance team was comfortable signing off on. Feyi's clerk went from forty five minutes a day on driver meals to about six. Nobody at the consulting firm three accounts over ever noticed a thing changed, because for them, nothing did.

What I would tell myself, back in that near miss on a Monday: the fast yes and the right yes are not the same shape of decision, and the fast one is the one you can almost never take back.

Five checks before Marrick's request gets anywhere near the model

This is a question about what to build first and what to refuse first, not a story about a habit with two settings, so ORDER fits, not a step-by-step recovery framework.

Two things worth saying out loud here, since this is exactly where an AI PM question earns its name. First, the alternative most candidates reach for by reflex is building Marrick a full custom policy tier that same sprint, because they are a paying pilot and the renewal call is Friday. That got ruled out on purpose. A permanent feature built for a sample size of one customer usually gets rebuilt anyway the moment a second customer's version of the same rule turns out slightly different, and building it fast under deadline pressure is exactly how a rule tuned for one account quietly breaks for everyone else. Second, the real bar for the eventual fix is not "never let a driver's meal through without a receipt." It is a calibrated one: the new per-role check has to clear its own held-out set of real Marrick receipts, checked outside the model call, at a precision Halsted's compliance team signs off on, most of the time, by design, before it ever reaches a live report. That trade is real too. Saying no cost Marrick roughly three more weeks of manual review, real hours their clerk did not have to spend before the pilot. It bought Cinch's audit accuracy for the other thirty nine customers already leaning on that same shared prompt.

And if you want to be sure it really works, try it somewhere else

Same five checks, a completely different product, a different industry, so the method proves itself instead of repeating a story you happened to prepare.

Pallister Civic Systems sells Ordinance, an AI tool that checks building permit applications against a city's own zoning and code rules, to city planning departments across the state.

O, outcome. What every option competes to protect: a permit check that applies a city's real rules the same honest way to every applicant, not a version that quietly bends because one city's planning director asked nicely.
R, reversibility. Writing "auto-approve any permit inside the Maple Street incentive district, skip the setback review" straight into Ordinance's shared compliance prompt runs on every city's permits the moment it ships, not just Cedar Falls'. Telling Cedar Falls no for now, and routing those permits to a fast-track human queue instead, costs a planning clerk a few extra minutes per file, and it is easy to undo the day the evidence changes.
D, dependency. Averill Rasmussen, who runs product for Ordinance, first needed to know whether Cedar Falls' incentive-district carve-out was a real pattern or a one-off. Two other pilot cities running their own downtown revitalization zones turned out to want almost the same thing, which is what actually justified building a real setting.
E, evidence. Before building anything, Averill tagged every permit that would have used the carve-out for six weeks, and reached out to Ordinance's other pilot cities directly to ask.
R, rank. Fast-track human queue for Cedar Falls now, count the real volume, check the other pilot cities, then build a real zoning-district setting, checked outside the model, before switching it on anywhere.

Swap the trigger and it still runs.
Speed: an interviewer caps the answer at ninety seconds. Skip straight to the rule: rank by what's hardest to undo, and name the manual queue over the shared-prompt hack as the top move, not the other way round.
Cost: engineering says the real per-role setting cannot ship for six weeks, not three. Don't cave and ship the shared-prompt hack in the meantime. Keep the manual queue running a little longer instead, and say plainly that the trade is real hours for the pilot customer against safety for everyone else.
The model got better, for real: say Cinch's receipt-reading accuracy jumps ten points overnight. That still is not the same claim as "a rule built for one customer is safe to bury in a shared path." A better model can still be handed an exception nobody remembers writing.

Where people run it wrong.
They treat "the customer is paying, so say yes fast" as the whole analysis, instead of asking what saying yes fast actually touches.
They build the customer a full permanent feature off one request, then rebuild it anyway once a second customer's version of the same ask turns out different.
They bury the exception in the model's own instructions because it is the fastest place to put it, not because it is the right place to put it.

How to use it live. Say the rule before naming a single option: "I rank a pilot's one-off request by what's hardest to undo if I get it wrong, not by how loud the ask is." That buys you the room to give the real answer for this specific request, instead of reciting "always say yes to keep the customer happy" on reflex.

Flashcards (click a card to flip it)

Check yourself Score: 0 / 0