How does a pilot for an internal tool differ from a customer pilot?

How to answer this, stage by stage

This is a yes-or-no about which parts of one internal tool earn the slower process, not a rule for every internal build, so PICK carries the answer.

A last note before the walkthrough ends: this pick is about where the rigor goes, not a blanket ban on ever moving fast internally. Most candidates hear "internal tool" and answer like the whole thing gets a pass. Name the one slice that can't afford a quiet miss, and you've shown judgment instead of reciting a rule of thumb.

Let's learn

Talbot Trust is a regional bank. About 14,000 people work there, and every one of them can send the IT helpdesk a ticket: a broken laptop, a VPN drop, a new access request, a password reset. Around 1,100 tickets land there every week.

Six people used to do that sorting by hand. Most tickets are boring: a cracked screen, a stuck mouse, a forgotten password. Those go in a normal queue, get looked at within a day, and nobody's hurt if it takes an extra hour.

A small slice, about 60 tickets a week, are different. They're Access & Identity tickets: someone needs a system unlocked, someone's leaving the bank and needs their access shut off, or someone thinks their account has been broken into. Before Relay, the triage lead hand-carried every one of these straight to the security team, the same day, no general queue.

Then Talbot Trust built Relay, a tool that reads the ticket text and does that sorting on its own.

Relay ran a two-week trial with 200 volunteers at head office. It routed 95 percent of tickets correctly, one number, covering every ticket type at once. Everyone liked the number. Relay went live for all 14,000 employees the following week.

Here is the turn. That extra five percent wrong is not the real problem. The real problem is that nobody had ever asked how that five percent was spread out. Ninety percent of Relay's tickets are mouse requests and password resets, where a mis-tag costs someone an hour. The Access & Identity tickets, a much smaller slice, were the ones actually carrying that five percent, and nobody had split the number out to check.

At its worst, an Access & Identity ticket, something that decides whether someone can still get into their accounts, sits quietly in the wrong queue with nobody watching for it, because the whole tool is being graded by one number that's mostly measuring mouse requests.

What I would leave alone. The other ninety-plus percent of Relay's tickets: hardware, general applications, password self-service. None of that needs a slower process. A wrong tag there is loud and cheap; someone notices within the hour and moves on.

The lesson. "Internal" is not a risk level. The risk lives in what a ticket controls, not in who's asking for it. If a slice of an internal tool can quietly get something wrong that a customer product would never be allowed to, it earns the customer-grade process, whether or not anyone outside the building ever sees it.

Most of the queue is cheap to get wrong. The risk lives in the small slice

Two days nobody noticed, until someone else did

You don't need this to answer the question. Read it if you want to feel why the split has to happen before a real ticket, not after a good pilot readout.

Sundeep Malhotra has spent six years running internal tool launches at Talbot Trust. The expense app. The meeting-room booking tool. Both shipped without drama, both because he'd learned to pilot the boring parts fast and the risky parts carefully. Relay was supposed to be no different.

Relay reads the text of every incoming helpdesk ticket and decides, on its own, what kind of problem it is and how fast it needs to move. Before it existed, a triage lead named Solene ran that job by hand, and Access & Identity tickets, the ones about who can get into what, never touched the general queue at all. She walked them straight to security. Same day, every time.

Sundeep piloted Relay the fast way. Two weeks, 200 volunteers at head office, one number at the end: 95 percent of tickets routed correctly. Leadership loved it. Relay went live for all 14,000 employees the following week, three weeks start to finish.

Nobody had asked Relay for a second number. Just the one.

Five weeks into full rollout, Tarek Yates, an analyst on the fourth floor, submitted a ticket titled "can't log into my email, weird stuff happening." He meant his account might be compromised. Relay read it, tagged it "Applications, routine," and dropped it into the general queue with a next-business-day SLA.

It sat there for two days.

Nobody at the helpdesk caught it. What caught it was Talbot Trust's fraud-monitoring system, a completely separate tool that watches for unusual transaction patterns. It flagged a strange run of wire-transfer approval attempts coming from Tarek's account and looped in security directly, days before anyone would have opened his ticket in the ordinary run of the queue.

Same tool, same kind of mistake, two very different sizes of wrong

Security pulled and hand-checked 80 Access & Identity tickets from the five weeks since Relay went live, a proper sample, not just Tarek's. Sixty-five were routed correctly. That's 81 percent, not 95. Twelve of the fifteen wrong ones had sat an average of two days before anyone caught them at all, instead of the same-day hand-carry the old process guaranteed. Tarek's was one of the twelve. The only reason anyone went looking was luck that had nothing to do with Relay.

I want to say the mistake was trusting the 95 percent. It wasn't, not exactly. The 95 percent was true. It just wasn't the number that mattered. It was measuring eleven hundred tickets a week where being wrong costs an hour, and hiding sixty a week where being wrong could cost someone their accounts.

So here is the choice I'd take back.

When Sundeep scoped the pilot, he ran every ticket type through the same fast process and graded the whole thing with the same one number. That made sense for the ninety percent that's mouse requests and password resets. I'd split it. Let the fast, light pilot cover everything except Access & Identity, and give that slice its own bar, its own staged rollout, and a kill switch, before Relay ever tags one of those on its own.

And the thing I'd tell myself, back in that scoping meeting: a tool being internal was never the reason it was safe to move fast. The reason was that most of what it touches is cheap to get wrong. The moment that stopped being true for one slice of tickets, the pilot should have stopped treating that slice like the rest of them.

PICK, spelled out for a bank helpdesk

This is a yes-or-no about which slice of one internal tool earns the slower process, not a rule for every internal build Talbot Trust ever ships, so PICK carries the weight here.

Try the same four letters on an expense desk

Chinonso Iwu builds internal finance tools at Bellwether Retail. Her team's newest one, Outlay, reads every submitted expense report and flags line items that look off before a person approves reimbursement, so an approver only has to look closely at what got flagged instead of every receipt.

Before wiring Outlay into every expense category at once, Chinonso split the pilot the same way Sundeep eventually did.

P. Fast, light pilot for domestic expenses, meals, mileage, office supplies, one flag-rate number, ship it in three weeks. Full customer-grade rigor, a defined miss tolerance, an audit trail, sign-off from legal and tax, for international travel and any spend category with a regulatory disclosure threshold attached.
I. A missed flag on a domestic lunch receipt is felt by one approver, who notices it on the monthly summary and mentions it in passing, no real cost. A missed flag on a sanctioned-country expense or a mishandled VAT claim is felt by the whole finance and legal team, and it isn't felt at all until an external tax audit finds it, sometimes years later.
C. The domestic miss is cheap and visible, someone notices within the month and it's a two-minute conversation. The regulated-spend miss is hidden and expensive, nothing internal ever flags it, so it survives quietly until an outside authority finds it, and by then it's a fine and a filing, not a coaching conversation.
K. Flip regulated categories to the fast process once Outlay holds a 100 percent catch rate on a quarterly, audit-style sample for two consecutive quarters, the same guarantee the manual review already provides today.

Flashcards (click a card to flip it)

Check yourself Score: 0 / 0