ConceptAdvancedShipping & Model Lifecycle / Pilot design and POC-to-production / #12
What integration work should happen during a pilot rather than after?
The direct answer
Build and prove the tool's real, automated connection to the system it will read from in production, during the pilot itself, on live data. If a pilot's numbers came from a person hand-carrying a file instead of a working connection, the pilot never tested whether the product can run itself, and that gap gets found at the worst possible moment: right after everyone already agreed it worked.
The ranking, by what breaks first if skipped
Build the live, automated data connection during the pilot, not a manual stand-in for it.Why: dependency. Nothing about production readiness means anything until the tool can get real data on its own.
Set up the tool's own login and permissions, not a person's.Why: reversibility. An auth gap found after rollout begins is the hardest one to unwind, because every site has its own security queue.
Run it on the real count volume at real speed, not a curated sample.Why: cheap to check now, and it's the only way to tell "the model works" from "the model works on data someone chose for it."
Put basic monitoring on the feed, so a missed load gets caught the same day.Why: without it, a broken pipeline looks exactly like a quiet, well-behaved one, until reconciliation is late.
Leave the dashboard's look and the alert wording for after the pilot.Why: none of it changes whether the tool can run itself.
How to answer this, stage by stage
Seven moves. The trap in this question is answering with a list of nice engineering habits, when it's really asking which piece of plumbing has to be tested before a pilot's win means anything for rollout.
1
Ground it in one real product
Say it like this
"Let me make this real. Say Bramston Goods is piloting TallyGate at its Plainfield distribution center. It reads cycle counts off the warehouse system and flags where they don't match what's on record, so nobody has to reconcile 38,000 SKUs by hand at month-end. Osric Vantham is the PM running the pilot, Tolu Ackerman leads the engineering side, and Elowen Feldkamp runs Plainfield's inventory desk. I'll answer against that."
Why this works
Grounds an abstract question about pilots in one real system, so the ranking that follows isn't hypothetical.
2
Name your method before you use it
Say it like this
"I'd use ORDER here. Rank the integration work by what has to be true before anything else about production readiness matters, not just list good engineering habits."
Why this works
Signals a plan up front, so the answer reads as a method, not a checklist pulled from memory.
3
Say what the question is actually testing
Say it like this
"This isn't really asking what to build eventually. It's asking which piece of plumbing, if it's still missing when the pilot ends, turns a real success into a rollout that stalls for months."
Why this works
Separates the real judgment call from a surface reading that treats the question as a generic engineering to-do list.
4
Give the ranked answer straight
Say it like this
"Build the automated connection to the real system first. Then the tool's own auth. Then basic monitoring on the feed. The dashboard, the notification wording, all of it, goes dead last."
Why this works
This is deliverable 0, said out loud, in the order that actually matters.
5
Show what has to be true before anything else
Say it like this
"None of the rest of it counts if TallyGate can't actually get Plainfield's counts on its own. A manual file drop proves the matching logic is smart. It proves nothing about whether the product can run itself once nobody's carrying the file anymore."
Why this works
Shows the order isn't arbitrary. One thing has to be real before the next one is worth testing.
6
Name what's hardest to take back
Say it like this
"The hardest thing to undo is a rollout date that leadership already believes, set on a pilot that quietly ran on a hand-carried file. By the time someone asks which service account TallyGate logs in with, the date's already on a calendar with thirty distribution centers under it."
Why this works
Names the one gap that turns a normal engineering task into a lasting schedule tax.
7
Back it with the numbers and close on the rule
Say it like this
"Here's what it looked like. Reconciling Plainfield by hand cost 60 hours of clerk time a month. With TallyGate, still fed by hand each night, it dropped to 4. Leadership approved rollout to 30 more centers off that number. Then a new engineer asked which login TallyGate used at Plainfield, and the honest answer was: none, a person did. The first wave of centers didn't go live for 15 weeks, nine weeks past target. So: build the real connection first, since nothing after it matters without it. Auth second. Monitoring third. Polish last, because none of it buys back nine weeks."
Why this works
Ends on the literal ranking the question asked for, backed by a number instead of just asserted.
Let's learn
TallyGate is a tool built for Bramston Goods, a big-box retailer. It reads a distribution center's cycle counts off the warehouse management system, the WMS, and flags where a scanned count doesn't match what the system already thinks is on the shelf, so nobody has to check every mismatch by hand.
Before TallyGate, at Bramston's Plainfield distribution center, reconciling the count against the system was a person's job. Elowen Feldkamp has run Plainfield's inventory desk for nine years. At month-end, her three-person team pulled the count file, checked it line by line against the WMS across roughly 38,000 SKUs, and chased down every mismatch by phone or by walking the floor. That took each of them two and a half days. Sixty hours of clerk time, every month, before anything else on their list.
Knowledge spark: what is a cycle count?
Instead of counting every item in a warehouse once a year, workers count a small slice of it constantly, a few aisles a day, on a rolling schedule. It catches errors faster, but it also means there's always a fresh count sitting somewhere waiting to be checked against the system.
Osric Vantham, the PM building TallyGate, ran a seven-week pilot at Plainfield. Every night, Elowen's team exported the day's count file from the WMS and emailed it to Osric's analyst, who loaded it into TallyGate by hand. That part never showed up in a status update, because it worked, and because it wasn't the part anyone was measuring.
What TallyGate actually did was good. Reconciliation dropped from sixty hours a month to about four. Leadership loved the number and approved rollout to thirty more distribution centers before the fiscal year closed.
A live data connection has to be real before auth is worth testing, before monitoring is worth building, before any of the polish matters at all.
The extra rollout planning was not the problem. Say that plainly, because it's the part most people skip past. What actually decided whether Bramston's rollout hit its date wasn't how good TallyGate's matching was. It was whether anyone had ever built the thing that gets real data into TallyGate without a person doing it every night.
Reconciliation time at Plainfield, before TallyGate vs. during the pilot
Clerk hours, manual reconciliationReview hours, pilot with hand-carried data
This number is real. It just never tested whether TallyGate could get the count on its own, only whether it was smart once someone handed it the count.
TallyGate didn't fail at Plainfield. It passed a test nobody meant to give it: can the model find mismatches, not can the product feed itself.
Here's what that costs at its worst. Bramston's WMS runs three different versions across its network, an older on-site version at some centers, a newer cloud one at others. Nobody had provisioned a service account, tested a permission scope, or checked what a live feed even looked like at any of the thirty rollout centers, because Plainfield's own connection had never been anything but a person and an email.
The choice I would take back
Early in the pilot, Osric's team chose to have an analyst hand-load the nightly file rather than spend the first two weeks getting IT to provision even one real, limited API connection at Plainfield. I would keep the pilot's timeline roughly the same. I'd just spend the first two weeks on the connection instead of the dashboard, so the sixty-to-four number meant what everyone thought it meant.
What I would leave alone. Whether TallyGate's mismatch alert shows up as a red flag or an orange badge is not worth arguing about in week one. Elowen's team reads either one the same way, in about two seconds. Spend the pilot's time on whether the data gets there, not on how the flag looks once it does.
The lesson. A pilot's win only means what it looks like it means if the data got there the way production will get it. Test the plumbing at the same time as the model, even at one site, or the number you show leadership is a number about the model alone.
Now here is the same thing as a story
The short version is above. Keep reading if you want to feel why a pilot that actually worked still cost Bramston nine weeks.
Plainfield's inventory desk gets its busiest three days a month, right at close.
Elowen Feldkamp has run that desk for nine years. Ask her which SKU is going to be off before the count file even finishes loading, and she'll usually be right, just from which aisle it sits in.
Osric Vantham joined Bramston's product team eighteen months before TallyGate. What he was good at, from his first project, was getting something in front of a warehouse floor fast, instead of spending a quarter in design reviews before anyone touched it.
He built TallyGate's pilot around Plainfield because Elowen's team already tracked their reconciliation time closely, which made the before-and-after easy to show. To get data into the tool fast, without waiting on an IT ticket queue, his analyst set up a simple routine: export the count file from the WMS each night, email it over, load it into TallyGate by hand before morning.
For six weeks, that was the whole pilot. Every status update opened with the same number, sixty hours down to four. Nobody asked how the data got into TallyGate, because the number kept being good.
Then, in the meeting where Bramston's leadership signed off on rolling TallyGate out to thirty more distribution centers, one of Tolu Ackerman's newer platform engineers asked something that wasn't meant as a challenge. "Which service account does TallyGate log into WMS with, at Plainfield?" He wasn't trying to slow anything down. He was just filling in a rollout runbook and needed a name to put in a field.
Nobody had an answer. There wasn't a service account. There was an analyst, an email, and a person deciding every night to keep doing it.
One of these is a Tuesday-afternoon fix. The other one was already believed by thirty distribution centers' worth of rollout plan.
So instead of the rollout starting on the date leadership had already approved, Tolu's team had to find out, one center at a time, whether Bramston's older on-site WMS or its newer cloud version was running there, provision a real service account for each, and get every one through a security review that nobody had scoped, because nobody had known it needed scoping.
It was never really about whether TallyGate worked. Plainfield already proved that. It was about what "it works" had quietly never included.
A month into the pilot, in the meeting where the team picked what to build next, someone had floated spending a sprint on the real connection instead of the reporting view Osric wanted for the leadership readout. The nightly file already worked, and the connection felt like something to get to once the pilot proved the concept. Nobody wrote down that the pilot would end, and the rollout would start, before that sprint ever got picked up.
I would go back and pick that sprint. Not instead of the reporting view, alongside it. The reporting view sells the pilot. The connection is what makes the sale mean anything past Plainfield.
With that change, here's the replay. Same seven-week pilot, but the first two weeks go to a real, narrow API connection at Plainfield instead of dashboard work, tested against the same live counts Elowen's team was already pulling. Osric's readout still shows sixty hours down to four, except now the number describes what production will actually do. When the rollout meeting happens, Tolu's engineer already has an answer to give the runbook, because Plainfield's answer already exists and every other center just needs the same thing copied and reviewed.
Fifteen weeks later, in the world that actually happened, the first wave of five centers finally went live. In the version where the connection got built during the pilot, that same wave hits its original six-week target, and the other twenty five follow within the quarter instead of trailing into the next one.
What I'd tell myself, back in the meeting where the sprint got skipped: the thing the pilot is supposed to prove isn't just that the model is smart. It's that nobody has to keep being the plumbing.
ORDER: which piece of plumbing has to be real first
GUARD would fit if the question asked who gets hurt by a bad flag. This question sits earlier than that, which piece of engineering work a pilot has to finish before its own success means anything. That's ORDER's job.
O, outcome. Every piece of integration work in this pilot competes for one thing: a rollout date that actually holds once thirty distribution centers are counting on it.
R, reversibility. The hardest thing to walk back is a rollout date leadership already approved, set on a pilot that quietly ran on a hand-carried file. Once that date is on thirty calendars, finding the gap doesn't just cost a fix, it costs the schedule.
D, dependency. Nothing else about production readiness counts until TallyGate can pull real data on its own. A manual workaround proves the matching logic works. It proves nothing about whether the product can run without a person carrying the file.
E, evidence. Cheap to check first: was the pilot's data flow automated, or was a person moving a file by hand every night, before anyone spends a dollar on the rest of the rollout.
R, rank. Build the real, automated connection first. Then the tool's own auth and permissions, not a person's login standing in for it. Then basic monitoring on the feed, so a missed load gets caught the same day. The dashboard and the alert wording go dead last, because none of it changes whether the tool can run itself.
Weeks to first-wave rollout, planned versus what actually happened once the gap was found
The moment the auth gap surfacedNine weeks of unplanned connection and security work
The line doesn't bend because the model got worse. It bends because the plumbing the pilot never tested had to get built after the date was already promised.
The check that keeps this ranking honest
If provisioning the real connection at the other thirty centers had taken a day each, this wouldn't rank first. It ranks first because each center's WMS version, permissions, and security review were genuinely different, so the gap compounded thirty times instead of getting fixed once.
Same order, a hospital piloting a scan-review tool instead of a warehouse
Windham General Hospital piloted ScanBridge, a tool that flags chest X-rays worth a second look before a radiologist signs off, so the busiest reads get a check without every scan needing one.
O. Every version of ScanBridge's rollout order protects one thing: that radiologists across Windham's network trust the flag enough to act on it without re-reading every scan themselves.
R. Running the pilot on scans a research assistant pulled and loaded by hand, then discovering at rollout that ScanBridge has never pulled a live scan from the hospital's own imaging system, is the hardest thing to undo. It doesn't cost a patch. It costs months of provisioning across every site that has to connect.
D. None of it matters until ScanBridge can pull a scan the moment it's taken, on its own, from the imaging system radiologists already use.
E. Cheap to check: was a person exporting and uploading scans during the pilot, or did the tool ever once pull one on its own.
R. Same order: build the live connection to the imaging system first. Then the tool's own clinical-system login, not a researcher's. Then an alert if a scan goes unflagged for too long. The report layout Dr. Ilja Csoban's team wanted goes last.
Swap the trigger and it still runs
Bramston needs the rollout to move faster, not slower, after a strong quarter. The order doesn't move. Speed makes the automated connection matter more, not less, since there's less time to discover the gap the hard way.
TallyGate's matching model gets noticeably more accurate. Doesn't reorder either. A better model still can't reach data nobody built a pipe for.
Bramston licenses TallyGate to a smaller retail partner with its own WMS. Doesn't reorder. The rule protects the same thing no matter whose warehouse system sits on the other end.
Where people run it wrong
Treating a pilot's headline number as proof the product works, when a hand-carried file only ever proves the model works once someone hands it clean data.
Building the dashboard leadership will see before the connection nobody will see, because the dashboard is the more visible thing to show progress on.
Testing the connection once, early, and never checking it again after the underlying system gets upgraded mid-pilot.
How to use it live
Say the outcome out loud before naming a single result. "Every piece of this pilot protects one thing, that the win still holds once nobody's hand-carrying the data anymore." Then ask what's still being carried by a person. Naming the outcome first turns a vague integration question into something you can rank line by line.
Flashcards (click a card to flip it)
1 · THE FRAMEWORK
Which framework fits ranking which integration work has to happen inside a pilot, and why not GUARD?
Tap to flip
ANSWER
ORDER, for ranking which piece of work is hardest to undo once it's discovered missing. GUARD is for who gets hurt by a bad flag, not for sequencing what gets built when.
2 · THE PERSON
Who is this answer about?
Tap to flip
ANSWER
Osric Vantham, the product manager running TallyGate's pilot at Bramston Goods's Plainfield distribution center, working with engineering lead Tolu Ackerman and inventory lead Elowen Feldkamp.
3 · THE HABIT
What did Osric's team stop asking about once the pilot's numbers looked good?
Tap to flip
ANSWER
They stopped asking how the data got into TallyGate. An analyst hand-loading a nightly file kept working, so nobody checked whether a real connection existed.
4 · THE DEPENDENCY
What has to be true before any other piece of integration work counts?
Tap to flip
ANSWER
TallyGate has to be able to pull real data on its own. A manual file drop proves the matching logic works. It proves nothing about whether the product can run without a person carrying the file.
5 · THE OLD DECISION
What decision would you take back, and why did it make sense at the time?
Tap to flip
ANSWER
Having an analyst hand-load the nightly count file instead of spending the pilot's first two weeks on a real connection. It made sense because it let the pilot start fast without waiting on an IT ticket queue.
6 · THE NUMBER
Reconciling Plainfield by hand cost ___ hours of clerk time a month. With TallyGate, still fed by hand, it dropped to ___.
Tap to flip
ANSWER
60 hours, down to 4. A real number, and one that never tested whether TallyGate could get the count on its own.
7 · THE REPLAY
Same seven-week pilot, but the connection gets built in the first two weeks instead of the dashboard. What changes?
Tap to flip
ANSWER
The readout still shows sixty hours down to four, except it now describes what production will actually do. The first wave of centers hits its six-week target instead of going live at week 15.
8 · THE TRANSFER
Section 4 runs ORDER again on a different product. Which one, and what plays the role of the manual file drop there?
Tap to flip
ANSWER
Windham General Hospital's ScanBridge. The equivalent gap is a research assistant loading scans by hand instead of ScanBridge pulling them live from the hospital's own imaging system.
Check yourself Score: 0 / 0
Fill in the blank
1. Reconciling Plainfield's counts by hand cost ______ hours of clerk time a month. During the pilot, still fed by a nightly file, it dropped to ______.
Show hint
It's the number leadership used to approve the rollout.
Show answer
60 hours, down to 4. A real drop, but one that never tested whether TallyGate could get the count without a person carrying it.
Multiple choice
2. Which piece of integration work does this answer say has to happen first, before anything else about a pilot's readiness?
A. Designing the leadership dashboard and reporting view
B. Building a real, automated connection to the system the tool reads from
C. Getting the model's matching accuracy above 95 percent
D. Writing the alert copy for a missed reconciliation
Show hint
Ask what every other piece of rollout work quietly assumes already exists.
Show answer
B. Every later piece of integration work assumes the tool can already get real data on its own. Build that first, or the rest gets tested on a workaround that won't scale.
Short answer, name the reversal
3. What old decision does this answer take back, and why did it make sense when the pilot started?
Show hint
Look for the decision that let the pilot start fast, not the one that made it look good.
Show answer
Model answer: Having an analyst hand-load the nightly count file instead of spending the first two weeks on a real API connection. It made sense because it let the pilot start immediately, without waiting on an IT provisioning queue.
True or false
4. True or false: since TallyGate's matching results were accurate during the pilot, the manual file drop was a safe thing to leave for after rollout began. Say why.
True
False
Show hint
Think about what actually cost the nine weeks: the matching accuracy, or the thing that fed it.
Show answer
False. The nine-week delay wasn't caused by TallyGate getting anything wrong. It was caused by nobody ever building the connection the pilot's own numbers assumed was there. Accuracy and plumbing are two different questions, and the pilot only answered one of them.
Short answer, apply it yourself
5. Pick a pilot you're running or have seen run. What piece of plumbing is currently a person doing something by hand, that would need to be real automation before you'd trust the pilot's result for a wider rollout?
Show hint
Look for the step nobody puts in the status update because it already works.
Show answer
Model answer: "A support-ticket triage tool where someone on the team was manually tagging which tickets counted as 'urgent' before the model saw them. The pilot's accuracy number was really measuring the tagging, not the model reading raw tickets on its own."
Short answer, the number question
6. If Plainfield's reconciliation had only dropped from 60 hours to 30, would building the real connection during the pilot still need to rank first? Say what changes and what doesn't.
Show hint
Dependency is about what has to be true first, not about how big the win looks.
Show answer
Model answer: "What changes: leadership might be less excited to approve a fast rollout off a smaller win, so there's more time to notice the gap. What doesn't change: the connection still has to be real before the number means anything for thirty other sites, because the ranking depends on what the tool can prove about itself, not on how impressive the number is."
From U2xAI Academy
From answering questions to owning outcomes.
A live workshop where you ship a working AI agent, defend a launch decision, and walk away with a portfolio recruiters can't wave off, not just more questions to study.
