ConceptIntermediateAI Opportunity & Model Strategy / Feasibility assessment and technical spikes / #18
What is the right team composition for a feasibility spike?
FLIPSa clipboard, a spike team, and the one role nobody had put on it
Loomcraft Textiles is piloting a computer-vision system that scans finished garments on the line for defects: loose stitching, uneven dye, torn seams. Marisol Ferreira runs quality control there, and had to decide who actually belonged on the feasibility spike team before an audit made the answer expensive.
The direct answer
Put the person who will actually act on the spike's answer inside the spike team from day one, not just the data scientist who can build it. For a defect-detection spike, that means a floor inspector or another domain expert sits in the room and reviews real flagged garments alongside the model's calls, because they catch the failure modes a benchmark accuracy number never shows.
Do this, in order
Include the person who will act on the spike's verdict, not just the person who builds it.Why: a benchmark number can look great while missing the exact defect type that matters most on the actual floor.
Have that domain expert review real flagged and unflagged garments during the spike, not after it.Why: catching a gap during the spike costs an afternoon; catching it after sign-off costs a customer complaint and an audit.
Get a name on record for who signs off on the spike's result.Why: without one person accountable for the verdict, "the team agreed" quietly becomes nobody's decision.
Keep the core team small, not everyone who might have an opinion.Why: a data scientist, an ML engineer, and one domain expert is enough; a large committee slows the spike without adding real judgment.
Say plainly when a solo spike is fine.Why: an early, throwaway feasibility check with no real stakes yet doesn't need the full team either.
How to answer this, stage by stage
Nobody is scoring whether you can name job titles. They're scoring whether you can say, concretely, which specific role catches the failure a clean benchmark number would have missed.
Stage 1
Scope it to one real spike
Say it like this
"Let's ground this in a real case: a fabric-defect detection spike at Loomcraft Textiles, where the team composition itself turned out to be the thing that decided whether the spike actually meant anything."
Why this works
Keeps the answer from turning into a generic org-chart discussion with nothing real behind it.
Stage 2
Say your structure out loud
Say it like this
"I'll run this as FLIPS. Find the person whose morning this is. Locate the habit they built. Identify the flip, the verb that snaps. Pinpoint the old decision. Show the replay."
Why this works
Naming the method up front tells the interviewer you have a repeatable way to reason about team design, not just an opinion about org charts.
Stage 3
Reframe: it isn't "who can build the spike," it's "who can catch what the spike misses"
Say it like this
"This isn't really a question about who has the skills to run a computer-vision spike. Data scientists can build that easily. It's a question about who in the room actually knows what a real defect looks like on the floor, well enough to notice when a benchmark number is quietly wrong."
Why this works
This is where a strong answer separates from someone who just lists job titles without saying what each one is actually for.
Stage 4
Give the flip
Say it like this
"Here's the flip: Marisol used to review every spike result herself, alongside the data science team. Once the spike team was scoped as data scientists and engineers only, to move faster, she stopped being in the room at all, and started trusting a summary slide instead."
Why this works
This is the direct answer to what actually goes wrong, named as a real behavior change, not a vague warning about "communication."
Stage 5
Prove it with the compressed evidence
Say it like this
"The spike team, data scientists and an ML engineer, reported 96 percent accuracy against their own labeled sample and called it ready. When a client audit reviewed 40 recent garments by hand three weeks later, it found three real defect types the model had never once flagged, including a loose-collar-seam pattern any floor inspector would have caught in seconds."
Why this works
Compresses the whole failure into the one gap nobody in the actual spike room was positioned to notice.
Stage 6
Name the AI-specific reasoning and the trade-off
Say it like this
"The honest reason this isn't a generic staffing question is that a model's accuracy score is only as good as the examples it was tested against, and a team without deep floor knowledge doesn't know which examples are actually the hard, rare ones worth testing. We accepted a slower spike, one more person's calendar to coordinate, in exchange for never finding out about a missed defect type from a client's own audit."
Why this works
This names the load-bearing, AI-specific judgment: an accuracy number means nothing without someone who knows what the hard cases actually look like.
Stage 7
Say what a smaller team is fine for
Say it like this
"For a rough, throwaway first look, just to see whether the idea is worth pursuing at all, a data scientist alone poking at a public defect-image dataset is completely fine. The floor inspector's seat matters once the spike's answer is going to drive a real go or no-go decision."
Why this works
Shows judgment about when the fuller team composition actually earns its cost, instead of applying the rule everywhere out of habit.
Stage 8
Close on the one decision
Say it like this
"So: put the person who will act on the answer inside the spike, not just the person who can build it. That's what separates a spike that produces a real decision from one that produces a slide."
Why this works
Restates the direct answer in one breath, tying the whole walkthrough back to the single decision it was built to defend.
Let's learn
Say we build a tool that flags fabric defects before they reach a customer: loose stitching, uneven dye patches, torn seams, caught by a camera on the line instead of a person's eye at the end of the shift.
Before the spike, Marisol Ferreira walked the finishing line herself every afternoon, checking maybe 60 garments by hand out of several thousand produced that day, catching most serious defects but knowing she was sampling, not seeing everything. The pitch for the AI tool was that it could check every single garment, not just a sample.
The five letters, held up as one page. Identify the flip is the step this question is really testing.
Here's the turn: the spike itself wasn't scoped wrong on purpose. It was scoped fast, by two data scientists and an ML engineer, to hit a tight timeline, and Marisol, who used to review every quality initiative personally, was told a summary would be enough this time. That is the actual thing that broke, not the model's code.
Real defects caught, spike team with and without a floor inspector
Same 12 defect types tested in the dry run. Three of them, including the loose-collar-seam pattern, only got caught once someone who actually worked the floor was in the room.
At its worst, the spike reports a clean, confident accuracy number, the tool ships across the whole finishing line, and the exact defect type nobody on the spike team knew to test for keeps slipping through, invisible, until a client's own quality audit finds it months later on garments already shipped.
A spike team that can build the model isn't the same thing as a spike team that knows what the model is supposed to catch. The gap between those two is invisible until someone outside the room finds it for you.
The choice I would take back
When the spike was scoped, the plan removed Marisol's seat on the working team to move faster, keeping her as a stakeholder who'd review a final summary instead. That made sense when the goal was speed and the model's code was the obvious bottleneck. It stopped making sense the moment the real bottleneck turned out to be knowing which defects were worth testing for in the first place, which was never a coding problem at all.
What I would leave alone: I wouldn't require a floor inspector's seat on an early, throwaway spike just to see whether computer vision could plausibly detect fabric defects at all, using a public image dataset with no real production stakes yet. That's a fine job for a data scientist working alone.
The lesson: we scoped the team around who could write the code fastest, when the real question was always who could tell us whether the code was even solving the right problem.
Now here is the same thing as a story
The short version above is what you'd say defending the team composition in a five-minute planning meeting. Read this one for what it felt like the week an audit found the gap before Marisol ever got the chance to.
The clipboard Marisol carries on the finishing line floor has a torn corner and a rubber band holding the pages together, nine years of daily inspection notes underneath the top sheet.
For years, any new quality initiative at Loomcraft went through her directly. She'd sit in the room, ask the annoying question about the rare case nobody else thought of, and sign off personally. When the AI defect-detection spike got scoped, the project lead, under real time pressure from a client deadline, decided to keep the working team lean: two data scientists and an ML engineer, with Marisol as a stakeholder who'd get a summary at the end.
The composition the spike should have had from day one, and the seat that got quietly left off it.
Nobody meant any harm by it. It felt like the sensible, fast way to run a technical spike: put the people who can build the thing in the room, and loop in everyone else once there's something concrete to look at.
The spike ran for two weeks. The model scored 96 percent accuracy against a labeled sample the data science team had put together themselves, mostly from garments already known to have common defects: uneven dye, obvious torn seams. It looked ready.
One number moved smoothly. The other one held steady, then snapped, three weeks later, on a client's own floor.
Then came an audit, not of the model, of the whole quality program, requested by one of Loomcraft's biggest clients after a routine spot-check on their end found a batch with a defect nobody on the line had flagged: a loose seam at the collar that, under wear, would open within a few washes.
Knowledge spark: why would a 96 percent accurate model still miss a whole defect type?
Accuracy is only ever measured against the examples it was tested on. If the test sample never included a loose-collar-seam garment, a 96 percent score says nothing about how the model handles that case, it simply never got asked. A high accuracy number and a real blind spot can sit side by side, invisible, until someone who knows the floor points at the exact thing missing from the sample.
Marisol pulled 40 recent garments and reviewed them by hand alongside the model's own calls from the same batch. Three defect types, including the loose-collar-seam pattern, had never once been flagged by the model, and none of them had appeared in the sample the data science team had used to test it.
Same 12 defect types, same dry run. One seat in the room was the entire difference.
The real question was never whether the data scientists had done good work. They had, on exactly the sample they'd been given. The real gap was that nobody who actually knew, from years on the floor, which defects were rare but serious had ever been asked what belonged in that sample in the first place.
One full-page image to carry the whole answer: team composition was never a dial you could tune slightly. It was a switch, and it had been left in the wrong position.
The fix was small: rerun the same spike, same model, same code, with Marisol sitting in for the two dry-run sessions and reviewing real flagged and unflagged garments alongside the team as it ran, not after. All three missed defect types got caught before sign-off this time, added to the training and test data, and the model shipped with a documented list of known-hard cases instead of a clean number that hid them.
What happens on a line when the right expertise never made it into the room in the first place, seen elsewhere too.
What I'd tell myself, hearing how close that loose-collar-seam garment came to reaching a customer: the spike team wasn't wrong to move fast. It was wrong to assume speed and expertise were the same seat.
If you want to remember it easily, here is another wayNot a script for putting every stakeholder on every spike. FLIPS is what tells you exactly which seat, once removed, turns a clean accuracy number into a blind spot.
F
Find the person. Whose morning is this?
Marisol Ferreira, who has run quality control at Loomcraft Textiles for nine years and used to personally sign off on every quality initiative.
A specific person with a specific, earned habit, not an abstract "the quality team."
L
Locate the habit. What did she stop doing because it worked?
Sitting inside every quality initiative's working sessions and asking about the rare case, in favor of reviewing a summary slide once the spike was already scoped without her.
The habit was rational and earned, built from years of catching real problems that way; it stopped only because the new spike was scoped to move fast.
I
Identify the flip. What verb snaps?
Reviewing the working sessions in person, catching gaps as they happen, versus reviewing a finished summary after the fact, with no way back into the room once the spike is done.
This is the hard step, and the one that explains the whole answer: a delegation flip. The senior expert handed the technical work down to a lean team, and by the time quality dropped, two people were doing one job, plus an audit neither wanted.
P
Pinpoint the old decision. Which choice only made sense before?
Removing Marisol's seat from the working team to move faster under a tight client deadline, keeping her only as a stakeholder who'd see a final summary.
Reasonable when the model's code looked like the obvious bottleneck. Wrong once the real bottleneck turned out to be knowing which defects were worth testing for.
S
Show the replay. Same trigger, new design.
The rerun spike includes Marisol in both dry-run sessions. All three previously missed defect types get caught and added to the test data before sign-off, instead of surfacing three weeks later in a client's own audit.
A countable result: 12 of 12 defect types caught in the dry run, up from 9 of 12, and zero surprises in the client's next audit.
The recap, one line per letter: find the person is Marisol, who used to review every initiative personally; locate the habit is sitting in the working sessions, not just reading a summary; identify the flip is a delegation flip, technical work handed down to a lean team with no domain seat; pinpoint the old decision is removing her seat to move faster under deadline pressure; and show the replay is a rerun spike catching all 12 defect types with her back in the room, instead of a client audit catching three of them later.
And if you want to be sure it really works, try it somewhere elseSame five letters, a recycling sorting line instead of a garment factory. Different flip family entirely, the same missing seat.
Dax Holloway works the sorting line at Harrow Bend Materials Recovery, where a new AI system flags likely contamination, food waste, hazardous items, mixed into recyclable material on the belt. The feasibility spike for that system was run entirely by an outside vendor's engineers, with no line worker consulted on which contamination types actually mattered most in practice. Mapped onto FLIPS: find the person is Dax, who has sorted this line for six years. Locate the habit is trusting his own eye for contamination types the model wasn't confident about. Identify the flip here is a workaround flip, not delegation: once the model started flagging things Dax knew from experience were false alarms, he quietly stopped acting on its flags at all and built his own private mental checklist instead, running his old process underneath the new tool without telling anyone. Pinpoint the old decision is the vendor's spike never including a line worker's review of which contamination types the model handled worst. Show the replay adds a rotating line-worker review seat to every contamination-type update, catching disagreements with Dax's own judgment before they turn into a system nobody actually trusts.
The same missing seat, in a completely different industry: the person who acts on the tool's calls was never in the room that built it.
Swap the trigger and it still runs.
Speed: an interviewer caps you at sixty seconds. Say "put the person who acts on the spike's answer inside the spike, not just the person who builds it," and stop.
Cost: there's no budget to add a domain expert's time to every spike. Say so honestly, and scope the smallest real version, a single half-day review session, rather than skipping the seat entirely.
The spike's first result looks great, for real: even a strong accuracy number is worth a domain expert's second look, since it can look great while quietly never having been tested against the one rare case that matters most.
Where people run it wrong.
They staff a spike purely for who can build it fastest, treating domain expertise as a review step instead of a seat at the table.
They let a clean benchmark number stand in for a real judgment call about what actually matters on the floor.
They add the domain expert only after something goes wrong, instead of before the spike ever runs.
How to use it live. The moment an interviewer asks about spike team composition, ask yourself: who, specifically, will act on this spike's answer once it's done, and are they in the room while it's being built? If not, that's the gap worth naming.
Flashcards (tap any card to flip it)
1 · THE FLIP FAMILY
What flip family is this?
Tap to flip
ANSWER
Delegation flip: technical work got handed down to a lean team without domain expertise, and once quality gaps surfaced, the senior expert had to take the whole review back personally, plus an audit.
2 · THE PERSON
Who is this answer about?
Tap to flip
ANSWER
Marisol Ferreira, who runs quality control at Loomcraft Textiles and used to sign off personally on every quality initiative before the defect-detection spike left her off the working team.
3 · THE HABIT
What did Marisol stop doing when the spike team was scoped without her?
Tap to flip
ANSWER
Sitting inside the working sessions and asking about the rare case. She was told a final summary would be enough this time.
4 · THE FLIP, IN THIS STORY
What's the two setting switch here?
Tap to flip
ANSWER
Reviewing the spike's working sessions in person, catching gaps as they happen, versus reviewing a finished summary after the fact with no way back into the room. No middle setting once the spike had already run.
5 · THE OLD DECISION
What decision would you take back?
Tap to flip
ANSWER
Removing Marisol's seat from the spike's working team to move faster under deadline pressure, keeping her only as a stakeholder who'd see a final summary.
6 · THE NUMBER
Fill in the blank: the spike team without a floor inspector caught ___ of 12 real defect types in the dry run; with one, it caught ___.
Tap to flip
ANSWER
9 of 12 without a floor inspector; 12 of 12 with one.
7 · THE REPLAY
Same spike, Marisol included in the dry-run sessions from the start. What changes?
Tap to flip
ANSWER
All three previously missed defect types, including the loose-collar-seam pattern, get caught and added to the training and test data before sign-off, instead of surfacing three weeks later in a client's audit.
8 · CROSS PRODUCT TRANSFER
Section 4 answers this same question again for a different product, with a different flip family. Which product, and which family?
Tap to flip
ANSWER
Harrow Bend Materials Recovery's contamination-detection line. The flip there is a workaround flip: line worker Dax Holloway quietly ignores the model's flags and runs his own private checklist underneath it.
Check yourself Score: 0 / 0
True or false
1. True or false: the spike's original 96 percent accuracy number was calculated incorrectly.
True
False
Show hint
Look at the knowledge spark about what an accuracy number actually measures.
Show answer
False. The number was calculated correctly against the sample it was tested on. The problem was that the sample never included the three missed defect types.
Multiple choice
2. Why did adding a floor inspector to the spike team catch defects the data scientists' sample missed?
A. The floor inspector rewrote the model's code to be more accurate.
B. The floor inspector knew, from years on the line, which rare defect types belonged in the test sample in the first place.
C. The floor inspector had access to a larger training dataset.
D. The client audit shared its findings with the data science team first.
Show hint
Look at the reframe stage in the walkthrough.
Show answer
B. The gap was never about coding skill; it was about knowing which real-world cases were worth testing for.
Fill in the blank
3. Fill in the blank: the client audit found a loose ___ seam defect that the model had never once flagged.
Show hint
Look at the story's description of the specific missed defect.
Show answer
Collar. The loose-collar-seam pattern was one of three defect types missing from the original spike's test sample.
Short answer, where it wouldn't matter
4. Name a situation where a spike would be fine without a floor inspector on the team.
Show hint
Look at "what I would leave alone."
Show answer
Model answer: An early, throwaway feasibility check using a public defect-image dataset, with no real production stakes yet, is fine for a data scientist to run alone.
Short answer, apply it yourself
5. Think of a project team you've been on where the person who'd actually use the result wasn't in the room while it was being built. What got missed because of that?
Show hint
Think about the gap between who can build something and who knows what it actually needs to catch.
Show answer
Model answer: A strong version names a specific missed case, the same way the loose-collar-seam defect is named here, not just a general feeling that something was off.
Short answer, work the number
6. If the spike team without a floor inspector had caught 11 of 12 defect types instead of 9, would adding the inspector still have mattered?
Show hint
Think about what one missed defect type can still cost, even if it's rare.
Show answer
Model answer: Yes. Even one missed defect type, if it's the loose-collar-seam pattern that fails under real wear, is still a real customer complaint and a real audit risk, not a rounding error.
Before you close the answer
Why this works
Tests whether you'll staff a spike purely for who can build it fastest, or notice that the person who'll act on the result belongs in the room while it's being built.
Follow-up traps
"Isn't this just normal cross-functional staffing, nothing AI-specific?" Response: no, because a model's accuracy score is only as good as the examples it's tested against, and a team without domain depth doesn't know which rare cases belong in that test sample at all.
"What if the domain expert doesn't have time to sit in every spike?" Response: scope the ask down, not away, even one half-day dry-run review session catches most of the gap a full-time seat would.
If pressed
The rebuilt test sample specifically included the three missed defect types at a rate matching their real, low frequency on the line, roughly 1 in 200 garments, rather than over-representing them just to make the model look thorough.
From U2xAI Academy
From answering questions to owning outcomes.
A live workshop where you ship a working AI agent, defend a launch decision, and walk away with a portfolio recruiters can't wave off, not just more questions to study.
