ConceptAdvancedAI Opportunity & Model Strategy / Build vs buy vs fine-tune decisions / #12
What data volume and quality do you need before fine-tuning is even worth considering?
BOUNDthe agreement number mattered more than the example count, and nobody was tracking it
Northtrail Outfitters sells outdoor gear online. ReasonTag reads the free text a customer types when they return something and tags the real reason: wrong size, damaged in transit, changed their mind. Perrin Adeyika is the AI PM deciding whether ReasonTag is ready to fine-tune, and Innes Fairweather is the data scientist whose blunt question in a planning call changed the whole estimate.
The direct answer
Don't fine-tune until two numbers both clear a bar: a real volume floor, a few thousand clean examples spread across every category, and a quality floor, two independent people tagging the same item the same way at least eight or nine times out of ten. Below that agreement number, more data does not help. It just teaches the model to be confidently wrong in the same pattern two disagreeing people already were.
Do this, in order
Check label agreement before counting examples at all.Why: a large pile of inconsistently tagged data trains a model to repeat the inconsistency, not fix it.
Set the volume floor per category, not as one company-wide total.Why: 3,600 total examples spread across 12 categories can still leave the rarest one with 40 real examples.
Test plain prompting against the same eval set you'd use for fine-tuning.Why: it shows whether a real gap exists before any training pipeline gets built.
Weigh the accuracy gain from fine-tuning against the cost of the training pipeline.Why: a few points of lift over prompting rarely earns its ongoing upkeep cost.
Re-check both floors whenever a new category gets added.Why: a 13th return reason resets the volume math and usually starts at zero agreed examples.
Sanity-check the total against something countable, like a normal week of returns.Why: catches an estimate that's quietly off by ten times.
How to answer this, stage by stage
Nobody is scoring whether you can name a big enough number. They're scoring whether you know there are two numbers, not one, and which of them actually breaks first.
Stage 1
Scope it to one real feature, with a real number already in hand
Say it like this
"Let's ground this. Northtrail Outfitters sells outdoor gear online. ReasonTag reads what a customer typed when they return something and tags the real reason, wrong size, damaged, changed their mind. Perrin's team has nine hundred labeled returns already and a plan to fine-tune next week."
Why this works
Keeps this from turning into an abstract lecture on data requirements with no real feature behind it.
Stage 2
Say your structure out loud before any numbers
Say it like this
"I'll run this as BOUND. Break it down, the actual equation behind 'enough data.' Own the numbers, my assumptions and where they come from. Use a range, not one falsely precise figure. Nail the sanity check, compare it to something countable. Direction, which assumption would swing the estimate most."
Why this works
Signals a real estimation method instead of a gut-feel number pulled from nowhere.
Stage 3
Reframe the question: volume was never the only number that mattered
Say it like this
"Everyone in that room was asking 'how many examples do we have.' Nobody was asking the harder question: if you gave the same nine hundred returns to two different taggers, would they tag them the same way? That number turned out to matter more."
Why this works
This is where the answer separates from a rule-of-thumb count with nothing behind it.
Stage 4
Give the one estimate: the actual equation and range
Say it like this
"Here's the math. Twelve return-reason categories, times a floor of about three hundred clean examples each, gets you to around thirty six hundred as a baseline. Add a buffer for categories that get confused with each other, wrong size versus not as described, and you're really looking at three to six thousand. None of that matters, though, unless two taggers agree on the label at least eighty five percent of the time."
Why this works
This is the direct answer, stated as real arithmetic instead of a vague "it depends."
Stage 5
Prove it with the compressed evidence
Say it like this
"At nine hundred examples, plain prompting already hit seventy eight percent on our eval set. Fine-tuning at that same stage only reached eighty one percent, barely worth the training pipeline. Once Innes fixed the tagging guide and the pool grew to four thousand two hundred examples at eighty eight percent agreement, fine-tuning reached ninety four, past the ninety two percent bar we needed to auto-route returns with nobody double-checking."
Why this works
Gives the interviewer a real before-and-after they can't easily poke holes in.
Stage 6
Name the AI-specific reasoning and the trade-off being accepted
Say it like this
"The honest reason volume alone didn't save us at nine hundred examples is that fine-tuning on inconsistently labeled data doesn't average out the disagreement, it bakes it in as confident-sounding output. We accepted months of extra labeling work and a slower launch in exchange for a model that could route returns with nobody checking every tag."
Why this works
This is the load-bearing judgment. It only makes sense because the model learns from the labels themselves, not from any generic software argument.
Stage 7
Say what skips this math, then close on one line
Say it like this
"I wouldn't run any of this for Northtrail's package-weight lookup, that's a plain database call, no labels involved. For ReasonTag, the answer holds: don't fine-tune until you clear a real volume floor and a real agreement floor, in that order, because the second one is the one people forget to check."
Why this works
Closes with real judgment about where this math doesn't apply, and restates the direct answer in one breath.
Let's learn
ReasonTag is a feature inside Northtrail's return flow that reads a customer's own words and tags the real reason an item is coming back, so the warehouse knows whether to restock it, refurbish it, or discard it.
The real fork was never "do we have enough data." It was two separate forks, stacked on top of each other.
Before this got checked, the planning doc for ReasonTag said one thing: nine hundred labeled returns, ready to fine-tune next week. Nine hundred sounded like plenty. Nobody in that document had written down what "enough" actually required.
Four requirements, and the plan only had a number for one of them.
Minimum labeled examples, built up from the equation
Month 3 poolCategory floorConfusion buffer
Nine hundred examples was less than a quarter of the recommended floor, before agreement even entered the picture.
When Perrin finally ran a quick audit, the real number wasn't the example count. It was this: handed the same hundred returns, two different taggers agreed on the label only 61 percent of the time. "Not as described" and "wrong size" kept getting swapped depending on who was tagging that day.
Knowledge spark: what is label agreement?
Give the same items to two people, separately, and check how often they pick the same tag. High agreement means the category is clear enough that a model has something consistent to learn. Low agreement means the humans themselves don't agree on the answer, so there's nothing stable underneath the label for a model to actually learn.
Same returns, same two taggers. The only thing that changed was a written guide for where one category ends and the next begins.
We didn't have a data problem. We had two people quietly disagreeing about what "not as described" even means.
Here's the turn: the nine hundred examples were never really the problem. Fine-tuning on them would have worked, technically, in the sense that the model would have learned something. What it would have learned was a coin-flip pattern between two disagreeing humans, delivered with total confidence.
Accuracy on the eval set, prompting versus fine-tuning, as the pool grew
Prompting onlyFine-tuned
At the first stage, fine-tuning barely beat prompting. The three-point gap wasn't worth a training pipeline. At the second stage, it was.
The choice I would take back
Launching ReasonTag's return categories without a written tagging guide, and letting whoever was on shift use their own judgment call on ambiguous returns. That made sense when Perrin was the only person tagging returns, since one person is automatically consistent with themselves. It stopped making sense the moment a second tagger joined and nobody checked whether the two of them agreed.
What I would leave alone: Northtrail's package weight and dimension autofill, a plain lookup against a product database, needs none of this. There's no label, no disagreement possible, nothing to fine-tune.
The lesson: a data floor has two numbers, not one. Volume tells you whether you have enough examples. Agreement tells you whether those examples mean anything consistent. Check agreement first, because no amount of volume fixes a category nobody can agree on.
Now here is the same thing as a story
The short version above is what you'd say out loud in the room. Read this one for what it actually felt like to nearly train a model on data that quietly disagreed with itself.
Perrin Adeyika had spent three months building ReasonTag's category list: twelve return reasons, pulled from a year of customer service notes. Nine hundred returns had been tagged by then, mostly by Perrin, working through the returns queue most mornings before the rest of the team logged on.
A labeled example isn't just text plus a tag. It's four things, and the plan only ever had two of them.
In a planning call, Innes Fairweather, the data scientist who'd be running the actual training job, asked the question nobody had: "You're not seriously fine-tuning on three hundred examples a category with no tagging guide, are you? Who else besides you has ever tagged one of these?"
The pool grew the whole time. What actually changed at month eight wasn't the count, it was the guide.
Nobody else had. Perrin asked two junior support reps to independently re-tag a random hundred returns from the existing pool, blind to each other's answers and to Perrin's original tags. They agreed with each other only 61 percent of the time. "Not as described" and "wrong size" were the two categories doing most of the disagreeing.
We weren't measuring how much data we had. We were measuring how much we agreed with ourselves.
Innes never had a fixed number for exactly when a category's boundary was clear enough to trust. It came down to a feeling with two settings: either the disagreement was random noise that a bigger tagging guide would fix, or the category itself was genuinely fuzzy and no guide would ever pin it down. Sixty one percent, with the same two categories colliding every time, said the first one.
Volume alone moved Northtrail from month one to month three. It took a written guide to actually clear the other axis.
Perrin and a support-ops lead spent two weeks writing a one-page guide: three worked examples per category, and a clear rule for the "not as described" versus "wrong size" collision, whichever the customer's own words named first. Re-tagging the same hundred returns against the new guide pushed agreement to 91 percent.
Back when ReasonTag's categories were first drafted, skipping a formal tagging guide wasn't an unreasonable call. Perrin was the only one tagging, so there was nothing to disagree with yet. It stopped being reasonable the moment a second person joined and nobody checked whether their tags matched.
Here's the replay: by month eight, the labeled pool had grown to 4,200 examples under the fixed guide, sitting at 88 percent agreement. Fine-tuning against that pool reached 94 percent on the held-out eval set, clearing the 92 percent bar the team had set before letting ReasonTag auto-route a return with nobody double-checking the tag.
One version of this story trains a model on nine hundred examples that two people can't even agree on, ships it anyway, and spends the next six months chasing down why the auto-routing keeps misfiring. The other spends two weeks on a tagging guide before touching a training job, and gets a number worth trusting on the other side.
What I'd tell myself, hearing Innes's question in that planning call: a big pile of labeled data feels like progress, but it's only progress if two strangers, handed the same item, would land on the same answer. Check that first. It's cheap, and it's the number that actually decides whether training is worth running at all.
BOUND, the five checks before anyone touches a training jobNot a script for talking yourself out of fine-tuning. BOUND is what tells you, honestly, whether the data underneath it can hold the weight.
B
Break it down. What's the actual equation behind "enough data"?
Minimum usable examples equals the number of categories times a per-category floor, plus a buffer for categories that get confused with each other. That volume only counts once a separate quality gate, label agreement, also clears its own bar.
Without stating the equation out loud, "enough data" stays a feeling instead of a number.
O
Own the numbers. Where does each assumption actually come from?
Twelve real return-reason categories, an assumed floor of about 300 clean examples per category, and an assumed agreement bar of 85 percent, based on how often two categories get confused with each other in practice.
This is the step most answers skip, stating a number with no source is a guess wearing a decision's clothes.
U
Use a range. What's the honest low and high?
3,000 examples if categories are cleanly separated, up to 6,000 if they overlap more. Agreement somewhere between 82 and 90 percent, not a single false-precision number.
A single number here would claim confidence nobody actually has yet.
N
Nail the sanity check. Does this survive a smell test?
3,600 examples is about 12 returns a day, every day, for a year, from one warehouse. Across Northtrail's four warehouses, that's roughly three months of normal return volume, not a special data-collection project.
If the number implied a multi-year data-collection effort for a routine feature, that would be the sign something's off.
D
Direction. Which single assumption swings the estimate most?
Label agreement. Below roughly 70 percent, no volume of data saves the fine-tune, because the model would just be learning to imitate two humans who disagree with each other nearly half the time.
This is the answer to the actual question: the number that decides everything is agreement, not example count.
The recap, one line per letter: break it down is the volume-times-categories-plus-buffer equation, own the numbers is stating where 300-per-category and 85 percent agreement actually come from, use a range is 3,000 to 6,000 examples instead of one false-precision figure, nail the sanity check is comparing it to a normal week of returns, and direction is naming label agreement as the one assumption that can make the whole effort pointless regardless of volume.
And if you want to be sure it really works, try it somewhere elseSame five letters, a school district instead of a retailer. This time the volume floor resets every semester, not the agreement rate.
Tobias Wrenfield is the AI PM at Fallowmere School District, which is piloting GradeNote, a tool that gives teachers a draft comment on a student's short-answer science quiz response before the teacher finalizes it. Mapped onto BOUND: break it down means the equation is examples times rubric criteria, roughly eight scoring criteria per quiz, times a floor of 250 examples each, for a baseline of 2,000. Own the numbers means naming that floor as an assumption borrowed from a similar grading-feedback pilot elsewhere, not a guarantee. Use a range gives 2,400 to 3,200 once a buffer for criteria that overlap is added, agreement assumed between 80 and 88 percent among teachers scoring the same answer. Nail the sanity check compares 2,400 examples to roughly three weeks of one grade level's quiz volume across the district, cheap to gather. Direction is different here from Northtrail's case: the swing factor isn't agreement, teachers turned out to agree reasonably well once shown the same rubric. It's category churn, the rubric changes every semester as units change, so the volume clock resets constantly in a way Northtrail's twelve stable return reasons never did.
Same fork as Northtrail's, but the branch that actually bites here is different: a rubric that resets every semester never accumulates enough examples to reach the floor.
Swap the trigger and it still runs.
Speed: an interviewer caps you at sixty seconds. Skip straight to "clear a volume floor per category and an agreement floor between two taggers, in that order, before fine-tuning is worth considering," and stop.
Cost: no budget for a formal tagging-agreement study. Say so honestly, and commit to a cheap version, ten items, two people, before spending anything on a training job.
The model got better, for real: say a future general model handles return-reason tagging at 90 percent with no fine-tuning at all. The volume and agreement floors still matter, they just become the bar prompting alone now has to clear instead.
Where people run it wrong.
They treat example count as the only number that matters, and never check whether two humans would label the same items the same way.
They set one volume floor for the whole dataset instead of per category, so a common category masks a rare one that's badly underrepresented.
They fine-tune anyway once "enough" data exists, without checking whether the accuracy gain over plain prompting is even worth the training pipeline's ongoing cost.
How to use it live. The moment an interviewer asks how much data you'd need, don't reach straight for a number. Ask yourself first: if I handed the same hundred items to two different people, would they tag them the same way? That question alone buys real thinking time, and it usually reveals which floor is actually missing.
Flashcards (tap any card to flip it)
1 · THE FRAMEWORK
What framework fits deciding whether enough data exists before fine-tuning?
Tap to flip
ANSWER
BOUND: break it down, own the numbers, use a range, nail the sanity check, direction. This question is a category H estimation question, so a flip family from FLIPS doesn't apply here.
2 · THE PEOPLE
Who is this answer about?
Tap to flip
ANSWER
Perrin Adeyika, the AI PM who owns ReasonTag at Northtrail Outfitters. Innes Fairweather is the data scientist whose question about tagging agreement reorders the whole plan.
3 · THE EQUATION
What's the actual build-up behind "enough data" here?
Tap to flip
ANSWER
Categories times a per-category floor, plus a confusion buffer, gated by a separate label agreement bar. 12 categories times roughly 300 each, plus a buffer, only counts once two taggers agree at least 85 percent of the time.
4 · THE RANGE
What's the honest low-to-high estimate, instead of one false-precise number?
Tap to flip
ANSWER
3,000 to 6,000 examples, with agreement somewhere between 82 and 90 percent, depending on how cleanly the categories separate from each other.
5 · THE OLD DECISION
What old decision does this answer take back?
Tap to flip
ANSWER
Launching ReasonTag's categories without a written tagging guide. Reasonable when Perrin alone did the tagging. Wrong once a second tagger joined and nobody checked whether their tags matched.
6 · THE NUMBER
Fill in the blank: two taggers agreed on the same return only ___ percent of the time, before the guide existed.
Tap to flip
ANSWER
61 percent. It rose to 91 percent after a one-page tagging guide resolved the "wrong size" versus "not as described" collision.
7 · THE REPLAY
Same nine hundred examples, checked for agreement first instead of fine-tuned immediately. What changes?
Tap to flip
ANSWER
Two weeks writing a tagging guide, agreement jumps to 91 percent, and the pool grows to 4,200 examples by month eight, at which point fine-tuning reaches 94 percent, clearing the 92 percent bar.
8 · CROSS PRODUCT TRANSFER
Section 4 runs BOUND again on a different product. Which one, and what swings the estimate there instead?
Tap to flip
ANSWER
Fallowmere School District's GradeNote. Unlike ReasonTag, agreement isn't the swing factor there, teachers agree reasonably well. It's rubric churn: the categories reset every semester, so volume never has time to build up.
Check yourself Score: 0 / 0
True or false
1. True or false: the real problem with ReasonTag's nine hundred examples was that there weren't enough of them.
True
False
Show hint
Look at what the audit of two taggers actually found.
Show answer
False. The real problem was label agreement, only 61 percent, not the count. More of the same inconsistent data would not have fixed it.
Multiple choice
2. Why did fine-tuning barely beat plain prompting at the 900-example stage?
A. The model being fine-tuned was too small for the task.
B. The labels themselves were inconsistent, so there was nothing stable underneath for fine-tuning to learn.
C. Nine hundred examples is always too few to fine-tune on, regardless of the task.
D. The eval set used for testing was too small to be reliable.
Show hint
Look at the label-agreement audit and what it found about the "not as described" versus "wrong size" categories.
Show answer
B. At 61 percent agreement, the labels themselves disagreed with each other nearly 4 times in 10, so fine-tuning had no consistent pattern to learn beyond what prompting already caught.
Fill in the blank
3. Fill in the blank: the recommended minimum, twelve categories at a 300-example floor plus a confusion buffer, came to about ___ examples.
Show hint
Look at the stacked bar chart in "Let's learn."
Show answer
4,200 examples. 3,600 from the category floor, plus a 600-example buffer for categories that get confused with each other.
Short answer, where it wouldn't matter
4. Name a feature at Northtrail where this entire data-floor question would NOT apply, and say why not.
Show hint
Look at "what I would leave alone."
Show answer
Model answer: Package weight and dimension autofill. It's a plain lookup against a product database, no labels and no disagreement possible, so there's nothing to fine-tune in the first place.
Short answer, apply it yourself
5. Think of a classification task you've seen proposed for fine-tuning. Would two people handed the same examples actually agree on the label? How would you check, cheaply?
Show hint
Think about a category boundary that's genuinely fuzzy versus one that's just never been written down clearly.
Show answer
Model answer: A support-ticket urgency classifier: hand the same 50 tickets to two agents separately and compare their urgency picks before assuming the existing tags are trustworthy enough to train on.
Short answer, work the number
6. If agreement had stayed at 61 percent even after the pool grew to 4,200 examples, would fine-tuning still be worth trying?
Show hint
Think about what fine-tuning actually learns from when the labels themselves are inconsistent.
Show answer
Model answer: No. More volume at the same low agreement rate just gives the model more examples of the same inconsistency to learn from. The honest next step would still be fixing the tagging guide, not adding more data.
Before you close the answer
Why this works
Tests whether you know "enough data" is two separate bars, volume and label agreement, and whether you'd actually check the harder one before committing to a training pipeline.
Follow-up traps
"Couldn't you just have three taggers vote and use the majority label?" Response: voting hides the disagreement instead of fixing it, the model still learns from an averaged-over coin flip unless the category boundary itself gets clarified in a guide.
"Isn't 85 percent agreement an arbitrary bar?" Response: it's an assumption stated out loud, not a law, tied to how often the two hardest-to-tell-apart categories were actually colliding in Northtrail's own data, and it should move if that collision rate changes.
If pressed
The eval set used to measure both prompting and fine-tuning was held out and re-tagged by a third, independent reviewer, specifically so the 92 percent bar wasn't measured against the same inconsistent labels the model might have been trained on.
From U2xAI Academy
From answering questions to owning outcomes.
A live workshop where you ship a working AI agent, defend a launch decision, and walk away with a portfolio recruiters can't wave off, not just more questions to study.
