ConceptIntermediateAI Opportunity & Model Strategy / When NOT to use AI / #22
List the questions you would ask before agreeing to any AI feature request.
SPARKsix questions decided both what to build and what to say no to, in the same meeting
Verrly builds ContractSift, a tool that flags risky clauses in vendor contracts before a lawyer signs off. Baldassare Yun, a team lead, pitches a version of it that flags "anything different from our standard." Oleander Frisch is the AI PM who runs every AI feature request, including this one, through the same six questions before agreeing to anything.
The direct answer
Ask, every time, before agreeing to anything: what's the real problem underneath the pitch, does usable data actually exist, what's the cost of a wrong output and can it be caught before real harm, is a genuinely simpler rule already enough, what would make this technically infeasible and has that been checked, and what's the real opportunity cost against everything else the same team capacity could build. Apply the same six questions every time, fast, and let the honest answers decide, not the enthusiasm in the room.
Do this, in order
Ask what the real problem is, separate from the specific AI solution being pitched.Why: a request usually names a solution before anyone's confirmed what result it's actually meant to produce.
Check whether usable data or signal genuinely exists for it.Why: no real data means no real model, no matter how good the pitch sounds.
Ask what a wrong output costs, and whether it gets caught before real harm.Why: an error that's cheap and catchable is a very different bet than one that's expensive and silent.
Check whether a simpler rule or template already does the job.Why: the AI-specific cost and risk only get justified once a simpler approach has genuinely been ruled out.
Ask what would make this technically infeasible, and whether that's been checked.Why: naming the failure condition up front catches a bad bet before real weeks get spent on it.
Weigh the real opportunity cost against everything else the same team capacity could build.Why: a good idea can still lose to a better use of the exact same engineering time.
How to answer this, stage by stage
Nobody is scoring whether you can recite six questions from memory. They're scoring whether you'd have caught the vague version of the pitch before it became three wasted weeks.
Stage 1
Scope it to one real request, not intake process design in the abstract
Say it like this
"Let me ground this. Say a team lead pitches an AI feature that flags 'anything different from our standard contract.' That's vague enough to mean almost anything, and it's a real example of the kind of request this checklist exists for."
Why this works
Keeps the answer from turning into abstract intake-process theory nobody can picture.
Stage 2
Say your structure out loud before any content
Say it like this
"I'll run this as SPARK. Situation, how requests get evaluated today. Payoff, the habit I want this checklist to build. Anchor, the actual six questions. Risk, what happens without them. Keep out, what this checklist deliberately isn't."
Why this works
Signals a repeatable intake method instead of a personal gut check dressed up as rigor.
Stage 3
Say plainly what's wrong with evaluating requests today
Say it like this
"Right now, requests get evaluated on how convincing the pitch sounds in the room. 'Flag anything different from standard' sounds reasonable out loud. It's also not a real scope, there's no data behind 'different,' and no bar for what counts as a miss."
Why this works
Shows the real gap the checklist is fixing, not just asserting that a checklist would help.
Stage 4
State the anchor: the six questions, asked out loud
Say it like this
"Here's what I'd actually ask, in the room, before agreeing to anything. What's the real problem underneath this? Does usable data exist? What does a wrong flag cost, and would we catch it before signing? Would a simpler rule already do this? What would make it infeasible? And what else could this same team time build instead?"
Why this works
This is the direct answer, made concrete enough to actually say out loud in a real meeting.
Stage 5
Prove it with the compressed example
Say it like this
"Running Baldassare's pitch through this: the real problem was 40 contracts a month costing lawyers 3.5 hours each. Real data existed, 1,200 past contracts with the legal team's own risk flags. A keyword rule, tested, only caught 45 percent of known risky clauses. That's how 'flag anything different' became a scoped, testable feature instead of a vague ask."
Why this works
This is where the story lives, compressed to the moment the checklist actually changed what got built.
Stage 6
Name why these specific questions are AI-specific, not generic intake
Say it like this
"These aren't generic project-intake questions with AI slapped on as an example. 'Does usable data exist' and 'what would make this infeasible' are questions that only make sense because a model needs real examples to learn from and can fail in ways a hand-written rule simply can't. A checklist for a database migration wouldn't ask either one."
Why this works
This is the load bearing judgment, distinguishing this from a repurposed general project checklist.
Stage 7
Say what this isn't, then close on one line
Say it like this
"This isn't a gate built to say no to everything. Baldassare's request passed, once it got scoped around real data and a real cost bar, and it shipped inside a month. The point is getting real answers fast, not slow-walking every idea out of habit."
Why this works
Closes with real judgment instead of turning the checklist into a bureaucratic wall, and restates the direct answer in one breath.
Let's learn
ContractSift is a tool meant to flag risky clauses in vendor contracts before a lawyer signs off, so legal doesn't have to manually cross-reference every contract against the company's own playbook by hand.
The real workflow the request was actually about, before anyone had scoped a single AI feature.
Before the checklist existed at Verrly, AI feature requests got evaluated on how convincing the pitch sounded in the room. Baldassare's original ask, "flag anything different from our standard contract," was exactly that kind of request: reasonable-sounding, and not actually a scoped problem yet.
Catch rate on known risky clauses, keyword rule versus tested model
Simpler rule, tested firstModel, only after the rule fell short
This test is exactly question four, "is a simpler rule already enough," run for real before anyone committed to building a model.
Running it through the checklist changed the shape of the request entirely. The real problem, question one, turned out to be 40 contracts a month costing legal 3.5 hours each, real, recurring, and worth solving. Question two found real data: 1,200 past contracts, already flagged by the legal team itself. Question four tested whether a simple keyword rule would do, and it caught less than half of known risky clauses, genuinely not enough.
The checklist didn't slow the idea down. It turned a vague, unfundable pitch into a scoped, testable feature in the same meeting.
Here's the turn: the checklist isn't a filter built to catch bad ideas and reject them. The turn is that most requests aren't bad, they're just underspecified, and the same six questions that kill a genuinely bad request are what turn a promising one into something a team can actually build and defend.
Knowledge spark: why does "does data exist" matter more for AI requests than other project requests?
A model learns its behavior from real examples. No usable data means no real way to build or evaluate it, no matter how clear the pitch is. A regular software feature, like a new settings page, doesn't need labeled historical examples to get built, so this question is specific to requests that need a model to learn something.
Legal team hours spent on contract review per month, before and after launch
Before launchAfter launch
This is what question one's "real problem" turned into, once it actually shipped: a real, measurable drop in recurring hours.
At its worst, skipping this checklist would have cost real engineering weeks building a version of ContractSift with no clear bar for success, "different from standard" with no data behind it and no way to test whether it was working.
The choice I would take back
Evaluating earlier AI feature requests mostly on how convincing they sounded in a meeting, with no consistent set of questions applied to all of them. That worked fine when requests were small and cheap. It stopped working once a vague, well-pitched request could burn real weeks before anyone checked whether the data even existed.
What I would leave alone: a request for a small UI tweak inside ContractSift, like reordering which flagged clauses show first, doesn't need this full checklist. The six questions are for requests that need a model to learn something new, not every product change that happens to touch an AI feature.
The lesson: most AI feature requests aren't bad ideas, they're underspecified ones. The same six questions that catch a genuinely bad request are what turn a promising one into something real, fast, in the same meeting it was pitched.
Now here is the same thing as a story
The short version above is what you'd say out loud in the room. Read this one for what it actually felt like to watch a vague pitch turn into a real, scoped feature in twenty minutes.
Oleander Frisch had run Verrly's AI roadmap for two years, long enough to have watched a few requests burn real weeks before anyone checked whether they were actually buildable. The six questions weren't a policy handed down from above, they were the thing Oleander started asking after the third time a promising-sounding request turned out to have no real data behind it.
Baldassare's pitch landed on a Tuesday, in the middle of a roadmap review. "We should flag anything in a vendor contract that's different from our standard template. Lawyers keep missing stuff." Heads nodded. It sounded reasonable.
The actual questions asked in that room, before anyone committed to anything.
Oleander didn't say yes or no. "What's actually going wrong today, concretely?" Baldassare paused, then landed on something real: forty contracts a month, legal spending about three and a half hours each cross-referencing a forty-page playbook by hand.
"Does data exist for what 'risky' actually looks like?" This one had a real answer too: twelve hundred past contracts, each already flagged by the legal team itself over the years. Usable, real, sitting in a shared drive nobody had thought to point a model at yet.
Same underlying goal, twenty minutes apart. The difference is entirely in what got asked in between.
The next question was the one that actually reshaped the pitch: "would a simple rule, flag anything matching known risky wording, already do this?" Nobody in the room knew. Oleander asked for a day to test it against the twelve hundred contracts.
We weren't testing whether Baldassare's idea was good. We were testing whether it was actually the shape it had been pitched as.
The keyword rule caught 45 percent of known risky clauses. Not nothing, genuinely useful as a baseline, but far short of what legal actually needed to trust it. That gap was the real evidence a model was worth building here, not a hunch, a real, measured shortfall.
Oleander never had a fixed rule for what made a request worth greenlighting on the spot versus sending back for more scoping. It came down to a feeling with two settings: the honest answers to the six questions add up to something real, or they don't yet. Baldassare's did, once they'd actually been asked out loud.
Even a well-scoped feature needs a plan for the day it's wrong. That plan got built into the design from the start, not bolted on after a real miss.
Back before the checklist existed, evaluating requests mostly on how convincing they sounded wasn't an unreasonable habit, most early requests really were small and cheap to be wrong about. It stopped being safe the moment a request could plausibly cost real engineering weeks before anyone checked whether it was even buildable.
The fork the checklist actually runs, every single time a request comes in.
Here's the replay: same pitch, same Tuesday, but with the six questions run before any commitment gets made. Twenty minutes later, the room has a scoped feature, a real precision bar to test against, and a required human sign-off step for the cases it gets wrong. It ships within a month, not because it was rushed, but because it started scoped instead of vague.
One version of this story spends real weeks building "flag anything different" and discovers, too late, there was never a real bar for success. The other spends twenty minutes asking six honest questions and ships something real inside a month.
What I'd tell myself, watching that first keyword-rule test come back at 45 percent: the questions were never about slowing anyone down. They were about finding out, fast, whether the exciting version of an idea was also the true one.
SPARK, run on a pitch before it ever became a commitmentNot a script for saying no to every AI feature request. SPARK is what turns a vague, well-meant pitch into either a real feature or an honest pass, in the same meeting.
S
Situation. What's actually true about how requests get evaluated today?
AI feature requests at Verrly used to get judged on how convincing the pitch sounded in the room, with no consistent set of questions applied across all of them.
A real, honest description of the habit, not a strawman version of it.
P
Payoff. What habit do I want this checklist to build, every time?
Not a verdict on any one request. The habit is asking the same six real questions before agreeing to anything, so requests get evaluated on substance instead of how well they're pitched.
Every future request starts with the same six questions, not a fresh improvisation.
A
Anchor. The one concrete thing everything else hangs on.
Six questions, asked in order: real problem, real data, cost of a wrong output, is a simpler rule enough, what makes it infeasible, and what else the same capacity could build.
This is the direct answer to the question, as an actual list you could say out loud in a real meeting.
R
Risk. What breaks the first time this checklist gets skipped?
A vague, well-pitched request burns real engineering weeks before anyone checks whether the data exists or a simpler rule would already do the job.
The cost of skipping the checklist doesn't show up in the pitch meeting, it shows up weeks later.
K
Keep out. What this checklist deliberately isn't.
Not a gate built to slow-walk every idea out of habit. Baldassare's request passed, once it was scoped, and shipped inside a month. A small UI tweak doesn't need the full six questions either.
Keeps the checklist from becoming its own kind of bureaucratic drag on genuinely good, well-scoped ideas.
The recap, one line per letter: situation is the honest habit of judging requests by how convincing they sound, payoff is the habit of asking the same six questions every time, anchor is the six questions themselves, risk is the real cost of skipping them, and keep out is making sure the checklist stays a fast filter, not a wall.
And if you want to be sure it really works, try it somewhere elseSame five letters, a customer-onboarding checklist tool instead of a contract reviewer. This time the request gets a fast no, not a scoped yes.
Rosalind Achebe runs product at Lanternhouse, a project-management app. A sales lead pitches an AI feature that predicts which trial users will churn, so the team can proactively reach out. Mapped onto SPARK: situation is that requests, like Verrly's, get evaluated on pitch quality alone today. Payoff is the same habit, asking the six questions before agreeing to anything. Anchor is running them: the real problem turns out to be real, lost trial conversions are worth real revenue. Real data exists, eighteen months of trial usage logs. But the cost of a wrong output is high and silent, a false "not at risk" prediction means a real at-risk user simply never gets an outreach at all, with nobody noticing the miss until the user has already churned. A simpler rule, flagging users with zero logins in five days, already catches most real churn signals in a quick test. Risk is greenlighting a complex model for a problem a simple rule already mostly solves. Keep out means this isn't rejecting the sales lead's real insight, churn is worth acting on, just not with the specific tool first proposed. Rank ends in a pass on the model, a yes on the simple rule, shipped the same week.
Swap the trigger and it still runs.
Speed: an interviewer caps you at sixty seconds. Skip straight to the six questions, said fast, and stop.
Cost: no time to test a simpler rule before the meeting. Ask the question anyway, and commit to testing it as the very next step before building anything bigger.
The model got better, for real: say a future model version could genuinely outperform any rule on this same task. Rerun the six questions anyway, a better model doesn't change whether real data exists or what a wrong output costs.
Where people run it wrong.
They evaluate requests on how exciting the demo sounds, instead of asking whether real data and a real cost bar exist.
They skip checking for a simpler rule, assuming a model must be the more sophisticated, therefore better, choice.
They treat the checklist as a way to say no to everything, instead of a fast path to a real yes when the answers support it.
How to use it live. The moment an interviewer describes any AI feature request, ask yourself first: does real data exist for it, and what does a wrong output actually cost. Those two questions alone buy real thinking time, and they're usually where the honest answer about the whole request is hiding.
Flashcards (tap any card to flip it)
1 · THE FRAMEWORK
What framework fits building a consistent checklist for evaluating AI feature requests?
Tap to flip
ANSWER
SPARK: situation, payoff, anchor, risk, keep out. It turns "how convincing was the pitch" into six consistent, real questions applied every time.
2 · THE PEOPLE
Who is this answer about?
Tap to flip
ANSWER
Oleander Frisch, the AI PM who runs the six-question checklist at Verrly. Baldassare Yun is the team lead whose vague pitch gets scoped through it.
3 · THE HABIT
What habit does this answer's payoff exist to build?
Tap to flip
ANSWER
Ask the same six real questions before agreeing to any AI feature request, so requests get evaluated on substance, not how well they're pitched.
4 · THE ANCHOR
What are the six questions, in order?
Tap to flip
ANSWER
What's the real problem, does real data exist, what does a wrong output cost and is it catchable, is a simpler rule already enough, what would make this infeasible, and what's the real opportunity cost.
5 · THE OLD DECISION
What old decision would this answer take back?
Tap to flip
ANSWER
Evaluating earlier AI feature requests mostly on how convincing they sounded in the room, with no consistent questions applied. Fine while requests were small and cheap. Wrong once a vague, well-pitched one could burn real weeks.
6 · THE NUMBER
Fill in the blank: a simple keyword rule caught only ___ percent of known risky clauses, versus ___ percent for the tested model.
Tap to flip
ANSWER
45 percent, versus 89 percent. The real evidence, from question four, that a model was genuinely worth building here.
7 · THE RISK, SURVIVED
What breaks if the checklist gets skipped, and how does the anchor survive it?
Tap to flip
ANSWER
A vague, well-pitched request burns real engineering weeks before anyone checks the data or a simpler alternative. Asking the six questions up front catches this in the same meeting, not weeks later.
8 · CROSS PRODUCT TRANSFER
Section 4 runs SPARK again on a different product. Which one, and how does the outcome differ?
Tap to flip
ANSWER
Lanternhouse's churn-prediction request. Unlike Baldassare's, this one gets a fast no on the model, since a simple login-based rule already catches most of the same signal, with the simple rule shipped instead.
Check yourself Score: 0 / 0
Multiple choice
1. Why did Baldassare's original pitch, "flag anything different from our standard," need the checklist before it could be built?
A. Oleander disagreed with the underlying goal of catching risky clauses.
B. It had no real scope yet, no confirmed data, and no bar for what counted as success or failure.
C. Verrly's leadership had banned all new AI features that quarter.
D. Baldassare hadn't gotten legal's approval to pitch it.
Show hint
Look at Stage 3 of the walkthrough.
Show answer
B. The pitch was underspecified, not bad. The checklist's questions turned it into something scoped and testable rather than rejecting the underlying goal.
True or false
2. True or false: the six questions in this checklist would apply just as well to a request for a new settings page with no AI involved.
True
False
Show hint
Look at Stage 6 of the walkthrough and the knowledge spark in "Let's learn."
Show answer
False. Questions like "does usable data exist" and "what would make this infeasible" are specific to requests that need a model to learn something, not a generic project checklist with AI as an example.
Fill in the blank
3. Fill in the blank: legal review hours dropped from about ___ hours a month before launch to about ___ hours a month by month 4.
Show hint
Look at the line chart in "Let's learn."
Show answer
140 hours, down to 54 hours. The real, measurable outcome that question one's "real problem" turned into once the feature actually shipped.
Short answer, where it wouldn't matter
4. Name a type of request at Verrly that would NOT need the full six-question checklist, and say why not.
Show hint
Look at "what I would leave alone."
Show answer
Model answer: A UI tweak, like reordering which flagged clauses display first. It doesn't need a model to learn anything new, so the AI-specific questions don't apply.
Short answer, apply it yourself
5. Think of an AI feature request or idea you've heard pitched. Run it through two of the six questions. What did you find?
Show hint
Pick "does usable data exist" and "is a simpler rule already enough" as a quick pair to test.
Show answer
Model answer: A pitch for an AI tool to auto-tag support tickets by urgency turned out to have real historical data (past ticket resolutions), but a simple rule based on keywords like "down" or "urgent" already caught most of the same signal in a quick check.
Short answer, work the number
6. If the keyword rule had caught 85 percent of known risky clauses instead of 45 percent, would the checklist still support building the model?
Show hint
Compare 85 percent against the model's own 89 percent, and think about what the gap would need to justify the added AI-specific cost and risk.
Show answer
Model answer: Probably not as clearly. An 85-versus-89 gap is small enough that the simpler, cheaper, more explainable rule might already be "enough" under question four, especially once the model's added maintenance cost is weighed in.
Before you close the answer
Why this works
Tests whether you have a consistent, repeatable way to evaluate AI feature requests, rather than a case-by-case gut feeling, and whether the questions themselves are genuinely specific to what makes an AI request different from any other product request.
Follow-up traps
"Doesn't this checklist just slow every request down?" Response: Baldassare's request passed and shipped inside a month, the checklist took twenty minutes and turned a vague pitch into a scoped, fundable one, it didn't add a bureaucratic delay.
"What if leadership wants to skip the checklist for a pet project?" Response: run it anyway, quickly, and bring the honest answers to that conversation, a real gap in the data or a real cost of being wrong is easier to act on than a vague objection.
If pressed
The keyword-rule benchmark used the same 1,200 historical contracts as the eventual model's training data, so the 45-versus-89 comparison was a fair, apples-to-apples test, not a rule tested on an easier or smaller sample.
From U2xAI Academy
From answering questions to owning outcomes.
A live workshop where you ship a working AI agent, defend a launch decision, and walk away with a portfolio recruiters can't wave off, not just more questions to study.
