Tell me about an AI feature you would kill today if you owned it.

Picwell is a photo-sharing app. CaptionMuse suggests three AI-written captions the moment someone posts a photo. Fenwick Osata is the AI PM who owned it, and this is the feature Fenwick would kill today, with real usage data behind the decision, not a hunch.

How to answer this, stage by stage

Nobody is scoring whether you can name a feature you dislike. They're scoring whether killing it is backed by real evidence, and whether you'd say it out loud about your own work.

Let's learn

CaptionMuse is a feature on Picwell that writes three suggested captions the instant someone posts a photo, meant to save people from staring at a blank caption box.

Nothing about the first month's dashboard predicted this. It took someone actually checking the right number.

At launch, CaptionMuse looked like a clear win. Suggestion impressions climbed steadily. Time spent on the post-composing screen went up too, which the team read as engagement. Nobody was tracking the one number that actually mattered: how many suggestions got used, unedited, as written.

Two metrics that looked like success, and the one that would have told the real story, missing entirely.

When Fenwick finally pulled the real number, six months in, it was 4 percent. Not a typo, not a rounding error. Ninety-six times out of a hundred, someone saw three AI-written captions and used none of them as written.

So Fenwick checked whether 4 percent was still ramping up, the way a new feature sometimes does before people discover it. It wasn't. The number sat at 4 percent in month one, 5 percent in month three, 4 percent again in month six. Flat, not early, just genuinely low.

Here's the turn: CaptionMuse was never badly built. Its suggestions were often genuinely decent, sometimes funny, occasionally exactly what someone might have written themselves. The turn is that "decent suggestion" and "something I want to actually use" turned out to be two different bars, and almost nobody was clearing the second one.

At its worst, this cost showed up as a real, ongoing inference bill, tens of thousands of dollars a month, spent generating three captions for every single post on the platform, for a feature 96 percent of people never used as intended. That budget had a real alternative use the whole time.

What I would leave alone: Picwell's AI-powered photo enhancement, sharpening and lighting correction, stays exactly as it is. Nobody's identity is wrapped up in whether their photo got auto-brightened, so there's no authenticity cost the way there is with someone else's words standing in for your own.

The lesson: a feature that produces genuinely good output can still be solving the wrong problem. Track whether people actually use what the model made, not just whether they saw it, and don't let six months pass before checking.

Now here is the same thing as a story

The short version above is what you'd say out loud in the room. Read this one for what it actually felt like to watch a feature everyone had called a launch success quietly fail a much simpler test.

Fenwick Osata had shipped CaptionMuse eight months earlier to genuine excitement. The launch review had gone well: impressions up, session time up, a few glowing internal Slack messages about how funny some of the generated captions were.

Marlone Byfield, an engineer on the team, asked an offhand question in a retro: "Do we actually know how many people use the suggestions as-is?" Nobody in the room had that number. It had never been on the dashboard.

Three honest guesses, before anyone had the real data to tell which ones were right.

Fenwick pulled it that afternoon, expecting something respectable, maybe 20 or 30 percent. The real number was 4. Not a bad week. Six months, averaged, flat the whole time.

The first instinct, understandably, was to assume the suggestions just weren't good enough yet. Fenwick's team ran a quick internal review of a sample and found most of the captions were fine, sometimes genuinely funny, occasionally better than what people ended up writing themselves.

Not a quality problem. A problem with what the feature had assumed the friction actually was.

So Fenwick did the thing that should have happened at launch: talked to real users. Not a survey, actual short conversations with a dozen people who'd used Picwell that week. The pattern was consistent and, in hindsight, obvious. "It's not really mine if the app wrote it." "I don't struggle with the caption, I struggle with whether the photo's even good enough to post." Nobody said the suggestions were bad. Almost everyone said using one felt wrong.

Fenwick never had a fixed threshold for when a shipped feature's usage number crossed from "still finding its audience" to "solving the wrong problem." It came down to a feeling with two settings: the number is still moving, or it's found its real, flat ceiling. Six unchanged months made that second setting unmistakable.

The fork the evidence actually pointed to, once someone finally looked.

Back at launch, shipping CaptionMuse default-on for everyone wasn't an unreasonable call. The team's working theory, that a blank caption box was the real blocker, was a genuinely plausible guess at the time, with no data yet to contradict it. It stopped being reasonable the moment six months of flat, real usage said the theory was wrong, and nobody had checked.

Killing the feature was never the end of the story. What replaced it is the part that actually mattered.

Here's the replay: CaptionMuse's default-on suggestions come off every post. A small, opt-in "give me an idea" button stays for the 4 percent who genuinely want it. The inference budget that used to write three unused captions per post gets redirected toward a "which of your last three photos is your best shot" comparison tool, aimed squarely at the friction the interviews actually surfaced.

One version of this story keeps a quietly unused feature running for years because its impressions chart never dipped. The other spends one honest afternoon of user interviews and redirects real budget toward the problem people were actually stuck on the whole time.

What I'd tell myself, hearing Marlone's offhand question in that retro: a dashboard full of numbers that look good will never tell you the one thing that matters, whether people actually wanted what you gave them. Somebody has to ask that question on purpose, because the launch metrics never will.

TRACE, run on a feature that looked fine on every chart except the one that matteredNot a script for finding fault with old work. TRACE is what turns "I have a hunch this isn't working" into a real, defensible decision to kill it.

The recap, one line per letter: timeline is the six-month gap between launch and actually checking the real number, recut is separating suggestion quality from usage, assume nothing is the trap of reading impressions as proof of real value, cause candidates is the honest shortlist of what might explain the flat 4 percent, and evidence test is the real user conversations that confirmed which one was true.

And if you want to be sure it really works, try it somewhere elseSame five letters, a fitness app instead of a photo app. This time the kill decision is about a workout-plan generator, not a caption writer.

Priya Wentz owns product at Pacefield, a fitness app. Pacefield shipped an AI feature that generates a full personalized workout plan the moment someone signs up. Mapped onto TRACE: timeline is that signup completion and initial plan-view rates looked strong for months, while the real number, how many people were still following their AI-generated plan after week three, sat unchecked. Recut is that the plans themselves tested as reasonable by trainers reviewing them, the real issue wasn't quality. Assume nothing means not trusting "plan generated" as proof the plan got used. Cause candidates are plan quality, plan rigidity (no easy way to adjust it as life happened), or the real problem being accountability, not planning. Evidence test is user interviews finding that people who quit weren't confused by their plan, they'd simply missed a day, felt they'd "broken" the AI's schedule, and never came back, the same authenticity-adjacent gap as CaptionMuse, but about commitment instead of voice. The kill decision replaces the rigid full-plan generator with a lighter, adjustable week-by-week suggestion that expects and absorbs missed days.

Swap the trigger and it still runs.
Speed: an interviewer caps you at sixty seconds. Skip straight to "the real usage number was flat and low for six months, here's what people actually wanted instead," and stop.
Cost: no time to run real user interviews before this conversation. Say so honestly, and name the interviews as the concrete next step rather than guessing at the cause.
The model got better, for real: say a future version of CaptionMuse wrote genuinely indistinguishable-from-human captions. The authenticity problem likely persists anyway, since the issue was never really about writing quality.

Where people run it wrong.
They track impressions and engagement time as proof of value, without ever checking whether the AI's actual output got used.
They assume a low usage number means the model needs to improve, instead of checking whether it's solving the right problem at all.
They treat killing a shipped feature as an admission of failure, instead of the accountable move once real evidence says it's not working.

How to use it live. The moment an interviewer asks about a feature you'd kill, don't reach for something you personally dislike. Reach for the one where the real usage number, not the impressions number, told a story nobody wanted to look at closely. That's usually where the honest answer is.

Flashcards (tap any card to flip it)

Check yourself Score: 0 / 0