Your provider deprecates the model behind your main feature in 60 days. Write the plan.

How to answer this, stage by stage

Seven moves, in the order I'd actually say them. This question tempts you to sound busy. Say less, and make every sentence defend the order.

Let's learn

What do you do when the ground your feature stands on tells you it is closing in sixty days?

Say a marketplace lets sellers upload photos when they list something for sale. A model reads each photo and writes two things: a short caption a seller can use as is, and alt text, the words a screen reader says out loud, and the words an image search engine reads when nobody ever clicks the photo itself. Eighty five thousand new listings a day pass through this model, and almost nobody at the company thinks about it.

For three years it just worked. A seller uploads three photos of a lamp, and two seconds later a caption is already sitting in the box, and alt text nobody but a screen reader or a search crawler will ever read directly. Nobody in the company thinks about the model behind it, the same way nobody thinks about the wiring behind a light switch.

Then an email arrives from the company that owns the model. In sixty days, the thing this feature calls every time a photo lands stops answering. Not slower. Not worse. Off.

Here is the turn. Sixty days sounds like the emergency. It is not the real one.

Say the team panics and rewrites every caption and every alt text entry it already has, all forty six million of them, in the first two weeks, to look ahead of the deadline. The new model gets most of it right. But on about eight in every hundred listings it swaps a small detail: the wrong material, the wrong colour, a chair called a bench. No error, no crash, nothing that shows up on a dashboard that week. Weeks later, search engines re-crawl those listings and rank some of them worse, because the new text matches fewer of the words a shopper actually typed. Getting that ranking back is not a quick fix. It can take months, and some of it never fully comes back.

None of the sixty days start with picking a model

The first two days go to two things running side by side. Legal sends a data agreement to the two vendors worth testing, because a seller's photo can have a home address sitting on a shipping label in the background, and nobody sends that anywhere without a signature first. At the same time, a small team starts building a real eval set: five hundred real listing photo sets, across twelve categories, each one scored by a real person for what the caption should actually say.

Then the quiet part. The winning candidate runs against three thousand real photos a day for two weeks, in the background. No seller's caption changes yet. The team just watches how close the new model gets to what the eval set says is right, and the prompt gets a small fix or two along the way.

What I would leave alone. Somewhere between ten and twenty percent of listings sell things where the photo barely matters: screws, cables, a four pack of batteries. The caption reads almost the same sentence no matter which model writes it. I would not spend one engineering hour hardening that part of the pipeline in sixty days. If the new model gets those slightly wrong, nobody notices and nobody's search ranking moves.

The lesson. The team built this pipeline assuming a vendor might slowly get worse someday, in a way there would be time to notice. Nobody planned for the version where a company just hands you a date and turns the key. The lesson isn't to trust a vendor less. It's to know, before you need it, which parts of your product would survive that vendor disappearing on a Tuesday, and which parts wouldn't.

The email, and the two ways it could have gone

You don't need this to answer the question. Read it slower, when you want to feel why the order matters and not just recite it.

Femi Adeyemi has run the guts of Listing Assist for eight months, since the engineer who built it left for another job. He learned it the hard way, reading three years of old incident logs before he changed a single line, and in eight months he's kept it running without one real outage. People trust him with the pipeline because he's careful with it, not because he's loud about it.

For most of those eight months, Listing Assist was the one thing nobody in his stand up ever asked about. Eighty five thousand listings a day, captions and alt text already sitting there before a seller finished scrolling past the upload screen. Boring, in the good way.

Then, on a Tuesday during sprint planning, an email lands, forwarded by the account manager, subject line something like model retirement notice. Femi reads it twice. Sixty days, then the thing this whole pipeline calls goes quiet for good.

His director reads it too, and her first instinct sounds like a good one. "While we're touching this," she says, "let's just rewrite everything now, the whole back catalogue, with the new model, before anyone asks why we waited." It sounds proactive. It's the kind of line that reads well in a status update.

Femi says no, and this is the part worth hearing. He doesn't say no because rewriting is a bad idea in general. He says no because of what it touches.

He drew this on the whiteboard instead of arguing about it

A mistake in tomorrow's listings costs a click to fix, and you'll catch it in the rollout before it ever reaches a real seller. A mistake spread across forty six million already indexed listings hides until a search engine gets around to re-crawling it, weeks later, and by then it isn't a bug ticket, it's a slow bleed in traffic nobody can point a finger at.

She agrees, and the plan runs the other way instead. The legal request goes out that afternoon. The eval set starts the next morning, five hundred photo sets, twelve categories, two people scoring them by hand for most of a week. The shadow test runs for two weeks after that, quiet, nothing a seller ever sees. By day thirty five the shared layer is in, one clean seam where six services used to wire straight into the vendor. The rollout goes category by category, and by day fifty two every new listing is running on the new model, eight days of slack left before the old one goes dark for good.

The forty six million already published listings never move. They're exactly what they always were: still indexed, still fine, still doing the job they were written to do three years ago.

What the rewrite everything path would have cost instead: roughly eight in every hundred rewritten listings quietly wrong, a slow ranking drop nobody notices for three weeks, and a repair that would have eaten the back half of the year, not the sixty days they actually had.

The thing Femi would tell himself, looking back: they built this pipeline like the vendor would only ever get worse, slowly, in a way there'd be time to notice. Nobody planned for the version where the vendor just hands you a date.

ORDER, run against a calendar instead of a wishlist

This is a prioritisation question wearing a technical costume. FLIPS looks for the moment a person's behaviour snaps after something in the product changes. Nothing about anyone's behaviour has changed yet here, the notice only just landed. The whole question is what order things happen in before day sixty, so the framework is ORDER.

ORDER, worked against this deadline

Run it somewhere a signature matters more than a search rank

A hospital's radiology department dictates notes into a transcription model from an outside vendor, and the vendor announces it is shutting that model down in forty five days. Same shape of question, a different kind of already written record.

Flashcards (click a card to flip it)

Check yourself Score: 0 / 0