How would you handle a rollout where cost scales faster than expected?

How to answer this, stage by stage

This is a yes or no about whether to pause or keep shipping while you fix it, not a rule for every cost line an AI feature ever runs over, so PICK carries the answer.

One more thing before the walkthrough ends: this isn't a blanket rule that every cost overrun needs a pause. Most candidates hear "costs more than planned" and reach for the same caution every time. Name whether the overrun gets better or worse as more people use the feature, and you've shown judgment instead of reciting a rule of thumb.

Let's learn

Here's what happens when a feature works so well that success itself becomes the expensive part.

Palette Foundry is a design tool. Inside it sits a feature called Skyfill: pick an empty patch of your canvas, type what you want there, and it paints something that matches the rest of your design.

The team built Skyfill with two models working together. A fast, cheap model paints the first draft. If that model isn't sure the fill will match the rest of the canvas, it quietly hands the job to a slower, much pricier model for a second pass, no button, no ask, it just happens. Before launch, the team tested this hand-off on 400 of their own design files, clean, finished, one style each. On those files, the hand-off fired 8 times in 100. At that rate, the cost model said Skyfill would run about $18,000 a month once a quarter of all users had it.

Real users are not working from finished files. A canvas mid-project is messy: three unfinished layers, two clashing styles, a placeholder nobody has replaced yet. The cheap model reads all of that as low confidence far more often, so the hand-off isn't firing 8 times in 100. It's firing on about half of every real fill. At the 25 percent stage, that puts the actual bill at about $63,000 a month. Three and a half times the plan.

Here is the turn. Three and a half times isn't the real problem. The real problem is that the more Skyfill gets used exactly the way it was meant to be used, on real, messy, in-progress work, the worse this number gets, because that's exactly the kind of canvas that trips the hand-off. Usage climbing was supposed to be the win. Instead, usage climbing is what makes it more expensive, at every stage still ahead.

At its worst: if nothing changes and Skyfill reaches everyone, the plan expected about $72,000 a month. The real pace works out to about $252,000 a month, for a feature that was supposed to be a normal line item, not the biggest number on the infrastructure bill.

What I would leave alone. The part of Skyfill that never needs any of this: filling a plain, mostly-empty background on a simple, already-tidy canvas. The cheap model reads those confidently almost every time, the hand-off barely ever fires there, and it never will, no matter how many more people use it.

The lesson. A cost forecast built from a clean test set only tells you the cost of clean work. The whole reason to ship an AI feature to real users is that their work isn't clean. Any cost model that skips that step is measuring the wrong thing from day one.

Same rollout, two very different sizes of what it costs to get this wrong

Two numbers that were never supposed to disagree

You don't need this to answer the question. Read it if you want to feel why holding the line at 25 percent beats fixing it while you keep climbing.

Julen Kastrup can read a cost model the way some people read a recipe. He knows exactly which line will run over before anyone's bought a single ingredient.

He's run the Skyfill pod at Palette Foundry for two years and priced the team's last AI feature himself, spreadsheet and all. Skyfill launched at 5 percent. Every Monday for the first six weeks, Julen opened the full weekly infra report line by line: fills served, hand-off rate, blended cost per fill, next to the number the launch model had promised. Off by a rounding error, every single week.

By week seven, at the 25 percent stage, the report started arriving with one line at the top: "Skyfill spend: within range." Julen started reading only that line. Six good weeks does that to a person, and there was a launch deck due.

Then came the quarterly audit. Oskar Weng, from finance, runs the infra reconciliation ahead of every board deck. He pulled Skyfill's actual GPU billing, compared it to what the automated dashboard had been reporting all along, and messaged Julen: "Your Skyfill number's off by a lot. The email's been marking it 'within range' using the 8 percent hand-off rate from launch. Real rate's running at half of all fills. Did you know that?"

Julen didn't stop the rollout. Launch was one week from the board deck, adoption numbers were the best story in the room, and pulling the ramp back felt like turning a win into a confession. So he told engineering to recalibrate the threshold without blocking the ramp for it, and Skyfill moved from 25 percent to 50 percent on schedule the next Monday, the fix running underneath it in parallel.

Recalibrating against real, messy user canvases turned out to take about three weeks, not because the fix itself was hard, but because it now had to be tested carefully against a population that kept growing underneath it. Every extra day of testing meant one more day of a bigger population paying the old, broken rate. By the time the new threshold shipped, those three weeks had cost Palette Foundry about $47,000 more than the plan ever budgeted for that stretch, on top of everyone already exposed to it.

Back at the original launch readiness review, an engineer had actually asked whether they should hold off calibrating until real user files existed. The answer, reasonable at the time, was that there weren't any real files yet, the clean set was all there was, and they'd revisit it once the beta numbers came in. Nobody put a date on revisiting it. The first few weeks looked fine, so nobody did.

Here is the replay. Same message from Oskar, new design: the rollout holds flat at 25 percent the moment Julen reads it. Recalibrating the threshold against real canvases, without a growing population underneath it, takes the team about a week. Total extra spend over the plan during that week: about $12,000, against the $47,000 the way it actually happened.

One design pays $12,000 to learn the truth. The other pays four times that to keep pretending it already knew it. The board deck moves eight days. Nobody outside the room ever needs to know the number was wrong at all.

And the thing I'd go back and tell myself, in that first launch review: the system we built to watch for trouble was only ever as honest as the number we typed into it on day one.

PICK, with a real dollar figure on both sides

This is a yes or no about whether to pause or keep shipping while you fix it, not a rule for every rollout Palette Foundry runs, so PICK carries the weight here.

Try the same four letters on a shoebox of old photographs

Iwan Dace runs product for Kinfolk Archive, an app three hundred thousand people use to restore and colorize old family photographs. Upload a scan, and a model repairs the damage and brings back the color.

Two changes are queued for the same release. A new album layout that groups restored photos by decade, pure interface, no model involved. And an expansion of the restoration model's escalation logic: a fast single-pass model handles typical scans, but a torn, badly faded, or multi-face photo gets handed to a slow, expensive multi-pass model instead, the one built to actually repair real damage.

Iwan split the release the way Julen eventually wished he had.

P. Fast, light rollout for the album layout, no model touched, ship it in a week. Full pause-and-fix treatment for the escalation logic: the threshold was calibrated on employees' own well-kept family photos, and real users are uploading exactly the water-damaged, decades-old, attic-stored photos the app was built for, at a rate the test set never captured.
I. A wrong album layout is felt by one user who regroups a folder by hand, a minute of annoyance. An unresolved escalation cost is felt by the whole company's per-photo economics, and it worsens specifically as more people upload the damaged photos Kinfolk Archive exists to fix.
C. The layout miss is cheap and loud, someone flags it the same day. The cost miss is quiet and expensive, restorations keep succeeding, users stay happy, and the wrong number sits inside a monthly infrastructure bill nobody reads line by line until finance closes the quarter.
K. Resume expanding restoration access once a threshold recalibrated against real degraded photos holds the escalation rate under a set line for three straight days. Below that line, the risk is close enough to normal photo processing. Above it, it stays paused.

Flashcards (click a card to flip it)

Check yourself Score: 0 / 0