How do you give useful feedback on model output without prescribing an implementation?

Furnora takes a photo of an empty room from a real estate listing and generates a version with furniture in it, so an agent can list a home as staged without paying for real furniture or a stylist. Obiageli Quarshie owns the staging model surface. Kelechi Amissah leads the team that builds it. When Obiageli found a pattern in how the model handled narrow rooms, she had to decide what to actually say to Kelechi about it, and her first two tries both cost the team real time before her third one worked.

How to answer this, stage by stage

Nobody is grading whether you can define a good bug report. They're grading whether you know why a PM handing an engineer a fix, instead of a failure, is a worse mistake than saying nothing useful at all.

Let's learn

A photo goes in, furniture comes back rendered into it, and it ships with nobody checking it first. That last part is why the model's mistakes matter.

Furnora turns an empty listing photo into a furnished one, so an agent can post a staged photo the same day instead of paying for real furniture and a two week wait.

Across the whole catalog, the model does this well. Average scale error, how close a piece of furniture's size is to what the room could actually hold, sits under 6 percent, and it has for months. That number looks fine on a dashboard.

Inside one slice of it, it isn't fine. Rooms narrower than nine and a half feet, common in older rowhomes and prewar apartment buildings, run a scale error closer to 22 percent. A sofa or a bed reads sized for a room a fifth wider than the one it's actually in, so it visually overlaps the far wall or floats above the floor line. That slice is a minority of listings, about a fifth of them, so it barely moves the average at all.

Here's the turn. The extra mistakes in narrow rooms were never really the problem. The problem was what Obiageli said about them. Her first note, "the narrow ones look off, can you fix the scaling," told Kelechi's team nothing they could act on, so they spent most of a week trying things that didn't touch it. Her second note, after that failed, told them exactly what to build, a preprocessing step that would rescale the furniture to the room's pixel width. They built it. It didn't fix the real problem, and it broke something new.

What I would leave alone: when Kelechi later found and flagged that the staging pipeline's image cache was serving a stale room mask on retries, I told him to fix the caching bug directly and how. That's not a modeling judgment call, it's a real engineering fact I actually understood, and being specific there cost nothing and wasted no one's week.

The lesson: "fix it" and "do it this way" both skip the one thing an engineer actually needs: a case they can point the model at and watch it fail, with nothing else attached.

The useful note sits in one corner only: specific about the failure, and silent about the fix. Everything else Obiageli tried landed somewhere else on this map.

Now here is the same thing as a story

The short version above is what you actually say out loud. Read this one for what it cost to learn it the slow way.

Every Monday morning, before anyone else is at their desk, Obiageli Quarshie pulls the past week's narrow-room stagings into one folder and scrolls through them. Four years running product for Furnora's staging model, and she can spot a scale mismatch in about a second: a loveseat a little too deep for its wall, a headboard that reads like it's floating an inch off the floor.

For most of those four years, that Monday scroll was a formality. The model was good. Agents loved it, a $1,800 in-person staging job replaced by a photo that shipped in under a minute, and Obiageli's Monday folder rarely held more than a couple of odd ones out of a hundred, the kind of thing you notice and forget.

Then Furnora signed a partner brokerage that specialized in older buildings, prewar walk-ups, narrow rowhomes cut into apartments a century ago. More of that housing stock started moving through the pipeline. And Monday after Monday, the folder held a few more than usual. Not most of them. Not even close. But enough that she stopped scrolling past.

She didn't say anything for six weeks. She told herself she was still getting a feel for whether it was a real pattern or just a run of bad luck in one batch. Then one Monday, instead of scrolling, she pulled all one hundred forty narrow-room stagings from that week and checked every one by hand. Thirty one of them, twenty two percent, had a sofa or bed that plainly didn't fit the room it was rendered into.

The same folder, checked properly, turned a feeling into a number. What Obiageli did with that number is the whole story.

She messaged Kelechi Amissah, who leads the staging model's engineering, that same afternoon. "The narrow ones look off, can you take a pass at the scaling?" That was the whole message.

Kelechi's team spent most of the next week on it, and none of it landed anywhere near the real problem, because "looks off" pointed at nothing in particular. They tried a longer denoising schedule. They adjusted the lighting model. One engineer, half joking, floated pulling narrow rooms out of auto-staging entirely and routing them to a human touch-up queue until someone figured it out. That idea died fast in the same meeting: narrow rooms are disproportionately the older, cheaper listings that can't afford real staging in the first place, about a fifth of everything Furnora processes, and routing a fifth of the product back to manual work would have re-created the exact cost problem Furnora exists to remove.

By the following Thursday, embarrassed that her first note hadn't worked, Obiageli swung the other way. She'd spent a night reading about how the model reads room geometry, and she came in with something that sounded like an answer. "Add a step before render that measures the room's pixel width from the depth map and rescales the furniture asset to match it." Kelechi's team, trusting a PM who'd clearly done some homework, built exactly that. Three weeks.

It shipped, and it even moved the number a little on that week's narrow-room batch. But a new problem showed up right behind it: furniture that looked subtly warped, an armrest bent at the wrong angle, a leg meeting the floor at an angle the room's own perspective didn't support. It happened in about one staged photo in eleven, and a warped sofa reads as broken in a way an over-large one doesn't. It looks wrong even to someone who's never thought about scale.

So Kelechi came back to Obiageli, and instead of patching the rescale hack again, he did the thing a PM rarely sees because it lives inside the model, not the product. He pulled the training set. Rooms under nine and a half feet wide made up about four percent of everything the staging model had ever been trained on, even though Furnora's own listing volume runs closer to nineteen percent narrow by that same cutoff. The model wasn't failing to measure the room. It measured it fine. It just had almost nothing to learn a correct sofa scale from at that width, so it fell back on a scale it had learned from thousands of wider rooms, applied with exactly the same confidence it uses everywhere else.

That's the moment Obiageli understood what she should have handed over in the first place. Not "fix the scaling," too vague to point at anything. Not "add a rescale step," a real, specific fix, but for a cause she'd guessed at without the context to know if it was right. What Kelechi actually needed was the twelve or so reproducible cases she already had sitting in her Monday folder: this room, this width, this sofa, off by this much, in fourteen of forty photos like it. Let him trace it back to wherever it actually lived.

What Obiageli handed over the third time. Three of the four items are about the failure. The fourth is a rule about what to leave out.

They rebuilt the fix properly: narrow-room examples reweighted in training, and a room-width signal added as a genuine input to the generation step, not a bolt-on afterward. The rescale hack came out entirely.

What I'd tell myself, standing in that Thursday meeting handing Kelechi a fix I had no real business specifying: I wasn't being helpful by being specific. I was borrowing his job to avoid admitting, out loud, that I didn't actually know what was broken yet.

PICK, the case you hand over, not the fix you write

Not a rule about being nice to engineers. PICK only earns its keep here if it makes you name, out loud, which of your two mistakes costs more, and it isn't the one that feels worse in the room.

One mistake shows up in days and costs a conversation. The other hides inside a real build for weeks before anyone finds it.

Four branches, one question asked before any note goes out: does this case earn a build, and does it earn one without telling the engineer how.

The trade worth saying out loud: the real fix, a room-width signal fed into generation, adds a small amount of inference time per photo, since the model now has to read and use that signal instead of just rendering blind. Worth it, since the alternative was a rescale hack that patched the number without fixing what was actually happening inside the model.

And if you want to be sure it really works, try it somewhere else

Same four letters, a crop health tool instead of a listing photo, and the fix a PM almost prescribed would have been just as wrong for just as specific a reason.

Thistlebeck runs a phone app that scans a tomato leaf and flags disease before a grower loses a field to it. Its classifier keeps confusing early blight with simple nutrient yellowing on leaves photographed in low afternoon light, a mistake that matters because one gets sprayed and the other gets fertilized, and doing the wrong one wastes a week either way.

The Thistlebeck PM had the same two options Obiageli did. One hands the engineer a case. The other hands them a guess about a camera problem that was never the real cause.

Position: describe the fourteen photos, same low light, same wrong call, without naming a mechanism. Impact: a vague "the classifier struggles in low light" gives the ML team nothing to chase; a prescribed "add a color-calibration step before scoring" can lock in a fix for a camera problem that was never the actual cause. Cost asymmetry: a clarifying conversation about which fourteen photos costs an afternoon; building and shipping a calibration step that turns out to fix nothing costs a growing season, since growers only find out it didn't work when the next round of misdiagnosed leaves shows up. Kill criteria: does the note name reproducible photos with nothing about how to fix the model, or does it already contain a proposed pipeline change.

Swap the trigger and it still runs.
Speed: an interviewer caps you at ninety seconds. Skip straight to it: ask for the reproducible case before anything else gets discussed.
Cost: no time for a back-and-forth conversation before the next sprint starts. Fine, but write the case, not a label, into the ticket itself: this room, this width, this sofa, this many times out of how many tried.
The model got better, for real: say a new checkpoint drops the narrow-room error on its own, no feedback needed. Still doesn't mean the discipline stops mattering. Ask what's the next slice hiding inside the new average.

Where people run it wrong.
They mistake being specific about the fix for being helpful, when specificity only helps if it's aimed at the failure, not the mechanism.
They treat a single striking example as a pattern, and send an engineer chasing something that only happened once.
They fix vague feedback that didn't land by getting more prescriptive instead of more precise about what actually broke.

How to use it live. If you're ever handed a piece of model behavior mid-interview and asked what you'd tell the engineer, buy yourself a second by asking one thing out loud: "how many times has this happened, and on what, exactly." That question is the whole method, said as a question instead of a rule.

Flashcards (tap any card to flip it)

Check yourself Score: 0 / 0