How to Attack an Execution and Metrics Round

Execution rounds test one thing above all else: can you define success, and diagnose failure, without guessing. The candidates who fail jump straight to a cause on the first slide. The ones who pass isolate first. That single habit, narrowing before concluding, is most of what separates a hire from a no-hire here. These questions come in a few recognisable shapes, define success, diagnose a metric that moved, estimate a number, plan a launch.

But the opening move is always the same regardless of shape. Clarify the metric and its exact definition before you reason about anything. Which metric, defined how, over what window. Get that pinned, then attack. First shape, define success. Pick a North Star that captures real value, not a vanity count. Signups and page views go up for reasons that have nothing to do with users getting value, so avoid them.

For a product, ask, if this number rises, are users genuinely better off? Then pair the North Star with guardrails, the counter-metrics that stop you from gaming it. If your North Star is engagement, guard on satisfaction, so you're not juicing the number with dark patterns. For an AI product, add a quality guardrail, hallucination-report rate, or a faithfulness score, because you can drive usage up while the answers quietly get worse.

Name the North Star and its guardrails together, always, because that pairing is the signal that you know a single metric can be gamed. Second shape, and the most common one, a metric moved, explain it. Here's the rule: isolate before you conclude. Do not name a cause until you've narrowed the problem. Walk it like this. Is the drop even real, or is it a measurement artefact?

A logging change, a tracking bug, a client that stopped emitting events, a holiday boundary. Rule that out first, because a broken event pipeline is a completely different fix from a real behaviour change. Then, internal or external? Internal is a release you shipped, a config change, a paywall tweak, a broken entry point. External is seasonality, a competitor launch, an upstream dependency that got slower.

Splitting internal from external cuts your hypothesis space in half. Then segment, by platform, by geography, by app version, by cohort, until the delta concentrates somewhere. The drop is almost never spread evenly, it's hiding in one segment, and your job is to find which one. Then funnel it. Are fewer users arriving, or fewer engaging once they're there? That question halves the remaining hypotheses again.

And only then, name the cause and act. A regression means roll back. A measurement artefact means fix logging and restate the number. A real external shift means an actual product response. You've reached a testable cause, and you never guessed. Third shape, estimation. Go top-down, with one stated assumption per line, and a sanity check against a known public number at the end.

Start from a big population, apply a chain of fractions, say every assumption out loud, then ask, is this order of magnitude plausible? The structure is what's scored, not the exact figure. So if they ask you to estimate ChatGPT's daily users, don't blurt a number, build it, start from people online, cut to the reachable regions, cut to those who've tried it, cut to those who use it daily, and state each cut as an assumption you'd refine with data.

A wrong final number with clean structure beats a lucky guess with none. Fourth shape, a launch. Stage it. Internal dogfood first, then a small canary percentage, then a ramp, with pre-agreed thresholds at each gate and a rollback you've actually tested. For an AI feature, one of those thresholds should be a quality or safety metric, not just engagement, because a feature can be sticky and still be doing harm.

And say what you'd watch during the ramp, the guardrail metrics, not just the North Star, so a regression trips a gate before it reaches everyone. A launch plan with no rollback and no thresholds is a hope, not a plan, and interviewers can tell the difference instantly. So what earns the points here? First, you isolate before concluding, you never guess the cause on slide one.

Second, you check the metric is even real before chasing it, because half of all drops are measurement, not behaviour. And third, you separate the metric from its measurement, and the North Star from a vanity number. Those distinctions are the whole craft of this round. Now the ways people lose it. The biggest trap is jumping to a root cause with no segmentation, it feels fast and decisive, and it's usually wrong.

Second trap, choosing DAU or message count as a North Star when neither actually reflects value, because both can rise while the product is failing. And the third, forgetting guardrails entirely, so your success metric can be gamed and you didn't even notice the blind spot. So here's the whole attack. Clarify the metric and its definition first. To define success, pick a value-capturing North Star and pair it with guardrails.

To diagnose, isolate before you conclude, is it real or measured, internal or external, then segment and funnel until the delta shows itself, and only then name the cause. To estimate, go top-down with stated assumptions and a sanity check. To launch, stage it with thresholds and a tested rollback. Memorise this. Define success with guardrails, and diagnose by isolating, real versus measured, internal versus external, then segment, before you ever name a cause.

---