Metrics & Estimation: North Star, Guardrails, Funnels, Guesstimates, Diagnosis

The execution round is where vague candidates fall apart, because you can't hand wave a metric. You either have the tool or you don't. So this video hands you five tools that between them answer basically every metrics question you'll get. These are the North Star, its guardrails, the funnel, the top down estimate, and the diagnosis method. Learn these five, and the whole execution round turns into a matter of picking the right one.

By the end of this you'll be able to define success for any product, estimate the size of almost anything out loud, and diagnose a metric that moved without guessing at the cause. Those three skills are the entire round, plus the metrics half of every product sense answer. Let's take them one at a time. Tool one, the North Star metric.

This is the single metric that best captures the value your product delivers to users. Not a vanity number like raw signups or page views. Here's the test. If this number goes up, are users genuinely getting more value? For Spotify it's time spent listening, not app opens. For an AI assistant it's something closer to tasks successfully completed per weekly active user, rather than messages sent.

Messages can even rise when the product is failing, because users repeat the same broken question three times. Picking a North Star is a judgement call, and interviewers score the judgement. A good North Star is value capturing, it's movable by the team, and it's hard to game. Tool two, guardrail metrics. Here's the thing about a North Star. You can almost always improve it in a way that quietly harms the product.

Guardrails are the counter metrics that must not get worse while you chase it. If your North Star is engagement, a guardrail is that you're not driving it with addictive dark patterns, so you might guard on user reported satisfaction. For an AI product, guardrails usually include a safety or quality rate, hallucination report rate, harmful output rate, latency, and cost per query.

The move in the room is this. The second you name a North Star, name its guardrails in the same breath. North Star is tasks completed per weekly active user, guardrails are hallucination report rate under one percent and p95 latency under three seconds. That combination shows you know metrics can be gamed, and that you're already watching the blind spot.

Naming a North Star with no guardrail is one of the fastest ways to look junior. Tool three, funnels. A funnel breaks a user journey into ordered stages, so you can see exactly where people drop off. A generic product funnel is aware, then acquired, then activated, then retained, then referred. A feature funnel is saw the feature, tried it, succeeded, came back.

The power of a funnel is diagnosis by localisation. Instead of saying engagement is low, you get 70 percent try it but only 20 percent succeed. That points straight at the quality of the AI output, not at discovery. When a question asks you to define success or find a problem, building the funnel turns one fuzzy number into a set of specific, addressable stages you can actually go fix.

And the reason a funnel matters more for AI features than for most is that the succeeded stage is doing heavy lifting. For a normal button, tried it basically means succeeded. For a generative feature, a user can try it and get a bad answer, so the gap between tried and succeeded is where the model quality really lives. If 80 percent try your assistant and only 25 percent get an answer they act on, no amount of marketing fixes that.

It's an output quality problem, and the funnel is what tells you so instead of letting you chase the wrong lever. Tool four, guesstimates, or top down estimation. Estimate the number of ChatGPT users worldwide. How big is this market. The method is always top down and structured, never a guessed number pulled from the air. You start from a big known population, apply a chain of reasonable fractions, and state every assumption out loud.

Then you sanity check the result against something you actually know. The final number matters far less than the structure and the stated assumptions, because interviewers are scoring whether your reasoning is sound and auditable. Let me build one live. Estimate global ChatGPT users. Start with about 5.5 billion people online. Assume maybe 40 percent are in regions and languages where ChatGPT is usable and known, so around 2.2 billion reachable.

Assume awareness and having tried it at, say, 25 percent, so about 550 million have tried it at least once. Assume monthly active among the triers at, say, 30 percent, so roughly 165 million monthly active. Now say clearly that each of those fractions is an assumption you would refine with real data. Then sanity check the result. You might note that real reported numbers are actually much higher, in the several hundred million weekly active range.

That tells you your awareness and active fractions were conservative, which is perfectly fine. The interviewer doesn't care whether you land on 150 million or 500 million. They care that you built it in layers, stated your assumptions, and checked the answer against reality. Tool five, and the big one, diagnosis. The most common execution question is, metric X moved, why?

And there's one rule that separates strong answers from weak ones. Isolate before you conclude. Don't jump to a cause. Narrow the problem first. Here's the walk. Start by clarifying and sizing it. Which exact metric and definition? What magnitude, is that 15 percent relative or absolute? Sudden or gradual? Over what window? Any known launches or holidays? And is the move even outside normal variance?

Size it before you chase it. Then, real or measurement? Before you assume a real behaviour change, rule out a logging bug, a tracking change, a client that stopped emitting events, a timezone or holiday boundary. A drop that's actually a broken event pipeline is a completely different fix. Then, internal or external? Internal is a release, a config or paywall change, a broken entry point.

External is a competitor launch, seasonality, an upstream dependency, maybe model latency spiked and users bounced. Then segment to localise. Break the metric by platform, by geo, by app version, and by cohort. For DAU that's new, plus retained, plus resurrected. The delta almost always concentrates somewhere. Your job is to find where. Then funnel it. Are fewer users arriving, or fewer succeeding once they're there?

That single question halves your hypotheses. And only then, confirm and act. Pull the one segment that explains most of the delta, correlate it with the release timeline and the error dashboards, and now name the cause. A regression means roll back. A measurement artefact means fix logging and restate. A real external shift means a product response. The highest value habit in this whole toolkit is that one thing.

Separate the metric moved from the underlying thing changed, and isolate the segment before you commit to a story. Let me put success and diagnosis together. Prompt, you launched an AI meeting notes feature, define success, then diagnose a 15 percent weekly usage drop. Success first. North Star is meetings with AI notes actually opened and used afterwards, per weekly active team, not raw notes generated, because generating notes nobody reads isn't value.

Guardrails are note accuracy, a sampled human rated faithfulness score, and that the feature isn't quietly reducing live attendance, which is a genuine second order risk with meeting summaries. Funnel is meeting recorded, notes generated, notes opened, notes shared or acted on. Now the drop. Size it. 15 percent relative, sudden, one week. Real or measurement. Check whether a client update broke the notes opened event before assuming behaviour changed.

Internal or external. Was there a release last week? Segment. Is the drop on one platform, one app version, one customer cohort? Funnel. Are fewer meetings generating notes, which is a capture bug, or are notes being generated but not opened, which is a value or quality problem? Say it localises to the notes opened stage, on the latest desktop version only.

That points at a broken link or a UI regression in that release, not a demand problem. So you roll back and restate. And notice, you reached a specific, testable cause without ever once guessing. A few myths that sink execution answers. More engagement is always the goal? No. Engagement rises when the product is failing and users are repeating a broken assistant, so pick a value capturing North Star and guard it.

The estimate needs to be accurate? No, it needs to be structured and explicit in its assumptions. An order of magnitude with clear reasoning beats a precise looking number with no method. Jump to the likely cause to save time? The fastest wrong answer is worse than a thirty second isolation, so segment and check measurement before you conclude. And one metric is enough?

A North Star with no guardrail invites gaming, so always pair them. Five tools. North Star for value. Guardrails for the blind spot. Funnel to localise. Top down structure for estimates. And always isolate before you conclude. That's the execution round, the estimation questions, and the metrics half of every design answer, all in one toolkit.