How would you sample feedback to avoid a vocal minority skewing your view?

Kestrel Mutual's claims copilot reads a claim file and drafts a plain-language summary for the adjuster handling it, with a thumbs up or down under every draft. Petra Vansova is the senior adjuster who also owns the feedback dashboard leadership reads every month.

How to answer this, stage by stage

Nobody's grading whether you know the phrase "sampling bias." They're grading whether you can show the actual arithmetic behind a trustworthy number.

Let's learn

Picture ten adjusters, and only one of them ever clicks anything at all.

Kestrel Mutual's claims copilot reads a claim file, the incident report, photos, and prior notes, and drafts a plain-language summary for the adjuster to review before making a decision. A thumbs-up and thumbs-down icon sits under every draft.

A trustworthy sample isn't collected. It's built, on purpose, in this order.

Before anyone looked closely, Kestrel's monthly dashboard reported one flat number: an average thumbs-up rate across every rating anyone ever submitted, no cap, no weighting, whoever clicked, counted.

The turn: the size of that 340 was never the problem. A flat number that size can look completely healthy while still describing almost nobody's actual experience.

At its worst: leadership reads a healthy 91% thumbs-up rate in a board deck, unaware it's built almost entirely on one adjuster's opinion about one claim type, while a different claim type Kestrel is actively expanding into has never been rated by anyone at all.

What I would leave alone: a single adjuster flagging one specific claim as flat-out wrong is still worth acting on immediately, on its own merits. The sampling design is about the aggregate quality number, not about ignoring an individual, specific, correct complaint.

The lesson: a bigger raw number of ratings isn't the same thing as a trustworthy one. The honest question was never "how do we get more feedback." It was "who exactly is allowed to count, and how much."

Now here is the same thing as a story

The short version above is what you'd say defending the redesigned dashboard to Kestrel's leadership. Read this one for how the gap actually got found.

Petra Vansova has adjusted auto and property claims at Kestrel Mutual for eleven years, and for the last two she's also owned the monthly quality dashboard for the claims copilot.

Both of these describe a real week at Kestrel. Only one of them ever showed up on the dashboard.

A new data analyst, three weeks into the team, was building her first version of the weekly report when she asked Petra a plain question: "wait, are we just counting whoever complained loudest?"

The adjusters actually rating things are, on average, the least typical part of the whole caseload.

Petra pulled the raw numbers to check. Two hundred twenty adjusters handle roughly 2,400 claims a week between them. Only 340 of those claims ever got a rating, and one adjuster, who specializes in a narrow slice of auto-body estimates, had submitted 96 of them himself.

Using the standard formula for a trustworthy percentage, a 95% confidence level with a 5% margin of error needs 385 ratings. Kestrel had 340 total, and a huge share of those came from one person.

Petra ran the sanity check out loud in the room: 340 sounds like a real number, until you notice a quarter of it came from one man who rates almost every claim he touches, on a claim type most adjusters barely see.

Petra's team redesigned the collection: cap any one adjuster's counted ratings at three per week, then actively prompt a random, fixed quota across five claim types and three tenure bands, targeting about 150 properly distributed ratings a week, an honest 8% margin instead of a fantasy 5% the current volume could never support.

Four small design choices, and together they turn 340 unequal opinions into one number worth trusting.

Run the same new hire's question forward under the new design: the answer is simply no, because no single adjuster can supply more than three of the counted ratings in any given week, whatever the raw click count behind the scenes says.

Petra built the flat, uncapped dashboard because it felt neutral, just averaging whatever came in. It took one new analyst's plain question to see that "count everything equally" isn't neutral at all when some people show up forty times more often than others.

BOUND, the actual arithmeticNot a vibe check. BOUND is what makes a sampling design defensible with real numbers, not just good intentions.

The recap, one line per letter: break it down is the standard sample-size equation, own the numbers is Kestrel's real 220 adjusters and 340 weekly ratings, use a range is the honest 150 to 385 span, nail the sanity check is one man supplying a quarter of the total, and direction is the margin-of-error choice that swings the estimate the most.

And if you want to be sure it really works, try it somewhere elseSame five letters, an HVAC dispatch floor instead of a claims desk. A different trade, the same arithmetic.

Ductwise suggests a likely cause to an HVAC technician mid-repair, based on a description of the symptoms. Reuben Okafor runs field service operations for the company using it, and a small handful of senior technicians rate almost every suggestion, while most technicians rate almost none.

Mapped onto BOUND: break it down is the same sample-size equation, applied to however many technicians and repair calls the company runs each week. Own the numbers might look like 85 technicians, 900 repair calls a week, and 60 ratings, 22 of them from one senior technician who always rates Ductwise's compressor-fault suggestions specifically. Use a range gives a target between roughly 40 and 95 properly distributed ratings a week, depending on the margin of error the team can live with. Nail the sanity check is noticing that one technician's specialty, compressor faults, is wildly overrepresented while duct-leak calls, a third of all repairs, have barely been rated at all. Direction is the same finding: the margin-of-error choice swings the needed count far more than anything else in the design.

The same three warning signs show up whether the caseload is insurance claims or HVAC repair tickets.

Swap the trigger and it still runs.
Speed: an interviewer caps you at sixty seconds. Say "cap each rater's contribution, then actively sample a fixed quota by group, at a margin of error the real volume supports," and stop.
Cost: there's no engineering time to build active solicitation prompts this quarter. Say so honestly, and start by simply capping and reweighting the existing organic ratings first, since even that beats a flat, uncapped average.
The model gets better, for real: if the copilot's summaries get consistently more accurate, the sampling design still matters, since even a small remaining error rate needs an honest, unskewed view to catch it at all.

Where people run it wrong.
They treat a bigger raw number of ratings as automatically more trustworthy, without checking who supplied them.
They demand a tight margin of error without checking whether real weekly volume can actually support it.
They stratify the analysis after the fact instead of designing the sample around the strata from the start.

How to use it live. When someone asks how you'd sample feedback fairly, ask yourself one question first: could any single person, on their own, move this week's number. If the honest answer is yes, that's the fix, right there.

The design only earns the word trusted once it's been checked the same way, week after week.

Flashcards (tap any card to flip it)

Check yourself Score: 0 / 0