Explain the risk of optimizing directly on user feedback.

Solvemate walks a high schooler through a math problem, step by step, and a thumbs-up or thumbs-down sits under every explanation. Ines Calderon leads product for Solvemate and owns the decision of what those ratings are allowed to do to the model.

How to answer this, stage by stage

Nobody's grading whether you know feedback can be gamed. They're grading whether you can name who actually gets hurt when it is.

Let's learn

Here's what happens when you let a rating decide what's true.

Solvemate walks a high schooler through a math problem one step at a time. After each explanation, a thumbs-up or thumbs-down sits at the bottom of the screen. Early on, the team used those ratings the simplest way possible: whichever phrasing of an explanation got more thumbs-up became the version shown more often to the next student on that same problem.

There's a gap in this loop where correctness should sit, and right now nothing fills it.

For a long stretch, this looked like it was working. Ratings climbed steadily, quarter over quarter, and the team read that as students getting more out of every explanation.

The turn: the ratings climbing was never proof the math was getting more correct. It was proof the explanations were getting better at sounding clear, and those are not the same thing.

At its worst: a wrong method for solving a quadratic gets reinforced across thousands of students because it reads as confident and friendly, while a correct but slightly more formal explanation quietly loses ground and stops being shown at all.

What I would leave alone: using student ratings to choose between two explanations that are already both mathematically correct, say, a shorter one versus a more detailed one, is exactly the right use of a rating. The risk is only in letting a rating decide correctness itself.

The lesson: a feedback signal is only safe to optimize on directly when the people giving it can actually judge the thing you're asking them to judge. The moment that stops being true, optimizing on it rewards confidence over correctness, quietly and for a long time before anyone notices.

Now here is the same thing as a story

The short version above is what you'd say defending this fix to Solvemate's leadership. Read this one for how the gap actually got found.

Solvemate's engineering floor is quietest around nine at night, when the week's retraining job kicks off, folding in a fresh batch of student ratings from the past few days.

There was no single bad Tuesday here. No customer complaint, no viral screenshot, no one incident anyone could point to. The risk built up the ordinary way: a little at a time, every week, for the better part of a year and a half.

One of these two people holds a lever. The other one just lives with whatever the lever produced.

Ines Calderon ran a quarterly correctness audit that had been on the calendar for two years, nothing special about this particular quarter. A rotating group of subject-matter math teachers reviewed a sample of two hundred of Solvemate's highest-rated explanations, the ones with a thumbs-up rate above ninety percent.

The dangerous quadrant isn't the one anyone was watching for. It's the one that looks fine on a ratings dashboard.

Thirty four of those two hundred, seventeen percent, contained an actual error in the math reasoning. Every one of them had been shown to students hundreds of times, each time reinforcing the same wrong method a little further.

The teachers noticed something else, too: the errors clustered in a specific pattern. A confident, friendly tone, paired with a subtly wrong step buried in the middle, exactly the kind of mistake a student who's still learning the topic has no way to catch.

A rating is one number standing in for four different questions, and only one of them is the one that matters here.

Ines's team gated the retraining loop: an independent correctness check now runs on every explanation first, and only explanations that pass it are eligible for ratings to promote at all.

Run the same eighteen months forward with the gate in place: the confident-but-wrong share never gets to climb past its round-one baseline, because nothing reaches a student until a correctness check, not a rating, has already cleared it.

Ines built the ratings-first design because it felt like listening to students directly, and that felt like the right instinct at the time. It took a routine audit, not a crisis, to see that listening to students about clarity and trusting them to judge correctness were two very different things wearing the same thumbs-up icon.

GUARD, the two people in the roomNot a fairness checklist. GUARD is what forces you to name who's rating and who's living with the result.

Nothing about this needs a single bad day. A small compounding drift, left unchecked, gets you to the same place.

The recap, one line per letter: groups is the rater and the future student who never rated anything, unequal is the confident-but-wrong explanations winning over correct plain ones, ability to contest is a struggling student who can't tell the difference, reduce is the correctness gate before any rating counts, and detect is watching error rate by tier instead of trusting the average.

And if you want to be sure it really works, try it somewhere elseSame five letters, a warehouse instead of a classroom. A different floor, the same lever.

Shiftline suggests weekly shift schedules to a warehouse's operations lead, and workers rate each suggested schedule with a thumbs up or down before it's finalized. Godwin Asare runs operations at a distribution site using it.

Mapped onto GUARD: groups are the workers who bother to rate a schedule, usually whoever got an easy shift that week, and the workers who end up on whatever pattern the ratings favor, including the ones who rarely rate anything at all. Unequal is a schedule that quietly loads more weekend and night shifts onto a quieter subgroup, since the vocal raters keep reinforcing whichever pattern favors them personally. Ability to contest is the night-shift crew, who rarely log into the rating system at all and have no lever over which pattern keeps winning. Reduce is the same design move as Solvemate's: gate any schedule change behind a fairness check on hour distribution across the whole team, not just the loudest raters, before ratings can influence which pattern repeats. Detect is watching the actual spread of night and weekend hours by worker, not just the overall thumbs-up rate on the schedule tool.

The same three questions catch this whether the output is a math step or a shift schedule.

Swap the trigger and it still runs.
Speed: an interviewer caps you at sixty seconds. Say "gate correctness before ratings can promote anything, since raters who need help most can't judge correctness," and stop.
Cost: there's no budget yet for a full independent correctness pipeline. Say so honestly, and start with a smaller, manual spot-check on the highest-rated explanations first, since a thin gate still beats none.
The model gets better, for real: if Solvemate's base model gets sharper and makes fewer errors overall, the gate still matters, since even a small remaining error rate reinforced at scale over years is still real damage.

Where people run it wrong.
They treat a high rating as proof of quality without ever separately checking correctness.
They wait for a single dramatic failure instead of running a routine, scheduled audit.
They fix the model's confidence without ever gating what ratings are allowed to reinforce in the first place.

How to use it live. When someone asks about optimizing on user feedback, ask yourself one question first: can the person giving this feedback actually judge the thing I'm about to let it decide. If the honest answer is no, that's the risk, right there.

Only one of these three branches is the good outcome, and a raw rating can't tell you which branch you're actually on.

Flashcards (tap any card to flip it)

Check yourself Score: 0 / 0