How do you evaluate a model's reliability on structured output for your schema?

PayLeap gives gig drivers a same-day cash advance against a photo of their pay stub. StubSnap is the part that reads the photo and turns it into six fields a decision engine trusts automatically: gross pay, deductions, net pay, pay period, and employer name. Selin Marchetti reconciles the flagged ones every morning.

How to answer this, stage by stage

Nobody is scoring whether you know the words "schema validation." They're scoring whether you'd have caught the one field going wrong before the record-level number ever moved.

Let's learn

Picture StubSnap before anyone has argued about how to grade it.

A driver photographs a pay stub. StubSnap reads it and hands PayLeap's decision engine six fields, and the engine either approves a same-day cash advance or asks for a person to look at it.

Before StubSnap, an ops team spot-checked about 30 stubs a day by hand against the photo, roughly one in twenty, and it took about 20 minutes a batch.

With StubSnap, every stub gets read, and the record either passes schema validation or it doesn't. Overall, that pass rate barely moved across a model version bump: 96 percent, then 94.

Five steps, and the fourth one, confidence scored, is the one this whole answer turns on.

Here's the turn: those 2 extra points of record failure are not the real problem. The real problem is that a record-level pass or fail says nothing about which one field is going bad, while everything else on the stub still reads fine.

At its worst, that gap sends a same-day cash advance out against a fabricated employer, or it makes a real driver wait a day over a field mistake that costs nothing. Either way, the tool ends up worse than no tool at all.

What I would leave alone: I wouldn't spend eval budget hardening gross pay. It's the largest, boldest number on every stub layout, and it's almost never the field that goes wrong.

The lesson: a single number that says "the record passed" can hide the one field that's already broken. The record isn't the unit that matters. The field is.

Now here is the same thing as a story

The short version above is what you'd say in a design review. Read this one for what it felt like the week a near miss changed what Selin checks every morning.

Selin Marchetti can tell in about two seconds whether a payroll number looks right, just from the shape of the stub. Six years reconciling driver pay will do that.

When StubSnap first shipped, she spot-checked ten stubs a morning against the photo, all six fields. For the first month it was never wrong on anything that mattered. By month three she checked three. By month five, most mornings, none.

Four fields, one schema. Employer name was always the one worth watching hardest, and nobody had drawn this before.

Then came a Thursday. Selin was cross-checking an unrelated fraud flag, a stub that had tripped a duplicate-employer rule, and she noticed the employer name StubSnap had extracted didn't match anything on the photo at all. It read the driver's own name in the employer field, off a stub from a payroll processor StubSnap had never seen before.

That stub had already passed schema validation. Green checkmark, on time, no flag. The only reason anyone caught it was a fraud rule built for something else entirely.

Selin pulled the last six weeks of employer name extractions herself that Thursday night. 41 stubs where the model's own confidence, a number that existed the whole time but nobody surfaced, had already dropped under 80 percent.

The drift started in week one. Nobody looked at this number until week four, by luck.

When the schema check was first designed, nobody suggested tracking each field on its own. "The record either validates or it doesn't," someone said in that planning meeting, and it was a reasonable thing to say when every field was reading well above 90 percent.

The real question was never whether StubSnap overall was reliable. It was whether any one field, the one a fraud decision actually leans on, had quietly stopped being trustworthy while the record-level number stayed calm.

The fix isn't a smarter model. It's a decision that already had a number to act on.

With a per-field bar in place, week three's dip to 79 percent on employer name would have started routing those stubs to a person automatically, two full weeks before the near miss, instead of the whole batch riding on a record-level flag that had no idea anything was wrong.

Two weeks earlier, or one green checkmark on a fraud rule that had nothing to do with any of this. That's the whole gap between catching it and getting lucky.

What I'd tell myself, the night Selin found that stub: a number that stays quiet isn't a number that's fine. It's a number nobody built a way to hear.

LEAD, mapped onto one eval designNot a script for distrusting every field equally. LEAD is what tells you which one to watch.

The four letters, held up as one page, with the early signal as the one worth remembering.

The recap, one line per letter: link is wrongful advances avoided and drivers approved on time, early signal is employer name's own confidence tracked apart from the record-level pass rate, abuse is loosening the schema or rubber-stamping shaky records, and decision is the three-tier routing bar that turns a number into an action.

Every one of these three would make the dashboard look better while StubSnap got worse.

And if you want to be sure it really works, try it somewhere elseSame four letters, a city permit office instead of a gig-pay app. The schema changes, but the seam sits in the same place.

Wendell Okoro runs the permitting desk for Civica Permits, a system that reads scanned construction-permit applications into a fixed schema: applicant name, parcel ID, permit type, and structure square footage. Mapped onto LEAD: link is permits routed to the right inspector without a manual re-key. Early signal is parcel ID's own field confidence, tracked apart from the overall form-pass rate, since a wrong parcel ID sends an inspector to the wrong property entirely. Abuse is a clerk marking a low-confidence parcel ID as "confirmed" just to clear a backlog queue. Decision is the same three-tier bar: high confidence auto-files, medium routes to a clerk, low kicks the scan back for a clearer photo.

Swap the trigger and it still runs.
Speed: an interviewer caps you at sixty seconds. Say "score every field on its own, and let the riskiest one set the bar," and stop.
Cost: no budget yet for a labeled golden set. Say so honestly, and commit to hand-labeling 200 real records in the first sprint instead of skipping the whole idea.
The model got better, for real: if a new version pushes employer name's confidence up to 97 percent, that's the moment to loosen the review bar deliberately, not the moment to stop watching the number.

Where people run it wrong.
They grade the whole JSON object as one pass or fail, so a single weak field hides inside a healthy-looking average.
They build the golden set from clean, synthetic examples instead of the messy real stub layouts the model actually sees in production.
They treat a confidence score as a nice-to-have UI detail instead of the number the whole review decision should run on.

How to use it live. The moment an interviewer asks how you'd evaluate structured output, ask yourself: if I only got to trust one number, would it be the record passing, or one field's own confidence? Say the second one out loud, and the rest of the answer follows.

Flashcards (tap any card to flip it)

Check yourself Score: 0 / 0