What does it mean for a model to be multimodal and what product surfaces does that unlock?

Vellacourt reads a job posting and rewrites a job seeker's resume and cover letter to fit it. Most postings get pasted in as plain text. Suvarna Rathnayake found hers taped to a laundromat bulletin board, a printed flyer for a warehouse job at Runcible Distribution, so she photographed it instead. Ozara Mainwaring runs product for Vellacourt, and for ten weeks nobody there knew that one small, creased line on flyers like Suvarna's kept vanishing on the way to the resume.

How to answer this, stage by stage

Nobody is grading whether you can define "multimodal" like a glossary entry. They're grading whether you can say, plainly, which pictures are worth building for and which ones are just a nicer looking text box.

Let's learn

Vellacourt reads a job posting and rewrites a job seeker's resume and cover letter to fit it. Type or paste the posting in, and out comes a resume and a cover letter built around that exact job, not a generic one.

Five steps. The middle one, turning a posting into fields Vellacourt can build with, is the step this whole answer turns on.

Vellacourt built the pasted-text path first. A job seeker copies a posting off a company site or a job board and pastes it in. About 22,000 resume and cover letter pairs a week come from that path, and extraction is close to perfect, because the text was already typed by a computer, clean, in order, nothing to misread.

Then Vellacourt added a second way in: take a photo or a screenshot instead of typing anything. Uploads grew fast, from about 400 a week to about 2,300 a week over ten weeks. Two very different kinds of pictures started arriving under the same button.

One picture is the only record of a job that exists. The other is the same words, in a picture instead of a paste box.

Some uploads were clean screenshots of a job board post, already digital text, just captured as an image instead of copied as words. Others were photos of real, physical flyers: taped to a wall, folded, lit by whatever light happened to be in the room. Vellacourt's own extraction accuracy split hard along that line.

The gap didn't show up as a wrong answer. It showed up as silence. Vellacourt's extraction gave every field a confidence score internally, but the product never surfaced it. A field read at 9 percent confidence and a field that was never on the page at all looked exactly the same to a job seeker: not there.

What it costs at its worst: Suvarna's flyer had one line in small print, cut slightly by a fold crease, "must currently hold valid forklift certification." She has one, current. Vellacourt's photo-upload flow read the header and the duties confidently, wrote her a generic resume about punctuality and stamina, and never mentioned her certification, because that line scored 9 percent confidence and got dropped with no flag, no note, nothing. She heard nothing back for eight days, until a friend mentioned the flyer's forklift line by name. She went and reread it herself, rewrote her cover letter's opening by hand, and resubmitted. Eleven days after her first, generic try, she got a callback.

What I would leave alone: the plain paste-the-text box. Roughly 22,000 tailored packages a week still start there. Pasting a posting captures everything already, and no picture makes that path faster or safer. Nothing about this problem touches it.

The lesson: test every new "let's add a picture" idea against one question, is the picture the only copy of the thing, or does it show something words alone would drop. If neither is true, it's a slower typing box wearing a camera icon, and the real cost isn't the feature, it's the confidence someone builds in a product that quietly wasn't reading what it claimed to read.

Now here is the same thing as a story

The short version above is what you'd actually say out loud. Read this one for the ten weeks that built up to a question nobody could answer on the spot.

Ozara Mainwaring had run product at Vellacourt for three years, most of it spent making the paste-the-text path faster: fewer clicks, a cleaner preview, a resume that read like it was written by someone who'd actually looked at the posting. It worked. Job seekers pasted a listing in, checked the preview, hit generate, and got something they'd actually send.

Photo upload was supposed to be the same idea, just easier. Not everyone finds a job online. Some people still find one on a corkboard.

Nobody decided, on any single day, to let a real requirement disappear without a word. A screen built for one kind of photo just never got built for the other.

For the first few weeks, photo upload looked like a clean win. Most people who tried it uploaded a screenshot, a job board post captured as an image instead of copied as text, because that's who tries a brand new button first: people already comfortable online. Extraction on those was nearly as good as pasted text. The team watched one blended accuracy number, and it barely moved.

Word of mouth carried the feature past the early, screen-savvy users. By week six, more uploads were coming from people who'd found a job the old way, on a wall, a windshield flyer, a printed sheet handed across a counter, and photographed it because someone told them the app could handle a picture now. Those were the uploads the blended number was quietly hiding.

Suvarna's flyer arrived in week eight. She'd walked past the Runcible Distribution posting at her laundromat a dozen times before she photographed it, dim overhead light, one fold crease running straight through the bottom corner where the requirements sat in small print. Vellacourt's model read the bold header and the main duties without any trouble. The forklift certification line, the one requirement that actually mattered most to that employer, scored a 9 percent confidence internally and got left out. Nothing on her screen said a line had been found and doubted. Nothing said a line existed there at all.

Everything else on the flyer made it through fine. The one line an employer actually screened for is the one that never reached her resume.

She sent the generic version. Eight days of silence followed. Then a friend, who'd walked past the same corkboard, asked her, almost in passing, whether she'd mentioned that Runcible specifically wanted forklift-certified people. She hadn't seen a line about that in her own resume, because she'd never seen it in the tailored version Vellacourt built for her. She went back to the actual flyer, found the small print herself, and rewrote her cover letter's opening line by hand to lead with her certification. Eleven days after her first, generic submission, she got a callback. Three days after that, an interview.

It surfaced properly a week later, almost by accident. The newest member of Vellacourt's quality team, three weeks into the job, was doing a routine sample of support threads and found Suvarna's, a short, mild note thanking Vellacourt "eventually." Curious, she pulled the internal extraction log behind it and found the forklift line sitting at 9 percent confidence, silently dropped, not flagged anywhere, not even in the team's own internal review tools. She asked Ozara a question nobody could answer on the spot: "If we saw this line and weren't sure, why didn't we just say so?"

The audit that followed pulled 500 photo-uploaded postings from all ten weeks since launch and reran them with confidence logging turned on. The rate of flyer-sourced tailorings carrying at least one silently dropped requirement had been climbing the whole time, invisible inside a blended number that mostly reflected the early, clean screenshots.

By week nine, about 39 percent of that week's roughly 2,300 photo uploads carried at least one silently dropped requirement, close to 900 tailored packages in a single week, out into the world, missing the one line that might have mattered most.

The decision Ozara would take back sits in a planning meeting months earlier, when the photo-upload team debated whether to keep the editable preview screen for images. Someone argued it broke the magic: type nothing, get a resume, that was the whole pitch. Someone else pointed out the preview only ever caught small things on pasted text anyway. Both were true, for pasted text. Nobody in that room was thinking about a flyer with a fold crease through the one line that mattered.

Run the same ten weeks again, with the preview screen kept for every photo upload and a visible flag on anything scored under, say, 40 percent confidence. Suvarna's flyer still gets photographed in the same dim laundromat light. The forklift line still comes back at 9 percent. But this time it shows up on her screen with a small mark: "not sure about this line, worth checking." She glances at the real flyer, sees the requirement, adds it herself. Her first submission is the good one. No eight days. No friend's remark. No second try.

What Ozara would tell herself, back in that planning meeting: the magic was never the missing screen. It was the resume actually matching the job. Taking out the one thing built to guarantee that, to protect a feeling of speed, was never a fair trade.

The photo test: PICK, one letter at a time

Not a way to decide whether pictures are good or bad. PICK is what forces you to say, out loud, which pictures are actually worth the engineering, and which ones just look impressive in a demo.

One cost is small and everyone can see it happening. The other one hides behind a resume that looks, on the surface, completely finished.

Three things worth saying directly, since this is where the real judgment sits. Vellacourt considered a stricter fix instead of confidence scoring and a review screen: raise the extraction threshold so anything below a high bar gets left blank across the board, no guessing at all. It lost, because plenty of low-contrast lines on real photos were still read correctly, and a blanket threshold would have thrown those out too, trading one kind of missing information for a different kind, instead of just telling the user what the model wasn't sure about. The AI-specific failure worth naming by name is a silent low-confidence drop: a model that is unsure behaves exactly like a model that found nothing, and nothing in the product told anyone the difference. The guardrail is confidence-scored extraction paired with a mandatory review step for anything under the bar, not an engineering log only the team itself ever looks at. And the trade-off is real and accepted on purpose: the review screen adds 20 to 30 seconds to every photo-sourced session, a small, deliberate cost against ever again generating a resume that quietly leaves out the one line an employer was actually screening for.

And if you want to be sure it really works, try it somewhere else

Same four letters, an insurance claim instead of a job flyer, and this time the picture that matters is a dented bumper, not a printed page.

Castleton Mutual lets a policyholder file an auto claim by typing a description of the damage or uploading a photo of it. Osaretin Braithewaite runs AI product there, and faced a version of the same question Ozara did: which claims actually need the photo, and which ones are just asking for a picture because a picture feels more thorough?

Same shape of pipeline as the resume one. The step that decides everything is still the model, and still for the same reason, what it was actually shown.

Mapped onto PICK: Position, a photo genuinely unlocks a new surface when it shows something a typed description can't capture as precisely, dent depth, the exact point of impact, the color of paint transferred from whatever hit the car. A typed "small dent, rear bumper" collapses three different repair tiers into one vague sentence. Impact, undersell it and an adjuster keeps asking policyholders to type longer and longer damage descriptions that still miss the details that set the price; oversell it and Castleton starts asking for photos of paperwork, an insurance card, a VIN, that were already typed into the claim form, adding friction for zero new information. Cost asymmetry, a demo of "upload a photo, get an instant estimate" is quick to build; making it reliable across bad phone cameras, night photos, and rain-streaked bumpers is the expensive part, and skipping it means a wrong repair tier that only surfaces when a shop's own estimate doesn't match. Kill criteria, the same test travels unchanged: does the photo carry something the typed description genuinely could not. Damage, yes. A document that was already typed elsewhere, no.

Swap the trigger and it still runs.
Speed: an interviewer caps you at ninety seconds. Skip straight to the test: does the picture carry something the words alone would lose? If not, it's a gimmick, full stop.
Cost: no budget to build confidence scoring this quarter. Keep the mandatory preview screen for every photo upload instead, so a person catches what the model quietly missed, not nobody.
The model got better, for real: say the extraction model doubles its accuracy on messy photos next quarter. Keep the review screen anyway. Being better at reading blur is not the same as never being wrong about the one line someone actually needed.

Where people run it wrong.
They add a camera icon to a screen that already has a working text box, because a photo demo feels more impressive, whether or not the photo adds anything real.
They let "the model read it" and "the model actually knows what it read" collapse into one thing, so a low-confidence guess looks exactly like a correct answer on the screen.
They test the feature on their own clean, well-lit phone photos, then ship it for flyers taped to a wall in bad light, and call the gap a rare edge case instead of the whole real world.

How to use it live. Before answering, ask yourself one thing out loud: if I made the user type this instead of photographing it, would they lose anything real? If you can't name what they'd lose, you haven't found a real surface yet, you've found a nicer looking text box.

Flashcards (tap any card to flip it)

Check yourself Score: 0 / 0