What is the latency cost of adding a reranking step, and when is it worth it?

Concordis Labs matched papers to reviewers with one merged pipeline, retrieval and reranking bolted together with a single on off switch. A fix for a slow search box quietly turned off the one step built to catch a real conflict of interest, and nobody noticed until an audit did.

How to answer this, stage by stage

Nobody is testing whether you know reranking makes a result list better. They are testing whether you will price the exact moment its extra time stops being free, or just say "reranking helps, so use it" and stop there.

Let's learn

Concordis reads a submitted paper and suggests which reviewers on a conference's list are actually qualified, and free of any conflict, to review it.

Before Concordis added a reranking step, matching ran on retrieval alone. It turned each paper into one number pattern called an embedding, compared that pattern against a similar pattern for every reviewer's bio, and returned the fifty closest matches in about 38 milliseconds. Fast enough that a chair typing a search barely noticed it happening at all.

Two stages, two very different clocks. Retrieval barely registers. Reranking is where all the extra time lives.

With reranking added, that same fifty name list gets read a second time, this time pairing the paper's real text against each candidate's actual bio and declared conflicts, before it is reordered. That second pass adds about 650 milliseconds on a normal day, and closer to 900 under load. On a background batch run that assigns thousands of papers overnight, nobody notices. Typed into the chair's live search box, it turns an instant list into one that visibly hangs.

Here is the part that matters: those extra 650 milliseconds were never the problem. What the team did about them was. Faced with a search box that suddenly felt slow, they had exactly one switch to flip, because retrieval and reranking had been built as a single merged step from day one. So they flipped it. Off. Everywhere. Including the overnight batch run that actually decides who reviews what.

A slow search box costs a second and gets noticed instantly. A missed conflict costs nothing to notice, right up until someone goes looking.

At its worst, this does not fail loud. Because retrieval and reranking shared one switch, a plain fix for a search box that "felt broken" also switched off the exact step trained to catch a reviewer sharing a recent advisor with a paper's author. Not because anyone decided conflicts stopped mattering. Because nobody built a way to turn the reranker off in one room without turning it off in the other.

The choice I would take back: in Concordis's very first release, built for a forty reviewer pilot workshop, retrieval and reranking ran as one pass because splitting them was not worth the extra code for such a small pool. Nobody revisited that decision once the product grew into conferences a hundred times the size.

What I'd leave alone: the very first filter Concordis runs, before retrieval even starts, a plain rule that drops any reviewer who marked a subfield "not interested." It is a yes or no fact sitting right there in the reviewer's own profile, not a close call, and reranking would only spend 650 milliseconds confirming what the rule already knows for free.

The lesson: a slow step is not a mistake by itself. Wiring it to only one switch is. The reranker was always right to exist. It just needed its own light switch, in the one room that actually needed it, instead of sharing a breaker with a room that did not.

Now here is the same thing as a story

The short version is above. Read the long version below for why one spot check, two weeks before a deadline, changed how Concordis decides which switch controls what.

Reidar Kvale built Concordis's very first matching pass himself, for a forty reviewer pilot workshop where the whole reviewer list fit on one screen. He knew exactly why every part of that first pipeline existed, because he had written each part to solve one specific complaint from that workshop's own organizers.

The next two years went well. Concordis grew from that one workshop into conferences with hundreds, then thousands of submissions, and every cycle, a chair could type a rough description of a paper into the dashboard's search box and watch a ranked reviewer list appear almost as fast as they could read it. Leadership pointed to Concordis in every renewal conversation. Reidar kept the pipeline exactly as he had first built it, retrieval and reranking running together as one pass, because nothing had ever given him a reason to pull them apart.

Then Concordis signed its first conference with more than three thousand submissions, and the reviewer pool grew past twelve hundred people to match. The habit thinned in three small beats nobody flagged at the time: the live search box started taking most of a second to answer instead of a blink, a few admins started scrolling the alphabetical reviewer list by hand for quick fixes instead of typing a real query, and Reidar, watching a dashboard full of complaints about "the search feeling broken," reached for the only switch the pipeline had and turned reranking off, for everyone, calling it temporary.

Nikoline Aasen's spot check, two weeks before reviews were due. Twenty three assignments were fine. Two were not, and nothing in Concordis had said so.

The trigger was small. Nikoline Aasen, the conference's integrity chair, pulled twenty five locked assignments at random, a routine check she ran before every deadline. Twenty three looked exactly right. Two did not. Both were papers assigned to a reviewer who shared a recent advisor with one of the authors, the kind of conflict a plain retrieval match cannot tell apart from genuine expertise, because both a real expert and a former advisor use almost the same words in their bio.

Nikoline sent Reidar a short message with both paper IDs and one question: "how did these two get past the conflict check?" Reidar spent an afternoon tracing it and found the real answer sitting in a changelog from four months earlier, a one line entry that said "reranking off, search feels slow, revisit later." Nobody had revisited it. The reranker that would have read those two reviewers' bios against the actual author list, and caught the shared advisor in plain text, had simply never been asked.

The real cost was never the extra 650 milliseconds. It was that for four months, every single locked assignment at every conference running on Concordis went out with no reranking check at all, and on exactly the audit that mattered, two of them turned out to be wrong.

Same spot check, rerun after the fix. Splitting the switch did not slow the batch run down. It just gave the search box its own setting.

The decision Reidar would take back happened two years earlier, in the week he shipped that first pilot pipeline. "It's one product, it should have one setting," he told himself, and for a forty person pool that was true enough to never get questioned. Nobody in that room asked what happens once the pool passes a thousand people and two very different kinds of users are staring at the same switch.

Run that spot check again with the fix in place: reranking stays on for every batch run, no matter how the search box is behaving that week, and the search box gets its own fast, retrieval only path that never touches the reranker at all. Nikoline's next spot check, twenty five more assignments pulled at random the following cycle, comes back clean. Zero conflicts missed, and nobody has to send a quiet email to two authors explaining a reviewer swap two weeks before their deadline.

One pipeline made reranking a single breaker for the whole house. The other gave it its own switch, in the one room that actually needed it.

What I would tell myself in that first pilot week: "one setting" is not simpler, it is just a decision you have not been asked to make twice yet.

PICK, so the decision holds up under questions

This is not a speed versus quality question wearing a technical costume. PICK is what keeps "the fast path felt right" a real, defensible decision instead of a guess that happened to work.

Two things worth stating directly, since this is where the real judgment sits. The alternative Reidar's team considered and dropped was widening the retrieval stage itself, moving from the top 50 candidates to the top 200, using the same embedding model, instead of adding a separate reranking pass. It lost because retrieval scores a paper and a reviewer as two single number patterns, and widening the list just offers more of the same kind of match. On Concordis's own 400 example set of hand labeled conflicts, going from top 50 to top 200 moved the catch rate from 70 percent to about 74 percent. Adding a cross encoder rerank over just the original top 50 moved it to 97 percent. More retrieval was not the mechanism, reading the paper and the reviewer together was. The AI specific failure worth naming by name is silent drift: as reviewer pools grow and new subfields show up that did not exist when the embedding model was trained, retrieval alone quietly starts favoring reviewers whose bios share vocabulary with a paper rather than reviewers who are actually the right fit or actually conflict free, with no error message anywhere. The guardrail is that same 400 example hand labeled set, rerun before every conference cycle, so a drop in catch rate gets caught by a scheduled check instead of by whichever integrity chair happens to go looking.

And if you want to be sure it really works, try it somewhere else

Same four letters, a customs manifest instead of a paper abstract, and this time the hidden cost is not a review nobody double checks, it is a shipment that clears a checkpoint it should never have passed.

Wharfline is a routing tool a mid size customs brokerage built for assigning incoming cargo shipments to specialized inspectors. A shipment's manifest, the goods it carries, its origin, its customs code, gets matched to the inspector actually certified to clear it. Junio Ferraz runs inspection operations at the port where Wharfline's first large build shipped. The brokerage handles about 900 shipments a day. Retrieval matches a manifest to inspector specialty profiles in about 30 milliseconds, fast enough to sit inside the queue view officers glance at all shift. Reranking cross checks the exact goods description against each inspector's actual certification history, catching things like a hazardous materials endorsement that lapsed last month, and adds close to 500 milliseconds per shipment.

Wharfline's pipeline has one box Concordis never needed, a filter that skips reranking entirely for shipments too small to inspect at all.

Same rank as before, a different lever: for Concordis, the hidden cost was a review nobody double checked. For Wharfline, it is a regulated shipment cleared by an inspector who was never certified to catch what is actually inside it. The fix is not a live versus batch split this time, since nothing about a shipment gets typed into a search box mid thought. It is a volume filter ahead of the reranker, skipping reranking entirely for shipments that never reach the certification sensitive tier, so the reranker's time gets spent only where a miss actually matters.

Swap the trigger and it still runs.
Speed: an interviewer caps you at ninety seconds. Skip straight to it, rerank the moment a real decision gets locked in, skip it on anything that is just a person looking something up, and never let a fast guess through unchecked on the cases where being wrong actually costs someone.
Cost: there is budget this quarter for either a faster reranker or a wider retrieval candidate pool, not both. The faster reranker wins for Wharfline specifically, because a wrongly cleared regulated shipment costs far more than a slightly wider net of candidate inspectors ever would.
The model got better, for real: say the reranker's latency gets cut in half next year. Real progress, and it narrows the gap. It does not remove the need for the volume filter, since a fast reranker still burns real compute on parcels nobody was ever going to send an inspector for.

Where people run it wrong.
They treat "reranking is more accurate" as reason enough to run it everywhere, without asking who is actually waiting on the answer.
They fix a slow interactive surface by turning reranking off system wide, taking the batch decision's own accuracy down with it.
They size the reranker's compute budget once, at launch volume, and never revisit it once real usage multiplies past that.

How to use it live. Say the real question out loud before naming a stage: "before I answer, is this the moment a real decision gets locked in, or a moment someone is just looking something up." That one question decides whether reranking's extra time is free or expensive.

Flashcards (tap any card to flip it)

Check yourself Score: 0 / 0