How do embedding and re-embedding costs enter a RAG cost model?

Larchmarket's product catalog holds 2.4 million listings, and every one of them needs a fresh embedding whenever its text changes. Embedding it all once was cheap. The real cost was buried in how the system decided when to do it again.

How to answer this, stage by stage

Nobody is grading whether you know what an embedding is. They're grading whether you'll treat "embeddings" as one cost or catch that it's really two, one of which quietly scales with catalog size instead of catalog change.

Let's learn

Kinbasket is the part of Larchmarket, an online marketplace, that decides what to show next to a listing you're looking at: "shoppers who viewed this also liked," built by comparing what each listing's text actually means instead of just matching its category tag.

Before anything like Kinbasket, Larchmarket's "similar items" panel was a plain rule: same category, same price band, sorted by rating. That worked fine for a phone case. It produced nonsense for anything unusual, a left-handed kitchen tool would get paired with any other left-handed item regardless of what either one actually did, because the rule never read what a listing said, only which box it was filed under.

Kinbasket reads what a listing actually says instead of stopping at its category tag, so a listing for "waterproof hiking boots, wide fit" correctly pairs with one that says "wide, weatherproof trail boot" even though the wording barely overlaps. Getting there means every one of Larchmarket's 2.4 million listings has to be turned into one of those rows of numbers, and kept turned into one, as sellers keep editing what they sell.

The turn: Kinbasket's recommendations were never the problem. Whether the embeddings were good was never really in question either. The real risk was how the system kept 2.4 million embeddings current as the catalog kept changing underneath it, and the honest way to do that turned out to cost a completely different amount depending on one design choice nobody had revisited since the catalog was sixty times smaller.

At its worst, this isn't really about $1,872 being an alarming number on its own; plenty of infrastructure lines cost more. At its worst, a re-embedding job that scales with catalog size instead of catalog change can quietly become the single largest line in a whole recommendation system's budget, bigger than search serving, bigger than the index hosting itself, while sitting under one vague dashboard row labeled "embeddings" that nobody ever broke apart to look at closely.

The honest range for delta re-embedding barely moves, $238 to $247. The naive alternative sits in a different world entirely: $2,091 a month for re-embedding work that mostly repeats itself.

What I'd leave alone: the query-time embedding line doesn't need this kind of scrutiny. It already runs on a small, cheap model, and its cost scales predictably with how many shoppers show up, not with how large the catalog gets, so there's no full-versus-delta trap hiding inside it.

The lesson: a pipeline that "just works" can still be quietly billing you for the same wrong decision every single night. Cost that scales with the size of a thing, instead of how often that thing actually changes, doesn't announce itself. It just gets a little bigger every month the thing it's attached to grows.

Now here is the same thing as a story

Read the long version below when you want to feel why a job that never broke still cost eight times more than it should have, not just be told that it did.

Ozias Kirchoff had built search indexing systems before he joined Larchmarket, and when Kinbasket needed its first embedding pipeline, he built the version he trusted most: wipe the vector index, rebuild it from scratch, every night, for every listing. No change events to track, no partial state that could ever drift out of sync with what a listing actually said. If something went wrong, tomorrow's rebuild fixed it automatically.

At 40,000 listings, mostly first-party goods before Larchmarket opened up to outside sellers, that nightly rebuild cost about $31 a month. Nobody blinked. It shipped, it worked, and Wenna Vantras, who owned Kinbasket's cost model, signed off on a line so small it barely needed its own row on a spreadsheet.

Third-party sellers started joining a few months later, and the catalog grew the way marketplaces grow: slowly, then all at once. By month four it had passed a quarter million listings. By month eight, nine hundred thousand. The nightly rebuild kept running exactly as designed, every listing, every night, and its cost climbed in a straight line right alongside the catalog, quietly, inside a general "AI infrastructure" budget line that nobody had ever asked Kinbasket to break out of.

Nobody noticed a single bad day. There wasn't one. The catalog kept growing, the job kept running, and the monthly line kept being a little bigger than the month before, for over a year, without ever once failing or throwing an error that would have made anyone stop and look at it.

It surfaced because a new engineer, walking through the recommendations codebase in her first week, found the nightly reindex job and asked Ozias a plain question in the team chat: why are we re-embedding all 2.4 million listings every night, when most of them haven't changed in months? Ozias didn't have a real answer beyond "that's how it's always worked." He went and pulled the job logs to find out for himself.

What he found: of the listings re-embedded the previous night, about 61 percent hadn't changed at all in the prior 30 days. The job wasn't broken. It had simply never been told the difference between a listing that changed and one that hadn't, because at 40,000 listings, with no change-tracking system built yet, that difference had never been worth building.

Ozias's first instinct was the sensible one: check whether the nightly job could just run less often instead, maybe every third night. Wenna talked him out of it fast, because a slower clock still doesn't know which listings actually need re-embedding, it just makes stale ones stale for longer between rebuilds. What they actually needed already existed: Larchmarket's catalog service emitted a change event every time a seller edited a listing's title, description, or attributes. Nobody on the recommendations team had ever wired Kinbasket's re-embedding job to listen for it.

The decision that opened the door went back to that very first design meeting, when the catalog was small enough that "just rebuild everything" wasn't a real cost, it was a convenience. Nobody in that meeting decided the nightly job should still be re-embedding sixty times as many listings a year and a half later without changing shape. It just kept working, the way a decision that never once fails stops looking like a decision anyone made.

Run that same first-week discovery again, with the change-triggered job in place from month one instead. The new engineer never gets curious about the nightly job, because there's nothing in the logs that looks strange: the re-embedding line tracks catalog churn, not catalog size, so it stays near $24 a month whether the catalog holds 40,000 listings or 2.4 million. The line that would have needed a hallway question never gets big enough to draw one.

One design let a job's cost scale with how big the catalog got. The other lets it scale with how much of the catalog actually changed, which is the only thing re-embedding was ever supposed to be paying for.

What Ozias would tell himself, back in that first design meeting: rebuilding everything nightly wasn't wrong when it cost $31 a month. It was a shortcut that had never been asked to survive a catalog sixty times its original size, and nobody had put a date on when to ask it to.

BOUND: the arithmetic behind Kinbasket's embedding line

Not a story question wearing a framework's clothes. This is an estimation problem, and BOUND is what keeps a nightly job that "just works" from quietly becoming the biggest line in the budget.

Three things worth stating directly, since this is where the real judgment sits. The alternative Ozias's original pipeline used, and the one Wenna's team moved away from, was the nightly full reindex, rejected once the logs showed 61 percent of nightly re-embeds were touching listings that hadn't changed in the prior month, at a cost of $1,872 a month for that wasted work alone. The AI-specific failure worth naming by name is silent embedding staleness: if a delta pipeline's change-detection lags or drops an event, Kinbasket keeps serving recommendations built off an out-of-date embedding, a discontinued color variant, an old category before a recategorization, and nothing about the recommendation looks obviously wrong, it's just quietly built on stale meaning. The guardrail is a staleness monitor comparing each embedding's last-updated timestamp against the listing's own last-modified timestamp, and forcing a re-embed the moment that gap crosses about 24 hours for high-traffic listings. That guardrail isn't free, it costs a small amount of extra monitoring and occasional forced re-embeds outside the normal delta flow, small next to what a nightly full reindex was costing to solve the same problem badly. And the bar Kinbasket holds itself to was never a fixed embedding cost per listing, no catalog this size and this changeable earns a fixed number. It's a range, rechecked whenever churn drifts, with re-embedding priced against what actually changed, not against how large the catalog happens to be that month.

And if you want to be sure it really works, try it somewhere else

Same five letters, a veterinary clinical library instead of a shopping catalog, and this time the lever isn't when re-embedding fires, it's how much text gets re-embedded each time it does.

Furlex is a RAG assistant Oakspindle Veterinary Group built so vets can query treatment protocols and drug interaction guidance mid-consult. Thaddea Nyquist runs digital operations for Oakspindle's clinic network.

The build-up: Furlex's library holds 42,000 documents, drug monographs, species-specific treatment guides, dosage tables, averaging about 1,800 tokens each once embedded. The one-time full embed, on a higher-accuracy clinical model at $0.50 a million tokens, runs about $37.80, amortized to roughly $2.10 a month. About 2,500 documents get a clinically meaningful edit every month, a revised dosage table, an updated interaction warning, new species guidance. That's the real recurring line.

Re-embedding a whole 1,800-token document every time one 180-token dosage paragraph gets revised costs about $2.25 a month across those 2,500 monthly edits. Re-embedding just the changed passage instead costs about $0.23, roughly a tenth as much. Furlex's total bill is nowhere near Kinbasket's, a few dollars a month against a few hundred, but the exact same trap is sitting inside it, just at a smaller sticker price: re-embedding more text than actually changed.

Same method, different lever: for Kinbasket, the lever that swung the estimate was when re-embedding fires, change-triggered versus scheduled. For Furlex, timing was never the issue, every re-embed there is already change-triggered. The lever is what unit gets re-embedded once it fires: the whole document, or just the passage a vet actually edited.

Swap the trigger and it still runs.
Speed: an interviewer caps you at ninety seconds. Skip straight to it: split the one-time embed from the recurring re-embed, trigger the recurring one off a real change, then check what unit of content actually gets re-embedded when it fires.
Cost: there's no budget this quarter for both a change-event pipeline and a paragraph-level chunking rebuild. The change-event pipeline wins first, since a schedule-triggered job scales with catalog size forever if nobody touches it; paragraph-level chunking is a smaller, later win once the bigger trap is closed.
The model got better, for real: say the catalog embedding model gets a price cut next quarter. That helps the catalog-embed and re-embed lines equally, dollar for dollar. It does nothing to fix a nightly-reindex design; a cheaper price times a wasteful trigger is still a wasteful trigger, just a slightly cheaper one.

Where people run it wrong.
They treat "embedding cost" as one line and never separate the one-time build from the recurring cost of keeping it current.
They pick the simplest re-embedding trigger, a schedule, because it's easiest to build, and never revisit it once the catalog outgrows the size where that was ever a safe default.
They compare the recurring line only to the total monthly bill, never to the catalog's own churn rate, so a line that looks small next to everything else is quietly doing far more work than it needs to.

How to use it live. Ask the real question before quoting a number: "before I give you a monthly figure, is this index rebuilt on a clock, or only when something in it actually changes?" That buys a beat to think, instead of repeating a comfortable total that was never checked against how the re-embedding job actually decides when to run.

Flashcards (tap any card to flip it)

Check yourself Score: 0 / 0