How would you migrate when the new model has a different context window?

How to answer this, stage by stage

This is a yes-or-no about how to handle the documents that don't fit the new window, not a rule for every model swap Casenote will ever run, so PICK carries the answer.

One more thing before the walkthrough ends: this isn't a rule that every shrinking context window needs a full rebuild. If a firm's documents were almost all short already, a simple truncation with a clear, hard-to-miss warning might be the right, cheaper call. Say which direction the miss compounds as usage grows, and you've shown judgment instead of reciting "always chunk long documents."

Let's learn

Every 120-page vendor contract, before Casenote, cost a mid-level associate about 5 hours: read every page, flag the indemnification clause, the termination terms, the liability cap, by hand.

Casenote reads the same kind of document and writes a summary flagging exactly those clauses. With the model the tool launched on, one pass covers up to about 300 pages, so nearly every contract, NDA, and discovery bundle gets read start to finish. The summary comes back in under a minute. Full review, summary plus a spot check of the flagged pages, drops to about 40 minutes.

Then the team moved to a new model. It's faster and cheaper per page, and better at spotting risky language in whatever it actually reads.

The new model's window only covers about 75 pages. To hit a hard cutover date, the pipeline shipped the same way it always had: feed in the document, read what fits, summarize it. Nobody added a step for what happens past page 75.

For most documents, nothing changed. About 83 percent of everything Casenote processes is under 75 pages anyway. But for the other 17 percent, roughly 153 documents a day, the model only ever sees the first 75 pages. It still returns a full, clean summary. A clause list. A risk-flag section. A confident paragraph at the top. That is exactly what the model is built to do with whatever sits inside its window. It has no way to say fifty-five more pages existed and it never saw them.

This is a specific kind of AI mistake: the model gives a fluent, confident, complete-looking answer built from only part of the input, and nothing in that answer flags what's missing. Call it silent truncation. The guardrail that actually catches it isn't a smarter model, it's a length check at the door: anything over 75 pages gets automatically routed into the chunking pipeline, and every summary prints exactly what share of the document it read.

Here is the turn. The missing pages were never really the mistake anyone would catch. The summary still looks complete. Nothing errors. The format doesn't change. A reviewer who has trusted a clean-looking Casenote summary for two years has no reason to start rereading the source document now, on this document, of all documents. The failure isn't in what the summary says. It's in what never made it into the summary, and nobody reading it knows that.

Chunking costs something too, and it's worth saying plainly. A 130-page contract that took one model call and about 40 seconds under simple truncation takes four calls and close to three minutes under the chunked pipeline, because each piece gets summarized on its own before they're combined. That's the trade being made on purpose: slower and pricier on the long documents, in exchange for actually reading all of them.

At its worst: leave this unfixed across every client firm using Casenote, and about 38,250 documents a year go out with only part of them actually read, every one of them looking complete.

What I would leave alone. Every document under 75 pages, which is most of what Casenote handles. They fit inside the new window the same as they fit the old one. Chunking them would only add cost and delay for no reason at all.

The lesson. A context window isn't a speed setting, it's a promise about how much of the document actually got read. Change it without checking your own document lengths first, and you're breaking that promise without telling anyone who was counting on it.

Same migration, two very different sizes of miss

The page nobody reread until it was almost too late

Read this version when you've got a few minutes. It lands harder, because nobody argues with a page someone actually reread by accident.

Lior Bassam has run the summarization product at Casenote for three years. He's shipped two model migrations before this one, both clean, no client so much as noticing a change in the summaries they got back.

The third migration looked the same going in. The new model tested better on every internal benchmark Casenote had: sharper clause detection, better at telling a real indemnification risk from routine boilerplate. The only catch was the context window, a hard ceiling around 75 pages instead of 300. Lior's team ran the numbers, figured most documents would never notice, and hit the cutover date on schedule.

For six weeks, the weekly migration health report told Lior exactly what he needed to hear. Documents processed, on target. Error rate, unchanged. Client complaints, zero. He read that top line every Monday morning and moved on to the next fire.

He stopped opening the full report somewhere around week three. It had said the same thing for three weeks running, and four other launches that quarter were competing for his attention.

Then, on a Tuesday in the sixth week, Nkemdirim Kanu, a senior paralegal at Ansdell & Pell, one of Casenote's oldest client firms, was pulling an old vendor agreement for an unrelated dispute. Nothing was wrong with it, as far as she knew. She just happened to be rereading it, page by page, for a completely different reason.

On page 94 of a 130-page vendor services agreement worth about 2.4 million dollars over its term, she found an indemnification clause with no dollar cap on it. Casenote's summary, written six weeks earlier, had said plainly: no indemnification concerns identified.

She wasn't hunting for a bug. She emailed Lior directly. "Your summary said this contract was clean on indemnification. It isn't. Can you check what happened here?"

That was the whole trigger. One email, about one contract, from someone who wasn't even looking for a problem.

Here's what Lior found. The vendor agreement ran 130 pages. Casenote's new model only ever read the first 75. The other 55 pages, including the clause Nkemdirim found, were never inside the model's window at all. It didn't skip the clause. It never had the chance to see it.

Lior pulled the real numbers that afternoon. Across every client firm, about 17 percent of documents were longer than 75 pages. Over the six weeks since cutover, that worked out to roughly 4,590 documents that had gone out with a summary covering only part of what was actually there.

I want to say the mistake was picking a model with a smaller window. It wasn't. The new model was genuinely better at the part it could see. The real mistake was treating window size like a performance number instead of a coverage promise, the same way you'd treat a faster processor, when it was actually a decision about how much of a client's document Casenote would ever read.

So here is the decision I would take back.

Months earlier, at the migration kickoff, an engineer had actually flagged the window difference and asked whether long documents needed a separate plan. The honest answer at the time was that the cutover date was fixed by the old model's own vendor retiring it, and there wasn't time to build chunking before then. The team agreed to ship the swap first and revisit long documents in a follow-up sprint. Nobody put a date on that sprint. The health report looked fine, so nobody chased it.

I would take that back. I'd build the chunking pipeline, the piece-by-piece read with a coverage check, before the cutover, not after it, even if that meant the migration itself slipped by three weeks.

And the part I'd tell myself, if I could go back to that kickoff meeting: we changed how much of every client's document actually got read, and we let a dashboard that only counted "processed successfully" stand in for actually checking that.

PICK, run all the way through the numbers

This is a yes-or-no about how to handle the documents that don't fit the new window, not a rule for every model swap Casenote will ever run, so PICK carries the weight here.

Try the same four letters when the window gets bigger, not smaller

Faysal Rane runs product for Threadline, a tool that reads a customer's full support history, every past ticket, every chat, and hands the next agent a summary instead of a stack of old tickets to reread. Threadline is migrating too, but the window is moving the other way: the new model holds about four times as much text as the old one.

The easy move looked obvious: now that everything fits, stop trimming the history and feed in every ticket a customer has ever filed. Faysal ran PICK on it before shipping that.

P. Restructure stays the answer, just aimed the other way. Keep the relevance filter that picks the most recent and most topically related tickets, even though the bigger window could technically hold a customer's entire history. Don't feed in everything just because there's finally room.
I. Feeding in everything is felt by every customer on a call: slower responses, and a summary quietly diluted by tickets from three years ago that have nothing to do with today's problem, a known pattern where a model loses track of what matters in the middle of a long input. Keeping the filter is felt by Threadline's engineers, an ongoing system to maintain instead of one they get to delete now that the window seems to make it unnecessary.
C. Feeding in everything is cheap to ship, delete the filter, done, and it looks like a win because there's finally room. The cost is hidden: per-ticket inference cost roughly triples, responses get slower, and answer quality drifts down in a way no single ticket ever flags as wrong. Keeping the filter costs a known, ongoing bit of maintenance, small next to that.
K. Drop the filter only where the evidence says it's safe: for the slice of customers whose entire history already sits well under half the new window, with no accuracy drop on Threadline's own evaluation set. Above that, keep filtering, no matter how much room the window has.

Flashcards (click a card to flip it)

Check yourself Score: 0 / 0