Describe the first internal AI tool you would build at a 500-person company.

Interviewer's question: "Describe the first internal AI tool you would build at a 500-person company." Gable Point Rx processes prior-authorization requests for a few dozen employer health plans. Adaeze Chukwu has led prior-auth review there for seven years.

How to answer this, stage by stage

Nobody is grading how many features you can list. They're grading whether you can name the one design decision the whole tool actually depends on.

Let's learn

CiteCheck is a tool that reads a prior-authorization request, the member's plan rider, and the current drug formulary, and drafts a determination with the exact clause and formulary line behind it.

Gable Point Rx handles prior-auth for a few dozen employer health plans, each with its own rider and its own formulary. Before CiteCheck, Adaeze and her team cross-reference a paper binder of plan riders against a PDF formulary by hand, about fourteen minutes a request, mostly spent finding the right clause rather than deciding anything.

The turn: the fourteen-to-four-minute drop is not, by itself, the win. What matters is whether reviewers still open the underlying document on the cases where the citation could be stale, and don't quietly start trusting every citation the same amount.

Same request, same reviewer. What changes is how much of the fourteen minutes was ever real judgment.

Four small parts. The second and third are the only reason this is a citation tool and not a guessing tool.

At its worst: a plan renews, its formulary updates, and CiteCheck keeps citing the prior year's version with full confidence, because nothing in the design ever asked it to check its own citation's age. A reviewer who has learned to trust the citation without opening the source signs off on a wrong determination, and a member either waits weeks longer for a drug they need, or gets approved for one their new plan no longer covers.

What I would leave alone: specialty-drug appeals, the hardest and highest-stakes case Gable Point handles, doesn't need CiteCheck on day one. Those cases already get a senior reviewer's full attention, and building for them first would delay the routine ninety percent that actually needs the help.

The lesson: the first internal AI tool at any company should be judged by whether its output can be checked in seconds, not by how accurate it claims to be. Accuracy you can't check is a rumor with good production values.

Now here is the same thing as a story

The short version above is what you'd say pitching CiteCheck to Gable Point's operations lead. Read this one for how the near miss actually happened.

Adaeze Chukwu has reviewed prior-auth requests at Gable Point for seven years. Hand her a rider and a formulary and she can find the relevant clause faster than most people can open the right PDF.

For CiteCheck's first three months, Adaeze opened the underlying plan document on every single determination, just to see if the tool's citation actually matched. It always did. By month four, she was opening it on maybe one in five. By month six, she mostly just glanced at the citation snippet CiteCheck showed inline and signed.

The near miss came on a Tuesday in March, right at the start of renewal season. A member's employer had renewed its plan two weeks earlier, moving a common blood pressure medication onto a tier that no longer needed prior approval at all. CiteCheck's index hadn't caught up yet. It cited the old formulary, confidently, and drafted a denial: prior approval required, not on file.

Same wrong citation, two different desks. Only one of them has a version date printed where a reviewer can actually see it.

Adaeze almost signed it. The citation looked exactly like every other one she'd stopped double-checking. What made her pause was a habit older than CiteCheck itself: renewal season was when she used to double-check everything, tool or no tool, because it was the time of year things moved underneath her.

Here's the decision I'd take back: CiteCheck's formulary index defaulted to "most recently processed" without ever showing which plan's renewal cycle it was still catching up on. That made sense at launch, when every formulary in the system really was current. It stopped making sense the first renewal season, when some plans had updated and others hadn't, and nothing on screen told a reviewer which was which.

Replayed with a version date on every citation: the same March request comes in. CiteCheck still cites the old formulary tier, but the citation now reads "Formulary v.2024, indexed 11 months ago" in plain text next to it. Adaeze sees the date, checks the plan's actual renewal record, finds the mismatch in ninety seconds, and approves the medication the same day instead of drafting a wrongful denial.

I let the index default to "most recent" because at the time, that word meant something true. It took a near-denial during renewal season to see that "most recent" and "current" quietly stop being the same word the moment a plan renews and the index hasn't caught up.

SPARK, mapped onto one citationNot a feature list. SPARK is what tells you why the citation, and not the recommendation, is the actual design.

Each of these is a real feature Gable Point could build eventually. None of them is this design's job today.

Only the bottom-right corner is where trusting the citation without a second look is actually safe.

The step that disappears isn't the reviewer's judgment. It's the fourteen minutes spent hunting for where to apply it.

The recap, one line per letter: situation is a reviewer cross-referencing a binder by hand, payoff is trusting the routine case while still checking the renewed one, anchor is the dated, versioned citation, risk is a confident citation from a stale formulary, and keep out is holding back on auto-approval and specialty appeals until the routine case is solid.

And if you want to be sure it really works, try it somewhere elseSame five letters, a university's course-scheduling office instead of a pharmacy benefit manager. A completely different domain, the same anchor-is-the-citation shape.

Alderfield State University's registrar office pre-checks whether a student's course substitution request satisfies a degree requirement. Rosalind Okafor has advised transfer students there for nine years.

Mapped onto SPARK: situation is Rosalind cross-referencing a student's transcript against a printed degree-requirement sheet by hand, about twenty minutes a request. Payoff is advisors trusting the tool's citation on common substitutions while still reading the actual requirement on an unusual major or a recent catalog change. The anchor is structurally the same: every substitution recommendation shows the exact degree-requirement clause and catalog year it used, not just "approved" or "denied." Risk is the same shape too: if the university updates its catalog mid-year and the tool cites last year's requirement with full confidence, an advisor who's stopped checking could approve a substitution that no longer satisfies the new requirement. Keep out is no automatic approval without an advisor's sign-off, no use for graduate-level degree audits yet, no direct student-facing citations without review.

A different shape entirely, branching instead of side by side. The citation is still what decides which branch a request takes.

Swap the trigger and it still runs.
Speed: an interviewer caps you at sixty seconds. Say "show the exact citation behind every recommendation, dated and versioned, so a reviewer can check it instead of trusting it," and stop.
Cost: if showing a full citation snippet is too expensive to render at first, ship just the clause number and date, and say so honestly, a thin citation still beats a bare recommendation.
The model gets better, for real: if CiteCheck's accuracy climbs high enough that reviewers rarely see a wrong citation, that's exactly when a stale one becomes easiest to miss, not a reason to stop showing the date.

Where people run it wrong.
They ship the recommendation first and plan to "add explainability later," by which point reviewers have already learned to trust it blindly.
They show a citation but not its version or date, so a stale one looks identical to a current one.
They let auto-approval creep in once accuracy looks good, removing the one place a stale citation ever gets caught.

How to use it live. When someone asks for the first internal AI tool at a company, ask yourself one question: can the person using it check the answer in seconds, or only trust it completely? Build the anchor around making the first one true.

Flashcards (tap any card to flip it)

Check yourself Score: 0 / 0