What is the risk of an internal-only launch phase, and what does it catch?

How to answer this, stage by stage

Eight moves. The number that matters shows up in stage five, and the whole answer turns on what a clean number can and can't be proof of.

Let's learn

Say Palewick Mail builds a small AI classifier that reads incoming mail and decides, in a fraction of a second, whether it is safe, spam, or someone trying to trick the person reading it.

The team ran it inside Palewick first, on their own 620 employees, for three weeks. It caught 638 of the 640 phishing emails that reached those employees' inboxes. That's 99.7 percent, the cleanest number the mail-security team had ever shipped with.

Then it went live for all 900,000 paying mailboxes. Three weeks in, a security audit checked it against 140 confirmed real phishing attempts from that window. It only caught 99 of them. Forty one got through.

Here's the turn. Forty one missed emails is not really the risk. The risk is what the team did before any of that ever happened. They saw 99.7 percent inside their own walls and treated it as proof the product was ready for everyone else. So they skipped the small test with real customers that existed to catch exactly this kind of miss.

At its worst, this costs more than 41 emails. Efua Amankwah runs a five-person bookkeeping firm and is a Palewick customer. She opened an email that looked exactly like the last four invoices from her firm's biggest supplier, same logo, same reference number, one line changed: a new account for the payment. She wired $14,200 that afternoon. Two days later the real supplier called asking why the invoice was still unpaid.

The internal number kept climbing. Whether an outside test happened at all only ever had two settings.

How ready a launch is isn't a dial you read off a clean number. It's a switch with two positions, and only one of them is proof.

Same classifier, two very different audiences. One grid is almost entirely clean. The other isn't.

What I would leave alone. The internal phase is still exactly right for catching pipeline bugs, does the classifier actually run, does the quarantine folder actually work, and for obvious bulk spam. A mass low-effort spam blast looks about the same whether an employee or a stranger receives it, so there's nothing broken in testing that on Palewick's own mail first.

The lesson. A clean number from people who already work at the company isn't proof the product is ready for people who don't. It's proof the product survived the one audience least likely to be fooled by it.

Now here is the same thing as a story

Pull this one out when there's more time, and you want the interviewer to feel the gap, not just note it down.

The rollout runbook lived in a shared doc with three checkboxes on it, and for two years Kerem Dogan never shipped a mail-security feature at Palewick until all three were ticked: internal, external beta, general release.

Kerem had shipped nine of those features already. Hand him a week of quarantine logs and he could tell you, before lunch, whether a miss was a fluke or the start of a pattern.

The first time the runbook really got tested was an auto-declutter feature for messy inboxes. The external beta ran on five percent of Palewick's customers for three weeks, and it agreed with the internal number almost exactly. Nobody thought twice about it. The second time, an attachment safety scanner, the beta shrank to one percent of customers for one week, because two years of these launches had taught the team that the beta mostly just confirmed what internal testing already showed. Nobody minded. It still counted as done.

This time, the phishing and spam classifier, internal testing came back at 99.7 percent. Cleaner than either of the two launches that got a real beta. At the go/no-go meeting, someone asked whether they really needed two more weeks with a slice of paying customers for a number that was already this good. Kerem agreed they didn't. They ticked all three boxes and shipped to all 900,000 mailboxes the same week.

Three weeks later, Efua Amankwah opened an email that looked exactly like the last four invoices from her firm's biggest supplier. Same logo. Same reference number. One line changed: a new account for the payment. She runs a five-person bookkeeping firm out of a strip mall, and she wired $14,200 that afternoon. Two days later the real supplier called asking why the invoice was still unpaid.

Kerem pulled the quarantine logs the way he always does. Not a fluke. A security review comparing 140 confirmed phishing attempts from Palewick's first three weeks live against what the classifier actually caught found it had missed 41 of them, almost all of the personalized kind, an invoice with one changed line, a message using a real vendor's name.

I want to say the internal number was wrong. It wasn't. Palewick's own 620 employees really do get plenty of real phishing, and the classifier really did catch 99.7 percent of it. But Palewick employees work at an email company. They notice a mismatched reply-to address before their coffee's cold, and none of the phishing that reached them that month was written to look like it came from one specific vendor they already trusted. Kerem never had a number telling him the beta was unnecessary. He had a feeling, and the feeling only had two settings: this number is clean enough to ship on, or it isn't. 99.7 percent flipped the switch. Nothing was going to flip it back before the next real customer opened an email.

So here is the decision I would take back.

At that go/no-go meeting, the runbook's three checkboxes had quietly become two and a half. Nobody had ever voted to remove the external beta. Two clean launches in a row had just made it feel like a formality, and on the third launch, the cleanest one yet, the team let an internal result stand in for a box that was never actually checked.

I would put the beta back as its own real gate, one that a good internal number can't excuse. Run the identical launch again with that gate real. Internal still comes back at 99.7 percent. The classifier still ships to a beta first, two percent of paying customers, about 18,000 mailboxes, for two weeks. In week one of that beta, a security review of the beta's own confirmed phishing attempts finds the same gap, personalized, vendor-styled attempts getting through at a much lower rate than the internal number promised. The team adds a check that flags a changed payment detail against a vendor's known history, retests, and reaches 95 percent on that category before opening to all 900,000 mailboxes, four weeks later than the original date. None of the roughly 18,000 beta customers lose money to the gap, because the beta is small enough that a miss shows up as a pattern in a security review, not as one customer's afternoon.

If I'm honest, agreeing to skip the beta at that meeting wasn't the real mistake. Two clean launches in a row will talk anyone into believing the third one doesn't need the same care. The real mistake was never writing down what the internal phase could actually prove, so that on the one day the number looked its best, nobody in the room could point to the sentence that said: this measures whether our own trained eyes get fooled, not whether a stranger's does.

FLIPS, run against the week Kerem shipped straight to everyone

The letters matter less than which one breaks first. Here's the same five steps, mapped onto Palewick's third launch.

FLIPS, five rows

"They should have tested more carefully" is a diagnosis anyone can offer after the fact. The harder part is naming the exact checkbox that got merged into another one, and showing there was no cheaper fix once a real customer had already found the gap.

And if you want to be sure it really works, try it somewhere else

Grovemark Learning is nowhere near email security. It builds an AI assistant that gives students written feedback on essay drafts, and it piloted the assistant entirely with teachers at Pinegate Academy before ever letting a student see its feedback directly. Same question, a different flip this time. Nobody stops requiring an outside check. The people running the pilot quietly narrow what they feed it.

F. Aleksandra Wrobel, curriculum lead at Grovemark Learning, ran the three-week teacher pilot at Pinegate.
L. Teachers stopped feeding the assistant truly rough student writing, first drafts with texting shorthand, no paragraph breaks, half-finished thoughts, and kept testing it on the exemplar essays they already knew graded consistently well.
I. A different flip from Kerem's. Nobody stopped checking. The forty teachers in the pilot quietly narrowed which essays they fed it, real classroom range in, only the clean, already-strong writing out, because picking a good example took less time than pulling a messy real one from the queue.
P. At the pilot's kickoff, the default rule was "use whatever recent assignments you already have graded," instead of requiring a fixed slice of real, unedited first drafts pulled straight from the queue.
S. Require every teacher's sample to include a slice of unedited first drafts pulled automatically, not hand-picked. Round two: the same essay, graded twice, comes back consistent only 58 percent of the time on that messier slice, a gap the clean sample never showed. The team fixes the scoring rule for informal writing by week two of the pilot, not after 3,000 students start seeing the feedback directly, and consistency on retest reaches 89 percent before student-facing launch.

Flashcards (click a card to flip it)

Eight fixed slots, pulled straight from the answer above.

Check yourself Score: 0 / 0