Rule-Based vs AI Lead Scoring: Run Both for 30 Days and Let the Data Pick

A vendor told me their AI scoring model would lift our conversion. Maybe it would. But the ask was to rip out a rule-based model reps already trusted, wire the AI score into routing, and find out whether it was better by watching the quarter play out. That is not a test. That is a bet with a rollback measured in months. I said no, and instead we ran both models side by side for 30 days without touching how a single lead got routed. At the end the data picked, and I had a defensible answer either way.

This is shadow mode, and it is the difference between “we think AI is better” and “AI scored 30 days of real leads and converted them 12 points higher, here is the table.” AI scoring often does win, because a model trained on your history finds interactions a human point-table never encodes. But “often” is not “yours, on your data, this quarter.” Salesforce Einstein needs roughly 1,000 leads and 120 conversions before its scores mean anything, and under about 50 conversions a rules-based model is still the right call. The only way to know which one wins on your pipeline is to run them together before you trust either.

The catch is that a 30-day gap only means something once enough leads have resolved. Read the Hot-band conversion gap on day nine and it swings on a handful of conversions; read it on day 30 and the error band has closed under the gap. That convergence is the reason the field stays inert until the run ends, and it is the whole argument against cutting over early.

What shadow mode is

Your live rule-based model keeps running and keeps controlling routing. Nothing changes for reps. In parallel, the AI model scores the exact same leads at the exact same moment, and you write its score to a second field that does nothing except sit there and get recorded. Every lead now carries two scores: the one that acted on it, and the one that would have.

Thirty days later you have a clean cohort of leads that both models scored, and you know what happened to each one: converted to opportunity, or did not. Now you can ask the only question that matters. Of the leads each model called “hot,” what share converted?

The setup, wired

You need one new field and one place to write to it. Keep the AI score out of every routing rule, every assignment flow, every alert. It rides along as a passenger with no hands on the wheel.

Lead.rule_score__c   -- existing, drives routing (unchanged)
Lead.ai_score__c     -- new, written on create/update, drives NOTHING
Lead.scored_at__c    -- timestamp so you can build a clean cohort later

Write ai_score__c on the same trigger that fires your rule score. If the AI call is an external model, call it async so a slow response never delays a lead. If the call fails, log it and move on. The AI score being briefly null is fine because it is not doing anything yet. That is the whole point of shadow mode: it can fail safely because it has no authority.

The build

Then you wait, and you resist every urge to peek and cut over early. Thirty days, or one full cycle of your fastest-converting segment, whichever gives you enough leads to trust the numbers.

The comparison that decides it

At day 30, pull the cohort and bucket both scores into the same bands. Then compute conversion within each band, per model.

-- For each model, within each score band:
-- what share of leads converted to opportunity?

WITH scored AS (
  SELECT lead_id,
         rule_band,          -- Hot / Warm / Cold from rule_score
         ai_band,            -- Hot / Warm / Cold from ai_score
         converted            -- 1 if became opp within window, else 0
  FROM lead_scoring_cohort
  WHERE scored_at BETWEEN :start AND :end
)
SELECT 'rule' AS model, rule_band AS band,
       COUNT(*) AS leads,
       ROUND(AVG(converted) * 100, 1) AS conversion_pct
FROM scored GROUP BY rule_band
UNION ALL
SELECT 'ai' AS model, ai_band AS band,
       COUNT(*) AS leads,
       ROUND(AVG(converted) * 100, 1) AS conversion_pct
FROM scored GROUP BY ai_band
ORDER BY model, band;

Here is the shape of a real result, rounded from a run I did.

The Hot and Cold rows tell the story at a glance:

Read the Hot row and the Cold row together. The AI model’s “Hot” bucket converted 34% against the rule model’s 22%, and it did that on fewer leads, which means it was more selective and more right. Its “Cold” bucket converted at 1% against 3%, so the leads it threw away were genuinely worse. That is the pattern you want: sharper at both ends. A model that only wins on the Hot row while dumping good leads into Cold is buying precision by losing volume, and you would see that in the Cold conversion staying high.

If the AI numbers do not beat the rules, you keep your rules, you tell the vendor exactly why with a table, and you spent 30 days and one custom field to avoid a bad cutover. That is a win too.

The faith cutover versus shadow mode

What separates this from a vibes migration

Three things keep shadow mode honest.

Same leads, same window. You are comparing the two models on identical inputs, so a good month or a bad month hits both equally and cancels out. Comparing this quarter’s AI results to last quarter’s rule results proves nothing because the market moved underneath you.

The passenger never touches the wheel. The moment the AI score influences routing, you have contaminated the test, because now the two models are seeing different downstream treatment. Keep it inert until the day you decide.

Conversion, not agreement. The tempting metric is “how often do the two models agree,” and it is useless. You do not care whether they agree. You care which one is right, and right means converted.

The traps that quietly break the test

Shadow mode is simple, which is why people are careless with it and get a clean-looking result that means nothing. Two traps recur.

The first is the peek-and-cutover. Someone sees the AI winning on day nine, gets excited, and wires it into routing early to “start capturing the upside.” The moment they do, the test is over, because the two models now see different downstream treatment and the remaining cohort is contaminated. Discipline here is not optional. The field stays inert until day 30, full stop, even when the early numbers look great, because early numbers on a small cohort are noise wearing a signal’s clothes.

The second is the wrong window. Thirty days is a default, not a law. What you need is enough converted-or-not outcomes to trust the conversion rates, and that depends on your sales cycle. If your fastest segment takes 60 days to convert a lead to an opportunity, a 30-day window measures almost nothing, because most leads have not resolved yet. Set the window to one full cycle of the segment you are testing, and make sure the Hot band has enough leads that a few coin-flips do not swing the percentage. A 34% Hot conversion on 260 leads is a signal. On 12 leads it is a rounding error.

Why the passenger design is the safe way to test AI at all

There is a broader lesson here that applies past scoring. The reason shadow mode is safe to run in production is that the new system has no authority. It observes, it records, it is compared, and it changes nothing until a human reads the evidence and decides. That is the same shape as the control layer for write-access agents, where the agent proposes into a queue and a human approves before anything applies. In both cases you get to evaluate an AI system against real data, at real scale, with a real blast radius of zero, because you separated the decision from the effect. Any time a vendor asks you to give a model authority before you have evidence it deserves it, the answer is to run it as a passenger first. Shadow mode is that principle applied to scoring.

What to do next

Add one field, write the AI score to it on the same trigger as your rule score, and route nothing off it. Set a calendar reminder for 30 days out with the query above pasted into it. When it fires, run the query, read the Hot and Cold rows, and let the table make the call. If AI wins, cut over with evidence. If it loses, you kept a working model and got a free benchmark. Either way you never bet the quarter on a slide. This is the same discipline as the pilot survival checklist: measure the outcome before you trust the machine.