Conversion propensity model
A first model with ROC-AUC 0.988 that used the session's own information, and the corrected version that only uses what is available before the decision.
ROC-AUC on test
0.878
95% CI: 0.869 – 0.887 (June 2017)
Top 5% lift
×7.8
The 5% of sessions with the highest score captures 38.8% of purchases
Sessions analyzed
393k
308k users, 6 months of Google Analytics
Conversion rate
1.3%
Highly imbalanced class
The problem
Using six months of Google Analytics data from the Google Merchandise Store (January–June 2017), the goal was to estimate, for each session, the probability that it ends in at least one transaction, so marketing could prioritize audiences and campaigns.
Only about 1.3% of sessions convert, so the imbalance drives almost every decision: how to split the data, which metric to report and how to interpret the score.
Conversion by channel
Before modeling I ran a decision-oriented exploratory analysis. The Referral channel converts at 4.7% versus just 0.8% for organic search, always showing each channel's n so noisy rates from small samples aren't trusted.
Hover over each bar to see the conversion rate and the sample size (n).
Class imbalance
Only 1.34% of the 392,892 sessions end in a purchase.
Conversion by device
Desktop concentrates the traffic and converts noticeably better than mobile or tablet.
On-site activity: converted vs. did not convert
Hits, pageviews and time on site, on a log scale because of their heavy skew. Pick a metric to compare the two groups.
Weekly evolution
Conversion rises 43% between the first and the last month even though weekly session volume barely changes (+6%).
Methodology
- 1
Temporal split
Training on everything before June 2017 and testing on June (63,578 sessions, 946 purchases). A random split would mix the future with the past.
- 2
Baseline and model
LightGBM with class_weight='balanced', num_leaves=20 and learning_rate=0.1, compared against a simple baseline. Tuning barely moved PR-AUC (+0.45%), and I documented it that way.
- 3
Decision threshold
Threshold chosen to maximize F1 on the temporal test set. Because class balancing shifts the probabilities, the score is used for ranking, not as a literal probability.
- 4
Explainability
Global and per-session SHAP to show why the model assigns each score, not just what the average per variable is.
Audit: what information is available at decision time
The original model used the session's own totals (hits, pageviews, time on site, bounces and derived variables), which is reasonable for describing sessions that have already ended. But the case posed was deciding who to include or exclude from campaigns, that is, before the purchase, and those values are only known once the session ends; the purchase itself inflates them, because buying means going through checkout and generating more pages and hits. The simplest test: using pageviews alone, with no model at all, already gives an AUC of 0.978.
I retrained with the same temporal split, the same hyperparameters and the same pipeline in three variants: the original, one with only what is known at the start of the session (A), and one that adds the user's history from previous sessions (B).
ROC-AUC on test by available information
Same temporal partition, same hyperparameters. The red bar uses data that does not exist at decision time.
Each variable on its own (AUC without a model)
End-of-session variables separate the classes almost by themselves; history variables give a moderate, honest signal.
How I built the history without looking at the future
g = df.groupby("fullVisitorID")
df["prev_sessions"] = g.cumcount()
for col in ["pageviews", "hits", "timeOnSite", "y"]:
filled = df[col].fillna(0)
# running total up to the current session, minus the current session
df[f"prev_{col}_sum"] = filled.groupby(df["fullVisitorID"]).cumsum() - filled
df["days_since_prev"] = g["date"].diff().dt.days.fillna(-1)
df["prev_converted"] = (df["prev_y_sum"] > 0).astype(int)Per-user cumulative sums minus the current row: each session only sees what happened before it.
Final result (variant B)
- ROC-AUC 0.878 (bootstrap 95% CI: 0.869–0.887) and PR-AUC 0.109 on a June base rate of 1.5%.
- The 5% of sessions with the highest score captures 38.8% of real purchases: ×7.8 lift over picking at random.
- Without the history (variant A) the lift drops to ×6.1, meaning that knowing who the user is adds real signal.
- The most important variables are the day of the week, the previous hits and time history, the visit number and the days since the previous session.
Deployment
Besides the notebook I built a minimal implementation to show how it would reach production.
- 1
API with FastAPI
/health, /model-info and /predict endpoints, plus batch scoring from CSV and a minimal web dashboard served by the same API.
- 2
Container and Cloud Run
A Dockerfile ready to deploy on Cloud Run.
- 3
Features from BigQuery
An example SQL reproduces the same variables from the native GA4 export to BigQuery.
- 4
Use by marketing
The score is used to rank sessions and prioritize the top N% in campaigns; the high/medium/low segments derive from the same threshold.
What I learned
- Define the prediction moment first (session start vs. end) and only then pick variables.
- Report a metric that answers the decision (top-k lift) and not just AUC, with confidence intervals to size the uncertainty.
- Compare the result against single-variable baselines: pageviews alone already gave an AUC of 0.978.
- Next step: anticipate the purchase before the person enters the site, using only their history of previous visits.