The Nightly Batch That Won't Fit
A subscription streaming service scores all 41 million subscribers nightly to decide who receives a retention offer. The model is a Driverless AI experiment exported as a MOJO. The job starts an H2O-3 cluster, imports a 22 GB parquet extract, loads the model with h2o.import_mojo(), and calls h2o.predict(). It takes 6 h 40 min on a 4-node cluster with 64 GB per node. The downstream campaign build starts at 05:00, and the extract only lands at 23:30. Separately, product wants a real-time endpoint for 150 requests per second at p99 under 80 ms for in-session offers. Over five months the nightly model's AUC, measured against 30-day-later labels, has fallen from 0.83 to 0.74. The feature plan_type has a PSI of 0.31 because two new plans launched. Retraining on only the last 60 days fixes the new plans but degrades the legacy base by 0.04 AUC.
1. How would you restructure scoring to meet both the batch window and the real-time requirement? Discuss MOJO versus POJO and where scoring should run.
Read the worked answer
Treat these as two different engineering problems that happen to share a model artefact.
Nightly batch. The 6 h 40 min is not model arithmetic; MOJO scoring is row-independent and roughly constant-time per row. The cost is moving 22 GB through a JVM cluster, serialising frames, and cluster overhead/GC. You do not need a distributed cluster for independent row scoring. Stream the parquet partition by partition through the MOJO in a plain process (or as a Spark UDF with the MOJO loaded per executor), so memory stays bounded and you parallelise across cores. Benchmark rows/second/thread first, then size the worker pool against the 5.5-hour window with headroom; the compute for 41M rows is minutes, the rest is I/O - so the real win is avoiding a full materialisation of the frame.
Real-time. Never call an H2O cluster per request: the REST round-trip plus cluster GC will blow any p99, let alone 80 ms. Embed the MOJO in the service process via h2o-genmodel.jar and EasyPredictModelWrapper, and move slow or aggregate features into a feature store so the request path performs only key lookups plus fast transforms. Precompute and cache anything that depends on history.
MOJO vs POJO. MOJO is the right artefact: it covers the DAI ensemble (XGBoost plus neural net) and the broader algorithm set, whereas POJOs support a narrower set and are tied to the H2O version that generated them. Pin the MOJO hash and the matching genmodel jar version together in the deployment.
2. Design the drift monitoring and retraining strategy given the `plan_type` shift and the legacy-segment regression.
Read the worked answer
Monitor three layers, because each fails differently:
- Inputs. PSI or KS per feature against the training distribution on a rolling window.
plan_typeat 0.31 is far past the usual 0.1 (minor) / 0.25 (major) bands and is a genuine population change, not noise. Also alert on null rates and join-key coverage - a broken upstream pipeline looks exactly like drift. - Scores. Mean score, decile shares, and the share crossing the offer threshold. These move before labels arrive and give you an early warning.
- Outcomes. Once 30-day labels land, per-segment AUC, calibration, and offer uplift. Slice by
plan_type, tenure and region - the aggregate 0.83 → 0.74 can hide a stable legacy segment and a collapsed new-plan segment, or vice versa.
Retraining: the 60-day experiment is the key finding - it is a segment trade-off, not an improvement. Fixes, in order: add the new plans to the feature pipeline and an explicit new-plan indicator; retrain on a pooled window (12–18 months) with recency weighting or a per-plan interaction so the model can learn plan-specific behaviour while retaining legacy history; then gate acceptance on per-segment out-of-time performance with a pre-registered threshold, rather than on aggregate AUC. If a single global model cannot hold both segments, consider a segmented model with a router, and document the choice.
3. What is your rollout and rollback plan, and how do you version models and scores?
Read the worked answer
Rollout. Shadow-score the candidate alongside production for one to two weeks without acting on it, comparing score distributions and segment-level metrics. Then a champion/challenger split (e.g. 5% of eligible subscribers on the challenger) with a pre-registered primary metric (incremental saves per thousand offers) and guardrails (offer cost, unsubscribe rate, complaint rate). Do not let the challenger take 100% on the strength of offline AUC alone - offline gains frequently fail to convert to incremental retention when the offer itself changes behaviour.
Rollback. Keep the previous MOJO deployable and the previous night's score table intact. If a guardrail trips, flip traffic back and re-score the affected cohort with the champion. Rollback must be a configuration change, not a rebuild - if reverting requires retraining, it is not a rollback plan.
Versioning. Record on every scored row: model ID and MOJO hash, training-data snapshot identifier, feature-pipeline version, and scoring timestamp. Version the genmodel jar alongside the MOJO in the same artefact repository, and tag the DAI experiment and its settings. Without the model version on the score row, the drift analysis itself becomes unreliable, because you cannot attribute a performance change to a segment shift versus a model swap. Keep the label-join job keyed on the same identifiers so that outcome monitoring, retraining data selection and audit all resolve to the same lineage.