The dashboard was green. Latency fine, error rate zero, uptime four nines. The model had also been quietly degrading for eleven weeks, because an upstream team had changed a field from "IN" to "India" and one feature had been silently null ever since. Nothing broke. Nothing paged. Revenue just got a bit worse every week, and eventually someone in finance asked a question that engineering could not answer.
That story is the whole reason MLOps interview questions exist as a category. A broken service throws errors; a broken model returns confident predictions that are wrong. Everything in this discipline — feature stores, drift monitoring, registries, shadow deployments — is an answer to that single asymmetry. This guide covers the questions that recur, in roughly the order interviewers ask them.
The lifecycle you should be able to draw
Be able to sketch this in ninety seconds and talk through it:
- Data and versioning. A dataset you can reproduce, with lineage from source to training snapshot. "We queried the warehouse in July" is not reproducible.
- Feature pipeline. One feature definition consumed by both training and serving. This is the single most important design decision in the whole loop.
- Training pipeline. Parameterised and schedulable, logging every run with its data version, code version, hyperparameters and metrics.
- Model registry. Versioned artifacts with stage transitions — staging, production, archived — and a human or automated approval gate.
- Deployment. Shadow first, then canary, with a fallback that always works.
- Monitoring and retraining. Drift detection on inputs and outputs, and a retraining trigger based on a signal rather than only a calendar.
Training-serving skew, the question that recurs most
"What is training-serving skew?" The feature value computed during training differs from the value computed at serving time — different code paths, different data freshness, or a transformation fitted on the full dataset before splitting.
"How do you prevent it?" The expected answer is a shared feature definition rather than good intentions. A feature store gives you one definition materialised into both an offline store for training and an online store for low-latency serving. If you do not have one, the same transformation code must be packaged and used by both paths.
"What if the feature needs recent data at serving time?" Now you are talking about point-in-time correctness: when generating training data you must use the feature value as it was at the time of the label event, not as it is now. Getting this wrong is the most common source of leakage in a production ML system, and being able to explain it precisely is a genuine senior signal.
Monitoring: what you actually watch
- Data drift. The input distribution moves. Measure with population stability index, KL divergence, or a Kolmogorov-Smirnov test per feature, and alert on a threshold rather than eyeballing a chart.
- Prediction drift. The output distribution moves. Cheap to compute and often the earliest signal you get.
- Concept drift. The relationship between inputs and outputs changes — the world moved, not the data. Only visible once labels arrive.
- Feature health. Null rates, cardinality changes, range violations. The eleven-week bug in the opening was a null-rate alert nobody had.
- Delayed labels. Many systems only learn the truth days or weeks later, so your real performance metric always lags. Say how you bridge the gap with proxy metrics in the meantime.
- Business metric. The one that actually matters, and the one drift alerts exist to protect.
The sentence that lands: "I alert on inputs, because output metrics arrive too late to prevent the damage."
Deployment strategies for models specifically
- Shadow deployment. The new model receives real traffic and its predictions are logged but not served. The only safe way to compare on production data, and it costs you inference twice.
- Canary. A fraction of traffic goes to the new model, with automated rollback on a metric breach.
- A/B test. Statistically valid comparison of business outcomes, which is a different thing from a canary and takes longer. Be able to say why.
- Champion-challenger. The challenger runs continuously in shadow and gets promoted when it wins over a defined window.
- Fallback. What serves when the model is unavailable or returns nonsense — a simpler model, a cached prediction, or a heuristic. Naming this unprompted is a strong signal.
Our CI/CD interview questions guide covers the deployment mechanics these sit on top of.
Reproducibility and the registry
"How do you reproduce a prediction from six months ago?" This is a favourite, usually framed as an incident: a customer disputes a decision and you must explain it. The answer needs four things versioned together — the code, the model artifact, the feature values as they were at that moment, and the environment. If any one is missing you cannot reproduce it, and in regulated contexts that is a compliance problem as well as an engineering one.
"What goes in a model registry?" The artifact, its version, the training run that produced it, the dataset version, the evaluation metrics, the approval status, and who approved it. Tools such as MLflow do this; the concept matters more than the tool in an interview.
Pipelines and orchestration
- Batch versus streaming features, and the consistency guarantee each gives you.
- Idempotent pipeline steps, because every orchestrator retries and a non-idempotent training step will double-count.
- Backfills. How you recompute a feature's history without breaking the online store.
- Orchestrator choice — Airflow, Dagster, Kubeflow, Prefect. Have an opinion but do not be dogmatic; interviewers are listening for whether you understand DAGs, retries, backfills and dependencies. Our Airflow interview questions guide covers the most commonly asked one.
- Cost. GPU hours for training, per-request inference cost at production QPS, and the storage cost of keeping every feature snapshot. Cost awareness is unusually well received in MLOps interviews.
The scenario questions
Expect at least one of these, and answer as an incident walkthrough rather than a lecture:
- "Your model's accuracy dropped 5% overnight. What do you do?" Check for a data pipeline failure or schema change first — sudden drops are almost always upstream, not the model. Then feature null rates, then recent deploys of anything, then drift, then labels. Roll back before you finish diagnosing.
- "Training works, production is worse. Why?" Training-serving skew, leakage in training, distribution shift, or a preprocessing mismatch. Name all four and say how you would distinguish them.
- "How often should you retrain?" Not a fixed answer. Retrain on a trigger — drift breaching a threshold or performance falling below a floor — with a scheduled fallback. Saying "monthly" without qualification is the weak answer.
- "How would you roll back a model?" Registry stage transition plus a traffic switch, with the previous version still warm. If your answer requires a retrain, you do not have a rollback.
The behavioral question inside the technical round
You will be asked about a model that failed in production. Have a real one: what it was, how you found out and how long that took, what the impact was, what you did first, and what you changed structurally so that class of failure would be caught next time.
"We monitored it more closely afterwards" is not a change. "We added a null-rate alert on every feature and a schema contract test on the upstream table" is. Our behavioral interview questions guide covers the shape.
Where each prep option actually helps
- Designing Machine Learning Systems by Chip Huyen — the single best preparation for this interview, especially on skew and monitoring.
- The Google Rules of Machine Learning document — free, short, and unusually direct about doing the simple thing first.
- Building one end-to-end pipeline yourself — even a small one, with a registry, a scheduled retrain and a drift check, is worth more than any amount of reading.
- MLflow, Feast and Evidently documentation — enough to speak concretely about registries, feature stores and drift reports.
- ChatGPT — good for generating incident scenarios to practise against. It will not notice that your answer never explained how you would find out.
- Greenroom — the spoken layer. Ari, the AI interviewer runs these scenario questions out loud and pushes on the detection step. Honest tradeoff: Ari will not review your DAGs, so build one for real.
Our machine learning engineer interview questions guide covers the modelling half and LLM production engineer interview questions covers the generative-AI variant of this same round.
Frequently asked questions
What are the most common MLOps interview questions?
They cluster around the production half of the lifecycle: what training-serving skew is and how you prevent it, how you detect data and concept drift, how you deploy a model safely using shadow and canary rollouts, what goes into a model registry, how you reproduce a prediction from six months ago, how often you should retrain, and a scenario where accuracy drops overnight and you have to diagnose it live.
How do you prevent training-serving skew?
Use one feature definition consumed by both paths rather than relying on discipline. A feature store materialises the same definition into an offline store for training and an online store for serving; without one, the identical transformation code must be packaged and used by both. You also need point-in-time correctness when building training data, meaning each feature is taken as it was at the moment of the label event rather than as it is now.
How do you monitor a machine learning model in production?
Alert on inputs rather than waiting for output metrics, because labels usually arrive too late to prevent damage. Track feature health such as null rates, cardinality changes and range violations; data drift using population stability index, KL divergence or a KS test per feature; prediction drift on the output distribution as an early cheap signal; and concept drift once labels land. Tie all of it back to the business metric the alerts exist to protect.
How often should you retrain a model?
Retrain on a trigger rather than only a calendar. The trigger is usually drift breaching a defined threshold or a performance metric falling below a floor, with a scheduled retrain as a fallback so the model never becomes arbitrarily stale. Answering with a fixed interval such as monthly, with no qualification, is the weak version of this answer.
What do you do if model accuracy drops suddenly in production?
Treat it as an upstream incident first, because sudden drops are almost always a data pipeline failure or a schema change rather than the model itself. Check feature null rates and schema, then recent deploys of any component, then drift, then label quality — and roll back to the previous model version before you finish diagnosing, since a registry stage transition plus a traffic switch is far faster than a fix.
How do you reproduce a prediction from six months ago?
You need four things versioned together: the code, the model artifact, the feature values as they were at that moment, and the environment. If any one is missing the prediction cannot be reconstructed, which is an engineering problem during an incident and a compliance problem in regulated contexts. This is why data versioning, a model registry and point-in-time feature retrieval exist as separate concerns.