---
title: Deep Learning Interview Questions and Answers (2026)
description: Real deep learning interview questions — backpropagation, vanishing gradients, regularization, optimizers, and debugging a model that will not learn.
url: https://usegreenroom.app/blog/deep-learning-interview-questions
last_updated: 2026-07-31
---
← Back to blog
AI & ML
# Deep learning interview questions and answers
July 31, 2026 · 13 min read
"Your loss isn't going down," the interviewer said. "Same value, four epochs." The candidate suggested a lower learning rate. Nothing. A bigger model. Nothing. Twenty minutes in, having proposed most of a textbook, he had not once asked what the loss value actually was — which, at exactly 0.693, would have told him the model was outputting 0.5 for everything and the labels were probably all one class.
Deep learning interview questions reward debugging instincts far more than architecture trivia. Anyone can name three optimizers; the hire is the person who knows what ln(2) means when it shows up as a loss. This guide covers the concepts that get asked — backprop, gradients, regularization, optimizers — and the diagnostic reasoning that decides the round.
## What deep learning interviews actually test
Across ML engineer, research engineer, data scientist and applied scientist loops, deep learning interview questions cluster into five bands:
- Mechanics — backpropagation, the chain rule, what a gradient is and what it is with respect to.
- Training pathologies — vanishing and exploding gradients, dead neurons, why activations and initialisation matter.
- Generalisation — overfitting, regularization, batch norm, dropout, and what each actually does.
- Optimization — SGD vs momentum vs Adam, learning rate schedules, batch size effects.
- Debugging — an open-ended "the model isn't learning" question. This is the band that separates candidates.
The five bands of deep learning interview questions, with debugging deciding most senior rounds.
## Backpropagation and gradients
The opening question, usually phrased as "explain backpropagation." The answer that scores is not a formula recital — it is: a forward pass computes the loss; backprop applies the chain rule backwards through the computation graph to get the gradient of the loss with respect to every parameter; the optimizer then steps each parameter against its gradient. The key insight to state explicitly is that backprop is reverse-mode automatic differentiation, and it is efficient precisely because one backward pass gets gradients for all parameters at once.
Common follow-ups, with the answers interviewers want:
- Why not compute gradients numerically? Finite differences need two forward passes per parameter — for a model with millions of parameters that is hopeless. (Numerical gradients are still useful for checking an analytical implementation.)
- What is the gradient with respect to? The parameters, not the input — unless you are doing adversarial examples or input optimisation, in which case say so.
- Why does gradient descent work at all on a non-convex loss? Honest answer: no guarantee of a global minimum, but in high dimensions most critical points are saddle points rather than bad local minima, and the minima that are found tend to generalise acceptably. Interviewers respect the candidate who says "we don't have a complete theory here" over one who overclaims.
## Vanishing and exploding gradients
The most-asked pathology question, and it deserves a mechanism, not a definition. In a deep network, backprop multiplies many partial derivatives together. If those factors are consistently below 1, the product shrinks exponentially with depth — early layers get almost no gradient and stop learning. If consistently above 1, it explodes and you get NaN.
Then the fixes, and why each works:
- Better activations. Sigmoid saturates with a maximum derivative of 0.25, so stacking them guarantees decay. ReLU has derivative 1 on the positive side, so it does not shrink the signal — at the cost of dead neurons where the unit outputs zero forever, which is what Leaky ReLU and GELU address.
- Careful initialisation. Xavier/Glorot for symmetric activations, He initialisation for ReLU. The point is keeping the variance of activations roughly constant across layers.
- Residual connections. The single biggest one. A skip connection gives the gradient a path with derivative 1 straight back to earlier layers, which is why ResNets trained at depths that previously failed outright.
- Normalisation layers. Batch norm and layer norm keep activations in a well-conditioned range.
- Gradient clipping. Specifically for explosion, and standard in RNN and transformer training.
A strong closing line: "vanishing gradients are why depth didn't work before 2015, and residual connections are most of why it does now."
## Overfitting, regularization and what each technique really does
Expect "how do you know you're overfitting, and what do you do?" The diagnosis is the gap between training and validation loss over time — training loss falling while validation loss rises. Then the toolkit, and interviewers probe whether you understand the mechanism:
- More data / augmentation — the most effective fix and the one people mention last.
- Dropout — randomly zeroes units during training, preventing co-adaptation; roughly an ensemble over subnetworks. The follow-up trap: it must be disabled at inference, which is what model.eval() does.
- Weight decay / L2 — penalises large weights, favouring simpler functions. L1 instead drives weights to exactly zero, giving sparsity.
- Early stopping — stop when validation loss stops improving. Cheap and surprisingly strong.
- Batch normalisation — mainly an optimisation aid (it stabilises and speeds training) with a mild regularising side effect from batch noise. Calling it purely a regulariser is a common error.
The batch norm follow-up is a favourite: why does batch norm behave differently at train and test time? During training it normalises using the current batch's statistics; at inference there may be no batch, so it uses running averages accumulated during training. Forgetting eval() means your test predictions depend on whatever else is in the batch — a real bug that has shipped in real systems. Our machine learning engineer interview questions guide covers the classical-ML side.
## Optimizers, learning rate and batch size
Know the progression and what each addition buys:
- SGD — step against the gradient of a mini-batch. Noisy, and the noise itself helps escape sharp regions.
- Momentum — accumulates a velocity over past gradients, damping oscillation across ravines and accelerating along consistent directions.
- RMSProp — per-parameter scaling by a running average of squared gradients, so rarely-updated parameters still move.
- Adam — momentum plus RMSProp, with bias correction. The default for a reason, though well-tuned SGD with momentum still generalises better on some vision benchmarks, and AdamW's decoupled weight decay is the standard choice for transformers.
Learning rate is the hyperparameter interviewers care most about — "if you could tune one thing, what?" The answer is the learning rate, every time. Be ready to describe warmup (why transformers need it: early updates with a poorly conditioned model can be destructive), cosine or step decay, and how you would find a starting value with a range test.
On batch size, the expected nuance: larger batches give less gradient noise and better hardware utilisation but often generalise slightly worse, and if you scale batch size you generally scale the learning rate with it. Say the practical constraint too — batch size is frequently chosen by what fits in GPU memory, and gradient accumulation simulates a larger one.
## The debugging question: "the model isn't learning"
Back to the intro. This open-ended question appears in most mid and senior loops, and there is a right funnel:
- Read the loss value, not just its trend. A binary classifier stuck at ~0.693 is outputting 0.5 for everything (that is ln 2); a categorical model stuck at ln(num_classes) is predicting uniform. This single observation localises the bug immediately.
- Overfit a single batch. The best diagnostic in deep learning. Take 8 examples and train until loss is near zero. If it cannot, the bug is in the model, loss or optimizer wiring — not the data volume or regularization.
- Check the data pipeline. Are labels aligned with inputs? Is normalisation applied? Are augmentations destroying the signal? Print and actually look at a batch.
- Check gradients. Are they zero (disconnected graph, missing requires_grad, a detach in the wrong place)? Are they NaN (exploding, or a log of zero)?
- Check the learning rate — by orders of magnitude, not by 10%.
- Check for train/eval mode mistakes and the classic missing zero_grad().
The single-batch drill is worth being able to write, because interviewers sometimes ask for it directly:
```
# Sanity check: a working model MUST be able to memorise 8 examples.
# If loss does not approach 0, the bug is wiring — not data volume.
x, y = next(iter(train_loader))
x, y = x[:8].to(device), y[:8].to(device)
model.train()
for step in range(300):
optimizer.zero_grad()
loss = criterion(model(x), y)
loss.backward()
if step == 0:
# Are gradients even reaching the first layer?
for name, p in model.named_parameters():
g = None if p.grad is None else p.grad.abs().mean().item()
print(name, g) # None = disconnected, 0.0 = dead, nan = exploding
optimizer.step()
print("final loss:", loss.item()) # expect ~0.0 on a healthy setup
```
The gradient printout is the part that earns credit. None means the parameter is not in the graph at all; exactly 0.0 means dead units or a detached path; nan means explosion upstream. Three symptoms, three different bugs, one loop.
Candidates who lead with "I'd try a bigger model" are guessing. Candidates who say "first I'd try to overfit a single batch, because that separates a wiring bug from a learning problem" have obviously done this for a living.
## Courses, papers, Kaggle, ChatGPT — where each fits
- The Deep Learning book (Goodfellow, Bengio, Courville) — the reference for the mechanics questions; free online.
- Andrej Karpathy's Neural Networks: Zero to Hero — building backprop from scratch is the fastest way to make the gradient questions unbluffable, and it is free.
- fast.ai and the Deep Learning Specialization — good for coverage; both skew toward getting something working over diagnosing why it does not.
- Original papers — ResNet, Batch Norm, Adam, Attention Is All You Need. Interviewers ask about these specific ideas, and reading the abstract and intro of each is a few hours well spent.
- Kaggle — real practice at getting a model to converge, which is exactly the debugging skill the hard question tests.
- ChatGPT — good for explaining a concept and reviewing code. It will not stare at a stuck loss of 0.693 and wait for you to notice what that number is.
- Greenroom — the spoken layer. Ari, the AI interviewer, runs the round out loud and pushes the diagnostic follow-up when your answer is a list of techniques rather than a hypothesis. Fair tradeoff: Ari will not train your model.
> The core truth: deep learning interviews are diagnostic interviews. Naming five regularization techniques is table stakes; saying "first I'd overfit a single batch to separate a wiring bug from a learning problem" is what gets written in the feedback.
## How to prepare for a deep learning interview
- Week 1: mechanics — implement backprop by hand for a two-layer network, on paper and in code. Explain the chain rule aloud in ninety seconds.
- Week 2: pathologies and regularization. Deliberately cause vanishing gradients with deep sigmoid stacks, then fix them with ReLU and residuals, and describe what changed.
- Week 3: optimization and debugging drills. Break a working training script five ways — wrong label alignment, missing zero_grad, eval mode, absurd learning rate, detached graph — and practise diagnosing each from symptoms alone.
- Final week: one project explained end to end in first person, and two full spoken mocks with diagnostic follow-ups.
Going deeper on the stack? The PyTorch interview questions guide and TensorFlow interview questions guide cover the frameworks, the NLP interview questions guide and computer vision interview questions guide cover the domains, and the ML engineer interview questions guide covers deployment and classical ML.
## Frequently asked questions
### What are the most common deep learning interview questions?
The most common questions cover explaining backpropagation and what gradients are computed with respect to, vanishing and exploding gradients and how residual connections and activation choice address them, how you detect and fix overfitting, what dropout and batch normalisation actually do and why they behave differently at inference, the differences between SGD, momentum, RMSProp and Adam, and an open-ended question about debugging a model that will not learn.
### How do you explain backpropagation in an interview?
Say that a forward pass computes the loss, then backpropagation applies the chain rule backwards through the computation graph to obtain the gradient of the loss with respect to every parameter, and the optimizer steps each parameter against its gradient. The key point to state explicitly is that this is reverse-mode automatic differentiation, which is efficient because a single backward pass produces gradients for all parameters at once.
### What causes vanishing gradients and how do you fix them?
Backpropagation multiplies many partial derivatives together, so if those factors are consistently below one the product shrinks exponentially with depth and early layers stop learning. Fixes include using ReLU-family activations instead of saturating sigmoids, proper initialisation such as He or Xavier, normalisation layers, and above all residual connections, which give gradients a path with derivative one straight back to earlier layers.
### Why does batch normalisation behave differently during training and inference?
During training batch normalisation normalises using the statistics of the current mini-batch. At inference there may be no meaningful batch, so it uses running averages of mean and variance accumulated during training. If you forget to switch the model to evaluation mode, predictions for one example depend on whatever else happens to be in the batch, which is a real and frequently shipped bug.
### How do you debug a deep learning model that is not learning?
First read the actual loss value rather than only its trend — a binary classifier stuck near 0.693 is predicting 0.5 for everything. Then try to overfit a single small batch, which separates a wiring bug from a learning problem. After that check data and label alignment, inspect gradients for zeros or NaNs, vary the learning rate by orders of magnitude, and check for train versus eval mode mistakes and a missing gradient reset.
### Which hyperparameter matters most in deep learning?
The learning rate. It has more impact on whether a model converges at all than architecture size, batch size or optimizer choice. Be ready to discuss finding a starting value with a range test, why transformers need warmup because early updates on a poorly conditioned model can be destructive, and common schedules such as cosine annealing and step decay.
Deep learning rounds are diagnostic conversations, and diagnosis is a spoken skill. Greenroom runs mock ML interviews out loud with Ari — debugging follow-ups included — and scores structure, clarity and depth. Free to start. Curious how it works? See how AI mock interviews work.