--- title: Observability Interview Questions and Answers (2026) description: Observability interview questions and answers — metrics, logs and traces, cardinality, SLOs and error budgets, alerting philosophy and debugging an unknown failure. url: https://usegreenroom.app/blog/observability-interview-questions last_updated: 2026-08-01 --- ← Back to blog Technical # Observability interview questions and answers August 1, 2026 · 12 min read ![Observability interview questions and answers — guide cover from Greenroom, the AI mock interviewer](/assets/blog/observability-interview-questions-hero.webp) Every dashboard was green. CPU normal, memory normal, error rate 0.02%, all services healthy. Meanwhile a specific set of users — on one mobile app version, in one region, hitting one endpoint — were failing to check out entirely, and had been for six hours. The dashboards were not wrong. They were just answering questions somebody had thought to ask in advance, and this was not one of them. That distinction is the whole subject of **observability interview questions**. Monitoring tells you whether the things you predicted are happening. Observability is whether you can answer a question you did not anticipate, from the data the system already emits. Interviewers probe the difference constantly, and the strongest candidates are the ones who can debug an unknown failure out loud. ## Monitoring versus observability The definition that scores: **monitoring answers known questions with predefined dashboards and alerts; observability is the property of a system that lets you ask new questions of it without shipping new code.** Then make it concrete: "our p99 is high" is monitoring. "Our p99 is high specifically for authenticated users on app version 4.2 in the Mumbai region calling the checkout endpoint" is observability, and it requires high-cardinality, wide events rather than pre-aggregated counters. Say the phrase "unknown unknowns" — it is the standard framing and it signals you have read the field. ![Diagram of observability interview topics — the three signals, cardinality and cost, SLOs and error budgets, alerting philosophy, distributed tracing and debugging an unknown failure](/assets/blog/observability-interview-questions-diagram.webp) Six areas. The differentiator is the last one: can you debug a failure nobody predicted, using only what the system already emits? ## The three signals - **Metrics.** Numeric time series, cheap to store, fast to query, excellent for trends and alerting. Their weakness is that they are aggregated, so you cannot drill into a specific request after the fact. - **Logs.** High detail per event, expensive at volume, and only useful at scale if they are structured. Say "structured logging" — a JSON line with consistent fields is queryable, a formatted English sentence is not. - **Traces.** A request's path across services with timing per span. The only signal that tells you *which hop* was slow in a distributed call, which is why they matter more as architectures fragment. The mature addition: these are converging into **wide structured events** — one rich record per unit of work, from which metrics and traces can be derived. Mentioning that trend, and OpenTelemetry as the vendor-neutral instrumentation standard, positions you well. ## Cardinality, the question that finds people out **"Why did our observability bill triple?"** Almost always cardinality. Cardinality is the number of unique combinations of label values on a metric. Metrics systems store one time series per unique combination, so adding a label with many possible values multiplies your storage. Adding `user_id` as a label to a metric is the classic catastrophe: a million users means a million time series per metric. The correct framing: **high cardinality belongs in logs and traces, not in metrics labels.** Keep metric labels low-cardinality — service, endpoint, status class, region — and put the user ID, request ID and trace ID in the structured event where it can be queried without creating a time series. Being able to explain this cleanly is one of the highest-signal answers in the whole round. Our [Prometheus interview questions](/blog/prometheus-interview-questions) guide covers the mechanics. ## SLIs, SLOs and error budgets - **SLI** — a service level indicator, the actual measurement: the proportion of requests served successfully under 300ms. - **SLO** — the target for that indicator: 99.9% over a rolling 28 days. - **Error budget** — what the SLO permits you to fail. At 99.9% you have roughly 43 minutes of failure a month, and that budget is a decision-making tool: budget remaining means ship faster, budget exhausted means stop feature work and fix reliability. - **SLA** — the contractual version, with financial consequences, and always looser than your internal SLO. The point interviewers listen for: **measure what users experience, not what servers report.** A service with 100% uptime that returns errors to a third of requests is not meeting its SLO, and a CPU graph would never have told you. Also be ready for "how do you choose an SLO?" The honest answer is from user impact and business tolerance, not by adding a nine to the current number — and that 100% is never the right target because it eliminates the budget you need to ship anything. ## Alerting philosophy - **Alert on symptoms, not causes.** Page on elevated error rate or latency that users feel; do not page on CPU at 90%, which may be perfectly fine. - **Every page must be actionable and urgent.** If the response is to acknowledge it and look in the morning, it should not have paged. - **Alert fatigue is a real failure mode.** A team that gets forty pages a night stops reading them, and the one that mattered gets missed. Track pages per on-call shift as a metric. - **The four golden signals** — latency, traffic, errors, saturation — as a starting framework, plus RED (rate, errors, duration) for services and USE (utilisation, saturation, errors) for resources. - **Multi-window burn-rate alerts.** Alert when the error budget is being consumed fast enough to matter, over both a short and a long window, so you catch both a sudden outage and a slow bleed without paging on noise. Our [SRE interview questions](/blog/site-reliability-engineer-interview-questions) guide covers the reliability practice around this. ## Distributed tracing - **Context propagation.** A trace ID created at the edge and passed through every hop, including across queues — which is the part people forget and the reason async paths show up as disconnected traces. - **Sampling.** Head-based sampling decides at the start and is cheap but may discard the interesting requests; tail-based sampling decides after the fact and can keep all errors and slow requests, at the cost of buffering. Knowing this distinction is a strong signal. - **Spans and attributes.** What belongs on a span, and why adding request-specific attributes there is correct while adding them to metric labels is not. - **What tracing tells you that logs cannot** — the time distribution across a request path, which identifies the slow hop rather than merely confirming slowness. ## The debugging question Almost every loop has one: "users report checkout is slow, dashboards look fine, what do you do?" Answer as a live investigation, narrating your reasoning: - Confirm and scope it — which users, which region, which client version, which endpoint, and since when. Narrowing before hypothesising is what separates strong answers. - Compare against a known-good baseline rather than an absolute threshold. - Check what changed: deploys, config, feature flags, upstream dependencies, traffic mix. - Move from metrics to traces to find the slow span, then to logs for that specific trace ID. - Look at percentiles rather than averages, because an average hides a bimodal distribution completely. - Say what you would add so the next occurrence is diagnosable in five minutes rather than six hours. That last step is what the interviewer is really waiting for. **The core truth:** observability rounds are graded on whether you can answer a question nobody prepared for. Every strong answer moves from "here is the dashboard" to "here is how I would narrow it down, and here is what I would instrument so it is obvious next time." ## Where each prep option actually helps - **The Google SRE books** — free online, and the source of golden signals, SLOs and error budgets as interviewers use them. - **Observability Engineering by Charity Majors and colleagues** — the clearest articulation of the wide-events and unknown-unknowns framing. - **The OpenTelemetry documentation** — the vendor-neutral standard, and worth knowing by name and shape. - **Instrument something yourself** — add tracing to a two-service app and find a slow span. It answers the debugging question for you. - **ChatGPT** — good for explaining PromQL or a sampling strategy. It will not narrow down your outage, which is the actual skill being tested. - **Greenroom** — the spoken layer. [Ari, the AI interviewer](/) runs the debugging scenario out loud, which is the round most people have never rehearsed aloud. Honest tradeoff: Ari will not read your dashboards, so build and break something too. ## Frequently asked questions ### What is the difference between monitoring and observability? Monitoring answers questions you predicted in advance using predefined dashboards and alerts, while observability is the property of a system that lets you ask new questions of it without shipping new code. Concretely, knowing that p99 latency is high is monitoring, whereas being able to determine that it is high specifically for authenticated users on one app version in one region calling one endpoint is observability — and that requires high-cardinality wide events rather than pre-aggregated counters. ### What are the three pillars of observability? Metrics, logs and traces. Metrics are cheap numeric time series that are excellent for trends and alerting but aggregated, so you cannot drill into a single request afterwards. Logs carry high detail per event, are expensive at volume, and are only queryable at scale if they are structured. Traces show a request's path and timing across services, which is the only way to identify which hop was slow. The modern framing is that these converge into wide structured events, instrumented via OpenTelemetry. ### What is cardinality and why does it matter in observability? Cardinality is the number of unique combinations of label values on a metric, and metrics systems store one time series per combination — so adding a high-cardinality label such as user ID can create millions of series and is the usual reason an observability bill explodes. The rule is that high-cardinality data belongs in logs and traces, not in metric labels: keep labels to low-cardinality dimensions such as service, endpoint, status class and region. ### What is the difference between an SLI, an SLO and an error budget? An SLI is the actual measurement, such as the proportion of requests served successfully under 300 milliseconds. An SLO is the target for that indicator, for example 99.9% over a rolling 28 days. The error budget is what the SLO permits you to fail — roughly 43 minutes a month at 99.9% — and it functions as a decision tool, where remaining budget means you can ship faster and an exhausted budget means feature work stops until reliability improves. An SLA is the looser contractual version with financial consequences. ### What should you alert on? Symptoms that users feel, such as elevated error rates or latency, rather than causes such as CPU at 90% which may be entirely fine. Every page must be both actionable and urgent — if the correct response is to look at it in the morning, it should not have paged. Useful frameworks are the four golden signals of latency, traffic, errors and saturation, plus multi-window burn-rate alerts that catch both a sudden outage and a slow error-budget bleed without generating noise. ### How do you debug an issue when all the dashboards look fine? Scope it before hypothesising: which users, which region, which client version, which endpoint and since when. Compare against a known-good baseline rather than an absolute threshold, check what changed across deploys, config, feature flags and upstream dependencies, then move from metrics to traces to find the slow span and from there to logs filtered by that trace ID. Look at percentiles rather than averages, since an average hides a bimodal distribution entirely, and finish by saying what you would instrument so the next occurrence is obvious in minutes. The observability round is a live investigation, and investigations are spoken. [Greenroom](https://usegreenroom.app/) runs the debugging scenario out loud with Ari, who keeps asking how you would narrow it down. Free to start. Curious how it works? See [how AI mock interviews work](/blog/ai-mock-interview).