---
title: Apache Spark Interview Questions & Architecture (2026)
description: Apache Spark interview questions on driver/executor architecture, RDDs vs DataFrames, lazy evaluation and shuffles — real answers, not just PySpark syntax.
url: https://usegreenroom.app/blog/spark-interview-questions
last_updated: 2026-08-20
---

← Back to blog

Data Engineering

# Apache Spark interview questions: architecture, RDDs and the shuffle

August 20, 2026 · 13 min read

![Apache Spark interview questions guide covering driver/executor architecture — cover from Greenroom, the AI mock interviewer](/assets/blog/spark-interview-questions-hero.webp)

The candidate had a working `groupBy().agg()` pipeline, a clean GitHub repo, and three months of "I use Spark daily" on their resume. Then the interviewer asked one question that wasn't about syntax at all: "When your driver calls `.collect()` on a 40GB DataFrame, what actually happens?" Silence. Then: "...it collects the data?" Correct, technically, the way "the ocean has water in it" is correct.

That's the gap this guide fills. **Apache Spark interview questions** at anything past entry level stop testing whether you can chain DataFrame methods and start testing whether you understand the machine underneath them — the driver, the executors, the cluster manager, the DAG, and the shuffle that quietly wrecks half the jobs in production. This post covers Spark's **architecture and core execution model** — RDDs, DataFrames, Datasets, lazy evaluation, partitioning and performance tuning — the depth that shows up whether you're writing Scala, Java, or PySpark. If you specifically need PySpark syntax and code-heavy examples, our [PySpark interview questions guide](/blog/pyspark-interview-questions) covers that half; this one covers what's happening underneath it.

## Spark architecture: driver, executors, cluster manager

Every Spark application has one **driver** process and many **executor** processes, coordinated by a **cluster manager**.

The **driver** runs your `main()` function, builds the logical plan from your code, and — critically — is where the `SparkContext`/`SparkSession` lives. It doesn't process data itself; it converts your transformations into a **DAG** (directed acyclic graph) of stages, splits each stage into **tasks**, and hands those tasks to executors. It also collects results back when you call an action like `.collect()` or `.count()`.

**Executors** are JVM processes that run on worker nodes. Each executor runs tasks in parallel across its allocated cores and holds data in memory (or spills to disk) for caching and shuffle operations. Executors are where the actual computation happens — the driver plans, the executors execute.

The **cluster manager** (YARN, Kubernetes, Mesos, or Spark's own Standalone manager) is responsible for allocating resources — it decides which physical machines your executors run on and enforces resource limits. It doesn't know anything about your Spark job's logic; it just hands out containers/pods on request.

**The follow-up interviewers actually ask:** "What happens if the driver dies?" The whole application dies — the driver is a single point of coordination, which is why `--deploy-mode cluster` (driver runs on the cluster, survives client disconnection) is the production default over `--deploy-mode client` (driver runs on your laptop or a gateway node, and dies if that connection drops).

## RDD vs DataFrame vs Dataset

This is the single most common **Apache Spark interview question**, and most candidates answer it as a history lesson instead of a decision framework.

**RDD** (Resilient Distributed Dataset) is Spark's original abstraction — a distributed collection of JVM objects, partitioned across the cluster, recomputable from its lineage if a partition is lost (that's the "resilient" part). RDDs give you full control but no schema and no built-in query optimization — Spark can't see inside your lambda to optimize it.

**DataFrame** is a distributed collection of rows with a known schema (column names and types), conceptually like a table. Because Spark knows the schema, the **Catalyst optimizer** can rewrite your query plan (predicate pushdown, column pruning, join reordering) before running it — this is why DataFrame code is almost always faster than equivalent RDD code, even though it looks like less work.

**Dataset** (Scala/Java only — there is no typed Dataset API in Python or R, since Python is dynamically typed) adds compile-time type safety on top of DataFrame: `Dataset[Person]` catches a typo in a field name at compile time instead of at runtime three hours into a cluster job. You lose a little of Catalyst's ability to optimize across the type boundary, which is the real tradeoff, not "Datasets are strictly better."

<div class="verdict"><strong>The core truth:</strong> the honest interview answer is a decision framework, not a ranking. Use DataFrames by default — they get you Catalyst's optimizations for free. Reach for RDDs only when you need low-level control Catalyst can't express. Use Datasets in Scala/Java when compile-time type safety is worth more to your team than the last few percent of optimizer freedom.</div>

## Transformations vs actions, and why laziness exists

Spark operations split into two categories, and the distinction is the reason Spark jobs behave the way they do.

**Transformations** (`map`, `filter`, `select`, `groupBy`, `join`) build up a plan but don't run anything. **Actions** (`collect`, `count`, `write`, `show`) trigger actual execution. Nothing computes until an action is called — this is **lazy evaluation**, and it's not an implementation detail, it's the entire optimization strategy.

Because Spark sees your *whole* chain of transformations before running any of it, it can optimize across the whole pipeline instead of executing each step eagerly: it can push a `filter` down before a `join` to shrink the data moved, or skip columns you never select. An interviewer asking "why is Spark lazy?" is testing whether you understand that laziness is what makes global optimization possible — not just a quirky API choice.

The DAG that laziness builds gets broken into **stages** at every point a **shuffle** is required, and each stage gets broken into **tasks** — one task per partition, run in parallel across executors. "Explain the difference between a job, a stage, and a task" comes up in nearly every one of these interviews: one action triggers one job; a job splits into stages at shuffle boundaries; a stage splits into one task per partition.

## Partitioning and the shuffle

If there's one concept that separates candidates who've read about Spark from candidates who've debugged it in production, it's the shuffle.

A **narrow transformation** (`map`, `filter`) computes each output partition from a single input partition — no data has to move between executors. A **wide transformation** (`groupByKey`, `join`, `distinct`, `repartition`) requires data with the same key to end up on the same partition, which means shuffling data across the network between executors — the single most expensive operation in Spark, involving disk I/O, serialization, and network transfer.

`spark.sql.shuffle.partitions` defaults to 200, regardless of your data size — a classic interview trap. Too few partitions on a huge dataset means each partition is enormous and executors run out of memory; too many partitions on a small dataset means task-scheduling overhead dominates the actual work. Tuning this number to your data size is one of the first real levers a Spark engineer reaches for.

`repartition()` triggers a full shuffle to change partition count (increase or decrease). `coalesce()` avoids a full shuffle by merging existing partitions — it can only *decrease* partition count, and it's the right choice when you're just reducing output file count after a filter, not rebalancing skewed data.

## Performance tuning: skew, broadcast joins, caching

This is where senior Spark interviews live — not "what is a DataFrame," but "the job used to take 20 minutes, now it takes 3 hours, what do you check first?"

**Data skew.** When one key holds a disproportionate share of the rows (one `customer_id` with 40% of all transactions), the task processing that partition runs far longer than every other task — the job's total time becomes the slowest task's time, not the average. The Spark UI's stage view shows this immediately: 199 tasks finish in seconds, one runs for 40 minutes. Fixes: salting the skewed key (appending a random suffix to spread it across partitions, then aggregating twice), or Spark 3's **adaptive query execution** (AQE), which can detect and split skewed partitions automatically at runtime.

**Broadcast joins.** A standard join between two large tables requires a shuffle on both sides. If one side is small enough to fit in each executor's memory, Spark can instead **broadcast** the small table whole to every executor and join locally — no shuffle at all. `spark.sql.autoBroadcastJoinThreshold` controls the size cutoff for this to happen automatically (10MB by default), and you can force it with a broadcast hint when Spark's size estimate is wrong. Knowing when a join *should* be a broadcast join, and why one isn't happening automatically, is a very common follow-up.

**Caching and persistence.** `.cache()` (an alias for `.persist(MEMORY_AND_DISK)`) stores a DataFrame in memory across actions so it isn't recomputed from scratch each time — but only pays off when you actually reuse that DataFrame multiple times. This is the exact trap from the opening scene: caching a DataFrame you only read once adds serialization overhead for zero benefit. `.persist()` with an explicit storage level (`MEMORY_ONLY`, `MEMORY_AND_DISK`, `DISK_ONLY`) lets you trade memory pressure for recomputation cost when data doesn't fully fit in RAM. Always `.unpersist()` when you're done — a full cluster's memory quietly eaten by stale cached DataFrames is a real production bug, not a theoretical one.

![A diagram of what Apache Spark interviews actually test, from architecture through performance debugging](/assets/blog/spark-interview-questions-diagram.webp)

## How this differs from a PySpark interview

Every real Spark job at a company runs on the same engine regardless of language — the driver/executor model, Catalyst, the shuffle, all identical whether you write Scala, Java, or Python. What changes with PySpark specifically is the **cost of the language boundary**: Python UDFs cross the JVM/Python process boundary for every row, which is slow enough that most PySpark interviews spend real time on "why do you avoid a Python UDF when a built-in `pyspark.sql.functions` call exists" — a question that simply doesn't apply if you're writing native Scala. If your interview loop is at a company running Scala natively (common at larger, older data platforms, and in the finance sector), expect more questions on Datasets' type safety and JVM memory tuning (`spark.executor.memory`, off-heap vs on-heap). If it's a Python-first data team, expect the PySpark-specific syntax and UDF questions instead — see our [PySpark interview questions guide](/blog/pyspark-interview-questions) for that half of the picture, worked in code.

## GeeksforGeeks lists, Databricks docs, and ChatGPT — where they fall short

Most Spark prep starts the same way: a GeeksforGeeks-style question dump, the official Databricks/Apache Spark documentation, or pasting "explain Spark architecture" into ChatGPT. All three are useful for building the mental model in this post — and none of them prepare you for the actual interview, which is verbal, live, and full of follow-ups you can't predict from a static list.

A question dump gives you the definition of a broadcast join. It doesn't ask you, out loud, with someone watching you think, "the interviewer just told you your broadcast join isn't triggering — what do you check first?" ChatGPT will happily explain the difference between `repartition` and `coalesce` in a clean paragraph you can read silently — it won't notice that you said "shuffle" three times without ever explaining what actually moves across the network, the way a real interviewer would catch and probe.

Ari, the AI interviewer behind Greenroom, runs a spoken mock interview that asks the follow-up a real Spark interviewer would ask next — not a script, an actual response to what you said. That's a different skill from recognizing the right answer on a page, and it's the skill that's actually being graded.

## Practise the architecture questions out loud

You can read this entire guide, nod at every section, and still freeze the first time someone asks you live to trace what happens between `.collect()` and a result appearing in your driver's console. Reading is recognition; interviewing is production under pressure with a stranger watching you think. [Greenroom](/) runs spoken data-engineering mock interviews, asks real follow-ups on shuffles, skew, and architecture tradeoffs, and gives feedback on how clearly you explained your reasoning — not just whether the final answer was technically right. It's a genuine gap in most prep: you can be fluent on a whiteboard and still lose ten seconds finding your words out loud.

Pair this with our guides on [PySpark interview questions](/blog/pyspark-interview-questions) for code-level practice, [Databricks interview questions](/blog/databricks-interview-questions) if the role runs on the Databricks platform specifically, and [Apache Kafka interview questions](/blog/kafka-interview-questions) if the pipeline reads from a Kafka topic before it ever reaches Spark. If the role's orchestration layer comes up too, see [Airflow interview questions](/blog/airflow-interview-questions).

## Frequently asked questions

### What are the most common Apache Spark interview questions?

The most common are explaining the driver/executor/cluster manager architecture, RDD vs DataFrame vs Dataset, transformations vs actions and why Spark is lazily evaluated, the difference between a job, a stage, and a task, narrow vs wide transformations and what triggers a shuffle, and performance-debugging scenarios involving data skew, broadcast joins, and caching.

### What is the difference between an RDD, a DataFrame, and a Dataset in Spark?

An RDD is a distributed collection of JVM objects with no schema and no automatic query optimization. A DataFrame is a distributed collection of rows with a known schema, which lets Spark's Catalyst optimizer rewrite the query plan for performance. A Dataset (available only in Scala and Java, not Python) adds compile-time type safety on top of a DataFrame. Use DataFrames by default; use RDDs for low-level control Catalyst can't express; use Datasets when compile-time type safety matters more than the optimizer's full freedom.

### Why is Spark lazily evaluated?

Transformations build up a logical plan without executing anything; only an action like `.collect()` or `.count()` triggers actual computation. Laziness lets Spark see the entire chain of transformations before running any of it, so it can optimize globally — pushing filters earlier, skipping unused columns, reordering joins — instead of executing each step eagerly and optimizing nothing.

### What causes a shuffle in Spark, and why is it expensive?

A shuffle happens whenever data with the same key needs to end up on the same partition, which wide transformations like `groupByKey`, `join`, `distinct`, and `repartition` require. It's expensive because it involves writing data to disk, serializing it, sending it across the network to other executors, and deserializing it again — by far the most costly kind of operation in a Spark job, and the first thing to look for when a job is slower than expected.

### How do you fix data skew in Spark?

First confirm it in the Spark UI's stage view — one task running far longer than the rest while most finish quickly is the signature of skew, usually caused by one key holding a disproportionate share of rows. Fixes include salting the skewed key (adding a random suffix to spread it across more partitions, then aggregating in two passes) or enabling Spark 3's adaptive query execution (AQE), which can detect and split skewed partitions automatically at runtime.

### How is a Spark interview different from a PySpark interview?

Architecture, the shuffle, Catalyst, and performance tuning are identical regardless of language — that's what this guide covers. PySpark-specific interviews add questions about the Python/JVM language boundary, particularly why Python UDFs are slow compared to built-in `pyspark.sql.functions` calls, since every UDF row crosses that boundary. Scala-native interviews instead lean on Dataset type safety and JVM memory tuning. Expect the mix that matches the team's actual language.

Apache Spark interviews reward people who can explain what's actually moving across the network, out loud, under a follow-up question. Greenroom runs spoken data-engineering mock interviews with real follow-ups and feedback on every answer. Free to start.