Spark real-time mode vs Flink: sub-second streaming in Spark, and what it still cannot do

Spark real-time mode is a trigger that makes a Structured Streaming query handle each record as it arrives, so a Kafka-to-Kafka job answers in milliseconds instead of the second or more a micro-batch takes. You switch it on with one line, .trigger(realTime="5 minutes"), and keep the rest of your Spark code. The cost is a cluster that holds every stage's tasks at once, and a narrower set of sources, sinks and operators. In particular, it does not read or write Delta tables.

It answers the main reason teams ran Apache Flink beside Spark: latency. It does not answer every reason, and the gaps are specific enough to check against your job.

What changes inside Spark

A micro-batch query wakes up, reads what has arrived, runs its stages one after another, commits and sleeps. The wait for the next batch and the stage-by-stage plan put a floor under latency.

Real-time mode keeps the batch but makes it long: 5 minutes by default. Inside it, 3 things change:

1. Records flow through as they arrive. The source does not wait for the batch to fill.
2. Every stage is scheduled at once. The reading tasks and the Spark ShuffleSpark ShuffleThe step that moves rows between executors so rows with the same key end up together, for a join, group by or repartition; it splits a job into stages. tasks run side by side for the whole batch.
3. Stages pass data straight on. A streaming shuffle hands rows to the next stage as soon as they are produced, instead of after the previous stage finishes.

The interval you give the trigger is not a latency setting. It is how long each long batch lasts, and so how often Spark checkpoints. A longer interval means fewer checkpoints but a longer replay after a failure. A shorter one checkpoints more often, which can cost latency.

How fast is it? The Apache Spark 4.1 release notes say a stateless query can get down to single-digit milliseconds. Databricks describes sub-100 ms processing, and a bank customer flagging card transactions from Kafka within 200 ms. Stateful logic and Python functions add time. The only number that counts is the one you measure.

The price: slots held for the whole run

Because every stage runs at once, the cluster must have a task slot for every task in every stage. Databricks' own examples:

• 8 slots: a stateless Kafka-to-Kafka query reading 8 partitions.
• 28 slots: the same read followed by a shuffle into 20 partitions (8 + 20).
• 48 slots: two shuffles of 20 each (8 + 20 + 20).

With fewer slots the query is not scheduled at all. Size yours:

Try itHow many task slots does a real-time query hold?

Kafka partitions (maxPartitions)

8

116256

Shuffle stages

None (stateless)1 shuffle2 shuffles

Shuffle partitions

20

120200

Cores per worker

4 cores8 cores16 cores

Kafka read8

Shuffle 120

Those slots are held even when no data is coming in. Databricks' tuning advice follows from that: aim for a utilisation target such as 50%, let one task read several Kafka partitions with maxPartitions, and keep shuffle partitions at the smallest number that does not fall behind. The usual instinct of a few hundred shuffle partitions, fine for a batch job, is expensive here. The shuffle partition calculator shows what a given number does in micro-batch mode, where the trade-off is different.

Turning it on

On Databricks, real-time mode for a direct Structured Streaming query runs on classic compute. Serverless notebooks and jobs do not offer it, though Lakeflow pipelines have their own real-time setting. The cluster needs:

• Databricks Runtime 16.4 LTS or above (Databricks recommends 18.1).
• Autoscaling, Photon and spot instances turned off.
• spark.databricks.streaming.realTimeMode.enabled set to true.

Then the query itself, in update mode, the only output mode it accepts:

query = (
    spark.readStream
    .format("kafka")
    .option("kafka.bootstrap.servers", broker_address)
    .option("subscribe", "payments")
    .option("maxPartitions", 8)
    .load()
    .selectExpr("CAST(key AS STRING) AS key", "CAST(value AS STRING) AS value")
    .writeStream
    .format("kafka")
    .option("kafka.bootstrap.servers", broker_address)
    .option("topic", "payments-scored")
    .option("checkpointLocation", checkpoint_location)
    .outputMode("update")
    .trigger(realTime="5 minutes")
    .start()
)

In Scala the trigger is RealTimeTrigger.apply(), which can also take the interval. From Runtime 16.4, checkpoints use a format that lets the same query switch between real-time and micro-batch mode, so you can try it on an existing stream.

Open-source Spark has it too, with less coverage. Spark 4.1 added real-time mode for stateless Scala queries; Spark 4.2 brought the trigger to PySpark, for stateless queries without Python UDFs (what else is in Spark 4.2).

What it runs, and what it does not

This is the part to read before you plan a migration. On Databricks today:

2 rows decide most designs. First, Delta is neither a source nor a sink. Real-time mode is for decisions on the message bus: read a Kafka topic, score or filter, write a Kafka topic. If the stream's job is to land rows in a lakehouse table, a micro-batch query (or Zerobus Ingest, which writes straight to Delta) is the tool. Second, foreachBatch is gone, and with it the usual way to MERGE each batch into a table (Idempotent Pipelines and Exactly-OnceIdempotent Pipelines and Exactly-OnceA pipeline step is idempotent when running it twice gives the same result as running it once. That is what makes retries safe, and it is how systems that deliver a message more than once still count it exactly once. patterns built on it need rewriting).

Most Python UDFs work but need care; Python UDTFs, Unity Catalog UDFs and grouped functions such as applyInPandas do not. Databricks recommends an Arrow batch size of 1 (spark.sql.execution.arrow.maxRecordsPerBatch) for latency. Pandas UDFs perform poorly at that size, so they need a larger batch and give back some of the gain.

Measuring latency

The usual batch duration metric means little when a batch lasts 5 minutes. Real-time mode reports 3 numbers per batch instead:

• sourceQueuingLatencyMs: from the record landing in Kafka to the query reading it.
• processingLatencyMs: from reading it to writing it on.
• e2eLatencyMs: the two together, from the bus to the sink.

If queuing latency is the large one, the query is short of slots or partitions. If processing latency is, look at UDFs and state. Two options that cut time between batches, asynchronous progress tracking and asynchronous state checkpointing, are off until you turn them on.

Real-time mode or Flink

Databricks' benchmark puts real-time mode up to 92% faster than Apache Flink on queries modelled on customer feature computation. It is the vendor's own test, with the queries published. The stronger argument is that you run one engine: the same DataFrame code for the batch backfill, the model training data and the live path.

Flink still fits better when the job needs what real-time mode leaves out:

• Event-time timers. For example, "alert if no heartbeat arrives within 30 seconds of event time". Real-time mode's timers fire only when data arrives or the batch ends.
• Session windows, or outer joins between two streams.
• Many small independent pipelines. Each real-time query holds its slots for its lifetime, while Flink by default lets one slot hold a whole pipeline of a job's operators.
• Exactly-once state with Flink's checkpoint and savepoint model, which a team already runs and trusts.

And plain micro-batches are still right for most streams. That covers ETL into Delta, anything where a few seconds is fine, and jobs where cost and fast recovery matter more than milliseconds. Databricks says the same: real-time mode needs dedicated compute, and longer batches mean longer replays.

Where it fits in a pipeline

A common shape is two queries on the same Kafka topic. A real-time query scores each payment and writes the decision to an output topic within milliseconds. A micro-batch query lands the same events in a Delta table every minute for reporting and model training. Sizing the topic for both readers is covered in how many Kafka partitions you need and the Kafka sizing calculator.