Batch vs streaming: do you actually need real-time data?

"The dashboard must be real-time." Ask one follow-up question and the requirement usually dissolves:

PM:  the dashboard must be real-time
you: how fresh, exactly?
PM:  people check it on Mondays

Nobody needs "real-time". They need a freshness number, and most stakeholders have never been asked for one.

Two delivery models

Batch runs on a schedule: cron, Airflow, dbt, plain SQL. It is cheap because compute only exists while the job runs, and it is operable because a failed run is re-run, not paged.

Streaming processes events in seconds: Kafka, Flink, Kinesis. It is genuinely necessary for a small set of problems, and it brings distributed-systems homework: event ordering, late data, replays, exactly-once semantics, and infrastructure that never sleeps.

The freshness interview

One question settles the architecture: "what decision changes if this data is an hour old?"

• Monthly report: daily batch. Anything more is theater.
• Morning dashboard: nightly batch, done before coffee.
• "Feels live" ops screen: micro-batch every few minutes.
• Fraud, alerts, pricing: genuine streaming. The real use case.

In most companies, the honest answer to "how fresh?" is "by tomorrow morning". That is a batch job, and the budget saved is real.

The middle almost nobody names

Micro-batching, a small batch every one to five minutes, delivers minutes-fresh data on boring batch tooling. Same SQL, same testability, same re-runs. For the large class of dashboards that should feel current, users cannot tell it from streaming, and the on-call rotation certainly can.

The trap

Three months of Kafka feeding a dashboard one person opens on Monday. The pipeline costs more than the insight it carries, and every incident costs a night. Start with batch. Move to micro-batch when someone actually waits on data. Stream when seconds change money.

The takeaway

Get the freshness number before the architecture. Batch by default, micro-batch when minutes matter, streaming when seconds do. The print-ready PDF above has the interview, the cost table and the trap on one page.