Big Data Fundamentals

When "Big" Stopped Being an Adjective

For most of this course, we have discussed systems that handle thousands or millions of requests. Big data is what happens when the data itself becomes the bottleneck, not because the queries are complex, but because there is simply too much of it for any single machine to store, process, or analyze in a reasonable time.

The global datasphere hit roughly 64 zettabytes in 2020. By 2025, that number reached approximately 181 zettabytes. Projections for 2028 exceed 390 zettabytes. A single zettabyte is one trillion gigabytes. This growth is not linear. It is exponential, driven by IoT sensors, video streaming, mobile devices, and machine-generated logs.

This session introduces the fundamental processing paradigms and storage architectures that handle data at this scale.

OLTP vs. OLAP

Before diving into big data architectures, you need to understand the two fundamental database workload types. Every system leans toward one or the other.

Batch vs. Stream Processing

Data processing at scale splits into two paradigms based on when the data is processed relative to when it arrives.

Batch processing collects data over a period (hours, a day), then processes it all at once. MapReduce and Apache Spark are the canonical tools. You run a job that reads a full dataset, transforms it, and writes results. The job might take minutes or hours. The output is complete and consistent, but it is always behind real time by at least the batch interval.

Stream processing processes data as it arrives, record by record or in micro-batches of milliseconds. Apache Flink, Kafka Streams, and Apache Spark Structured Streaming are the primary tools. Latency is seconds or less. But handling out-of-order events, late arrivals, and exactly-once semantics is significantly more complex.

Most production systems use both. The Lambda architecture runs a batch layer for accurate historical data and a speed layer for approximate real-time results. The Kappa architecture simplifies this by using a single stream processing layer for everything, replaying the event log when historical reprocessing is needed.

Data Lake vs. Data Warehouse vs. Data Lakehouse

Where you store data at scale depends on its structure, who needs it, and how it will be queried.

A data lake is cheap storage for everything. Raw files in any format (JSON, CSV, Parquet, images, logs) dumped into object storage like S3 or Azure Data Lake Storage. Schema is applied when reading ("schema-on-read"), not when writing. This makes ingestion fast and flexible, but querying is slow without additional tooling. Data lakes tend to become "data swamps" without governance.

A data warehouse is structured storage optimized for analytical queries. Data is cleaned, transformed, and loaded into a predefined schema ("schema-on-write"). Columnar storage formats enable fast aggregations. Snowflake, BigQuery, and Redshift are the dominant players. Warehouses are fast for BI queries but expensive for raw storage and inflexible for unstructured data.

A data lakehouse combines both. It adds a transaction layer (Delta Lake, Apache Iceberg, Apache Hudi) on top of data lake storage. This gives you ACID transactions, schema enforcement, and query optimization on cheap object storage. You get warehouse-like performance without copying data into a separate warehouse system.

Kafka as the Integration Backbone

Apache Kafka sits at the center of most modern data pipelines. It is a distributed event streaming platform that acts as a durable, high-throughput buffer between data producers and consumers. Kafka does not process data. It transports it reliably and lets multiple downstream systems consume the same events independently.

A Kafka topic is an ordered, append-only log of events. Producers write to topics. Consumers read from topics at their own pace. Because the log is persistent (stored on disk with configurable retention), a new consumer can start reading from the beginning. A real-time dashboard can read events as they arrive. A batch pipeline can read the same events hours later. Neither interferes with the other.

This decoupling is why Kafka is the backbone. It separates "when data is produced" from "when and how data is consumed," which lets teams build and evolve their processing pipelines independently.

A Modern Data Pipeline

Putting these pieces together, here is what a modern data architecture looks like for a company that needs both real-time insights and historical analytics.

Data flows left to right. All sources publish to Kafka. Stream processors consume from Kafka for real-time needs. Batch processors consume the same data (or read from the lakehouse) for historical analysis. The lakehouse serves as the central storage layer, feeding both the warehouse for BI and ML pipelines for model training.

The critical insight: Kafka decouples producers from consumers. Adding a new downstream system (a fraud detection engine, a recommendation model, a compliance audit) does not require changing any upstream source. You just add a new consumer group to the relevant Kafka topic.

This second diagram shows a common pattern: Change Data Capture (CDC) using Debezium to stream database changes into Kafka, which then feeds into the analytics layer. This lets you keep your OLTP database optimized for transactions while building your OLAP layer from the same data, without any direct coupling between the two systems.

Further Reading

• What is a Data Lakehouse?, Databricks
• Data Generation Volume Worldwide 2010-2029, Statista
• Apache Kafka Documentation, apache.org
• Data Warehouse vs Data Lake vs Data Lakehouse, Monte Carlo
• Apache Flink Documentation, apache.org