Apache Kafka Explained: Architecture and Use Cases

What Apache Kafka is, and who this is for

The Kafka documentation calls Kafka an event streaming platform that does three things: publish and subscribe to streams of events, store those streams durably for as long as you configure, and process them as they occur or later. It runs as a cluster of servers (brokers, plus controllers that manage cluster metadata) and is used through client libraries and connectors. Continuous capture of sensor data from IoT devices and equipment is one of the use cases the project lists itself.

This explainer is for developers, data engineers and architects who are designing the data path for building management (BMS), CAFM and CMMS systems or other telemetry, and who need to know what Kafka adds between devices and databases. On 27 October 2026 the latest release was Apache Kafka 4.4.0, published on 22 October 2026. For how Kafka fits into a full device-to-database design, see Designing an IoT Data Pipeline with Kafka; for the overall picture, start at the time-series database overview.

Topics, partitions, keys and ordering

An event (also called a record or message) has a key, a value, a timestamp and optional headers. Events are written to a topic, a named stream such as bms.telemetry.v1. A topic can have many producers and many consumers, and reading an event does not delete it.

Each topic is split into partitions, and each partition is an append-only log stored on one broker (the leader) and copied to others. Partitions are how Kafka scales: different partitions are written and read in parallel on different brokers, and the partition count sets the maximum parallelism of a consumer group. The documentation describes a replication factor of 3 as a common production setting.

Ordering holds only inside a partition. Events with the same key are written to the same partition, and a consumer reads each partition in the order the events were written. Across partitions there is no global order. For telemetry this means choosing the key deliberately: keying by point or asset keeps each series in order; a random or empty key spreads load but gives up per-series order.

Adding partitions later is possible, but the operations guide warns that with the default hash-based partitioner the mapping of keys to partitions changes, so new events for a key can land in a different partition from its older events, and Kafka does not move existing data. Size the partition count for expected growth when you create the topic.

bin/kafka-topics.sh --bootstrap-server localhost:9092 --create \
    --topic bms.telemetry.v1 --partitions 12 --replication-factor 3 \
    --config retention.ms=1209600000 --config min.insync.replicas=2

Offsets, consumer groups and delivery guarantees

Every event in a partition has an offset, a sequence number. A consumer's position in a partition is simply the offset of the next event to read, which Kafka stores for each consumer group. Within a group, each partition is read by exactly one consumer at a time, so adding consumers (up to the partition count) spreads the work. Different groups read independently: a database sink, an alarm service and an archive job can each consume the same topic at their own pace.

Because events stay until retention removes them, a consumer can rewind. The kafka-consumer-groups.sh tool shows each group's lag (how far it is behind the end of each partition) and can reset offsets to the earliest offset, to a date and time, or by a duration. That is how you reload a day of readings into a rebuilt table after fixing a bug in a sink.

Delivery and offset commits. By default Kafka gives at-least-once delivery: after a failure, a consumer restarts from its last committed offset and can see some events again. Whether that means duplicates or losses depends on when it commits: after processing gives at least once, before processing gives at most once. Auto-commit, the client default, commits periodically in the background rather than when your processing finishes. Producers can be made idempotent, which stops producer retries from duplicating records in the log, and transactional, and Kafka Streams offers exactly-once processing between Kafka topics. None of that extends to an external database. A database sink is at-least-once with idempotent writes; it should only be described as exactly once when offsets are stored in the same transaction as the data and a fencing check rejects writes from a consumer that has lost its partitions in a rebalance. Commit modes, rebalances, retries, fencing and a failure-recovery table are in Kafka Consumer Idempotency for Sensor Streams.

Share groups. Kafka now also documents share groups, an alternative to consumer groups in which several consumers can share a partition and acknowledge records individually, closer to a traditional work queue. For ordered telemetry, classic consumer groups remain the natural fit, because a shared partition is no longer read in order by a single consumer. Their documented status by Kafka version, including dead-letter support in 4.4.0, is compared in Kafka vs RabbitMQ.

bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 \
    --describe --group tsdb-sink

bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 \
    --reset-offsets --group tsdb-sink --topic bms.telemetry.v1 \
    --to-datetime 2026-10-26T00:00:00.000 --execute

Retention, compaction and tiered storage

Retention is configured per topic with cleanup.policy:

• delete (the default) discards whole log segments once they pass retention.ms (default 604800000 ms, seven days) or retention.bytes (no size limit by default; the limit applies per partition). The documentation describes retention as an SLA on how soon consumers must read their data.
• compact keeps at least the latest value for each key. A compacted topic keyed by point ID works as a "last known value" or configuration feed, for example the current setpoint or calibration for each point, but it is not a history of readings.
• delete,compact applies both: segments past retention are removed, and the rest are compacted.

Tiered storage lets brokers keep recent segments on local disk and move completed segments to a remote store such as S3 or HDFS, with separate local and total retention settings. Kafka does not ship the remote storage implementation itself; a plugin must be configured. Even with long retention, Kafka reads by offset or timestamp position, so it is a replay source, not a historian you query.

KRaft: cluster metadata without ZooKeeper

Older Kafka clusters depended on Apache ZooKeeper to hold cluster metadata. In KRaft mode, Kafka manages metadata itself: some servers take the controller role and form a quorum using the Raft protocol, while brokers handle data requests. The upgrade notes state that Apache Kafka 4.0 only supports KRaft mode and that ZooKeeper mode has been removed; ZooKeeper-based clusters have to be migrated to KRaft first, and the last bridge release for that migration is Kafka 3.9.

• Controllers are typically 3 or 5 servers; a majority must be available, so three controllers tolerate one failure and five tolerate two.
• A server can be a broker, a controller or both ("combined"). Combined servers are simpler for development, but the documentation does not recommend combined mode in critical deployments.
• Since Kafka 4.1, controller membership can be changed dynamically (kraft.version 1), replacing the static controller.quorum.voters list with controller.quorum.bootstrap.servers.

If you are reading older tutorials that start ZooKeeper before Kafka, they describe a mode that current releases no longer support.

Kafka Connect and stream processing

Kafka Connect is Kafka's framework for moving data between Kafka and other systems without writing a consumer by hand. Source connectors bring data in (for example from an MQTT broker or a database's change log); sink connectors write topics out (for example to PostgreSQL, a time-series database, ClickHouse or object storage). Connect runs standalone or as a distributed cluster of workers, is managed through a REST API, tracks connector offsets itself, and can route records that fail conversion to a dead letter queue topic. Many connectors come from vendors and the community with their own licences, so read each connector's licence before relying on it; this is not legal advice, so take advice for your situation where the terms matter.

For processing in flight, the Kafka Streams library builds applications that read topics, filter, join, aggregate over time windows and write results to other topics, for example a one-minute average per AHU or a stream of threshold breaches. Heavier analytics still belongs in a database or warehouse; see Batch vs Stream Ingestion for IoT Data for where to draw that line.

Why Kafka is not a database

Kafka is durable storage for event streams, which leads some teams to ask whether it can be the system of record for telemetry. For most building and IoT data it should not be the only store:

• Access is sequential. Consumers read a partition from an offset forwards. There is no index by point and time range, no SQL and no aggregation; "average chiller kW per hour last month" means reading and computing over everything in range.
• Retention is a window. Data is deleted by age or size, or compacted to the latest value per key. Compaction is not downsampling.
• No relational integrity. Work orders, assets and points need constraints, updates and joins; that is a relational database's job.

Sizing, security and operations

• Sizing retention. Estimate messages per day times bytes per message times replication factor times retention days. As an illustration only: 20,000 BMS points at one reading a minute is 28.8 million messages a day; at an assumed 200 bytes per JSON message that is about 5.8 GB a day before compression, about 17 GB a day with three replicas, and roughly 120 GB for seven days. Measure your real message size and compression ratio before buying disks.
• Durability. The topic configuration reference describes a typical scenario for stronger durability: replication factor 3, min.insync.replicas=2 and producers using acks=all, so that a majority of replicas must persist a write before it counts as successful. Whether telemetry needs that depends on how much loss you can accept and on cluster size.
• Security. The documentation is explicit that security is optional and that non-secured clusters are supported, so it must be switched on: TLS for encryption, SASL (SCRAM, Kerberos or OAUTHBEARER) or TLS certificates for authentication, and ACLs to restrict which clients can read or write each topic.
• Monitoring. Watch consumer lag per group, under-replicated partitions, disk use against retention, and controller quorum health. Lag that keeps growing means the sink cannot keep up or has stopped.
• Operating cost. A production cluster means at least three brokers and a controller quorum, upgrades and capacity planning. Managed Kafka services remove some of that work; compare their pricing models on the vendors' own pages.

When not to use Kafka

• One producer, one consumer, modest volume. A gateway writing a few thousand points every minute straight into PostgreSQL or a time-series database, as in How to Store IoT Sensor Data, needs no streaming platform in between.
• Device connectivity. Sensors and controllers on constrained networks should speak MQTT or a field protocol, not the Kafka client protocol; see Kafka vs MQTT.
• Task queues. Per-message routing, priorities and acknowledgements per job suit a message broker better; see Kafka vs RabbitMQ.
• Querying history. Dashboards, reports and analytics need a database or warehouse; Kafka only feeds them.
• No one to run it. Without the skills or budget to operate a replicated cluster (or pay for a managed one), the reliability you gain may be lost to misconfiguration.

Frequently asked questions

Sources

• Apache Kafka 4.4 documentation: Introduction
• Apache Kafka 4.4 documentation: Design
• Apache Kafka 4.4 documentation: Topic Configs
• Apache Kafka 4.4 documentation: Basic Kafka Operations
• Apache Kafka 4.4 documentation: KRaft
• Apache Kafka 4.4 documentation: Upgrading
• Apache Kafka 4.4 documentation: Tiered Storage
• Apache Kafka 4.4 documentation: Kafka Connect overview
• Apache Kafka 4.4 documentation: Kafka Connect user guide
• Apache Kafka 4.4 documentation: Security overview
• Apache Kafka downloads
• Apache Kafka 4.4 documentation: Consumer Configs
• Apache Kafka 4.4 javadoc: ConsumerRebalanceListener (partitions reassigned before revocation)
• Apache Kafka 4.4 javadoc: KafkaConsumer

Checked 27 October 2026.

How we research these articles: our editorial method. Research-based from official documentation; SQL examples are run on a local database where the page says so. CodeWithSQL earns nothing from the vendors mentioned.