Asynchronous Events With Kafka
Kafka is a distributed event log: producers append records to topics, and consumers read those records at their own pace. The key reliability property comes from persistence on the broker side plus consumer offset tracking, which lets systems recover after outages without replaying everything from scratch.
A practical example: an order service publishes an OrderPlaced event. A billing service consumes it and writes invoices, while a shipping service consumes the same event and creates shipment tasks. If billing is down for 30 minutes, it can resume by reading from the last committed offset, assuming offsets and retention settings match the recovery window.
Kafka’s resilience depends on configuration choices such as replication factor, topic retention, and consumer offset commit strategy. Those choices interact with failure modes like broker loss, consumer crashes, and slow downstream dependencies, which is where many teams stumble.
Common Pain Points And Misreads
Teams often treat Kafka as a “message queue replacement” and then design around queue-like assumptions. Kafka topics are append-only logs, so the system’s behavior depends on how long data remains available (retention) and how consumers track progress (offsets).
Another frequent misread involves delivery semantics. Kafka does not magically prevent duplicates; it provides at-least-once delivery patterns when consumers retry or when failures occur after processing but before offset commits. If downstream systems are not idempotent, duplicates become data corruption, not just extra work.
Supporting technologies also matter. Schema evolution usually relies on a schema registry pattern (for example, Confluent Schema Registry or an equivalent), and event contracts need versioning rules. Without a disciplined schema strategy, consumers break during upgrades, and “resilience” turns into constant hotfixing.
Operational dependencies show up in monitoring and storage. If you set retention to 1 hour but your consumer can be paused for 6 hours during maintenance, recovery becomes impossible for that consumer. I once saw a team set retention to match a perceived “latency need” and later discovered their batch consumer ran nightly with occasional delays; the mismatch created gaps that looked like data loss.
Solutions And Advice For Resilience
Design Events For Replays
Model events so they can be replayed safely. Use stable event identifiers (for example, an eventId derived from the source transaction) and make downstream writes idempotent by storing processed IDs or using unique constraints. For relational sinks, a unique index on (eventId, eventType) often turns duplicates into no-ops.
Choose a clear contract for schema evolution. A common approach is backward-compatible changes: add optional fields, avoid renaming, and keep semantics consistent across versions. If you use Avro with a schema registry, enforce compatibility rules at registration time; a version like Avro schema v3 that breaks compatibility should fail fast rather than corrupt consumers.
Plan for ordering only within partitions. If you need ordering by customer, partition by customerId so all events for that key land in the same partition. Ordering across keys requires additional logic, and teams sometimes assume Kafka preserves global order.
Pick Offsets And Semantics
Decide how consumers commit offsets. With auto-commit, offsets may advance even when processing fails, which can create data loss from the application’s perspective. With manual commit, you can commit only after successful processing, but you must handle crash windows where processing succeeded and commit did not.
Many teams use consumer groups so multiple instances share partitions. That improves throughput but changes failure behavior: if one instance crashes, another instance takes over its partitions and reprocesses from the last committed offset. That reprocessing is expected; the system’s correctness depends on idempotency and replay safety.
For exactly-once processing, Kafka offers transactional APIs, but end-to-end exactly-once depends on sink capabilities and correct configuration. Treat “exactly-once” as a system property, not a single setting, and test it with failure injection rather than trusting defaults.
Configure Retention And Replication
Set replication factor based on fault tolerance goals and broker count. A replication factor of 3 is common for production clusters, but the right choice depends on your operational tolerance for broker loss and your ability to maintain quorum during maintenance. If you run fewer brokers, replication factor choices can reduce availability.
Align retention with consumer recovery time. If you expect a consumer to be offline for up to 8 hours, retention must exceed that window plus any worst-case lag. Also account for compaction if you use log compaction; compaction keeps the latest record per key, which changes replay behavior for consumers that rely on full history.
Watch topic-level settings like max message size and segment behavior. Oversized events can fail production writes, and segment settings can affect disk usage patterns. I’ve seen teams hit max.message.bytes after adding a new field to an event payload; the fix involved both schema changes and producer-side validation.
Build Observability And Backpressure
Track consumer lag, processing error rates, and end-to-end latency from event creation to sink write. Consumer lag alone does not show whether processing is stuck; a consumer can be “caught up” on offsets while downstream writes fail. Instrument the consumer pipeline so you can separate poll/deserialize time from business logic time and sink latency.
Use dead-letter topics for poison messages. A common pattern is to route deserialization failures or schema incompatibilities to a dedicated topic with the original payload and error metadata. That keeps the main pipeline moving while preserving evidence for remediation.
Backpressure needs explicit handling. If downstream systems slow down, your consumer processing threads can pile up, increasing memory usage and increasing the chance of timeouts. A bounded work queue and a clear retry policy (with jittered exponential backoff) help keep failure modes predictable.
Case Examples With Realistic Constraints
Retail Promotions Pipeline
A retail platform publishes PromotionApplied events from a checkout service. A promotions analytics consumer aggregates counts per store and day. During a schema upgrade, one consumer instance runs with an older schema version and fails to deserialize new fields.
The team routes failed records to a dead-letter topic and pauses only the affected consumer group. After updating the consumer, they replay from the last committed offset and verify that idempotent writes prevent double-counting. The outcome is not “zero downtime”; it is controlled degradation with preserved data for later reprocessing.
They also adjust retention from 2 hours to 24 hours because the incident response process took longer than expected. The change reduces the risk of losing events during future maintenance windows.
Payments And Idempotent Sinks
A payments service emits PaymentAuthorized events. A ledger service writes ledger entries to a database. A consumer crash occurs after writing to the database but before committing offsets, so the same event is processed again when the consumer restarts.
Instead of trying to prevent the crash window, the ledger service uses an idempotency key: it stores eventId and rejects duplicates via a unique constraint. The second processing attempt fails fast at the database layer, and the consumer still commits offsets after handling the duplicate case.
In this scenario, resilience comes from designing for retries and duplicates rather than relying on perfect timing. The team measures duplicate rate during chaos tests and confirms it stays within expected bounds.
Decision Checklist For Kafka Reliability
| Area | Question To Answer | What To Set Or Build | Failure Mode Covered |
|---|---|---|---|
| Event Contract | Can consumers replay old events after upgrades? | Backward-compatible schema rules + versioning | Schema drift breaks deserialization |
| Idempotency | Do sinks tolerate duplicates? | Unique keys, dedupe tables, or idempotent upserts | At-least-once reprocessing |
| Offsets | Do offsets advance only after success? | Manual commit after processing + error handling | Crash between processing and commit |
| Retention | Can consumers recover within outage windows? | Retention > max expected downtime + lag buffer | Data gaps after long pauses |
| Replication | What broker loss can you tolerate? | Replication factor and min ISR policies | Broker failure during writes |
Step-by-step checklist for a new topic: define the event schema and compatibility rules, add eventId and partition key strategy, choose consumer commit behavior, set retention based on recovery time, then run a failure test that kills a consumer mid-processing and verifies sink correctness.
If you skip the failure test, you usually discover the gap during an incident, when the team has less time to reason about offsets, retries, and downstream side effects.
Common Mistakes That Break Trust
One mistake is treating Kafka as a guarantee of business correctness. Kafka can deliver records reliably, but it cannot infer whether your sink writes are idempotent or whether your processing logic is safe under retries.
Another mistake involves mixing operational and business concerns in the same topic. If you publish both “state changes” and “audit logs” together, consumers may need different retention and different replay strategies, and the combined topic becomes harder to manage.
Teams also underestimate consumer lag interpretation. A stable lag metric can hide slow processing if offsets commit late or if the consumer pauses due to errors. Track error rates and processing durations per stage, not only lag.
Finally, avoid “set-and-forget” defaults. For example, a consumer library version like 3.6.x may change default behaviors around cooperative rebalancing or error handling compared with older versions; the exact effect depends on configuration, so you need to read the release notes and test rebalances in a staging cluster.
FAQ
How Do Consumer Groups Affect Ordering?
Ordering is preserved within a partition. A consumer group spreads partitions across instances, so events for the same partition key stay ordered, while events across different keys or partitions can interleave.
What Happens When Offsets Are Not Committed?
After a restart, the consumer resumes from the last committed offset, so records processed after that point may be reprocessed. Correctness then depends on idempotent handling in the consumer and sink.
How Should Retention Be Chosen?
Retention should cover the maximum time you expect consumers to be paused plus worst-case lag growth. If retention is shorter than that window, consumers can miss older events and cannot replay them.
Do Kafka Guarantees Remove the Need For Idempotency?
No. Failures can occur after processing but before offset commit, and retries can produce duplicates. Idempotency in downstream writes is the practical defense.
When Are Dead-Letter Topics Appropriate?
Dead-letter topics fit when a record cannot be processed due to deserialization errors, schema incompatibility, or validation failures. The main pipeline continues, and you can replay the dead-letter topic after fixing the consumer.
Author's Insight
Resilient event-driven systems with Kafka rely on a chain of assumptions: event contracts must survive schema evolution, consumers must commit offsets in a way that matches processing success, and sinks must tolerate duplicates. Kafka’s persistence and replication help with broker failures, but business correctness comes from idempotency and replay-safe design.
When teams treat “reliability” as a single Kafka setting, they usually discover gaps around offset commit timing, retention windows, and downstream side effects. A practical approach is to run failure injection tests that simulate consumer crashes, broker restarts, and schema mismatches, then verify sink outcomes rather than only message delivery.
Even with careful design, exact outcomes depend on your consumer library behavior and your sink’s constraints, so you need staging tests that mirror production configuration.
Key Takeaways
- Kafka provides a durable event log; business resilience comes from replay-safe event design and idempotent sinks.
- Offsets and commit timing determine whether reprocessing happens; duplicates are expected under failures.
- Retention must match consumer recovery time, not just perceived latency needs.
- Observability should cover lag, processing errors, and end-to-end latency, not lag alone.
- Dead-letter topics and failure injection tests turn “unknown failure modes” into measurable, fixable behaviors.
