Moving a call onto a log does not remove the coupling between two services. It moves the coupling into time, and the consumer’s spare capacity decides how much time.

Take a chat app, a made-up one, that shows each member a list of their conversations with an unread count and a preview of the latest message. Build it twice. In the first version, sending a message calls the counter service before the send returns. In the second, sending a message appends an event to a partitioned log, and a consumer updates the counts a moment later. On a normal day the two are hard to tell apart.

Now give each a 30-second outage. In the synchronous version, take one counter shard down: every send and every read of the counts on that shard fails for those 30 seconds, about 1,100 failed sends per outage in the simulation behind this post, each shown to the person who pressed send. In the event version, crash one of the three counter consumers without a clean shutdown and bring it back 30 seconds later with its own partitions. Nothing fails. Instead, with the consumers at 90 percent utilization, its third of the conversations shows stale rows for about five minutes. A member who looks at their list in that window sees a wrong row for a busy conversation about three times out of four. A sender who looks a second after sending almost never sees the message they just sent. And about 150 events that the consumer applied in its last few seconds come back half a minute later and are applied a second time.

So where did the coupling go, and how big is it?

Everything here comes from a small simulation, described near the end with a version you can run in a second, so the numbers are a model’s numbers. The mechanisms are not new. The original Kafka paper already says that a consumer that crashes without a clean shutdown can hand its successor duplicate messages, “after the last offset successfully committed,” and that order is guaranteed only within a partition (Kreps, Narkhede, and Rao, 2011, section 3.3). What the simulation adds is how large these effects are, who sees them, and which common fixes hold up.

Headroom is a staleness budget

Say the consumer can apply μ events a second and the log brings it λ. Its utilization is ρ = λ/μ, and its headroom is 1 – ρ. Stop it for D seconds. The backlog is λD events, and it drains only at the spare rate, μ – λ, so catching up takes λD/(μ – λ) = ρD/(1 – ρ) seconds after the consumer returns. Counting from the moment it stopped, its rows are stale for about D/(1 – ρ) seconds. The outage is multiplied by one over the headroom.

The simulated consumer is the same queue the formula describes, so the agreement is expected, and two details roughly cancel at the median. Events applied after the last offset commit are delivered again, up to 5 seconds’ worth, which can stretch the window by up to ρ × 5/(1 – ρ) seconds, 45 at 90 percent. And I counted rows as stale until the oldest unapplied event was under 2 seconds old for 10 seconds, which ends the window about ρ × 2/(1 – ρ) seconds early. At 95 percent the same crash produced anything from 8 to 13 minutes, because a backlog that drains at 5 percent of capacity is at the mercy of every burst of traffic.

Who sees it

A stale row is not always a wrong one. Call the age of the oldest unapplied event the lag. A read returns a wrong value only if an event that changes that row is still waiting. If events change a row at rate r and the lag is L seconds, the chance is 1 – e^(-rL). Over the stale window the lag climbs from 0 to about D and falls back, spending about equal time at every level, so the share of wrong reads is close to 1 – (1 – e^(-rD))/(rD). It depends on r and D, not on ρ. Utilization sets how long the window lasts, not how bad it is inside it.

A row changes about 1.25 times per message in its conversation: every message changes the preview, and a quarter of them bring this member’s read. For busy conversations, 6 messages a minute, that predicts about 74 percent of random reads wrong, and the simulation gave 75 to 78 percent for the crashed consumer’s conversations. For active ones, a message a minute, it predicts 26 percent, simulated 26 to 27. For quiet ones, a message a day, almost none.

Many real reads are not random. They come right after writes. A sender who looks at their own list one second after sending saw a wrong row 99 to 100 percent of the time, since their own message was the event still waiting. (Many clients hide this case by drawing the sender’s own message locally.) A recipient who opens the app five seconds after a notification saw a wrong row about 90 percent of the time. Even with no crash, at 95 percent utilization about 4 percent of those sender reads were already wrong.

That is the trade. The strict synchronous version gave writers and readers an error for 30 seconds. The event version gives no one an error, and readers get a plausible wrong answer for five minutes, with no reason to doubt it.

“Then keep it synchronous”

The comparison is fair only if the synchronous design gets the same outage, so I took the counter shard for the same conversations down for the same 30 seconds and tried three behaviors. In all three, reads of the counts on that shard fail while it is down.

• Strict. The send fails. About 1,100 failed sends per outage. Afterward nothing is wrong.
• Retry inside the request, for up to 10 seconds. About 730 sends fail. The ones sent in the last 10 seconds of the outage wait and succeed late, by 5 seconds at the median.
• Best effort. The send succeeds and the counter update is dropped. No send fails. But averaged over the next two and a half minutes, about one read in five of a busy or active conversation was wrong, and each dropped update lingers: the count until that member’s next read, the preview until the next message. After nine outages, 316 rows were still wrong when the runs ended, most of them in quiet conversations whose next event may be days away.

The event version left 5 to 8 rows wrong after the same nine crashes, all of them duplicates in quiet conversations. It is not free of outages either. The send still has to append to the log, so if the log is down, sends fail there too, unless an outbox moves that wait into time as well. And it kept its reads only because its counts live in a store that stayed up. Had that store gone down instead, its reads would have failed too, and the consumer would still have had a backlog to clear. Synchronous calls are not free of duplicates either: a client that retries after an ambiguous timeout can store the same message twice. Strict synchronous calls trade availability for correctness. Best-effort ones lose updates without telling anyone. Events keep every update but deliver some of them late and some twice.

“Then deduplicate”

The duplicates come from the offset commit. By default a Kafka consumer commits its offsets automatically every 5 seconds, so a consumer that dies uncleanly has applied up to about 5 seconds of events, more if a batch is slow, that its successor will see again. At 90 percent utilization that was about 150 events per crash, between 36 and 263 depending on how long before the crash the last commit happened.

The usual fix is to remember the ids of applied events and skip repeats, with a time to live (TTL) so the store stays bounded. A TTL store catches a repeat only if it arrives within the TTL, so what matters is how long each path takes to deliver an event again. These numbers are at 80 percent utilization:

• A crash where the consumer restarts with its partitions within 30 seconds, as a static group member can: repeats arrive 30 to 35 seconds after the first application.
• A crash where the group waits out the session timeout, 45 seconds by default, and reassigns the partitions: 45 to 50 seconds.
• A stall. A handler hangs, and the group takes its partitions away after max.poll.interval.ms, 5 minutes by default, and for a static member only once its session has also expired: 300 to 350 seconds.
• A replay of the log to fix a bug or rebuild a view: minutes to days.

A 30-second TTL counted every repeat twice on the crash and stall paths. One minute covered both crashes and missed the stall. Ten minutes covered all three. As it must, the share applied twice equaled the share of repeats older than the TTL, so the TTL has to exceed the delay of the longest path plus the commit interval.

Two cases no practical TTL fixed.

The hung handler’s late write. The hung handler looked the event id up before it hung, and when its call finally returned, after the replacement had applied the same event, it wrote the event anyway. Any dedup check that runs outside the transaction of the write leaves this check-then-act gap, whatever its window. A dedup row written in the same transaction as the update, with the id as a unique key, would catch it, and so does the offset check below.

The replay. Replaying the crashed consumer’s whole log into the live counters applied 97 percent of the replayed events twice with a 30-second TTL and 39 percent with a 10-minute one, the share older than the TTL. Only a TTL longer than the oldest replayed event, about 17 minutes here and the whole retention in production, would have skipped them all, and then the replay would have changed nothing. What it left depended on when you looked. A replay re-applies old resets along with old increments, and the TTL skipped exactly the recent events, so at first it rewound busy and active rows by ten minutes: 100 seconds after the replay, 10,088 rows were wrong with the TTL and 1,930 without any dedup, summed over nine runs. Those healed at the next read. Once every event had been applied, the TTL left 698 rows wrong and no dedup 1,668, all of them messages counted twice in quiet conversations. The TTL changed which rows broke and when, not whether the replay was safe.

Two designs held on every path:

• Store each partition’s last applied offset in the same transaction as the update, as a conditional write, and skip anything at or below it, which needs the offsets and the rows in one database. The successor and the hung handler both find the stored offset already past their event. The Kafka consumer documentation describes this as a way to get “exactly once” semantics, stronger than the default “at-least once.” It catches only a second delivery of the same record, though. An event published twice, by an outbox relay that restarts or an application that sends it again after a timeout, gets a new offset and passes. And it skips a replay into the live view entirely, which is safe but fixes nothing, so a replay meant to repair a view should rebuild a new one and switch reads over.
• Make the update monotone. Count as unread the other member’s messages whose sequence number is above this member’s highest read, and take the highest sequence number as the preview. This needs sequence numbers assigned at the source, and with one sequence per sender it reduces to two maxima: the other member’s latest message and the latest of theirs this member has read. Applying such an update twice, or out of order, changes nothing.

Both left zero wrong rows, by construction, which is the point. They depend on a property of the write, not on how late a repeat arrives. Kreps and colleagues already suggested deduplicating by offset or by a unique key in the message, and Kafka’s own transactions do not change this for a counter in a database, because they cover writes back into Kafka. A TTL store also costs memory that grows with its window, at least the event rate times the TTL times the size of an id: a gigabyte for 100,000 events a second, a 10-minute TTL and 16-byte ids, before any index.

“Kafka keeps the order”

It keeps the order within a partition, and only there. The key decides which events share a partition, so the key decides which orders hold.

For this part the read model matters. Until now a read covered every message sent before it, which makes a counter that resets to zero on a read exact. Here a recipient’s device gets each message typically 0.2 to 1 second after it is sent, and a read covers only what the device has shown. These runs already skip repeats by stored offsets, so only order and the reset are at work.

Key every event by conversation and a read never overtook the message it covers. In the simulation that holds by construction, because each partition is applied in order and a message enters the log the moment it is sent. In a real app it holds only if the message event is in the log before anyone can see the message. With a dual write, or an outbox relay that lags, a read event can land first even on the same key.

Key each event by the user who caused it, the sender for a message and the reader for a read, and the two can land on different partitions. In normal operation that rarely mattered, because reads come seconds after their messages: under 0.5 percent of reads overtook their message, even at 95 percent utilization. A crash changed that. About 16 percent of the reads that covered a message sent during the stale window were applied before it, at 80, 90 and 95 percent alike, about 1,000 extra reordered pairs per crash at 80 percent.

And a counter that resets to zero on a read is wrong even without reordering. A message can already be in the log, and counted, before it reaches the reader’s device, and the reset erases it. With conversation keys and no crash at all, that counter left 293 rows wrong at the end of nine runs, and a synchronous counter that resets the same way would too. These do not pile up: each is fixed at that member’s next read, and new ones appear. With user keys, the nine crashes added about 210 more such rows at 80 percent utilization and 1,740 at 95, about 24 and 190 per crash. Most were in active conversations, which also heal at the member’s next read. The quiet ones, about 11 and 43 per crash, can stay wrong for days. The monotone counter, which compares sequence numbers instead of resetting, left none in any run, which is also how the simulation computes the right answer.

The key is a contract. Choosing it chooses which events keep their order, and the shape of the update decides what happens when that order breaks anyway.

What to watch

• The age of the oldest unapplied event, in seconds, per partition. Lag counted in messages hides how long readers have been seeing old rows.
• Headroom, 1 – ρ, per consumer, measured under a backlog, and what it becomes when partitions move. If one of three consumers stays down past the session timeout, the other two each take half again as much load, and above two thirds utilization they cannot catch up until it returns. And a consumer that is down for more than a share 1 – ρ of the time never catches up. I did not simulate either case. Both follow from the same arithmetic.

How the simulation works

A made-up chat app with 103,600 two-member conversations: 600 busy ones at 6 messages a minute, 3,000 active ones at one a minute, and 100,000 quiet ones at one a day. Recipients open a conversation after half the messages they get, 20 seconds later on average. Events go to 12 partitions, keyed by conversation unless stated, and three consumers each apply their events one at a time with exponential service times and commit offsets every 5 seconds. A read is wrong if its row differs from the state with every event applied once, at the moment it happened. Each setting ran with nine seeds. The script below gives nearly the same staleness numbers for a single consumer, 61, 147 and 299 seconds, with the standard library in under a second.

"""How long does a 30 s consumer crash leave a projection stale? Standard library only.
One consumer applies events one at a time, in arrival order. Events arrive at random, 50 per second, and service
times are random too, sized so that the consumer's utilization is rho. Offsets are committed every 5 s.
At t = 60 s the consumer dies uncleanly and comes back 30 s later at the last committed offset, so the events it
applied after that commit are applied again. The projection counts as stale until the oldest event not yet applied
is less than 2 s old, and stays that way for 10 s.
"""
import random
import statistics
def run(rho, seed, rate=50.0, crash=60.0, down=30.0, commit_every=5.0, end=1500.0):
    rng = random.Random(seed)
    arrivals, t = [], 0.0
    while (t := t + rng.expovariate(rate)) < end:
        arrivals.append(t)
    mu, n = rate / rho, len(arrivals)
    applied = [None] * n                       # when each event was first applied
    free, i = 0.0, 0
    while i < n:                               # normal work until the crash
        finish = max(arrivals[i], free) + rng.expovariate(mu)
        if finish > crash:
            break                              # not finished before the crash, so it runs again after
        applied[i], free, i = finish, finish, i + 1
    phase = rng.uniform(0, commit_every)
    last_commit = phase + commit_every * ((crash - phase) // commit_every)
    committed = sum(1 for x in applied[:i] if x <= last_commit)
    free = crash + down                        # back, from the committed offset
    for j in range(committed, n):
        free = max(arrivals[j], free) + rng.expovariate(mu)
        if j >= i:
            applied[j] = free
    t, j, since = crash + down, i, None
    while t < end:                             # stale until the lag stays under 2 s for 10 s
        while j < n and applied[j] <= t:
            j += 1
        lag = t - arrivals[j] if j < n and arrivals[j] <= t else 0.0
        if lag < 2.0:
            since = t if since is None else since
            if t - since >= 10.0:
                return since - crash
        else:
            since = None
        t += 0.05
    return None
print("utilization  30/(1-rho)  stale period, median of 9 runs")
for rho in (0.5, 0.8, 0.9):
    stale = statistics.median(run(rho, seed) for seed in range(1, 10))
    print(f"{rho:>11}  {30 / (1 - rho):>8.0f} s  {stale:>29.0f} s")

Where this is wrong

• It is a toy: exponential service times, random arrivals, and one event at a time, with each consumer taking its four partitions in time order. A real backlog changes a consumer’s rate, up as batches grow or down as the backlog loads the database, so measure it under a backlog. The time-ordered processing understates reordering under user keys, and the model knows nothing of real latency tails from batching, fetch waits, or garbage collection.
• The crashed consumer returns with its own partitions, as a static group member does. With the default dynamic membership, the crash also triggers a rebalance, and under eager assignment every consumer in the group pauses for it.
• The synchronous versions have no retry queue and no reconciliation job. A nightly job that recomputes counts would erase most of the best-effort errors, and most of the event version’s too.
• Kafka’s newer consumer group protocol, generally available since Kafka 4.0 and still opt-in for clients, sets the session timeout on the broker: 45 seconds by default, within a default range of 45 to 60.

One rule

For every call you turn into an event, write down the consumer’s headroom, 1 – ρ, measured under a backlog, and multiply the longest stop you consider plausible by 1/(1 – ρ). That is roughly how long readers will see stale rows. Every update must also be safe to apply twice or out of order, whenever it arrives, and only a monotone one was: an offset or a dedup row written in the same transaction as the update stops repeats, but not reordering.

Events and asynchrony do not remove coupling. They move it from space into time. A synchronous call couples the sender to the counter for as long as the counter is down. An event couples the reader to the consumer for that long times 1/(1 – ρ). Moving to events can lower the Release Coupling Ratio, the share of teams that must deploy together for one change. The coupling in time is paid for in headroom.

Chapter 6 of Beyond Microservices, “Event-Driven and Async Patterns in the Real World,” covers the same move of coupling from space into time, and the contracts, idempotency, and replay discipline it calls for.

References

Kreps, J., Narkhede, N., and Rao, J. “Kafka: a Distributed Messaging System for Log Processing.” NetDB 2011.

Apache Kafka 4.4 documentation, consumer configurations. https://kafka.apache.org/44/configuration/consumer-configs/

Apache Kafka 4.4 documentation, broker configurations. https://kafka.apache.org/44/configuration/broker-configs/

Apache Kafka 4.4 documentation, the consumer rebalance protocol. https://kafka.apache.org/44/operations/consumer-rebalance-protocol/

Apache Kafka 4.4 Javadoc, KafkaConsumer, “Storing Offsets Outside Kafka.” https://kafka.apache.org/44/javadoc/org/apache/kafka/clients/consumer/KafkaConsumer.html