← The Notebook

    Field Notes · Systems

    Do You Use Kafka, or Just Pay for It?

    By Bakare Praise··8 min read

    The conversation always goes the same way. Two services need to stop calling each other directly, someone says "let's put a message broker in between," and within a week the architecture diagram has a Kafka cluster on it. Nobody asked for a distributed commit log. Somebody asked for two services to stop calling each other directly.

    I want to be careful here, because this is not a Kafka-is-bad essay. Kafka is superb at the thing it does. The problem is that the thing it does is narrower and stranger than the way it gets used, and a lot of teams end up running a log to do a queue's job. They pay for partitions, consumer groups and rebalances to get behaviour a database table would have given them for free.

    There are really only three shapes of asynchronous work. Nearly every "we need a broker" conversation is one of them, and they want completely different tools.

    Shape one: a work queue

    "Do this task, once, soon, and tell me if it failed."

    Send a receipt. Transcode a video. Charge a card. Each message is a unit of work with an owner and a fate: it succeeds, it fails, it retries, or it eventually gives up somewhere a human can find it.

    What this shape demands is per-message accounting. You want competing consumers pulling from one pool so you can scale by adding workers. You want an ack when work completes and a nack when it doesn't. You want a retry policy with backoff, and a dead-letter queue where the poison messages go to be looked at on Monday. You want to ask "how many are waiting?" and get a number.

    RabbitMQ does this natively. So does SQS. So, for a very large number of systems, does Postgres:

    SELECT id, payload
    FROM jobs
    WHERE status = 'pending' AND run_after <= now()
    ORDER BY run_after
    FOR UPDATE SKIP LOCKED
    LIMIT 20;
    

    SKIP LOCKED is the line that makes this real rather than clever. Each worker locks the rows it takes and every other worker steps over them, so you get competing consumers without a broker and without a new operational surface. The part people underrate is that it all happens inside the same transaction as your business data. No dual-write problem. No "the row committed but the message didn't."

    I am not claiming this scales to a million messages a second. I am claiming that most systems calling themselves event-driven are doing a few hundred a minute, and the table is faster to build, easier to inspect with the SQL you already know, and possible to debug at 3am without learning a new CLI.

    Shape two: a batch

    "Do this to ten thousand rows at 2am."

    Settle yesterday's transactions. Expire stale sessions. Email everyone whose trial ends tomorrow. This shape needs no broker at all, and it is the one that gets skipped, because a cron job doesn't look like architecture.

    It should. A batch is the most debuggable thing in your system. It has a beginning and an end. You can re-run it. You can diff its output against the last run. You can read the whole process in one file instead of reconstructing it from six services and a trace. When it breaks at 2am it breaks in one place, and it is still broken at 9am when you are awake to look at it, instead of having silently retried into a corrupted state.

    The instinct to stream everything is usually a requirement nobody stated out loud. Before accepting "it has to be real-time," ask what actually happens if it is five minutes late. Often: nothing. Nobody is watching. The number appears on a dashboard that a human opens twice a day. That is a batch, and calling it a stream costs you a cluster and buys you a slightly fresher dashboard.

    Batching also has a property queues don't: it amortises. One INSERT of a thousand rows is not a thousand times the cost of one insert. One API call carrying five hundred records is not five hundred round trips. When throughput becomes a problem, the first fix is almost never a bigger broker. It is doing more per operation.

    Shape three: a log

    "Many consumers, each at their own position, replayable from the beginning."

    This is Kafka's actual shape, and it is genuinely different from the other two. A log is an ordered, durable, retained sequence that consumers read at their own pace. Nobody consumes a message and destroys it. Each consumer group tracks its own offset in the same stream.

    That single property, that the data stays and readers are independent, is what buys you the things Kafka is famous for:

    • Replay. A new service starts from offset zero and rebuilds its own view of six months of history. A bug corrupts a downstream projection; you fix the code, reset the offset, and reprocess.
    • Fan-out to unknown consumers. Publish payment.settled and the analytics team, the ledger service and the fraud model each read it without your producer knowing any of them exist.
    • Ordering per key. Everything for account A lands in one partition and is processed in order, while other accounts proceed in parallel.
    • Throughput. Sequential writes to a partitioned log, at volumes queues do struggle with.

    If your system needs those, the cluster is worth it. If it does not, you are paying for all of them anyway: in partition counts you have to guess up front and cannot lower later, in consumer-group rebalances that stall processing while assignments shuffle, in offset management, in retention tuning, in a schema registry, in the fact that a single poison message blocks its partition because there is no built-in dead-letter path. Kafka's guarantee is at-least-once, so duplicate handling remains entirely your problem. See The Retry That Charged You Twice for what that costs to do properly.

    A queue asks "who will do this?" A log asks "what happened, and who wants to know?" Two different questions. One of them is usually not yours.

    The diagnostic

    Here is how I actually decide. Answer honestly, about the system you have rather than the one you are imagining in eighteen months:

    1. Do you ever replay? Not "could we," but have you, or is it written down as something you will do? If the answer is no, retention is buying you nothing.
    2. How many independent consumers does this have? Not consumer instances, but independent interests, each at its own position. One is a queue. Several, owned by different teams, is a log.
    3. Does order matter, and at what grain? Globally is nearly impossible anyway. Per key is Kafka's sweet spot. Not at all means you have been handed a constraint you never needed.
    4. When one message fails, what should happen? If the answer is "retry it, then set it aside and carry on," you want a queue with a DLQ, and you will end up building that on top of Kafka by hand.
    5. Is the consumer's job to do work, or to know something? Work is a queue. Knowledge is a log.
    6. What is the actual volume? Write down the number. Not peak-of-peak with a 10x safety factor, but today's number. It is usually two orders of magnitude below where the tooling choice starts to matter.
    7. Who operates it at 3am? If the honest answer is "whoever is awake," that argues hard for the thing already in your stack.

    Score it and most systems come out as a work queue with one consumer, moderate volume, no replay, and a real need for dead-lettering. That is RabbitMQ, or SQS, or a table. It is not a log.

    What that looked like in practice

    At MAP Systems I built event-driven microservices for a platform carrying chat, voice and video, and the broker was RabbitMQ. That was the right call, and it was the right call for an unglamorous reason: almost every message was a task with an owner. Deliver this. Notify that. Transcode this recording. Each one needed to be acked, retried on failure, and dead-lettered when it kept failing, and each one was consumed exactly once by whichever worker was free.

    Nothing in that workload wanted replay. No second team was waiting to independently reinterpret a chat delivery from three weeks ago. The consumer's job was to do work, not to know something, and once you can say that sentence about your system, the choice stops being a matter of taste.

    The scaling that mattered came from the shape of the design, not the brand of the broker: isolating services so they failed independently, caching in Redis what didn't need recomputing, batching what didn't need to be immediate. A log in the middle of that would have added operational weight to a problem that did not have a log's shape.

    The honest summary

    Kafka is not a more powerful RabbitMQ, and RabbitMQ is not a lightweight Kafka. They answer different questions, and the expensive mistake is not picking the "weaker" tool. It is picking the tool whose question you were never asking.

    So before the cluster: say out loud what you have. If you can describe your requirement without using the words replay, retention or multiple independent consumers, you have a queue. Possibly you have a cron job.

    There is no shame in that. The boring answer holds up at 3am, and the work is the same either way.


    ← All notes

    Also in the notebook


    Colophon

    "I invented the term object-oriented, and I can tell you I did not have C++ in mind."
    — Alan Kay

    Print Run

    Three years in circulation across fintech, logistics, med-tech, ed-tech and e-commerce. Readership measured in uptime, not clicks — a new edition ships whenever the backend does.

    Office

    England, United Kingdom.
    [email protected]


    © 2026 Bakare Praise Stephen. All rights reserved.
    Visits are logged with approximate location and time. IP addresses are not stored.

    Vol. I · No. 1 · Filed from England

    Return to the masthead ↑