27 August 2026 · 7 min read · recloud engineering

SQS vs Kinesis: work pools, fan-out, and when each wins

SQS — one queue, competing workers Queue each message → one worker Worker A Worker B Worker C Kinesis — one stream, every group reads all Stream ordered shards every record → every group Billing rating Usage analytics Anomaly detection

The two services get lumped together as "async messaging", but they implement opposite philosophies, and picking the wrong one bakes an architecture problem in early. The distinction fits in two sentences. SQS is a work pool: many identical workers compete for messages, each message is processed by exactly one of them, and when it's deleted it's gone. Kinesis is a log: records are appended in order per shard, every consumer group reads the full stream independently, and the data stays there for replay whether zero or five applications have read it.

Multiple workers on the same queue — the SQS sweet spot

Scaling SQS is gloriously dumb: add workers. Visibility timeout hides a message while one worker holds it; finish and delete, or crash and it reappears for someone else. Redrive policies move poison messages to a DLQ after N attempts. This is the right home for jobs — "provision this service", "send this email", "generate this invoice" — where the only question is did it happen once, eventually.

# the whole consumer contract
receive (message invisible for 120s)
  → do the work (idempotently — at-least-once is the deal)
  → delete
# crash anywhere: message returns, another worker retries

What SQS won't give you: a second application reading the same messages. One consumer's delete is everyone's delete. The moment two systems want the data, either you bolt SNS or EventBridge in front and fan out to one queue per consumer — a perfectly good pattern — or you've discovered your data isn't a job at all.

Multiple readers of the same facts — the Kinesis sweet spot

Some data is not a task list but a record of what happened: RADIUS accounting, order state transitions, flow telemetry. Billing rates it, analytics aggregates it, anomaly detection watches it — three readers, same records, different pace. That's a stream. Each consumer group keeps its own checkpoint; a new consumer can start from a day ago and catch up; per-key ordering (partition key = subscriber, say) guarantees a Stop never overtakes its Start.

The costs are equally structural: shard management (workers scale with shards, not freely; hot partition keys make hot shards), checkpointing machinery instead of visibility timeouts, and a pricing model you must actually read. Enhanced fan-out helps consumer counts; it doesn't absolve you of shard-key design.

The decision rule

  • Command → SQS. "Do this once." Competing workers, DLQs, no replay, no ceremony. One queue per consumer type via SNS/EventBridge if several systems need to react.
  • Fact → Kinesis (or Kafka, same philosophy). "This happened." Many readers, per-key ordering, replay as a first-class feature.

The pipelines that age worst are the hybrids: facts shoved into SQS (second consumer arrives, data's already deleted) and commands shoved into Kinesis (one slow record blocks its whole shard behind it). In our RADIUS backend both live side by side and neither is exotic: accounting records flow down a stream because billing and analytics both drink from it; provisioning commands sit in a queue because each must run exactly once. Name the data honestly — command or fact — and the service picks itself.

Need a hand with this in production?

recloud is a group of software and network engineers specialising in Cisco Systems and Juniper, working with Australian ISPs, network operators and enterprises. See backend engineering, Kubernetes services. Or contact us.