14 September 2026 · 7 min read · recloud engineering

Grafana Loki on S3: logs without the Elasticsearch bill

The traditional log stack indexes every word of every line, which is why it eats a cluster of stateful nodes and a budget line nobody defends happily. Loki's wager is the opposite: index almost nothing — just a small set of labels and time ranges — compress the raw lines into chunks, and put everything in object storage. You grep at query time with parallel workers instead of paying to index at ingest time. For the log shapes an operator actually has — syslog from hundreds of network devices, RADIUS and BNG session logs, Kubernetes pod output — that trade is a bargain.

Everything in one bucket

Modern Loki (the TSDB index format) keeps both the index and the chunks in S3 — no Cassandra, no DynamoDB index store, no local state that matters. The storage half of the config is compact:

storage_config:
  tsdb_shipper:
    active_index_directory: /data/tsdb-index
    cache_location:         /data/tsdb-cache
  aws:
    s3: s3://ap-southeast-2/acme-loki
compactor:
  working_directory: /data/compactor
  retention_enabled: true
limits_config:
  retention_period: 2160h        # 90 days, and S3 makes 900 affordable

The components that look stateful are shippers and caches — lose a node and the bucket still holds the truth. Retention is the compactor's job, not a lifecycle rule: it rewrites chunks as it deletes, so keep exactly one compactor running and give it room to work.

Labels are the whole design

Loki lives or dies on label cardinality. Labels are the index; every unique combination is a stream with its own chunks. The discipline:

Label what you filter by: site, device_role (bng, olt, core), app, severity. Tens of values each.
Never label what identifies: subscriber IDs, session IDs, IPs, request IDs. Those live in the line and LogQL greps them: {device_role="bng", site="me1"} |= "INH0012345".

Break this rule and you recreate the Elasticsearch cost model inside Loki, one tiny chunk at a time. It's the same instinct as every scaling story we write: know which dimension is allowed to explode, and make sure it's the cheap one.

Deployment shape and the honest caveats

Skip both the single binary (no isolation) and the full microservices constellation (twelve deployments for a mid-size operator is cosplay). The simple-scalable split — a write target, a read target, a backend target — matches real operator volumes and scales each path independently on Kubernetes; queries fan out across read replicas, and "search 30 days of BNG logs" parallelises against S3 instead of hammering hot disks.

Caveats, because there are always caveats: needle-in-a-haystack searches over long ranges are slower than an indexed store — that's the trade you accepted, mitigate with time bounds and label filters. S3 request costs, not storage, dominate a busy cluster's bill — chunk caching (memcached) pays for itself quickly. And alerting on logs works (the ruler evaluates LogQL), but metrics belong in a metrics store — which is exactly where Thanos and the same S3 bucket pattern come in.