16 September 2026 · 8 min read · recloud engineering

Thanos on S3: years of Prometheus metrics, one global view

Prometheus is the right collector and the wrong archive. Local TSDB on local disk means retention measured in weeks, one more failure domain per site, and — for an operator with Prometheus in every POP — no single place to ask "show me subscriber sessions across all sites, last quarter". Thanos fixes precisely those three things without replacing Prometheus at all: it bolts on beside it and turns an S3 bucket into the long-term store.

The moving parts, in the order they earn their place

Sidecar — runs next to each Prometheus, uploads every finished 2-hour block to the bucket, and serves the freshest data over gRPC. Prometheus keeps a couple of days locally; the bucket keeps years.
Querier — one PromQL endpoint that fans out to every sidecar and store, merges the answers, and deduplicates HA pairs: run two Prometheis per site with an external label like replica, and the querier folds their double-scraped series into one clean result. HA stops being a Grafana headache.
Store gateway — serves historical blocks straight from S3, caching index headers locally. This is what makes "last quarter" a query instead of an archaeology project.
Compactor — the quiet hero. It compacts small blocks, applies retention, and downsamples to 5-minute and 1-hour resolutions, so a year-long dashboard renders from a few thousand points instead of millions. Run exactly one per bucket — it assumes it owns the data.

Pointing everything at the bucket is one small file every component shares:

# objstore.yml
type: S3
config:
  bucket:   acme-metrics
  endpoint: s3.ap-southeast-2.amazonaws.com
  sse_config:
    type: SSE-S3

Retention policy that matches how humans look at graphs

Keep raw resolution for 30–90 days (incident forensics), 5-minute for a year (capacity reviews), 1-hour forever-ish (the board slide with five years of subscriber growth). Storage maths on S3 are gentle — downsampled metrics for an entire operator network cost less per month than one engineer-hour — so the constraint is query memory, not the bucket: give the store gateway and querier real limits and enable the query-frontend's result caching before a five-year dashboard teaches you why.

Operational notes from real fleets

External labels are load-bearing. site, replica, cluster on each Prometheus are what dedup and block organisation key on. Choose once, change never.
The compactor is a singleton with opinions. Halted compaction (usually overlapping blocks from label mistakes) silently degrades queries weeks later — alert on thanos_compact_halted.
Multi-site links are query paths. The querier reaches sidecars over gRPC across your WAN; on an operator backbone that's fine, just monitor it like the dependency it is.

And when Thanos is the wrong answer

Fairness requires the comparison: VictoriaMetrics delivers most of the same outcome — long retention, one query endpoint, lower resource use — as a single system that Prometheus simply remote-writes into, and we deploy it for teams who want fewer moving parts. Thanos wins when you want Prometheus untouched at the edge, S3 as the only stateful thing you babysit, per-component scaling, and object-storage economics all the way down. Both beat the default, which is throwing away your metrics every fortnight. Pair it with Loki on the same bucket pattern and the whole observability stack's state lives in object storage — the cheapest, most durable disk you'll ever not manage.