8 September 2026 · 7 min read · recloud engineering

S3 object events and time partitioning that ages well

S3 stops being "a place files go" and becomes a data platform the day you add two disciplines: events, so things happen when objects land, and time partitioning, so five years of data stays cheap to store and fast to query. Both are ten-minute decisions that are miserable to retrofit — flow records, log archives and accounting exports all live or die by them.

Events: let the bucket drive the pipeline

Every PutObject can emit a notification. The rules that keep it sane in production:

Notify a queue, not a function. S3 → SQS → workers gives you retries, DLQs and backpressure for free; S3 → Lambda directly gives you a stampede when someone backfills a month of data. (The work-pool rules apply verbatim — an object landing is a job.)
Prefer EventBridge when more than one thing cares. Classic notifications allow limited overlapping prefix rules per bucket; EventBridge takes the single notification and fans it out by pattern to as many targets as the platform grows.
Design for at-least-once and no ordering. Duplicate events happen; key your processing on the object key (a natural idempotency token) and never assume part-2 arrives after part-1.

Partitioning: encode time in the key, hive-style

s3://acme-flows/netflow/dt=2026-09-08/hour=14/bng-01-1409-a7f3.parquet

That one convention pays four ways:

Query pruning. Athena, Trino, Spark and ClickHouse all understand dt= partitions — a "last 24 hours" query touches 24 prefixes, not five years of them.
Lifecycle rules become policy. Transition to Infrequent Access at 30 days, Glacier at 180, expire at 7 years — expressed on prefixes, executed by S3, audited by nobody at 2am.
Backfills and reprocessing get boundaries. "Re-run the 8th of September" is a prefix listing, not a scan.
Parallelism scales per prefix. S3's per-prefix request limits (3,500 writes / 5,500 reads per second) partition along with you — the old advice about random key prefixes has been obsolete for years; date prefixes are fine.

One decision deserves a paragraph of thought: event time or arrival time? Partition by when the data happened (event time) and late-arriving records rewrite yesterday's prefix — annoying for immutable pipelines. Partition by arrival and yesterday's query must peek at today's prefix to be complete. For billing-grade data we usually choose arrival time for the raw layer (immutable, append-only, easy to reason about) and event time for the curated layer the queries hit.

The small-files problem — solve it with the events you already have

A BNG fleet emitting flow records every minute yields thousands of kilobyte-sized objects per hour, and every downstream reader pays per-object overhead forever. The fix composes the two halves of this article: object-created events queue keys into SQS, a compactor drains the queue, and once an hour it rewrites each hot prefix as a handful of large Parquet files — raw partitions in, curated partitions out, lifecycle rules aging both. That's the whole architecture; it fits on an index card and it will still be correct when the bucket holds a petabyte.