Storage & Data

Three ways to be given somewhere to put things.

The difference is what the API's unit of work is, and that single fact cascades into everything else.

Three different questions that people constantly mash into the word “reliable”. Your bank:

• Durability — will the bank still know I have £5,000 next year? (Did the bytes survive?)
• Availability — is the ATM working right now? (Can I reach the bytes?)
• Consistency — if I withdraw at one ATM and my partner checks the balance at another one second later, do they see the new number? (Do all readers agree?)

They are independent. A system can be perfectly durable and completely unavailable (your data is safe in a region that is on fire and unreachable). It can be highly available and inconsistent (every replica answers instantly, some with stale values).

CAP is the referee: when the network splits the building in half, you may keep answering (possibly with stale notes) or stay correct (by refusing to answer). Not both. PACELC is the honest follow-up question: fine — but what do you do the other 99.99% of the time, when the network is healthy? There you trade latency against consistency, and that trade is the one you live with every day.

Partitions are rare. Latency is constant. So the useful question is almost never “what happens under partition” — it is “what does a read cost me when everything is fine?”

OLTP (online transaction processing) is a shop till: thousands of tiny, precise operations per second, each touching one or two rows, each of which must be right. OLAP (online analytical processing) is the end-of-quarter review: one enormous question — “average basket size by region by week for three years” — that reads two columns out of a billion rows. Same data; opposite physics.

Two rules that will save you repeatedly:

1. Never point your application's request path at a warehouse. Warehouse queries are billed and engineered for seconds-to-minutes and scanned bytes. A dashboard that refreshes per page-view against a columnar engine is a bill and a latency problem at the same time.
2. Never run big analytics on your OLTP primary. A single unindexed aggregate can evict the working set from cache and take the whole transactional workload down with it. Route it to a read replica at minimum; export it to the warehouse properly.

The modern hybrid worth knowing: HTAP-ish features — a columnar cache or engine attached to the transactional database (Spanner's columnar engine, AlloyDB's columnar engine, Cosmos DB analytical store / Fabric mirroring) — that let you run analytics on operational data without an ETL hop. Useful, but they narrow the gap rather than close it.

S3 is a bucket per namespace, a key per object, and a class per object that you can change at any time — by hand, by lifecycle rule, or automatically. Everything else in S3 is a feature bolted to that: versioning keeps old copies, replication copies to another bucket, and the classes are the temperature dial from Session 1.

Source note: AWS guidance covers S3, EBS and Glacier at catalogue depth and doesn't enumerate the classes. From here on this section is AWS documentation and current pricing.

The lifecycle rule is the mechanism, so here it is at S3's actual API shape:

// S3 lifecycle configuration — the config IS the mechanism
{ "Rules": [{
    "ID": "logs-age-out", "Status": "Enabled",
    "Filter": { "Prefix": "inference-logs/" },
    "Transitions": [
      { "Days": 30,  "StorageClass": "STANDARD_IA" },
      { "Days": 180, "StorageClass": "DEEP_ARCHIVE" }],
    "Expiration": { "Days": 2555 },
    "AbortIncompleteMultipartUpload": { "DaysAfterInitiation": 7 }
}]}

Storage is often not the biggest line. The four others:

• Requests. PUTs cost roughly 10× GETs, and archive-class PUTs cost more again (~$0.05/1,000). A pipeline writing millions of tiny objects pays more in requests than in bytes.
• Retrieval. Only on IA and colder — but at $0.03/GB (Glacier Instant), reading 10 TB back costs $300 on top of everything else.
• Egress. Data out to the internet is ~$0.09/GB for the first 10 TB, falling in tiers. Serving public assets straight from a bucket instead of through a CDN is a classic unforced error — CloudFront-to-origin transfer is free.
• Minimum billable object size. IA classes bill a 128 KB floor per object; archive classes add ~40 KB metadata. Ten million 20 KB objects in Standard-IA bill as if they were 128 KB each.

You are renting a disk over the network and pretending it's local. Two dials: how big and how fast. The generational shift across all three clouds has been decoupling those two dials — older volume types tied speed to size (want more IOPS? buy more GB you don't need); modern ones let you provision capacity and performance independently.

Renting a flat versus owning a house. The landlord fixes the boiler, replaces the roof and re-paints the exterior (patching, backups, failover). You still choose the furniture and you're still responsible for the mess you make inside (schema, indexes, queries, connection handling). And there are things you're simply not allowed to do — no knocking through walls: no OS shell, no arbitrary extensions, no superuser.

Managed is right almost always. The genuine exceptions:

• You need an extension or a version the service doesn't offer. Common with exotic Postgres extensions, or engine builds pinned by a vendor application.
• You need OS-level access — a custom collation, a filesystem-level tool, an agent that must run on the DB host.
• Regulatory or data-residency constraints the managed service doesn't satisfy in your region.
• Cost at very large, very steady scale, where a well-run self-managed cluster on reserved instances genuinely undercuts the managed premium — and you have the DBA capacity to run it properly. Be honest about the second half.

Everything else — “we want control”, “we already know how to run Postgres”, “managed is expensive” — is usually a decision to spend engineer-hours on undifferentiated work. The patterns catalog calls the managed option Database-as-a-Service and treats it as the default for cloud-native applications for exactly this reason.

A restaurant with one kitchen. Adding more waiters (read replicas) serves more customers reading the menu and collecting food, and it's cheap. But every order still passes through one kitchen. To cook more you either build a bigger kitchen (vertical scaling — finite) or open separate kitchens each responsible for different dishes (sharding — complex) or use a fundamentally different design where every branch can cook independently and they reconcile (distributed SQL).

All three shapes distribute data the same way: hash or range the key, assign to a partition, put partitions on machines. The API you get is a direct consequence.

The schema is the mechanism here, so this is the one snippet that belongs in Pass 2:

# The whole design decision, in six lines. Access pattern first:
#   "get all inference jobs for one tenant, newest first"
PartitionKey: "TENANT#{tenant_id}"       # WHERE the data lives — spreads load
SortKey:      "JOB#{iso8601_ts}#{job_id}"  # ORDER inside the partition
# → query(PK = TENANT#42, SK begins_with "JOB#14201B-07") is ONE partition read.
# → "all jobs across all tenants by status" is NOT answerable. Add a GSI, or
#   accept a full scan. Decide this now, not after launch.

1. Indexes multiply writes. Every global secondary index is a second table maintained synchronously. Three GSIs means a single logical write costs roughly four writes. This is the most common reason a DynamoDB bill lands at 3× the estimate.
2. Item size is a multiplier, not a footnote. A write unit covers 1 KB. A 3.5 KB item costs 4 write units — so a schema change that adds a verbose field to a hot table can quadruple the bill without a single extra request.
3. Consistency is priced. A strongly-consistent read costs twice an eventually-consistent one, on DynamoDB and Cosmos DB alike. Defaulting the whole application to strong is a decision to double your read bill, usually for no user-visible benefit.
4. Cross-region replication multiplies writes again. A global table writes in every replica region; three regions means roughly 3× the write charges plus cross-region transfer.

A cache is a bet that you'll be asked the same question again soon. It's the notepad next to the phone: you write down the answer you just looked up, because looking it up again costs more than reading your own handwriting. And like a notepad, the danger isn't losing it — it's trusting it after the answer has changed.

Everything so far answers “what is the state of this one thing right now?”. The warehouse answers “what happened across everything, over time?”. Different question, different physics (columnar, from S1), different service, and — critically — a different billing model: you generally pay for bytes scanned or for reserved compute slots, not for rows stored.

Two numbers decide the warehouse billing model. On-demand / per-TB-scanned wins for sporadic, exploratory use — you pay nothing between queries. Capacity / slot / reservation wins once queries are frequent and predictable, and it also caps the blast radius of somebody's accidental unfiltered scan. The migration point usually arrives the month after a dashboard goes live and starts auto-refreshing every five minutes.