Cache Is Also A First-Class Citizen

Everything an object crosses on its way through a system is a boundary, and every boundary is a contract. A contract works only while both sides still agree on the shape of what passes across it, and shapes do not hold still: a field gets added, another gets dropped, a third gets split in two. So any contract meant to outlive the moment it was written needs some way to say which version of itself it is. We have grown good at giving most of our contracts that stamp, and oddly casual about giving it to one of them.

The idea of versioning is hardly new and is close to ubiquitous across frameworks, yet it is far more often left as an open-ended “architectural decision” that somebody will get around to implementing. Most commonly I have seen teams version their APIs and RPCs, and if they are being thorough, the events they publish onto a bus. If you write games, what about the assets you ship, the character models a client already has on disk? If you work on the web, chunk versioning has been handled for you by your framework for so long that you have probably never thought about it. You keep a tidy history of migrations for your database schemas, which is not quite versioning but is close enough to count. Through all of this, you almost certainly never think about versioning your cache at all.

The reason the cache gets skipped is a habit of mind, not a technical fact. Contracts that face outward get treated as explicit, because a stranger is on the other end and a broken shape is their outage as much as yours. Contracts that stay inside the house get treated as implicit, because you own both ends and you quietly assume you will simply remember. That split is the real decision most teams make, usually without noticing they made it.

For a long time the database sat comfortably on the implicit side of that line. It does not anymore. Teams have started versioning database schemas properly, even though a schema sits firmly inside the boundary of a single logical service. Neon lets you branch a database the way you branch code, because a branch is a copy-on-write clone whose writes are stored as a delta against the shared base, and a root branch can be restored to any moment still inside its configured history window. Atlas comes at the same problem from the schema side. You declare the shape you want and it works out the migration that gets you there, keeping a versioned migration directory when you want that history written down, and tracing column-level lineage in its cloud tier when you need to know where a value came from. Put the two together and a pull request can carry its own database, with its own schema diff, isolated from production until you say otherwise.

So the database has climbed out of the implicit bucket. That leaves the cache as the one inward-facing contract still living on trust. The neglect carries its own evolution tax, and it is the kind that can take your user experience, your infrastructure bill and your reliability down together. The reasoning I hear most often is that a cache is internal, and ephemeral anyway, so whatever you lose you can always get back “easily.”

I have written previously about the decisions teams make around serialising and versioning their cached data, and where the bytes themselves go wrong: how a cache’s default serializer, Redisson’s Kryo//NOTE 01Redisson (https://redisson.pro/docs/) is the Redis and Valkey client most Java backends reach for. Since December 2022 its out-of-the-box codec has been Kryo 5, a fast positional binary format that writes values with no field names on the wire, which is exactly what makes it break when a class changes shape., breaks the first time a class gains a field, and how an evolution-safe format like JSON, Protobuf or Avro, with a version stamp for the changes too large to absorb quietly, keeps the payload readable across versions. That is a real and necessary fix. It is also the easy half, because a serializer that can parse the new shape says nothing about whether the new shape can be filled.

From a readable byte to a safe rollout

A cache never really stores your objects, it stores bytes: every time you put a data model into Redis you serialise it first, flattening the object into an encoded blob through a codec, and reading it back runs that in reverse. Which codec does the encoding is what decides whether an object written by last week’s code can still be read by this week’s. Changing the codec makes the bytes readable again. It does nothing to make the rollout safe, and those are different problems. Evolving a schema that is already live is an operational discipline, and the ecosystems that took it seriously built real machinery for it. Kafka puts a schema registry//NOTE 02A schema registry, Confluent's and its peers, stores every schema centrally and checks each new one for compatibility at publish time, so a producer cannot ship a change that would break the consumers already running. The schema travels as a small id, not inline with every message. between producers and consumers, so a producer cannot publish a shape the consumers cannot read, and the two sides deploy on their own clocks. A database hands you a migration tool that versions every change and rolls it forward deterministically, one ordered step at a time. A cache has none of that. It has a time-to-live and your good intentions. So before arguing that a cache deserves to be treated as a first-class citizen, it is worth walking one real schema change all the way through a cache and watching where it actually hurts.

Stop treating caching as an afterthought

Picture the object at the centre of all this: a user profile a service caches because it gets read constantly and is expensive to rebuild from scratch.

public class UserProfile {
    public String userId;
    public String name;
    public String email;
    public String role;   // exactly one role
}

The codec question is settled here. This is stored through an evolution-safe format, so the class can gain a field without the next read throwing. Now make the smallest change a growing product ever asks for: one role per user stops being enough, and a user needs to hold several. The safe move is not to touch the old field but to add a new one beside it, and to teach the reader to map the old single value onto the new list when an old payload comes back.

public class UserProfile {
    public List<String> roles;

    // v1 wrote a single "role". Map it forward when we read old bytes.
    @JsonSetter("role")
    public void setLegacyRole(String single) {
        if (single != null) this.roles = List.of(single);
    }
}

That seems like a fair call, and here it was completely safe. A user who had one role still has exactly one role, wrapping it in a list loses nothing, and there is no database round trip or missing feature hiding behind it. Serving the adapted old data was simply correct.

But what happens when it is not that clean? What happens when the new shape needs data the cache never held in the first place? Suppose v2 adds a payment tier to the profile. The old cached profile never carried one, because that value lives in an entirely different service, so there is nothing to wrap and nothing to adapt. The value was never in those bytes. That is no longer a compatibility problem. It is a hydration problem, and it behaves very differently.

public class UserProfile {
    public String userId;
    public String name;
    public String email;
    public List<String> roles;

    // New in v2, and never present in a single cached byte.
    // The only source of truth for this lives in the payments service.
    public PaymentTier paymentTier;
}

It matters because of why the cache existed in the first place, which is that the profile was an aggregate rather than a record.

Stitching those together on every read is expensive, and something like a profile gets read far more often than it gets written, to the point where a server-side client might read it on every page view. Caching it was the right call, and it remains the right call. Which leaves exactly one question with no comfortable answer attached: who is on the hook for actually delivering the value that new field represents? The cache cannot answer that. Something has to.

The cruel part is where in the release this bites. The change looks finished long before it is dangerous.

Weighing the options

The tempting move is to pre-catch the gap, so when the tier is missing you hand everyone a default tier and get on with your day. It feels safe, and it is a lie. You shipped a feature and then quietly declined to deliver it, and the user ends up looking at a badge that is wrong rather than one that is broken, which is worse precisely because it looks fine.

The other move is to fetch the tier synchronously on read. That is correct, but it charges a steep penalty for what amounts to a hard invalidation on the next read of every key. Every read that misses the new field now bypasses the cache entirely and re-aggregates across all three origin services at once, so the cache you built to protect those services is now aiming a firehose at them//NOTE 03This is the cache stampede, also called the thundering herd: a mass of requests all miss at once and hammer the origin together. Naming it helps, because every fix below is really a way to keep the herd from arriving in one instant.. A net UX regression, shipped on purpose, by people who thought they were being careful.

Maybe shadow writes then, pre-warming a v2 keyspace ahead of the release. On a team of one that sounds clever. On a fast-moving cross-functional team it becomes an operational burden nobody signed up for, because somebody has to answer a series of increasingly awkward questions. How do you guarantee when the shadow ends? What happens to the dead code once the shadow is promoted to the hot path? Who watches it in the meantime, and who is empowered to make those calls? It gets worse with more than one of you, because two developers shadowing two changes at once produce overlapping keyspaces that drift out of sync the moment one of them writes without knowing about the other. That is not a migration strategy. It is a second production system nobody agreed to run.

Or perhaps we keep it very simple, because after all it is only a cache, and a cache is ephemeral by nature. Let the TTL run out, let it serve stale data until then, and trust that it will serve correct data one day. That is not a strategy, it is a hope, and it quietly decides that a wrong answer for the length of a TTL is acceptable without anyone actually deciding that. So nothing that follows is clever. Each option trades one thing for another, and the entire job is knowing which trade you just made.

What I would reach for is a background job that fetches the missing piece under some back pressure, writing the fresh value back with the remaining TTL so the key still expires exactly when it was always going to. Stale while it rehydrates//NOTE 04This is the cache-side cousin of HTTP's stale-while-revalidate (RFC 5861): serve the stale response immediately, refresh it out of band. Same bargain, different layer..

UserProfile p = cache.get(key);            // v1 bytes, adapted to the v2 shape
if (p.paymentTier == null) {               // the v2 field the old bytes never carried
    hydrator.submit(() -> {                // serve stale now, fix the cache off-thread
        PaymentTier tier = paymentService.fetch(key);
        long ttl = cache.remainingTtl(key); // keep the original expiry
        cache.set(key, p.withTier(tier), ttl);
    });
}
return p;                                   // the request thread leaves immediately

The shape of the fix is always the same and only the throttle changes. What you must not do is let every cold key rush the origin the instant the new version ships, because that is the stampede wearing a different hat. Three levers keep it bounded, and they compose, and one of them carries a technique worth naming, probabilistic early expiration//NOTE 05Letting only a random fraction of reads trigger the upgrade is a form of probabilistic early expiration. Each read rolls the dice, so the refreshes spread across the staleness window instead of all firing in the first second after a deploy..

• A single-flight lock. When a key needs rebuilding, one worker takes a short-lived lock on that key, does the fetch, and writes the fresh value back, while every other request for the same key is handed the adapted stale value instead of queueing behind the database. One origin query per hot key, not one per reader.
• A probabilistic upgrade. Rather than every legacy read attempting the fetch, each one rolls the dice and only a small fraction do, which stretches the refreshes across the staleness window instead of delivering them all in one spike.
• A bounded worker pool. Ten workers means at most ten origin queries in flight no matter how many million keys happen to be cold, so the pool size becomes a hard cap on the load your migration can generate, chosen by you rather than by the traffic.

Composed, the whole read path is still short, because those three levers are three lines in the same handler rather than three separate code paths.

UserProfile p = cache.get(key);                    // v1 bytes, adapted to the v2 shape
if (p.paymentTier == null                          // the v2 field the old bytes never carried
        && ThreadLocalRandom.current().nextDouble() < 0.05) {   // lever 2: only a fraction try
    hydrator.submit(() -> {                         // lever 3: a fixed-size pool caps the fan-out
        if (locks.tryLock("hydrate:" + key)) {      // lever 1: one winner per key, the rest serve stale
            try {
                PaymentTier tier = paymentService.fetch(key);
                cache.set(key, p.withTier(tier), cache.remainingTtl(key)); // keep the original expiry
            } finally {
                locks.unlock("hydrate:" + key);
            }
        }
    });
}
return p;                                           // the request thread leaves immediately

None of this has to be a manual cutover, and that is the whole point. Tie the read path to the deploy and the cache heals itself as traffic flows. While only the old code is live it reads and writes v1. During the rollout the new code reads v2 and falls back to v1, upgrading each key it touches. Once the deploy settles it reads and writes v2 alone, and the last v1 keys expire on their own TTL. The database sees a trickle, never a flood, and nobody stands at a console flipping a switch at the exact wrong second.

Redisson is not a thin Redis client but a wide framework full of distributed data structures and services, which is why I argued at length before that its one narrow codec default was such a poor fit. Given how much it does, it is fair to ask whether it already handles this hydration for you. The honest answer is that it does not. There is a read-through loader, but it fires on a miss and the caller waits on it, which is the exact opposite of serving something stale. There is write-behind, which batches your writes and flushes them after a delay, but the backlog it accumulates is an unbounded queue sitting in Redis//NOTE 06Redisson's write-behind defaults are a 1000ms delay and a batch size of 50, but writeBehindBatchSize bounds each flush, not the queue behind it. The local cache options (eviction policy, sync strategy, reconnection strategy) contain no refresh knob at all., not something that pushes back when you produce faster than it drains. The local cache options let you tune eviction, synchronisation and reconnection, and not one of them mentions refreshing anything.

What Redisson does hand you is every ingredient. Distributed locks and expirable semaphores, so exactly one refresher wins the race for a given key. A cluster-wide rate limiter. Bounded executor pools to run the work inside. You get the building blocks. Assembling them into something that serves stale safely is work you have to do yourself.

Even then, the right answer depends entirely on where you are standing. Ten thousand users in a single region is not ten million sharded across ten geographies, and an approach that scales beautifully across one may not survive the other. Know your blast radius before you choose: how many keys are involved, how many services sit behind them, and how bad the worst case gets if you are wrong. What matters is that you make the call on purpose rather than inherit it by default.

What the streaming world does instead

Everything in the last two sections was safety I had to assemble by hand, out of locks and pools and a carefully ordered read path. It is worth stopping to notice that one corner of our industry does not build any of that by hand, because it decided long ago that a schema changing under a live reader is a platform problem rather than an application one. That corner is streaming, and the machine it built for exactly this is the schema registry.

When a Kafka producer serialises a record with Avro, it does not put the schema on the wire at all. It registers the schema with a central registry, gets back a small numeric id, and writes only that id in front of the bytes. The part that matters is not the id, it is what the registry does before it hands one back. You set a compatibility mode//NOTE 07Confluent's registry supports BACKWARD, where new code reads old data, FORWARD, where old code reads new data, and FULL, which is both, set per subject. It checks a proposed schema against the ones already registered and rejects anything that violates the mode. Documented at https://docs.confluent.io/platform/current/schema-registry/fundamentals/schema-evolution.html. on the subject, and the registry refuses to accept a new schema that would break it. A change that would leave the running consumers unable to read is rejected at publish time, inside the producer’s own deploy, before a single incompatible byte reaches anyone.

Sit that next to the cache for a moment. The cache accepts anything. You serialise a v2 object, Redis stores the bytes without an opinion, and the incompatibility you just introduced waits quietly until a reader trips over it on some later deploy. The registry takes that same failure and moves it from the worst place it can happen, a read in production, to the best place it can happen, the build that caused it. It is the same class of bug, caught an entire release cycle earlier, by shared infrastructure instead of by whoever is on call.

There is a second thing the registry buys that the cache never had, which is that producers and consumers stop having to move in lockstep. Because any consumer can resolve any schema the registry has accepted, a producer can ship v2 while half the consumers are still running v1, and nothing has to coordinate the two beyond the compatibility rule that was already enforced. That is the rolling-deploy coexistence problem from earlier in this piece, the one a cache makes you solve by hand with versioned keyspaces and lazy hydration, except here it is simply a property of the platform that nobody has to think about.

# an Avro record carries a schema id, not the schema itself
value.serializer=io.confluent.kafka.serializers.KafkaAvroSerializer
schema.registry.url=http://registry:8081

# compression is applied per batch, not per value
compression.type=zstd

Even the compression story inverts. Compression can be a liability for a cache because it runs once per value, so whatever setup a compressor needs gets paid on every single read and write. Kafka compresses whole record batches at once, thousands of records to a batch, so that same setup is amortised down to nothing, and an algorithm that is expensive one value at a time turns close to free a batch at a time. Where the compression lives turns out to matter as much as which compression you chose.

None of this is exotic, and none of it is new. It is a registry, a compatibility check, and a place to keep the schema, and streaming teams have run it for years without ever calling it heroic. The cache got none of it. There is no registry standing between the writer and the reader, no gate that turns away an unsafe change, no platform promise that an old reader survives a new write. The cache simply stores your bytes and trusts you, which is the whole reason every guard in the last two sections had to be built by hand.

Own the default you ship

There is a philosophy underneath the Kryo default, and it is worth saying out loud, because a great deal of infrastructure quietly runs on it. Kryo 5//NOTE 08Kryo 5 is documented at https://github.com/EsotericSoftware/kryo. It is a general-purpose Java serialization library built for speed and a compact wire format, used well beyond Redisson. is the fast, compact binary codec Redisson reaches for out of the box, the one I pulled apart before for breaking the moment a cached class changes shape. The reasoning that made it the default is that a cache is temporary, volatile data: if the schema changes you deploy new code, and if an old key will not deserialize you catch the exception, treat it as a cache miss, and fetch a fresh copy from the database. On a whiteboard that is clean. In production it is the precise move that turns a schema change into a stampede, because “treat it as a miss” at scale means every key misses in the same instant.

So let me be plain about the assumption buried in that word. A cache is not ephemeral in the sense that gets used to excuse it. It exists to deliver value, the performance a user can actually feel, the materialisation of a computation too expensive to repeat on demand. The moment you treat it as disposable you start making decisions that throw that value away, and somewhere downstream of those decisions sits real money and a real person waiting on a page that is quietly wrong.

None of this is a new idea. Our industry has known for decades that the moment you introduce a cache you are breaking a structural rule, because you are choosing to duplicate state across separate physical systems. Martin Fowler lays the consequence out in Microservice Trade-Offs: distribute your services and you inherit eventual consistency, the need for caching grows along with the architecture, and cache invalidation is waiting at the end of it. The inconsistency never announces itself. Business logic goes on making decisions against stale data, and by the time anyone investigates, the window has long since closed. Developers, he writes, have to “figure out how to detect when things are out of sync before doing anything the code will regret.”

The moment a cached copy of your data outlives the deployment that created it, it stops being a temporary performance hack and hardens into a high-stakes interface contract. We meticulously version our APIs and run strict database migrations because we are afraid of breaking things. The cache can break every one of those same things, a crash on read, a poisoned interface, a stampeded database, and we hand it a TTL and look away. If you are the one choosing the default your teammates inherit, that choice is not neutral, and “it is only a cache” is how it becomes someone else’s incident on a Friday night. The split between the contracts we version and the ones we do not was never real. Your cache deserves a migration strategy. It deserves a version.