Redis Persistence Wars: RDB, AOF, and Hybrid

Introduction & hook

Imagine this: It’s 3 AM. PagerDuty screams. Your lead generation service, powered by a critical Redis instance, is down. You scramble to restart it, but the new leads that came in just minutes before the crash are gone. Vanished. Your carefully crafted user sessions? Poof. Leaderboard scores? Reset. This isn’t just a bug; it’s a direct hit to your business, your users, and your sleep schedule.

Why did this happen? Because Redis, at its core, is an in-memory database. Its lightning-fast performance comes from keeping all data in RAM. But RAM is volatile. A server reboot, a power outage, or a Redis process crash means all unsaved data is lost forever. This is where Redis persistence steps in. It’s the strategy we use to safeguard our in-memory data, ensuring that even if the server dies, our application’s state can be resurrected. We’re going to dissect the three primary ways Redis fights data loss: RDB (Redis Database) snapshots, the AOF (Append-Only File), and the modern Hybrid approach. Each has its battle scars and its triumphs, and choosing the right one is crucial for your system’s resilience.

How it works (the visual example)

Let’s build a mental model. Think of your Redis server as a bustling kitchen, constantly preparing dishes (processing commands).

RDB Persistence: The Recipe Book

Imagine a world-renowned chef (your Redis server) who creates exquisite dishes (data). Every few hours, or after a certain number of orders, the chef stops and writes down every single recipe for every dish currently in their memory into a giant, concise recipe book. This book is a RDB snapshot. It’s a binary file containing a compact, point-in-time representation of your entire dataset.

How does Redis create this snapshot without stopping the chef from cooking? This is where the magic of fork() and Copy-on-Write (CoW) comes in. When it’s time to save, the main Redis process (the parent chef) fork()s itself, creating an identical child process (an apprentice chef). Critically, this child initially shares all memory pages with the parent. If the parent chef continues to modify a recipe (write to memory), the operating system steps in, makes a private copy of that specific memory page for the parent, and lets the parent modify its copy. The child process continues to see the original, unmodified page. This ensures the child can write the snapshot of the data as it was at the fork() moment, without blocking the parent or being affected by ongoing writes. The child writes this snapshot to disk, then exits. Simple, efficient, and genius.

AOF Persistence: The Detailed Journal

Now, consider a meticulous journalist (your Redis server) who, instead of writing a full summary periodically, notes down every single action in a detailed journal. “User A added item X to cart,” “User B updated score Z.” This journal is the Append-Only File. Every write command (SET, LPUSH, HSET, etc.) that modifies the dataset is simply appended as a command string to this file.

When Redis restarts, it doesn’t need a snapshot. It simply reads the AOF from top to bottom and re-executes every command, meticulously rebuilding the exact state of the data as it was before the crash. This means extremely high durability, as you lose only the commands that weren’t written to the AOF. However, AOF files can grow very large, very quickly, making recovery slow. This is where AOF rewrite mechanics save the day. Periodically, Redis will fork() a background process (another journalist apprentice). This apprentice reads the current in-memory state of the data and writes a new, optimized, minimal sequence of commands to a temporary AOF. For example, if you incremented a counter 1000 times, the old AOF would have 1000 INCR commands. The new AOF would just have one SET command with the final value. Once the rewrite is complete, Redis atomically swaps the old AOF with the new, compact one. The main process is mostly unaffected, just like with RDB.

Hybrid Persistence: Best of Both Worlds

The Hybrid AOF (available since Redis 4.0) combines these two. When an AOF rewrite happens, the new AOF file starts with an RDB preamble – essentially, a compact RDB snapshot of the dataset at the time of the rewrite. After this RDB preamble, subsequent commands are appended in the usual AOF format. On restart, Redis first loads the RDB snapshot blazingly fast, then replays only the short AOF commands that occurred after the snapshot. This dramatically speeds up recovery compared to a pure AOF file replay while maintaining the high durability of AOF.

Real-world use cases

Choosing the right persistence strategy isn’t about finding a one-size-fits-all answer; it’s about understanding your application’s tolerance for data loss and downtime.

RDB-only is your go-to for disaster recovery and analytical processing. If you’re using Redis as a cache where the original data can be rebuilt from a source of truth (like a database), or for large-scale data ingestion where losing a few minutes of data isn’t catastrophic but quick recovery of the bulk dataset is paramount, RDB is excellent. Think leaderboards where daily resets are common, temporary session stores that can tolerate some loss, or batch processing results. It’s also fantastic for creating backups: just copy the RDB file.
- Anti-pattern: Using RDB-only for high-durability requirements like financial transactions, critical user settings, or shopping carts where losing even seconds of data is unacceptable. Your data loss window here is determined by your save configuration, which can be minutes or hours.
AOF-only is for when durability is king. If your Redis instance holds user session data, real-time messaging queues, or any data that simply cannot be lost, AOF with careful fsync configuration is your best bet. It provides a much smaller data loss window, potentially down to zero.
- Anti-pattern: Running AOF with appendfsync always on a high-throughput, latency-sensitive application. This can turn every write into a blocking disk operation, throttling your Redis performance and causing unacceptable latency spikes. Also, running AOF without regular rewrites on a write-heavy instance will lead to gigantic AOF files and agonizingly slow recovery times.
Hybrid AOF is the recommended default for most production systems seeking fast recovery with high durability. It blends the speed of RDB loading with the minimal data loss of AOF. This is ideal for almost any critical application where you need both quick restarts and minimal data loss, offering a robust compromise.
- Anti-pattern: While rare, if your application has exceptionally low write volume and extreme sensitivity to any background process, the occasional fork() for the RDB preamble might be a consideration, though for most systems, it’s negligible compared to the benefits.

Implementation & code

Configuring Redis persistence isn’t about writing application code; it’s about setting up your redis.conf correctly. Let’s look at robust, production-ready configurations versus naive approaches.

First, the naive defaults often look like this:

# A naive RDB-only setup (default, but often not enough durability)
save 900 1    # Save if 1 change in 15 minutes
save 300 10   # Save if 10 changes in 5 minutes
save 60 10000 # Save if 10000 changes in 1 minute
appendonly no # AOF is disabled

This default RDB setup means you could lose up to 15 minutes of data. For many applications, that’s a disaster waiting to happen. If appendonly no and there are no save rules, you have no persistence at all.

For a robust, production-ready system, we generally aim for Hybrid AOF with a sensible fsync policy.

# Production-ready Hybrid AOF configuration
appendonly yes
appendfilename "appendonly.aof"

# This is the CRITICAL part: How often Redis fsyncs the AOF to disk.
# everysec: Best balance. Max 1 second data loss. (Recommended default)
# always: Highest durability (0 data loss), but can be very slow.
# no: Lowest durability (OS flushes), but fastest. (Dangerous!)
appendfsync everysec

# AOF rewrite configuration. Critical for managing AOF file size.
# Rewrite AOF when it's 100% larger than its previous size,
# but only if the AOF is at least 64mb.
auto-aof-rewrite-percentage 100
auto-aof-rewrite-min-size 64mb

# Enable RDB snapshots *within* the AOF rewrite. This is the Hybrid mode.
# When a rewrite happens, the AOF will start with an RDB snapshot.
# This setting is usually enabled by default if AOF is on and Redis >= 4.0.
# The actual config entry for this is `aof-use-rdb-preamble yes`
# It's typically the default behavior with `appendonly yes` in modern Redis.
# You don't usually need to explicitly set it unless you want to disable it.
# (For clarity, I'm noting its conceptual importance here, even if it's default.)

# You can disable RDB explicit saves if AOF is your primary persistence,
# as the RDB preamble provides RDB snapshotting during AOF rewrites.
# However, many prefer to keep RDB saves as a redundant backup strategy.
# save "" # Uncomment this to disable explicit RDB snapshots

Why this configuration is robust:

appendonly yes: Activates AOF, ensuring command logging.
appendfsync everysec: This is the workhorse. It tells Redis to flush the AOF buffer to disk at least once per second in a background thread. This gives you high durability (you might lose up to 1 second of data if a crash happens between flushes) without blocking your main Redis operations for every single write.
auto-aof-rewrite-percentage 100 and auto-aof-rewrite-min-size 64mb: These settings prevent your AOF from growing infinitely large. When the AOF grows to double its size (and is at least 64MB), Redis will automatically trigger a background rewrite. This keeps the AOF file compact, speeding up restarts and reducing disk space usage. The magic here is the aof-use-rdb-preamble yes (which is the default on modern Redis if AOF is on), turning your AOF rewrites into efficient Hybrid snapshots.

Senior-level insights & gotchas

Even with the right configuration, persistence has subtle behaviors that can bite you at scale.

The fork() tax is real: Both RDB snapshots and AOF rewrites trigger a fork() call. For small datasets, this is nearly instant. But on a Redis instance with tens or hundreds of gigabytes, fork() can take hundreds of milliseconds, even seconds. During this time, the main Redis thread is blocked. No commands are processed. This leads to latency spikes.
- Why it happens: When fork() is called, the OS duplicates the parent’s page table. While modern Linux kernels are heavily optimized for CoW and don’t copy all memory, the process of setting up the new page table entries can be expensive for very large memory footprints.
- Mitigation: Monitor redis_fork_milliseconds in INFO persistence. Schedule RDB saves and AOF rewrites during off-peak hours if possible. Consider using a replica-no-disk setup for RDB backups if latency is absolutely critical. Ensure your server has enough free memory before the fork().
Copy-on-Write memory overhead: While fork() doesn’t immediately duplicate all memory, heavy writes during an RDB save or AOF rewrite will trigger CoW. If your application has a high write volume while persistence is running, Redis’s memory usage can temporarily spike. If you have a 100GB dataset and 20% of it is modified during a snapshot, you’ll need an additional 20GB of RAM for the copied pages. Running out of memory during a save can cause the process to be killed by the OOM killer, leading to an outage.
- Mitigation: Provision your Redis servers with ample head-room memory (e.g., 50-100% more than your peak dataset size). Monitor used_memory_rss and total_system_memory.
fsync policies and durability vs. latency:
- appendfsync always: Guarantees 0 data loss (barring OS/hardware bugs), as every write command is flushed to disk before acknowledgement. But this turns Redis into a disk-bound process, killing latency, especially on slow disks. Only use for extremely sensitive data and if you can tolerate very low throughput.
- appendfsync everysec: The sweet spot. Up to 1 second of data loss. Provides excellent durability with good performance. The fsync happens in a background thread, so it doesn’t block the main Redis event loop. If the background fsync thread takes too long (e.g., slow disk), Redis might temporarily fall back to blocking for fsync to avoid extreme data loss. This is tunable with no-appendfsync-on-rewrite yes/no.
- appendfsync no: Redis delegates fsync to the operating system. Data loss can be seconds to minutes, depending on the OS sync interval. Use only if performance is paramount and data loss is highly tolerable, even then, it’s generally ill-advised for critical data.
AOF rewrite strategy: Don’t rely on default auto-aof-rewrite-min-size for very small instances. If your dataset never hits 64MB, a rewrite might never trigger, leading to an ever-growing AOF. Tune auto-aof-rewrite-min-size or trigger BGREWRITEAOF manually. Also, consider setting no-appendfsync-on-rewrite yes to prevent the main thread from blocking on fsync calls during an AOF rewrite. This prioritizes the main thread’s responsiveness over strict background AOF durability during the rewrite, as the main AOF file isn’t being written to by the background process.
Recovery time benchmarks: RDB files typically load much faster than AOF files of comparable logical data size, because RDB is a direct memory dump, while AOF needs to parse and re-execute commands. The Hybrid AOF significantly mitigates this for AOF, making recovery times much closer to RDB because it only replays the incremental commands after the RDB preamble. For massive AOF files (e.g., hundreds of GBs) on older Redis versions or pure AOF, recovery could take hours.
Disk I/O contention: Both RDB saves and AOF fsync operations write to disk. On busy systems, these disk operations can contend with other processes or even Redis’s own read/write operations, degrading performance. Use fast SSDs, monitor disk I/O metrics (iops, bandwidth, latency), and consider dedicated storage for your persistence files.

Summary & production checklist

Navigating Redis persistence means balancing durability with performance and recovery time. Don’t let your Redis be the weak link in your data chain.

Assess Durability Needs: How much data loss can your application actually tolerate? Minutes? Seconds? Zero? This dictates your strategy.
Default to Hybrid AOF: For most critical production systems, appendonly yes with appendfsync everysec provides the best balance of durability, performance, and recovery time.
Tune AOF Rewrites: Configure auto-aof-rewrite-percentage and auto-aof-rewrite-min-size to keep your AOF file lean and efficient.
Monitor fork() Latency: Watch redis_fork_milliseconds from INFO persistence. Schedule persistence operations if these latencies are impacting your SLAs.
Provision Memory Generously: Ensure your Redis server has sufficient RAM to handle temporary CoW memory spikes during RDB saves or AOF rewrites.
Understand fsync Trade-offs: Be explicit about your appendfsync policy. everysec is usually right, always is almost always too slow, no is almost always too risky.
Test Recovery Routinely: Simulate a crash and verify your Redis instance recovers correctly and within acceptable timeframes. Don’t wait for a real outage to discover issues.
Backup Persistence Files: Regularly copy your *.rdb and *.aof files to offsite, redundant storage. This is your ultimate safety net.