The Danger Waiting Silently: While Everything Seems Fine...
For system administrators, disaster recovery (DR) tests are usually routine and predictable processes. At least, you might think so. You took a backup of your production environment and transferred it to your test server. The base backup opened without any issues, and everything looks fine. Now comes the turn of applying (replaying) the WAL (Write-Ahead Log) files, which will process the transactions that occurred after the backup into the database.
You are monitoring the logs; WAL files are being pulled from the archive one by one, being processed… And then suddenly everything stops.
At first, you might think that this is a large transaction block and the system is busy writing it to disk. Five minutes pass, ten minutes pass, it becomes half an hour… You look at the server's resource utilization; the CPU is idle, disk I/O is almost zero. However, the recovery process does not end, and the database somehow fails to start up and give the “database system is ready to accept connections” message. Gigabytes of WAL files waiting to be processed are lined up at the door, but no one inside is touching them. The most annoying part of it all is that the error logs are spotless. There is no crash, no error message; just silence and a frozen startup process.
The Real Problem Itself: MultiXactOffsetSLRU
Since the logs are not telling you anything, the next step is to investigate at the operating system level. When you examine what the stalled PostgreSQL startup process (PID) is waiting for, you may encounter the following situation: the process has entered a futex_wait state, indicating a lock wait at the operating system level. When examined using PostgreSQL’s internal monitoring tools—or through pg_stat_activity if hot standby is enabled—you can see the wait event where the process is stuck:
LWLock: MultiXactOffsetSLRU
The system requested a lock for a MultiXact operation, but this lock has deadlocked in an infinite loop. Data cannot be read, written, and naturally the remaining WAL files cannot be processed.
While investigating the root cause, you may come across Bug #19490 in the PostgreSQL community’s bug reports (pgsql-bugs), describing this exact scenario.
So why did this lockup occur seemingly out of nowhere? The answer lies in an innocent-looking version difference. For example, while your production server may be running version 16.8, the backup server used for the restore test may be running version 16.14 because it was installed more recently. You might assume that a minor version difference is backward-compatible. However, a code change introduced to the SimpleLruWriteAll() function in PostgreSQL 16.12 triggers this hidden deadlock during WAL replay across minor versions.
If your WAL files suddenly stop being processed during a restore operation and the system freezes without any warning, you are not alone. Let’s look at how to overcome this deadlock and bring the database back online.
Searching for a Temporary Solution: Downgrading
After identifying the issue, the first thing that might come to mind is to bring the PostgreSQL version on the test server to the same level as production, that is, to 16.8 (or to 16.11, where the bug does not yet exist). It is theoretically possible to downgrade only the binary files without touching the data directory. However, dealing with downgrading packages and trying to resolve operating system dependencies is the last thing you would want to do. You need a much cleaner and more permanent solution.
Rescuing Process: PostgreSQL 16.15
Fortunately, the PostgreSQL community identified Bug #19490 and addressed the issue. Released on August 13, 2026, PostgreSQL 16.15 provides a solution for systems affected by this problem. The release notes confirm that the deadlock involving SLRU locks during WAL replay has been resolved.
All you need to do is advance the server. Since this is a “minor” version update, you do not need to deal with database files or complex migration scenarios:
- Stop the locked PostgreSQL service on the test server.
- Use your operating system’s package manager (apt, yum, etc.) to update the postgresql-16 package to version 16.15.
- Restart the service.
Breaking of the Silence and Happy Ending
After starting the PostgreSQL service, monitor the log file. Then the moment of resolution will arrive: gigabytes of WAL files that have remained untouched for hours will begin processing within seconds. pg_stat_slru will reset, and the wait events will return to normal. There will be no deadlock or stalling. Within just a few minutes, you will see the expected line in the log file:
LOG: database system is ready to accept connections
Your database will start up healthily, full data consistency will be ensured, and your stressfully started restore test will conclude successfully.
Lessons to be Learned (Best Practices)
These kinds of restore tests serve as a reminder of a few golden rules regarding database management:
- Do Not Underestimate Minor Releases: Do not assume that “16.x and 16.y are essentially the same.” In the database world, small code changes can have major and unpredictable effects, especially in low-level core mechanisms such as WAL replay and SLRU.
- Version Synchronization Is Essential: Ensure that servers used for restore testing or maintained for Disaster Recovery (DR) run the exact same minor version as your production servers.
- Stay Up to Date: If a version difference is unavoidable or you are setting up a new server, always use the latest stable release announced by the PostgreSQL community—in this case, version 16.15.
If your restore process freezes for no apparent reason and the logs provide no clues, a silently waiting MultiXactOffsetSLRU lock may be lurking in the background. The solution is within reach: update your server to version 16.15, then sit back and watch the logs flow.
