The Quietly Lurking Danger: When Everything Seems Fine...
For system administrators, disaster recovery (DR) tests are usually routine and predictable processes. At least, you might think so. You took a backup of your production environment and transferred it to your test server. The base backup opened without any issues, and everything looks fine. Now comes the turn of applying (replaying) the WAL (Write-Ahead Log) files, which will process the transactions that occurred after the backup into the database.
You are monitoring the logs; WAL files are being pulled from the archive one by one, processed... And then suddenly everything stops.
At first, you might think this is a large transaction block and the system is busy writing it to disk. Five minutes pass, ten minutes pass, it becomes half an hour… You check the server's resource usage; the CPU is relaxed, disk I/O is almost zero. However, the recovery process never ends, and the database just won't start up and give the “database system is ready to accept connections” message. Gigabytes of WAL files waiting to be processed are lined up outside, but no one inside is touching them. The most frustrating part of the job is that the error logs are spotless. There are no crashes, no error messages; just silence and a frozen startup process.
The Real Problem Itself: MultiXactOffsetSLRU
Since the logs aren't telling you anything, let's drop down to the operating system level. When you examine what the hanging PostgreSQL startup process (PID) is waiting for, you might encounter this scenario: The process is a futex_wait has entered the (OS-level deadlock) state. With PostgreSQL's own internal monitoring tools (or if hot standby is enabled pg_stat_activity When you look through, you see the wait event it is stuck on:
LWLock: MultiXactOffsetSLRU
The system requested a lock for a MultiXact operation, but this lock has deadlocked in an infinite loop. Data cannot be read, written, and naturally, the remaining WAL files cannot be processed.
While investigating the root cause, if you browse through the PostgreSQL community error logs (pgsql-bugs), you will find exactly this scenario described Bug #19490 You can face the truth.
So why did this crash happen to you out of the blue? The answer is hidden in the seemingly innocent difference in “versions.” For example, your production server 16.8 when working on the version, because your backup server on which you performed the restore test is newly installed 16.14 version. You might think, “Anyway, it's just a minor version difference, it is backward compatible.”. However, in PostgreSQL 16.12 The code change added to the SimpleLruWriteAll() function triggers this latent deadlock bug during WAL replay operations across minor versions.
If you also experience the issue where WAL files suddenly stop processing in the middle of a restore operation and the system silently hangs, You are not alone. Let's see how you can get out of this deadlock and get the database running.
Searching for a Temporary Solution: Downgrading
After identifying the issue, the first thing that might come to mind is to bring the PostgreSQL version on the test server to the same level as production, that is, to 16.8 (or to 16.11 where the bug does not yet exist). It is theoretically possible to downgrade only the binary files without touching the data directory. However, dealing with package downgrades and trying to resolve operating system dependencies is the last thing you would want to do. You need a much cleaner and more permanent solution.
Rescuer Process: PostgreSQL 16.15
Fortunately, the PostgreSQL community recognized this issue (Bug #19490) and took action. August 13, 2026 published on the date of PostgreSQL 16.15 its version has become the solution for those suffering precisely from this problem. Reviewing the release notes, it is seen that the deadlock on SLRU locks during WAL replay has been resolved.
All you need to do is move the server forward. Since this is a “minor” version update, you do not need to deal with database files or complex migration scenarios:
- Stop the locked PostgreSQL service on the test server.
- via your operating system's package manager (apt/yum etc.) postgresql-16 by updating your package 16.15 Downgrade to version.
- Restart the service.
Breaking the Silence and Happy Ending
After starting the PostgreSQL service, observe the log file. And that moment of resolution will come… Gigabytes of WAL files, waiting untouched for hours, will begin to be processed in seconds. pg_stat_slru It will be reset and wait events will return completely to normal. There will be no deadlocks and no hanging. Within just a few minutes, you will see the expected line in the log file:
LOG: database system is ready to accept connections
Your database will start up healthily, full data consistency will be ensured, and your stressfully started restore test will conclude successfully.
Lessons to be Learned (Best Practices)
These kinds of restore tests serve as a reminder of a few golden rules regarding database management:
- Don't Underestimate Minor Versions: “16.x and 16.y are the same anyway” Do not say that. In the database world, small code changes can lead to unpredictably large effects, especially in low-level kernel mechanisms like WAL replay and SLRU.
- Version Synchronization is Required: Compare the versions of the servers where you perform restore tests or keep as Disaster Recovery (DR) with your production servers. on the exact same minor version Make sure to keep holding it.
- Stay Up to Date: If version differences are inevitable or you are setting up a brand new server, always upgrade to the latest known stable version announced by the PostgreSQL community (in our case, 16.15).
If your restore process is freezing for no reason and the logs aren't telling you anything, maybe in the background waiting quietly and silently MultiXactOffsetSLRU the lock exists. The solution is not very far away; your server Update to 16.15 and sit back and watch the logs flow.
