{"id":41105,"date":"2026-08-27T10:14:05","date_gmt":"2026-08-27T07:14:05","guid":{"rendered":"https:\/\/profelis.com.tr\/?p=41105"},"modified":"2026-08-27T10:24:20","modified_gmt":"2026-08-27T07:24:20","slug":"how-a-disaster-recovery-drill-can-turn-into-a-problem-silent-hang-during-postgresql-restore-bug-19490","status":"publish","type":"post","link":"https:\/\/profelis.com.tr\/en\/2026\/08\/27\/felaket-kurtarma-tatbikati-nasil-soruna-donusebilir-postgresql-restore-surecinde-sessiz-kilitlenme-bug-19490\/","title":{"rendered":"How Can a Disaster Recovery Drill Turn into a Problem? Silent Lockup During the PostgreSQL Restore Process (Bug #19490)"},"content":{"rendered":"
\n\t\t\t\t
\n\t\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t

The Danger Waiting Silently: While Everything Seems Fine...<\/h2>\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t

For system administrators, disaster recovery (DR) tests are usually routine and predictable processes. At least, you might think so. You took a backup of your production environment and transferred it to your test server. The base backup opened without any issues, and everything looks fine. Now comes the turn of applying (replaying) the WAL (Write-Ahead Log) files, which will process the transactions that occurred after the backup into the database.<\/span><\/p>

You are monitoring the logs; WAL files are being pulled from the archive one by one, being processed\u2026 And then suddenly everything stops.<\/span><\/p>

At first, you might think that this is a large transaction block and the system is busy writing it to disk. Five minutes pass, ten minutes pass, it becomes half an hour\u2026 You look at the server's resource utilization; the CPU is idle, disk I\/O is almost zero. However, the recovery process does not end, and the database somehow fails to start up and give the \u201cdatabase system is ready to accept connections\u201d message. Gigabytes of WAL files waiting to be processed are lined up at the door, but no one inside is touching them. The most annoying part of it all is that the error logs are spotless. There is no crash, no error message; just silence and a frozen startup process.<\/span><\/p>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t

\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\"\"\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t\t

The Real Problem Itself: MultiXactOffsetSLRU<\/h2>\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t

Since the logs are not telling you anything, the next step is to investigate at the operating system level. When you examine what the stalled PostgreSQL startup process (PID) is waiting for, you may encounter the following situation: the process has entered a <\/span>futex_wait<\/span> state, indicating a lock wait at the operating system level. When examined using PostgreSQL\u2019s internal monitoring tools\u2014or through pg_stat_activity<\/span> if hot standby is enabled\u2014you can see the wait event where the process is stuck:<\/span><\/p>

LWLock: MultiXactOffsetSLRU<\/b><\/span><\/p>

The system requested a lock for a MultiXact operation, but this lock has deadlocked in an infinite loop. Data cannot be read, written, and naturally the remaining WAL files cannot be processed.<\/span><\/p>

While investigating the root cause, you may come across <\/span>Bug #19490<\/b><\/span> in the PostgreSQL community\u2019s bug reports (pgsql-bugs), describing this exact scenario.<\/span><\/p>

So why did this lockup occur seemingly out of nowhere? The answer lies in an innocent-looking version difference. For example, while your production server may be running version <\/span>16.8<\/b><\/span>, the backup server used for the restore test may be running version <\/span>16.14<\/b><\/span> because it was installed more recently. You might assume that a minor version difference is backward-compatible. However, a code change introduced to the <\/strong><\/span>SimpleLruWriteAll() function in PostgreSQL 16.12 triggers this hidden deadlock during WAL replay across minor versions.<\/strong><\/p>

If your WAL files suddenly stop being processed during a restore operation and the system freezes without any warning, you are not alone.<\/strong> Let\u2019s look at how to overcome this deadlock and bring the database back online.<\/span><\/p>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t

\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\"\"\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t\t

Searching for a Temporary Solution: Downgrading<\/h2>\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t

After identifying the issue, the first thing that might come to mind is to bring the PostgreSQL version on the test server to the same level as production, that is, to 16.8 (or to 16.11, where the bug does not yet exist). It is theoretically possible to downgrade only the binary files without touching the data directory. However, dealing with downgrading packages and trying to resolve operating system dependencies is the last thing you would want to do. You need a much cleaner and more permanent solution.<\/span><\/p>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t

\n\t\t\t\t\t

Rescuing Process: PostgreSQL 16.15<\/h2>\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t

Fortunately, the PostgreSQL community identified Bug #19490 and addressed the issue. Released on <\/span>August 13, 2026<\/b><\/span>, <\/span>PostgreSQL 16.15<\/b><\/span> provides a solution for systems affected by this problem. The release notes confirm that the deadlock involving SLRU locks during WAL replay has been resolved.<\/strong><\/span><\/p>

All you need to do is advance the server. Since this is a \u201cminor\u201d version update, you do not need to deal with database files or complex migration scenarios:<\/span><\/p>

  1. Stop the locked PostgreSQL service on the test server.<\/span><\/li>
  2. Use your operating system\u2019s package manager (apt, yum, etc.) to update the <\/span>postgresql-16<\/span> package to version <\/span>16.15<\/b><\/span>.<\/span><\/li>
  3. Restart the service.<\/span><\/li><\/ol>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t
    \n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\"\"\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t
    \n\t\t\t\t\t

    Breaking of the Silence and Happy Ending<\/h2>\t\t\t\t<\/div>\n\t\t\t\t
    \n\t\t\t\t\t\t\t\t\t

    After starting the PostgreSQL service, monitor the log file. Then the moment of resolution will arrive: gigabytes of WAL files that have remained untouched for hours will begin processing within seconds. <\/span>pg_stat_slru<\/span> will reset, and the wait events will return to normal. There will be no deadlock or stalling. Within just a few minutes, you will see the expected line in the log file:<\/span><\/p>

    LOG: database system is ready to accept connections<\/span><\/p>

    Your database will start up healthily, full data consistency will be ensured, and your stressfully started restore test will conclude successfully.<\/span><\/p>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t

    \n\t\t\t\t\t

    Lessons to be Learned (Best Practices)<\/h2>\t\t\t\t<\/div>\n\t\t\t\t
    \n\t\t\t\t\t\t\t\t\t

    These kinds of restore tests serve as a reminder of a few golden rules regarding database management:<\/span><\/p>

    1. Do Not Underestimate Minor Releases:<\/b> Do not assume that \u201c16.x and 16.y are essentially the same.\u201d<\/span><\/i> In the database world, small code changes can have major and unpredictable effects, especially in low-level core mechanisms such as WAL replay and SLRU.<\/span><\/li>
    2. Version Synchronization Is Essential:<\/b> Ensure that servers used for restore testing or maintained for Disaster Recovery (DR) run <\/span>the exact same minor version<\/b> as your production servers.<\/span><\/li>
    3. Stay Up to Date:<\/b> If a version difference is unavoidable or you are setting up a new server, always use the latest stable release announced by the PostgreSQL community\u2014in this case, version 16.15.<\/span><\/li><\/ol>

      If your restore process freezes for no apparent reason and the logs provide no clues, a silently waiting <\/span><\/span>MultiXactOffsetSLRU<\/span> lock<\/span> may be lurking in the background. The solution is within reach: <\/span>update your server to version 16.15<\/b><\/span>, then sit back and watch the logs flow.<\/span><\/p>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<\/div>","protected":false},"excerpt":{"rendered":"

      Explore the cause of the silent lockup during the PostgreSQL restore process, Bug #19490, and the solution introduced in PostgreSQL 16.15, along with the technical details.<\/p>","protected":false},"author":16,"featured_media":41111,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[45,75,129],"tags":[60,50],"class_list":["post-41105","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-yazi","category-postgresql","category-veritabani","tag-postgresql","tag-veritabani"],"_links":{"self":[{"href":"https:\/\/profelis.com.tr\/en\/wp-json\/wp\/v2\/posts\/41105","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/profelis.com.tr\/en\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/profelis.com.tr\/en\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/profelis.com.tr\/en\/wp-json\/wp\/v2\/users\/16"}],"replies":[{"embeddable":true,"href":"https:\/\/profelis.com.tr\/en\/wp-json\/wp\/v2\/comments?post=41105"}],"version-history":[{"count":15,"href":"https:\/\/profelis.com.tr\/en\/wp-json\/wp\/v2\/posts\/41105\/revisions"}],"predecessor-version":[{"id":41127,"href":"https:\/\/profelis.com.tr\/en\/wp-json\/wp\/v2\/posts\/41105\/revisions\/41127"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/profelis.com.tr\/en\/wp-json\/wp\/v2\/media\/41111"}],"wp:attachment":[{"href":"https:\/\/profelis.com.tr\/en\/wp-json\/wp\/v2\/media?parent=41105"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/profelis.com.tr\/en\/wp-json\/wp\/v2\/categories?post=41105"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/profelis.com.tr\/en\/wp-json\/wp\/v2\/tags?post=41105"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}