Skip to content

Always On (active-active cluster)

Several mail servers work on the same data; if one server fails, the proxy sends clients to a live server, with no account move or takeover.

Global administrator and rootAdmin console › Tools and Migration › Profelis Always On

Setup: Always On setup.

Cluster status

  1. Open Tools and Migration › Profelis Always On. The top bar shows the cluster ID, ZooKeeper status, Live nodes x / y, this server, routing, Sync (green all in sync when all nodes agree, red N issue(s) otherwise) and the Storage backend (Ceph, Garage, S3 or NFS). The node list below has Node, ZooKeeper, LDAP, MaxScale, Storage, This server and Server ID (first 8 characters; hover for the full ID). If the server has not joined a cluster, a red warning appears and the list stays empty.
    • LDAP: In sync (the multi-master servers agree; up to 60 seconds of difference is normal), Lagging (this server is not receiving the other's changes) or Down.
    • MaxScale: the database server's role in MaxScale: Master, Slave, Synced, Donor, Joining, Maintenance or Down. Hover for the Galera state and cluster size.
    • Storage: OK when the node's S3 endpoint (local RGW/Garage) answers, otherwise Degraded or Down. With NFS only this server's shared directory is checked.
    • The checks refresh every 10 seconds; a check that does not answer shows ? in the cell and the node list keeps working.
  2. With Auto Refresh on, a server that stops shows as down within 10 seconds.
  3. Flush Caches clears the cache cluster-wide (see below).

Profelis Always On: a two-node cluster, 2/2 live nodes

Profelis Always On: a two-node cluster, 2/2 live nodes

Cluster-wide cache flush

Flush Caches clears the cache on all nodes (for example after a COS change). zmprov flushCache on a single server does not affect the other nodes.

Taking a node out for maintenance

As root (the install links profelis-alwayson-companion under /usr/local/sbin; on older installs use the full path /opt/profelis-alwayson/bin/profelis-alwayson-companion):

profelis-alwayson-companion drain --reason maintenance
profelis-alwayson-companion status
profelis-alwayson-companion undrain

While drained, new sessions and mail delivery go to the other nodes; existing sessions are not cut. The helper monitors health: after three consecutive failures it drains the node itself and brings it back when it recovers.

Rolling restart

When a setting that needs a mailboxd restart changes, do not restart the nodes by hand one after another. On any node, as root:

profelis-alwayson-companion rolling-restart

The nodes are drained one at a time, running requests get 30 seconds, mailboxd is restarted, the node is brought back once its health checks pass, then the next node follows. The cluster keeps serving throughout. For the current node only: profelis-alwayson-companion restart-self.

  • The command does not start if fewer than two nodes are live; only one restart runs in the cluster at a time.
  • If a node does not come back healthy, it stays drained and the run stops without touching the other nodes. Fix the cause, then profelis-alwayson-companion undrain.
  • To see the plan first: rolling-restart --dry-run.

The info line on the Profelis Always On screen points to this command as well.

Health checks on a full-HA cluster

On a cluster with a witness (setup), as root on each server:

scripts/alwayson/alwayson-install --phase verify

The check covers Galera (2 servers + arbitrator = 3), ZooKeeper (leader/follower), OpenSearch (3 nodes), LDAP replication, the message store (Ceph HEALTH_OK/HEALTH_WARN or Garage) and S3 bucket access. Ceph status: ceph --id profelis -s; S3 access: zmjava com.profelis.store.s3.S3Check (zimbra user, EXT_JAR_PATH=/opt/zimbra/lib/ext/com_profelis_s3store/com_profelis_s3store.jar).

When a server goes down, no action is needed: the remaining server keeps reading and writing (Ceph shows HEALTH_WARN). When the server returns, the database, directory, search index and message store resynchronise by themselves.

Directory (LDAP) replication check

On each mail server, as the zimbra user:

/opt/zimbra/libexec/zmreplchk
/opt/zimbra/libexec/zmldapmmrtool -q

Expected: contextCSN identical on both servers and status "In Sync". ldap_url lists both servers (the local one first); if one fails, mailboxd falls over to the other. After ldap_url changes, mailboxd needs a restart (see rolling restart).

LDAP password rotation

The seven LDAP passwords (ldap_root_password, ldap_replication_password, zimbra_ldap_password, ldap_postfix_password, ldap_amavis_password, ldap_nginx_password, ldap_bes_searcher_password) are rotated across the cluster with the manual phase:

alwayson-install --yes --phase ldap-passwd

The phase is aware of multi-master replication. zmldappasswd -l alone does not update the replication consumer credentials in the MMR configuration; the phase also fixes them (olcDatabase={3}mdb), so replication does not stop with Invalid credentials (49). Check with zmreplchk afterwards.

The MaxScale administrator account is hardened with alwayson-install --yes --phase maxscale-admin.

The Always On administration screen (1.0.7) shows per-node LDAP, MaxScale and Storage sync columns and a "Sync" line, so a node that has fallen out of step is visible at a glance.

Message store (S3) failover

Profelis S3 requires Zimbra PFE 10 (it is not available on Zimbra 9).

On every call the S3 store tries the local endpoint first and the other server's endpoint if that fails (profelis_s3_endpoints). If one RGW goes down, reads and writes move to the other without interruption; LMTP 250 is returned only after the write is confirmed, and if it cannot be written the MTA retries. No new message is lost; only the single request in flight at that moment may be delayed.

Failover expectations (single machine lost):

Event Expected Measured
Mail server down Drops from the live node list; sign-in and delivery continue on the other server Second server: 95 % of sign-ins, mail 30/30; Galera rejoin about 5 s, Ceph HEALTH_OK about 4 min
First server down Same; database writes continue on the other node through MaxScale 216/216 sign-ins, mail 20/20; Ceph HEALTH_OK about 6 min
Witness down No impact; Galera/ZooKeeper quorum 2/3 91-93 % of sign-ins, mail 118/118; ZooKeeper leader moves, rejoin about 2 min
RGW only stopped S3 requests move to the other endpoint about 97 % of sign-ins, 72/72; the S3 store fails over within the same PUT

No mail was lost in any scenario. Full table: Failover behaviour.

Known gap

The companion does not monitor S3/RGW health; the blob_mount check is meaningless with Ceph. Monitor Ceph/RGW separately with alwayson-install --phase verify and ceph --id profelis -s.

Scope

There are never two writers on the same mailbox at once; write access is serialised with a distributed lock. The health of the shared storage and database cluster must be monitored separately.