Always On setup¶

The Profelis installer sets up an active-active cluster, where two or more mail servers work on the same data, together with the Always On module: firewall, ZooKeeper, database (Galera), directory replication, shared message store (Ceph, Garage, enterprise S3, or NFS in a lab), search index (OpenSearch) and the Always On settings.

Lab first

Always On changes the database, file store and directory layout. Do the first installation in a test environment before production and take snapshots of the servers.

Prerequisites¶

All mail servers run Zimbra PFE 10 and use the same Zimbra directory (LDAP): the second server is installed as an additional server pointing to the first server's LDAP (mailbox, MTA, proxy). The installer checks this and stops with instructions if not.
The Zimbra patch package (zmpatch) including 604 must be applied. Without it the installer does not enable Always On.
The servers must reach each other on: 2181/2888/3888 (ZooKeeper), 9200/9300 (OpenSearch), 3306/4567/4568/4444 (Galera), 7306 (lab database), 2049/111/20048 (NFS), 3300/6789/6800-7000/7480 (Ceph), 3900/3901/3903 (Garage). The installer adds rules that open these ports only to cluster members; Zimbra's internal ports (11211, 7025 and others, LDAP 389/636) and the admin port 7071 are opened only to cluster members and to the [firewall] admin_cidr range; the proxy front end (443/80/993/995/143/110/587/25/465) and SSH stay open.

One-command installation¶

On the first server (database, NFS, ZooKeeper and OpenSearch run here):

sudo ./install.sh --yes --modules alwayson --alwayson-nodes mail1.zpe.profelis.net,mail2.zpe.profelis.net

The profile is written to /etc/profelis-alwayson/profile.conf and the installation continues.
Without --yes and without a profile, the installer asks "Set up the Always On cluster now?".
With a witness server add --alwayson-witness witness.zpe.profelis.net; for a shared IP add --alwayson-vip 192.0.2.10.

On the second server: the profile contains the shared database password; copy it unchanged and run the same installer:

sudo install -d -m 0700 /etc/profelis-alwayson
sudo scp mail1:/etc/profelis-alwayson/profile.conf /etc/profelis-alwayson/profile.conf
sudo ./install.sh --yes --modules alwayson

Order: witness (if any) → first server → other servers.

Verification¶

Each server ends with dogrulama: ok=… fail=0. Manually: scripts/alwayson/alwayson-install --phase verify.
profelis-alwayson-companion status shows live nodes and health checks (LMTP, HTTPS, shared store, index); drain must be no (hayir). Note: the install links the short command under /usr/local/sbin; on older installs use the full path /opt/profelis-alwayson/bin/profelis-alwayson-companion.
In the admin console, Tools and Migration › Profelis Always On: Live nodes equals the number of servers, ZooKeeper is OK and each node is active.

Always On screen on a two-node cluster: ZooKeeper OK, 2/2 live nodes

Full HA: setup with a witness¶

PFE 10 only

Profelis S3 and the full-HA message store (ceph, garage, s3) require Zimbra PFE 10. On Zimbra 9 the installer and alwayson-install refuse these stores with a clear message, and the S3 store extension is not built or shipped in the Zimbra 9 package; use nfs there.

A two-server setup without a witness is for labs: if the first server fails, the cluster stops. With a witness, losing any single machine (a mail server or the witness) stops neither reads nor writes:

Component	Mail servers	Witness
Database	MariaDB Galera (synchronous) + local MaxScale	Galera arbitrator (garbd)
ZooKeeper	ensemble member	ensemble member
Directory (LDAP)	multi-master (MMR)	—
Search (OpenSearch)	data node, each index has a copy on the other server	voting node
Message store	Ceph RGW / Garage node, local endpoint	Ceph / Garage node (holds a full copy)

Message store options (--alwayson-blob):

ceph (default): Ceph (cephadm, Tentacle). Every message is written synchronously as 3 copies (size=3, min_size=2); large enterprise setups, scale-out, erasure coding (EC) option.
garage: Garage v2, one zone per machine, 3 full copies; simpler operations, small and medium setups.
s3: existing enterprise S3 (Ceph RGW, NetApp StorageGRID, Dell ECS…); no storage component is installed and the witness can stay small.
nfs: lab only; the NFS server is a single point of failure.

MinIO is not supported: the community edition is archived and receives no security updates.

MaxScale licence

The database proxy MaxScale is licensed under the BSL; it is free to use with fewer than 3 database servers (in this design: 2 mail servers). If you plan to run 3 or more database servers, check the MariaDB licence first.

The witness is a storage member

With Ceph and Garage the witness keeps a copy of the data; its disk must be the same size as the mail servers'. A small witness is only possible with s3.

No data loss

A message is not considered delivered before the S3 store confirms the write (LMTP 250 is returned only after the S3 write succeeded; on failure the MTA retries). Galera and Ceph replicate synchronously; there is no asynchronous replication anywhere.

Order: witness → first server → other servers.

On the first server generate the profile and install:
```
sudo ./install.sh --yes --modules alwayson \
  --alwayson-nodes mail1.zpe.profelis.net,mail2.zpe.profelis.net \
  --alwayson-witness witness.zpe.profelis.net --alwayson-witness-ip 192.0.2.30 \
  --alwayson-blob ceph --alwayson-set blob.devices=/dev/sdb,/dev/sdc \
  --alwayson-set firewall.admin_cidr=203.0.113.0/24
```
Instead of flags, sudo ./install.sh asks step by step (role, nodes, witness, store, memory, modules), shows a summary for confirmation and runs the installation to the end. The guided mode also asks for the IP addresses or CIDR ranges allowed to reach the admin console (7071); leaving it empty opens it to the cluster IPs only (a warning is shown). Non-interactive: --alwayson-admin-cidr 203.0.113.0/24 or --alwayson-set firewall.admin_cidr=....

Guided installation: menu, role selection and summary (the dialog is in Turkish) (1/3)

Guided installation: menu, role selection and summary (the dialog is in Turkish) (2/3)

Guided installation: menu, role selection and summary (the dialog is in Turkish) (3/3)
The profile (/etc/profelis-alwayson/profile.conf) carries the cluster's shared passwords and keys: readable by root only (chmod 600). Copy it unchanged to the witness and the other servers.

On the witness (no Zimbra needed; an unrelated Zimbra on the witness is left untouched):

sudo ./install.sh          # role: witness  (or: scripts/alwayson/alwayson-install --yes)

Continue the installation on the first server, then on the other servers run sudo ./install.sh --yes --modules alwayson.
On the first server rebuild the search index and verify: scripts/alwayson/alwayson-install --phase reindex,verify.

Each step prints [i/N] faz: …; completed steps are skipped as "zaten kuruldu" (already installed). If a step fails, the installer prints the command to resume (--phase …).

LDAP multi-master prerequisites¶

In the ldap-mmr phase the installer adds zimbra-ldap to the other mail servers and enables multi-master replication (zmldapenable-mmr). Before you start:

These seven local config keys must be set on the first server: ldap_root_password, ldap_replication_password, zimbra_ldap_password, ldap_postfix_password, ldap_amavis_password, ldap_nginx_password, ldap_bes_searcher_password. If the profile has an [ldap] section, the values are written from it into the local config; without that section the installer stops on any empty key (except ldap_bes_searcher_password).
The LDAP certificate chain, including a self-signed root, must be in /opt/zimbra/conf/ca on every server (replication over TLS is verified against this directory). With a public certificate authority, add its root certificate there and create the hash-named link (c_rehash); the chain must be identical on both servers.
When replication has finished, contextCSN is identical on both servers and the entry counts match; the installer does not change the ldap_url order before checking this.

Moving a lab cluster to full HA¶

On the first server of an existing two-server cluster:

sudo ./install.sh --yes --modules alwayson --alwayson-upgrade-fullha witness.zpe.profelis.net \
  --alwayson-witness-ip 192.0.2.30 --alwayson-blob ceph

The profile is updated (the old one is backed up). Copy the profile to the witness and the other server; install the witness; on the other server run su - zimbra -c 'zmmailboxdctl stop' (no writes while the database moves); then run the installation on the first server and then on the other server. During the move:

ZooKeeper goes from one node to three members, OpenSearch from one node to a cluster (the index is rebuilt: --phase reindex).
The database moves from Zimbra's own MariaDB to Galera after a backup (/var/lib/profelis-alwayson/zimbra-db-before-galera.sql.gz).
Existing message files stay readable; to move them online into the S3 store run --phase blob-migrate on the first server (rate-limited, resumable).

Sizing (~20 TB of mail)¶

Store	Protection	Raw / logical	Raw for 20 TB (with headroom)	When one machine is lost
Ceph, replicated `size=3` (default)	3 synchronous copies	3×	~90-100 TB	reads + writes continue
Ceph, EC `k=4 m=2` (at least 6 servers)	erasure coding	1.5×	~45-50 TB	reads + writes continue
Ceph, EC `k=2 m=1` (3 servers)	erasure coding	1.5×	~45-50 TB	read only: not suitable for HA
Garage `replication_factor=3`	3 full copies	3×	~90-100 TB	reads + writes continue
Enterprise S3	vendor	vendor	vendor	vendor

Recommended at this scale: an existing enterprise S3, or Ceph with at least 5 storage servers (6 × 3.84 TB NVMe/SSD each; ~38 TB usable with size=3), RGW index pool on a separate SSD class.
Mail blobs are small objects: the default is a replicated pool; EC splits every object and adds latency, recommended only for an archive tier.
Network: client network 10/25 GbE, separate cluster network 25 GbE (blob.ceph_cluster_network).
Migration time (75 KB average message, 16 threads): ~6-7 hours per TB; 20 TB in ~5-6 days, online and rate-limited. Measure with a single mailbox first.
Measured on the lab: blob-migrate moved 5.4 MB in 481 items in about 20 s. The lab is too small to give a reliable throughput figure; measure with a real mailbox first. Reindexing 28 accounts took about 11 min.
Erasure-coded (EC) pools add overhead for small objects; the RGW index pool must always stay replicated and on SSD. The witness disk must be the same size as the mail servers' disks (see the warning above).

Memory (RAM) plan¶

Approximate values measured on the idle lab cluster (per service, per machine); they grow with load in production:

Service	Memory
OpenSearch	heap 1 GB (`index.opensearch_heap`); outside the lab size it by memory and data volume
Ceph MGR	190-255 MB (RSS)
Ceph MON	180-200 MB (RSS)
Ceph RGW	~160 MB (RSS)
Ceph OSD	150-170 MB (RSS) on the idle lab; the default target is 4 GB per OSD, cap it on a small machine
MaxScale	~18 MB (RSS)
Garbd (Galera arbitrator)	~9 MB (RSS)
Galera (MariaDB)	as set by `db.innodb_buffer_pool`

Keep the OpenSearch data disk below 85 % full: above that the replicas cannot be allocated, and losing a node then turns the index red. --phase verify warns about this.

Add the services above to Zimbra's own requirements on each mail server; the witness runs only the voting/arbitrator services (plus Ceph/Garage when it is a storage member).

Failover behaviour¶

Measured on the lab cluster by isolating a machine from the network (the virtual machines were not powered off). In every scenario no mail was lost.

Lost machine	Sign-ins	Mail delivery	Recovery
Second server	95 % of sign-ins succeeded through the first server; about 30 s stall while Galera re-forms	30/30 delivered	Galera rejoin about 5 s; Ceph `HEALTH_OK` in about 4 min; Always On re-registers in about 50 s
First server (after the proxy fix)	216/216 sign-ins through the second server (about 3.3 s with a header token, about 0.4 s with a cookie)	20/20 delivered (the slowest about 140 s late through Postfix deferral)	MaxScale promotes the second node to master and keeps it (stickiness); Ceph `HEALTH_OK` in about 6 min
Witness	91-93 % of sign-ins	118/118 delivered	ZooKeeper leader moves; rejoin about 2 min
Directory (slapd) only, one server	about 98 %	60/60	automatic
RGW only, one server	about 97 %	72/72	the S3 store fails over to the other endpoint within the same PUT

Notes:

Before the proxy fix, when the first server was lost only about 1 % of sign-ins succeeded through the second server. The fix sets short upstream timeouts on the proxy (zimbraReverseProxyUpstreamConnectTimeout 3 s and related attributes); the installer applies them.
Stopping a single Ceph daemon with ceph orch daemon stop needs --force (the orchestrator otherwise refuses because it would reduce availability).
In the lab an RGW failure was detected in 8.6 s and recovered in 10.4 s.

Security hardening¶

The firewall phase restricts Zimbra's internal ports (11211, 7025/7026, 7072-7074, 7110/7143/7993/7995, 7780, 8443, 389/636) to the cluster addresses. Port 7071 (administration) is open only to the cluster and to [firewall] admin_cidr.
The swatch log-injection remote code execution hardening runs before and after zmsetup. Run install.sh --pre-zmsetup between the Zimbra packages and zmsetup; the installation fails if the system is still unpatched.
The Profelis Guard baseline is refreshed after the installation.

For operations see Always On.

Scope

Existing local message files are not moved to the shared store automatically (see Storage tiers).
The search index moves to OpenSearch; reindex existing accounts: zmprov rim <account> start.
When testing searches with non-ASCII letters from the command line, use a UTF-8 locale for the zimbra user (LC_ALL=C.UTF-8 zmmailbox …); webmail is not affected.